2024-06-11T18:58:34.046 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-11T18:58:34.046 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861 --description rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} --name lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi --owner scheduled_lflores@teuthology --verbose -- /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/orig.config.yaml 2024-06-11T18:58:34.083 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-11T18:58:34.167 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861 branch: wip-lflores-testing-2-2024-06-10-1925 description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} email: null first_in_suite: false job_id: '7749861' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lflores-testing-2-2024-06-10-1925 ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request sha1: 37bbe1d8d788b7c9a71280e7d510c295a5853f0b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 37bbe1d8d788b7c9a71280e7d510c295a5853f0b selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lflores-testing-2-2024-06-10-1925 sha1: 37bbe1d8d788b7c9a71280e7d510c295a5853f0b owner: scheduled_lflores@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 7590 sha1: 37bbe1d8d788b7c9a71280e7d510c295a5853f0b sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-lflores-testing-2-2024-06-10-1925 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 37bbe1d8d788b7c9a71280e7d510c295a5853f0b targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK0TZDceOPUGWUQRpUOa8Qwzy83q1SX5RxcIfpi+gjo81nXvkXCMRXobXA0qXn6sNtxm9amEOuS9vjHnFc4vFXg= smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMXyP0QtX3p4dtUVOlrQCBBXLrLpo7du/6MR/BSjYpo0W7Qe1FnWlTVl1GM8ZQkMr1YCNDw+IlhiE3b7xhPp+K0= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-06-10_22:21:08 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-11T18:58:34.167 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa; will attempt to use it 2024-06-11T18:58:34.168 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa/tasks 2024-06-11T18:58:34.168 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-11T18:58:34.169 INFO:teuthology.task.internal:Checking packages... 2024-06-11T18:58:34.190 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '37bbe1d8d788b7c9a71280e7d510c295a5853f0b' 2024-06-11T18:58:34.191 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-11T18:58:34.191 INFO:teuthology.packaging:ref: None 2024-06-11T18:58:34.191 INFO:teuthology.packaging:tag: None 2024-06-11T18:58:34.191 INFO:teuthology.packaging:branch: wip-lflores-testing-2-2024-06-10-1925 2024-06-11T18:58:34.191 INFO:teuthology.packaging:sha1: 37bbe1d8d788b7c9a71280e7d510c295a5853f0b 2024-06-11T18:58:34.191 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lflores-testing-2-2024-06-10-1925 2024-06-11T18:58:34.337 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-4216.g37bbe1d8 2024-06-11T18:58:34.339 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-11T18:58:34.346 INFO:teuthology.task.internal:no buildpackages task found 2024-06-11T18:58:34.346 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-11T18:58:34.352 INFO:teuthology.task.internal:Saving configuration 2024-06-11T18:58:34.367 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-11T18:58:34.373 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-11T18:58:34.392 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-11 18:53:36.620542', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK0TZDceOPUGWUQRpUOa8Qwzy83q1SX5RxcIfpi+gjo81nXvkXCMRXobXA0qXn6sNtxm9amEOuS9vjHnFc4vFXg='} 2024-06-11T18:58:34.412 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-11 18:53:36.619165', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMXyP0QtX3p4dtUVOlrQCBBXLrLpo7du/6MR/BSjYpo0W7Qe1FnWlTVl1GM8ZQkMr1YCNDw+IlhiE3b7xhPp+K0='} 2024-06-11T18:58:34.412 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-11T18:58:34.417 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2024-06-11T18:58:34.417 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-06-11T18:58:34.417 INFO:teuthology.run_tasks:Running task console_log... 2024-06-11T18:58:34.466 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa455b5fbe0>, signals=[15]) 2024-06-11T18:58:34.466 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-11T18:58:34.470 INFO:teuthology.task.internal:Opening connections... 2024-06-11T18:58:34.471 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-06-11T18:58:34.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T18:58:34.547 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-06-11T18:58:34.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T18:58:34.619 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-11T18:58:34.625 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-06-11T18:58:34.642 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-06-11T18:58:34.643 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-06-11T18:58:34.700 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-06-11T18:58:34.700 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-06-11T18:58:34.700 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-06-11T18:58:34.700 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-06-11T18:58:34.700 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-06-11T18:58:34.700 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-06-11T18:58:34.700 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-11T18:58:34.700 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-06-11T18:58:34.700 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-06-11T18:58:34.700 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-11T18:58:34.700 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-06-11T18:58:34.701 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-11T18:58:34.701 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-11T18:58:34.701 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-11T18:58:34.701 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-06-11T18:58:34.725 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-06-11T18:58:34.742 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-06-11T18:58:34.742 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-06-11T18:58:34.800 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-06-11T18:58:34.800 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-06-11T18:58:34.800 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-06-11T18:58:34.800 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-06-11T18:58:34.800 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-06-11T18:58:34.800 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-06-11T18:58:34.800 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-11T18:58:34.800 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-06-11T18:58:34.800 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-06-11T18:58:34.800 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-11T18:58:34.801 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-06-11T18:58:34.801 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-11T18:58:34.801 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-11T18:58:34.801 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-11T18:58:34.801 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-06-11T18:58:34.823 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-11T18:58:34.831 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-11T18:58:34.836 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-11T18:58:34.836 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-06-11T18:58:34.838 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-06-11T18:58:34.856 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-11T18:58:34.861 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-11T18:58:34.861 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-06-11T18:58:34.895 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-06-11T18:58:34.989 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-11T18:58:35.040 INFO:teuthology.run_tasks:Running task kernel... 2024-06-11T18:58:35.054 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-11T18:58:35.054 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.f': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-11T18:58:35.054 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-11T18:58:35.055 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-11T18:58:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T18:58:35.071 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-06-11T18:58:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T18:58:35.072 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-06-11T18:58:35.129 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-437.el9.x86_64 2024-06-11T18:58:35.129 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-437.el9.x86_64 2024-06-11T18:58:35.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-06-11T18:58:35.130 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-437.el9.x86_64 2024-06-11T18:58:35.130 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.14.0-437.el9.x86_64 2024-06-11T18:58:35.131 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum install -y kernel 2024-06-11T18:58:38.356 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.1 MB 00:01 2024-06-11T18:58:42.840 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 1.3 MB/s | 8.1 MB 00:06 2024-06-11T18:58:46.662 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 20 MB 00:07 2024-06-11T18:58:49.800 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 3.3 MB/s | 20 MB 00:05 2024-06-11T18:58:52.033 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.3 MB 00:01 2024-06-11T18:58:53.606 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 17 kB 00:00 2024-06-11T18:58:54.884 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-06-11T18:58:56.204 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.3 MB 00:02 2024-06-11T18:58:58.715 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 17 kB 00:01 2024-06-11T18:58:59.934 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-06-11T18:58:59.997 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-06-11T18:59:01.999 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-11T18:59:01.999 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-11T18:59:02.093 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T18:59:02.099 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T18:59:02.099 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-06-11T18:59:02.099 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T18:59:02.099 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T18:59:02.099 INFO:teuthology.orchestra.run.smithi117.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-11T18:59:02.100 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-06-11T18:59:02.100 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-11T18:59:02.100 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-11T18:59:02.100 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-11T18:59:02.100 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T18:59:02.100 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T18:59:02.100 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T18:59:02.100 INFO:teuthology.orchestra.run.smithi117.stdout:Install 4 Packages 2024-06-11T18:59:02.100 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T18:59:02.101 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 81 M 2024-06-11T18:59:02.101 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 124 M 2024-06-11T18:59:02.101 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T18:59:02.242 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T18:59:02.243 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T18:59:02.247 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T18:59:02.283 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:02.284 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:02.287 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:02.399 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:02.400 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:02.418 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:03.181 INFO:teuthology.orchestra.run.smithi117.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 970 kB/s | 989 kB 00:01 2024-06-11T18:59:03.202 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:03.203 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T18:59:03.865 INFO:teuthology.orchestra.run.smithi117.stdout:(2/4): kernel-core-5.14.0-457.el9.x86_64.rpm 9.4 MB/s | 16 MB 00:01 2024-06-11T18:59:04.515 INFO:teuthology.orchestra.run.smithi117.stdout:(3/4): kernel-modules-core-5.14.0-457.el9.x86_6 22 MB/s | 29 MB 00:01 2024-06-11T18:59:04.815 INFO:teuthology.orchestra.run.smithi117.stdout:(4/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 13 MB/s | 35 MB 00:02 2024-06-11T18:59:04.823 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T18:59:04.823 INFO:teuthology.orchestra.run.smithi117.stdout:Total 30 MB/s | 81 MB 00:02 2024-06-11T18:59:05.207 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-06-11T18:59:05.535 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T18:59:05.878 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T18:59:05.878 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T18:59:06.839 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T18:59:06.840 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T18:59:07.238 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-11T18:59:07.238 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-11T18:59:07.355 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T18:59:07.366 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T18:59:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-06-11T18:59:07.366 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T18:59:07.366 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T18:59:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-11T18:59:07.367 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-06-11T18:59:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-11T18:59:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-11T18:59:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-11T18:59:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T18:59:07.367 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T18:59:07.367 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T18:59:07.367 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-06-11T18:59:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T18:59:07.367 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 81 M 2024-06-11T18:59:07.367 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 124 M 2024-06-11T18:59:07.367 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T18:59:07.482 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T18:59:07.482 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T18:59:07.486 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T18:59:07.960 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 104.251.122.5) 2024-06-11T18:59:07.985 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 104.251.122.5) 2024-06-11T18:59:08.001 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 104.251.122.5) 2024-06-11T18:59:08.280 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 104.251.122.5) 2024-06-11T18:59:08.304 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 104.251.122.5) 2024-06-11T18:59:08.331 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 199.223.246.113) 2024-06-11T18:59:08.336 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 104.251.122.5) 2024-06-11T18:59:08.337 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 199.223.246.113) 2024-06-11T18:59:08.353 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 199.223.246.113) 2024-06-11T18:59:08.390 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 163.237.219.7) 2024-06-11T18:59:08.394 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 163.237.219.7) 2024-06-11T18:59:08.410 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 163.237.219.7) 2024-06-11T18:59:08.520 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 163.237.219.7) 2024-06-11T18:59:08.527 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 163.237.219.7) 2024-06-11T18:59:08.551 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 216.158.133.254) 2024-06-11T18:59:08.551 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 163.237.219.7) 2024-06-11T18:59:08.554 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 216.158.133.254) 2024-06-11T18:59:08.565 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 216.158.133.254) 2024-06-11T18:59:08.653 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 216.158.133.254) 2024-06-11T18:59:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 216.158.133.254) 2024-06-11T18:59:08.680 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 216.158.133.254) 2024-06-11T18:59:08.693 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:08.706 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:08.763 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:08.784 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:08.784 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:08.804 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:08.819 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 64.50.233.100) 2024-06-11T18:59:08.819 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 64.50.233.100) 2024-06-11T18:59:08.885 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 64.50.233.100) 2024-06-11T18:59:08.928 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 64.50.233.100) 2024-06-11T18:59:08.946 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 64.50.233.100) 2024-06-11T18:59:08.981 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 64.50.233.100) 2024-06-11T18:59:09.087 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 140.211.166.134) 2024-06-11T18:59:09.105 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 140.211.166.134) 2024-06-11T18:59:09.140 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 140.211.166.134) 2024-06-11T18:59:09.286 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T18:59:09.435 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 140.211.166.134) 2024-06-11T18:59:09.457 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 140.211.166.134) 2024-06-11T18:59:09.488 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 140.211.166.134) 2024-06-11T18:59:09.498 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 64.50.236.52) 2024-06-11T18:59:09.519 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 64.50.236.52) 2024-06-11T18:59:09.580 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 64.50.236.52) 2024-06-11T18:59:09.667 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 64.50.236.52) 2024-06-11T18:59:09.699 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 64.50.236.52) 2024-06-11T18:59:09.728 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-11T18:59:09.730 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 64.50.236.52) 2024-06-11T18:59:09.758 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-11T18:59:09.784 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-11T18:59:09.872 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-11T18:59:09.876 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-11T18:59:09.902 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-11T18:59:09.918 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 206.82.17.213) 2024-06-11T18:59:09.920 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 206.82.17.213) 2024-06-11T18:59:09.926 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-11T18:59:09.991 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 206.82.17.213) 2024-06-11T18:59:10.014 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 206.82.17.213) 2024-06-11T18:59:10.015 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 206.82.17.213) 2024-06-11T18:59:10.015 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-11T18:59:10.036 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 206.82.17.213) 2024-06-11T18:59:10.072 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ord.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 166.78.229.131) 2024-06-11T18:59:10.073 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ord.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 166.78.229.131) 2024-06-11T18:59:10.090 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ord.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 166.78.229.131) 2024-06-11T18:59:10.199 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ord.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 166.78.229.131) 2024-06-11T18:59:10.205 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ord.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 166.78.229.131) 2024-06-11T18:59:10.258 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ord.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 166.78.229.131) 2024-06-11T18:59:10.293 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 72.4.120.219) 2024-06-11T18:59:10.293 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 72.4.120.219) 2024-06-11T18:59:10.311 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 72.4.120.219) 2024-06-11T18:59:10.311 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 72.4.120.219) 2024-06-11T18:59:10.311 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 72.4.120.219) 2024-06-11T18:59:10.329 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 72.4.120.219) 2024-06-11T18:59:11.274 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 255 kB/s | 989 kB 00:03 2024-06-11T18:59:11.825 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-457.el9.x86_64.rpm 3.6 MB/s | 16 MB 00:04 2024-06-11T18:59:11.904 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 104.251.122.5) 2024-06-11T18:59:11.940 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T18:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 199.223.246.113) 2024-06-11T18:59:12.558 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 6.7 MB/s | 35 MB 00:05 2024-06-11T18:59:12.561 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 104.251.122.5) 2024-06-11T18:59:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-11T18:59:12.679 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 163.237.219.7) 2024-06-11T18:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 163.237.219.7) 2024-06-11T18:59:12.813 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 216.158.133.254) 2024-06-11T18:59:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-11T18:59:12.840 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 216.158.133.254) 2024-06-11T18:59:12.929 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:12.970 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T18:59:13.089 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 64.50.233.100) 2024-06-11T18:59:13.124 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 64.50.233.100) 2024-06-11T18:59:13.484 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 140.211.166.134) 2024-06-11T18:59:13.643 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 140.211.166.134) 2024-06-11T18:59:13.823 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 64.50.236.52) 2024-06-11T18:59:13.886 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 64.50.236.52) 2024-06-11T18:59:14.008 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-11T18:59:14.071 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-11T18:59:14.178 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 206.82.17.213) 2024-06-11T18:59:14.242 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 206.82.17.213) 2024-06-11T18:59:14.361 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://ord.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 166.78.229.131) 2024-06-11T18:59:14.425 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://ord.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 166.78.229.131) 2024-06-11T18:59:14.443 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-457.el9.x86_64.rpm (IP: 72.4.120.219) 2024-06-11T18:59:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-457.el9.x86_6 5.8 MB/s | 29 MB 00:05 2024-06-11T18:59:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T18:59:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:Total 9.0 MB/s | 81 MB 00:08 2024-06-11T18:59:17.001 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T18:59:17.368 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T18:59:17.368 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T18:59:17.598 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-11T18:59:17.731 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-11T18:59:18.353 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T18:59:18.354 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T18:59:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T18:59:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-11T18:59:21.534 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-11T18:59:22.520 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-11T18:59:24.184 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-11T18:59:24.333 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-11T18:59:29.103 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-11T18:59:29.222 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-11T18:59:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-11T18:59:51.949 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-11T18:59:51.989 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-11T18:59:52.251 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-11T18:59:52.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-11T18:59:52.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-11T18:59:52.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-11T18:59:53.035 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-11T18:59:53.035 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T18:59:53.036 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T18:59:53.036 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-11T18:59:53.036 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-11T18:59:53.036 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-11T18:59:53.036 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-11T18:59:53.036 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T18:59:53.036 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T18:59:53.393 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-06-11T18:59:53.472 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-11T18:59:53.472 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-11T18:59:53.473 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-11T18:59:53.473 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-11T18:59:53.473 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-11T18:59:53.473 INFO:teuthology.task.kernel:Installing distro kernel on mon.f... 2024-06-11T18:59:53.473 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2024-06-11T18:59:53.473 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-06-11T18:59:53.526 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-11T18:59:53.526 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-11T18:59:53.527 DEBUG:teuthology.orchestra.run.smithi117:> sudo rpm -qi grub2-tools 2024-06-11T18:59:53.565 INFO:teuthology.orchestra.run.smithi117.stdout:Name : grub2-tools 2024-06-11T18:59:53.565 INFO:teuthology.orchestra.run.smithi117.stdout:Epoch : 1 2024-06-11T18:59:53.565 INFO:teuthology.orchestra.run.smithi117.stdout:Version : 2.06 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:Release : 68.el9 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: x86_64 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:Group : Unspecified 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:Size : 8264254 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:License : GPLv3+ 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:Packager : builder@centos.org 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:Vendor : CentOS 2024-06-11T18:59:53.566 INFO:teuthology.orchestra.run.smithi117.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-11T18:59:53.567 INFO:teuthology.orchestra.run.smithi117.stdout:Summary : Support tools for GRUB. 2024-06-11T18:59:53.567 INFO:teuthology.orchestra.run.smithi117.stdout:Description : 2024-06-11T18:59:53.567 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T18:59:53.567 INFO:teuthology.orchestra.run.smithi117.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-11T18:59:53.567 INFO:teuthology.orchestra.run.smithi117.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-11T18:59:53.567 INFO:teuthology.orchestra.run.smithi117.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-11T18:59:53.567 INFO:teuthology.orchestra.run.smithi117.stdout:hardware devices. 2024-06-11T18:59:53.567 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T18:59:53.567 INFO:teuthology.orchestra.run.smithi117.stdout:This subpackage provides tools for support of all platforms. 2024-06-11T18:59:53.568 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-11T18:59:53.568 INFO:teuthology.task.kernel:Updating grub on smithi117 to boot 5.14.0-457.el9.x86_64 2024-06-11T18:59:53.568 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-11T18:59:54.250 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-06-11T18:59:55.600 INFO:teuthology.orchestra.run.smithi117.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-11T18:59:55.641 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-06-11T18:59:55.643 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-06-11T18:59:55.658 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.a3oSzW07Ls 2024-06-11T18:59:55.658 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.a3oSzW07Ls 2024-06-11T18:59:55.724 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.a3oSzW07Ls 2024-06-11T18:59:55.856 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.a3oSzW07Ls is 6KB 2024-06-11T18:59:55.905 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.a3oSzW07Ls 2024-06-11T18:59:55.919 DEBUG:teuthology.orchestra.run.smithi117:> sudo /bin/ls /boot/loader/entries || true 2024-06-11T18:59:55.979 INFO:teuthology.orchestra.run.smithi117.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-11T18:59:55.979 INFO:teuthology.orchestra.run.smithi117.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-11T18:59:55.979 INFO:teuthology.orchestra.run.smithi117.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-11T18:59:55.980 INFO:teuthology.orchestra.run.smithi117.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-11T18:59:55.981 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-11T18:59:56.111 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-06-11T19:00:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-11T19:00:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-11T19:00:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-11T19:00:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-11T19:00:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-11T19:00:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-11T19:00:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-11T19:00:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:00:04.533 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:00:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-11T19:00:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-11T19:00:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-11T19:00:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-11T19:00:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:00:04.533 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:00:04.912 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-06-11T19:00:04.996 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-11T19:00:04.997 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-11T19:00:04.997 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-11T19:00:04.997 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-11T19:00:04.997 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-11T19:00:04.997 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-11T19:00:04.997 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-06-11T19:00:04.997 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-06-11T19:00:05.050 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-11T19:00:05.051 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-11T19:00:05.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-06-11T19:00:05.081 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-06-11T19:00:05.081 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-06-11T19:00:05.081 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-06-11T19:00:05.081 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2024-06-11T19:00:05.081 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-06-11T19:00:05.081 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-11T19:00:05.081 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-06-11T19:00:05.081 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2024-06-11T19:00:05.081 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-06-11T19:00:05.082 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-11T19:00:05.082 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-11T19:00:05.082 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-11T19:00:05.082 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-11T19:00:05.082 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-06-11T19:00:05.082 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-06-11T19:00:05.082 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-11T19:00:05.082 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-06-11T19:00:05.082 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-06-11T19:00:05.082 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:00:05.082 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-11T19:00:05.082 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-11T19:00:05.083 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-11T19:00:05.083 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-06-11T19:00:05.083 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:00:05.083 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-06-11T19:00:05.084 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-11T19:00:05.084 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-457.el9.x86_64 2024-06-11T19:00:05.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-11T19:00:05.644 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-06-11T19:00:07.066 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-11T19:00:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-06-11T19:00:07.096 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-06-11T19:00:07.111 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.thZukoNQLk 2024-06-11T19:00:07.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.thZukoNQLk 2024-06-11T19:00:07.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.thZukoNQLk 2024-06-11T19:00:07.310 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.thZukoNQLk is 6KB 2024-06-11T19:00:07.358 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.thZukoNQLk 2024-06-11T19:00:07.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-06-11T19:00:07.437 INFO:teuthology.orchestra.run.smithi098.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-11T19:00:07.437 INFO:teuthology.orchestra.run.smithi098.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-11T19:00:07.437 INFO:teuthology.orchestra.run.smithi098.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-11T19:00:07.437 INFO:teuthology.orchestra.run.smithi098.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-11T19:00:07.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-11T19:00:07.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-06-11T19:00:26.133 INFO:teuthology.task.kernel:Checking client mon.f for new kernel version... 2024-06-11T19:00:26.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-06-11T19:00:26.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T19:00:37.612 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-11T19:00:37.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-06-11T19:00:37.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T19:01:00.831 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-06-11T19:01:09.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-06-11T19:01:09.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T19:01:12.098 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-06-11T19:01:21.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-06-11T19:01:21.108 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T19:01:28.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-06-11T19:01:39.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-06-11T19:01:40.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-06-11T19:01:40.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T19:01:40.523 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-06-11T19:01:40.882 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-06-11T19:01:40.882 INFO:teuthology.task.kernel:Checking kernel version of mon.f, want "5.14.0-457.el9.x86_64"... 2024-06-11T19:01:40.883 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-06-11T19:01:40.897 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-457.el9.x86_64 2024-06-11T19:01:40.897 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-11T19:01:40.897 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-11T19:01:40.898 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-11T19:01:41.898 INFO:teuthology.task.kernel:Enabling kdb on mon.f... 2024-06-11T19:01:41.899 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-11T19:01:41.968 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2024-06-11T19:01:42.012 DEBUG:teuthology.parallel:result is None 2024-06-11T19:01:51.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-06-11T19:01:51.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T19:01:51.825 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-06-11T19:01:52.363 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-06-11T19:01:52.363 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-457.el9.x86_64"... 2024-06-11T19:01:52.363 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-06-11T19:01:52.378 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-457.el9.x86_64 2024-06-11T19:01:52.378 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-11T19:01:52.378 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-11T19:01:52.378 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-11T19:01:53.379 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-11T19:01:53.379 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-11T19:01:53.502 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-06-11T19:01:53.533 DEBUG:teuthology.parallel:result is None 2024-06-11T19:01:53.534 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-11T19:01:53.541 INFO:teuthology.task.internal:Creating test directory... 2024-06-11T19:01:53.541 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-11T19:01:53.543 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-11T19:01:53.563 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-11T19:01:53.569 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-11T19:01:53.578 INFO:teuthology.task.internal:Creating archive directory... 2024-06-11T19:01:53.578 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-11T19:01:53.598 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-11T19:01:53.691 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-11T19:01:53.697 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-11T19:01:53.698 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-11T19:01:53.700 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-11T19:01:53.728 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T19:01:53.730 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T19:01:53.739 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T19:01:53.742 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-11T19:01:53.743 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-11T19:01:53.750 INFO:teuthology.task.internal:Configuring sudo... 2024-06-11T19:01:53.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-11T19:01:53.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-11T19:01:53.811 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-11T19:01:53.819 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-11T19:01:53.819 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-11T19:01:53.849 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-11T19:01:53.868 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-11T19:01:53.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-11T19:01:53.993 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-11T19:01:54.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-11T19:01:54.114 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:01:54.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-11T19:01:54.181 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-11T19:01:54.232 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-11T19:01:54.278 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-11T19:01:54.361 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-11T19:01:54.404 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:01:54.404 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-11T19:01:54.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-06-11T19:01:54.476 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-06-11T19:01:54.514 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-11T19:01:54.578 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-11T19:01:55.070 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-11T19:01:55.079 INFO:teuthology.task.internal:Starting timer... 2024-06-11T19:01:55.079 INFO:teuthology.run_tasks:Running task pcp... 2024-06-11T19:01:55.096 INFO:teuthology.run_tasks:Running task selinux... 2024-06-11T19:01:55.104 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-06-11T19:01:55.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-06-11T19:01:55.185 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-06-11T19:01:55.187 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-06-11T19:01:55.262 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-06-11T19:01:55.264 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-11T19:01:55.264 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-06-11T19:01:55.288 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-06-11T19:01:55.289 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-06-11T19:01:55.309 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-06-11T19:01:55.309 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi117.front.sepia.ceph.com': 'permissive'} 2024-06-11T19:01:55.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-11T19:01:55.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T19:01:55.353 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-11T19:01:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T19:01:55.381 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-11T19:01:55.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-06-11T19:01:55.419 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-06-11T19:01:55.448 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-11T19:01:55.457 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-11T19:01:55.460 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-11T19:01:55.600 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-11T19:01:55.611 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-11T19:01:55.612 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi117.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-11T19:06:17.695 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi117.front.sepia.ceph.com')] 2024-06-11T19:06:17.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-06-11T19:06:17.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T19:06:17.774 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-06-11T19:06:17.811 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-06-11T19:06:17.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-06-11T19:06:17.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-11T19:06:17.885 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-06-11T19:06:17.969 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-06-11T19:06:17.969 INFO:teuthology.run_tasks:Running task clock... 2024-06-11T19:06:17.981 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-11T19:06:17.986 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-11T19:06:17.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-11T19:06:17.989 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-11T19:06:17.989 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-11T19:06:18.015 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-11T19:06:18.030 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-11T19:06:18.038 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-11T19:06:18.052 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-11T19:06:18.053 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-06-11T19:06:18.065 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-06-11T19:06:18.076 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-06-11T19:06:18.079 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-11T19:06:18.087 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-06-11T19:06:18.093 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-11T19:06:18.101 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-11T19:06:18.115 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-11T19:06:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-06-11T19:06:18.153 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-11T19:06:18.154 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-06-11T19:06:18.154 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-11T19:06:18.154 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-11T19:06:18.154 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-11T19:06:18.154 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-11T19:06:18.168 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-06-11T19:06:18.172 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-11T19:06:18.172 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-06-11T19:06:18.172 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-11T19:06:18.172 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-11T19:06:18.172 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-11T19:06:18.173 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-11T19:06:18.173 INFO:teuthology.run_tasks:Running task install... 2024-06-11T19:06:18.183 DEBUG:teuthology.task.install:project ceph 2024-06-11T19:06:18.183 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '37bbe1d8d788b7c9a71280e7d510c295a5853f0b'}} 2024-06-11T19:06:18.183 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '37bbe1d8d788b7c9a71280e7d510c295a5853f0b'} 2024-06-11T19:06:18.183 INFO:teuthology.task.install:Using flavor: default 2024-06-11T19:06:18.189 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-11T19:06:18.189 INFO:teuthology.task.install:extra packages: [] 2024-06-11T19:06:18.189 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '37bbe1d8d788b7c9a71280e7d510c295a5853f0b', 'tag': None, 'wait_for_package': False} 2024-06-11T19:06:18.190 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=37bbe1d8d788b7c9a71280e7d510c295a5853f0b 2024-06-11T19:06:18.196 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '37bbe1d8d788b7c9a71280e7d510c295a5853f0b', 'tag': None, 'wait_for_package': False} 2024-06-11T19:06:18.196 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=37bbe1d8d788b7c9a71280e7d510c295a5853f0b 2024-06-11T19:06:18.348 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/centos/9/flavors/default/ 2024-06-11T19:06:18.348 INFO:teuthology.task.install.rpm:Package version is 19.0.0-4216.g37bbe1d8 2024-06-11T19:06:18.412 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/centos/9/flavors/default/ 2024-06-11T19:06:18.412 INFO:teuthology.task.install.rpm:Package version is 19.0.0-4216.g37bbe1d8 2024-06-11T19:06:18.462 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-11T19:06:18.462 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:06:18.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-11T19:06:18.489 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-11T19:06:18.490 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-11T19:06:18.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-11T19:06:18.584 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-06-10-1925/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-11T19:06:18.584 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:06:18.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-11T19:06:18.611 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-11T19:06:18.612 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/37bbe1d8d788b7c9a71280e7d510c295a5853f0b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-11T19:06:18.633 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-11T19:06:18.678 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-11T19:06:18.697 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-06-11T19:06:18.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-06-11T19:06:18.753 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-11T19:06:18.818 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-06-11T19:06:18.819 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-06-11T19:06:18.935 INFO:teuthology.orchestra.run.smithi098.stdout:45 files removed 2024-06-11T19:06:18.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw 2024-06-11T19:06:19.049 INFO:teuthology.orchestra.run.smithi117.stdout:45 files removed 2024-06-11T19:06:19.078 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw 2024-06-11T19:06:19.617 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 222 kB/s | 84 kB 00:00 2024-06-11T19:06:19.826 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 178 kB/s | 84 kB 00:00 2024-06-11T19:06:20.001 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-06-11T19:06:20.239 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 32 kB/s | 12 kB 00:00 2024-06-11T19:06:20.427 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-06-11T19:06:20.607 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-06-11T19:06:22.231 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.1 MB 00:01 2024-06-11T19:06:22.344 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.1 MB 00:01 2024-06-11T19:06:26.846 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 20 MB 00:03 2024-06-11T19:06:30.304 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 20 MB 00:06 2024-06-11T19:06:34.583 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.3 MB 00:04 2024-06-11T19:06:36.344 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 17 kB 00:00 2024-06-11T19:06:37.639 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-06-11T19:06:38.605 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.3 MB 00:04 2024-06-11T19:06:40.368 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 17 kB 00:00 2024-06-11T19:06:41.213 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-06-11T19:06:42.836 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-06-11T19:06:45.247 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:06:45.262 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:06:45.262 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-06-11T19:06:45.262 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:06:45.262 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:06:45.262 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 11 M 2024-06-11T19:06:45.262 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 3.4 M 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 3.1 M 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 5.4 M 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 21 M 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 25 k 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 748 k 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-11T19:06:45.263 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 504 k 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 5.4 M 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 45 k 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 136 k 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 161 k 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 321 k 2024-06-11T19:06:45.264 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 300 k 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 100 k 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout:Installing weak dependencies: 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout:Install 30 Packages 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 2 Packages 2024-06-11T19:06:45.265 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:06:45.266 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 61 M 2024-06-11T19:06:45.266 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:06:46.045 INFO:teuthology.orchestra.run.smithi117.stdout:(1/32): ceph-base-19.0.0-4216.g37bbe1d8.el9.x86 9.8 MB/s | 5.4 MB 00:00 2024-06-11T19:06:46.095 INFO:teuthology.orchestra.run.smithi117.stdout:(2/32): ceph-selinux-19.0.0-4216.g37bbe1d8.el9. 506 kB/s | 25 kB 00:00 2024-06-11T19:06:46.187 INFO:teuthology.orchestra.run.smithi117.stdout:(3/32): libcephfs2-19.0.0-4216.g37bbe1d8.el9.x8 8.0 MB/s | 748 kB 00:00 2024-06-11T19:06:46.246 INFO:teuthology.orchestra.run.smithi117.stdout:(4/32): libradosstriper1-19.0.0-4216.g37bbe1d8. 8.5 MB/s | 504 kB 00:00 2024-06-11T19:06:46.460 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-06-11T19:06:46.596 INFO:teuthology.orchestra.run.smithi117.stdout:(5/32): librgw2-19.0.0-4216.g37bbe1d8.el9.x86_6 15 MB/s | 5.4 MB 00:00 2024-06-11T19:06:46.646 INFO:teuthology.orchestra.run.smithi117.stdout:(6/32): python3-ceph-argparse-19.0.0-4216.g37bb 908 kB/s | 45 kB 00:00 2024-06-11T19:06:46.714 INFO:teuthology.orchestra.run.smithi117.stdout:(7/32): python3-ceph-common-19.0.0-4216.g37bbe1 2.0 MB/s | 136 kB 00:00 2024-06-11T19:06:46.763 INFO:teuthology.orchestra.run.smithi117.stdout:(8/32): python3-cephfs-19.0.0-4216.g37bbe1d8.el 3.2 MB/s | 161 kB 00:00 2024-06-11T19:06:46.831 INFO:teuthology.orchestra.run.smithi117.stdout:(9/32): python3-rados-19.0.0-4216.g37bbe1d8.el9 4.7 MB/s | 321 kB 00:00 2024-06-11T19:06:46.898 INFO:teuthology.orchestra.run.smithi117.stdout:(10/32): python3-rbd-19.0.0-4216.g37bbe1d8.el9. 4.4 MB/s | 300 kB 00:00 2024-06-11T19:06:46.965 INFO:teuthology.orchestra.run.smithi117.stdout:(11/32): python3-rgw-19.0.0-4216.g37bbe1d8.el9. 1.5 MB/s | 100 kB 00:00 2024-06-11T19:06:47.001 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:06:47.207 INFO:teuthology.orchestra.run.smithi117.stdout:(12/32): ceph-radosgw-19.0.0-4216.g37bbe1d8.el9 6.3 MB/s | 11 MB 00:01 2024-06-11T19:06:47.232 INFO:teuthology.orchestra.run.smithi117.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 400 kB/s | 107 kB 00:00 2024-06-11T19:06:47.232 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:06:47.233 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:06:47.316 INFO:teuthology.orchestra.run.smithi117.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-06-11T19:06:47.332 INFO:teuthology.orchestra.run.smithi117.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2024-06-11T19:06:47.383 INFO:teuthology.orchestra.run.smithi117.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-06-11T19:06:47.450 INFO:teuthology.orchestra.run.smithi117.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 630 kB/s | 42 kB 00:00 2024-06-11T19:06:47.475 INFO:teuthology.orchestra.run.smithi117.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-06-11T19:06:47.509 INFO:teuthology.orchestra.run.smithi117.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-06-11T19:06:47.534 INFO:teuthology.orchestra.run.smithi117.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 613 kB/s | 36 kB 00:00 2024-06-11T19:06:47.560 INFO:teuthology.orchestra.run.smithi117.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 940 kB/s | 47 kB 00:00 2024-06-11T19:06:47.610 INFO:teuthology.orchestra.run.smithi117.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.0 MB/s | 309 kB 00:00 2024-06-11T19:06:47.635 INFO:teuthology.orchestra.run.smithi117.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-06-11T19:06:47.660 INFO:teuthology.orchestra.run.smithi117.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-11T19:06:47.694 INFO:teuthology.orchestra.run.smithi117.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-06-11T19:06:47.744 INFO:teuthology.orchestra.run.smithi117.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 16 MB/s | 839 kB 00:00 2024-06-11T19:06:47.778 INFO:teuthology.orchestra.run.smithi117.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-06-11T19:06:47.870 INFO:teuthology.orchestra.run.smithi117.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-06-11T19:06:47.912 INFO:teuthology.orchestra.run.smithi117.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-06-11T19:06:48.112 INFO:teuthology.orchestra.run.smithi117.stdout:(30/32): ceph-common-19.0.0-4216.g37bbe1d8.el9. 8.1 MB/s | 21 MB 00:02 2024-06-11T19:06:48.187 INFO:teuthology.orchestra.run.smithi117.stdout:(31/32): librados2-19.0.0-4216.g37bbe1d8.el9.x8 11 MB/s | 3.4 MB 00:00 2024-06-11T19:06:48.354 INFO:teuthology.orchestra.run.smithi117.stdout:(32/32): librbd1-19.0.0-4216.g37bbe1d8.el9.x86_ 7.1 MB/s | 3.1 MB 00:00 2024-06-11T19:06:48.359 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:06:48.359 INFO:teuthology.orchestra.run.smithi117.stdout:Total 20 MB/s | 61 MB 00:03 2024-06-11T19:06:48.458 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:06:48.493 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:06:48.494 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:06:48.820 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:06:48.855 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:06:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-06-11T19:06:48.855 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:06:48.855 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:06:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 11 M 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 3.4 M 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 3.1 M 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 5.4 M 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 21 M 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 25 k 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 748 k 2024-06-11T19:06:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 504 k 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 5.4 M 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 45 k 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 136 k 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 161 k 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-11T19:06:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 321 k 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 300 k 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 100 k 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout:Install 30 Packages 2024-06-11T19:06:48.858 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 2 Packages 2024-06-11T19:06:48.859 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:06:48.859 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 61 M 2024-06-11T19:06:48.859 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:06:49.064 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:06:49.065 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:06:50.345 INFO:teuthology.orchestra.run.smithi098.stdout:(1/32): ceph-radosgw-19.0.0-4216.g37bbe1d8.el9. 11 MB/s | 11 MB 00:00 2024-06-11T19:06:50.395 INFO:teuthology.orchestra.run.smithi098.stdout:(2/32): ceph-selinux-19.0.0-4216.g37bbe1d8.el9. 505 kB/s | 25 kB 00:00 2024-06-11T19:06:50.470 INFO:teuthology.orchestra.run.smithi098.stdout:(3/32): ceph-base-19.0.0-4216.g37bbe1d8.el9.x86 5.1 MB/s | 5.4 MB 00:01 2024-06-11T19:06:50.498 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:06:50.512 INFO:teuthology.orchestra.run.smithi098.stdout:(4/32): libcephfs2-19.0.0-4216.g37bbe1d8.el9.x8 6.2 MB/s | 748 kB 00:00 2024-06-11T19:06:50.546 INFO:teuthology.orchestra.run.smithi098.stdout:(5/32): libradosstriper1-19.0.0-4216.g37bbe1d8. 6.5 MB/s | 504 kB 00:00 2024-06-11T19:06:50.590 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-06-11T19:06:50.621 INFO:teuthology.orchestra.run.smithi098.stdout:(6/32): python3-ceph-argparse-19.0.0-4216.g37bb 601 kB/s | 45 kB 00:00 2024-06-11T19:06:50.634 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-06-11T19:06:50.668 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-06-11T19:06:50.690 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-11T19:06:50.715 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-11T19:06:50.721 INFO:teuthology.orchestra.run.smithi098.stdout:(7/32): python3-ceph-common-19.0.0-4216.g37bbe1 1.3 MB/s | 136 kB 00:00 2024-06-11T19:06:50.838 INFO:teuthology.orchestra.run.smithi098.stdout:(8/32): librgw2-19.0.0-4216.g37bbe1d8.el9.x86_6 17 MB/s | 5.4 MB 00:00 2024-06-11T19:06:50.864 INFO:teuthology.orchestra.run.smithi098.stdout:(9/32): python3-cephfs-19.0.0-4216.g37bbe1d8.el 1.1 MB/s | 161 kB 00:00 2024-06-11T19:06:50.892 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-11T19:06:50.906 INFO:teuthology.orchestra.run.smithi098.stdout:(10/32): python3-rados-19.0.0-4216.g37bbe1d8.el 4.7 MB/s | 321 kB 00:00 2024-06-11T19:06:50.931 INFO:teuthology.orchestra.run.smithi098.stdout:(11/32): python3-rbd-19.0.0-4216.g37bbe1d8.el9. 4.4 MB/s | 300 kB 00:00 2024-06-11T19:06:50.946 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-11T19:06:50.956 INFO:teuthology.orchestra.run.smithi098.stdout:(12/32): python3-rgw-19.0.0-4216.g37bbe1d8.el9. 1.9 MB/s | 100 kB 00:00 2024-06-11T19:06:51.010 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:06:51.016 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:06:51.327 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-06-11T19:06:51.378 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 7/34 2024-06-11T19:06:51.462 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 7/34 2024-06-11T19:06:51.515 INFO:teuthology.orchestra.run.smithi098.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 183 kB/s | 107 kB 00:00 2024-06-11T19:06:51.516 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:06:51.551 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:19.0.0-4216.g37bbe1d8.el9.x86_64 8/34 2024-06-11T19:06:51.579 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 9/34 2024-06-11T19:06:51.676 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 9/34 2024-06-11T19:06:51.705 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-06-11T19:06:51.748 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:19.0.0-4216.g37bbe1d8.el9. 11/34 2024-06-11T19:06:51.757 INFO:teuthology.orchestra.run.smithi098.stdout:(14/32): ceph-common-19.0.0-4216.g37bbe1d8.el9. 9.0 MB/s | 21 MB 00:02 2024-06-11T19:06:51.782 INFO:teuthology.orchestra.run.smithi098.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 194 kB/s | 160 kB 00:00 2024-06-11T19:06:51.799 INFO:teuthology.orchestra.run.smithi098.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 159 kB/s | 45 kB 00:00 2024-06-11T19:06:51.826 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:19.0.0-4216.g37bbe1d8.el9.x86_64 12/34 2024-06-11T19:06:51.848 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:19.0.0-4216.g37bbe1d8.el9.x86_6 13/34 2024-06-11T19:06:51.897 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:19.0.0-4216.g37bbe1d8.el9.x86_6 13/34 2024-06-11T19:06:51.931 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-06-11T19:06:52.043 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-06-11T19:06:52.066 INFO:teuthology.orchestra.run.smithi098.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-06-11T19:06:52.074 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-06-11T19:06:52.100 INFO:teuthology.orchestra.run.smithi098.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 141 kB/s | 42 kB 00:00 2024-06-11T19:06:52.116 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-06-11T19:06:52.125 INFO:teuthology.orchestra.run.smithi098.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 898 kB/s | 292 kB 00:00 2024-06-11T19:06:52.148 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-06-11T19:06:52.175 INFO:teuthology.orchestra.run.smithi098.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-06-11T19:06:52.177 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-06-11T19:06:52.201 INFO:teuthology.orchestra.run.smithi098.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 358 kB/s | 36 kB 00:00 2024-06-11T19:06:52.210 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-06-11T19:06:52.226 INFO:teuthology.orchestra.run.smithi098.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 471 kB/s | 47 kB 00:00 2024-06-11T19:06:52.285 INFO:teuthology.orchestra.run.smithi098.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-06-11T19:06:52.310 INFO:teuthology.orchestra.run.smithi098.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.2 MB/s | 49 kB 00:00 2024-06-11T19:06:52.335 INFO:teuthology.orchestra.run.smithi098.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-06-11T19:06:52.361 INFO:teuthology.orchestra.run.smithi098.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-06-11T19:06:52.394 INFO:teuthology.orchestra.run.smithi098.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-06-11T19:06:52.470 INFO:teuthology.orchestra.run.smithi098.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 839 kB 00:00 2024-06-11T19:06:52.520 INFO:teuthology.orchestra.run.smithi098.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-06-11T19:06:52.534 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-06-11T19:06:52.574 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:19.0.0-4216.g37bbe1d8.el9.x86_64 22/34 2024-06-11T19:06:52.595 INFO:teuthology.orchestra.run.smithi098.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-06-11T19:06:52.655 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:19.0.0-4216.g37bbe1d8.el9.x86_64 22/34 2024-06-11T19:06:52.679 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 23/34 2024-06-11T19:06:52.715 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-06-11T19:06:52.862 INFO:teuthology.orchestra.run.smithi098.stdout:(31/32): librbd1-19.0.0-4216.g37bbe1d8.el9.x86_ 9.2 MB/s | 3.1 MB 00:00 2024-06-11T19:06:52.987 INFO:teuthology.orchestra.run.smithi098.stdout:(32/32): librados2-19.0.0-4216.g37bbe1d8.el9.x8 6.6 MB/s | 3.4 MB 00:00 2024-06-11T19:06:52.995 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:06:52.995 INFO:teuthology.orchestra.run.smithi098.stdout:Total 15 MB/s | 61 MB 00:04 2024-06-11T19:06:53.092 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:06:53.128 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:06:53.128 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:06:53.195 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-06-11T19:06:53.235 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 26/34 2024-06-11T19:06:53.306 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 26/34 2024-06-11T19:06:53.375 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 27/34 2024-06-11T19:06:53.404 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x8 28/34 2024-06-11T19:06:53.699 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:06:53.699 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:06:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:06:55.281 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-06-11T19:06:55.309 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-06-11T19:06:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-06-11T19:06:55.357 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-11T19:06:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-11T19:06:55.568 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-11T19:06:55.621 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-11T19:06:55.692 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x86_64 29/34 2024-06-11T19:06:55.767 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x86_64 29/34 2024-06-11T19:06:56.004 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-06-11T19:06:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 7/34 2024-06-11T19:06:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 7/34 2024-06-11T19:06:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-4216.g37bbe1d8.el9.x86_64 8/34 2024-06-11T19:06:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 9/34 2024-06-11T19:06:56.321 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x86_64 29/34 2024-06-11T19:06:56.326 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 9/34 2024-06-11T19:06:56.355 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-06-11T19:06:56.363 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:19.0.0-4216.g37bbe1d8.el9.x86_64 30/34 2024-06-11T19:06:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-4216.g37bbe1d8.el9. 11/34 2024-06-11T19:06:56.457 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:19.0.0-4216.g37bbe1d8.el9.x86_64 30/34 2024-06-11T19:06:56.458 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-11T19:06:56.458 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-11T19:06:56.458 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:06:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-4216.g37bbe1d8.el9.x86_64 12/34 2024-06-11T19:06:56.480 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:19.0.0-4216.g37bbe1d8.el9.x86_64 31/34 2024-06-11T19:06:56.499 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-4216.g37bbe1d8.el9.x86_6 13/34 2024-06-11T19:06:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-4216.g37bbe1d8.el9.x86_6 13/34 2024-06-11T19:06:56.606 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-06-11T19:06:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-06-11T19:06:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-06-11T19:06:56.815 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-06-11T19:06:56.847 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-06-11T19:06:56.876 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-06-11T19:06:56.909 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-06-11T19:06:57.234 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-06-11T19:06:57.280 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-4216.g37bbe1d8.el9.x86_64 22/34 2024-06-11T19:06:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-4216.g37bbe1d8.el9.x86_64 22/34 2024-06-11T19:06:57.402 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 23/34 2024-06-11T19:06:57.437 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-06-11T19:06:57.920 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-06-11T19:06:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 26/34 2024-06-11T19:06:58.004 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 26/34 2024-06-11T19:06:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 27/34 2024-06-11T19:06:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x8 28/34 2024-06-11T19:07:00.406 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x86_64 29/34 2024-06-11T19:07:00.450 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x86_64 29/34 2024-06-11T19:07:01.018 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x86_64 29/34 2024-06-11T19:07:01.054 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-4216.g37bbe1d8.el9.x86_64 30/34 2024-06-11T19:07:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-4216.g37bbe1d8.el9.x86_64 30/34 2024-06-11T19:07:01.136 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-11T19:07:01.136 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-11T19:07:01.136 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:01.163 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-4216.g37bbe1d8.el9.x86_64 31/34 2024-06-11T19:07:06.063 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:19.0.0-4216.g37bbe1d8.el9.x86_64 31/34 2024-06-11T19:07:06.063 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-06-11T19:07:06.063 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-06-11T19:07:06.063 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-06-11T19:07:06.063 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-06-11T19:07:06.063 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-06-11T19:07:06.063 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-06-11T19:07:06.063 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-06-11T19:07:06.063 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:07.013 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 32/34 2024-06-11T19:07:07.039 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 32/34 2024-06-11T19:07:07.039 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:07:07.039 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-11T19:07:07.040 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-11T19:07:07.040 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-11T19:07:07.040 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:07.042 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-11T19:07:07.107 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-11T19:07:07.108 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-11T19:07:08.229 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-11T19:07:08.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/34 2024-06-11T19:07:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2/34 2024-06-11T19:07:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 3/34 2024-06-11T19:07:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:19.0.0-4216.g37bbe1d8.el9.x86_64 4/34 2024-06-11T19:07:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 5/34 2024-06-11T19:07:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:19.0.0-4216.g37bbe1d8.el9.x86_6 6/34 2024-06-11T19:07:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 7/34 2024-06-11T19:07:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:19.0.0-4216.g37bbe1d8.el9. 8/34 2024-06-11T19:07:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x8 9/34 2024-06-11T19:07:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:19.0.0-4216.g37bbe1d8.el9.x86_64 10/34 2024-06-11T19:07:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:19.0.0-4216.g37bbe1d8.el9.x86_64 11/34 2024-06-11T19:07:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 12/34 2024-06-11T19:07:08.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 13/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-06-11T19:07:08.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-06-11T19:07:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-06-11T19:07:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-06-11T19:07:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-06-11T19:07:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-06-11T19:07:08.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 31/34 2024-06-11T19:07:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-06-11T19:07:08.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:19.0.0-4216.g37bbe1d8.el9.x86_64 33/34 2024-06-11T19:07:09.021 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-06-11T19:07:09.021 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:09.021 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.022 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-11T19:07:09.023 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.024 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.024 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:09.024 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-11T19:07:09.024 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-11T19:07:09.024 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-11T19:07:09.024 INFO:teuthology.orchestra.run.smithi117.stdout: re2-1:20211101-3.el9.x86_64 2024-06-11T19:07:09.024 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-11T19:07:09.024 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:09.024 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:07:09.380 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-test 2024-06-11T19:07:09.959 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:27 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:07:10.084 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:07:10.085 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:10.085 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-06-11T19:07:10.085 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:10.086 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:07:10.086 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 49 M 2024-06-11T19:07:10.086 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-06-11T19:07:10.086 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 163 k 2024-06-11T19:07:10.086 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-11T19:07:10.086 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-11T19:07:10.086 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:10.086 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:07:10.086 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:10.087 INFO:teuthology.orchestra.run.smithi117.stdout:Install 4 Packages 2024-06-11T19:07:10.087 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:10.087 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 50 M 2024-06-11T19:07:10.087 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 212 M 2024-06-11T19:07:10.087 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:07:10.437 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:10.484 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 206.82.17.213) 2024-06-11T19:07:10.580 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 206.82.17.213) 2024-06-11T19:07:10.608 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 216.158.133.254) 2024-06-11T19:07:10.676 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 216.158.133.254) 2024-06-11T19:07:10.709 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 199.223.246.113) 2024-06-11T19:07:10.732 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-4216.g37bbe1d8.el9.x86_64 31/34 2024-06-11T19:07:10.732 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-06-11T19:07:10.732 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-06-11T19:07:10.732 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-06-11T19:07:10.733 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-06-11T19:07:10.733 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-06-11T19:07:10.733 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-06-11T19:07:10.733 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-06-11T19:07:10.733 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:10.741 INFO:teuthology.orchestra.run.smithi117.stdout:(1/4): libcephsqlite-19.0.0-4216.g37bbe1d8.el9. 430 kB/s | 163 kB 00:00 2024-06-11T19:07:10.742 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] xmlstarlet-1.6.1-20.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:10.850 INFO:teuthology.orchestra.run.smithi117.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 587 kB/s | 64 kB 00:00 2024-06-11T19:07:11.183 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 104.251.122.5) 2024-06-11T19:07:11.521 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 104.251.122.5) 2024-06-11T19:07:11.559 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 64.50.233.100) 2024-06-11T19:07:11.661 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 64.50.233.100) 2024-06-11T19:07:11.728 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 32/34 2024-06-11T19:07:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 32/34 2024-06-11T19:07:11.752 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:07:11.752 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-11T19:07:11.752 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-11T19:07:11.753 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-11T19:07:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-11T19:07:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-11T19:07:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-11T19:07:11.850 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 140.211.166.134) 2024-06-11T19:07:12.211 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 140.211.166.134) 2024-06-11T19:07:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 163.237.219.7) 2024-06-11T19:07:12.725 INFO:teuthology.orchestra.run.smithi117.stdout:(3/4): ceph-test-19.0.0-4216.g37bbe1d8.el9.x86_ 21 MB/s | 49 MB 00:02 2024-06-11T19:07:12.812 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 163.237.219.7) 2024-06-11T19:07:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-11T19:07:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/34 2024-06-11T19:07:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2/34 2024-06-11T19:07:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 3/34 2024-06-11T19:07:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-4216.g37bbe1d8.el9.x86_64 4/34 2024-06-11T19:07:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 5/34 2024-06-11T19:07:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-4216.g37bbe1d8.el9.x86_6 6/34 2024-06-11T19:07:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 7/34 2024-06-11T19:07:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-4216.g37bbe1d8.el9. 8/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x8 9/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-4216.g37bbe1d8.el9.x86_64 10/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-4216.g37bbe1d8.el9.x86_64 11/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 12/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 13/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-06-11T19:07:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-06-11T19:07:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-06-11T19:07:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-06-11T19:07:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-06-11T19:07:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-06-11T19:07:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-06-11T19:07:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-06-11T19:07:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-06-11T19:07:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-06-11T19:07:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-06-11T19:07:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 31/34 2024-06-11T19:07:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-06-11T19:07:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-4216.g37bbe1d8.el9.x86_64 33/34 2024-06-11T19:07:13.025 INFO:teuthology.orchestra.run.smithi117.stdout:(4/4): socat-1.7.4.1-6.el9.x86_64.rpm 114 kB/s | 304 kB 00:02 2024-06-11T19:07:13.029 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:07:13.029 INFO:teuthology.orchestra.run.smithi117.stdout:Total 17 MB/s | 50 MB 00:02 2024-06-11T19:07:13.055 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:07:13.076 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:07:13.077 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:07:13.449 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:07:13.449 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:07:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-06-11T19:07:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:13.838 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-06-11T19:07:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.838 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:07:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-11T19:07:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-11T19:07:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-11T19:07:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-11T19:07:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-11T19:07:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-11T19:07:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-11T19:07:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-11T19:07:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-11T19:07:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-11T19:07:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-11T19:07:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-11T19:07:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-11T19:07:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-11T19:07:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-11T19:07:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-06-11T19:07:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-11T19:07:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:13.844 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:07:13.985 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:07:14.060 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-11T19:07:14.099 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-11T19:07:14.127 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:19.0.0-4216.g37bbe1d8.el9.x86_64 3/4 2024-06-11T19:07:14.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-test 2024-06-11T19:07:14.780 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:28 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:07:14.905 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:07:14.905 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:14.905 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-06-11T19:07:14.906 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:14.906 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:07:14.906 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 49 M 2024-06-11T19:07:14.906 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-06-11T19:07:14.906 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 163 k 2024-06-11T19:07:14.906 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-11T19:07:14.906 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-11T19:07:14.906 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:14.906 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:07:14.906 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:14.906 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-06-11T19:07:14.906 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:14.907 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 50 M 2024-06-11T19:07:14.907 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 212 M 2024-06-11T19:07:14.907 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:07:15.254 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:15.319 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://ftp-chi.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 64.50.236.52) 2024-06-11T19:07:15.427 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): libcephsqlite-19.0.0-4216.g37bbe1d8.el9. 652 kB/s | 163 kB 00:00 2024-06-11T19:07:15.429 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] xmlstarlet-1.6.1-20.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:15.490 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 64.50.236.52) 2024-06-11T19:07:15.544 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 552 kB/s | 64 kB 00:00 2024-06-11T19:07:15.545 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T19:07:15.631 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 162.255.226.75) 2024-06-11T19:07:15.677 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 206.82.17.213) 2024-06-11T19:07:15.771 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 206.82.17.213) 2024-06-11T19:07:15.831 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 163.237.219.7) 2024-06-11T19:07:15.946 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 163.237.219.7) 2024-06-11T19:07:15.973 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 216.158.133.254) 2024-06-11T19:07:16.039 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 216.158.133.254) 2024-06-11T19:07:16.074 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 64.50.233.100) 2024-06-11T19:07:16.169 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 64.50.233.100) 2024-06-11T19:07:16.803 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 187 kB/s | 304 kB 00:01 2024-06-11T19:07:17.553 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): ceph-test-19.0.0-4216.g37bbe1d8.el9.x86_ 21 MB/s | 49 MB 00:02 2024-06-11T19:07:17.560 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:07:17.560 INFO:teuthology.orchestra.run.smithi098.stdout:Total 19 MB/s | 50 MB 00:02 2024-06-11T19:07:17.586 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:07:17.607 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:07:17.607 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:07:17.983 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:07:17.984 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:07:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:07:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-11T19:07:18.644 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-11T19:07:18.672 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-4216.g37bbe1d8.el9.x86_64 3/4 2024-06-11T19:07:18.775 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:19.0.0-4216.g37bbe1d8.el9.x86_64 3/4 2024-06-11T19:07:18.857 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:19.0.0-4216.g37bbe1d8.el9.x86_64 4/4 2024-06-11T19:07:21.276 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-test-2:19.0.0-4216.g37bbe1d8.el9.x86_64 4/4 2024-06-11T19:07:21.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/4 2024-06-11T19:07:21.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2/4 2024-06-11T19:07:21.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-11T19:07:21.558 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-11T19:07:21.558 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:21.558 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:07:21.558 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:21.558 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:21.558 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-11T19:07:21.558 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-11T19:07:21.559 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:21.559 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:07:21.840 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph 2024-06-11T19:07:22.427 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:40 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:07:22.565 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:07:22.568 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:22.568 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-06-11T19:07:22.568 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:22.568 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:07:22.568 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 6.6 k 2024-06-11T19:07:22.568 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-06-11T19:07:22.568 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 2.4 M 2024-06-11T19:07:22.568 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 1.5 M 2024-06-11T19:07:22.569 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 274 k 2024-06-11T19:07:22.569 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 4.7 M 2024-06-11T19:07:22.569 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 17 M 2024-06-11T19:07:22.569 INFO:teuthology.orchestra.run.smithi117.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-11T19:07:22.569 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-11T19:07:22.569 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-11T19:07:22.569 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-11T19:07:22.569 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-11T19:07:22.569 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-11T19:07:22.569 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-11T19:07:22.569 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-11T19:07:22.569 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-11T19:07:22.570 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:07:22.571 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:22.572 INFO:teuthology.orchestra.run.smithi117.stdout:Install 37 Packages 2024-06-11T19:07:22.572 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:22.572 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 30 M 2024-06-11T19:07:22.572 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 108 M 2024-06-11T19:07:22.572 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:07:23.108 INFO:teuthology.orchestra.run.smithi117.stdout:(1/37): ceph-19.0.0-4216.g37bbe1d8.el9.x86_64.r 33 kB/s | 6.6 kB 00:00 2024-06-11T19:07:23.308 INFO:teuthology.orchestra.run.smithi117.stdout:(2/37): ceph-mds-19.0.0-4216.g37bbe1d8.el9.x86_ 5.9 MB/s | 2.4 MB 00:00 2024-06-11T19:07:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-4216.g37bbe1d8.el9.x86_64 3/4 2024-06-11T19:07:23.403 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-4216.g37bbe1d8.el9.x86_64 4/4 2024-06-11T19:07:23.617 INFO:teuthology.orchestra.run.smithi117.stdout:(3/37): ceph-mgr-19.0.0-4216.g37bbe1d8.el9.x86_ 2.1 MB/s | 1.5 MB 00:00 2024-06-11T19:07:23.734 INFO:teuthology.orchestra.run.smithi117.stdout:(4/37): ceph-mgr-modules-core-19.0.0-4216.g37bb 2.3 MB/s | 274 kB 00:00 2024-06-11T19:07:23.771 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:23.918 INFO:teuthology.orchestra.run.smithi117.stdout:(5/37): ceph-mon-19.0.0-4216.g37bbe1d8.el9.x86_ 5.8 MB/s | 4.7 MB 00:00 2024-06-11T19:07:23.918 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:24.051 INFO:teuthology.orchestra.run.smithi117.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 799 kB/s | 253 kB 00:00 2024-06-11T19:07:24.052 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:07:24.235 INFO:teuthology.orchestra.run.smithi117.stdout:(7/37): ceph-osd-19.0.0-4216.g37bbe1d8.el9.x86_ 18 MB/s | 17 MB 00:00 2024-06-11T19:07:24.260 INFO:teuthology.orchestra.run.smithi117.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 511 kB/s | 106 kB 00:00 2024-06-11T19:07:24.318 INFO:teuthology.orchestra.run.smithi117.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-06-11T19:07:24.369 INFO:teuthology.orchestra.run.smithi117.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.7 MB/s | 1.2 MB 00:00 2024-06-11T19:07:24.370 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] lua-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lua-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:24.402 INFO:teuthology.orchestra.run.smithi117.stdout:(11/37): python3-urllib3-1.26.5-5.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-06-11T19:07:24.403 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] python3-mako-1.1.4-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-mako-1.1.4-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:07:24.444 INFO:teuthology.orchestra.run.smithi117.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 646 kB/s | 135 kB 00:00 2024-06-11T19:07:24.445 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:24.545 INFO:teuthology.orchestra.run.smithi117.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 347 kB/s | 35 kB 00:00 2024-06-11T19:07:24.578 INFO:teuthology.orchestra.run.smithi117.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 981 kB/s | 172 kB 00:00 2024-06-11T19:07:24.603 INFO:teuthology.orchestra.run.smithi117.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 805 kB/s | 188 kB 00:00 2024-06-11T19:07:24.629 INFO:teuthology.orchestra.run.smithi117.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-06-11T19:07:24.654 INFO:teuthology.orchestra.run.smithi117.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 708 kB/s | 77 kB 00:00 2024-06-11T19:07:24.655 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:24.679 INFO:teuthology.orchestra.run.smithi117.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-06-11T19:07:24.730 INFO:teuthology.orchestra.run.smithi117.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-06-11T19:07:24.763 INFO:teuthology.orchestra.run.smithi117.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 518 kB/s | 43 kB 00:00 2024-06-11T19:07:24.789 INFO:teuthology.orchestra.run.smithi117.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.0 MB/s | 172 kB 00:00 2024-06-11T19:07:24.814 INFO:teuthology.orchestra.run.smithi117.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 120 kB/s | 22 kB 00:00 2024-06-11T19:07:24.839 INFO:teuthology.orchestra.run.smithi117.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-06-11T19:07:24.889 INFO:teuthology.orchestra.run.smithi117.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 912 kB/s | 23 kB 00:00 2024-06-11T19:07:24.915 INFO:teuthology.orchestra.run.smithi117.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 851 kB/s | 19 kB 00:00 2024-06-11T19:07:24.940 INFO:teuthology.orchestra.run.smithi117.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 140 kB/s | 18 kB 00:00 2024-06-11T19:07:24.965 INFO:teuthology.orchestra.run.smithi117.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-06-11T19:07:24.991 INFO:teuthology.orchestra.run.smithi117.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-06-11T19:07:25.016 INFO:teuthology.orchestra.run.smithi117.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-06-11T19:07:25.041 INFO:teuthology.orchestra.run.smithi117.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-06-11T19:07:25.066 INFO:teuthology.orchestra.run.smithi117.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-06-11T19:07:25.092 INFO:teuthology.orchestra.run.smithi117.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-06-11T19:07:25.117 INFO:teuthology.orchestra.run.smithi117.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-06-11T19:07:25.142 INFO:teuthology.orchestra.run.smithi117.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-06-11T19:07:25.197 INFO:teuthology.orchestra.run.smithi117.stdout:(35/37): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-06-11T19:07:25.217 INFO:teuthology.orchestra.run.smithi117.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-06-11T19:07:25.242 INFO:teuthology.orchestra.run.smithi117.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-06-11T19:07:25.246 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:07:25.246 INFO:teuthology.orchestra.run.smithi117.stdout:Total 11 MB/s | 30 MB 00:02 2024-06-11T19:07:25.393 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:07:25.433 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:07:25.433 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:07:25.856 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:07:25.857 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:07:25.932 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-test-2:19.0.0-4216.g37bbe1d8.el9.x86_64 4/4 2024-06-11T19:07:25.932 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/4 2024-06-11T19:07:25.932 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2/4 2024-06-11T19:07:25.932 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-11T19:07:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-11T19:07:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:26.185 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:07:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-11T19:07:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-11T19:07:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:26.186 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:07:26.448 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph 2024-06-11T19:07:27.039 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:41 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:07:27.151 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:07:27.174 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:07:27.177 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-06-11T19:07:27.177 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:27.177 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:07:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 6.6 k 2024-06-11T19:07:27.177 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-06-11T19:07:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 2.4 M 2024-06-11T19:07:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 1.5 M 2024-06-11T19:07:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 274 k 2024-06-11T19:07:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 4.7 M 2024-06-11T19:07:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 17 M 2024-06-11T19:07:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-11T19:07:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-11T19:07:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-11T19:07:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-11T19:07:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-11T19:07:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-11T19:07:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-11T19:07:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-11T19:07:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-11T19:07:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-11T19:07:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections 2024-06-11T19:07:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-11T19:07:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-11T19:07:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-11T19:07:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-11T19:07:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-11T19:07:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-11T19:07:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-11T19:07:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-11T19:07:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:27.180 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:07:27.180 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:27.180 INFO:teuthology.orchestra.run.smithi098.stdout:Install 37 Packages 2024-06-11T19:07:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 30 M 2024-06-11T19:07:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 108 M 2024-06-11T19:07:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:07:27.204 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-06-11T19:07:27.252 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-06-11T19:07:27.281 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-06-11T19:07:27.353 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-06-11T19:07:27.414 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-06-11T19:07:27.455 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-06-11T19:07:27.548 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-06-11T19:07:27.634 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-06-11T19:07:27.683 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-06-11T19:07:27.717 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-06-11T19:07:27.731 INFO:teuthology.orchestra.run.smithi098.stdout:(1/37): ceph-19.0.0-4216.g37bbe1d8.el9.x86_64.r 36 kB/s | 6.6 kB 00:00 2024-06-11T19:07:27.747 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-06-11T19:07:27.781 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-06-11T19:07:27.815 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-06-11T19:07:27.864 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-06-11T19:07:27.899 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-06-11T19:07:27.941 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-06-11T19:07:27.955 INFO:teuthology.orchestra.run.smithi098.stdout:(2/37): ceph-mgr-19.0.0-4216.g37bbe1d8.el9.x86_ 3.8 MB/s | 1.5 MB 00:00 2024-06-11T19:07:27.981 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-06-11T19:07:28.013 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-06-11T19:07:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-06-11T19:07:28.228 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-06-11T19:07:28.305 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-06-11T19:07:28.359 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/37 2024-06-11T19:07:28.390 INFO:teuthology.orchestra.run.smithi098.stdout:(3/37): ceph-mon-19.0.0-4216.g37bbe1d8.el9.x86_ 7.1 MB/s | 4.7 MB 00:00 2024-06-11T19:07:28.405 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-06-11T19:07:28.462 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-06-11T19:07:28.481 INFO:teuthology.orchestra.run.smithi098.stdout:(4/37): ceph-mgr-modules-core-19.0.0-4216.g37bb 2.9 MB/s | 274 kB 00:00 2024-06-11T19:07:28.517 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-06-11T19:07:28.719 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-06-11T19:07:28.774 INFO:teuthology.orchestra.run.smithi098.stdout:(5/37): ceph-mds-19.0.0-4216.g37bbe1d8.el9.x86_ 1.9 MB/s | 2.4 MB 00:01 2024-06-11T19:07:28.776 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-06-11T19:07:28.799 INFO:teuthology.orchestra.run.smithi098.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 799 kB/s | 253 kB 00:00 2024-06-11T19:07:28.799 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:28.800 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:07:28.854 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-06-11T19:07:28.952 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-06-11T19:07:28.966 INFO:teuthology.orchestra.run.smithi098.stdout:(7/37): ceph-osd-19.0.0-4216.g37bbe1d8.el9.x86_ 17 MB/s | 17 MB 00:01 2024-06-11T19:07:28.991 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-06-11T19:07:29.041 INFO:teuthology.orchestra.run.smithi098.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 4.6 MB/s | 1.2 MB 00:00 2024-06-11T19:07:29.116 INFO:teuthology.orchestra.run.smithi098.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.6 MB/s | 125 kB 00:00 2024-06-11T19:07:29.118 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-06-11T19:07:29.142 INFO:teuthology.orchestra.run.smithi098.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 311 kB/s | 106 kB 00:00 2024-06-11T19:07:29.143 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] lua-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lua-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:29.192 INFO:teuthology.orchestra.run.smithi098.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 598 kB/s | 135 kB 00:00 2024-06-11T19:07:29.218 INFO:teuthology.orchestra.run.smithi098.stdout:(12/37): python3-urllib3-1.26.5-5.el9.noarch.rp 2.1 MB/s | 215 kB 00:00 2024-06-11T19:07:29.219 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-mako-1.1.4-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-mako-1.1.4-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:07:29.219 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:29.280 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-4216.g37bbe1d8.el9. 32/37 2024-06-11T19:07:29.284 INFO:teuthology.orchestra.run.smithi098.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-06-11T19:07:29.301 INFO:teuthology.orchestra.run.smithi098.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 416 kB/s | 35 kB 00:00 2024-06-11T19:07:29.310 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:19.0.0-4216.g37bbe1d8.el9.x86_64 33/37 2024-06-11T19:07:29.336 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:19.0.0-4216.g37bbe1d8.el9.x86_64 33/37 2024-06-11T19:07:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:07:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-11T19:07:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-11T19:07:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-11T19:07:29.337 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:29.360 INFO:teuthology.orchestra.run.smithi098.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-06-11T19:07:29.385 INFO:teuthology.orchestra.run.smithi098.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 893 kB/s | 172 kB 00:00 2024-06-11T19:07:29.411 INFO:teuthology.orchestra.run.smithi098.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 468 kB/s | 51 kB 00:00 2024-06-11T19:07:29.411 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:29.445 INFO:teuthology.orchestra.run.smithi098.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 495 kB/s | 42 kB 00:00 2024-06-11T19:07:29.495 INFO:teuthology.orchestra.run.smithi098.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-06-11T19:07:29.528 INFO:teuthology.orchestra.run.smithi098.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.4 MB/s | 172 kB 00:00 2024-06-11T19:07:29.553 INFO:teuthology.orchestra.run.smithi098.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-06-11T19:07:29.578 INFO:teuthology.orchestra.run.smithi098.stdout:(22/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 324 kB/s | 43 kB 00:00 2024-06-11T19:07:29.629 INFO:teuthology.orchestra.run.smithi098.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2024-06-11T19:07:29.654 INFO:teuthology.orchestra.run.smithi098.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.0 MB/s | 23 kB 00:00 2024-06-11T19:07:29.680 INFO:teuthology.orchestra.run.smithi098.stdout:(25/37): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-06-11T19:07:29.705 INFO:teuthology.orchestra.run.smithi098.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-06-11T19:07:29.730 INFO:teuthology.orchestra.run.smithi098.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-06-11T19:07:29.756 INFO:teuthology.orchestra.run.smithi098.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-06-11T19:07:29.781 INFO:teuthology.orchestra.run.smithi098.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-06-11T19:07:29.806 INFO:teuthology.orchestra.run.smithi098.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-06-11T19:07:29.823 INFO:teuthology.orchestra.run.smithi098.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 390 kB/s | 16 kB 00:00 2024-06-11T19:07:29.840 INFO:teuthology.orchestra.run.smithi098.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-06-11T19:07:29.865 INFO:teuthology.orchestra.run.smithi098.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-06-11T19:07:29.890 INFO:teuthology.orchestra.run.smithi098.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-06-11T19:07:29.916 INFO:teuthology.orchestra.run.smithi098.stdout:(35/37): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-06-11T19:07:29.932 INFO:teuthology.orchestra.run.smithi098.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-06-11T19:07:29.958 INFO:teuthology.orchestra.run.smithi098.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-06-11T19:07:29.961 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:07:29.961 INFO:teuthology.orchestra.run.smithi098.stdout:Total 11 MB/s | 30 MB 00:02 2024-06-11T19:07:30.110 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:07:30.150 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:07:30.150 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:07:30.582 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:07:30.583 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:07:30.777 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 34/37 2024-06-11T19:07:30.804 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 34/37 2024-06-11T19:07:30.804 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:07:30.804 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-11T19:07:30.804 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-11T19:07:30.804 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-11T19:07:30.804 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:31.244 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:19.0.0-4216.g37bbe1d8.el9.x86_64 35/37 2024-06-11T19:07:31.268 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:19.0.0-4216.g37bbe1d8.el9.x86_64 35/37 2024-06-11T19:07:31.268 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:07:31.268 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-11T19:07:31.268 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-11T19:07:31.268 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-11T19:07:31.268 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:31.495 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:19.0.0-4216.g37bbe1d8.el9.x86_64 36/37 2024-06-11T19:07:31.518 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:19.0.0-4216.g37bbe1d8.el9.x86_64 36/37 2024-06-11T19:07:31.518 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:07:31.518 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-11T19:07:31.518 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-11T19:07:31.518 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-11T19:07:31.519 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:31.583 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:19.0.0-4216.g37bbe1d8.el9.x86_64 37/37 2024-06-11T19:07:31.875 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:07:31.936 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-06-11T19:07:31.979 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-06-11T19:07:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-06-11T19:07:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-06-11T19:07:32.154 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-06-11T19:07:32.187 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-06-11T19:07:32.280 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-06-11T19:07:32.358 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-06-11T19:07:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-06-11T19:07:32.433 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-06-11T19:07:32.463 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-06-11T19:07:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-06-11T19:07:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-06-11T19:07:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-06-11T19:07:32.615 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-06-11T19:07:32.649 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-06-11T19:07:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-06-11T19:07:32.738 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-06-11T19:07:32.812 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-06-11T19:07:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-06-11T19:07:33.031 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-06-11T19:07:33.092 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/37 2024-06-11T19:07:33.130 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-06-11T19:07:33.180 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-06-11T19:07:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-06-11T19:07:33.270 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-2:19.0.0-4216.g37bbe1d8.el9.x86_64 37/37 2024-06-11T19:07:33.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/37 2024-06-11T19:07:33.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2/37 2024-06-11T19:07:33.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:19.0.0-4216.g37bbe1d8.el9.x86_64 3/37 2024-06-11T19:07:33.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:19.0.0-4216.g37bbe1d8.el9.x86_64 4/37 2024-06-11T19:07:33.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 5/37 2024-06-11T19:07:33.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-4216.g37bbe1d8.el9. 6/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-06-11T19:07:33.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-06-11T19:07:33.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-06-11T19:07:33.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-06-11T19:07:33.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-06-11T19:07:33.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-06-11T19:07:33.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-06-11T19:07:33.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-06-11T19:07:33.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-06-11T19:07:33.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-06-11T19:07:33.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-06-11T19:07:33.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-06-11T19:07:33.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-06-11T19:07:33.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-06-11T19:07:33.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-06-11T19:07:33.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-06-11T19:07:33.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-06-11T19:07:33.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-06-11T19:07:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-06-11T19:07:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-06-11T19:07:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-06-11T19:07:33.644 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-06-11T19:07:33.692 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-06-11T19:07:33.818 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-06-11T19:07:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-4216.g37bbe1d8.el9. 32/37 2024-06-11T19:07:34.011 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-4216.g37bbe1d8.el9.x86_64 33/37 2024-06-11T19:07:34.037 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-4216.g37bbe1d8.el9.x86_64 33/37 2024-06-11T19:07:34.037 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:07:34.037 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-11T19:07:34.037 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-11T19:07:34.038 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-11T19:07:34.038 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:34.329 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-06-11T19:07:34.329 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:34.329 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:07:34.329 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:34.329 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: lua-5.4.4-4.el9.x86_64 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-11T19:07:34.330 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-11T19:07:34.331 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-11T19:07:34.332 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-11T19:07:34.332 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-11T19:07:34.332 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-11T19:07:34.332 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-11T19:07:34.332 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-11T19:07:34.332 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-11T19:07:34.332 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-11T19:07:34.332 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-11T19:07:34.332 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:34.332 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:07:34.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-base 2024-06-11T19:07:35.217 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:53 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:07:35.300 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-base-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:07:35.341 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:07:35.342 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-06-11T19:07:35.342 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:07:35.393 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install cephadm 2024-06-11T19:07:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 34/37 2024-06-11T19:07:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 34/37 2024-06-11T19:07:35.506 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:07:35.506 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-11T19:07:35.506 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-11T19:07:35.506 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-11T19:07:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-4216.g37bbe1d8.el9.x86_64 35/37 2024-06-11T19:07:35.964 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:53 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:07:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-4216.g37bbe1d8.el9.x86_64 35/37 2024-06-11T19:07:35.978 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:07:35.978 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-11T19:07:35.978 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-11T19:07:35.978 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-11T19:07:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:36.088 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:07:36.088 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-06-11T19:07:36.089 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:36.089 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:07:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 764 k 2024-06-11T19:07:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:36.089 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:07:36.089 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:36.089 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-06-11T19:07:36.089 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:36.089 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 764 k 2024-06-11T19:07:36.089 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 770 k 2024-06-11T19:07:36.089 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:07:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-4216.g37bbe1d8.el9.x86_64 36/37 2024-06-11T19:07:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-4216.g37bbe1d8.el9.x86_64 36/37 2024-06-11T19:07:36.229 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:07:36.229 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-11T19:07:36.229 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-11T19:07:36.229 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-11T19:07:36.229 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-4216.g37bbe1d8.el9.x86_64 37/37 2024-06-11T19:07:36.385 INFO:teuthology.orchestra.run.smithi117.stdout:cephadm-19.0.0-4216.g37bbe1d8.el9.noarch.rpm 2.5 MB/s | 764 kB 00:00 2024-06-11T19:07:36.385 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:07:36.386 INFO:teuthology.orchestra.run.smithi117.stdout:Total 2.5 MB/s | 764 kB 00:00 2024-06-11T19:07:36.386 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:07:36.391 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:07:36.391 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:07:36.397 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:07:36.397 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:07:36.474 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:07:36.979 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:07:37.071 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:07:37.453 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:07:37.689 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:07:37.690 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:37.690 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:07:37.690 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:07:37.690 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:37.690 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:07:37.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-immutable-object-cache 2024-06-11T19:07:38.052 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-2:19.0.0-4216.g37bbe1d8.el9.x86_64 37/37 2024-06-11T19:07:38.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/37 2024-06-11T19:07:38.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2/37 2024-06-11T19:07:38.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-4216.g37bbe1d8.el9.x86_64 3/37 2024-06-11T19:07:38.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-4216.g37bbe1d8.el9.x86_64 4/37 2024-06-11T19:07:38.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 5/37 2024-06-11T19:07:38.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-4216.g37bbe1d8.el9. 6/37 2024-06-11T19:07:38.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-06-11T19:07:38.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-06-11T19:07:38.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-06-11T19:07:38.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-06-11T19:07:38.052 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-06-11T19:07:38.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-06-11T19:07:38.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-06-11T19:07:38.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-06-11T19:07:38.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-06-11T19:07:38.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-06-11T19:07:38.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-06-11T19:07:38.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-06-11T19:07:38.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-06-11T19:07:38.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-06-11T19:07:38.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-06-11T19:07:38.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-06-11T19:07:38.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-06-11T19:07:38.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-06-11T19:07:38.475 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:56 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:07:38.601 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:07:38.602 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:38.602 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-06-11T19:07:38.602 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:38.602 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:07:38.602 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 145 k 2024-06-11T19:07:38.602 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:38.602 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:07:38.603 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:38.603 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-06-11T19:07:38.603 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:38.603 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 145 k 2024-06-11T19:07:38.603 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 435 k 2024-06-11T19:07:38.603 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:07:38.965 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-immutable-object-cache-19.0.0-4216.g37bbe1 399 kB/s | 145 kB 00:00 2024-06-11T19:07:38.966 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:07:38.966 INFO:teuthology.orchestra.run.smithi117.stdout:Total 398 kB/s | 145 kB 00:00 2024-06-11T19:07:38.966 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:07:38.974 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:07:38.974 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:07:39.032 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:07:39.032 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:07:39.125 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-06-11T19:07:39.125 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:39.126 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:07:39.126 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:39.126 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:39.126 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:39.126 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:07:39.126 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:39.126 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:39.126 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-11T19:07:39.127 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-11T19:07:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-11T19:07:39.129 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-11T19:07:39.129 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-11T19:07:39.129 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-11T19:07:39.129 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:39.129 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:07:39.179 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:07:39.241 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-4216.g37bbe1d8. 1/1 2024-06-11T19:07:39.266 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-4216.g37bbe1d8. 1/1 2024-06-11T19:07:39.266 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:07:39.266 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-11T19:07:39.266 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:39.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base 2024-06-11T19:07:40.008 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:54 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:07:40.091 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-base-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:07:40.132 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:07:40.133 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-06-11T19:07:40.133 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:07:40.154 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-4216.g37bbe1d8. 1/1 2024-06-11T19:07:40.154 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:40.154 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:07:40.154 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:40.154 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:40.154 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:07:40.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install cephadm 2024-06-11T19:07:40.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr 2024-06-11T19:07:40.752 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:54 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:07:40.877 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:07:40.878 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:40.878 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-06-11T19:07:40.878 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:40.878 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:07:40.878 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 764 k 2024-06-11T19:07:40.878 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:40.878 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:07:40.878 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:40.878 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-06-11T19:07:40.878 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:40.878 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 764 k 2024-06-11T19:07:40.879 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 770 k 2024-06-11T19:07:40.879 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:07:40.954 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:58 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:07:41.042 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-mgr-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:07:41.083 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:07:41.084 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-06-11T19:07:41.084 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:07:41.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-dashboard 2024-06-11T19:07:41.164 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm-19.0.0-4216.g37bbe1d8.el9.noarch.rpm 2.6 MB/s | 764 kB 00:00 2024-06-11T19:07:41.165 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:07:41.165 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.6 MB/s | 764 kB 00:00 2024-06-11T19:07:41.165 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:07:41.170 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:07:41.171 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:07:41.177 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:07:41.177 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:07:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:07:41.717 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:59 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:07:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:07:41.818 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:07:41.842 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:07:41.843 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:41.843 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-06-11T19:07:41.843 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:41.843 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:07:41.843 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 3.5 M 2024-06-11T19:07:41.843 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-06-11T19:07:41.843 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-11T19:07:41.843 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 34 k 2024-06-11T19:07:41.843 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 16 k 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout:Install 12 Packages 2024-06-11T19:07:41.844 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:41.845 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 8.7 M 2024-06-11T19:07:41.845 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 104 M 2024-06-11T19:07:41.845 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:07:42.219 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:07:42.352 INFO:teuthology.orchestra.run.smithi117.stdout:(1/12): ceph-grafana-dashboards-19.0.0-4216.g37 165 kB/s | 34 kB 00:00 2024-06-11T19:07:42.377 INFO:teuthology.orchestra.run.smithi117.stdout:(2/12): ceph-prometheus-alerts-19.0.0-4216.g37b 71 kB/s | 16 kB 00:00 2024-06-11T19:07:42.428 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] protobuf-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/protobuf-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:42.435 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] python3-protobuf-3.14.0-13.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-protobuf-3.14.0-13.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:07:42.466 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:07:42.466 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:42.466 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:07:42.466 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:07:42.466 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:42.467 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:07:42.644 INFO:teuthology.orchestra.run.smithi117.stdout:(3/12): ceph-mgr-dashboard-19.0.0-4216.g37bbe1d 7.1 MB/s | 3.5 MB 00:00 2024-06-11T19:07:42.645 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:42.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-immutable-object-cache 2024-06-11T19:07:42.861 INFO:teuthology.orchestra.run.smithi117.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 556 kB/s | 269 kB 00:00 2024-06-11T19:07:42.895 INFO:teuthology.orchestra.run.smithi117.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 16 MB/s | 548 kB 00:00 2024-06-11T19:07:42.920 INFO:teuthology.orchestra.run.smithi117.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 774 kB/s | 19 kB 00:00 2024-06-11T19:07:43.004 INFO:teuthology.orchestra.run.smithi117.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-06-11T19:07:43.029 INFO:teuthology.orchestra.run.smithi117.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-06-11T19:07:43.054 INFO:teuthology.orchestra.run.smithi117.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-06-11T19:07:43.088 INFO:teuthology.orchestra.run.smithi117.stdout:(10/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.4 MB/s | 1.0 MB 00:00 2024-06-11T19:07:43.113 INFO:teuthology.orchestra.run.smithi117.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-06-11T19:07:43.260 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:57 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:07:43.385 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:07:43.386 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:43.386 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-06-11T19:07:43.386 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:43.386 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:07:43.386 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 145 k 2024-06-11T19:07:43.387 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:43.387 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:07:43.387 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:43.387 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-06-11T19:07:43.387 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:43.387 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 145 k 2024-06-11T19:07:43.387 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 435 k 2024-06-11T19:07:43.387 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:07:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-immutable-object-cache-19.0.0-4216.g37bbe1 469 kB/s | 145 kB 00:00 2024-06-11T19:07:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:07:43.697 INFO:teuthology.orchestra.run.smithi117.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 821 kB/s | 863 kB 00:01 2024-06-11T19:07:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:Total 466 kB/s | 145 kB 00:00 2024-06-11T19:07:43.697 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:07:43.702 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:07:43.702 INFO:teuthology.orchestra.run.smithi117.stdout:Total 4.7 MB/s | 8.7 MB 00:01 2024-06-11T19:07:43.704 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:07:43.704 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:07:43.753 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:07:43.760 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:07:43.760 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:07:43.770 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:07:43.770 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:07:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:07:43.948 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:07:43.949 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:07:43.972 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-4216.g37bbe1d8. 1/1 2024-06-11T19:07:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-4216.g37bbe1d8. 1/1 2024-06-11T19:07:43.997 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:07:43.997 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-11T19:07:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:44.476 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:07:44.547 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-06-11T19:07:44.591 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-06-11T19:07:44.637 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-06-11T19:07:44.668 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-06-11T19:07:44.749 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-06-11T19:07:44.867 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-4216.g37bbe1d8. 1/1 2024-06-11T19:07:44.867 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:44.868 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:07:44.868 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:07:44.868 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:44.868 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:07:44.883 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-06-11T19:07:44.970 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-06-11T19:07:45.014 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-06-11T19:07:45.043 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-06-11T19:07:45.075 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-4216.g37bbe1d8.el9 10/12 2024-06-11T19:07:45.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr 2024-06-11T19:07:45.677 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:59 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:07:45.765 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-mgr-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:07:45.807 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:07:45.808 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-06-11T19:07:45.808 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:07:45.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-dashboard 2024-06-11T19:07:46.123 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-4216.g37bbe1d8.el 11/12 2024-06-11T19:07:46.158 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-4216.g37bbe1d8.el9.noa 12/12 2024-06-11T19:07:46.437 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:00 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:07:46.563 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:07:46.564 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-06-11T19:07:46.564 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:46.564 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:07:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 3.5 M 2024-06-11T19:07:46.564 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-06-11T19:07:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-11T19:07:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 34 k 2024-06-11T19:07:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 16 k 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-06-11T19:07:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:46.566 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 8.7 M 2024-06-11T19:07:46.566 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 104 M 2024-06-11T19:07:46.566 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:07:47.034 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): ceph-prometheus-alerts-19.0.0-4216.g37b 84 kB/s | 16 kB 00:00 2024-06-11T19:07:47.069 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] protobuf-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/protobuf-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:47.134 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): ceph-grafana-dashboards-19.0.0-4216.g37 115 kB/s | 34 kB 00:00 2024-06-11T19:07:47.136 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-protobuf-3.14.0-13.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-protobuf-3.14.0-13.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:07:47.293 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): ceph-mgr-dashboard-19.0.0-4216.g37bbe1d 7.8 MB/s | 3.5 MB 00:00 2024-06-11T19:07:47.318 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:47.410 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-4216.g37bbe1d8.el9.noa 12/12 2024-06-11T19:07:47.410 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-4216.g37bbe1d8.el 1/12 2024-06-11T19:07:47.411 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-4216.g37bbe1d8.el9.noa 2/12 2024-06-11T19:07:47.411 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-4216.g37bbe1d8.el9 3/12 2024-06-11T19:07:47.411 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-06-11T19:07:47.411 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-06-11T19:07:47.411 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-06-11T19:07:47.411 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-06-11T19:07:47.411 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-06-11T19:07:47.411 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-06-11T19:07:47.411 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-06-11T19:07:47.411 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-06-11T19:07:47.703 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-06-11T19:07:47.703 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:47.703 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:07:47.703 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-11T19:07:47.703 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:07:47.704 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:07:47.704 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:07:47.704 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-11T19:07:47.704 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-11T19:07:47.704 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-11T19:07:47.704 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-11T19:07:47.704 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-11T19:07:47.704 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-11T19:07:47.704 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-11T19:07:47.704 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-11T19:07:47.704 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:47.704 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:07:47.932 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-11T19:07:48.002 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 311 kB/s | 269 kB 00:00 2024-06-11T19:07:48.035 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:01 2024-06-11T19:07:48.102 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.1 MB/s | 863 kB 00:00 2024-06-11T19:07:48.152 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 167 kB/s | 19 kB 00:00 2024-06-11T19:07:48.186 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.9 MB/s | 548 kB 00:00 2024-06-11T19:07:48.211 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.5 MB/s | 144 kB 00:00 2024-06-11T19:07:48.228 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 733 kB/s | 31 kB 00:00 2024-06-11T19:07:48.254 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-06-11T19:07:48.304 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 10 MB/s | 2.0 MB 00:00 2024-06-11T19:07:48.308 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:07:48.309 INFO:teuthology.orchestra.run.smithi098.stdout:Total 5.0 MB/s | 8.7 MB 00:01 2024-06-11T19:07:48.356 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:07:48.373 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:07:48.373 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:07:48.504 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:06 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:07:48.560 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:07:48.560 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:07:48.630 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:07:48.631 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================= 2024-06-11T19:07:48.631 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-06-11T19:07:48.631 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================= 2024-06-11T19:07:48.631 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:07:48.631 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 7.4 M 2024-06-11T19:07:48.631 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-06-11T19:07:48.631 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-11T19:07:48.631 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-11T19:07:48.631 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================= 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout:Install 12 Packages 2024-06-11T19:07:48.632 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:07:48.633 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 38 M 2024-06-11T19:07:48.633 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 209 M 2024-06-11T19:07:48.633 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:07:48.895 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:48.896 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:07:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-06-11T19:07:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-06-11T19:07:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-06-11T19:07:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-06-11T19:07:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-06-11T19:07:49.446 INFO:teuthology.orchestra.run.smithi117.stdout:(1/12): ceph-mgr-diskprediction-local-19.0.0-42 12 MB/s | 7.4 MB 00:00 2024-06-11T19:07:49.447 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] flexiblas-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:49.495 INFO:teuthology.orchestra.run.smithi117.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 288 kB/s | 195 kB 00:00 2024-06-11T19:07:49.497 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] flexiblas-netlib-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:49.521 INFO:teuthology.orchestra.run.smithi117.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 438 kB/s | 33 kB 00:00 2024-06-11T19:07:49.522 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-06-11T19:07:49.555 INFO:teuthology.orchestra.run.smithi117.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 549 kB/s | 18 kB 00:00 2024-06-11T19:07:49.596 INFO:teuthology.orchestra.run.smithi117.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 850 kB/s | 35 kB 00:00 2024-06-11T19:07:49.611 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-06-11T19:07:49.672 INFO:teuthology.orchestra.run.smithi117.stdout:(6/12): libgfortran-11.4.1-3.el9.x86_64.rpm 939 kB/s | 803 kB 00:00 2024-06-11T19:07:49.727 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-06-11T19:07:49.797 INFO:teuthology.orchestra.run.smithi117.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.9 MB/s | 244 kB 00:00 2024-06-11T19:07:49.917 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-06-11T19:07:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-4216.g37bbe1d8.el9 10/12 2024-06-11T19:07:50.289 INFO:teuthology.orchestra.run.smithi117.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.8 MB/s | 3.0 MB 00:00 2024-06-11T19:07:50.389 INFO:teuthology.orchestra.run.smithi117.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 250 kB 00:00 2024-06-11T19:07:50.606 INFO:teuthology.orchestra.run.smithi117.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 4.7 MB/s | 4.7 MB 00:01 2024-06-11T19:07:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-4216.g37bbe1d8.el 11/12 2024-06-11T19:07:51.050 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-4216.g37bbe1d8.el9.noa 12/12 2024-06-11T19:07:51.139 INFO:teuthology.orchestra.run.smithi117.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.8 MB/s | 5.1 MB 00:01 2024-06-11T19:07:52.272 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-4216.g37bbe1d8.el9.noa 12/12 2024-06-11T19:07:52.272 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-4216.g37bbe1d8.el 1/12 2024-06-11T19:07:52.272 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-4216.g37bbe1d8.el9.noa 2/12 2024-06-11T19:07:52.272 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-4216.g37bbe1d8.el9 3/12 2024-06-11T19:07:52.272 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-06-11T19:07:52.272 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-06-11T19:07:52.272 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-06-11T19:07:52.272 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-06-11T19:07:52.273 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-06-11T19:07:52.273 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-06-11T19:07:52.273 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-06-11T19:07:52.273 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-06-11T19:07:52.582 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-06-11T19:07:52.582 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:52.582 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:07:52.582 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-11T19:07:52.582 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:07:52.582 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:07:52.582 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:07:52.583 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-11T19:07:52.583 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-11T19:07:52.583 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-11T19:07:52.583 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-11T19:07:52.583 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-11T19:07:52.583 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-11T19:07:52.583 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-11T19:07:52.583 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-11T19:07:52.583 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:52.583 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:07:52.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-11T19:07:53.395 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:07 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:07:53.521 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:07:53.523 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-06-11T19:07:53.523 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-06-11T19:07:53.523 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-06-11T19:07:53.523 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:07:53.523 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 7.4 M 2024-06-11T19:07:53.523 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-06-11T19:07:53.523 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-11T19:07:53.523 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-11T19:07:53.523 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-06-11T19:07:53.524 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:07:53.525 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 38 M 2024-06-11T19:07:53.525 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 209 M 2024-06-11T19:07:53.525 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:07:53.738 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:53.745 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:53.888 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 958 kB/s | 195 kB 00:00 2024-06-11T19:07:53.889 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] flexiblas-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:53.938 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-06-11T19:07:53.939 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] flexiblas-netlib-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:54.480 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 55 kB/s | 33 kB 00:00 2024-06-11T19:07:54.481 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:07:54.997 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 35 kB/s | 18 kB 00:00 2024-06-11T19:07:55.022 INFO:teuthology.orchestra.run.smithi117.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.5 MB/s | 16 MB 00:04 2024-06-11T19:07:55.027 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:07:55.027 INFO:teuthology.orchestra.run.smithi117.stdout:Total 5.9 MB/s | 38 MB 00:06 2024-06-11T19:07:55.106 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-42 5.2 MB/s | 7.4 MB 00:01 2024-06-11T19:07:55.164 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.4 MB/s | 3.0 MB 00:01 2024-06-11T19:07:55.385 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:07:55.413 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:07:55.414 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:07:55.648 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): openblas-0.3.21-2.el9.x86_64.rpm 54 kB/s | 35 kB 00:00 2024-06-11T19:07:55.838 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:07:55.838 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:07:55.990 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 296 kB/s | 244 kB 00:00 2024-06-11T19:07:56.356 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 3.8 MB/s | 4.7 MB 00:01 2024-06-11T19:07:56.597 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:07:56.673 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-11T19:07:56.712 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-11T19:07:56.801 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-11T19:07:56.846 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-11T19:07:56.873 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 284 kB/s | 250 kB 00:00 2024-06-11T19:07:56.948 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.9 MB/s | 5.1 MB 00:01 2024-06-11T19:07:57.023 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-11T19:07:57.053 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-11T19:07:57.130 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-11T19:07:57.774 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-11T19:07:57.899 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-11T19:07:57.907 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-06-11T19:07:57.910 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:07:57.910 INFO:teuthology.orchestra.run.smithi098.stdout:Total 8.7 MB/s | 38 MB 00:04 2024-06-11T19:07:58.257 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:07:58.283 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:07:58.283 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:07:58.703 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:07:58.704 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:07:59.109 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-11T19:07:59.451 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:07:59.523 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-11T19:07:59.563 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-11T19:07:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-11T19:07:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-11T19:07:59.890 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-11T19:07:59.922 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-11T19:08:00.005 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-11T19:08:00.165 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-11T19:08:00.209 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-4216.g37bbe 12/12 2024-06-11T19:08:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-11T19:08:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-11T19:08:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-11T19:08:02.716 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-4216.g37bbe 12/12 2024-06-11T19:08:02.716 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-4216.g37bbe 1/12 2024-06-11T19:08:02.716 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-11T19:08:02.716 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-11T19:08:02.716 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-11T19:08:02.716 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-11T19:08:02.716 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-11T19:08:02.716 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-11T19:08:02.716 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-11T19:08:02.717 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-11T19:08:02.717 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-11T19:08:02.717 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-11T19:08:03.053 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-11T19:08:03.086 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-4216.g37bbe 12/12 2024-06-11T19:08:03.143 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-11T19:08:03.143 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:03.143 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:08:03.143 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:03.144 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:03.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-rook 2024-06-11T19:08:04.113 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:22 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:04.241 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:04.242 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:04.242 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-06-11T19:08:04.242 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:04.242 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:08:04.242 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 49 k 2024-06-11T19:08:04.242 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-06-11T19:08:04.242 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:04.243 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:08:04.244 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:04.244 INFO:teuthology.orchestra.run.smithi117.stdout:Install 13 Packages 2024-06-11T19:08:04.244 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:04.244 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 2.2 M 2024-06-11T19:08:04.244 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 26 M 2024-06-11T19:08:04.244 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:08:04.434 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] python3-jsonpatch-1.21-16.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-16.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:08:04.434 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] python3-jsonpointer-2.0-4.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-2.0-4.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:08:04.544 INFO:teuthology.orchestra.run.smithi117.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 102 kB/s | 19 kB 00:00 2024-06-11T19:08:04.569 INFO:teuthology.orchestra.run.smithi117.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 121 kB/s | 26 kB 00:00 2024-06-11T19:08:04.569 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] python3-oauthlib-3.1.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-oauthlib-3.1.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:08:04.570 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] python3-pyasn1-0.4.8-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.4.8-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:08:04.594 INFO:teuthology.orchestra.run.smithi117.stdout:(3/13): ceph-mgr-rook-19.0.0-4216.g37bbe1d8.el9 208 kB/s | 49 kB 00:00 2024-06-11T19:08:04.695 INFO:teuthology.orchestra.run.smithi117.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-06-11T19:08:04.711 INFO:teuthology.orchestra.run.smithi117.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-06-11T19:08:04.737 INFO:teuthology.orchestra.run.smithi117.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 1.3 MB/s | 32 kB 00:00 2024-06-11T19:08:04.762 INFO:teuthology.orchestra.run.smithi117.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 798 kB/s | 54 kB 00:00 2024-06-11T19:08:04.787 INFO:teuthology.orchestra.run.smithi117.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 297 kB/s | 15 kB 00:00 2024-06-11T19:08:04.812 INFO:teuthology.orchestra.run.smithi117.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-06-11T19:08:04.863 INFO:teuthology.orchestra.run.smithi117.stdout:(10/13): python3-google-auth-2.29.0-2.el9.noarc 2.0 MB/s | 205 kB 00:00 2024-06-11T19:08:04.888 INFO:teuthology.orchestra.run.smithi117.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.9 MB/s | 90 kB 00:00 2024-06-11T19:08:04.913 INFO:teuthology.orchestra.run.smithi117.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-06-11T19:08:04.946 INFO:teuthology.orchestra.run.smithi117.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-06-11T19:08:04.949 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:04.949 INFO:teuthology.orchestra.run.smithi117.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-06-11T19:08:05.014 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:08:05.029 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:08:05.029 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:08:05.117 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:08:05.117 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:08:05.592 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-4216.g37bbe 12/12 2024-06-11T19:08:05.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-4216.g37bbe 1/12 2024-06-11T19:08:05.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-11T19:08:05.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-11T19:08:05.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-11T19:08:05.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-11T19:08:05.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-11T19:08:05.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-11T19:08:05.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-11T19:08:05.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-11T19:08:05.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-11T19:08:05.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-11T19:08:05.621 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:08:05.786 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-11T19:08:05.840 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-11T19:08:05.904 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-11T19:08:05.941 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-11T19:08:05.982 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-11T19:08:06.024 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-11T19:08:06.024 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:06.024 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:08:06.024 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:08:06.024 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-11T19:08:06.024 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-11T19:08:06.024 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-11T19:08:06.024 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-11T19:08:06.024 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-11T19:08:06.024 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-11T19:08:06.024 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-11T19:08:06.024 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-11T19:08:06.025 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-11T19:08:06.025 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-11T19:08:06.025 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-11T19:08:06.025 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:06.025 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:06.099 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-11T19:08:06.220 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 7/13 2024-06-11T19:08:06.267 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-11T19:08:06.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-rook 2024-06-11T19:08:06.991 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-11T19:08:07.007 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:21 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:07.039 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-11T19:08:07.073 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-11T19:08:07.114 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-11T19:08:07.137 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:07.138 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:07.138 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-06-11T19:08:07.138 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:07.138 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:08:07.138 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 49 k 2024-06-11T19:08:07.138 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:07.139 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:08:07.140 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:07.140 INFO:teuthology.orchestra.run.smithi098.stdout:Install 13 Packages 2024-06-11T19:08:07.140 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:07.140 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.2 M 2024-06-11T19:08:07.140 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-06-11T19:08:07.140 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:08:07.143 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-rook-2:19.0.0-4216.g37bbe1d8.el9.noarch 13/13 2024-06-11T19:08:07.516 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-jsonpatch-1.21-16.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-16.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:08:07.516 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-jsonpointer-2.0-4.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-2.0-4.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:08:07.606 INFO:teuthology.orchestra.run.smithi098.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 152 kB/s | 26 kB 00:00 2024-06-11T19:08:07.631 INFO:teuthology.orchestra.run.smithi098.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 99 kB/s | 19 kB 00:00 2024-06-11T19:08:07.632 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-oauthlib-3.1.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-oauthlib-3.1.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:08:07.632 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-pyasn1-0.4.8-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.4.8-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:08:07.747 INFO:teuthology.orchestra.run.smithi098.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-06-11T19:08:07.773 INFO:teuthology.orchestra.run.smithi098.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-06-11T19:08:07.798 INFO:teuthology.orchestra.run.smithi098.stdout:(5/13): ceph-mgr-rook-19.0.0-4216.g37bbe1d8.el9 137 kB/s | 49 kB 00:00 2024-06-11T19:08:07.823 INFO:teuthology.orchestra.run.smithi098.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 1.3 MB/s | 32 kB 00:00 2024-06-11T19:08:07.849 INFO:teuthology.orchestra.run.smithi098.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 710 kB/s | 54 kB 00:00 2024-06-11T19:08:07.874 INFO:teuthology.orchestra.run.smithi098.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.2 MB/s | 279 kB 00:00 2024-06-11T19:08:07.899 INFO:teuthology.orchestra.run.smithi098.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 198 kB/s | 15 kB 00:00 2024-06-11T19:08:07.907 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-4216.g37bbe1d8.el9.noarch 13/13 2024-06-11T19:08:07.907 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-rook-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/13 2024-06-11T19:08:07.907 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-11T19:08:07.907 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-11T19:08:07.907 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-11T19:08:07.907 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-11T19:08:07.907 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-11T19:08:07.907 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-11T19:08:07.907 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-11T19:08:07.907 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-11T19:08:07.907 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 10/13 2024-06-11T19:08:07.908 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-11T19:08:07.908 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-11T19:08:07.958 INFO:teuthology.orchestra.run.smithi098.stdout:(10/13): python3-google-auth-2.29.0-2.el9.noarc 1.8 MB/s | 205 kB 00:00 2024-06-11T19:08:07.991 INFO:teuthology.orchestra.run.smithi098.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.8 MB/s | 90 kB 00:00 2024-06-11T19:08:08.017 INFO:teuthology.orchestra.run.smithi098.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-06-11T19:08:08.050 INFO:teuthology.orchestra.run.smithi098.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-06-11T19:08:08.053 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:08.053 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-06-11T19:08:08.117 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:08:08.132 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:08:08.132 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:08:08.218 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:08:08.219 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:08:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-11T19:08:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:08.226 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:08:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:08:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-11T19:08:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-11T19:08:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-11T19:08:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-11T19:08:08.226 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-11T19:08:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-11T19:08:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-11T19:08:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-11T19:08:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-11T19:08:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-11T19:08:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-11T19:08:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-11T19:08:08.227 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:08.227 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:08.482 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-cephadm 2024-06-11T19:08:08.735 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:08:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-11T19:08:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-11T19:08:09.040 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-11T19:08:09.064 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:27 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-11T19:08:09.128 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-11T19:08:09.192 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:09.193 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:09.193 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-06-11T19:08:09.193 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:09.193 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:08:09.193 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 143 k 2024-06-11T19:08:09.193 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-06-11T19:08:09.193 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-11T19:08:09.193 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-11T19:08:09.193 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-11T19:08:09.194 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-11T19:08:09.194 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-11T19:08:09.194 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:09.194 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:08:09.194 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:09.194 INFO:teuthology.orchestra.run.smithi117.stdout:Install 6 Packages 2024-06-11T19:08:09.194 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:09.195 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 7.0 M 2024-06-11T19:08:09.195 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 33 M 2024-06-11T19:08:09.195 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:08:09.229 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-11T19:08:09.332 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 7/13 2024-06-11T19:08:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-11T19:08:09.976 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] python3-babel-2.9.1-2.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-babel-2.9.1-2.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:08:09.976 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:08:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-11T19:08:10.152 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-11T19:08:10.157 INFO:teuthology.orchestra.run.smithi117.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 955 kB/s | 248 kB 00:00 2024-06-11T19:08:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-11T19:08:10.191 INFO:teuthology.orchestra.run.smithi117.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 16 MB/s | 548 kB 00:00 2024-06-11T19:08:10.216 INFO:teuthology.orchestra.run.smithi117.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-06-11T19:08:10.233 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-11T19:08:10.241 INFO:teuthology.orchestra.run.smithi117.stdout:(4/6): ceph-mgr-cephadm-19.0.0-4216.g37bbe1d8.e 415 kB/s | 143 kB 00:00 2024-06-11T19:08:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-4216.g37bbe1d8.el9.noarch 13/13 2024-06-11T19:08:10.267 INFO:teuthology.orchestra.run.smithi117.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-06-11T19:08:10.384 INFO:teuthology.orchestra.run.smithi117.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-06-11T19:08:10.385 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:10.386 INFO:teuthology.orchestra.run.smithi117.stdout:Total 5.9 MB/s | 7.0 MB 00:01 2024-06-11T19:08:10.455 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:08:10.466 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:08:10.467 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:08:10.564 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:08:10.564 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:08:10.868 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:08:11.025 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-11T19:08:11.034 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-4216.g37bbe1d8.el9.noarch 13/13 2024-06-11T19:08:11.034 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/13 2024-06-11T19:08:11.035 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-11T19:08:11.035 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-11T19:08:11.035 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-11T19:08:11.035 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-11T19:08:11.035 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-11T19:08:11.035 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-11T19:08:11.035 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-11T19:08:11.035 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-11T19:08:11.035 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 10/13 2024-06-11T19:08:11.035 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-11T19:08:11.035 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-11T19:08:11.077 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-11T19:08:11.354 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-11T19:08:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-11T19:08:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-11T19:08:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-11T19:08:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:11.356 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:11.601 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-11T19:08:11.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-cephadm 2024-06-11T19:08:11.705 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-11T19:08:11.768 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-11T19:08:11.797 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 6/6 2024-06-11T19:08:12.189 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:26 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:12.316 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:12.316 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-06-11T19:08:12.317 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:12.317 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:08:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 143 k 2024-06-11T19:08:12.317 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-06-11T19:08:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-11T19:08:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-11T19:08:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-11T19:08:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-11T19:08:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-11T19:08:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:12.317 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:08:12.317 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:12.318 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-06-11T19:08:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:12.318 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 7.0 M 2024-06-11T19:08:12.318 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 33 M 2024-06-11T19:08:12.318 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:08:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 6/6 2024-06-11T19:08:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/6 2024-06-11T19:08:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-11T19:08:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-11T19:08:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-11T19:08:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-11T19:08:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-11T19:08:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:12.806 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:08:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:08:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-11T19:08:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-11T19:08:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-11T19:08:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-11T19:08:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-11T19:08:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:12.807 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:12.881 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-babel-2.9.1-2.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-babel-2.9.1-2.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:08:12.883 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-11T19:08:13.048 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-fuse 2024-06-11T19:08:13.070 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): ceph-mgr-cephadm-19.0.0-4216.g37bbe1d8.e 537 kB/s | 143 kB 00:00 2024-06-11T19:08:13.104 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 16 MB/s | 548 kB 00:00 2024-06-11T19:08:13.129 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-06-11T19:08:13.154 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.3 MB/s | 51 kB 00:00 2024-06-11T19:08:13.596 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 313 kB/s | 248 kB 00:00 2024-06-11T19:08:13.626 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:31 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:13.750 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:13.751 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-06-11T19:08:13.751 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:13.751 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:08:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 893 k 2024-06-11T19:08:13.751 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-06-11T19:08:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-11T19:08:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:13.751 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:08:13.751 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:13.752 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-06-11T19:08:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:13.752 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 973 k 2024-06-11T19:08:13.752 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 2.8 M 2024-06-11T19:08:13.752 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:08:13.932 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:08:14.027 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 620 kB/s | 80 kB 00:00 2024-06-11T19:08:14.032 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.9 MB/s | 6.0 MB 00:01 2024-06-11T19:08:14.033 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:14.033 INFO:teuthology.orchestra.run.smithi098.stdout:Total 4.1 MB/s | 7.0 MB 00:01 2024-06-11T19:08:14.104 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:08:14.115 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:08:14.115 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:08:14.213 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:08:14.213 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:08:14.410 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-fuse-19.0.0-4216.g37bbe1d8.el9.x86_ 1.7 MB/s | 893 kB 00:00 2024-06-11T19:08:14.411 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:14.412 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.4 MB/s | 973 kB 00:00 2024-06-11T19:08:14.424 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:08:14.446 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:08:14.446 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:08:14.524 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:08:14.524 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:08:14.535 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:08:14.718 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-11T19:08:14.721 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:08:14.773 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-11T19:08:14.854 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-11T19:08:14.918 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2/2 2024-06-11T19:08:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-11T19:08:15.385 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-11T19:08:15.448 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-11T19:08:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 6/6 2024-06-11T19:08:15.568 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2/2 2024-06-11T19:08:15.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/2 2024-06-11T19:08:15.835 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-11T19:08:15.835 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:15.835 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:08:15.835 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-11T19:08:15.835 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:15.835 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:16.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-volume 2024-06-11T19:08:16.231 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 6/6 2024-06-11T19:08:16.231 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/6 2024-06-11T19:08:16.231 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-11T19:08:16.231 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-11T19:08:16.231 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-11T19:08:16.231 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-11T19:08:16.504 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-11T19:08:16.504 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:16.504 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:08:16.504 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:08:16.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-11T19:08:16.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-11T19:08:16.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-11T19:08:16.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-11T19:08:16.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-11T19:08:16.504 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:16.505 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:16.673 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:34 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:16.765 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-fuse 2024-06-11T19:08:16.805 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:16.806 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:16.806 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-06-11T19:08:16.806 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:16.806 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:08:16.806 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 273 k 2024-06-11T19:08:16.806 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:16.806 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:08:16.806 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:16.806 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-06-11T19:08:16.806 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:16.806 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 273 k 2024-06-11T19:08:16.807 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.3 M 2024-06-11T19:08:16.807 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:08:17.048 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-volume-19.0.0-4216.g37bbe1d8.el9.noarch.rp 1.1 MB/s | 273 kB 00:00 2024-06-11T19:08:17.049 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:17.049 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.1 MB/s | 273 kB 00:00 2024-06-11T19:08:17.049 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:08:17.057 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:08:17.058 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:08:17.087 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:08:17.087 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:08:17.300 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:08:17.334 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:31 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:17.376 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-volume-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:08:17.392 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-volume-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:08:17.392 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:08:17.392 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-11T19:08:17.392 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:17.459 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:17.459 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:17.459 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-06-11T19:08:17.459 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:17.459 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:08:17.459 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 893 k 2024-06-11T19:08:17.459 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-06-11T19:08:17.460 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-11T19:08:17.460 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:17.460 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:08:17.460 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:17.460 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-06-11T19:08:17.460 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:17.460 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 973 k 2024-06-11T19:08:17.460 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.8 M 2024-06-11T19:08:17.461 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:08:17.562 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-11T19:08:17.690 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 493 kB/s | 80 kB 00:00 2024-06-11T19:08:17.974 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-fuse-19.0.0-4216.g37bbe1d8.el9.x86_ 2.0 MB/s | 893 kB 00:00 2024-06-11T19:08:17.975 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:17.975 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.8 MB/s | 973 kB 00:00 2024-06-11T19:08:17.987 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:08:18.009 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:08:18.009 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:08:18.088 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:08:18.088 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:08:18.294 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:08:18.333 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-volume-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:08:18.333 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:18.333 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:08:18.333 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:08:18.333 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:18.333 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:18.442 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-11T19:08:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2/2 2024-06-11T19:08:18.660 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados-devel 2024-06-11T19:08:19.172 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2/2 2024-06-11T19:08:19.172 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/2 2024-06-11T19:08:19.249 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:37 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:19.378 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 127 k 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 127 k 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 456 k 2024-06-11T19:08:19.379 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:08:19.454 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-11T19:08:19.454 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:19.454 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:08:19.454 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-11T19:08:19.454 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:19.454 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:19.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-volume 2024-06-11T19:08:19.795 INFO:teuthology.orchestra.run.smithi117.stdout:librados-devel-19.0.0-4216.g37bbe1d8.el9.x86_64 305 kB/s | 127 kB 00:00 2024-06-11T19:08:19.796 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:19.796 INFO:teuthology.orchestra.run.smithi117.stdout:Total 304 kB/s | 127 kB 00:00 2024-06-11T19:08:19.796 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:08:19.802 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:08:19.802 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:08:19.866 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:08:19.866 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:08:20.027 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:08:20.143 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:20.319 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:34 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:20.447 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:20.448 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:20.448 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-06-11T19:08:20.448 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:20.448 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:08:20.448 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-4216.g37bbe1d8.el9 ceph-noarch 273 k 2024-06-11T19:08:20.448 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:20.448 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:08:20.448 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:20.449 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-06-11T19:08:20.449 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:20.449 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 273 k 2024-06-11T19:08:20.449 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.3 M 2024-06-11T19:08:20.449 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:08:20.580 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:20.820 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:20.820 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:20.821 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:08:20.821 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:08:20.821 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:20.821 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:21.089 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs2 2024-06-11T19:08:21.171 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-volume-19.0.0-4216.g37bbe1d8.el9.noarch.rp 378 kB/s | 273 kB 00:00 2024-06-11T19:08:21.171 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:21.172 INFO:teuthology.orchestra.run.smithi098.stdout:Total 378 kB/s | 273 kB 00:00 2024-06-11T19:08:21.172 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:08:21.181 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:08:21.181 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:08:21.205 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:08:21.206 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:08:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:08:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:08:21.531 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:08:21.532 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:08:21.532 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-11T19:08:21.532 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:21.678 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:39 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:21.764 INFO:teuthology.orchestra.run.smithi117.stdout:Package libcephfs2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:21.808 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:21.809 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-06-11T19:08:21.809 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:21.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs-devel 2024-06-11T19:08:22.480 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:40 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-4216.g37bbe1d8.el9.noarch 1/1 2024-06-11T19:08:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:22.503 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:08:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-4216.g37bbe1d8.el9.noarch 2024-06-11T19:08:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:22.504 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:22.605 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:22.606 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:22.606 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-06-11T19:08:22.606 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:22.606 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:08:22.606 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 32 k 2024-06-11T19:08:22.606 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:22.606 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:08:22.606 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:22.606 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-06-11T19:08:22.607 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:22.607 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 32 k 2024-06-11T19:08:22.607 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 142 k 2024-06-11T19:08:22.607 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:08:22.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados-devel 2024-06-11T19:08:23.004 INFO:teuthology.orchestra.run.smithi117.stdout:libcephfs-devel-19.0.0-4216.g37bbe1d8.el9.x86_6 80 kB/s | 32 kB 00:00 2024-06-11T19:08:23.004 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:23.005 INFO:teuthology.orchestra.run.smithi117.stdout:Total 80 kB/s | 32 kB 00:00 2024-06-11T19:08:23.005 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:08:23.008 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:08:23.008 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:08:23.017 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:08:23.017 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:08:23.098 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:08:23.210 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:23.389 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:37 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:23.519 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:23.520 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-06-11T19:08:23.520 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:23.521 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:08:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 127 k 2024-06-11T19:08:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:23.521 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:08:23.521 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:23.521 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-06-11T19:08:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:23.521 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 127 k 2024-06-11T19:08:23.521 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 456 k 2024-06-11T19:08:23.521 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:08:23.631 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:23.835 INFO:teuthology.orchestra.run.smithi098.stdout:librados-devel-19.0.0-4216.g37bbe1d8.el9.x86_64 403 kB/s | 127 kB 00:00 2024-06-11T19:08:23.836 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:23.836 INFO:teuthology.orchestra.run.smithi098.stdout:Total 401 kB/s | 127 kB 00:00 2024-06-11T19:08:23.836 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:08:23.842 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:08:23.842 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:08:23.861 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:23.862 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:23.862 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:08:23.862 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:08:23.862 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:23.862 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:23.904 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:08:23.905 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:08:24.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados2 2024-06-11T19:08:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:08:24.158 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:24.572 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:24.620 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:42 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:24.704 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:24.745 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:24.746 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-06-11T19:08:24.747 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:24.812 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librbd1 2024-06-11T19:08:24.835 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:24.835 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:24.836 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:08:24.836 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:08:24.836 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:24.836 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:25.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs2 2024-06-11T19:08:25.388 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:43 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:25.472 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:25.514 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:25.515 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-06-11T19:08:25.515 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:25.574 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rados 2024-06-11T19:08:25.717 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:39 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:25.804 INFO:teuthology.orchestra.run.smithi098.stdout:Package libcephfs2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:25.848 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:25.850 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-06-11T19:08:25.850 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:25.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs-devel 2024-06-11T19:08:26.168 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:44 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:26.263 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rados-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:26.306 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:26.307 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-06-11T19:08:26.307 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:26.366 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rgw 2024-06-11T19:08:26.493 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:40 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:26.619 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:26.620 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-06-11T19:08:26.620 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:26.620 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:08:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 32 k 2024-06-11T19:08:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:26.621 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:08:26.621 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:26.621 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-06-11T19:08:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:26.621 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 32 k 2024-06-11T19:08:26.621 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 142 k 2024-06-11T19:08:26.621 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:08:26.869 INFO:teuthology.orchestra.run.smithi098.stdout:libcephfs-devel-19.0.0-4216.g37bbe1d8.el9.x86_6 128 kB/s | 32 kB 00:00 2024-06-11T19:08:26.870 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:26.870 INFO:teuthology.orchestra.run.smithi098.stdout:Total 127 kB/s | 32 kB 00:00 2024-06-11T19:08:26.870 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:08:26.873 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:08:26.874 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:08:26.885 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:08:26.885 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:08:26.958 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:44 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:26.963 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:08:27.054 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:27.097 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:27.098 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-06-11T19:08:27.098 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:27.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-cephfs 2024-06-11T19:08:27.466 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:27.684 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:08:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:08:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:27.684 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:27.747 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:45 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:27.830 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-cephfs-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:27.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados2 2024-06-11T19:08:27.872 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:27.873 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-06-11T19:08:27.873 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:27.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rbd 2024-06-11T19:08:28.434 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:42 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:28.519 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:28.519 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:46 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:28.562 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:28.563 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-06-11T19:08:28.563 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:28.603 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:28.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librbd1 2024-06-11T19:08:28.645 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:28.646 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-06-11T19:08:28.646 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:28.709 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-fuse 2024-06-11T19:08:29.199 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:43 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:29.282 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:47 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-06-11T19:08:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:29.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rados 2024-06-11T19:08:29.409 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:29.410 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:29.410 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-06-11T19:08:29.410 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:29.410 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:08:29.410 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 85 k 2024-06-11T19:08:29.410 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:29.410 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:08:29.410 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:29.410 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-06-11T19:08:29.410 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:29.411 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 85 k 2024-06-11T19:08:29.411 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 231 k 2024-06-11T19:08:29.411 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:08:29.616 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-fuse-19.0.0-4216.g37bbe1d8.el9.x86_64.rpm 416 kB/s | 85 kB 00:00 2024-06-11T19:08:29.616 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:29.617 INFO:teuthology.orchestra.run.smithi117.stdout:Total 414 kB/s | 85 kB 00:00 2024-06-11T19:08:29.617 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:08:29.624 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:08:29.624 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:08:29.686 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:08:29.687 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:08:29.827 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:08:29.928 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:29.968 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:43 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:30.052 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rados-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:30.095 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:30.096 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-06-11T19:08:30.096 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:30.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rgw 2024-06-11T19:08:30.324 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:30.597 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:30.597 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:30.597 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:08:30.597 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:08:30.597 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:30.597 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:30.736 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:44 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:30.820 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rgw-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:30.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-mirror 2024-06-11T19:08:30.863 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:30.864 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-06-11T19:08:30.864 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:30.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-cephfs 2024-06-11T19:08:31.430 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:49 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:31.496 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:45 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:31.560 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:31.560 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:31.560 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-06-11T19:08:31.561 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:31.561 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:08:31.561 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 3.1 M 2024-06-11T19:08:31.561 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:31.561 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:08:31.561 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:31.561 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-06-11T19:08:31.561 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:31.561 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 3.1 M 2024-06-11T19:08:31.561 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 13 M 2024-06-11T19:08:31.561 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:08:31.579 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-cephfs-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:31.621 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:31.622 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-06-11T19:08:31.622 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:31.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rbd 2024-06-11T19:08:31.956 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-mirror-19.0.0-4216.g37bbe1d8.el9.x86_64.rpm 7.8 MB/s | 3.1 MB 00:00 2024-06-11T19:08:31.957 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:31.957 INFO:teuthology.orchestra.run.smithi117.stdout:Total 7.8 MB/s | 3.1 MB 00:00 2024-06-11T19:08:31.957 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:08:31.967 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:08:31.967 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:08:32.048 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:08:32.048 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:08:32.254 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:46 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:32.337 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 is already installed. 2024-06-11T19:08:32.378 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:32.380 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-06-11T19:08:32.380 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:32.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-fuse 2024-06-11T19:08:32.475 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:08:32.533 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:32.559 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:32.559 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:08:32.559 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-11T19:08:32.559 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-11T19:08:32.559 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-11T19:08:32.559 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:33.019 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:47 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:33.147 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:33.148 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-06-11T19:08:33.148 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:33.148 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:08:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 85 k 2024-06-11T19:08:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:33.148 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:08:33.148 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:33.148 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-06-11T19:08:33.148 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:33.148 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 85 k 2024-06-11T19:08:33.149 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 231 k 2024-06-11T19:08:33.149 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:08:33.349 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-fuse-19.0.0-4216.g37bbe1d8.el9.x86_64.rpm 427 kB/s | 85 kB 00:00 2024-06-11T19:08:33.350 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:33.350 INFO:teuthology.orchestra.run.smithi098.stdout:Total 424 kB/s | 85 kB 00:00 2024-06-11T19:08:33.350 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:08:33.357 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:08:33.357 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:08:33.419 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:08:33.420 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:08:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:08:33.663 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:33.670 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:33.670 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:33.670 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:08:33.670 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:08:33.670 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:33.670 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:33.898 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-nbd 2024-06-11T19:08:34.031 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:34.294 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:34.294 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:34.294 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:08:34.294 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:08:34.295 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:34.295 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:34.490 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:52 ago on Tue 11 Jun 2024 07:06:42 PM UTC. 2024-06-11T19:08:34.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror 2024-06-11T19:08:34.618 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-06-11T19:08:34.619 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:34.619 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-06-11T19:08:34.619 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:34.619 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-06-11T19:08:34.619 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 170 k 2024-06-11T19:08:34.619 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:34.619 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-06-11T19:08:34.619 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-06-11T19:08:34.619 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-06-11T19:08:34.619 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:34.619 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 170 k 2024-06-11T19:08:34.620 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 494 k 2024-06-11T19:08:34.620 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-06-11T19:08:35.152 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:49 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:35.223 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-nbd-19.0.0-4216.g37bbe1d8.el9.x86_64.rpm 282 kB/s | 170 kB 00:00 2024-06-11T19:08:35.224 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:35.224 INFO:teuthology.orchestra.run.smithi117.stdout:Total 282 kB/s | 170 kB 00:00 2024-06-11T19:08:35.224 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-06-11T19:08:35.231 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-06-11T19:08:35.231 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-06-11T19:08:35.285 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:35.286 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-06-11T19:08:35.287 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:35.287 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:08:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 3.1 M 2024-06-11T19:08:35.287 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:35.288 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:08:35.288 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:35.288 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-06-11T19:08:35.288 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:35.288 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 3.1 M 2024-06-11T19:08:35.288 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 13 M 2024-06-11T19:08:35.289 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:08:35.299 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-06-11T19:08:35.300 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-06-11T19:08:35.463 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-06-11T19:08:35.545 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:35.969 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-nbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:36.096 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-mirror-19.0.0-4216.g37bbe1d8.el9.x86_64.rpm 3.8 MB/s | 3.1 MB 00:00 2024-06-11T19:08:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:Total 3.8 MB/s | 3.1 MB 00:00 2024-06-11T19:08:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:08:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:08:36.107 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:08:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:08:36.189 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:08:36.219 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:36.219 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:36.219 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-06-11T19:08:36.219 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:08:36.219 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:36.219 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-06-11T19:08:36.453 DEBUG:teuthology.parallel:result is None 2024-06-11T19:08:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:08:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:36.731 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-11T19:08:36.731 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-11T19:08:36.732 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-11T19:08:36.732 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-11T19:08:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:37.777 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:08:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:08:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:37.778 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:37.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd 2024-06-11T19:08:38.573 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:52 ago on Tue 11 Jun 2024 07:06:46 PM UTC. 2024-06-11T19:08:38.703 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-06-11T19:08:38.704 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:38.704 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-06-11T19:08:38.704 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:38.704 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-06-11T19:08:38.704 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-4216.g37bbe1d8.el9 ceph 170 k 2024-06-11T19:08:38.704 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:38.704 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-06-11T19:08:38.704 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-06-11T19:08:38.704 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-06-11T19:08:38.704 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:38.704 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 170 k 2024-06-11T19:08:38.704 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 494 k 2024-06-11T19:08:38.705 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-06-11T19:08:39.376 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-nbd-19.0.0-4216.g37bbe1d8.el9.x86_64.rpm 253 kB/s | 170 kB 00:00 2024-06-11T19:08:39.376 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-06-11T19:08:39.377 INFO:teuthology.orchestra.run.smithi098.stdout:Total 253 kB/s | 170 kB 00:00 2024-06-11T19:08:39.377 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-06-11T19:08:39.385 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-06-11T19:08:39.385 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-06-11T19:08:39.454 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-06-11T19:08:39.454 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-06-11T19:08:39.612 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-06-11T19:08:39.724 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:40.166 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-nbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:40.425 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 1/1 2024-06-11T19:08:40.426 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:40.426 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-06-11T19:08:40.426 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-4216.g37bbe1d8.el9.x86_64 2024-06-11T19:08:40.426 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:40.426 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-06-11T19:08:40.656 DEBUG:teuthology.parallel:result is None 2024-06-11T19:08:40.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=37bbe1d8d788b7c9a71280e7d510c295a5853f0b 2024-06-11T19:08:40.883 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-11T19:08:40.904 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-4216.g37bbe1d8.el9 2024-06-11T19:08:40.905 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-4216.g37bbe1d8.el9 2024-06-11T19:08:40.905 INFO:teuthology.task.install:The correct ceph version 19.0.0-4216.g37bbe1d8 is installed. 2024-06-11T19:08:40.907 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=37bbe1d8d788b7c9a71280e7d510c295a5853f0b 2024-06-11T19:08:41.050 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-11T19:08:41.077 INFO:teuthology.orchestra.run.smithi117.stdout:19.0.0-4216.g37bbe1d8.el9 2024-06-11T19:08:41.077 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-4216.g37bbe1d8.el9 2024-06-11T19:08:41.078 INFO:teuthology.task.install:The correct ceph version 19.0.0-4216.g37bbe1d8 is installed. 2024-06-11T19:08:41.080 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-11T19:08:41.081 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:08:41.081 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-11T19:08:41.107 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:08:41.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-11T19:08:41.149 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-11T19:08:41.150 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:08:41.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-06-11T19:08:41.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-11T19:08:41.238 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:08:41.238 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-06-11T19:08:41.267 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-11T19:08:41.335 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-11T19:08:41.336 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:08:41.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-11T19:08:41.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-11T19:08:41.436 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:08:41.436 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-11T19:08:41.463 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-11T19:08:41.527 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-11T19:08:41.527 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:08:41.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-06-11T19:08:41.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-11T19:08:41.616 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:08:41.616 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-06-11T19:08:41.640 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-11T19:08:41.704 INFO:teuthology.run_tasks:Running task ceph... 2024-06-11T19:08:41.781 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-06-11T19:08:41.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2024-06-11T19:08:41.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /var/log/ceph 2024-06-11T19:08:41.807 INFO:tasks.ceph:Disabling ceph logrotate... 2024-06-11T19:08:41.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-11T19:08:41.848 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-11T19:08:41.871 INFO:tasks.ceph:Creating extra log directories... 2024-06-11T19:08:41.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-11T19:08:41.912 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-11T19:08:41.955 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-06-11T19:08:41.955 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-06-11T19:08:41.956 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861', 'branch': 'wip-lflores-testing-2-2024-06-10-1925', 'description': 'rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests}', 'email': None, 'first_in_suite': False, 'job_id': '7749861', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lflores-testing-2-2024-06-10-1925'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'sha1': '37bbe1d8d788b7c9a71280e7d510c295a5853f0b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '37bbe1d8d788b7c9a71280e7d510c295a5853f0b'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lflores-testing-2-2024-06-10-1925', 'sha1': '37bbe1d8d788b7c9a71280e7d510c295a5853f0b'}}, 'owner': 'scheduled_lflores@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'], ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 7590, 'sha1': '37bbe1d8d788b7c9a71280e7d510c295a5853f0b', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-lflores-testing-2-2024-06-10-1925', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '37bbe1d8d788b7c9a71280e7d510c295a5853f0b', 'targets': {'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK0TZDceOPUGWUQRpUOa8Qwzy83q1SX5RxcIfpi+gjo81nXvkXCMRXobXA0qXn6sNtxm9amEOuS9vjHnFc4vFXg=', 'smithi117.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMXyP0QtX3p4dtUVOlrQCBBXLrLpo7du/6MR/BSjYpo0W7Qe1FnWlTVl1GM8ZQkMr1YCNDw+IlhiE3b7xhPp+K0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '861a8dcf7aa816a26e13f039336f7ed0a9aec0fa', 'timestamp': '2024-06-10_22:21:08', 'tube': 'smithi', 'user': 'lflores', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841'} 2024-06-11T19:08:41.957 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-06-11T19:08:41.977 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-06-11T19:08:41.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-06-11T19:08:42.033 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/run/ceph 2024-06-11T19:08:42.058 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:08:42.058 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-06-11T19:08:42.111 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-11T19:08:42.111 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-06-11T19:08:42.165 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-11T19:08:42.166 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-11T19:08:42.166 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 673 Links: 1 2024-06-11T19:08:42.166 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-11T19:08:42.166 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-06-11T19:08:42.166 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-06-11 19:08:36.975403465 +0000 2024-06-11T19:08:42.166 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-06-11 19:05:37.056791750 +0000 2024-06-11T19:08:42.166 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-06-11 19:05:37.056791750 +0000 2024-06-11T19:08:42.166 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-06-11 19:05:37.056791750 +0000 2024-06-11T19:08:42.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-11T19:08:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-06-11T19:08:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-06-11T19:08:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000260673 s, 2.0 MB/s 2024-06-11T19:08:42.229 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-11T19:08:42.286 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-06-11T19:08:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-11T19:08:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-11T19:08:42.341 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 692 Links: 1 2024-06-11T19:08:42.341 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-11T19:08:42.342 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-06-11T19:08:42.342 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-06-11 19:08:36.976403454 +0000 2024-06-11T19:08:42.342 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-06-11 19:05:37.388788167 +0000 2024-06-11T19:08:42.342 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-06-11 19:05:37.388788167 +0000 2024-06-11T19:08:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-06-11 19:05:37.388788167 +0000 2024-06-11T19:08:42.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-11T19:08:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-06-11T19:08:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-06-11T19:08:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000179071 s, 2.9 MB/s 2024-06-11T19:08:42.405 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-11T19:08:42.461 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-06-11T19:08:42.517 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-11T19:08:42.517 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-11T19:08:42.517 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 710 Links: 1 2024-06-11T19:08:42.517 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-11T19:08:42.517 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-06-11T19:08:42.517 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-06-11 19:08:36.976403454 +0000 2024-06-11T19:08:42.517 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-06-11 19:05:37.727784508 +0000 2024-06-11T19:08:42.517 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-06-11 19:05:37.727784508 +0000 2024-06-11T19:08:42.517 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-06-11 19:05:37.727784508 +0000 2024-06-11T19:08:42.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-11T19:08:42.579 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-06-11T19:08:42.579 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-06-11T19:08:42.579 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000207781 s, 2.5 MB/s 2024-06-11T19:08:42.580 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-11T19:08:42.637 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-06-11T19:08:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-11T19:08:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-11T19:08:42.692 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 731 Links: 1 2024-06-11T19:08:42.692 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-11T19:08:42.692 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-06-11T19:08:42.692 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-06-11 19:08:36.976403454 +0000 2024-06-11T19:08:42.692 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-06-11 19:05:38.097780516 +0000 2024-06-11T19:08:42.692 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-06-11 19:05:38.097780516 +0000 2024-06-11T19:08:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-06-11 19:05:38.097780516 +0000 2024-06-11T19:08:42.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-11T19:08:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-06-11T19:08:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-06-11T19:08:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000206942 s, 2.5 MB/s 2024-06-11T19:08:42.756 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-11T19:08:42.812 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-06-11T19:08:42.813 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:08:42.813 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-06-11T19:08:42.830 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-11T19:08:42.830 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-06-11T19:08:42.887 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-11T19:08:42.887 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-11T19:08:42.887 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-06-11T19:08:42.887 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-11T19:08:42.887 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-06-11T19:08:42.887 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-06-11 19:08:32.811515472 +0000 2024-06-11T19:08:42.887 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-06-11 19:05:33.629816119 +0000 2024-06-11T19:08:42.887 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-06-11 19:05:33.629816119 +0000 2024-06-11T19:08:42.887 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-06-11 19:05:33.629816119 +0000 2024-06-11T19:08:42.887 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-11T19:08:42.952 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-06-11T19:08:42.952 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-06-11T19:08:42.952 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000229913 s, 2.2 MB/s 2024-06-11T19:08:42.953 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-11T19:08:43.010 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-06-11T19:08:43.066 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-11T19:08:43.066 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-11T19:08:43.066 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-06-11T19:08:43.067 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-11T19:08:43.067 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-06-11T19:08:43.067 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-06-11 19:08:32.812515452 +0000 2024-06-11T19:08:43.067 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-06-11 19:05:33.976808563 +0000 2024-06-11T19:08:43.067 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-06-11 19:05:33.976808563 +0000 2024-06-11T19:08:43.067 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-06-11 19:05:33.976808563 +0000 2024-06-11T19:08:43.067 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-11T19:08:43.131 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-06-11T19:08:43.131 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-06-11T19:08:43.131 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.0002054 s, 2.5 MB/s 2024-06-11T19:08:43.132 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-11T19:08:43.189 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-06-11T19:08:43.246 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-11T19:08:43.246 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-11T19:08:43.247 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-06-11T19:08:43.247 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-11T19:08:43.247 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-06-11T19:08:43.247 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-06-11 19:08:32.812515452 +0000 2024-06-11T19:08:43.247 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-06-11 19:05:34.311801267 +0000 2024-06-11T19:08:43.247 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-06-11 19:05:34.311801267 +0000 2024-06-11T19:08:43.247 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-06-11 19:05:34.311801267 +0000 2024-06-11T19:08:43.247 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-11T19:08:43.309 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-06-11T19:08:43.310 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-06-11T19:08:43.310 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000208317 s, 2.5 MB/s 2024-06-11T19:08:43.311 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-11T19:08:43.367 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-06-11T19:08:43.423 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-11T19:08:43.423 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-11T19:08:43.423 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-06-11T19:08:43.423 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-11T19:08:43.423 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-06-11T19:08:43.423 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-06-11 19:08:32.813515432 +0000 2024-06-11T19:08:43.423 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-06-11 19:05:34.643794038 +0000 2024-06-11T19:08:43.423 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-06-11 19:05:34.643794038 +0000 2024-06-11T19:08:43.423 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-06-11 19:05:34.643794038 +0000 2024-06-11T19:08:43.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-11T19:08:43.486 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-06-11T19:08:43.486 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-06-11T19:08:43.486 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000211555 s, 2.4 MB/s 2024-06-11T19:08:43.487 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-11T19:08:43.544 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-06-11T19:08:43.544 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-06-11T19:08:43.544 INFO:tasks.ceph:Generating config... 2024-06-11T19:08:43.545 INFO:tasks.ceph:[client] debug monc = 20 2024-06-11T19:08:43.545 INFO:tasks.ceph:[client] debug ms = 1 2024-06-11T19:08:43.546 INFO:tasks.ceph:[global] debug ms = 1 2024-06-11T19:08:43.546 INFO:tasks.ceph:[global] debug objecter = 20 2024-06-11T19:08:43.546 INFO:tasks.ceph:[global] debug rados = 20 2024-06-11T19:08:43.546 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-06-11T19:08:43.546 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-06-11T19:08:43.546 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-06-11T19:08:43.546 INFO:tasks.ceph:[global] ms type = async 2024-06-11T19:08:43.546 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-06-11T19:08:43.546 INFO:tasks.ceph:[mgr] debug ms = 1 2024-06-11T19:08:43.546 INFO:tasks.ceph:[mon] debug mon = 20 2024-06-11T19:08:43.546 INFO:tasks.ceph:[mon] debug ms = 1 2024-06-11T19:08:43.546 INFO:tasks.ceph:[mon] debug paxos = 20 2024-06-11T19:08:43.546 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-06-11T19:08:43.547 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-06-11T19:08:43.547 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-06-11T19:08:43.547 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-06-11T19:08:43.547 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-06-11T19:08:43.547 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-06-11T19:08:43.547 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-06-11T19:08:43.547 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-06-11T19:08:43.547 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-06-11T19:08:43.547 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-06-11T19:08:43.547 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-06-11T19:08:43.547 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-06-11T19:08:43.547 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-06-11T19:08:43.547 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-06-11T19:08:43.547 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-06-11T19:08:43.547 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-06-11T19:08:43.548 INFO:tasks.ceph:[osd] debug ms = 1 2024-06-11T19:08:43.548 INFO:tasks.ceph:[osd] debug osd = 20 2024-06-11T19:08:43.548 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-06-11T19:08:43.548 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-06-11T19:08:43.548 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-06-11T19:08:43.548 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-06-11T19:08:43.548 INFO:tasks.ceph:[osd] osd class default list = * 2024-06-11T19:08:43.548 INFO:tasks.ceph:[osd] osd class load list = * 2024-06-11T19:08:43.548 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-06-11T19:08:43.548 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-06-11T19:08:43.549 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-06-11T19:08:43.549 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-06-11T19:08:43.549 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-06-11T19:08:43.549 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-06-11T19:08:43.549 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-06-11T19:08:43.549 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-06-11T19:08:43.549 INFO:tasks.ceph:Setting up mon.a... 2024-06-11T19:08:43.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-06-11T19:08:43.620 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.keyring 2024-06-11T19:08:43.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-06-11T19:08:43.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-11T19:08:43.733 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.98'), ('mon.b', '[v2:172.21.15.98:3301,v1:172.21.15.98:6790]'), ('mon.c', '[v2:172.21.15.98:3302,v1:172.21.15.98:6791]'), ('mon.d', '[v2:172.21.15.98:3303,v1:172.21.15.98:6792]'), ('mon.e', '[v2:172.21.15.98:3304,v1:172.21.15.98:6793]'), ('mon.f', '172.21.15.117'), ('mon.g', '[v2:172.21.15.117:3301,v1:172.21.15.117:6790]'), ('mon.h', '[v2:172.21.15.117:3302,v1:172.21.15.117:6791]'), ('mon.i', '[v2:172.21.15.117:3303,v1:172.21.15.117:6792]')] 2024-06-11T19:08:43.733 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790],[v2:172.21.15.98:3302,v1:172.21.15.98:6791],[v2:172.21.15.98:3303,v1:172.21.15.98:6792],[v2:172.21.15.98:3304,v1:172.21.15.98:6793],172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790],[v2:172.21.15.117:3302,v1:172.21.15.117:6791],[v2:172.21.15.117:3303,v1:172.21.15.117:6792]', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mon.d': {}, 'mon.e': {}, 'mon.f': {}, 'mon.g': {}, 'mon.h': {}, 'mon.i': {}} 2024-06-11T19:08:43.734 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:08:43.734 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-06-11T19:08:43.787 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.98 --addv b '[v2:172.21.15.98:3301,v1:172.21.15.98:6790]' --addv c '[v2:172.21.15.98:3302,v1:172.21.15.98:6791]' --addv d '[v2:172.21.15.98:3303,v1:172.21.15.98:6792]' --addv e '[v2:172.21.15.98:3304,v1:172.21.15.98:6793]' --add f 172.21.15.117 --addv g '[v2:172.21.15.117:3301,v1:172.21.15.117:6790]' --addv h '[v2:172.21.15.117:3302,v1:172.21.15.117:6791]' --addv i '[v2:172.21.15.117:3303,v1:172.21.15.117:6792]' --print /home/ubuntu/cephtest/ceph.monmap 2024-06-11T19:08:43.891 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: generated fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = quincy 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 0 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:min_mon_release 17 (quincy) 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:election_strategy: 3 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:3: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:08:43.892 INFO:teuthology.orchestra.run.smithi098.stdout:4: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:08:43.893 INFO:teuthology.orchestra.run.smithi098.stdout:5: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:08:43.893 INFO:teuthology.orchestra.run.smithi098.stdout:6: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:08:43.893 INFO:teuthology.orchestra.run.smithi098.stdout:7: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:08:43.893 INFO:teuthology.orchestra.run.smithi098.stdout:8: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:08:43.893 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (9 monitors) 2024-06-11T19:08:43.894 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-06-11T19:08:43.909 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9daa5251-9ae7-4a07-8f69-80112d2163e1... 2024-06-11T19:08:43.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-06-11T19:08:43.952 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-06-11T19:08:43.992 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-06-11T19:08:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-06-11T19:08:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-11T19:08:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-06-11T19:08:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-06-11T19:08:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-06-11T19:08:43.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-06-11T19:08:43.993 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-11T19:08:43.994 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-06-11T19:08:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-06-11T19:08:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-06-11T19:08:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-06-11T19:08:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-06-11T19:08:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790],[v2:172.21.15.98:3302,v1:172.21.15.98:6791],[v2:172.21.15.98:3303,v1:172.21.15.98:6792],[v2:172.21.15.98:3304,v1:172.21.15.98:6793],172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790],[v2:172.21.15.117:3302,v1:172.21.15.117:6791],[v2:172.21.15.117:3303,v1:172.21.15.117:6792]" 2024-06-11T19:08:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-06-11T19:08:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: debug objecter = 20 2024-06-11T19:08:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: debug rados = 20 2024-06-11T19:08:43.995 INFO:teuthology.orchestra.run.smithi098.stdout: mon client directed command retry = 5 2024-06-11T19:08:43.996 INFO:teuthology.orchestra.run.smithi098.stdout: mon election default strategy = 3 2024-06-11T19:08:43.996 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject socket failures = 5000 2024-06-11T19:08:43.996 INFO:teuthology.orchestra.run.smithi098.stdout: ms type = async 2024-06-11T19:08:43.996 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:08:43.996 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.996 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-06-11T19:08:43.996 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-06-11T19:08:43.996 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.996 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-06-11T19:08:43.996 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-06-11T19:08:43.996 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-06-11T19:08:43.996 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug trim objects = true 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.997 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: bluefs allocator = hybrid 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore allocator = hybrid 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-06-11T19:08:43.998 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore zero block detection = True 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 20 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 20 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 10 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: osd class default list = * 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: osd class load list = * 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify cached snaps = True 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify missing on start = True 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-06-11T19:08:43.999 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock override recovery settings = True 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout: osd op queue = debug_random 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout: osd op queue cut off = debug_random 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:44.000 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-06-11T19:08:44.001 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: mon min osdmap epochs = 25 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune interval = 2 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune min = 15 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune txsize = 2 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: mon scrub inject crc mismatch = 0.01 2024-06-11T19:08:44.002 INFO:teuthology.orchestra.run.smithi098.stdout: mon scrub inject missing keys = 0.05 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout: mon scrub interval = 300 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on pool no app = False 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout: paxos service trim min = 5 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout: debug monc = 20 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-06-11T19:08:44.003 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2024-06-11T19:08:44.004 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2024-06-11T19:08:44.004 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.d] 2024-06-11T19:08:44.004 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.e] 2024-06-11T19:08:44.004 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.f] 2024-06-11T19:08:44.004 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.g] 2024-06-11T19:08:44.004 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.h] 2024-06-11T19:08:44.004 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.i] 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = cephx 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2024-06-11T19:08:44.005 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2024-06-11T19:08:44.006 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: mon_allow_pool_size_one = true 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2024-06-11T19:08:44.007 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = "172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790],[v2:172.21.15.98:3302,v1:172.21.15.98:6791],[v2:172.21.15.98:3303,v1:172.21.15.98:6792],[v2:172.21.15.98:3304,v1:172.21.15.98:6793],172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790],[v2:172.21.15.117:3302,v1:172.21.15.117:6791],[v2:172.21.15.117:3303,v1:172.21.15.117:6792]" 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: debug objecter = 20 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: debug rados = 20 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: mon client directed command retry = 5 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: mon election default strategy = 3 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: ms inject socket failures = 5000 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: ms type = async 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2024-06-11T19:08:44.008 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 600 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock profile = high_recovery_ops 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug trim objects = true 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.009 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: bluefs allocator = hybrid 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore allocator = hybrid 2024-06-11T19:08:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore block size = 96636764160 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore fsck on mount = True 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore zero block detection = True 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluefs = 20 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluestore = 20 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 20 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: debug rocksdb = 10 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd backfillfull_ratio = 0.85 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd full ratio = 0.9 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd nearfull ratio = 0.8 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: osd class default list = * 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: osd class load list = * 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify cached snaps = True 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify missing on start = True 2024-06-11T19:08:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: osd failsafe full ratio = 0.95 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock override recovery settings = True 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = bluestore 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: osd op queue = debug_random 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: osd op queue cut off = debug_random 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.012 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 20 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2024-06-11T19:08:44.013 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: # 1m isn't quite enough 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: mon_down_mkfs_grace = 2m 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_filestore_osds = false 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: mon min osdmap epochs = 25 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune interval = 2 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune min = 15 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune txsize = 2 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: mon scrub inject crc mismatch = 0.01 2024-06-11T19:08:44.014 INFO:teuthology.orchestra.run.smithi117.stdout: mon scrub inject missing keys = 0.05 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout: mon scrub interval = 300 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on pool no app = False 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout: paxos service trim min = 5 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout: debug monc = 20 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2024-06-11T19:08:44.015 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.b] 2024-06-11T19:08:44.016 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.c] 2024-06-11T19:08:44.016 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.d] 2024-06-11T19:08:44.016 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.e] 2024-06-11T19:08:44.016 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.f] 2024-06-11T19:08:44.016 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.g] 2024-06-11T19:08:44.016 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.h] 2024-06-11T19:08:44.016 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.i] 2024-06-11T19:08:44.017 INFO:tasks.ceph:Creating admin key on mon.a... 2024-06-11T19:08:44.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-06-11T19:08:44.110 INFO:tasks.ceph:Copying monmap to all nodes... 2024-06-11T19:08:44.111 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:08:44.111 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-06-11T19:08:44.126 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:08:44.126 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-06-11T19:08:44.181 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-06-11T19:08:44.181 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:08:44.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2024-06-11T19:08:44.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-11T19:08:44.255 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:08:44.255 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-06-11T19:08:44.310 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi117.front.sepia.ceph.com 2024-06-11T19:08:44.310 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:08:44.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.keyring 2024-06-11T19:08:44.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-11T19:08:44.345 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:08:44.345 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-06-11T19:08:44.400 INFO:tasks.ceph:Setting up mon nodes... 2024-06-11T19:08:44.400 INFO:tasks.ceph:Setting up mgr nodes... 2024-06-11T19:08:44.400 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-06-11T19:08:44.495 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-06-11T19:08:44.498 INFO:tasks.ceph:Setting up mds nodes... 2024-06-11T19:08:44.498 INFO:tasks.ceph_client:Setting up client nodes... 2024-06-11T19:08:44.498 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-11T19:08:44.538 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-06-11T19:08:44.550 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-06-11T19:08:44.551 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-06-11T19:08:44.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-06-11T19:08:44.575 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-06-11T19:08:44.575 INFO:tasks.ceph:role: osd.0 2024-06-11T19:08:44.575 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2024-06-11T19:08:44.575 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-11T19:08:44.648 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-11T19:08:44.648 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-11T19:08:44.648 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-11T19:08:44.648 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-11T19:08:44.648 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-11T19:08:44.648 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-06-11T19:08:44.648 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-11T19:08:44.648 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-11T19:08:44.648 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-11T19:08:44.648 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-11T19:08:45.153 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-06-11T19:08:45.155 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-06-11T19:08:45.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-06-11T19:08:45.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-06-11T19:08:45.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-06-11T19:08:45.357 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-06-11T19:08:45.357 INFO:tasks.ceph:role: osd.1 2024-06-11T19:08:45.358 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2024-06-11T19:08:45.358 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-11T19:08:45.430 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-11T19:08:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-11T19:08:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-11T19:08:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-11T19:08:45.430 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-11T19:08:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-06-11T19:08:45.430 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-11T19:08:45.430 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-11T19:08:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-11T19:08:45.430 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-11T19:08:45.883 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-06-11T19:08:45.885 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-06-11T19:08:45.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-06-11T19:08:45.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-06-11T19:08:46.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-06-11T19:08:46.094 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-06-11T19:08:46.094 INFO:tasks.ceph:role: osd.2 2024-06-11T19:08:46.094 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2024-06-11T19:08:46.094 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-11T19:08:46.165 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-11T19:08:46.165 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-11T19:08:46.165 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-11T19:08:46.165 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-11T19:08:46.165 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-11T19:08:46.165 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-06-11T19:08:46.165 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-11T19:08:46.166 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-11T19:08:46.166 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-11T19:08:46.166 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-11T19:08:46.648 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-06-11T19:08:46.650 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-06-11T19:08:46.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-06-11T19:08:46.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-06-11T19:08:46.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-11T19:08:46.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:46.874+0000 7f1e2d7fc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-06-11T19:08:46.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:46.880+0000 7f1e2d7fc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-06-11T19:08:46.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:46.880+0000 7f1e2d7fc640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-11T19:08:46.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:46.880+0000 7f1e2d7fc640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-11T19:08:46.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:46.880+0000 7f1e2d7fc640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-11T19:08:46.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:46.882+0000 7f1e2d7fc640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-06-11T19:08:49.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-11T19:08:49.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-11T19:08:49.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:49.847+0000 7f1c3c024640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-06-11T19:08:49.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:49.847+0000 7f1c3c024640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-06-11T19:08:49.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:49.847+0000 7f1c3c024640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-11T19:08:49.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:49.847+0000 7f1c3c024640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-11T19:08:49.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:49.847+0000 7f1c3c024640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-11T19:08:49.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:49.848+0000 7f1c3c024640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-06-11T19:08:52.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-11T19:08:52.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-11T19:08:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:52.809+0000 7fdfc7a1f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-06-11T19:08:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:52.809+0000 7fdfc7a1f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-06-11T19:08:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:52.809+0000 7fdfc7a1f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-11T19:08:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:52.809+0000 7fdfc7a1f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-11T19:08:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:52.809+0000 7fdfc7a1f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-11T19:08:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:08:52.810+0000 7fdfc7a1f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-06-11T19:08:55.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-11T19:08:55.684 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-06-11T19:08:55.714 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-06-11T19:08:55.714 INFO:tasks.ceph:role: osd.3 2024-06-11T19:08:55.714 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com 2024-06-11T19:08:55.714 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-11T19:08:55.793 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-11T19:08:55.793 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-11T19:08:55.794 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-11T19:08:55.794 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-11T19:08:55.794 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-11T19:08:55.794 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-06-11T19:08:55.794 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-11T19:08:55.794 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-11T19:08:55.794 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-11T19:08:55.794 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-11T19:08:56.320 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-06-11T19:08:56.323 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-06-11T19:08:56.323 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-06-11T19:08:56.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-06-11T19:08:56.513 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-06-11T19:08:56.581 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-06-11T19:08:56.581 INFO:tasks.ceph:role: osd.4 2024-06-11T19:08:56.581 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com 2024-06-11T19:08:56.582 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-11T19:08:56.657 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-11T19:08:56.657 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-11T19:08:56.658 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-11T19:08:56.658 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-11T19:08:56.658 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-11T19:08:56.658 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-06-11T19:08:56.658 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-11T19:08:56.658 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-11T19:08:56.658 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-11T19:08:56.658 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-11T19:08:57.108 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-06-11T19:08:57.110 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-06-11T19:08:57.110 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-06-11T19:08:57.207 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-06-11T19:08:57.235 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-06-11T19:08:57.301 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-06-11T19:08:57.301 INFO:tasks.ceph:role: osd.5 2024-06-11T19:08:57.301 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com 2024-06-11T19:08:57.301 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-11T19:08:57.373 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-11T19:08:57.374 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-11T19:08:57.374 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-11T19:08:57.374 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-11T19:08:57.374 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-11T19:08:57.374 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-06-11T19:08:57.374 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-11T19:08:57.374 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-11T19:08:57.374 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-11T19:08:57.374 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-11T19:08:57.859 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-06-11T19:08:57.861 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-06-11T19:08:57.862 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-06-11T19:08:58.013 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-06-11T19:08:58.045 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-11T19:08:58.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:08:58.158+0000 7f3664f8e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-06-11T19:08:58.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:08:58.160+0000 7f3664f8e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-06-11T19:08:58.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:08:58.160+0000 7f3664f8e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-11T19:08:58.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:08:58.160+0000 7f3664f8e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-11T19:08:58.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:08:58.160+0000 7f3664f8e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-11T19:08:58.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:08:58.168+0000 7f3664f8e640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-06-11T19:09:01.021 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-11T19:09:01.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-11T19:09:01.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:01.141+0000 7fc7cd5f3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-06-11T19:09:01.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:01.141+0000 7fc7cd5f3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-06-11T19:09:01.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:01.141+0000 7fc7cd5f3640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-11T19:09:01.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:01.141+0000 7fc7cd5f3640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-11T19:09:01.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:01.141+0000 7fc7cd5f3640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-11T19:09:01.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:01.141+0000 7fc7cd5f3640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-06-11T19:09:03.993 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-11T19:09:04.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-11T19:09:04.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:04.113+0000 7f9e15e21640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-06-11T19:09:04.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:04.113+0000 7f9e15e21640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-06-11T19:09:04.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:04.113+0000 7f9e15e21640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-11T19:09:04.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:04.113+0000 7f9e15e21640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-11T19:09:04.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:04.113+0000 7f9e15e21640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-11T19:09:04.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:04.114+0000 7f9e15e21640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-06-11T19:09:06.962 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-11T19:09:06.991 INFO:tasks.ceph:Reading keys from all nodes... 2024-06-11T19:09:06.991 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:09:06.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-06-11T19:09:07.022 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:09:07.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-06-11T19:09:07.093 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:09:07.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-06-11T19:09:07.162 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:09:07.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-06-11T19:09:07.192 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:09:07.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-06-11T19:09:07.262 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:09:07.262 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-06-11T19:09:07.330 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:09:07.330 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-06-11T19:09:07.397 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:09:07.397 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-06-11T19:09:07.454 INFO:tasks.ceph:Adding keys to all mons... 2024-06-11T19:09:07.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2024-06-11T19:09:07.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo tee -a /etc/ceph/ceph.keyring 2024-06-11T19:09:07.497 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2024-06-11T19:09:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC+oGhmypoyNBAAaUmrDhkjdyquvnaq7C9vaw== 2024-06-11T19:09:07.498 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2024-06-11T19:09:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDBoGhmGRyPMhAASOoZxO6PzbOALDLbc6ZzGQ== 2024-06-11T19:09:07.498 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2024-06-11T19:09:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDEoGhmD8pJMBAAgfnMf3iE5p30g0MW1vdN4g== 2024-06-11T19:09:07.498 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2024-06-11T19:09:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC8oGhmaG2hHRAAk94FJvBPqL9/1iAnpS3ZpQ== 2024-06-11T19:09:07.498 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2024-06-11T19:09:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDKoGhmgOmCCRAAWeWAWjHZUoEHiBc9bR+OXg== 2024-06-11T19:09:07.498 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2024-06-11T19:09:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDNoGhmMkSGCBAAT1l5C7au2A3x/FCKUq/YqA== 2024-06-11T19:09:07.498 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2024-06-11T19:09:07.499 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDQoGhmb6HZBhAAemeS0pIkTVmf/jnUqaf5Xw== 2024-06-11T19:09:07.499 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-06-11T19:09:07.499 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC8oGhm4ZMqIBAA+Q/XSHPWL8PUlr6E2xouqA== 2024-06-11T19:09:07.519 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.0] 2024-06-11T19:09:07.520 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQC+oGhmypoyNBAAaUmrDhkjdyquvnaq7C9vaw== 2024-06-11T19:09:07.520 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.1] 2024-06-11T19:09:07.520 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQDBoGhmGRyPMhAASOoZxO6PzbOALDLbc6ZzGQ== 2024-06-11T19:09:07.520 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.2] 2024-06-11T19:09:07.520 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQDEoGhmD8pJMBAAgfnMf3iE5p30g0MW1vdN4g== 2024-06-11T19:09:07.520 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.x] 2024-06-11T19:09:07.520 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQC8oGhmaG2hHRAAk94FJvBPqL9/1iAnpS3ZpQ== 2024-06-11T19:09:07.520 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.3] 2024-06-11T19:09:07.520 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQDKoGhmgOmCCRAAWeWAWjHZUoEHiBc9bR+OXg== 2024-06-11T19:09:07.520 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.4] 2024-06-11T19:09:07.520 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQDNoGhmMkSGCBAAT1l5C7au2A3x/FCKUq/YqA== 2024-06-11T19:09:07.520 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.5] 2024-06-11T19:09:07.521 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQDQoGhmb6HZBhAAemeS0pIkTVmf/jnUqaf5Xw== 2024-06-11T19:09:07.521 INFO:teuthology.orchestra.run.smithi117.stdout:[client.0] 2024-06-11T19:09:07.521 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQC8oGhm4ZMqIBAA+Q/XSHPWL8PUlr6E2xouqA== 2024-06-11T19:09:07.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-11T19:09:07.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-11T19:09:07.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-11T19:09:07.617 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-11T19:09:07.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-11T19:09:07.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-11T19:09:07.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-11T19:09:07.724 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-11T19:09:07.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-11T19:09:07.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-11T19:09:07.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-11T19:09:07.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-11T19:09:07.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-11T19:09:07.884 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-11T19:09:07.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-11T19:09:07.937 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-11T19:09:07.991 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-06-11T19:09:07.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-06-11T19:09:08.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-11T19:09:08.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-06-11T19:09:08.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-06-11T19:09:08.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-11T19:09:08.300 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-06-11T19:09:08.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-06-11T19:09:08.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-11T19:09:08.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-06-11T19:09:08.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-d 2024-06-11T19:09:08.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i d --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-11T19:09:08.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-d 2024-06-11T19:09:08.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-e 2024-06-11T19:09:08.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i e --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-11T19:09:08.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-e 2024-06-11T19:09:08.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-f 2024-06-11T19:09:08.911 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i f --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-11T19:09:09.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-f 2024-06-11T19:09:09.041 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-g 2024-06-11T19:09:09.106 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i g --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-11T19:09:09.203 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-g 2024-06-11T19:09:09.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-h 2024-06-11T19:09:09.293 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i h --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-11T19:09:09.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-h 2024-06-11T19:09:09.413 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-i 2024-06-11T19:09:09.478 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i i --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-11T19:09:09.574 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-i 2024-06-11T19:09:09.598 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-06-11T19:09:09.601 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-06-11T19:09:09.653 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-06-11T19:09:09.654 INFO:tasks.ceph.mon.a:Restarting daemon 2024-06-11T19:09:09.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-06-11T19:09:09.658 INFO:tasks.ceph.mon.a:Started 2024-06-11T19:09:09.658 INFO:tasks.ceph.mon.b:Restarting daemon 2024-06-11T19:09:09.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-06-11T19:09:09.663 INFO:tasks.ceph.mon.b:Started 2024-06-11T19:09:09.663 INFO:tasks.ceph.mon.c:Restarting daemon 2024-06-11T19:09:09.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-06-11T19:09:09.665 INFO:tasks.ceph.mon.c:Started 2024-06-11T19:09:09.665 INFO:tasks.ceph.mon.d:Restarting daemon 2024-06-11T19:09:09.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-06-11T19:09:09.667 INFO:tasks.ceph.mon.d:Started 2024-06-11T19:09:09.667 INFO:tasks.ceph.mon.e:Restarting daemon 2024-06-11T19:09:09.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-06-11T19:09:09.670 INFO:tasks.ceph.mon.e:Started 2024-06-11T19:09:09.670 INFO:tasks.ceph.mon.f:Restarting daemon 2024-06-11T19:09:09.670 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-06-11T19:09:09.696 INFO:tasks.ceph.mon.f:Started 2024-06-11T19:09:09.696 INFO:tasks.ceph.mon.g:Restarting daemon 2024-06-11T19:09:09.696 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-06-11T19:09:09.698 INFO:tasks.ceph.mon.g:Started 2024-06-11T19:09:09.699 INFO:tasks.ceph.mon.h:Restarting daemon 2024-06-11T19:09:09.699 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-06-11T19:09:09.701 INFO:tasks.ceph.mon.h:Started 2024-06-11T19:09:09.701 INFO:tasks.ceph.mon.i:Restarting daemon 2024-06-11T19:09:09.701 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-06-11T19:09:09.703 INFO:tasks.ceph.mon.i:Started 2024-06-11T19:09:09.703 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-06-11T19:09:09.703 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-06-11T19:09:09.703 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-06-11T19:09:09.705 INFO:tasks.ceph.mgr.x:Started 2024-06-11T19:09:09.705 DEBUG:tasks.ceph:set 0 configs 2024-06-11T19:09:09.706 INFO:tasks.ceph:Setting crush tunables to default 2024-06-11T19:09:09.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd crush tunables default 2024-06-11T19:09:09.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.833+0000 7fbc5d41e640 10 monclient: get_monmap_and_config 2024-06-11T19:09:09.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.833+0000 7fbc5d41e640 10 monclient: build_initial_monmap 2024-06-11T19:09:09.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.833+0000 7fbc5d41e640 10 monclient: monmap: 2024-06-11T19:09:09.833 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:09.833 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:09.833 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:09.834154+0000 2024-06-11T19:09:09.833 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:09.834154+0000 2024-06-11T19:09:09.833 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:09.834 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.833+0000 7fbc5d41e640 1 Processor -- start 2024-06-11T19:09:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.833+0000 7fbc5d41e640 1 -- start start 2024-06-11T19:09:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.833+0000 7fbc5d41e640 10 monclient: init 2024-06-11T19:09:09.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 10 monclient: _add_conns ranks=[1,4,6,10,2,3,8,0,9,5,7] 2024-06-11T19:09:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fbc580a6260 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 10 monclient(hunting): picked mon.noname-c con 0x7fbc5806c890 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 10 monclient(hunting): picked mon.noname-d con 0x7fbc580a2d00 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc577fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc580a2460 con 0x7fbc580a6260 2024-06-11T19:09:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fbc580a05b0 con 0x7fbc5806c890 2024-06-11T19:09:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fbc580a6640 con 0x7fbc580a2d00 2024-06-11T19:09:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc567fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc5d41e640 10 monclient(hunting): authenticate will time out at 759.641418s 2024-06-11T19:09:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc577fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:44344/0 (socket says 172.21.15.98:44344) 2024-06-11T19:09:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 learned_addr learned my addr 172.21.15.98:0/1580816983 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc577fe640 10 monclient(hunting): get_auth_request con 0x7fbc580a2d00 auth_method 0 2024-06-11T19:09:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc577fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc567fc640 10 monclient(hunting): get_auth_request con 0x7fbc5806c890 auth_method 0 2024-06-11T19:09:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc567fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc567fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 msgr2=0x7fbc5813ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:09:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 msgr2=0x7fbc5813ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 msgr2=0x7fbc5813da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 msgr2=0x7fbc5813da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:09.834+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:10.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:10.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc577fe640 10 monclient(hunting): get_auth_request con 0x7fbc580a2d00 auth_method 0 2024-06-11T19:09:10.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:10.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:10.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:10.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:10.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc567fc640 10 monclient(hunting): get_auth_request con 0x7fbc5806c890 auth_method 0 2024-06-11T19:09:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc567fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc567fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 msgr2=0x7fbc5813ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 msgr2=0x7fbc5813ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 msgr2=0x7fbc5813da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:09:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 msgr2=0x7fbc5813da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:09:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:10.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.035+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:09:10.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:10.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc577fe640 10 monclient(hunting): get_auth_request con 0x7fbc580a2d00 auth_method 0 2024-06-11T19:09:10.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:10.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:10.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:10.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:10.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc567fc640 10 monclient(hunting): get_auth_request con 0x7fbc5806c890 auth_method 0 2024-06-11T19:09:10.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:10.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc567fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:10.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc567fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:10.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 msgr2=0x7fbc5813ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:10.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 msgr2=0x7fbc5813ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:10.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:10.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:09:10.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 msgr2=0x7fbc5813da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:09:10.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 msgr2=0x7fbc5813da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:10.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:10.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:10.436+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:09:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc567fc640 10 monclient(hunting): get_auth_request con 0x7fbc5806c890 auth_method 0 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc567fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc567fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc577fe640 10 monclient(hunting): get_auth_request con 0x7fbc580a2d00 auth_method 0 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 msgr2=0x7fbc5813da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 msgr2=0x7fbc5813da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 msgr2=0x7fbc5813ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 msgr2=0x7fbc5813ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:09:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:11.237+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:09:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 10 monclient(hunting): tick 2024-06-11T19:09:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:09:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 1 monclient(hunting): continuing hunt 2024-06-11T19:09:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:09:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 msgr2=0x7fbc5813ffd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:09:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbc580a2d00 0x7fbc5813ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 msgr2=0x7fbc5813da80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:09:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc5806c890 0x7fbc5813da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 1 -- 172.21.15.98:0/1580816983 >> v1:172.21.15.117:6789/0 conn(0x7fbc580a6260 legacy=0x7fbc580a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 10 monclient: _add_conns ranks=[0,7,8,9,2,3,5,10] 2024-06-11T19:09:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc44003850 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 10 monclient(hunting): picked mon.noname-i con 0x7fbc44007440 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 10 monclient(hunting): picked mon.noname-e con 0x7fbc44009f80 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 1 -- 172.21.15.98:0/1580816983 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc5806c650 con 0x7fbc44003850 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 1 -- 172.21.15.98:0/1580816983 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fbc580a05b0 con 0x7fbc44007440 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 1 -- 172.21.15.98:0/1580816983 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbc580a6640 con 0x7fbc44009f80 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc54ff9640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc577fe640 10 monclient(hunting): get_auth_request con 0x7fbc44009f80 auth_method 0 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc577fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 msgr2=0x7fbc4400c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:09:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 msgr2=0x7fbc4400c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc567fc640 10 monclient(hunting): get_auth_request con 0x7fbc44007440 auth_method 0 2024-06-11T19:09:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc567fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc567fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 msgr2=0x7fbc440098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 msgr2=0x7fbc440098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:12.835+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:13.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:13.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc577fe640 10 monclient(hunting): get_auth_request con 0x7fbc44009f80 auth_method 0 2024-06-11T19:09:13.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:13.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:13.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:13.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 msgr2=0x7fbc4400c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:13.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 msgr2=0x7fbc4400c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:13.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:13.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:09:13.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:13.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc567fc640 10 monclient(hunting): get_auth_request con 0x7fbc44007440 auth_method 0 2024-06-11T19:09:13.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:13.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc567fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:13.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc567fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:13.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 msgr2=0x7fbc440098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:09:13.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 msgr2=0x7fbc440098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:13.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:13.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.036+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:09:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc577fe640 10 monclient(hunting): get_auth_request con 0x7fbc44009f80 auth_method 0 2024-06-11T19:09:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 msgr2=0x7fbc4400c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 msgr2=0x7fbc4400c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:09:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc567fc640 10 monclient(hunting): get_auth_request con 0x7fbc44007440 auth_method 0 2024-06-11T19:09:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc567fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc567fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 msgr2=0x7fbc440098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:09:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 msgr2=0x7fbc440098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:13.437+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:09:14.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:14.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc577fe640 10 monclient(hunting): get_auth_request con 0x7fbc44009f80 auth_method 0 2024-06-11T19:09:14.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:14.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:14.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:14.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 msgr2=0x7fbc4400c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:14.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 msgr2=0x7fbc4400c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:14.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:14.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:09:14.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:14.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc567fc640 10 monclient(hunting): get_auth_request con 0x7fbc44007440 auth_method 0 2024-06-11T19:09:14.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:14.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc567fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:14.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc567fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:14.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 msgr2=0x7fbc440098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:14.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc567fc640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 msgr2=0x7fbc440098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:14.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:14.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:14.238+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:09:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.050+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1638301700 0 0) 0x7fbc5806c650 con 0x7fbc44003850 2024-06-11T19:09:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.050+0000 7fbc55ffb640 10 monclient(hunting): _negotiate 2024-06-11T19:09:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.050+0000 7fbc55ffb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.050+0000 7fbc55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.050+0000 7fbc55ffb640 10 monclient(hunting): my global_id is 4113 2024-06-11T19:09:15.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.050+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc40003520 con 0x7fbc44003850 2024-06-11T19:09:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2690168558 0 0) 0x7fbc40003520 con 0x7fbc44003850 2024-06-11T19:09:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 msgr2=0x7fbc4400c440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:09:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc44009f80 0x7fbc4400c440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 msgr2=0x7fbc440098d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc44007440 0x7fbc440098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient: _finish_hunting 0 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 reap_dead start 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4400dd50 con 0x7fbc44003850 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7fbc48001a40 con 0x7fbc44003850 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc5d41e640 5 monclient: authenticate success, global_id 4113 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc5d41e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient: dump: 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.065 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 >> v1:172.21.15.98:6789/0 conn(0x7fbc44003850 legacy=0x7fbc44005c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient: _add_conns ranks=[7,5,4,8,1,2,6,3,0] 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc40004200 0x7fbc40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient(hunting): picked mon.i con 0x7fbc40004200 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbc40004e10 0x7fbc40005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient(hunting): picked mon.h con 0x7fbc40004e10 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc40009bf0 0x7fbc4000c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient(hunting): picked mon.c con 0x7fbc40009bf0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fbc580a05b0 con 0x7fbc40009bf0 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fbc580a6640 con 0x7fbc40004e10 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fbc4000c990 con 0x7fbc40004200 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.055+0000 7fbc55ffb640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc5d41e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc40009bf0 0x7fbc4000c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 10 monclient(hunting): get_auth_request con 0x7fbc40009bf0 auth_method 0 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc56ffd640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc40004200 0x7fbc40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbc40004e10 0x7fbc40005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc56ffd640 10 monclient(hunting): get_auth_request con 0x7fbc40004200 auth_method 0 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc56ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc56ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc567fc640 10 monclient(hunting): get_auth_request con 0x7fbc40004e10 auth_method 0 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc567fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc567fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 10 monclient(hunting): handle_auth_done global_id 4113 payload 293 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc40004200 msgr2=0x7fbc40004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc40004200 0x7fbc40004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbc40004e10 msgr2=0x7fbc40005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbc40004e10 0x7fbc40005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 10 monclient: _finish_hunting 0 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 1 monclient: found mon.c 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 1 -- 172.21.15.98:0/1580816983 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc40003520 con 0x7fbc40009bf0 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:15.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.056+0000 7fbc567fc640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbc40004e10 0x7fbc40005290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc577fe640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc40009bf0 0x7fbc4000c080 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbc580a0960 tx=0x7fbc40005830 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ea4edb421501e98e server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc50009700 con 0x7fbc40009bf0 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc55ffb640 10 monclient: dump: 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.070 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc5d41e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc50030070 con 0x7fbc40009bf0 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc55ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:15.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc5d41e640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc55ffb640 1 -- 172.21.15.98:0/1580816983 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc50035860 con 0x7fbc40009bf0 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc5d41e640 4 set_mon_vals no callback set 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.057+0000 7fbc55ffb640 10 monclient: dump: 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.072 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.058+0000 7fbc5d41e640 10 monclient: shutdown 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.058+0000 7fbc5d41e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.058+0000 7fbc5d41e640 1 -- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc40009bf0 msgr2=0x7fbc4000c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.058+0000 7fbc5d41e640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc40009bf0 0x7fbc4000c080 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbc580a0960 tx=0x7fbc40005830 comp rx=0 tx=0).stop 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.058+0000 7fbc5d41e640 1 -- 172.21.15.98:0/1580816983 shutdown_connections 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.058+0000 7fbc5d41e640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbc40009bf0 0x7fbc4000c080 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.058+0000 7fbc5d41e640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbc40004200 0x7fbc40004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.058+0000 7fbc5d41e640 1 --2- 172.21.15.98:0/1580816983 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbc40004e10 0x7fbc40005290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.058+0000 7fbc5d41e640 1 -- 172.21.15.98:0/1580816983 >> 172.21.15.98:0/1580816983 conn(0x7fbc580a95b0 msgr2=0x7fbc580a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:15.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.058+0000 7fbc5d41e640 1 -- 172.21.15.98:0/1580816983 shutdown_connections 2024-06-11T19:09:15.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.058+0000 7fbc5d41e640 1 -- 172.21.15.98:0/1580816983 wait complete. 2024-06-11T19:09:15.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.059+0000 7fbc5d41e640 10 monclient: build_initial_monmap 2024-06-11T19:09:15.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.059+0000 7fbc5d41e640 10 monclient: monmap: 2024-06-11T19:09:15.074 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:15.074 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:15.074 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:15.074 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:15.074 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:15.074 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.059+0000 7fbc5d41e640 1 librados: starting msgr at 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.059+0000 7fbc5d41e640 1 librados: starting objecter 2024-06-11T19:09:15.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 1 Processor -- start 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 1 -- start start 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 1 librados: setting wanted keys 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 1 librados: calling monclient init 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 10 monclient: init 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 10 monclient: _add_conns ranks=[2,5,8,0,7,3,1,4,6] 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbc40004200 0x7fbc5809e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 10 monclient(hunting): picked mon.noname-b con 0x7fbc40004200 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbc40004e10 0x7fbc581b9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 10 monclient(hunting): picked mon.noname-h con 0x7fbc40004e10 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc40009bf0 0x7fbc581b9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 10 monclient(hunting): picked mon.noname-e con 0x7fbc40009bf0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbc580a1d50 con 0x7fbc40004200 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fbc5806cd10 con 0x7fbc40004e10 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbc580a1ed0 con 0x7fbc40009bf0 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc5d41e640 10 monclient(hunting): authenticate will time out at 764.868695s 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc56ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbc40004200 0x7fbc5809e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc56ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbc40004200 0x7fbc5809e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:51170/0 (socket says 172.21.15.98:51170) 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc56ffd640 1 -- 172.21.15.98:0/938483716 learned_addr learned my addr 172.21.15.98:0/938483716 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc56ffd640 10 monclient(hunting): get_auth_request con 0x7fbc40004200 auth_method 0 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc56ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc577fe640 1 --2- 172.21.15.98:0/938483716 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc40009bf0 0x7fbc581b9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc567fc640 1 --2- 172.21.15.98:0/938483716 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbc40004e10 0x7fbc581b9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc567fc640 10 monclient(hunting): get_auth_request con 0x7fbc40004e10 auth_method 0 2024-06-11T19:09:15.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc567fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc567fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc577fe640 10 monclient(hunting): get_auth_request con 0x7fbc40009bf0 auth_method 0 2024-06-11T19:09:15.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc577fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:15.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.060+0000 7fbc577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc56ffd640 10 monclient(hunting): handle_auth_done global_id 4106 payload 824 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc56ffd640 1 -- 172.21.15.98:0/938483716 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc40009bf0 msgr2=0x7fbc581b9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc56ffd640 1 --2- 172.21.15.98:0/938483716 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc40009bf0 0x7fbc581b9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc56ffd640 1 -- 172.21.15.98:0/938483716 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbc40004e10 msgr2=0x7fbc581b9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc56ffd640 1 --2- 172.21.15.98:0/938483716 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbc40004e10 0x7fbc581b9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc56ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc56ffd640 1 monclient: found mon.noname-b 2024-06-11T19:09:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:15.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc56ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:15.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc56ffd640 1 -- 172.21.15.98:0/938483716 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc580a5b40 con 0x7fbc40004200 2024-06-11T19:09:15.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc56ffd640 1 --2- 172.21.15.98:0/938483716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbc40004200 0x7fbc5809e9a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbc4800f770 tx=0x7fbc4800fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:15.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc3f7fe640 10 client.?.objecter ms_handle_connect 0x7fbc40004200 2024-06-11T19:09:15.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.061+0000 7fbc3f7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:15.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 1 -- 172.21.15.98:0/938483716 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc4800fe80 con 0x7fbc40004200 2024-06-11T19:09:15.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:15.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 10 monclient: dump: 2024-06-11T19:09:15.082 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:15.082 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.082 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.082 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.082 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:15.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc5d41e640 5 monclient: authenticate success, global_id 4106 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 1 -- 172.21.15.98:0/938483716 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc48018070 con 0x7fbc40004200 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 1 -- 172.21.15.98:0/938483716 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc480052e0 con 0x7fbc40004200 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc5d41e640 10 monclient: _renew_subs 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc5d41e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc5d41e640 1 -- 172.21.15.98:0/938483716 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc580a8a30 con 0x7fbc40004200 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc557fa640 4 set_mon_vals no callback set 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 10 monclient: dump: 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.084 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:15.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc5d41e640 10 client.4106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc5d41e640 10 monclient: _renew_subs 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc5d41e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc5d41e640 1 -- 172.21.15.98:0/938483716 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc5809f5c0 con 0x7fbc40004200 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc5d41e640 1 librados: init done 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 1 -- 172.21.15.98:0/938483716 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbc4800d6d0 con 0x7fbc40004200 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 1 -- 172.21.15.98:0/938483716 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbc48010040 con 0x7fbc40004200 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 10 client.4106.objecter ms_dispatch 0x7fbc58000b90 osd_map(1..1 src has 1..1) 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 3 client.4106.objecter handle_osd_map got epochs [1,1] > 0 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 3 client.4106.objecter handle_osd_map decoding full epoch 1 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.064+0000 7fbc3f7fe640 20 client.4106.objecter dump_active .. 0 homeless 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.065+0000 7fbc5d41e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.065+0000 7fbc5d41e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:15.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.065+0000 7fbc5d41e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:15.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.065+0000 7fbc5d41e640 1 -- 172.21.15.98:0/938483716 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc580a48b0 con 0x7fbc40004200 2024-06-11T19:09:15.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.067+0000 7fbc3f7fe640 1 -- 172.21.15.98:0/938483716 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fbc480035f0 con 0x7fbc40004200 2024-06-11T19:09:15.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.067+0000 7fbc3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:15.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.067+0000 7fbc3f7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:15.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.134+0000 7fbc5d41e640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-06-11T19:09:15.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.134+0000 7fbc5d41e640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-06-11T19:09:15.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.134+0000 7fbc5d41e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:15.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.134+0000 7fbc5d41e640 1 -- 172.21.15.98:0/938483716 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fbc581a5890 con 0x7fbc40004200 2024-06-11T19:09:15.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc3f7fe640 1 -- 172.21.15.98:0/938483716 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fbc4800db50 con 0x7fbc40004200 2024-06-11T19:09:15.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-06-11T19:09:15.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc3f7fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-06-11T19:09:15.139 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-06-11T19:09:15.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 10 librados: watch_flush enter 2024-06-11T19:09:15.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 10 librados: watch_flush exit 2024-06-11T19:09:15.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 20 client.4106.objecter shutdown clearing up homeless session... 2024-06-11T19:09:15.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 10 client.4106.objecter successfully canceled tick 2024-06-11T19:09:15.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 10 monclient: shutdown 2024-06-11T19:09:15.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:15.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 1 -- 172.21.15.98:0/938483716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbc40004200 msgr2=0x7fbc5809e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 1 --2- 172.21.15.98:0/938483716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbc40004200 0x7fbc5809e9a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbc4800f770 tx=0x7fbc4800fc30 comp rx=0 tx=0).stop 2024-06-11T19:09:15.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 1 -- 172.21.15.98:0/938483716 shutdown_connections 2024-06-11T19:09:15.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 1 --2- 172.21.15.98:0/938483716 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc40009bf0 0x7fbc581b9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 1 --2- 172.21.15.98:0/938483716 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbc40004e10 0x7fbc581b9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 1 --2- 172.21.15.98:0/938483716 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbc40004200 0x7fbc5809e9a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.138+0000 7fbc5d41e640 1 -- 172.21.15.98:0/938483716 >> 172.21.15.98:0/938483716 conn(0x7fbc580a95b0 msgr2=0x7fbc580a4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:15.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.139+0000 7fbc5d41e640 1 -- 172.21.15.98:0/938483716 shutdown_connections 2024-06-11T19:09:15.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.139+0000 7fbc5d41e640 1 -- 172.21.15.98:0/938483716 wait complete. 2024-06-11T19:09:15.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:15.139+0000 7fbc5d41e640 1 librados: shutdown 2024-06-11T19:09:15.151 INFO:tasks.ceph:check_enable_crimson: False 2024-06-11T19:09:15.151 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-06-11T19:09:15.151 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:09:15.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-06-11T19:09:15.172 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:15.172+0000 7f37f0440040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-11T19:09:15.176 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:09:15.176 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-06-11T19:09:15.241 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-06-11T19:09:15.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-06-11T19:09:15.306 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:09:15.306 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-06-11T19:09:15.331 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:09:15.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-06-11T19:09:15.333 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:15.333+0000 7f37f0440040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-11T19:09:15.357 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:09:15.358 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-06-11T19:09:15.390 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:15.390+0000 7f37f0440040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-11T19:09:15.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new ef1075e3-3e1b-4904-b14b-90751df6c38c 0 2024-06-11T19:09:15.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.553+0000 7fa71c099640 10 monclient: get_monmap_and_config 2024-06-11T19:09:15.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.553+0000 7fa71c099640 10 monclient: build_initial_monmap 2024-06-11T19:09:15.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.553+0000 7fa71c099640 10 monclient: monmap: 2024-06-11T19:09:15.554 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-06-11T19:09:15.554 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.554 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:09:15.554499+0000 2024-06-11T19:09:15.554 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:09:15.554499+0000 2024-06-11T19:09:15.554 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:15.554 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:15.554 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:15.554 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:15.554 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:15.554 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:15.554 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:15.555 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:15.555 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:15.555 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:15.555 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:15.555 INFO:teuthology.orchestra.run.smithi117.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:15.555 INFO:teuthology.orchestra.run.smithi117.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:15.555 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 1 Processor -- start 2024-06-11T19:09:15.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 1 -- start start 2024-06-11T19:09:15.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 10 monclient: init 2024-06-11T19:09:15.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:15.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 10 monclient: _add_conns ranks=[0,5,7,8,4,6,1,2,9,10,3] 2024-06-11T19:09:15.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7140a52a0 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:15.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa71409d520 0x7fa71413a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 10 monclient(hunting): picked mon.noname-h con 0x7fa71409d520 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:15.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 0x7fa71413cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 10 monclient(hunting): picked mon.noname-i con 0x7fa714073830 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:15.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa71409d2e0 con 0x7fa7140a52a0 2024-06-11T19:09:15.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fa71409daa0 con 0x7fa71409d520 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa7140a5680 con 0x7fa714073830 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.554+0000 7fa71c099640 10 monclient(hunting): authenticate will time out at 778.366887s 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa719634640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa71409d520 0x7fa71413a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 0x7fa71413cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa719634640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa71409d520 0x7fa71413a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.117:50672/0 (socket says 172.21.15.117:50672) 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 0x7fa71413cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.117:48372/0 (socket says 172.21.15.117:48372) 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa719634640 1 -- 172.21.15.117:0/1707612420 learned_addr learned my addr 172.21.15.117:0/1707612420 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa719634640 10 monclient(hunting): get_auth_request con 0x7fa71409d520 auth_method 0 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa719634640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.565 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa719634640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa719634640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 10 monclient(hunting): get_auth_request con 0x7fa714073830 auth_method 0 2024-06-11T19:09:15.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa719634640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa719634640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa719634640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:15.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:15.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-06-11T19:09:15.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 1 -- 172.21.15.117:0/1707612420 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa71409d520 msgr2=0x7fa71413a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:15.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 1 --2- 172.21.15.117:0/1707612420 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa71409d520 0x7fa71413a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 1 -- 172.21.15.117:0/1707612420 >> v1:172.21.15.98:6789/0 conn(0x7fa7140a52a0 legacy=0x7fa7140a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 10 monclient: _finish_hunting 0 2024-06-11T19:09:15.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 1 monclient: found mon.noname-i 2024-06-11T19:09:15.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:15.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:09:15.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa71a636640 1 -- 172.21.15.117:0/1707612420 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71409e9d0 con 0x7fa714073830 2024-06-11T19:09:15.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.555+0000 7fa719634640 1 --2- 172.21.15.117:0/1707612420 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa71409d520 0x7fa71413a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:15.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71a636640 1 --2- 172.21.15.117:0/1707612420 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 0x7fa71413cf30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa70c009df0 tx=0x7fa70c02bd50 comp rx=0 tx=0).ready entity=mon.7 client_cookie=a4a9aaf4eb893e44 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:15.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 1 -- 172.21.15.117:0/1707612420 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa70c039070 con 0x7fa714073830 2024-06-11T19:09:15.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 10 monclient: dump: 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.568 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 10 monclient: _finish_auth 0 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 10 monclient: _check_auth_tickets 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 1 -- 172.21.15.117:0/1707612420 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa70c02cd70 con 0x7fa714073830 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71c099640 5 monclient: authenticate success, global_id 4120 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71c099640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:15.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71c099640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 1 -- 172.21.15.117:0/1707612420 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa70c0327b0 con 0x7fa714073830 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71c099640 4 set_mon_vals no callback set 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa718e33640 10 monclient: dump: 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.570 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71c099640 10 monclient: shutdown 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71c099640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71c099640 1 -- 172.21.15.117:0/1707612420 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 msgr2=0x7fa71413cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71c099640 1 --2- 172.21.15.117:0/1707612420 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 0x7fa71413cf30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa70c009df0 tx=0x7fa70c02bd50 comp rx=0 tx=0).stop 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71c099640 1 -- 172.21.15.117:0/1707612420 shutdown_connections 2024-06-11T19:09:15.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71c099640 1 --2- 172.21.15.117:0/1707612420 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 0x7fa71413cf30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71c099640 1 --2- 172.21.15.117:0/1707612420 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa71409d520 0x7fa71413a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.556+0000 7fa71c099640 1 -- 172.21.15.117:0/1707612420 >> 172.21.15.117:0/1707612420 conn(0x7fa7140a4780 msgr2=0x7fa7140a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.557+0000 7fa71c099640 1 -- 172.21.15.117:0/1707612420 shutdown_connections 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.557+0000 7fa71c099640 1 -- 172.21.15.117:0/1707612420 wait complete. 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.557+0000 7fa71c099640 10 monclient: build_initial_monmap 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.557+0000 7fa71c099640 10 monclient: monmap: 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-06-11T19:09:15.572 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.557+0000 7fa71c099640 1 librados: starting msgr at 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.557+0000 7fa71c099640 1 librados: starting objecter 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.557+0000 7fa71c099640 1 Processor -- start 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.557+0000 7fa71c099640 1 -- start start 2024-06-11T19:09:15.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.557+0000 7fa71c099640 1 librados: setting wanted keys 2024-06-11T19:09:15.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.557+0000 7fa71c099640 1 librados: calling monclient init 2024-06-11T19:09:15.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.557+0000 7fa71c099640 10 monclient: init 2024-06-11T19:09:15.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:15.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 10 monclient: _add_conns ranks=[7,3,4,1,5,2,0,6,8] 2024-06-11T19:09:15.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 0x7fa7141cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 10 monclient(hunting): picked mon.noname-i con 0x7fa714073830 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:15.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa71409d520 0x7fa7141cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 10 monclient(hunting): picked mon.noname-g con 0x7fa71409d520 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:15.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7140a52a0 0x7fa714072220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 10 monclient(hunting): picked mon.noname-c con 0x7fa7140a52a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:15.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fa7140a21a0 con 0x7fa71409d520 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa7140a2020 con 0x7fa7140a52a0 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa71413d470 con 0x7fa714073830 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71c099640 10 monclient(hunting): authenticate will time out at 778.370374s 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 0x7fa7141cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 0x7fa7141cbad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.117:48382/0 (socket says 172.21.15.117:48382) 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719634640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa71409d520 0x7fa7141cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 1 -- 172.21.15.117:0/2764090119 learned_addr learned my addr 172.21.15.117:0/2764090119 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 10 monclient(hunting): get_auth_request con 0x7fa714073830 auth_method 0 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719634640 10 monclient(hunting): get_auth_request con 0x7fa71409d520 auth_method 0 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719634640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719634640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719634640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719634640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719634640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719634640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71a636640 1 --2- 172.21.15.117:0/2764090119 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7140a52a0 0x7fa714072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71a636640 10 monclient(hunting): get_auth_request con 0x7fa7140a52a0 auth_method 0 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71a636640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71a636640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71a636640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71a636640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71a636640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa71a636640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 1 -- 172.21.15.117:0/2764090119 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7140a52a0 msgr2=0x7fa714072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 1 --2- 172.21.15.117:0/2764090119 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7140a52a0 0x7fa714072220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 1 -- 172.21.15.117:0/2764090119 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa71409d520 msgr2=0x7fa7141cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 1 --2- 172.21.15.117:0/2764090119 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa71409d520 0x7fa7141cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 10 monclient: _finish_hunting 0 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 1 monclient: found mon.noname-i 2024-06-11T19:09:15.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:15.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:09:15.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 1 -- 172.21.15.117:0/2764090119 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa714134640 con 0x7fa714073830 2024-06-11T19:09:15.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719634640 1 --2- 172.21.15.117:0/2764090119 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa71409d520 0x7fa7141cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:15.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa719e35640 1 --2- 172.21.15.117:0/2764090119 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 0x7fa7141cbad0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa71000c3a0 tx=0x7fa71000c860 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:15.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa70a7fc640 10 client.?.objecter ms_handle_connect 0x7fa714073830 2024-06-11T19:09:15.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.558+0000 7fa70a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:15.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 1 -- 172.21.15.117:0/2764090119 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa710017070 con 0x7fa714073830 2024-06-11T19:09:15.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:09:15.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 10 monclient: dump: 2024-06-11T19:09:15.578 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:15.578 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.579 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa71c099640 5 monclient: authenticate success, global_id 4129 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa71c099640 10 monclient: _renew_subs 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 1 -- 172.21.15.117:0/2764090119 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7100035f0 con 0x7fa714073830 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa71c099640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa71c099640 1 -- 172.21.15.117:0/2764090119 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7140a1360 con 0x7fa714073830 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 1 -- 172.21.15.117:0/2764090119 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7100010e0 con 0x7fa714073830 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa71c099640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70bfff640 4 set_mon_vals no callback set 2024-06-11T19:09:15.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 10 monclient: dump: 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.581 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.582 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.582 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.582 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa71c099640 10 monclient: _renew_subs 2024-06-11T19:09:15.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa71c099640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:09:15.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa71c099640 1 -- 172.21.15.117:0/2764090119 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fa71413a230 con 0x7fa714073830 2024-06-11T19:09:15.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa71c099640 1 librados: init done 2024-06-11T19:09:15.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 1 -- 172.21.15.117:0/2764090119 <== mon.7 v2:172.21.15.117:3303/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa7100050d0 con 0x7fa714073830 2024-06-11T19:09:15.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 1 -- 172.21.15.117:0/2764090119 <== mon.7 v2:172.21.15.117:3303/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fa710011ad0 con 0x7fa714073830 2024-06-11T19:09:15.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 10 client.4129.objecter ms_dispatch 0x7fa714000b90 osd_map(2..2 src has 1..2) 2024-06-11T19:09:15.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 3 client.4129.objecter handle_osd_map got epochs [2,2] > 0 2024-06-11T19:09:15.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 3 client.4129.objecter handle_osd_map decoding full epoch 2 2024-06-11T19:09:15.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.559+0000 7fa70a7fc640 20 client.4129.objecter dump_active .. 0 homeless 2024-06-11T19:09:15.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.560+0000 7fa71c099640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:15.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.560+0000 7fa71c099640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:15.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.560+0000 7fa71c099640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:09:15.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.560+0000 7fa71c099640 1 -- 172.21.15.117:0/2764090119 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7140a1910 con 0x7fa714073830 2024-06-11T19:09:15.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.562+0000 7fa70a7fc640 1 -- 172.21.15.117:0/2764090119 <== mon.7 v2:172.21.15.117:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fa710020020 con 0x7fa714073830 2024-06-11T19:09:15.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.562+0000 7fa70a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:15.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.562+0000 7fa70a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:15.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.634+0000 7fa71c099640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ef1075e3-3e1b-4904-b14b-90751df6c38c", "id": 0}] 2024-06-11T19:09:15.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.634+0000 7fa71c099640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ef1075e3-3e1b-4904-b14b-90751df6c38c", "id": 0}] 2024-06-11T19:09:15.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.634+0000 7fa71c099640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:09:15.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.634+0000 7fa71c099640 1 -- 172.21.15.117:0/2764090119 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_command({"prefix": "osd new", "uuid": "ef1075e3-3e1b-4904-b14b-90751df6c38c", "id": 0} v 0) -- 0x7fa714139e00 con 0x7fa714073830 2024-06-11T19:09:15.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.640+0000 7fa70a7fc640 1 -- 172.21.15.117:0/2764090119 <== mon.7 v2:172.21.15.117:3303/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ef1075e3-3e1b-4904-b14b-90751df6c38c", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fa710011590 con 0x7fa714073830 2024-06-11T19:09:15.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.640+0000 7fa70a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ef1075e3-3e1b-4904-b14b-90751df6c38c", "id": 0}] 2024-06-11T19:09:15.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.640+0000 7fa70a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:15.641 INFO:teuthology.orchestra.run.smithi117.stdout:0 2024-06-11T19:09:15.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 10 librados: watch_flush enter 2024-06-11T19:09:15.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 10 librados: watch_flush exit 2024-06-11T19:09:15.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 20 client.4129.objecter shutdown clearing up homeless session... 2024-06-11T19:09:15.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 10 client.4129.objecter successfully canceled tick 2024-06-11T19:09:15.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 10 monclient: shutdown 2024-06-11T19:09:15.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:15.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 1 -- 172.21.15.117:0/2764090119 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 msgr2=0x7fa7141cbad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 1 --2- 172.21.15.117:0/2764090119 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 0x7fa7141cbad0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa71000c3a0 tx=0x7fa71000c860 comp rx=0 tx=0).stop 2024-06-11T19:09:15.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 1 -- 172.21.15.117:0/2764090119 shutdown_connections 2024-06-11T19:09:15.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 1 --2- 172.21.15.117:0/2764090119 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7140a52a0 0x7fa714072220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 1 --2- 172.21.15.117:0/2764090119 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa71409d520 0x7fa7141cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 1 --2- 172.21.15.117:0/2764090119 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa714073830 0x7fa7141cbad0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 1 -- 172.21.15.117:0/2764090119 >> 172.21.15.117:0/2764090119 conn(0x7fa7140a4780 msgr2=0x7fa71413c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:15.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 1 -- 172.21.15.117:0/2764090119 shutdown_connections 2024-06-11T19:09:15.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 1 -- 172.21.15.117:0/2764090119 wait complete. 2024-06-11T19:09:15.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.641+0000 7fa71c099640 1 librados: shutdown 2024-06-11T19:09:15.652 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 985893e9-bd60-46fc-b913-7502ff73628d 1 2024-06-11T19:09:15.657 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:15.658+0000 7f37f0440040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-11T19:09:15.706 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:15.706+0000 7f37f0440040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-11T19:09:15.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.751+0000 7fd11d3f5640 10 monclient: get_monmap_and_config 2024-06-11T19:09:15.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.751+0000 7fd11d3f5640 10 monclient: build_initial_monmap 2024-06-11T19:09:15.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.751+0000 7fd11d3f5640 10 monclient: monmap: 2024-06-11T19:09:15.751 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-06-11T19:09:15.751 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.751 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:09:15.752158+0000 2024-06-11T19:09:15.751 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:09:15.752158+0000 2024-06-11T19:09:15.751 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:15.751 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:15.752 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:15.752 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:15.752 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:15.752 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:15.752 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:15.752 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:15.752 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:15.752 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:15.752 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:15.752 INFO:teuthology.orchestra.run.smithi117.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:15.752 INFO:teuthology.orchestra.run.smithi117.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:15.752 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.751+0000 7fd11d3f5640 1 Processor -- start 2024-06-11T19:09:15.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.751+0000 7fd11d3f5640 1 -- start start 2024-06-11T19:09:15.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.751+0000 7fd11d3f5640 10 monclient: init 2024-06-11T19:09:15.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:15.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 10 monclient: _add_conns ranks=[7,9,2,1,0,10,6,8,4,5,3] 2024-06-11T19:09:15.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd1180a2f40 0x7fd1180a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 10 monclient(hunting): picked mon.noname-i con 0x7fd1180a2f40 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:15.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd1180a4df0 0x7fd1180a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 10 monclient(hunting): picked mon.noname-a con 0x7fd1180a4df0 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1181366d0 0x7fd118136b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 10 monclient(hunting): picked mon.noname-b con 0x7fd1181366d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd1180a5710 con 0x7fd1180a4df0 2024-06-11T19:09:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd1180a3320 con 0x7fd1181366d0 2024-06-11T19:09:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd1180a2460 con 0x7fd1180a2f40 2024-06-11T19:09:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd11d3f5640 10 monclient(hunting): authenticate will time out at 778.564434s 2024-06-11T19:09:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd1180a2f40 0x7fd1180a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd1180a2f40 0x7fd1180a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.117:48398/0 (socket says 172.21.15.117:48398) 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 1 -- 172.21.15.117:0/1616241180 learned_addr learned my addr 172.21.15.117:0/1616241180 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 10 monclient(hunting): get_auth_request con 0x7fd1180a2f40 auth_method 0 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd1167fc640 1 --2- 172.21.15.117:0/1616241180 >> v2:172.21.15.98:3300/0 conn(0x7fd1180a4df0 0x7fd1180a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 1 -- 172.21.15.117:0/1616241180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1181366d0 msgr2=0x7fd118136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd1177fe640 1 --2- 172.21.15.117:0/1616241180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1181366d0 0x7fd118136b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 1 --2- 172.21.15.117:0/1616241180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1181366d0 0x7fd118136b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 1 -- 172.21.15.117:0/1616241180 >> v2:172.21.15.98:3300/0 conn(0x7fd1180a4df0 msgr2=0x7fd1180a51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 1 --2- 172.21.15.117:0/1616241180 >> v2:172.21.15.98:3300/0 conn(0x7fd1180a4df0 0x7fd1180a51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 1 monclient: found mon.noname-i 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd116ffd640 1 -- 172.21.15.117:0/1616241180 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1180a0ed0 con 0x7fd1180a2f40 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd1167fc640 10 monclient: get_auth_request con 0x7fd1180a4df0 auth_method 0 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd1167fc640 1 --2- 172.21.15.117:0/1616241180 >> v2:172.21.15.98:3300/0 conn(0x7fd1180a4df0 0x7fd1180a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd1177fe640 10 monclient: get_auth_request con 0x7fd1181366d0 auth_method 0 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.752+0000 7fd1177fe640 1 --2- 172.21.15.117:0/1616241180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1181366d0 0x7fd118136b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd116ffd640 1 --2- 172.21.15.117:0/1616241180 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd1180a2f40 0x7fd1180a48b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd10c0098a0 tx=0x7fd10c02b7c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=6a5dee8a5039473d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 1 -- 172.21.15.117:0/1616241180 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd10c038070 con 0x7fd1180a2f40 2024-06-11T19:09:15.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 10 monclient: dump: 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.768 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd11d3f5640 5 monclient: authenticate success, global_id 4138 2024-06-11T19:09:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd11d3f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:15.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 1 -- 172.21.15.117:0/1616241180 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd10c031990 con 0x7fd1180a2f40 2024-06-11T19:09:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd11d3f5640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd11d3f5640 4 set_mon_vals no callback set 2024-06-11T19:09:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 1 -- 172.21.15.117:0/1616241180 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd10c0323b0 con 0x7fd1180a2f40 2024-06-11T19:09:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:09:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.753+0000 7fd115ffb640 10 monclient: dump: 2024-06-11T19:09:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.754+0000 7fd11d3f5640 10 monclient: shutdown 2024-06-11T19:09:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.754+0000 7fd11d3f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.754+0000 7fd11d3f5640 1 -- 172.21.15.117:0/1616241180 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd1180a2f40 msgr2=0x7fd1180a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.754+0000 7fd11d3f5640 1 --2- 172.21.15.117:0/1616241180 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd1180a2f40 0x7fd1180a48b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd10c0098a0 tx=0x7fd10c02b7c0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.754+0000 7fd11d3f5640 1 -- 172.21.15.117:0/1616241180 shutdown_connections 2024-06-11T19:09:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.754+0000 7fd11d3f5640 1 --2- 172.21.15.117:0/1616241180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1181366d0 0x7fd118136b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.754+0000 7fd11d3f5640 1 --2- 172.21.15.117:0/1616241180 >> v2:172.21.15.98:3300/0 conn(0x7fd1180a4df0 0x7fd1180a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.754+0000 7fd11d3f5640 1 --2- 172.21.15.117:0/1616241180 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd1180a2f40 0x7fd1180a48b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.754+0000 7fd11d3f5640 1 -- 172.21.15.117:0/1616241180 >> 172.21.15.117:0/1616241180 conn(0x7fd1180a95b0 msgr2=0x7fd1180a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.754+0000 7fd11d3f5640 1 -- 172.21.15.117:0/1616241180 shutdown_connections 2024-06-11T19:09:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.754+0000 7fd11d3f5640 1 -- 172.21.15.117:0/1616241180 wait complete. 2024-06-11T19:09:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.755+0000 7fd11d3f5640 10 monclient: build_initial_monmap 2024-06-11T19:09:15.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.755+0000 7fd11d3f5640 10 monclient: monmap: 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:15.772 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.755+0000 7fd11d3f5640 1 librados: starting msgr at 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.755+0000 7fd11d3f5640 1 librados: starting objecter 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.755+0000 7fd11d3f5640 1 Processor -- start 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.755+0000 7fd11d3f5640 1 -- start start 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.755+0000 7fd11d3f5640 1 librados: setting wanted keys 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.755+0000 7fd11d3f5640 1 librados: calling monclient init 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.755+0000 7fd11d3f5640 10 monclient: init 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 10 monclient: _add_conns ranks=[5,8,1,7,2,3,0,4,6] 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd1180a2f40 0x7fd1181cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 10 monclient(hunting): picked mon.noname-h con 0x7fd1180a2f40 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:15.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1180a4df0 0x7fd1181cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 10 monclient(hunting): picked mon.noname-e con 0x7fd1180a4df0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1181366d0 0x7fd11806bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 10 monclient(hunting): picked mon.noname-f con 0x7fd1181366d0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd1180a1250 con 0x7fd1181366d0 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd118137060 con 0x7fd1180a2f40 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd1180a8170 con 0x7fd1180a4df0 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 10 monclient(hunting): authenticate will time out at 778.568093s 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd1180a2f40 0x7fd1181cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd1180a2f40 0x7fd1181cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.117:50676/0 (socket says 172.21.15.117:50676) 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1177fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1181366d0 0x7fd11806bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 1 -- 172.21.15.117:0/2774473134 learned_addr learned my addr 172.21.15.117:0/2774473134 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 10 monclient(hunting): get_auth_request con 0x7fd1180a2f40 auth_method 0 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1177fe640 10 monclient(hunting): get_auth_request con 0x7fd1181366d0 auth_method 0 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1177fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1177fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1167fc640 1 --2- 172.21.15.117:0/2774473134 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1180a4df0 0x7fd1181cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1167fc640 10 monclient(hunting): get_auth_request con 0x7fd1180a4df0 auth_method 0 2024-06-11T19:09:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1167fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1167fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:15.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 10 monclient(hunting): handle_auth_done global_id 4127 payload 824 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 1 -- 172.21.15.117:0/2774473134 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1180a4df0 msgr2=0x7fd1181cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 1 --2- 172.21.15.117:0/2774473134 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1180a4df0 0x7fd1181cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 1 -- 172.21.15.117:0/2774473134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1181366d0 msgr2=0x7fd11806bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 1 --2- 172.21.15.117:0/2774473134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1181366d0 0x7fd11806bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 1 monclient: found mon.noname-h 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 1 -- 172.21.15.117:0/2774473134 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd118126d30 con 0x7fd1180a2f40 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd116ffd640 1 --2- 172.21.15.117:0/2774473134 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd1180a2f40 0x7fd1181cd8d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd10c0090b0 tx=0x7fd10c0046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 10 client.?.objecter ms_handle_connect 0x7fd1180a2f40 2024-06-11T19:09:15.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:15.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 1 -- 172.21.15.117:0/2774473134 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd10c042020 con 0x7fd1180a2f40 2024-06-11T19:09:15.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:09:15.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 10 monclient: dump: 2024-06-11T19:09:15.778 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:15.778 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.778 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.778 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.778 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.778 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:15.778 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.778 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.779 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.779 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.779 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.779 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.779 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.779 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.779 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.779 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 5 monclient: authenticate success, global_id 4127 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 1 -- 172.21.15.117:0/2774473134 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd10c004370 con 0x7fd1180a2f40 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 10 monclient: _renew_subs 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd11d3f5640 1 -- 172.21.15.117:0/2774473134 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1180a7cb0 con 0x7fd1180a2f40 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 1 -- 172.21.15.117:0/2774473134 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd10c038040 con 0x7fd1180a2f40 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd0ff7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:15.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd0ff7fe640 10 monclient: dump: 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:15.781 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd11d3f5640 10 client.4127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd11d3f5640 10 monclient: _renew_subs 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd11d3f5640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd11d3f5640 1 -- 172.21.15.117:0/2774473134 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1181ce350 con 0x7fd1180a2f40 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd11d3f5640 1 librados: init done 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.756+0000 7fd1157fa640 4 set_mon_vals no callback set 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd0ff7fe640 1 -- 172.21.15.117:0/2774473134 <== mon.5 v2:172.21.15.117:3302/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd10c045020 con 0x7fd1180a2f40 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd0ff7fe640 1 -- 172.21.15.117:0/2774473134 <== mon.5 v2:172.21.15.117:3302/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fd10c030070 con 0x7fd1180a2f40 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd0ff7fe640 10 client.4127.objecter ms_dispatch 0x7fd118000b90 osd_map(3..3 src has 1..3) 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd0ff7fe640 3 client.4127.objecter handle_osd_map got epochs [3,3] > 0 2024-06-11T19:09:15.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd0ff7fe640 3 client.4127.objecter handle_osd_map decoding full epoch 3 2024-06-11T19:09:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd0ff7fe640 20 client.4127.objecter dump_active .. 0 homeless 2024-06-11T19:09:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd11d3f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd11d3f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd11d3f5640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.757+0000 7fd11d3f5640 1 -- 172.21.15.117:0/2774473134 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1180a1590 con 0x7fd1180a2f40 2024-06-11T19:09:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.760+0000 7fd0ff7fe640 1 -- 172.21.15.117:0/2774473134 <== mon.5 v2:172.21.15.117:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fd1180a1590 con 0x7fd1180a2f40 2024-06-11T19:09:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.760+0000 7fd0ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:15.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.760+0000 7fd0ff7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:15.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.832+0000 7fd11d3f5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "985893e9-bd60-46fc-b913-7502ff73628d", "id": 1}] 2024-06-11T19:09:15.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.832+0000 7fd11d3f5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "985893e9-bd60-46fc-b913-7502ff73628d", "id": 1}] 2024-06-11T19:09:15.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.832+0000 7fd11d3f5640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:15.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.832+0000 7fd11d3f5640 1 -- 172.21.15.117:0/2774473134 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_command({"prefix": "osd new", "uuid": "985893e9-bd60-46fc-b913-7502ff73628d", "id": 1} v 0) -- 0x7fd118137390 con 0x7fd1180a2f40 2024-06-11T19:09:15.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.838+0000 7fd0ff7fe640 1 -- 172.21.15.117:0/2774473134 <== mon.5 v2:172.21.15.117:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "985893e9-bd60-46fc-b913-7502ff73628d", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fd10c04c030 con 0x7fd1180a2f40 2024-06-11T19:09:15.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.838+0000 7fd0ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "985893e9-bd60-46fc-b913-7502ff73628d", "id": 1}] 2024-06-11T19:09:15.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.838+0000 7fd0ff7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:15.838 INFO:teuthology.orchestra.run.smithi117.stdout:1 2024-06-11T19:09:15.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.838+0000 7fd11d3f5640 10 librados: watch_flush enter 2024-06-11T19:09:15.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 10 librados: watch_flush exit 2024-06-11T19:09:15.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 20 client.4127.objecter shutdown clearing up homeless session... 2024-06-11T19:09:15.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 10 client.4127.objecter successfully canceled tick 2024-06-11T19:09:15.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 10 monclient: shutdown 2024-06-11T19:09:15.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:15.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 1 -- 172.21.15.117:0/2774473134 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd1180a2f40 msgr2=0x7fd1181cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:15.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 1 --2- 172.21.15.117:0/2774473134 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd1180a2f40 0x7fd1181cd8d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd10c0090b0 tx=0x7fd10c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 1 -- 172.21.15.117:0/2774473134 shutdown_connections 2024-06-11T19:09:15.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 1 --2- 172.21.15.117:0/2774473134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1181366d0 0x7fd11806bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 1 --2- 172.21.15.117:0/2774473134 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1180a4df0 0x7fd1181cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 1 --2- 172.21.15.117:0/2774473134 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd1180a2f40 0x7fd1181cd8d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:15.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 1 -- 172.21.15.117:0/2774473134 >> 172.21.15.117:0/2774473134 conn(0x7fd1180a95b0 msgr2=0x7fd1180a83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:15.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 1 -- 172.21.15.117:0/2774473134 shutdown_connections 2024-06-11T19:09:15.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 1 -- 172.21.15.117:0/2774473134 wait complete. 2024-06-11T19:09:15.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.839+0000 7fd11d3f5640 1 librados: shutdown 2024-06-11T19:09:15.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 2b0b4d24-8009-4cec-80e4-c0fa2d705879 2 2024-06-11T19:09:15.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.985+0000 7fccdd3b5640 10 monclient: get_monmap_and_config 2024-06-11T19:09:15.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.985+0000 7fccdd3b5640 10 monclient: build_initial_monmap 2024-06-11T19:09:15.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.986+0000 7fccdd3b5640 10 monclient: monmap: 2024-06-11T19:09:15.986 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-06-11T19:09:15.986 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:15.986 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:09:15.986876+0000 2024-06-11T19:09:15.986 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:09:15.986876+0000 2024-06-11T19:09:15.986 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:15.986 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:15.986 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:15.986 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:15.986 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:15.987 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:15.987 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:15.987 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:15.987 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:15.987 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:15.987 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:15.987 INFO:teuthology.orchestra.run.smithi117.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:15.987 INFO:teuthology.orchestra.run.smithi117.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:15.987 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:15.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.986+0000 7fccdd3b5640 1 Processor -- start 2024-06-11T19:09:15.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.986+0000 7fccdd3b5640 1 -- start start 2024-06-11T19:09:15.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.986+0000 7fccdd3b5640 10 monclient: init 2024-06-11T19:09:15.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.986+0000 7fccdd3b5640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:15.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 10 monclient: _add_conns ranks=[1,9,2,3,6,4,0,7,5,10,8] 2024-06-11T19:09:15.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fccd81347f0 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:15.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fccd809e6b0 0x7fccd813dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 10 monclient(hunting): picked mon.noname-a con 0x7fccd809e6b0 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:15.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd813e210 0x7fccd8140600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:15.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 10 monclient(hunting): picked mon.noname-b con 0x7fccd813e210 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:15.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccd809d410 con 0x7fccd81347f0 2024-06-11T19:09:15.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fccd80687f0 con 0x7fccd809e6b0 2024-06-11T19:09:15.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fccd809eaf0 con 0x7fccd813e210 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccdd3b5640 10 monclient(hunting): authenticate will time out at 778.799091s 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd6ffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fccd81347f0 0x7fccd8134bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:35088/0 (socket says 172.21.15.117:35088) 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd6ffd640 1 -- 172.21.15.117:0/1129664020 learned_addr learned my addr 172.21.15.117:0/1129664020 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd67fc640 1 --2- 172.21.15.117:0/1129664020 >> v2:172.21.15.98:3300/0 conn(0x7fccd809e6b0 0x7fccd813dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 1 --2- 172.21.15.117:0/1129664020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd813e210 0x7fccd8140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd67fc640 10 monclient(hunting): get_auth_request con 0x7fccd809e6b0 auth_method 0 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd67fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 10 monclient(hunting): get_auth_request con 0x7fccd813e210 auth_method 0 2024-06-11T19:09:15.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd5ffb640 1 -- 172.21.15.117:0/1129664020 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298445801 0 0) 0x7fccd809d410 con 0x7fccd81347f0 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd5ffb640 10 monclient(hunting): _negotiate 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd5ffb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd5ffb640 10 monclient(hunting): my global_id is 4114 2024-06-11T19:09:16.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd5ffb640 1 -- 172.21.15.117:0/1129664020 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccc0003520 con 0x7fccd81347f0 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 10 monclient(hunting): handle_auth_done global_id 4115 payload 274 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 1 -- 172.21.15.117:0/1129664020 >> v2:172.21.15.98:3300/0 conn(0x7fccd809e6b0 msgr2=0x7fccd813dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 1 --2- 172.21.15.117:0/1129664020 >> v2:172.21.15.98:3300/0 conn(0x7fccd809e6b0 0x7fccd813dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 1 -- 172.21.15.117:0/1129664020 >> v1:172.21.15.117:6789/0 conn(0x7fccd81347f0 legacy=0x7fccd8134bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 10 monclient: _finish_hunting 0 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 1 monclient: found mon.noname-b 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd77fe640 1 -- 172.21.15.117:0/1129664020 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd809d6f0 con 0x7fccd813e210 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.987+0000 7fccd67fc640 1 --2- 172.21.15.117:0/1129664020 >> v2:172.21.15.98:3300/0 conn(0x7fccd809e6b0 0x7fccd813dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd77fe640 1 --2- 172.21.15.117:0/1129664020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd813e210 0x7fccd8140600 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fccd0009d90 tx=0x7fccd002bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b74516ce54a70c5f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 1 -- 172.21.15.117:0/1129664020 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fccd0039070 con 0x7fccd813e210 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 10 monclient: dump: 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.002 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.003 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.003 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:16.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:16.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:16.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 1 -- 172.21.15.117:0/1129664020 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccd002cdb0 con 0x7fccd813e210 2024-06-11T19:09:16.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccdd3b5640 5 monclient: authenticate success, global_id 4115 2024-06-11T19:09:16.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccdd3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:16.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:16.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccdd3b5640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:16.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccdd3b5640 4 set_mon_vals no callback set 2024-06-11T19:09:16.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 1 -- 172.21.15.117:0/1129664020 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fccd00327b0 con 0x7fccd813e210 2024-06-11T19:09:16.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.988+0000 7fccd5ffb640 10 monclient: dump: 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.007 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.989+0000 7fccdd3b5640 10 monclient: shutdown 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.989+0000 7fccdd3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.989+0000 7fccdd3b5640 1 -- 172.21.15.117:0/1129664020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd813e210 msgr2=0x7fccd8140600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.989+0000 7fccdd3b5640 1 --2- 172.21.15.117:0/1129664020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd813e210 0x7fccd8140600 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fccd0009d90 tx=0x7fccd002bd90 comp rx=0 tx=0).stop 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.989+0000 7fccdd3b5640 1 -- 172.21.15.117:0/1129664020 shutdown_connections 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.989+0000 7fccdd3b5640 1 --2- 172.21.15.117:0/1129664020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd813e210 0x7fccd8140600 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.989+0000 7fccdd3b5640 1 --2- 172.21.15.117:0/1129664020 >> v2:172.21.15.98:3300/0 conn(0x7fccd809e6b0 0x7fccd813dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.989+0000 7fccdd3b5640 1 -- 172.21.15.117:0/1129664020 >> 172.21.15.117:0/1129664020 conn(0x7fccd80a4780 msgr2=0x7fccd80a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:16.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.989+0000 7fccdd3b5640 1 -- 172.21.15.117:0/1129664020 shutdown_connections 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.989+0000 7fccdd3b5640 1 -- 172.21.15.117:0/1129664020 wait complete. 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.989+0000 7fccdd3b5640 10 monclient: build_initial_monmap 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.989+0000 7fccdd3b5640 10 monclient: monmap: 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:16.009 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 1 librados: starting msgr at 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 1 librados: starting objecter 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 1 Processor -- start 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 1 -- start start 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 1 librados: setting wanted keys 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 1 librados: calling monclient init 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 10 monclient: init 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:16.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 10 monclient: _add_conns ranks=[6,5,8,3,0,4,2,7,1] 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fccd809e6b0 0x7fccd81cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 10 monclient(hunting): picked mon.noname-d con 0x7fccd809e6b0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fccd81347f0 0x7fccd8133730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 10 monclient(hunting): picked mon.noname-h con 0x7fccd81347f0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fccd813e210 0x7fccd8133c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 10 monclient(hunting): picked mon.noname-e con 0x7fccd813e210 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fccd80a56f0 con 0x7fccd81347f0 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fccd80a5570 con 0x7fccd809e6b0 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fccd8068ac0 con 0x7fccd813e210 2024-06-11T19:09:16.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccdd3b5640 10 monclient(hunting): authenticate will time out at 778.802780s 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccd67fc640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fccd81347f0 0x7fccd8133730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccd67fc640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fccd81347f0 0x7fccd8133730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.117:50684/0 (socket says 172.21.15.117:50684) 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccd67fc640 1 -- 172.21.15.117:0/532969458 learned_addr learned my addr 172.21.15.117:0/532969458 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccd67fc640 10 monclient(hunting): get_auth_request con 0x7fccd81347f0 auth_method 0 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccd67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccd67fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.990+0000 7fccd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd77fe640 1 --2- 172.21.15.117:0/532969458 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fccd813e210 0x7fccd8133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd6ffd640 1 --2- 172.21.15.117:0/532969458 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fccd809e6b0 0x7fccd81cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd6ffd640 10 monclient(hunting): get_auth_request con 0x7fccd809e6b0 auth_method 0 2024-06-11T19:09:16.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd77fe640 10 monclient(hunting): get_auth_request con 0x7fccd813e210 auth_method 0 2024-06-11T19:09:16.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 10 monclient(hunting): handle_auth_done global_id 4136 payload 824 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 1 -- 172.21.15.117:0/532969458 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fccd813e210 msgr2=0x7fccd8133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 1 --2- 172.21.15.117:0/532969458 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fccd813e210 0x7fccd8133c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 1 -- 172.21.15.117:0/532969458 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fccd809e6b0 msgr2=0x7fccd81cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 1 --2- 172.21.15.117:0/532969458 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fccd809e6b0 0x7fccd81cd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 10 monclient: _finish_hunting 0 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 1 monclient: found mon.noname-h 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 1 -- 172.21.15.117:0/532969458 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd813d540 con 0x7fccd81347f0 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd67fc640 1 --2- 172.21.15.117:0/532969458 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fccd81347f0 0x7fccd8133730 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00ec10 tx=0x7fcccc00c520 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:16.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 10 client.?.objecter ms_handle_connect 0x7fccd81347f0 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 1 -- 172.21.15.117:0/532969458 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcccc017070 con 0x7fccd81347f0 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 10 monclient: dump: 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.015 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 1 -- 172.21.15.117:0/532969458 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcccc00cc30 con 0x7fccd81347f0 2024-06-11T19:09:16.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccdd3b5640 5 monclient: authenticate success, global_id 4136 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccdd3b5640 10 monclient: _renew_subs 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccdd3b5640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccdd3b5640 1 -- 172.21.15.117:0/532969458 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccd811f840 con 0x7fccd81347f0 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 1 -- 172.21.15.117:0/532969458 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcccc00cde0 con 0x7fccd81347f0 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccdd3b5640 10 client.4136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccbf7fe640 10 monclient: dump: 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.017 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccdd3b5640 10 monclient: _renew_subs 2024-06-11T19:09:16.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccdd3b5640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccdd3b5640 1 -- 172.21.15.117:0/532969458 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fccd811e8e0 con 0x7fccd81347f0 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccd57fa640 4 set_mon_vals no callback set 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.991+0000 7fccdd3b5640 1 librados: init done 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.992+0000 7fccbf7fe640 1 -- 172.21.15.117:0/532969458 <== mon.5 v2:172.21.15.117:3302/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcccc01a070 con 0x7fccd81347f0 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.992+0000 7fccbf7fe640 1 -- 172.21.15.117:0/532969458 <== mon.5 v2:172.21.15.117:3302/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fcccc00f040 con 0x7fccd81347f0 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.992+0000 7fccbf7fe640 10 client.4136.objecter ms_dispatch 0x7fccd8000b90 osd_map(4..4 src has 1..4) 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.992+0000 7fccbf7fe640 3 client.4136.objecter handle_osd_map got epochs [4,4] > 0 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.992+0000 7fccbf7fe640 3 client.4136.objecter handle_osd_map decoding full epoch 4 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.992+0000 7fccbf7fe640 20 client.4136.objecter dump_active .. 0 homeless 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.992+0000 7fccdd3b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.992+0000 7fccdd3b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.992+0000 7fccdd3b5640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.992+0000 7fccdd3b5640 1 -- 172.21.15.117:0/532969458 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccd8134bd0 con 0x7fccd81347f0 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.994+0000 7fccbf7fe640 1 -- 172.21.15.117:0/532969458 <== mon.5 v2:172.21.15.117:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fcccc021070 con 0x7fccd81347f0 2024-06-11T19:09:16.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.994+0000 7fccbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:15.994+0000 7fccbf7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:16.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.070+0000 7fccdd3b5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2b0b4d24-8009-4cec-80e4-c0fa2d705879", "id": 2}] 2024-06-11T19:09:16.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.070+0000 7fccdd3b5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2b0b4d24-8009-4cec-80e4-c0fa2d705879", "id": 2}] 2024-06-11T19:09:16.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.070+0000 7fccdd3b5640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:16.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.070+0000 7fccdd3b5640 1 -- 172.21.15.117:0/532969458 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_command({"prefix": "osd new", "uuid": "2b0b4d24-8009-4cec-80e4-c0fa2d705879", "id": 2} v 0) -- 0x7fccd811f4b0 con 0x7fccd81347f0 2024-06-11T19:09:16.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.076+0000 7fccbf7fe640 1 -- 172.21.15.117:0/532969458 <== mon.5 v2:172.21.15.117:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2b0b4d24-8009-4cec-80e4-c0fa2d705879", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fcccc004020 con 0x7fccd81347f0 2024-06-11T19:09:16.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.076+0000 7fccbf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2b0b4d24-8009-4cec-80e4-c0fa2d705879", "id": 2}] 2024-06-11T19:09:16.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.076+0000 7fccbf7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:16.076 INFO:teuthology.orchestra.run.smithi117.stdout:2 2024-06-11T19:09:16.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.076+0000 7fccdd3b5640 10 librados: watch_flush enter 2024-06-11T19:09:16.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.076+0000 7fccdd3b5640 10 librados: watch_flush exit 2024-06-11T19:09:16.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.076+0000 7fccdd3b5640 20 client.4136.objecter shutdown clearing up homeless session... 2024-06-11T19:09:16.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.076+0000 7fccdd3b5640 10 client.4136.objecter successfully canceled tick 2024-06-11T19:09:16.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.077+0000 7fccdd3b5640 10 monclient: shutdown 2024-06-11T19:09:16.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.077+0000 7fccdd3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:16.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.077+0000 7fccdd3b5640 1 -- 172.21.15.117:0/532969458 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fccd81347f0 msgr2=0x7fccd8133730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.077+0000 7fccdd3b5640 1 --2- 172.21.15.117:0/532969458 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fccd81347f0 0x7fccd8133730 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00ec10 tx=0x7fcccc00c520 comp rx=0 tx=0).stop 2024-06-11T19:09:16.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.077+0000 7fccdd3b5640 1 -- 172.21.15.117:0/532969458 shutdown_connections 2024-06-11T19:09:16.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.077+0000 7fccdd3b5640 1 --2- 172.21.15.117:0/532969458 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fccd813e210 0x7fccd8133c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.077+0000 7fccdd3b5640 1 --2- 172.21.15.117:0/532969458 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fccd81347f0 0x7fccd8133730 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.077+0000 7fccdd3b5640 1 --2- 172.21.15.117:0/532969458 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fccd809e6b0 0x7fccd81cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.077+0000 7fccdd3b5640 1 -- 172.21.15.117:0/532969458 >> 172.21.15.117:0/532969458 conn(0x7fccd80a4780 msgr2=0x7fccd80a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:16.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.077+0000 7fccdd3b5640 1 -- 172.21.15.117:0/532969458 shutdown_connections 2024-06-11T19:09:16.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.077+0000 7fccdd3b5640 1 -- 172.21.15.117:0/532969458 wait complete. 2024-06-11T19:09:16.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.077+0000 7fccdd3b5640 1 librados: shutdown 2024-06-11T19:09:16.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 73793608-9e04-4837-bd2b-1b130a4e88d1 3 2024-06-11T19:09:16.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.185+0000 7f6cfb9d4640 10 monclient: get_monmap_and_config 2024-06-11T19:09:16.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.185+0000 7f6cfb9d4640 10 monclient: build_initial_monmap 2024-06-11T19:09:16.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.185+0000 7f6cfb9d4640 10 monclient: monmap: 2024-06-11T19:09:16.186 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-06-11T19:09:16.186 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.186 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:09:16.186817+0000 2024-06-11T19:09:16.186 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:09:16.186817+0000 2024-06-11T19:09:16.186 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:16.186 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.186 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:16.186 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.186+0000 7f6cfb9d4640 1 Processor -- start 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.186+0000 7f6cfb9d4640 1 -- start start 2024-06-11T19:09:16.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.186+0000 7f6cfb9d4640 10 monclient: init 2024-06-11T19:09:16.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:16.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 10 monclient: _add_conns ranks=[3,7,9,5,2,4,8,1,6,10,0] 2024-06-11T19:09:16.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cf41347f0 0x7f6cf409e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 10 monclient(hunting): picked mon.noname-g con 0x7f6cf41347f0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:16.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6cf4134bd0 0x7f6cf40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 10 monclient(hunting): picked mon.noname-i con 0x7f6cf4134bd0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:16.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6cf4135010 0x7f6cf40a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 10 monclient(hunting): picked mon.noname-a con 0x7f6cf4135010 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:16.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf40687f0 con 0x7f6cf4135010 2024-06-11T19:09:16.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf409d410 con 0x7f6cf41347f0 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f6cf409d6f0 con 0x7f6cf4134bd0 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cfb9d4640 10 monclient(hunting): authenticate will time out at 778.999273s 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cf41347f0 0x7f6cf409e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf8f6f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6cf4134bd0 0x7f6cf40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cf41347f0 0x7f6cf409e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:58132/0 (socket says 172.21.15.117:58132) 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 1 -- 172.21.15.117:0/1736009650 learned_addr learned my addr 172.21.15.117:0/1736009650 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 10 monclient(hunting): get_auth_request con 0x7f6cf41347f0 auth_method 0 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf8f6f640 10 monclient(hunting): get_auth_request con 0x7f6cf4134bd0 auth_method 0 2024-06-11T19:09:16.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf8f6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf8f6f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf8f6f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf8f6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf8f6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf8f6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 10 monclient(hunting): handle_auth_done global_id 4125 payload 274 2024-06-11T19:09:16.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 1 -- 172.21.15.117:0/1736009650 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6cf4134bd0 msgr2=0x7f6cf40a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:16.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 1 --2- 172.21.15.117:0/1736009650 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6cf4134bd0 0x7f6cf40a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9f71640 1 --2- 172.21.15.117:0/1736009650 >> v2:172.21.15.98:3300/0 conn(0x7f6cf4135010 0x7f6cf40a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 1 -- 172.21.15.117:0/1736009650 >> v2:172.21.15.98:3300/0 conn(0x7f6cf4135010 msgr2=0x7f6cf40a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:16.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 1 --2- 172.21.15.117:0/1736009650 >> v2:172.21.15.98:3300/0 conn(0x7f6cf4135010 0x7f6cf40a3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 10 monclient: _finish_hunting 0 2024-06-11T19:09:16.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 1 monclient: found mon.noname-g 2024-06-11T19:09:16.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:16.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:09:16.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf9770640 1 -- 172.21.15.117:0/1736009650 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf409d9d0 con 0x7f6cf41347f0 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.187+0000 7f6cf8f6f640 1 --2- 172.21.15.117:0/1736009650 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6cf4134bd0 0x7f6cf40a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6cf9770640 1 --2- 172.21.15.117:0/1736009650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cf41347f0 0x7f6cf409e6b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009870 tx=0x7f6ce802b860 comp rx=0 tx=0).ready entity=mon.3 client_cookie=42ad22305e8bbcdd server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 1 -- 172.21.15.117:0/1736009650 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ce8031a30 con 0x7f6cf41347f0 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 10 monclient: dump: 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.201 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 10 monclient: _finish_auth 0 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 1 -- 172.21.15.117:0/1736009650 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce8037070 con 0x7f6cf41347f0 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6cfb9d4640 5 monclient: authenticate success, global_id 4125 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6cfb9d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6cfb9d4640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 1 -- 172.21.15.117:0/1736009650 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ce8031c30 con 0x7f6cf41347f0 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6cfb9d4640 4 set_mon_vals no callback set 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.188+0000 7f6ce7fff640 10 monclient: dump: 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 10 monclient: shutdown 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/1736009650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cf41347f0 msgr2=0x7f6cf409e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 1 --2- 172.21.15.117:0/1736009650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cf41347f0 0x7f6cf409e6b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009870 tx=0x7f6ce802b860 comp rx=0 tx=0).stop 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/1736009650 shutdown_connections 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 1 --2- 172.21.15.117:0/1736009650 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6cf4134bd0 0x7f6cf40a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 1 --2- 172.21.15.117:0/1736009650 >> v2:172.21.15.98:3300/0 conn(0x7f6cf4135010 0x7f6cf40a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 1 --2- 172.21.15.117:0/1736009650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cf41347f0 0x7f6cf409e6b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/1736009650 >> 172.21.15.117:0/1736009650 conn(0x7f6cf40a4780 msgr2=0x7f6cf40a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/1736009650 shutdown_connections 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/1736009650 wait complete. 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 10 monclient: build_initial_monmap 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.189+0000 7f6cfb9d4640 10 monclient: monmap: 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 1 librados: starting msgr at 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 1 librados: starting objecter 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 1 Processor -- start 2024-06-11T19:09:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 1 -- start start 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 1 librados: setting wanted keys 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 1 librados: calling monclient init 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 10 monclient: init 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 10 monclient: _add_conns ranks=[4,8,0,7,6,3,1,2,5] 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6cf41347f0 0x7f6cf4133490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 10 monclient(hunting): picked mon.noname-c con 0x7f6cf41347f0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6cf4134bd0 0x7f6cf41339d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 10 monclient(hunting): picked mon.noname-e con 0x7f6cf4134bd0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4135010 0x7f6cf411ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 10 monclient(hunting): picked mon.noname-a con 0x7f6cf4135010 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf4068c40 con 0x7f6cf4135010 2024-06-11T19:09:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f6cf4068ac0 con 0x7f6cf41347f0 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6cf40a4110 con 0x7f6cf4134bd0 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cfb9d4640 10 monclient(hunting): authenticate will time out at 779.002648s 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf9f71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4135010 0x7f6cf411ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf9f71640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4135010 0x7f6cf411ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.117:44116/0 (socket says 172.21.15.117:44116) 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf9770640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6cf41347f0 0x7f6cf4133490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf9f71640 1 -- 172.21.15.117:0/882027855 learned_addr learned my addr 172.21.15.117:0/882027855 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf9f71640 10 monclient(hunting): get_auth_request con 0x7f6cf4135010 auth_method 0 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf9f71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf9f71640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf9f71640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf8f6f640 1 --2- 172.21.15.117:0/882027855 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6cf4134bd0 0x7f6cf41339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf9770640 10 monclient(hunting): get_auth_request con 0x7f6cf41347f0 auth_method 0 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf9770640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf9770640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf9770640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf8f6f640 10 monclient(hunting): get_auth_request con 0x7f6cf4134bd0 auth_method 0 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf8f6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf8f6f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.190+0000 7f6cf8f6f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9f71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9f71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9f71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf8f6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf8f6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf8f6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 1 -- 172.21.15.117:0/882027855 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6cf4134bd0 msgr2=0x7f6cf41339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 1 --2- 172.21.15.117:0/882027855 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6cf4134bd0 0x7f6cf41339d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 1 -- 172.21.15.117:0/882027855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4135010 msgr2=0x7f6cf411ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 1 --2- 172.21.15.117:0/882027855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4135010 0x7f6cf411ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 10 monclient: _finish_hunting 0 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 1 monclient: found mon.noname-c 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 1 -- 172.21.15.117:0/882027855 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf4122b90 con 0x7f6cf41347f0 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9f71640 1 --2- 172.21.15.117:0/882027855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4135010 0x7f6cf411ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cf9770640 1 --2- 172.21.15.117:0/882027855 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6cf41347f0 0x7f6cf4133490 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8033000 tx=0x7f6ce80046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 10 client.?.objecter ms_handle_connect 0x7f6cf41347f0 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 1 -- 172.21.15.117:0/882027855 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ce8041070 con 0x7f6cf41347f0 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 10 monclient: dump: 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.213 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.213 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:16.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 1 -- 172.21.15.117:0/882027855 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce80315c0 con 0x7f6cf41347f0 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cfb9d4640 5 monclient: authenticate success, global_id 4117 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cfb9d4640 10 monclient: _renew_subs 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cfb9d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/882027855 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf4121670 con 0x7f6cf41347f0 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 1 -- 172.21.15.117:0/882027855 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ce8031be0 con 0x7f6cf41347f0 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 10 monclient: dump: 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.215 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce77fe640 4 set_mon_vals no callback set 2024-06-11T19:09:16.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cfb9d4640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cfb9d4640 10 monclient: _renew_subs 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6ce5ffb640 1 -- 172.21.15.117:0/882027855 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6ce8003a60 con 0x7f6cf41347f0 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cfb9d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/882027855 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf41228b0 con 0x7f6cf41347f0 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.191+0000 7f6cfb9d4640 1 librados: init done 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.192+0000 7f6ce5ffb640 1 -- 172.21.15.117:0/882027855 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f6ce8003d50 con 0x7f6cf41347f0 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.192+0000 7f6ce5ffb640 10 client.4117.objecter ms_dispatch 0x7f6cf4000b90 osd_map(5..5 src has 1..5) 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.192+0000 7f6ce5ffb640 3 client.4117.objecter handle_osd_map got epochs [5,5] > 0 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.192+0000 7f6ce5ffb640 3 client.4117.objecter handle_osd_map decoding full epoch 5 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.192+0000 7f6ce5ffb640 20 client.4117.objecter dump_active .. 0 homeless 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.192+0000 7f6cfb9d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.192+0000 7f6cfb9d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.192+0000 7f6cfb9d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:16.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.192+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/882027855 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf409c9e0 con 0x7f6cf41347f0 2024-06-11T19:09:16.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.195+0000 7f6ce5ffb640 1 -- 172.21.15.117:0/882027855 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f6cf409c9e0 con 0x7f6cf41347f0 2024-06-11T19:09:16.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.195+0000 7f6ce5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.195+0000 7f6ce5ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:16.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.269+0000 7f6cfb9d4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "73793608-9e04-4837-bd2b-1b130a4e88d1", "id": 3}] 2024-06-11T19:09:16.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.269+0000 7f6cfb9d4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "73793608-9e04-4837-bd2b-1b130a4e88d1", "id": 3}] 2024-06-11T19:09:16.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.269+0000 7f6cfb9d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:16.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.269+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/882027855 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd new", "uuid": "73793608-9e04-4837-bd2b-1b130a4e88d1", "id": 3} v 0) -- 0x7f6cf409cda0 con 0x7f6cf41347f0 2024-06-11T19:09:16.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.275+0000 7f6ce5ffb640 1 -- 172.21.15.117:0/882027855 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "73793608-9e04-4837-bd2b-1b130a4e88d1", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f6cf409cda0 con 0x7f6cf41347f0 2024-06-11T19:09:16.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.275+0000 7f6ce5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "73793608-9e04-4837-bd2b-1b130a4e88d1", "id": 3}] 2024-06-11T19:09:16.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.275+0000 7f6ce5ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:16.275 INFO:teuthology.orchestra.run.smithi117.stdout:3 2024-06-11T19:09:16.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 10 librados: watch_flush enter 2024-06-11T19:09:16.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 10 librados: watch_flush exit 2024-06-11T19:09:16.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 20 client.4117.objecter shutdown clearing up homeless session... 2024-06-11T19:09:16.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 10 client.4117.objecter successfully canceled tick 2024-06-11T19:09:16.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 10 monclient: shutdown 2024-06-11T19:09:16.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:16.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/882027855 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6cf41347f0 msgr2=0x7f6cf4133490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 1 --2- 172.21.15.117:0/882027855 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6cf41347f0 0x7f6cf4133490 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8033000 tx=0x7f6ce80046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/882027855 shutdown_connections 2024-06-11T19:09:16.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 1 --2- 172.21.15.117:0/882027855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf4135010 0x7f6cf411ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 1 --2- 172.21.15.117:0/882027855 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6cf4134bd0 0x7f6cf41339d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 1 --2- 172.21.15.117:0/882027855 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6cf41347f0 0x7f6cf4133490 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/882027855 >> 172.21.15.117:0/882027855 conn(0x7f6cf40a4780 msgr2=0x7f6cf41c0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:16.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/882027855 shutdown_connections 2024-06-11T19:09:16.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 1 -- 172.21.15.117:0/882027855 wait complete. 2024-06-11T19:09:16.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.276+0000 7f6cfb9d4640 1 librados: shutdown 2024-06-11T19:09:16.286 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 60e657ae-e2a2-471e-b998-fba98d9c94e9 4 2024-06-11T19:09:16.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.385+0000 7fad583a4640 10 monclient: get_monmap_and_config 2024-06-11T19:09:16.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.385+0000 7fad583a4640 10 monclient: build_initial_monmap 2024-06-11T19:09:16.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.385+0000 7fad583a4640 10 monclient: monmap: 2024-06-11T19:09:16.385 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-06-11T19:09:16.385 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.386 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:09:16.386081+0000 2024-06-11T19:09:16.386 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:09:16.386081+0000 2024-06-11T19:09:16.386 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:16.386 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.386 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:16.386 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:16.386 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:16.386 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:16.386 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:16.386 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:16.386 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:16.386 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:16.387 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:16.387 INFO:teuthology.orchestra.run.smithi117.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:16.387 INFO:teuthology.orchestra.run.smithi117.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:16.387 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.385+0000 7fad583a4640 1 Processor -- start 2024-06-11T19:09:16.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.385+0000 7fad583a4640 1 -- start start 2024-06-11T19:09:16.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.385+0000 7fad583a4640 10 monclient: init 2024-06-11T19:09:16.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:16.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 10 monclient: _add_conns ranks=[7,6,5,0,10,4,2,9,1,8,3] 2024-06-11T19:09:16.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a0cd0 0x7fad500a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 10 monclient(hunting): picked mon.noname-i con 0x7fad500a0cd0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:16.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fad50138c80 0x7fad50139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 10 monclient(hunting): picked mon.noname-d con 0x7fad50138c80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:16.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fad500a2250 0x7fad500a26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 10 monclient(hunting): picked mon.noname-h con 0x7fad500a2250 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:16.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fad500739b0 con 0x7fad500a2250 2024-06-11T19:09:16.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fad5009ecf0 con 0x7fad50138c80 2024-06-11T19:09:16.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fad5009efd0 con 0x7fad500a0cd0 2024-06-11T19:09:16.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:16.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad583a4640 10 monclient(hunting): authenticate will time out at 779.198418s 2024-06-11T19:09:16.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56941640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fad500a2250 0x7fad500a26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a0cd0 0x7fad500a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56941640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fad500a2250 0x7fad500a26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.117:50686/0 (socket says 172.21.15.117:50686) 2024-06-11T19:09:16.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a0cd0 0x7fad500a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.117:48422/0 (socket says 172.21.15.117:48422) 2024-06-11T19:09:16.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56941640 1 -- 172.21.15.117:0/998389859 learned_addr learned my addr 172.21.15.117:0/998389859 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:09:16.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56941640 10 monclient(hunting): get_auth_request con 0x7fad500a2250 auth_method 0 2024-06-11T19:09:16.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56941640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56941640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56941640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 10 monclient(hunting): get_auth_request con 0x7fad500a0cd0 auth_method 0 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56941640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56941640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56941640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad5593f640 1 --2- 172.21.15.117:0/998389859 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fad50138c80 0x7fad50139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad5593f640 10 monclient(hunting): get_auth_request con 0x7fad50138c80 auth_method 0 2024-06-11T19:09:16.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad5593f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad5593f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad5593f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 10 monclient(hunting): handle_auth_done global_id 4156 payload 274 2024-06-11T19:09:16.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 1 -- 172.21.15.117:0/998389859 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fad50138c80 msgr2=0x7fad50139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:16.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 1 --2- 172.21.15.117:0/998389859 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fad50138c80 0x7fad50139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 1 -- 172.21.15.117:0/998389859 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fad500a2250 msgr2=0x7fad500a26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:16.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 1 --2- 172.21.15.117:0/998389859 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fad500a2250 0x7fad500a26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 10 monclient: _finish_hunting 0 2024-06-11T19:09:16.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 1 monclient: found mon.noname-i 2024-06-11T19:09:16.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:16.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:09:16.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56140640 1 -- 172.21.15.117:0/998389859 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5009f2b0 con 0x7fad500a0cd0 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.386+0000 7fad56941640 1 --2- 172.21.15.117:0/998389859 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fad500a2250 0x7fad500a26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad56140640 1 --2- 172.21.15.117:0/998389859 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a0cd0 0x7fad500a10b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fad40009830 tx=0x7fad4002b6b0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=2c8e3932a8621119 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 1 -- 172.21.15.117:0/998389859 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad40037070 con 0x7fad500a0cd0 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 10 monclient: dump: 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.401 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 10 monclient: _finish_auth 0 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 10 monclient: _check_auth_tickets 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:16.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad583a4640 5 monclient: authenticate success, global_id 4156 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad583a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 1 -- 172.21.15.117:0/998389859 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad40030980 con 0x7fad500a0cd0 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad583a4640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad583a4640 4 set_mon_vals no callback set 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 1 -- 172.21.15.117:0/998389859 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad400313a0 con 0x7fad500a0cd0 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.387+0000 7fad5513e640 10 monclient: dump: 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.403 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 10 monclient: shutdown 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:16.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 1 -- 172.21.15.117:0/998389859 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a0cd0 msgr2=0x7fad500a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 1 --2- 172.21.15.117:0/998389859 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a0cd0 0x7fad500a10b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fad40009830 tx=0x7fad4002b6b0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 1 -- 172.21.15.117:0/998389859 shutdown_connections 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 1 --2- 172.21.15.117:0/998389859 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fad500a2250 0x7fad500a26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 1 --2- 172.21.15.117:0/998389859 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fad50138c80 0x7fad50139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 1 --2- 172.21.15.117:0/998389859 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a0cd0 0x7fad500a10b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 1 -- 172.21.15.117:0/998389859 >> 172.21.15.117:0/998389859 conn(0x7fad500a7850 msgr2=0x7fad500a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 1 -- 172.21.15.117:0/998389859 shutdown_connections 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 1 -- 172.21.15.117:0/998389859 wait complete. 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 10 monclient: build_initial_monmap 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.388+0000 7fad583a4640 10 monclient: monmap: 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-06-11T19:09:16.405 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 1 librados: starting msgr at 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 1 librados: starting objecter 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 1 Processor -- start 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 1 -- start start 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 1 librados: setting wanted keys 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 1 librados: calling monclient init 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 10 monclient: init 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 10 monclient: _add_conns ranks=[8,7,2,6,1,4,0,3,5] 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad500a0cd0 0x7fad501cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 10 monclient(hunting): picked mon.noname-e con 0x7fad500a0cd0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a2250 0x7fad501ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 10 monclient(hunting): picked mon.noname-i con 0x7fad500a2250 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:16.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad50138c80 0x7fad5012f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 10 monclient(hunting): picked mon.noname-b con 0x7fad50138c80 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fad500a1770 con 0x7fad50138c80 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fad500a15f0 con 0x7fad500a2250 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fad500a2bf0 con 0x7fad500a0cd0 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad583a4640 10 monclient(hunting): authenticate will time out at 779.201790s 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad5593f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a2250 0x7fad501ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad5593f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a2250 0x7fad501ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.117:48438/0 (socket says 172.21.15.117:48438) 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad5593f640 1 -- 172.21.15.117:0/2092122981 learned_addr learned my addr 172.21.15.117:0/2092122981 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:09:16.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad5593f640 10 monclient(hunting): get_auth_request con 0x7fad500a2250 auth_method 0 2024-06-11T19:09:16.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad5593f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad5593f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad5593f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad5593f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad5593f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.389+0000 7fad5593f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad5593f640 10 monclient(hunting): handle_auth_done global_id 4165 payload 824 2024-06-11T19:09:16.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad5593f640 1 -- 172.21.15.117:0/2092122981 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad500a0cd0 msgr2=0x7fad501cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad5593f640 1 --2- 172.21.15.117:0/2092122981 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad500a0cd0 0x7fad501cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad5593f640 1 -- 172.21.15.117:0/2092122981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad50138c80 msgr2=0x7fad5012f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad56941640 1 --2- 172.21.15.117:0/2092122981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad50138c80 0x7fad5012f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad5593f640 1 --2- 172.21.15.117:0/2092122981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad50138c80 0x7fad5012f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad5593f640 10 monclient: _finish_hunting 0 2024-06-11T19:09:16.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad5593f640 1 monclient: found mon.noname-i 2024-06-11T19:09:16.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad5593f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:16.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad5593f640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:09:16.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad5593f640 1 -- 172.21.15.117:0/2092122981 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad501319f0 con 0x7fad500a2250 2024-06-11T19:09:16.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad5593f640 1 --2- 172.21.15.117:0/2092122981 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a2250 0x7fad501ce2b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00d7d0 tx=0x7fad4c00dc90 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:16.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 10 client.?.objecter ms_handle_connect 0x7fad500a2250 2024-06-11T19:09:16.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:16.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 1 -- 172.21.15.117:0/2092122981 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad4c018020 con 0x7fad500a2250 2024-06-11T19:09:16.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 10 monclient: dump: 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.411 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 10 monclient: _finish_auth 0 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 1 -- 172.21.15.117:0/2092122981 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad4c009b30 con 0x7fad500a2250 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad583a4640 5 monclient: authenticate success, global_id 4165 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 1 -- 172.21.15.117:0/2092122981 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad4c009ce0 con 0x7fad500a2250 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad583a4640 10 monclient: _renew_subs 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad583a4640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad583a4640 1 -- 172.21.15.117:0/2092122981 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad501315a0 con 0x7fad500a2250 2024-06-11T19:09:16.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 10 monclient: dump: 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.413 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.414 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.414 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.414 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.414 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.414 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad5493d640 4 set_mon_vals no callback set 2024-06-11T19:09:16.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad583a4640 10 client.4165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:16.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad583a4640 10 monclient: _renew_subs 2024-06-11T19:09:16.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad583a4640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:09:16.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad583a4640 1 -- 172.21.15.117:0/2092122981 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fad50139180 con 0x7fad500a2250 2024-06-11T19:09:16.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad583a4640 1 librados: init done 2024-06-11T19:09:16.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 1 -- 172.21.15.117:0/2092122981 <== mon.7 v2:172.21.15.117:3303/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fad4c01b020 con 0x7fad500a2250 2024-06-11T19:09:16.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 1 -- 172.21.15.117:0/2092122981 <== mon.7 v2:172.21.15.117:3303/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fad4c00f040 con 0x7fad500a2250 2024-06-11T19:09:16.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 10 client.4165.objecter ms_dispatch 0x7fad50000b90 osd_map(6..6 src has 1..6) 2024-06-11T19:09:16.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 3 client.4165.objecter handle_osd_map got epochs [6,6] > 0 2024-06-11T19:09:16.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 3 client.4165.objecter handle_osd_map decoding full epoch 6 2024-06-11T19:09:16.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.390+0000 7fad46ffd640 20 client.4165.objecter dump_active .. 0 homeless 2024-06-11T19:09:16.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.391+0000 7fad583a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.391+0000 7fad583a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.391+0000 7fad583a4640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:09:16.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.391+0000 7fad583a4640 1 -- 172.21.15.117:0/2092122981 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad50138720 con 0x7fad500a2250 2024-06-11T19:09:16.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.393+0000 7fad46ffd640 1 -- 172.21.15.117:0/2092122981 <== mon.7 v2:172.21.15.117:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fad50138720 con 0x7fad500a2250 2024-06-11T19:09:16.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.393+0000 7fad46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.393+0000 7fad46ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:16.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.464+0000 7fad583a4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "60e657ae-e2a2-471e-b998-fba98d9c94e9", "id": 4}] 2024-06-11T19:09:16.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.464+0000 7fad583a4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "60e657ae-e2a2-471e-b998-fba98d9c94e9", "id": 4}] 2024-06-11T19:09:16.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.464+0000 7fad583a4640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:09:16.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.464+0000 7fad583a4640 1 -- 172.21.15.117:0/2092122981 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_command({"prefix": "osd new", "uuid": "60e657ae-e2a2-471e-b998-fba98d9c94e9", "id": 4} v 0) -- 0x7fad500a2f80 con 0x7fad500a2250 2024-06-11T19:09:16.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.470+0000 7fad46ffd640 1 -- 172.21.15.117:0/2092122981 <== mon.7 v2:172.21.15.117:3303/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "60e657ae-e2a2-471e-b998-fba98d9c94e9", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fad4c022070 con 0x7fad500a2250 2024-06-11T19:09:16.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.470+0000 7fad46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "60e657ae-e2a2-471e-b998-fba98d9c94e9", "id": 4}] 2024-06-11T19:09:16.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.470+0000 7fad46ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:16.470 INFO:teuthology.orchestra.run.smithi117.stdout:4 2024-06-11T19:09:16.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 10 librados: watch_flush enter 2024-06-11T19:09:16.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 10 librados: watch_flush exit 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 20 client.4165.objecter shutdown clearing up homeless session... 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 10 client.4165.objecter successfully canceled tick 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 10 monclient: shutdown 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 1 -- 172.21.15.117:0/2092122981 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a2250 msgr2=0x7fad501ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 1 --2- 172.21.15.117:0/2092122981 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a2250 0x7fad501ce2b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00d7d0 tx=0x7fad4c00dc90 comp rx=0 tx=0).stop 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 1 -- 172.21.15.117:0/2092122981 shutdown_connections 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 1 --2- 172.21.15.117:0/2092122981 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad50138c80 0x7fad5012f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 1 --2- 172.21.15.117:0/2092122981 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad500a2250 0x7fad501ce2b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 1 --2- 172.21.15.117:0/2092122981 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad500a0cd0 0x7fad501cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 1 -- 172.21.15.117:0/2092122981 >> 172.21.15.117:0/2092122981 conn(0x7fad500a7850 msgr2=0x7fad501bf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 1 -- 172.21.15.117:0/2092122981 shutdown_connections 2024-06-11T19:09:16.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 1 -- 172.21.15.117:0/2092122981 wait complete. 2024-06-11T19:09:16.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.471+0000 7fad583a4640 1 librados: shutdown 2024-06-11T19:09:16.474 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:16.475+0000 7f37f0440040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-11T19:09:16.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new dae66a65-2ca2-4f8a-98e3-52b190abee7d 5 2024-06-11T19:09:16.567 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:16.567+0000 7f37f0440040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-11T19:09:16.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.577+0000 7f6a911c6640 10 monclient: get_monmap_and_config 2024-06-11T19:09:16.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.577+0000 7f6a911c6640 10 monclient: build_initial_monmap 2024-06-11T19:09:16.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.577+0000 7f6a911c6640 10 monclient: monmap: 2024-06-11T19:09:16.577 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-06-11T19:09:16.577 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.577 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:09:16.578806+0000 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:09:16.578806+0000 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:16.578 INFO:teuthology.orchestra.run.smithi117.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:16.579 INFO:teuthology.orchestra.run.smithi117.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:16.579 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 1 Processor -- start 2024-06-11T19:09:16.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 1 -- start start 2024-06-11T19:09:16.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 10 monclient: init 2024-06-11T19:09:16.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:16.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 10 monclient: _add_conns ranks=[2,10,3,5,7,0,9,1,4,6,8] 2024-06-11T19:09:16.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a8c0a0d10 0x7f6a8c0a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 10 monclient(hunting): picked mon.noname-b con 0x7f6a8c0a0d10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:16.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6a8c138c80 0x7f6a8c139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 10 monclient(hunting): picked mon.noname-f con 0x7f6a8c138c80 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:16.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a8c0a22d0 0x7f6a8c0a26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 10 monclient(hunting): picked mon.noname-g con 0x7f6a8c0a22d0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:16.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f6a8c09f010 con 0x7f6a8c138c80 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6a8c076a00 con 0x7f6a8c0a0d10 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f6a8c09ed30 con 0x7f6a8c0a22d0 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.578+0000 7f6a911c6640 10 monclient(hunting): authenticate will time out at 779.390990s 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a8c0a22d0 0x7f6a8c0a26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8a59c640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6a8c138c80 0x7f6a8c139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a8c0a22d0 0x7f6a8c0a26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:58140/0 (socket says 172.21.15.117:58140) 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8a59c640 10 monclient(hunting): get_auth_request con 0x7f6a8c138c80 auth_method 0 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 1 -- 172.21.15.117:0/796101488 learned_addr learned my addr 172.21.15.117:0/796101488 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 10 monclient(hunting): get_auth_request con 0x7f6a8c0a22d0 auth_method 0 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8ad9d640 1 --2- 172.21.15.117:0/796101488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a8c0a0d10 0x7f6a8c0a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8ad9d640 10 monclient(hunting): get_auth_request con 0x7f6a8c0a0d10 auth_method 0 2024-06-11T19:09:16.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 10 monclient(hunting): handle_auth_done global_id 4134 payload 274 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 1 -- 172.21.15.117:0/796101488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a8c0a0d10 msgr2=0x7f6a8c0a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 1 --2- 172.21.15.117:0/796101488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a8c0a0d10 0x7f6a8c0a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 1 -- 172.21.15.117:0/796101488 >> v2:172.21.15.117:3300/0 conn(0x7f6a8c138c80 msgr2=0x7f6a8c139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 1 --2- 172.21.15.117:0/796101488 >> v2:172.21.15.117:3300/0 conn(0x7f6a8c138c80 0x7f6a8c139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 10 monclient: _finish_hunting 0 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 1 monclient: found mon.noname-g 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8b59e640 1 -- 172.21.15.117:0/796101488 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a8c0a1630 con 0x7f6a8c0a22d0 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.579+0000 7f6a8a59c640 1 --2- 172.21.15.117:0/796101488 >> v2:172.21.15.117:3300/0 conn(0x7f6a8c138c80 0x7f6a8c139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:16.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a8b59e640 1 --2- 172.21.15.117:0/796101488 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a8c0a22d0 0x7f6a8c0a26d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009e80 tx=0x7f6a8402bd80 comp rx=0 tx=0).ready entity=mon.3 client_cookie=d06887ecbbc8b620 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 1 -- 172.21.15.117:0/796101488 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6a8402cb90 con 0x7f6a8c0a22d0 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 10 monclient: dump: 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.593 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 10 monclient: _finish_auth 0 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 1 -- 172.21.15.117:0/796101488 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a8402cd50 con 0x7f6a8c0a22d0 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a911c6640 5 monclient: authenticate success, global_id 4134 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a911c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:16.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a911c6640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 1 -- 172.21.15.117:0/796101488 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6a84034860 con 0x7f6a8c0a22d0 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a911c6640 4 set_mon_vals no callback set 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a89d9b640 10 monclient: dump: 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a911c6640 10 monclient: shutdown 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a911c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a911c6640 1 -- 172.21.15.117:0/796101488 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a8c0a22d0 msgr2=0x7f6a8c0a26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.580+0000 7f6a911c6640 1 --2- 172.21.15.117:0/796101488 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a8c0a22d0 0x7f6a8c0a26d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009e80 tx=0x7f6a8402bd80 comp rx=0 tx=0).stop 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.581+0000 7f6a911c6640 1 -- 172.21.15.117:0/796101488 shutdown_connections 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.581+0000 7f6a911c6640 1 --2- 172.21.15.117:0/796101488 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a8c0a22d0 0x7f6a8c0a26d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.581+0000 7f6a911c6640 1 --2- 172.21.15.117:0/796101488 >> v2:172.21.15.117:3300/0 conn(0x7f6a8c138c80 0x7f6a8c139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.581+0000 7f6a911c6640 1 --2- 172.21.15.117:0/796101488 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6a8c0a0d10 0x7f6a8c0a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.581+0000 7f6a911c6640 1 -- 172.21.15.117:0/796101488 >> 172.21.15.117:0/796101488 conn(0x7f6a8c0a7850 msgr2=0x7f6a8c0a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.581+0000 7f6a911c6640 1 -- 172.21.15.117:0/796101488 shutdown_connections 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.581+0000 7f6a911c6640 1 -- 172.21.15.117:0/796101488 wait complete. 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.582+0000 7f6a911c6640 10 monclient: build_initial_monmap 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.582+0000 7f6a911c6640 10 monclient: monmap: 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:16.597 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.582+0000 7f6a911c6640 1 librados: starting msgr at 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.582+0000 7f6a911c6640 1 librados: starting objecter 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.582+0000 7f6a911c6640 1 Processor -- start 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.582+0000 7f6a911c6640 1 -- start start 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.582+0000 7f6a911c6640 1 librados: setting wanted keys 2024-06-11T19:09:16.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.582+0000 7f6a911c6640 1 librados: calling monclient init 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.582+0000 7f6a911c6640 10 monclient: init 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 10 monclient: _add_conns ranks=[8,1,5,4,2,0,3,6,7] 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6a8c0a0d10 0x7f6a8c1cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 10 monclient(hunting): picked mon.noname-e con 0x7f6a8c0a0d10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a8c0a22d0 0x7f6a8c1ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 10 monclient(hunting): picked mon.noname-f con 0x7f6a8c0a22d0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6a8c138c80 0x7f6a8c12ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 10 monclient(hunting): picked mon.noname-h con 0x7f6a8c138c80 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c0a2c10 con 0x7f6a8c0a22d0 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f6a8c0a3d70 con 0x7f6a8c138c80 2024-06-11T19:09:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6a8c0a2d90 con 0x7f6a8c0a0d10 2024-06-11T19:09:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a911c6640 10 monclient(hunting): authenticate will time out at 779.395692s 2024-06-11T19:09:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8b59e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6a8c138c80 0x7f6a8c12ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8a59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a8c0a22d0 0x7f6a8c1ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8b59e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6a8c138c80 0x7f6a8c12ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.117:50688/0 (socket says 172.21.15.117:50688) 2024-06-11T19:09:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8a59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a8c0a22d0 0x7f6a8c1ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51100/0 (socket says 172.21.15.117:51100) 2024-06-11T19:09:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8b59e640 1 -- 172.21.15.117:0/3831394642 learned_addr learned my addr 172.21.15.117:0/3831394642 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:09:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8b59e640 10 monclient(hunting): get_auth_request con 0x7f6a8c138c80 auth_method 0 2024-06-11T19:09:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8b59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8b59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8a59c640 10 monclient(hunting): get_auth_request con 0x7f6a8c0a22d0 auth_method 0 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.583+0000 7f6a8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8ad9d640 1 --2- 172.21.15.117:0/3831394642 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6a8c0a0d10 0x7f6a8c1cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8ad9d640 10 monclient(hunting): get_auth_request con 0x7f6a8c0a0d10 auth_method 0 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8b59e640 10 monclient(hunting): handle_auth_done global_id 4154 payload 824 2024-06-11T19:09:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8b59e640 1 -- 172.21.15.117:0/3831394642 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6a8c0a0d10 msgr2=0x7f6a8c1cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8b59e640 1 --2- 172.21.15.117:0/3831394642 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6a8c0a0d10 0x7f6a8c1cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8b59e640 1 -- 172.21.15.117:0/3831394642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a8c0a22d0 msgr2=0x7f6a8c1ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8b59e640 1 --2- 172.21.15.117:0/3831394642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a8c0a22d0 0x7f6a8c1ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8b59e640 10 monclient: _finish_hunting 0 2024-06-11T19:09:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8b59e640 1 monclient: found mon.noname-h 2024-06-11T19:09:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8b59e640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:09:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8b59e640 1 -- 172.21.15.117:0/3831394642 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a8c132d90 con 0x7f6a8c138c80 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8b59e640 1 --2- 172.21.15.117:0/3831394642 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6a8c138c80 0x7f6a8c12ef90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6a84036040 tx=0x7f6a84004a40 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 10 client.?.objecter ms_handle_connect 0x7f6a8c138c80 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 1 -- 172.21.15.117:0/3831394642 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6a84042020 con 0x7f6a8c138c80 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 10 monclient: dump: 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.603 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a911c6640 5 monclient: authenticate success, global_id 4154 2024-06-11T19:09:16.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 1 -- 172.21.15.117:0/3831394642 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a840046c0 con 0x7f6a8c138c80 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 1 -- 172.21.15.117:0/3831394642 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6a8403e8c0 con 0x7f6a8c138c80 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a911c6640 10 monclient: _renew_subs 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a911c6640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a911c6640 1 -- 172.21.15.117:0/3831394642 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a8c0a4590 con 0x7f6a8c138c80 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a8959a640 4 set_mon_vals no callback set 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 10 monclient: dump: 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.605 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a911c6640 10 client.4154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a911c6640 10 monclient: _renew_subs 2024-06-11T19:09:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a911c6640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a911c6640 1 -- 172.21.15.117:0/3831394642 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a8c132ab0 con 0x7f6a8c138c80 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a911c6640 1 librados: init done 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 1 -- 172.21.15.117:0/3831394642 <== mon.5 v2:172.21.15.117:3302/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6a84005d10 con 0x7f6a8c138c80 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 1 -- 172.21.15.117:0/3831394642 <== mon.5 v2:172.21.15.117:3302/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f6a8403a040 con 0x7f6a8c138c80 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.584+0000 7f6a737fe640 10 client.4154.objecter ms_dispatch 0x7f6a8c000b90 osd_map(7..7 src has 1..7) 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.585+0000 7f6a737fe640 3 client.4154.objecter handle_osd_map got epochs [7,7] > 0 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.585+0000 7f6a737fe640 3 client.4154.objecter handle_osd_map decoding full epoch 7 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.585+0000 7f6a737fe640 20 client.4154.objecter dump_active .. 0 homeless 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.585+0000 7f6a911c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.585+0000 7f6a911c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.585+0000 7f6a911c6640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.585+0000 7f6a911c6640 1 -- 172.21.15.117:0/3831394642 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a8c139180 con 0x7f6a8c138c80 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.589+0000 7f6a737fe640 1 -- 172.21.15.117:0/3831394642 <== mon.5 v2:172.21.15.117:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f6a8402cc70 con 0x7f6a8c138c80 2024-06-11T19:09:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.589+0000 7f6a737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.589+0000 7f6a737fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:16.614 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:16.615+0000 7f37f0440040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-11T19:09:16.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.660+0000 7f6a911c6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dae66a65-2ca2-4f8a-98e3-52b190abee7d", "id": 5}] 2024-06-11T19:09:16.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.660+0000 7f6a911c6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dae66a65-2ca2-4f8a-98e3-52b190abee7d", "id": 5}] 2024-06-11T19:09:16.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.660+0000 7f6a911c6640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:16.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.660+0000 7f6a911c6640 1 -- 172.21.15.117:0/3831394642 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_command({"prefix": "osd new", "uuid": "dae66a65-2ca2-4f8a-98e3-52b190abee7d", "id": 5} v 0) -- 0x7f6a8c073920 con 0x7f6a8c138c80 2024-06-11T19:09:16.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.668+0000 7f6a737fe640 1 -- 172.21.15.117:0/3831394642 <== mon.5 v2:172.21.15.117:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dae66a65-2ca2-4f8a-98e3-52b190abee7d", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f6a84004250 con 0x7f6a8c138c80 2024-06-11T19:09:16.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.668+0000 7f6a737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dae66a65-2ca2-4f8a-98e3-52b190abee7d", "id": 5}] 2024-06-11T19:09:16.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.668+0000 7f6a737fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:16.668 INFO:teuthology.orchestra.run.smithi117.stdout:5 2024-06-11T19:09:16.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 10 librados: watch_flush enter 2024-06-11T19:09:16.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 10 librados: watch_flush exit 2024-06-11T19:09:16.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 20 client.4154.objecter shutdown clearing up homeless session... 2024-06-11T19:09:16.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 10 client.4154.objecter successfully canceled tick 2024-06-11T19:09:16.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 10 monclient: shutdown 2024-06-11T19:09:16.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:16.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 1 -- 172.21.15.117:0/3831394642 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6a8c138c80 msgr2=0x7f6a8c12ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 1 --2- 172.21.15.117:0/3831394642 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6a8c138c80 0x7f6a8c12ef90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6a84036040 tx=0x7f6a84004a40 comp rx=0 tx=0).stop 2024-06-11T19:09:16.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 1 -- 172.21.15.117:0/3831394642 shutdown_connections 2024-06-11T19:09:16.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 1 --2- 172.21.15.117:0/3831394642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a8c0a22d0 0x7f6a8c1ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 1 --2- 172.21.15.117:0/3831394642 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6a8c138c80 0x7f6a8c12ef90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 1 --2- 172.21.15.117:0/3831394642 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6a8c0a0d10 0x7f6a8c1cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.669+0000 7f6a911c6640 1 -- 172.21.15.117:0/3831394642 >> 172.21.15.117:0/3831394642 conn(0x7f6a8c0a7850 msgr2=0x7f6a8c09d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:16.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.670+0000 7f6a911c6640 1 -- 172.21.15.117:0/3831394642 shutdown_connections 2024-06-11T19:09:16.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.670+0000 7f6a911c6640 1 -- 172.21.15.117:0/3831394642 wait complete. 2024-06-11T19:09:16.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-06-11T19:09:16.670+0000 7f6a911c6640 1 librados: shutdown 2024-06-11T19:09:16.680 INFO:tasks.ceph.osd.0:Restarting daemon 2024-06-11T19:09:16.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-06-11T19:09:16.683 INFO:tasks.ceph.osd.0:Started 2024-06-11T19:09:16.683 INFO:tasks.ceph.osd.1:Restarting daemon 2024-06-11T19:09:16.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-06-11T19:09:16.685 INFO:tasks.ceph.osd.1:Started 2024-06-11T19:09:16.685 INFO:tasks.ceph.osd.2:Restarting daemon 2024-06-11T19:09:16.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-06-11T19:09:16.687 INFO:tasks.ceph.osd.2:Started 2024-06-11T19:09:16.687 INFO:tasks.ceph.osd.3:Restarting daemon 2024-06-11T19:09:16.688 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-06-11T19:09:16.690 INFO:tasks.ceph.osd.3:Started 2024-06-11T19:09:16.690 INFO:tasks.ceph.osd.4:Restarting daemon 2024-06-11T19:09:16.690 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-06-11T19:09:16.693 INFO:tasks.ceph.osd.4:Started 2024-06-11T19:09:16.693 INFO:tasks.ceph.osd.5:Restarting daemon 2024-06-11T19:09:16.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-06-11T19:09:16.696 INFO:tasks.ceph.osd.5:Started 2024-06-11T19:09:16.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-11T19:09:16.795 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:16.794+0000 7f37f0440040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-11T19:09:16.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.810+0000 7f23651f0640 10 monclient: get_monmap_and_config 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.810+0000 7f23651f0640 10 monclient: build_initial_monmap 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.810+0000 7f23651f0640 10 monclient: monmap: 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:16.811172+0000 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:16.811172+0000 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:16.811 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:16.811 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:16.811 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:16.811 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:16.811 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:16.811 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:16.811 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.810+0000 7f23651f0640 1 Processor -- start 2024-06-11T19:09:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.810+0000 7f23651f0640 1 -- start start 2024-06-11T19:09:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.810+0000 7f23651f0640 10 monclient: init 2024-06-11T19:09:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 10 monclient: _add_conns ranks=[2,3,10,7,4,6,9,1,0,8,5] 2024-06-11T19:09:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23600a52a0 0x7f23600a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 10 monclient(hunting): picked mon.noname-b con 0x7f23600a52a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f236009d520 0x7f23600a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 10 monclient(hunting): picked mon.noname-g con 0x7f236009d520 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2360073830 0x7f23600a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 10 monclient(hunting): picked mon.noname-f con 0x7f2360073830 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:16.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f236009daa0 con 0x7f2360073830 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f236009d2e0 con 0x7f23600a52a0 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f23600a5680 con 0x7f236009d520 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f23651f0640 10 monclient(hunting): authenticate will time out at 766.618458s 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23600a52a0 0x7f23600a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23600a52a0 0x7f23600a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:51188/0 (socket says 172.21.15.98:51188) 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 1 -- 172.21.15.98:0/1673373158 learned_addr learned my addr 172.21.15.98:0/1673373158 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 10 monclient(hunting): get_auth_request con 0x7f23600a52a0 auth_method 0 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235e59c640 1 --2- 172.21.15.98:0/1673373158 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f236009d520 0x7f23600a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235e59c640 10 monclient(hunting): get_auth_request con 0x7f236009d520 auth_method 0 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235f59e640 1 --2- 172.21.15.98:0/1673373158 >> v2:172.21.15.117:3300/0 conn(0x7f2360073830 0x7f23600a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235f59e640 10 monclient(hunting): get_auth_request con 0x7f2360073830 auth_method 0 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235f59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235f59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 10 monclient(hunting): handle_auth_done global_id 4160 payload 274 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 1 -- 172.21.15.98:0/1673373158 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f236009d520 msgr2=0x7f23600a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 1 --2- 172.21.15.98:0/1673373158 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f236009d520 0x7f23600a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 1 -- 172.21.15.98:0/1673373158 >> v2:172.21.15.117:3300/0 conn(0x7f2360073830 msgr2=0x7f23600a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 1 --2- 172.21.15.98:0/1673373158 >> v2:172.21.15.117:3300/0 conn(0x7f2360073830 0x7f23600a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 10 monclient: _finish_hunting 0 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 1 monclient: found mon.noname-b 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235ed9d640 1 -- 172.21.15.98:0/1673373158 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f236009e9d0 con 0x7f23600a52a0 2024-06-11T19:09:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.811+0000 7f235f59e640 1 --2- 172.21.15.98:0/1673373158 >> v2:172.21.15.117:3300/0 conn(0x7f2360073830 0x7f23600a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235ed9d640 1 --2- 172.21.15.98:0/1673373158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23600a52a0 0x7f23600a1910 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f23500098a0 tx=0x7f235002b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61c8541046ee292b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 1 -- 172.21.15.98:0/1673373158 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2350038070 con 0x7f23600a52a0 2024-06-11T19:09:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 10 monclient: dump: 2024-06-11T19:09:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.825 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 10 monclient: _finish_auth 0 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 10 monclient: _check_auth_tickets 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 1 -- 172.21.15.98:0/1673373158 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2350031980 con 0x7f23600a52a0 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f23651f0640 5 monclient: authenticate success, global_id 4160 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f23651f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f23651f0640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f23651f0640 4 set_mon_vals no callback set 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 1 -- 172.21.15.98:0/1673373158 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23500323a0 con 0x7f23600a52a0 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.812+0000 7f235dd9b640 10 monclient: dump: 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.828 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.813+0000 7f23651f0640 10 monclient: shutdown 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.813+0000 7f23651f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.813+0000 7f23651f0640 1 -- 172.21.15.98:0/1673373158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23600a52a0 msgr2=0x7f23600a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.813+0000 7f23651f0640 1 --2- 172.21.15.98:0/1673373158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23600a52a0 0x7f23600a1910 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f23500098a0 tx=0x7f235002b7b0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.813+0000 7f23651f0640 1 -- 172.21.15.98:0/1673373158 shutdown_connections 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.813+0000 7f23651f0640 1 --2- 172.21.15.98:0/1673373158 >> v2:172.21.15.117:3300/0 conn(0x7f2360073830 0x7f23600a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.813+0000 7f23651f0640 1 --2- 172.21.15.98:0/1673373158 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f236009d520 0x7f23600a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.813+0000 7f23651f0640 1 --2- 172.21.15.98:0/1673373158 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f23600a52a0 0x7f23600a1910 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.813+0000 7f23651f0640 1 -- 172.21.15.98:0/1673373158 >> 172.21.15.98:0/1673373158 conn(0x7f23600a4780 msgr2=0x7f23600a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.813+0000 7f23651f0640 1 -- 172.21.15.98:0/1673373158 shutdown_connections 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.813+0000 7f23651f0640 1 -- 172.21.15.98:0/1673373158 wait complete. 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.814+0000 7f23651f0640 10 monclient: build_initial_monmap 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.814+0000 7f23651f0640 10 monclient: monmap: 2024-06-11T19:09:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.814+0000 7f23651f0640 1 librados: starting msgr at 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.814+0000 7f23651f0640 1 librados: starting objecter 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.814+0000 7f23651f0640 1 Processor -- start 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.814+0000 7f23651f0640 1 -- start start 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.814+0000 7f23651f0640 1 librados: setting wanted keys 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.814+0000 7f23651f0640 1 librados: calling monclient init 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.814+0000 7f23651f0640 10 monclient: init 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.814+0000 7f23651f0640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 10 monclient: _add_conns ranks=[6,2,1,4,3,0,5,8,7] 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2360073830 0x7f23601cd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 10 monclient(hunting): picked mon.noname-d con 0x7f2360073830 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f236009d520 0x7f23601cde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 10 monclient(hunting): picked mon.noname-b con 0x7f236009d520 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23600a52a0 0x7f236006f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 10 monclient(hunting): picked mon.noname-f con 0x7f23600a52a0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235ed9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2360073830 0x7f23601cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f23600a1fd0 con 0x7f23600a52a0 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f23600a1e50 con 0x7f236009d520 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f23600a2150 con 0x7f2360073830 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 10 monclient(hunting): authenticate will time out at 766.622189s 2024-06-11T19:09:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f236009d520 0x7f23601cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f236009d520 0x7f23601cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:51198/0 (socket says 172.21.15.98:51198) 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 1 -- 172.21.15.98:0/4013712245 learned_addr learned my addr 172.21.15.98:0/4013712245 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 10 monclient(hunting): get_auth_request con 0x7f236009d520 auth_method 0 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235ed9d640 10 monclient(hunting): get_auth_request con 0x7f2360073830 auth_method 0 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235ed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235f59e640 1 --2- 172.21.15.98:0/4013712245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23600a52a0 0x7f236006f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235f59e640 10 monclient(hunting): get_auth_request con 0x7f23600a52a0 auth_method 0 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235f59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235f59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 10 monclient(hunting): handle_auth_done global_id 4169 payload 824 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 1 -- 172.21.15.98:0/4013712245 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2360073830 msgr2=0x7f23601cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 1 --2- 172.21.15.98:0/4013712245 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2360073830 0x7f23601cd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 1 -- 172.21.15.98:0/4013712245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23600a52a0 msgr2=0x7f236006f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 1 --2- 172.21.15.98:0/4013712245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23600a52a0 0x7f236006f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 10 monclient: _finish_hunting 0 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 1 monclient: found mon.noname-b 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 1 -- 172.21.15.98:0/4013712245 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23600a7790 con 0x7f236009d520 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235ed9d640 1 --2- 172.21.15.98:0/4013712245 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2360073830 0x7f23601cd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235e59c640 1 --2- 172.21.15.98:0/4013712245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f236009d520 0x7f23601cde80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f234800e8b0 tx=0x7f234800ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 10 client.?.objecter ms_handle_connect 0x7f236009d520 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 1 -- 172.21.15.98:0/4013712245 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2348017070 con 0x7f236009d520 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 10 monclient: dump: 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 1 -- 172.21.15.98:0/4013712245 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f234800cdb0 con 0x7f236009d520 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 1 -- 172.21.15.98:0/4013712245 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23480138e0 con 0x7f236009d520 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23577fe640 10 monclient: dump: 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:16.838 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f23651f0640 5 monclient: authenticate success, global_id 4169 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.815+0000 7f235d59a640 4 set_mon_vals no callback set 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23651f0640 10 monclient: _renew_subs 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23651f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23651f0640 1 -- 172.21.15.98:0/4013712245 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23600a0890 con 0x7f236009d520 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23651f0640 10 client.4169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23651f0640 10 monclient: _renew_subs 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23651f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23651f0640 1 -- 172.21.15.98:0/4013712245 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2360071450 con 0x7f236009d520 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23651f0640 1 librados: init done 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23577fe640 1 -- 172.21.15.98:0/4013712245 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f234801a070 con 0x7f236009d520 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23577fe640 1 -- 172.21.15.98:0/4013712245 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f234800f040 con 0x7f236009d520 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23577fe640 10 client.4169.objecter ms_dispatch 0x7f2360000b90 osd_map(8..8 src has 1..8) 2024-06-11T19:09:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23577fe640 3 client.4169.objecter handle_osd_map got epochs [8,8] > 0 2024-06-11T19:09:16.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23577fe640 3 client.4169.objecter handle_osd_map decoding full epoch 8 2024-06-11T19:09:16.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.816+0000 7f23577fe640 20 client.4169.objecter dump_active .. 0 homeless 2024-06-11T19:09:16.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.817+0000 7f23651f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.817+0000 7f23651f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.817+0000 7f23651f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:16.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.817+0000 7f23651f0640 1 -- 172.21.15.98:0/4013712245 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23600a1910 con 0x7f236009d520 2024-06-11T19:09:16.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.819+0000 7f23577fe640 1 -- 172.21.15.98:0/4013712245 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f234800c940 con 0x7f236009d520 2024-06-11T19:09:16.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.819+0000 7f23577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:16.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.819+0000 7f23577fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:16.859 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:16.860+0000 7f37f0440040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-11T19:09:16.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.878+0000 7f23651f0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:16.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.878+0000 7f23651f0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:16.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.878+0000 7f23651f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:16.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.878+0000 7f23651f0640 1 -- 172.21.15.98:0/4013712245 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2360069270 con 0x7f236009d520 2024-06-11T19:09:16.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.878+0000 7f23577fe640 1 -- 172.21.15.98:0/4013712245 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f234800cb20 con 0x7f236009d520 2024-06-11T19:09:16.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.878+0000 7f23577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:16.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.878+0000 7f23577fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:09:16.878 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","created":"2024-06-11T19:09:15.016586+0000","modified":"2024-06-11T19:09:16.662340+0000","last_up_change":"0.000000","last_in_change":"2024-06-11T19:09:16.662340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ef1075e3-3e1b-4904-b14b-90751df6c38c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"985893e9-bd60-46fc-b913-7502ff73628d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2b0b4d24-8009-4cec-80e4-c0fa2d705879","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"73793608-9e04-4837-bd2b-1b130a4e88d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60e657ae-e2a2-471e-b998-fba98d9c94e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dae66a65-2ca2-4f8a-98e3-52b190abee7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-11T19:09:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 10 librados: watch_flush enter 2024-06-11T19:09:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 10 librados: watch_flush exit 2024-06-11T19:09:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 20 client.4169.objecter shutdown clearing up homeless session... 2024-06-11T19:09:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 10 client.4169.objecter successfully canceled tick 2024-06-11T19:09:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 10 monclient: shutdown 2024-06-11T19:09:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 1 -- 172.21.15.98:0/4013712245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f236009d520 msgr2=0x7f23601cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 1 --2- 172.21.15.98:0/4013712245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f236009d520 0x7f23601cde80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f234800e8b0 tx=0x7f234800ed70 comp rx=0 tx=0).stop 2024-06-11T19:09:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 1 -- 172.21.15.98:0/4013712245 shutdown_connections 2024-06-11T19:09:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 1 --2- 172.21.15.98:0/4013712245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23600a52a0 0x7f236006f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 1 --2- 172.21.15.98:0/4013712245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f236009d520 0x7f23601cde80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 1 --2- 172.21.15.98:0/4013712245 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2360073830 0x7f23601cd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:16.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 1 -- 172.21.15.98:0/4013712245 >> 172.21.15.98:0/4013712245 conn(0x7f23600a4780 msgr2=0x7f23600a61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:16.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 1 -- 172.21.15.98:0/4013712245 shutdown_connections 2024-06-11T19:09:16.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 1 -- 172.21.15.98:0/4013712245 wait complete. 2024-06-11T19:09:16.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:16.879+0000 7f23651f0640 1 librados: shutdown 2024-06-11T19:09:16.889 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-06-11T19:09:16.889 INFO:tasks.ceph:Waiting for OSDs to come up 2024-06-11T19:09:16.911 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:16.911+0000 7f37f0440040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-11T19:09:17.025 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:17.025+0000 7f37f0440040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-11T19:09:17.285 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-06-11T19:09:17.285+0000 7fc6e811a640 -1 Falling back to public interface 2024-06-11T19:09:17.290 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-06-11T19:09:17.290+0000 7fae1c56e640 -1 Falling back to public interface 2024-06-11T19:09:17.290 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-06-11T19:09:17.290+0000 7f87cf60a640 -1 Falling back to public interface 2024-06-11T19:09:17.297 INFO:tasks.ceph.osd.4.smithi117.stderr:2024-06-11T19:09:17.297+0000 7f5e58e1c640 -1 Falling back to public interface 2024-06-11T19:09:17.297 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-06-11T19:09:17.298+0000 7ff68d3da640 -1 Falling back to public interface 2024-06-11T19:09:17.304 INFO:tasks.ceph.osd.5.smithi117.stderr:2024-06-11T19:09:17.304+0000 7f37583dc640 -1 Falling back to public interface 2024-06-11T19:09:17.406 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:17.406+0000 7f37f0440040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-11T19:09:17.493 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-11T19:09:17.543 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:17.543+0000 7f37f0440040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-06-11T19:09:17.586 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.586+0000 7f4bdd9a4640 10 monclient: get_monmap_and_config 2024-06-11T19:09:17.586 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.586+0000 7f4bdd9a4640 10 monclient: build_initial_monmap 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.586+0000 7f4bdd9a4640 10 monclient: monmap: 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:09:17.587588+0000 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:09:17.587588+0000 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:17.587 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:17.588 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:17.588 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:17.588 INFO:teuthology.misc.health.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:17.588 INFO:teuthology.misc.health.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:17.588 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:17.588 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 1 Processor -- start 2024-06-11T19:09:17.588 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 1 -- start start 2024-06-11T19:09:17.588 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 10 monclient: init 2024-06-11T19:09:17.596 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:17.596 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 10 monclient: _add_conns ranks=[3,9,0,2,1,7,8,5,10,6,4] 2024-06-11T19:09:17.596 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4bd80a3b70 0x7f4bd8138d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:17.596 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 10 monclient(hunting): picked mon.noname-g con 0x7f4bd80a3b70 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bd8139260 0x7f4bd8139640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 10 monclient(hunting): picked mon.noname-a con 0x7f4bd8139260 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bd80a3f50 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bd806d700 con 0x7f4bd80a3f50 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4bd8139b80 con 0x7f4bd8139260 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f4bd806df90 con 0x7f4bd80a3b70 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bdd9a4640 10 monclient(hunting): authenticate will time out at 767.394698s 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd67fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bd8139260 0x7f4bd8139640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd67fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bd8139260 0x7f4bd8139640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35818/0 (socket says 172.21.15.98:35818) 2024-06-11T19:09:17.597 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd77fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4bd80a3f50 0x7f4bd80a43a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56592/0 (socket says 172.21.15.98:56592) 2024-06-11T19:09:17.598 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd67fc640 1 -- 172.21.15.98:0/3397921078 learned_addr learned my addr 172.21.15.98:0/3397921078 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:17.598 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd67fc640 10 monclient(hunting): get_auth_request con 0x7f4bd8139260 auth_method 0 2024-06-11T19:09:17.598 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:17.598 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd67fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:17.598 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:17.598 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:17.598 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:17.598 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd6ffd640 1 --2- 172.21.15.98:0/3397921078 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4bd80a3b70 0x7f4bd8138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:17.598 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:17.598 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd6ffd640 10 monclient(hunting): get_auth_request con 0x7f4bd80a3b70 auth_method 0 2024-06-11T19:09:17.598 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:17.598 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.587+0000 7f4bd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd67fc640 10 monclient(hunting): handle_auth_done global_id 4167 payload 274 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd67fc640 1 -- 172.21.15.98:0/3397921078 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4bd80a3b70 msgr2=0x7f4bd8138d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd67fc640 1 --2- 172.21.15.98:0/3397921078 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4bd80a3b70 0x7f4bd8138d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd67fc640 1 -- 172.21.15.98:0/3397921078 >> v1:172.21.15.98:6789/0 conn(0x7f4bd80a3f50 legacy=0x7f4bd80a43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd67fc640 10 monclient: _finish_hunting 0 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd67fc640 1 monclient: found mon.noname-a 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd5ffb640 1 -- 172.21.15.98:0/3397921078 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2883796951 0 0) 0x7f4bd806d700 con 0x7f4bd80a3f50 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:17.599 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd67fc640 1 -- 172.21.15.98:0/3397921078 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd806e550 con 0x7f4bd8139260 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd5ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd67fc640 1 --2- 172.21.15.98:0/3397921078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd8139260 0x7f4bd8139640 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009870 tx=0x7f4bc002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4fed4f378a593db server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd5ffb640 1 -- 172.21.15.98:0/3397921078 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bc0031a30 con 0x7f4bd8139260 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.588+0000 7f4bd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bd5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bd5ffb640 10 monclient: dump: 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:17.600 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bd5ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bd5ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 5 monclient: authenticate success, global_id 4167 2024-06-11T19:09:17.601 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bd5ffb640 1 -- 172.21.15.98:0/3397921078 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc0037070 con 0x7f4bd8139260 2024-06-11T19:09:17.602 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:17.602 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bd5ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:17.602 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bd5ffb640 1 -- 172.21.15.98:0/3397921078 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bc0031bf0 con 0x7f4bd8139260 2024-06-11T19:09:17.602 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:17.602 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 4 set_mon_vals no callback set 2024-06-11T19:09:17.602 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:17.602 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bd5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:17.602 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bd5ffb640 10 monclient: dump: 2024-06-11T19:09:17.602 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-06-11T19:09:17.602 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:17.602 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:17.602 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:17.603 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 10 monclient: shutdown 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/3397921078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd8139260 msgr2=0x7f4bd8139640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 1 --2- 172.21.15.98:0/3397921078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd8139260 0x7f4bd8139640 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009870 tx=0x7f4bc002b860 comp rx=0 tx=0).stop 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/3397921078 shutdown_connections 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 1 --2- 172.21.15.98:0/3397921078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd8139260 0x7f4bd8139640 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 1 --2- 172.21.15.98:0/3397921078 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4bd80a3b70 0x7f4bd8138d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/3397921078 >> 172.21.15.98:0/3397921078 conn(0x7f4bd806c900 msgr2=0x7f4bd80a9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/3397921078 shutdown_connections 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.589+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/3397921078 wait complete. 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.590+0000 7f4bdd9a4640 10 monclient: build_initial_monmap 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.590+0000 7f4bdd9a4640 10 monclient: monmap: 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-06-11T19:09:17.604 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:17.605 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.590+0000 7f4bdd9a4640 1 librados: starting msgr at 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.590+0000 7f4bdd9a4640 1 librados: starting objecter 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.590+0000 7f4bdd9a4640 1 Processor -- start 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.590+0000 7f4bdd9a4640 1 -- start start 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.590+0000 7f4bdd9a4640 1 librados: setting wanted keys 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.590+0000 7f4bdd9a4640 1 librados: calling monclient init 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.590+0000 7f4bdd9a4640 10 monclient: init 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 10 monclient: _add_conns ranks=[7,1,5,2,3,0,4,8,6] 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4bd80a3b70 0x7f4bd81cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 10 monclient(hunting): picked mon.noname-i con 0x7f4bd80a3b70 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bd80a3f50 0x7f4bd8137f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 10 monclient(hunting): picked mon.noname-f con 0x7f4bd80a3f50 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd8139260 0x7f4bd81384d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 10 monclient(hunting): picked mon.noname-h con 0x7f4bd8139260 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd80a4ab0 con 0x7f4bd80a3f50 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f4bd806d3b0 con 0x7f4bd8139260 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f4bd80a4c30 con 0x7f4bd80a3b70 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bdd9a4640 10 monclient(hunting): authenticate will time out at 767.398241s 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd77fe640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd8139260 0x7f4bd81384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4bd80a3b70 0x7f4bd81cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd67fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bd80a3f50 0x7f4bd8137f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd77fe640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd8139260 0x7f4bd81384d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:38374/0 (socket says 172.21.15.98:38374) 2024-06-11T19:09:17.607 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd77fe640 1 -- 172.21.15.98:0/862972100 learned_addr learned my addr 172.21.15.98:0/862972100 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd67fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bd80a3f50 0x7f4bd8137f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:32842/0 (socket says 172.21.15.98:32842) 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd77fe640 10 monclient(hunting): get_auth_request con 0x7f4bd8139260 auth_method 0 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 10 monclient(hunting): get_auth_request con 0x7f4bd80a3b70 auth_method 0 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd67fc640 10 monclient(hunting): get_auth_request con 0x7f4bd80a3f50 auth_method 0 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd67fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:17.608 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 10 monclient(hunting): handle_auth_done global_id 4192 payload 824 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 1 -- 172.21.15.98:0/862972100 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd8139260 msgr2=0x7f4bd81384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 1 --2- 172.21.15.98:0/862972100 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd8139260 0x7f4bd81384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 1 -- 172.21.15.98:0/862972100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bd80a3f50 msgr2=0x7f4bd8137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 1 --2- 172.21.15.98:0/862972100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bd80a3f50 0x7f4bd8137f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 1 monclient: found mon.noname-i 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 1 -- 172.21.15.98:0/862972100 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd81cd950 con 0x7f4bd80a3b70 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd77fe640 1 --2- 172.21.15.98:0/862972100 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd8139260 0x7f4bd81384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bd6ffd640 1 --2- 172.21.15.98:0/862972100 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4bd80a3b70 0x7f4bd81cd410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800e850 tx=0x7f4bc800ed10 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bcffff640 10 client.?.objecter ms_handle_connect 0x7f4bd80a3b70 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.591+0000 7f4bcffff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 1 -- 172.21.15.98:0/862972100 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bc8017070 con 0x7f4bd80a3b70 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:09:17.610 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 10 monclient: dump: 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:17.611 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 10 monclient: _finish_auth 0 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 1 -- 172.21.15.98:0/862972100 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc800cdb0 con 0x7f4bd80a3b70 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bdd9a4640 5 monclient: authenticate success, global_id 4192 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bdd9a4640 10 monclient: _renew_subs 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bdd9a4640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/862972100 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd8072ca0 con 0x7f4bd80a3b70 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 1 -- 172.21.15.98:0/862972100 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bc80138e0 con 0x7f4bd80a3b70 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bd57fa640 4 set_mon_vals no callback set 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 10 monclient: dump: 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:17.613 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bdd9a4640 10 client.4192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bdd9a4640 10 monclient: _renew_subs 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bdd9a4640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/862972100 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd81253b0 con 0x7f4bd80a3b70 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bdd9a4640 1 librados: init done 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 1 -- 172.21.15.98:0/862972100 <== mon.7 v2:172.21.15.117:3303/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4bc801a070 con 0x7f4bd80a3b70 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 1 -- 172.21.15.98:0/862972100 <== mon.7 v2:172.21.15.117:3303/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f4bc800f040 con 0x7f4bd80a3b70 2024-06-11T19:09:17.614 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 10 client.4192.objecter ms_dispatch 0x7f4bd8000b90 osd_map(8..8 src has 1..8) 2024-06-11T19:09:17.615 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 3 client.4192.objecter handle_osd_map got epochs [8,8] > 0 2024-06-11T19:09:17.615 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 3 client.4192.objecter handle_osd_map decoding full epoch 8 2024-06-11T19:09:17.615 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.592+0000 7f4bcffff640 20 client.4192.objecter dump_active .. 0 homeless 2024-06-11T19:09:17.615 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.593+0000 7f4bdd9a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:17.615 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.593+0000 7f4bdd9a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:17.615 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.593+0000 7f4bdd9a4640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:09:17.615 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.593+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/862972100 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd81250d0 con 0x7f4bd80a3b70 2024-06-11T19:09:17.615 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.595+0000 7f4bcffff640 1 -- 172.21.15.98:0/862972100 <== mon.7 v2:172.21.15.117:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f4bc8021070 con 0x7f4bd80a3b70 2024-06-11T19:09:17.615 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.595+0000 7f4bcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:17.615 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.595+0000 7f4bcffff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:17.653 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.654+0000 7f4bdd9a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:17.654 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.654+0000 7f4bdd9a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:17.654 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.654+0000 7f4bdd9a4640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:09:17.654 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.654+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/862972100 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4bd8138d20 con 0x7f4bd80a3b70 2024-06-11T19:09:17.654 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.654+0000 7f4bcffff640 1 -- 172.21.15.98:0/862972100 <== mon.7 v2:172.21.15.117:3303/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f4bc80134c0 con 0x7f4bd80a3b70 2024-06-11T19:09:17.654 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.654+0000 7f4bcffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:17.655 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.654+0000 7f4bcffff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:17.655 INFO:teuthology.misc.health.smithi098.stdout: 2024-06-11T19:09:17.655 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":8,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","created":"2024-06-11T19:09:15.016586+0000","modified":"2024-06-11T19:09:16.662340+0000","last_up_change":"0.000000","last_in_change":"2024-06-11T19:09:16.662340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ef1075e3-3e1b-4904-b14b-90751df6c38c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"985893e9-bd60-46fc-b913-7502ff73628d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2b0b4d24-8009-4cec-80e4-c0fa2d705879","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"73793608-9e04-4837-bd2b-1b130a4e88d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"60e657ae-e2a2-471e-b998-fba98d9c94e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dae66a65-2ca2-4f8a-98e3-52b190abee7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-11T19:09:17.656 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 10 librados: watch_flush enter 2024-06-11T19:09:17.656 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 10 librados: watch_flush exit 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 20 client.4192.objecter shutdown clearing up homeless session... 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 10 client.4192.objecter successfully canceled tick 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 10 monclient: shutdown 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/862972100 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4bd80a3b70 msgr2=0x7f4bd81cd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 1 --2- 172.21.15.98:0/862972100 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4bd80a3b70 0x7f4bd81cd410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800e850 tx=0x7f4bc800ed10 comp rx=0 tx=0).stop 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/862972100 shutdown_connections 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 1 --2- 172.21.15.98:0/862972100 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd8139260 0x7f4bd81384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 1 --2- 172.21.15.98:0/862972100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bd80a3f50 0x7f4bd8137f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 1 --2- 172.21.15.98:0/862972100 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4bd80a3b70 0x7f4bd81cd410 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/862972100 >> 172.21.15.98:0/862972100 conn(0x7f4bd806c900 msgr2=0x7f4bd80a2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/862972100 shutdown_connections 2024-06-11T19:09:17.657 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 1 -- 172.21.15.98:0/862972100 wait complete. 2024-06-11T19:09:17.658 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:17.655+0000 7f4bdd9a4640 1 librados: shutdown 2024-06-11T19:09:17.664 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-06-11T19:09:17.802 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:17.803+0000 7f37f0440040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-11T19:09:17.855 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:17.855+0000 7f37f0440040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-11T19:09:17.970 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:17.971+0000 7f37f0440040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-11T19:09:18.263 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:18.263+0000 7f37f0440040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-11T19:09:18.309 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:18.310+0000 7f37f0440040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-11T19:09:18.466 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:18.466+0000 7f37f0440040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-11T19:09:18.600 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:18.600+0000 7f37f0440040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-11T19:09:18.649 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:18.650+0000 7f37f0440040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-11T19:09:18.720 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:18.720+0000 7f37f0440040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-11T19:09:18.764 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:18.765+0000 7f37f0440040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-11T19:09:19.239 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:19.239+0000 7f37f0440040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-11T19:09:19.353 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:19.353+0000 7f37f0440040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-11T19:09:20.369 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:20.369+0000 7f378ff5a640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-06-11T19:09:20.369 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:20.369+0000 7f378ff5a640 -1 mgr.server handle_report got status from non-daemon mon.e 2024-06-11T19:09:20.369 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:20.369+0000 7f378ff5a640 -1 mgr.server handle_report got status from non-daemon mon.g 2024-06-11T19:09:20.370 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:20.369+0000 7f378ff5a640 -1 mgr.server handle_report got status from non-daemon mon.f 2024-06-11T19:09:20.370 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:20.369+0000 7f378ff5a640 -1 mgr.server handle_report got status from non-daemon mon.i 2024-06-11T19:09:20.370 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:20.370+0000 7f378ff5a640 -1 mgr.server handle_report got status from non-daemon mon.h 2024-06-11T19:09:20.370 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:20.370+0000 7f378ff5a640 -1 mgr.server handle_report got status from non-daemon mon.d 2024-06-11T19:09:20.370 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-06-11T19:09:20.370+0000 7f378ff5a640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-06-11T19:09:20.764 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-06-11T19:09:20.764+0000 7f87cf60a640 -1 osd.0 0 log_to_monitors true 2024-06-11T19:09:20.766 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-06-11T19:09:20.767+0000 7fc6e811a640 -1 osd.1 0 log_to_monitors true 2024-06-11T19:09:20.791 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-06-11T19:09:20.792+0000 7fae1c56e640 -1 osd.2 0 log_to_monitors true 2024-06-11T19:09:20.796 INFO:tasks.ceph.osd.4.smithi117.stderr:2024-06-11T19:09:20.797+0000 7f5e58e1c640 -1 osd.4 0 log_to_monitors true 2024-06-11T19:09:20.800 INFO:tasks.ceph.osd.5.smithi117.stderr:2024-06-11T19:09:20.800+0000 7f37583dc640 -1 osd.5 0 log_to_monitors true 2024-06-11T19:09:20.823 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-06-11T19:09:20.823+0000 7ff68d3da640 -1 osd.3 0 log_to_monitors true 2024-06-11T19:09:22.383 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-06-11T19:09:22.384+0000 7ff689370640 -1 osd.3 0 waiting for initial osdmap 2024-06-11T19:09:22.384 INFO:tasks.ceph.osd.4.smithi117.stderr:2024-06-11T19:09:22.384+0000 7f5e5475c640 -1 osd.4 0 waiting for initial osdmap 2024-06-11T19:09:22.386 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-06-11T19:09:22.387+0000 7f87cb5a0640 -1 osd.0 0 waiting for initial osdmap 2024-06-11T19:09:22.386 INFO:tasks.ceph.osd.4.smithi117.stderr:2024-06-11T19:09:22.387+0000 7f5e50584640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-11T19:09:22.389 INFO:tasks.ceph.osd.5.smithi117.stderr:2024-06-11T19:09:22.390+0000 7f3754372640 -1 osd.5 0 waiting for initial osdmap 2024-06-11T19:09:22.436 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-06-11T19:09:22.436+0000 7ff685198640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-11T19:09:22.436 INFO:tasks.ceph.osd.5.smithi117.stderr:2024-06-11T19:09:22.436+0000 7f375019a640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-11T19:09:22.543 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-06-11T19:09:22.543+0000 7f87c73c8640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-11T19:09:23.647 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-06-11T19:09:23.648+0000 7fae18504640 -1 osd.2 0 waiting for initial osdmap 2024-06-11T19:09:23.653 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-06-11T19:09:23.653+0000 7fc6e40b0640 -1 osd.1 0 waiting for initial osdmap 2024-06-11T19:09:23.663 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-06-11T19:09:23.664+0000 7fae10324640 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-11T19:09:23.672 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-06-11T19:09:23.673+0000 7fc6dbed0640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-11T19:09:24.269 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-11T19:09:24.367 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.368+0000 7f0686c4d640 10 monclient: get_monmap_and_config 2024-06-11T19:09:24.368 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.368+0000 7f0686c4d640 10 monclient: build_initial_monmap 2024-06-11T19:09:24.368 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.368+0000 7f0686c4d640 10 monclient: monmap: 2024-06-11T19:09:24.368 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-06-11T19:09:24.368 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:24.368 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:09:24.369147+0000 2024-06-11T19:09:24.368 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:09:24.369147+0000 2024-06-11T19:09:24.368 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:24.368 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:24.369 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:24.369 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:24.369 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:24.369 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:24.369 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:24.369 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:24.369 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:24.369 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:24.369 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:24.369 INFO:teuthology.misc.health.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:24.369 INFO:teuthology.misc.health.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:24.370 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:24.379 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.368+0000 7f0686c4d640 1 Processor -- start 2024-06-11T19:09:24.379 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 1 -- start start 2024-06-11T19:09:24.379 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 10 monclient: init 2024-06-11T19:09:24.379 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:24.379 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 10 monclient: _add_conns ranks=[6,2,8,0,7,5,4,3,9,10,1] 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0680075ef0 0x7f06801355a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 10 monclient(hunting): picked mon.noname-d con 0x7f0680075ef0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0680072f70 0x7f0680073370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 10 monclient(hunting): picked mon.noname-b con 0x7f0680072f70 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f06800a3240 0x7f06800a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 10 monclient(hunting): picked mon.noname-e con 0x7f06800a3240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f06800a10e0 con 0x7f0680072f70 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f06800762d0 con 0x7f0680075ef0 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f06800a12b0 con 0x7f06800a3240 2024-06-11T19:09:24.380 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f0686c4d640 10 monclient(hunting): authenticate will time out at 774.176720s 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f067ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0680072f70 0x7f0680073370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f06851ea640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f06800a3240 0x7f06800a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f06849e9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0680075ef0 0x7f06801355a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f067ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0680072f70 0x7f0680073370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:53724/0 (socket says 172.21.15.98:53724) 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f067ffff640 1 -- 172.21.15.98:0/1654823642 learned_addr learned my addr 172.21.15.98:0/1654823642 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f067ffff640 10 monclient(hunting): get_auth_request con 0x7f0680072f70 auth_method 0 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f067ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f067ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f067ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f06849e9640 10 monclient(hunting): get_auth_request con 0x7f0680075ef0 auth_method 0 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f06849e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:24.381 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f06849e9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f06849e9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f06851ea640 10 monclient(hunting): get_auth_request con 0x7f06800a3240 auth_method 0 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f06851ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f06851ea640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f06851ea640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f067ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f067ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.369+0000 7f067ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06849e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06849e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06849e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:24.382 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 10 monclient(hunting): handle_auth_done global_id 4202 payload 274 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 1 -- 172.21.15.98:0/1654823642 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0680075ef0 msgr2=0x7f06801355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 1 --2- 172.21.15.98:0/1654823642 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0680075ef0 0x7f06801355a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 1 -- 172.21.15.98:0/1654823642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0680072f70 msgr2=0x7f0680073370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 1 --2- 172.21.15.98:0/1654823642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0680072f70 0x7f0680073370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 10 monclient: _finish_hunting 0 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 1 monclient: found mon.noname-e 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 1 -- 172.21.15.98:0/1654823642 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068006f460 con 0x7f06800a3240 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06849e9640 1 --2- 172.21.15.98:0/1654823642 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0680075ef0 0x7f06801355a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f067ffff640 1 --2- 172.21.15.98:0/1654823642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0680072f70 0x7f0680073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:24.383 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.370+0000 7f06851ea640 1 --2- 172.21.15.98:0/1654823642 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f06800a3240 0x7f06800a36c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f06800a1460 tx=0x7f067002b6a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=f3d8d32e70c1a334 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:24.384 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 1 -- 172.21.15.98:0/1654823642 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0670037070 con 0x7f06800a3240 2024-06-11T19:09:24.384 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:24.384 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:24.384 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 10 monclient: dump: 2024-06-11T19:09:24.384 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-06-11T19:09:24.384 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:24.384 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:24.384 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:24.384 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:24.384 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:24.384 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:24.384 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:24.385 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:24.385 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:24.385 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:24.385 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:24.385 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:24.385 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:24.385 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:24.385 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:24.385 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:24.385 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:24.385 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:24.386 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 1 -- 172.21.15.98:0/1654823642 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0670030980 con 0x7f06800a3240 2024-06-11T19:09:24.386 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 5 monclient: authenticate success, global_id 4202 2024-06-11T19:09:24.386 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:24.386 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:24.386 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 1 -- 172.21.15.98:0/1654823642 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0670030bb0 con 0x7f06800a3240 2024-06-11T19:09:24.386 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:24.386 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 4 set_mon_vals no callback set 2024-06-11T19:09:24.386 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:24.386 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:24.386 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f067f7fe640 10 monclient: dump: 2024-06-11T19:09:24.386 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-06-11T19:09:24.386 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:24.387 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:24.387 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:24.387 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:24.387 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:24.387 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:24.387 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:24.387 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:24.388 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:24.388 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:24.388 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:24.388 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:24.388 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:24.388 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:24.388 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:24.388 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 10 monclient: shutdown 2024-06-11T19:09:24.388 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:24.388 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 1 -- 172.21.15.98:0/1654823642 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f06800a3240 msgr2=0x7f06800a36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:24.388 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 1 --2- 172.21.15.98:0/1654823642 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f06800a3240 0x7f06800a36c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f06800a1460 tx=0x7f067002b6a0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.389 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 1 -- 172.21.15.98:0/1654823642 shutdown_connections 2024-06-11T19:09:24.389 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 1 --2- 172.21.15.98:0/1654823642 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f06800a3240 0x7f06800a36c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.389 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 1 --2- 172.21.15.98:0/1654823642 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0680072f70 0x7f0680073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.389 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 1 --2- 172.21.15.98:0/1654823642 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0680075ef0 0x7f06801355a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.389 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 1 -- 172.21.15.98:0/1654823642 >> 172.21.15.98:0/1654823642 conn(0x7f06800a7e60 msgr2=0x7f06800a4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:24.389 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 1 -- 172.21.15.98:0/1654823642 shutdown_connections 2024-06-11T19:09:24.389 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.371+0000 7f0686c4d640 1 -- 172.21.15.98:0/1654823642 wait complete. 2024-06-11T19:09:24.389 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.372+0000 7f0686c4d640 10 monclient: build_initial_monmap 2024-06-11T19:09:24.390 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.372+0000 7f0686c4d640 10 monclient: monmap: 2024-06-11T19:09:24.390 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-06-11T19:09:24.390 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:24.390 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:24.390 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-06-11T19:09:24.390 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:24.390 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:24.390 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:24.390 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:24.390 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:24.390 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:24.390 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.372+0000 7f0686c4d640 1 librados: starting msgr at 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.372+0000 7f0686c4d640 1 librados: starting objecter 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.372+0000 7f0686c4d640 1 Processor -- start 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.372+0000 7f0686c4d640 1 -- start start 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.372+0000 7f0686c4d640 1 librados: setting wanted keys 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.372+0000 7f0686c4d640 1 librados: calling monclient init 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.372+0000 7f0686c4d640 10 monclient: init 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:24.391 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 10 monclient: _add_conns ranks=[0,3,4,1,8,2,5,6,7] 2024-06-11T19:09:24.392 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0680072f70 0x7f06801cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:24.392 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 10 monclient(hunting): picked mon.noname-a con 0x7f0680072f70 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:24.392 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0680075ef0 0x7f06801cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:24.392 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 10 monclient(hunting): picked mon.noname-g con 0x7f0680075ef0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:24.392 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06800a3240 0x7f06801ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:24.392 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 10 monclient(hunting): picked mon.noname-c con 0x7f06800a3240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:24.392 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:24.392 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f068006f180 con 0x7f0680072f70 2024-06-11T19:09:24.392 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:24.392 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f06800738b0 con 0x7f0680075ef0 2024-06-11T19:09:24.392 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f06800a3c00 con 0x7f06800a3240 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f0686c4d640 10 monclient(hunting): authenticate will time out at 774.180535s 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06800a3240 0x7f06801ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06849e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0680072f70 0x7f06801cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06800a3240 0x7f06801ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:37324/0 (socket says 172.21.15.98:37324) 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 1 -- 172.21.15.98:0/1933016746 learned_addr learned my addr 172.21.15.98:0/1933016746 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 10 monclient(hunting): get_auth_request con 0x7f06800a3240 auth_method 0 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06849e9640 10 monclient(hunting): get_auth_request con 0x7f0680072f70 auth_method 0 2024-06-11T19:09:24.393 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06849e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06849e9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06849e9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06849e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06849e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06849e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f067ffff640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0680075ef0 0x7f06801cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 10 monclient(hunting): handle_auth_done global_id 4207 payload 824 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 1 -- 172.21.15.98:0/1933016746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0680075ef0 msgr2=0x7f06801cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0680075ef0 0x7f06801cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.394 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 1 -- 172.21.15.98:0/1933016746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0680072f70 msgr2=0x7f06801cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:24.395 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0680072f70 0x7f06801cd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.395 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 10 monclient: _finish_hunting 0 2024-06-11T19:09:24.395 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 1 monclient: found mon.noname-c 2024-06-11T19:09:24.395 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:24.395 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:24.395 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06851ea640 1 -- 172.21.15.98:0/1933016746 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06800a6cb0 con 0x7f06800a3240 2024-06-11T19:09:24.395 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.373+0000 7f06849e9640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0680072f70 0x7f06801cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:24.395 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f06851ea640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06800a3240 0x7f06801ce230 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f06700097d0 tx=0x7f06700046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:24.395 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 10 client.?.objecter ms_handle_connect 0x7f06800a3240 2024-06-11T19:09:24.395 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:24.396 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 1 -- 172.21.15.98:0/1933016746 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0670037050 con 0x7f06800a3240 2024-06-11T19:09:24.396 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:24.396 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:24.396 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 10 monclient: dump: 2024-06-11T19:09:24.396 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-06-11T19:09:24.396 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:24.396 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:24.396 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:24.396 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:24.396 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:24.396 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:24.396 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:24.397 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:24.397 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:24.397 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:24.397 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:24.397 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:24.397 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:24.397 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:24.397 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:24.397 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 10 monclient: _finish_auth 0 2024-06-11T19:09:24.397 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:09:24.398 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:24.398 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f0686c4d640 5 monclient: authenticate success, global_id 4207 2024-06-11T19:09:24.398 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 1 -- 172.21.15.98:0/1933016746 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0670043070 con 0x7f06800a3240 2024-06-11T19:09:24.398 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f0686c4d640 10 monclient: _renew_subs 2024-06-11T19:09:24.398 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f0686c4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:24.398 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f0686c4d640 1 -- 172.21.15.98:0/1933016746 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0680121290 con 0x7f06800a3240 2024-06-11T19:09:24.398 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:24.398 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 1 -- 172.21.15.98:0/1933016746 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0670030eb0 con 0x7f06800a3240 2024-06-11T19:09:24.398 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:24.399 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:24.399 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067effd640 4 set_mon_vals no callback set 2024-06-11T19:09:24.399 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f0686c4d640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:24.399 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f067d7fa640 10 monclient: dump: 2024-06-11T19:09:24.399 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-06-11T19:09:24.399 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:24.399 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:24.399 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:24.399 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:24.399 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:24.399 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:24.399 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:24.400 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:24.400 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:24.400 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:24.400 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:24.400 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:24.400 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:24.400 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:24.400 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:24.400 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f0686c4d640 10 monclient: _renew_subs 2024-06-11T19:09:24.401 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f0686c4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:24.401 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f0686c4d640 1 -- 172.21.15.98:0/1933016746 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f068011fb30 con 0x7f06800a3240 2024-06-11T19:09:24.401 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.374+0000 7f0686c4d640 1 librados: init done 2024-06-11T19:09:24.401 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.375+0000 7f067d7fa640 1 -- 172.21.15.98:0/1933016746 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 4) ==== 49355+0+0 (secure 0 0 0) 0x7f0670003610 con 0x7f06800a3240 2024-06-11T19:09:24.401 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.375+0000 7f0686c4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:24.401 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.375+0000 7f0686c4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:24.401 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.375+0000 7f0686c4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:24.401 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.375+0000 7f0686c4d640 1 -- 172.21.15.98:0/1933016746 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06800a3fa0 con 0x7f06800a3240 2024-06-11T19:09:24.401 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.375+0000 7f067d7fa640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f066003fd20 0x7f06600421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:24.402 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.375+0000 7f067d7fa640 1 -- 172.21.15.98:0/1933016746 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(11..11 src has 1..11) ==== 3047+0+0 (secure 0 0 0) 0x7f0670072a60 con 0x7f06800a3240 2024-06-11T19:09:24.402 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.375+0000 7f067d7fa640 10 client.4207.objecter ms_dispatch 0x7f0680000b90 osd_map(11..11 src has 1..11) 2024-06-11T19:09:24.402 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.375+0000 7f067d7fa640 3 client.4207.objecter handle_osd_map got epochs [11,11] > 0 2024-06-11T19:09:24.402 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.375+0000 7f067d7fa640 3 client.4207.objecter handle_osd_map decoding full epoch 11 2024-06-11T19:09:24.402 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.375+0000 7f067d7fa640 20 client.4207.objecter dump_active .. 0 homeless 2024-06-11T19:09:24.402 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.375+0000 7f06849e9640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f066003fd20 0x7f06600421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:24.402 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.376+0000 7f06849e9640 10 monclient: get_auth_request con 0x7f066003fd20 auth_method 0 2024-06-11T19:09:24.402 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.376+0000 7f067d7fa640 10 client.4207.objecter ms_handle_connect 0x7f066003fd20 2024-06-11T19:09:24.402 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.376+0000 7f06849e9640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f066003fd20 0x7f06600421e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0678009560 tx=0x7f0678009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:24.403 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.381+0000 7f067d7fa640 1 -- 172.21.15.98:0/1933016746 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f068006f460 con 0x7f06800a3240 2024-06-11T19:09:24.403 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.381+0000 7f067d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:24.403 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.381+0000 7f067d7fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:24.564 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.564+0000 7f0686c4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:24.564 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.564+0000 7f0686c4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:24.564 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.564+0000 7f0686c4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:24.564 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.564+0000 7f0686c4d640 1 -- 172.21.15.98:0/1933016746 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f068006cf80 con 0x7f06800a3240 2024-06-11T19:09:24.565 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.565+0000 7f067d7fa640 1 -- 172.21.15.98:0/1933016746 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+7549 (secure 0 0 0) 0x7f067003cce0 con 0x7f06800a3240 2024-06-11T19:09:24.565 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.565+0000 7f067d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:24.565 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.565+0000 7f067d7fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:24.565 INFO:teuthology.misc.health.smithi098.stdout: 2024-06-11T19:09:24.565 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","created":"2024-06-11T19:09:15.016586+0000","modified":"2024-06-11T19:09:23.379750+0000","last_up_change":"2024-06-11T19:09:23.379750+0000","last_in_change":"2024-06-11T19:09:16.662340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ef1075e3-3e1b-4904-b14b-90751df6c38c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6811","nonce":4087730304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6815","nonce":4087730304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6823","nonce":4087730304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6819","nonce":4087730304}]},"public_addr":"172.21.15.98:6811/4087730304","cluster_addr":"172.21.15.98:6815/4087730304","heartbeat_back_addr":"172.21.15.98:6823/4087730304","heartbeat_front_addr":"172.21.15.98:6819/4087730304","state":["exists","up"]},{"osd":1,"uuid":"985893e9-bd60-46fc-b913-7502ff73628d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2b0b4d24-8009-4cec-80e4-c0fa2d705879","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"73793608-9e04-4837-bd2b-1b130a4e88d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6809","nonce":2561136926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6811","nonce":2561136926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6815","nonce":2561136926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6813","nonce":2561136926}]},"public_addr":"172.21.15.117:6809/2561136926","cluster_addr":"172.21.15.117:6811/2561136926","heartbeat_back_addr":"172.21.15.117:6815/2561136926","heartbeat_front_addr":"172.21.15.117:6813/2561136926","state":["exists","up"]},{"osd":4,"uuid":"60e657ae-e2a2-471e-b998-fba98d9c94e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6801","nonce":3877218228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6803","nonce":3877218228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6807","nonce":3877218228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6805","nonce":3877218228}]},"public_addr":"172.21.15.117:6801/3877218228","cluster_addr":"172.21.15.117:6803/3877218228","heartbeat_back_addr":"172.21.15.117:6807/3877218228","heartbeat_front_addr":"172.21.15.117:6805/3877218228","state":["exists","up"]},{"osd":5,"uuid":"dae66a65-2ca2-4f8a-98e3-52b190abee7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6817","nonce":987023138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6819","nonce":987023138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6823","nonce":987023138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6821","nonce":987023138}]},"public_addr":"172.21.15.117:6817/987023138","cluster_addr":"172.21.15.117:6819/987023138","heartbeat_back_addr":"172.21.15.117:6823/987023138","heartbeat_front_addr":"172.21.15.117:6821/987023138","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-11T19:09:24.567 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.566+0000 7f0686c4d640 10 librados: watch_flush enter 2024-06-11T19:09:24.567 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.566+0000 7f0686c4d640 10 librados: watch_flush exit 2024-06-11T19:09:24.567 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 20 client.4207.objecter shutdown clearing up homeless session... 2024-06-11T19:09:24.567 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 10 client.4207.objecter successfully canceled tick 2024-06-11T19:09:24.568 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 -- 172.21.15.98:0/1933016746 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f066003fd20 msgr2=0x7f06600421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:24.568 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f066003fd20 0x7f06600421e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0678009560 tx=0x7f0678009190 comp rx=0 tx=0).stop 2024-06-11T19:09:24.568 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 10 monclient: shutdown 2024-06-11T19:09:24.568 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:24.568 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 -- 172.21.15.98:0/1933016746 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06800a3240 msgr2=0x7f06801ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:24.568 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06800a3240 0x7f06801ce230 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f06700097d0 tx=0x7f06700046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.568 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 -- 172.21.15.98:0/1933016746 shutdown_connections 2024-06-11T19:09:24.568 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f066003fd20 0x7f06600421e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.568 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06800a3240 0x7f06801ce230 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.568 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0680075ef0 0x7f06801cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.568 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 --2- 172.21.15.98:0/1933016746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0680072f70 0x7f06801cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:24.568 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 -- 172.21.15.98:0/1933016746 >> 172.21.15.98:0/1933016746 conn(0x7f06800a7e60 msgr2=0x7f06800a5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:24.569 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 -- 172.21.15.98:0/1933016746 shutdown_connections 2024-06-11T19:09:24.569 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 -- 172.21.15.98:0/1933016746 wait complete. 2024-06-11T19:09:24.569 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:24.567+0000 7f0686c4d640 1 librados: shutdown 2024-06-11T19:09:24.576 DEBUG:teuthology.misc:4 of 6 OSDs are up 2024-06-11T19:09:31.181 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-11T19:09:31.282 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.282+0000 7f732047a640 10 monclient: get_monmap_and_config 2024-06-11T19:09:31.282 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.282+0000 7f732047a640 10 monclient: build_initial_monmap 2024-06-11T19:09:31.282 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.282+0000 7f732047a640 10 monclient: monmap: 2024-06-11T19:09:31.282 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-06-11T19:09:31.282 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:31.282 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:09:31.283470+0000 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:09:31.283470+0000 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:31.283 INFO:teuthology.misc.health.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:31.284 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:31.292 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 1 Processor -- start 2024-06-11T19:09:31.292 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 1 -- start start 2024-06-11T19:09:31.292 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 10 monclient: init 2024-06-11T19:09:31.292 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:31.292 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 10 monclient: _add_conns ranks=[10,9,0,3,1,4,6,2,7,8,5] 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7318139020 0x7f731809ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 10 monclient(hunting): picked mon.noname-f con 0x7f7318139020 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7318139400 0x7f73180a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 10 monclient(hunting): picked mon.noname-a con 0x7f7318139400 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73180a6350 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f731809d6d0 con 0x7f73180a6350 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f731809d1a0 con 0x7f7318139400 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f731809d3f0 con 0x7f7318139020 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f732047a640 10 monclient(hunting): authenticate will time out at 781.090841s 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.283+0000 7f731da15640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7318139400 0x7f73180a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:31.293 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731ea17640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f73180a6350 0x7f73180a6780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43276/0 (socket says 172.21.15.98:43276) 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7318139400 0x7f73180a5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39300/0 (socket says 172.21.15.98:39300) 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731ea17640 1 -- 172.21.15.98:0/1906243349 learned_addr learned my addr 172.21.15.98:0/1906243349 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 10 monclient(hunting): get_auth_request con 0x7f7318139400 auth_method 0 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731e216640 1 --2- 172.21.15.98:0/1906243349 >> v2:172.21.15.117:3300/0 conn(0x7f7318139020 0x7f731809ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731e216640 10 monclient(hunting): get_auth_request con 0x7f7318139020 auth_method 0 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731e216640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:31.294 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731e216640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:31.295 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731e216640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:31.295 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 10 monclient(hunting): handle_auth_done global_id 4248 payload 274 2024-06-11T19:09:31.295 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 1 -- 172.21.15.98:0/1906243349 >> v2:172.21.15.117:3300/0 conn(0x7f7318139020 msgr2=0x7f731809ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:31.295 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 1 --2- 172.21.15.98:0/1906243349 >> v2:172.21.15.117:3300/0 conn(0x7f7318139020 0x7f731809ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.295 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 1 -- 172.21.15.98:0/1906243349 >> v1:172.21.15.98:6789/0 conn(0x7f73180a6350 legacy=0x7f73180a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:31.295 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 10 monclient: _finish_hunting 0 2024-06-11T19:09:31.295 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 1 monclient: found mon.noname-a 2024-06-11T19:09:31.295 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:31.295 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:31.295 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.284+0000 7f731da15640 1 -- 172.21.15.98:0/1906243349 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f731809e310 con 0x7f7318139400 2024-06-11T19:09:31.295 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731da15640 1 --2- 172.21.15.98:0/1906243349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7318139400 0x7f73180a5e10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f73180a8fd0 tx=0x7f731402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b97b746fcb9e5f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:31.295 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 1 -- 172.21.15.98:0/1906243349 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7314038070 con 0x7f7318139400 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 10 monclient: dump: 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:31.296 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:31.297 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:31.297 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:31.297 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:31.297 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:31.297 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:31.297 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:31.297 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 10 monclient: _finish_auth 0 2024-06-11T19:09:31.297 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 10 monclient: _check_auth_tickets 2024-06-11T19:09:31.297 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:31.297 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 1 -- 172.21.15.98:0/1906243349 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7314031970 con 0x7f7318139400 2024-06-11T19:09:31.297 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 5 monclient: authenticate success, global_id 4248 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 1 -- 172.21.15.98:0/1906243349 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7314031ba0 con 0x7f7318139400 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 4 set_mon_vals no callback set 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f731d214640 10 monclient: dump: 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.298 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 10 monclient: shutdown 2024-06-11T19:09:31.299 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 1 -- 172.21.15.98:0/1906243349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7318139400 msgr2=0x7f73180a5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 1 --2- 172.21.15.98:0/1906243349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7318139400 0x7f73180a5e10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f73180a8fd0 tx=0x7f731402b7a0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 1 -- 172.21.15.98:0/1906243349 shutdown_connections 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 1 --2- 172.21.15.98:0/1906243349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7318139400 0x7f73180a5e10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 1 --2- 172.21.15.98:0/1906243349 >> v2:172.21.15.117:3300/0 conn(0x7f7318139020 0x7f731809ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 1 -- 172.21.15.98:0/1906243349 >> 172.21.15.98:0/1906243349 conn(0x7f73180abd50 msgr2=0x7f73180a7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 1 -- 172.21.15.98:0/1906243349 shutdown_connections 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.285+0000 7f732047a640 1 -- 172.21.15.98:0/1906243349 wait complete. 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.286+0000 7f732047a640 10 monclient: build_initial_monmap 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.286+0000 7f732047a640 10 monclient: monmap: 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:31.300 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-06-11T19:09:31.301 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:31.301 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:31.301 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:31.301 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:31.301 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:31.301 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:31.301 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:31.301 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:31.301 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:31.301 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:31.301 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:31.301 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.286+0000 7f732047a640 1 librados: starting msgr at 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.286+0000 7f732047a640 1 librados: starting objecter 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.286+0000 7f732047a640 1 Processor -- start 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.286+0000 7f732047a640 1 -- start start 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.286+0000 7f732047a640 1 librados: setting wanted keys 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.286+0000 7f732047a640 1 librados: calling monclient init 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.286+0000 7f732047a640 10 monclient: init 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 10 monclient: _add_conns ranks=[4,7,8,5,6,2,0,1,3] 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f73180a6350 0x7f73181cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 10 monclient(hunting): picked mon.noname-c con 0x7f73180a6350 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7318139020 0x7f73181ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 10 monclient(hunting): picked mon.noname-i con 0x7f7318139020 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:31.302 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7318139400 0x7f7318123100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 10 monclient(hunting): picked mon.noname-e con 0x7f7318139400 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f73180a88f0 con 0x7f73180a6350 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f731809d9b0 con 0x7f7318139020 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f7318139810 con 0x7f7318139400 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f732047a640 10 monclient(hunting): authenticate will time out at 781.094316s 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731e216640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f73180a6350 0x7f73181cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7318139400 0x7f7318123100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7318139400 0x7f7318123100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:47684/0 (socket says 172.21.15.98:47684) 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731e216640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f73180a6350 0x7f73181cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:37334/0 (socket says 172.21.15.98:37334) 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 1 -- 172.21.15.98:0/793822140 learned_addr learned my addr 172.21.15.98:0/793822140 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:31.303 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 10 monclient(hunting): get_auth_request con 0x7f7318139400 auth_method 0 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731e216640 10 monclient(hunting): get_auth_request con 0x7f73180a6350 auth_method 0 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731e216640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731e216640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731e216640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731e216640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731e216640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731e216640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:31.304 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731da15640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7318139020 0x7f73181ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731da15640 10 monclient(hunting): get_auth_request con 0x7f7318139020 auth_method 0 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731da15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731da15640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731da15640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 1 -- 172.21.15.98:0/793822140 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7318139020 msgr2=0x7f73181ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7318139020 0x7f73181ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 1 -- 172.21.15.98:0/793822140 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f73180a6350 msgr2=0x7f73181cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f73180a6350 0x7f73181cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 10 monclient: _finish_hunting 0 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 1 monclient: found mon.noname-e 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:31.305 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 1 -- 172.21.15.98:0/793822140 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73181270c0 con 0x7f7318139400 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731e216640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f73180a6350 0x7f73181cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731da15640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7318139020 0x7f73181ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f731ea17640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7318139400 0x7f7318123100 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f731000cb10 tx=0x7f731000a060 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f730effd640 10 client.?.objecter ms_handle_connect 0x7f7318139400 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.287+0000 7f730effd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 1 -- 172.21.15.98:0/793822140 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7310017070 con 0x7f7318139400 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 10 monclient: dump: 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.306 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 10 monclient: _finish_auth 0 2024-06-11T19:09:31.307 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f732047a640 5 monclient: authenticate success, global_id 4229 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 1 -- 172.21.15.98:0/793822140 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7310007070 con 0x7f7318139400 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f732047a640 10 monclient: _renew_subs 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f732047a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f732047a640 1 -- 172.21.15.98:0/793822140 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73180aa5e0 con 0x7f7318139400 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 1 -- 172.21.15.98:0/793822140 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7310007710 con 0x7f7318139400 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f731ca13640 4 set_mon_vals no callback set 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f732047a640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f730effd640 10 monclient: dump: 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-06-11T19:09:31.308 INFO:teuthology.misc.health.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr: 2024-06-11T19:09:31.309 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f732047a640 10 monclient: _renew_subs 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f732047a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f732047a640 1 -- 172.21.15.98:0/793822140 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f7318124dd0 con 0x7f7318139400 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.288+0000 7f732047a640 1 librados: init done 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f730effd640 1 -- 172.21.15.98:0/793822140 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f73100078b0 con 0x7f7318139400 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f732047a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f732047a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f732047a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f732047a640 1 -- 172.21.15.98:0/793822140 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f731809ddd0 con 0x7f7318139400 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f730effd640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f72fc03fde0 0x7f72fc0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f730effd640 1 -- 172.21.15.98:0/793822140 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 3993+0+0 (secure 0 0 0) 0x7f7310050d40 con 0x7f7318139400 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f730effd640 10 client.4229.objecter ms_dispatch 0x7f7318000b90 osd_map(15..15 src has 1..15) 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f730effd640 3 client.4229.objecter handle_osd_map got epochs [15,15] > 0 2024-06-11T19:09:31.310 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f730effd640 3 client.4229.objecter handle_osd_map decoding full epoch 15 2024-06-11T19:09:31.311 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f730effd640 20 client.4229.objecter dump_active .. 0 homeless 2024-06-11T19:09:31.311 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f731e216640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f72fc03fde0 0x7f72fc0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:31.311 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.289+0000 7f731e216640 10 monclient: get_auth_request con 0x7f72fc03fde0 auth_method 0 2024-06-11T19:09:31.311 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.290+0000 7f730effd640 10 client.4229.objecter ms_handle_connect 0x7f72fc03fde0 2024-06-11T19:09:31.311 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.290+0000 7f731e216640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f72fc03fde0 0x7f72fc0422a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7308006fd0 tx=0x7f7308008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:31.311 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.294+0000 7f730effd640 1 -- 172.21.15.98:0/793822140 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f731809d6d0 con 0x7f7318139400 2024-06-11T19:09:31.311 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.294+0000 7f730effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:31.311 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.294+0000 7f730effd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:31.475 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.475+0000 7f732047a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:31.475 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.475+0000 7f732047a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:31.475 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.475+0000 7f732047a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:31.475 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.475+0000 7f732047a640 1 -- 172.21.15.98:0/793822140 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7318126ea0 con 0x7f7318139400 2024-06-11T19:09:31.476 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.476+0000 7f730effd640 1 -- 172.21.15.98:0/793822140 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+10473 (secure 0 0 0) 0x7f7310007480 con 0x7f7318139400 2024-06-11T19:09:31.476 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.476+0000 7f730effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:31.476 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.476+0000 7f730effd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:31.476 INFO:teuthology.misc.health.smithi098.stdout: 2024-06-11T19:09:31.476 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":15,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","created":"2024-06-11T19:09:15.016586+0000","modified":"2024-06-11T19:09:27.673561+0000","last_up_change":"2024-06-11T19:09:24.656699+0000","last_in_change":"2024-06-11T19:09:16.662340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-11T19:09:25.381456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ef1075e3-3e1b-4904-b14b-90751df6c38c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6811","nonce":4087730304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6815","nonce":4087730304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6823","nonce":4087730304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6819","nonce":4087730304}]},"public_addr":"172.21.15.98:6811/4087730304","cluster_addr":"172.21.15.98:6815/4087730304","heartbeat_back_addr":"172.21.15.98:6823/4087730304","heartbeat_front_addr":"172.21.15.98:6819/4087730304","state":["exists","up"]},{"osd":1,"uuid":"985893e9-bd60-46fc-b913-7502ff73628d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6801","nonce":3886067876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6803","nonce":3886067876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6807","nonce":3886067876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6805","nonce":3886067876}]},"public_addr":"172.21.15.98:6801/3886067876","cluster_addr":"172.21.15.98:6803/3886067876","heartbeat_back_addr":"172.21.15.98:6807/3886067876","heartbeat_front_addr":"172.21.15.98:6805/3886067876","state":["exists","up"]},{"osd":2,"uuid":"2b0b4d24-8009-4cec-80e4-c0fa2d705879","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6810","nonce":3751501255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6814","nonce":3751501255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6822","nonce":3751501255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6818","nonce":3751501255}]},"public_addr":"172.21.15.98:6810/3751501255","cluster_addr":"172.21.15.98:6814/3751501255","heartbeat_back_addr":"172.21.15.98:6822/3751501255","heartbeat_front_addr":"172.21.15.98:6818/3751501255","state":["exists","up"]},{"osd":3,"uuid":"73793608-9e04-4837-bd2b-1b130a4e88d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6809","nonce":2561136926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6811","nonce":2561136926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6815","nonce":2561136926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6813","nonce":2561136926}]},"public_addr":"172.21.15.117:6809/2561136926","cluster_addr":"172.21.15.117:6811/2561136926","heartbeat_back_addr":"172.21.15.117:6815/2561136926","heartbeat_front_addr":"172.21.15.117:6813/2561136926","state":["exists","up"]},{"osd":4,"uuid":"60e657ae-e2a2-471e-b998-fba98d9c94e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6801","nonce":3877218228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6803","nonce":3877218228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6807","nonce":3877218228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6805","nonce":3877218228}]},"public_addr":"172.21.15.117:6801/3877218228","cluster_addr":"172.21.15.117:6803/3877218228","heartbeat_back_addr":"172.21.15.117:6807/3877218228","heartbeat_front_addr":"172.21.15.117:6805/3877218228","state":["exists","up"]},{"osd":5,"uuid":"dae66a65-2ca2-4f8a-98e3-52b190abee7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6817","nonce":987023138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6819","nonce":987023138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6823","nonce":987023138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6821","nonce":987023138}]},"public_addr":"172.21.15.117:6817/987023138","cluster_addr":"172.21.15.117:6819/987023138","heartbeat_back_addr":"172.21.15.117:6823/987023138","heartbeat_front_addr":"172.21.15.117:6821/987023138","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.744223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.800796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.792849+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.854633+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.844451+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.763441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-11T19:09:31.478 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 10 librados: watch_flush enter 2024-06-11T19:09:31.478 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 10 librados: watch_flush exit 2024-06-11T19:09:31.478 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 20 client.4229.objecter shutdown clearing up homeless session... 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 10 client.4229.objecter successfully canceled tick 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 -- 172.21.15.98:0/793822140 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f72fc03fde0 msgr2=0x7f72fc0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f72fc03fde0 0x7f72fc0422a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7308006fd0 tx=0x7f7308008040 comp rx=0 tx=0).stop 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 10 monclient: shutdown 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 -- 172.21.15.98:0/793822140 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7318139400 msgr2=0x7f7318123100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7318139400 0x7f7318123100 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f731000cb10 tx=0x7f731000a060 comp rx=0 tx=0).stop 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 -- 172.21.15.98:0/793822140 shutdown_connections 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f72fc03fde0 0x7f72fc0422a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7318139400 0x7f7318123100 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7318139020 0x7f73181ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 --2- 172.21.15.98:0/793822140 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f73180a6350 0x7f73181cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 -- 172.21.15.98:0/793822140 >> 172.21.15.98:0/793822140 conn(0x7f73180abd50 msgr2=0x7f73180a9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:31.479 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 -- 172.21.15.98:0/793822140 shutdown_connections 2024-06-11T19:09:31.480 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 -- 172.21.15.98:0/793822140 wait complete. 2024-06-11T19:09:31.480 INFO:teuthology.misc.health.smithi098.stderr:2024-06-11T19:09:31.478+0000 7f732047a640 1 librados: shutdown 2024-06-11T19:09:31.487 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-06-11T19:09:31.487 INFO:tasks.ceph:Creating RBD pool 2024-06-11T19:09:31.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd pool create rbd 8 2024-06-11T19:09:31.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.585+0000 7f63e5b18640 10 monclient: get_monmap_and_config 2024-06-11T19:09:31.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.585+0000 7f63e5b18640 10 monclient: build_initial_monmap 2024-06-11T19:09:31.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.585+0000 7f63e5b18640 10 monclient: monmap: 2024-06-11T19:09:31.585 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:31.586 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:31.586 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:31.586541+0000 2024-06-11T19:09:31.586 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:31.586541+0000 2024-06-11T19:09:31.586 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:31.586 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:31.586 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:31.586 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:31.586 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:31.586 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:31.586 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:31.586 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:31.587 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:31.587 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:31.587 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:31.587 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:31.587 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:31.587 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:31.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 1 Processor -- start 2024-06-11T19:09:31.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 1 -- start start 2024-06-11T19:09:31.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 10 monclient: init 2024-06-11T19:09:31.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:31.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 10 monclient: _add_conns ranks=[7,0,6,9,5,4,2,10,8,3,1] 2024-06-11T19:09:31.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f63e00a4a70 0x7f63e00a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:31.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 10 monclient(hunting): picked mon.noname-i con 0x7f63e00a4a70 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:31.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63e00a2450 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:31.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a5fd0 0x7f63e0140c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:31.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 10 monclient(hunting): picked mon.noname-d con 0x7f63e00a5fd0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63e0139820 con 0x7f63e00a2450 2024-06-11T19:09:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f63e00a70b0 con 0x7f63e00a5fd0 2024-06-11T19:09:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f63e00a6e00 con 0x7f63e00a4a70 2024-06-11T19:09:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63e5b18640 10 monclient(hunting): authenticate will time out at 781.393811s 2024-06-11T19:09:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63deffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f63e00a2450 0x7f63e00a58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43292/0 (socket says 172.21.15.98:43292) 2024-06-11T19:09:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63dffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a5fd0 0x7f63e0140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63deffd640 1 -- 172.21.15.98:0/3627177729 learned_addr learned my addr 172.21.15.98:0/3627177729 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:09:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63dffff640 10 monclient(hunting): get_auth_request con 0x7f63e00a5fd0 auth_method 0 2024-06-11T19:09:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63dffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.586+0000 7f63dffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63df7fe640 1 --2- 172.21.15.98:0/3627177729 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f63e00a4a70 0x7f63e00a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63df7fe640 10 monclient(hunting): get_auth_request con 0x7f63e00a4a70 auth_method 0 2024-06-11T19:09:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63df7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 10 monclient(hunting): handle_auth_done global_id 4200 payload 274 2024-06-11T19:09:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63de7fc640 1 -- 172.21.15.98:0/3627177729 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1193974142 0 0) 0x7f63e0139820 con 0x7f63e00a2450 2024-06-11T19:09:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 1 -- 172.21.15.98:0/3627177729 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f63e00a4a70 msgr2=0x7f63e00a4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 1 --2- 172.21.15.98:0/3627177729 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f63e00a4a70 0x7f63e00a4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 1 -- 172.21.15.98:0/3627177729 >> v1:172.21.15.98:6789/0 conn(0x7f63e00a2450 legacy=0x7f63e00a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 10 monclient: _finish_hunting 0 2024-06-11T19:09:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 1 monclient: found mon.noname-d 2024-06-11T19:09:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:09:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 1 -- 172.21.15.98:0/3627177729 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e00a5390 con 0x7f63e00a5fd0 2024-06-11T19:09:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63de7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:09:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.587+0000 7f63dffff640 1 --2- 172.21.15.98:0/3627177729 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a5fd0 0x7f63e0140c40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009dc0 tx=0x7f63d802bd50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=f2d8048bbcc8bfb4 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 1 -- 172.21.15.98:0/3627177729 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63d8039070 con 0x7f63e00a5fd0 2024-06-11T19:09:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:09:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 10 monclient: dump: 2024-06-11T19:09:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:31.602 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:31.602 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:31.602 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:31.602 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:31.602 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:31.602 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:31.602 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:31.602 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:31.602 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:31.602 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:31.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 5 monclient: authenticate success, global_id 4200 2024-06-11T19:09:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 1 -- 172.21.15.98:0/3627177729 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63d802cd70 con 0x7f63e00a5fd0 2024-06-11T19:09:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 4 set_mon_vals no callback set 2024-06-11T19:09:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 1 -- 172.21.15.98:0/3627177729 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63d80327b0 con 0x7f63e00a5fd0 2024-06-11T19:09:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:09:31.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63de7fc640 10 monclient: dump: 2024-06-11T19:09:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:31.605 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:31.605 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:31.605 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:31.605 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:31.605 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:31.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 10 monclient: shutdown 2024-06-11T19:09:31.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:31.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 1 -- 172.21.15.98:0/3627177729 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a5fd0 msgr2=0x7f63e0140c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:31.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 1 --2- 172.21.15.98:0/3627177729 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a5fd0 0x7f63e0140c40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009dc0 tx=0x7f63d802bd50 comp rx=0 tx=0).stop 2024-06-11T19:09:31.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 1 -- 172.21.15.98:0/3627177729 shutdown_connections 2024-06-11T19:09:31.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 1 --2- 172.21.15.98:0/3627177729 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a5fd0 0x7f63e0140c40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 1 --2- 172.21.15.98:0/3627177729 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f63e00a4a70 0x7f63e00a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 1 -- 172.21.15.98:0/3627177729 >> 172.21.15.98:0/3627177729 conn(0x7f63e00ab830 msgr2=0x7f63e00a79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:31.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 1 -- 172.21.15.98:0/3627177729 shutdown_connections 2024-06-11T19:09:31.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.588+0000 7f63e5b18640 1 -- 172.21.15.98:0/3627177729 wait complete. 2024-06-11T19:09:31.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.589+0000 7f63e5b18640 10 monclient: build_initial_monmap 2024-06-11T19:09:31.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.589+0000 7f63e5b18640 10 monclient: monmap: 2024-06-11T19:09:31.606 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:31.606 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:31.606 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:31.606 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:31.606 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:31.606 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:31.607 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.589+0000 7f63e5b18640 1 librados: starting msgr at 2024-06-11T19:09:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.589+0000 7f63e5b18640 1 librados: starting objecter 2024-06-11T19:09:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.589+0000 7f63e5b18640 1 Processor -- start 2024-06-11T19:09:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.589+0000 7f63e5b18640 1 -- start start 2024-06-11T19:09:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.589+0000 7f63e5b18640 1 librados: setting wanted keys 2024-06-11T19:09:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.589+0000 7f63e5b18640 1 librados: calling monclient init 2024-06-11T19:09:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.589+0000 7f63e5b18640 10 monclient: init 2024-06-11T19:09:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 10 monclient: _add_conns ranks=[2,6,4,8,5,3,1,0,7] 2024-06-11T19:09:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e00a2450 0x7f63e0136eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 10 monclient(hunting): picked mon.noname-b con 0x7f63e00a2450 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a4a70 0x7f63e01373f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 10 monclient(hunting): picked mon.noname-d con 0x7f63e00a4a70 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63e00a5fd0 0x7f63e0122fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 10 monclient(hunting): picked mon.noname-c con 0x7f63e00a5fd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f63e00a2d50 con 0x7f63e00a2450 2024-06-11T19:09:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f63e00a2a70 con 0x7f63e00a5fd0 2024-06-11T19:09:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f63e00a8340 con 0x7f63e00a4a70 2024-06-11T19:09:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63e5b18640 10 monclient(hunting): authenticate will time out at 781.397346s 2024-06-11T19:09:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63deffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a4a70 0x7f63e01373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e00a2450 0x7f63e0136eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:31.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e00a2450 0x7f63e0136eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:53726/0 (socket says 172.21.15.98:53726) 2024-06-11T19:09:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63deffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a4a70 0x7f63e01373f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46750/0 (socket says 172.21.15.98:46750) 2024-06-11T19:09:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 1 -- 172.21.15.98:0/2294550364 learned_addr learned my addr 172.21.15.98:0/2294550364 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 10 monclient(hunting): get_auth_request con 0x7f63e00a2450 auth_method 0 2024-06-11T19:09:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63dffff640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63e00a5fd0 0x7f63e0122fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63dffff640 10 monclient(hunting): get_auth_request con 0x7f63e00a5fd0 auth_method 0 2024-06-11T19:09:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63dffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:31.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63dffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63deffd640 10 monclient(hunting): get_auth_request con 0x7f63e00a4a70 auth_method 0 2024-06-11T19:09:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63deffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63deffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:31.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-06-11T19:09:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 1 -- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a4a70 msgr2=0x7f63e01373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a4a70 0x7f63e01373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 1 -- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63e00a5fd0 msgr2=0x7f63e0122fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63e00a5fd0 0x7f63e0122fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:09:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 1 monclient: found mon.noname-b 2024-06-11T19:09:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 1 -- 172.21.15.98:0/2294550364 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e0126ee0 con 0x7f63e00a2450 2024-06-11T19:09:31.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63dffff640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63e00a5fd0 0x7f63e0122fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:31.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63deffd640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a4a70 0x7f63e01373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:31.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63df7fe640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e00a2450 0x7f63e0136eb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f63d000d800 tx=0x7f63d000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:31.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63c7fff640 10 client.?.objecter ms_handle_connect 0x7f63e00a2450 2024-06-11T19:09:31.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.590+0000 7f63c7fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:31.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 1 -- 172.21.15.98:0/2294550364 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63d000bb30 con 0x7f63e00a2450 2024-06-11T19:09:31.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:31.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:31.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 10 monclient: dump: 2024-06-11T19:09:31.613 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:31.613 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:31.613 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.614 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.614 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:31.614 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:31.614 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:31.614 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:31.614 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:31.614 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:31.614 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:31.614 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:31.614 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:31.614 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:31.615 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:31.615 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:31.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 10 monclient: _finish_auth 0 2024-06-11T19:09:31.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:31.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:31.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 1 -- 172.21.15.98:0/2294550364 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63d0019070 con 0x7f63e00a2450 2024-06-11T19:09:31.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63e5b18640 5 monclient: authenticate success, global_id 4232 2024-06-11T19:09:31.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63e5b18640 10 monclient: _renew_subs 2024-06-11T19:09:31.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63e5b18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:31.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63e5b18640 1 -- 172.21.15.98:0/2294550364 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63e0126c00 con 0x7f63e00a2450 2024-06-11T19:09:31.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 1 -- 172.21.15.98:0/2294550364 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63d000bd10 con 0x7f63e00a2450 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63e5b18640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63c7fff640 10 monclient: dump: 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:31.616 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:31.617 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:31.617 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:31.617 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:31.617 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:31.617 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:31.617 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:31.617 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:31.617 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:31.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63e5b18640 10 monclient: _renew_subs 2024-06-11T19:09:31.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63ddffb640 4 set_mon_vals no callback set 2024-06-11T19:09:31.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63e5b18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:31.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63e5b18640 1 -- 172.21.15.98:0/2294550364 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f63e0126330 con 0x7f63e00a2450 2024-06-11T19:09:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.591+0000 7f63e5b18640 1 librados: init done 2024-06-11T19:09:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63c7fff640 1 -- 172.21.15.98:0/2294550364 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f63d000b6c0 con 0x7f63e00a2450 2024-06-11T19:09:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63e5b18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63e5b18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63e5b18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63e5b18640 1 -- 172.21.15.98:0/2294550364 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63e0138e10 con 0x7f63e00a2450 2024-06-11T19:09:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63c7fff640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f63cc03fdb0 0x7f63cc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63c7fff640 1 -- 172.21.15.98:0/2294550364 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3993+0+0 (secure 0 0 0) 0x7f63d000f040 con 0x7f63e00a2450 2024-06-11T19:09:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63c7fff640 10 client.4232.objecter ms_dispatch 0x7f63e0000b90 osd_map(15..15 src has 1..15) 2024-06-11T19:09:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63c7fff640 3 client.4232.objecter handle_osd_map got epochs [15,15] > 0 2024-06-11T19:09:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63c7fff640 3 client.4232.objecter handle_osd_map decoding full epoch 15 2024-06-11T19:09:31.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63c7fff640 20 client.4232.objecter dump_active .. 0 homeless 2024-06-11T19:09:31.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63deffd640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f63cc03fdb0 0x7f63cc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:31.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.592+0000 7f63deffd640 10 monclient: get_auth_request con 0x7f63cc03fdb0 auth_method 0 2024-06-11T19:09:31.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.593+0000 7f63c7fff640 10 client.4232.objecter ms_handle_connect 0x7f63cc03fdb0 2024-06-11T19:09:31.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.593+0000 7f63deffd640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f63cc03fdb0 0x7f63cc042270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f63d4007f30 tx=0x7f63d4007ec0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:31.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.597+0000 7f63c7fff640 1 -- 172.21.15.98:0/2294550364 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f63d0022020 con 0x7f63e00a2450 2024-06-11T19:09:31.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.597+0000 7f63c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:31.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.597+0000 7f63c7fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:31.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.781+0000 7f63e5b18640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-06-11T19:09:31.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.782+0000 7f63e5b18640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-06-11T19:09:31.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.782+0000 7f63e5b18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:31.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:31.782+0000 7f63e5b18640 1 -- 172.21.15.98:0/2294550364 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f63e006c300 con 0x7f63e00a2450 2024-06-11T19:09:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.702+0000 7f63c7fff640 1 -- 172.21.15.98:0/2294550364 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (secure 0 0 0) 0x7f63d0014d70 con 0x7f63e00a2450 2024-06-11T19:09:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.702+0000 7f63c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-06-11T19:09:32.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.702+0000 7f63c7fff640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-06-11T19:09:32.703 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'rbd' created 2024-06-11T19:09:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.705+0000 7f63e5b18640 10 librados: watch_flush enter 2024-06-11T19:09:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.706+0000 7f63e5b18640 10 librados: watch_flush exit 2024-06-11T19:09:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.706+0000 7f63e5b18640 20 client.4232.objecter shutdown clearing up homeless session... 2024-06-11T19:09:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.706+0000 7f63e5b18640 10 client.4232.objecter successfully canceled tick 2024-06-11T19:09:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.706+0000 7f63e5b18640 1 -- 172.21.15.98:0/2294550364 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f63cc03fdb0 msgr2=0x7f63cc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.706+0000 7f63e5b18640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f63cc03fdb0 0x7f63cc042270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f63d4007f30 tx=0x7f63d4007ec0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.706+0000 7f63e5b18640 10 monclient: shutdown 2024-06-11T19:09:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.706+0000 7f63e5b18640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.706+0000 7f63e5b18640 1 -- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e00a2450 msgr2=0x7f63e0136eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.706+0000 7f63e5b18640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e00a2450 0x7f63e0136eb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f63d000d800 tx=0x7f63d000dcc0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.707+0000 7f63e5b18640 1 -- 172.21.15.98:0/2294550364 shutdown_connections 2024-06-11T19:09:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.707+0000 7f63e5b18640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f63cc03fdb0 0x7f63cc042270 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f63d4007f30 tx=0x7f63d4007ec0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.707+0000 7f63e5b18640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63e00a5fd0 0x7f63e0122fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.707+0000 7f63e5b18640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f63e00a4a70 0x7f63e01373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.707+0000 7f63e5b18640 1 --2- 172.21.15.98:0/2294550364 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e00a2450 0x7f63e0136eb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.707+0000 7f63e5b18640 1 -- 172.21.15.98:0/2294550364 >> 172.21.15.98:0/2294550364 conn(0x7f63e00ab830 msgr2=0x7f63e01bf250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.707+0000 7f63e5b18640 1 -- 172.21.15.98:0/2294550364 shutdown_connections 2024-06-11T19:09:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.707+0000 7f63e5b18640 1 -- 172.21.15.98:0/2294550364 wait complete. 2024-06-11T19:09:32.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.708+0000 7f63e5b18640 1 librados: shutdown 2024-06-11T19:09:32.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-06-11T19:09:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.822+0000 7f7c9d254640 10 monclient: get_monmap_and_config 2024-06-11T19:09:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.822+0000 7f7c9d254640 10 monclient: build_initial_monmap 2024-06-11T19:09:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.822+0000 7f7c9d254640 10 monclient: monmap: 2024-06-11T19:09:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:32.823307+0000 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:32.823307+0000 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:32.824 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:32.824 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:32.824 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:32.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.822+0000 7f7c9d254640 1 Processor -- start 2024-06-11T19:09:32.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.822+0000 7f7c9d254640 1 -- start start 2024-06-11T19:09:32.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.822+0000 7f7c9d254640 10 monclient: init 2024-06-11T19:09:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 10 monclient: _add_conns ranks=[7,8,5,3,10,2,4,9,6,1,0] 2024-06-11T19:09:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7c98135b60 0x7f7c98135f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 10 monclient(hunting): picked mon.noname-i con 0x7f7c98135b60 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7c980a5a10 0x7f7c980a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 10 monclient(hunting): picked mon.noname-e con 0x7f7c980a5a10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7c980a6350 0x7f7c9813fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 10 monclient(hunting): picked mon.noname-h con 0x7f7c980a6350 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f7c980a8580 con 0x7f7c980a6350 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f7c980a2c60 con 0x7f7c98135b60 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f7c980a2f40 con 0x7f7c980a5a10 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9d254640 10 monclient(hunting): authenticate will time out at 782.630394s 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7c980a5a10 0x7f7c980a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7c980a5a10 0x7f7c980a5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:47692/0 (socket says 172.21.15.98:47692) 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 1 -- 172.21.15.98:0/2995888386 learned_addr learned my addr 172.21.15.98:0/2995888386 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 10 monclient(hunting): get_auth_request con 0x7f7c980a5a10 auth_method 0 2024-06-11T19:09:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9759e640 1 --2- 172.21.15.98:0/2995888386 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7c980a6350 0x7f7c9813fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c96d9d640 1 --2- 172.21.15.98:0/2995888386 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7c98135b60 0x7f7c98135f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9759e640 10 monclient(hunting): get_auth_request con 0x7f7c980a6350 auth_method 0 2024-06-11T19:09:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9759e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9759e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c96d9d640 10 monclient(hunting): get_auth_request con 0x7f7c98135b60 auth_method 0 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c96d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 1 -- 172.21.15.98:0/2995888386 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7c98135b60 msgr2=0x7f7c98135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 1 --2- 172.21.15.98:0/2995888386 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7c98135b60 0x7f7c98135f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 1 -- 172.21.15.98:0/2995888386 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7c980a6350 msgr2=0x7f7c9813fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 1 --2- 172.21.15.98:0/2995888386 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7c980a6350 0x7f7c9813fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 10 monclient: _finish_hunting 0 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 1 monclient: found mon.noname-e 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.823+0000 7f7c9659c640 1 -- 172.21.15.98:0/2995888386 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c980a3220 con 0x7f7c980a5a10 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c9659c640 1 --2- 172.21.15.98:0/2995888386 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7c980a5a10 0x7f7c980a5e10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7c800098a0 tx=0x7f7c8002b7a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=c8111841945de524 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 1 -- 172.21.15.98:0/2995888386 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c80038070 con 0x7f7c980a5a10 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 10 monclient: dump: 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:32.837 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 10 monclient: _finish_auth 0 2024-06-11T19:09:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:09:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 1 -- 172.21.15.98:0/2995888386 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c80031970 con 0x7f7c980a5a10 2024-06-11T19:09:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c9d254640 5 monclient: authenticate success, global_id 4238 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c9d254640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c9d254640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 1 -- 172.21.15.98:0/2995888386 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c80031ba0 con 0x7f7c980a5a10 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c9d254640 4 set_mon_vals no callback set 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.824+0000 7f7c95d9b640 10 monclient: dump: 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 10 monclient: shutdown 2024-06-11T19:09:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 1 -- 172.21.15.98:0/2995888386 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7c980a5a10 msgr2=0x7f7c980a5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 1 --2- 172.21.15.98:0/2995888386 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7c980a5a10 0x7f7c980a5e10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7c800098a0 tx=0x7f7c8002b7a0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 1 -- 172.21.15.98:0/2995888386 shutdown_connections 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 1 --2- 172.21.15.98:0/2995888386 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7c980a6350 0x7f7c9813fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 1 --2- 172.21.15.98:0/2995888386 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7c980a5a10 0x7f7c980a5e10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 1 --2- 172.21.15.98:0/2995888386 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7c98135b60 0x7f7c98135f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 1 -- 172.21.15.98:0/2995888386 >> 172.21.15.98:0/2995888386 conn(0x7f7c980aba20 msgr2=0x7f7c980a7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 1 -- 172.21.15.98:0/2995888386 shutdown_connections 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 1 -- 172.21.15.98:0/2995888386 wait complete. 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 10 monclient: build_initial_monmap 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.825+0000 7f7c9d254640 10 monclient: monmap: 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:32.841 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.826+0000 7f7c9d254640 1 librados: starting msgr at 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.826+0000 7f7c9d254640 1 librados: starting objecter 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.826+0000 7f7c9d254640 1 Processor -- start 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.826+0000 7f7c9d254640 1 -- start start 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.826+0000 7f7c9d254640 1 librados: setting wanted keys 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.826+0000 7f7c9d254640 1 librados: calling monclient init 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.826+0000 7f7c9d254640 10 monclient: init 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.826+0000 7f7c9d254640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.826+0000 7f7c9d254640 10 monclient: _add_conns ranks=[7,3,4,0,1,8,2,5,6] 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.826+0000 7f7c9d254640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7c980a5a10 0x7f7c981cd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.826+0000 7f7c9d254640 10 monclient(hunting): picked mon.noname-i con 0x7f7c980a5a10 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c980a6350 0x7f7c98134dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 10 monclient(hunting): picked mon.noname-g con 0x7f7c980a6350 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7c98135b60 0x7f7c981bbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 10 monclient(hunting): picked mon.noname-c con 0x7f7c98135b60 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f7c980a6f50 con 0x7f7c980a6350 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7c98050490 con 0x7f7c98135b60 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f7c980a7290 con 0x7f7c980a5a10 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 10 monclient(hunting): authenticate will time out at 782.634064s 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c96d9d640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7c980a5a10 0x7f7c981cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7c98135b60 0x7f7c981bbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c96d9d640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7c980a5a10 0x7f7c981cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:45886/0 (socket says 172.21.15.98:45886) 2024-06-11T19:09:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c96d9d640 1 -- 172.21.15.98:0/3019621333 learned_addr learned my addr 172.21.15.98:0/3019621333 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 10 monclient(hunting): get_auth_request con 0x7f7c98135b60 auth_method 0 2024-06-11T19:09:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9659c640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c980a6350 0x7f7c98134dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c96d9d640 10 monclient(hunting): get_auth_request con 0x7f7c980a5a10 auth_method 0 2024-06-11T19:09:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c96d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9659c640 10 monclient(hunting): get_auth_request con 0x7f7c980a6350 auth_method 0 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 10 monclient(hunting): handle_auth_done global_id 4252 payload 824 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 1 -- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7c980a5a10 msgr2=0x7f7c981cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7c980a5a10 0x7f7c981cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 1 -- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c980a6350 msgr2=0x7f7c98134dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c980a6350 0x7f7c98134dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 10 monclient: _finish_hunting 0 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 1 monclient: found mon.noname-c 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 1 -- 172.21.15.98:0/3019621333 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c981224a0 con 0x7f7c98135b60 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9759e640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7c98135b60 0x7f7c981bbde0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7c90007d90 tx=0x7f7c9000e4b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 10 client.?.objecter ms_handle_connect 0x7f7c98135b60 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 1 -- 172.21.15.98:0/3019621333 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c9000edd0 con 0x7f7c98135b60 2024-06-11T19:09:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 10 monclient: dump: 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 1 -- 172.21.15.98:0/3019621333 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c90014890 con 0x7f7c98135b60 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 5 monclient: authenticate success, global_id 4252 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 10 monclient: _renew_subs 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 1 -- 172.21.15.98:0/3019621333 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c981cdd80 con 0x7f7c98135b60 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 1 -- 172.21.15.98:0/3019621333 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c90013380 con 0x7f7c98135b60 2024-06-11T19:09:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c8f7fe640 10 monclient: dump: 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9559a640 4 set_mon_vals no callback set 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 10 monclient: _renew_subs 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 1 -- 172.21.15.98:0/3019621333 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c980a0f70 con 0x7f7c98135b60 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.827+0000 7f7c9d254640 1 librados: init done 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.828+0000 7f7c8f7fe640 1 -- 172.21.15.98:0/3019621333 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f7c9001b030 con 0x7f7c98135b60 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.828+0000 7f7c9d254640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.828+0000 7f7c9d254640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.828+0000 7f7c9d254640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.828+0000 7f7c9d254640 1 -- 172.21.15.98:0/3019621333 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c980a4620 con 0x7f7c98135b60 2024-06-11T19:09:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.828+0000 7f7c8f7fe640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7c7803fdb0 0x7f7c78042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.829+0000 7f7c8f7fe640 1 -- 172.21.15.98:0/3019621333 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(16..16 src has 1..16) ==== 4335+0+0 (secure 0 0 0) 0x7f7c90019020 con 0x7f7c98135b60 2024-06-11T19:09:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.829+0000 7f7c8f7fe640 10 client.4252.objecter ms_dispatch 0x7f7c98000b90 osd_map(16..16 src has 1..16) 2024-06-11T19:09:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.829+0000 7f7c8f7fe640 3 client.4252.objecter handle_osd_map got epochs [16,16] > 0 2024-06-11T19:09:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.829+0000 7f7c8f7fe640 3 client.4252.objecter handle_osd_map decoding full epoch 16 2024-06-11T19:09:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.829+0000 7f7c96d9d640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7c7803fdb0 0x7f7c78042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.829+0000 7f7c8f7fe640 20 client.4252.objecter dump_active .. 0 homeless 2024-06-11T19:09:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.829+0000 7f7c96d9d640 10 monclient: get_auth_request con 0x7f7c7803fdb0 auth_method 0 2024-06-11T19:09:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.829+0000 7f7c8f7fe640 10 client.4252.objecter ms_handle_connect 0x7f7c7803fdb0 2024-06-11T19:09:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.829+0000 7f7c96d9d640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7c7803fdb0 0x7f7c78042270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c880079e0 tx=0x7f7c88008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.833+0000 7f7c8f7fe640 1 -- 172.21.15.98:0/3019621333 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7c90022060 con 0x7f7c98135b60 2024-06-11T19:09:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.833+0000 7f7c8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:32.833+0000 7f7c8f7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.016+0000 7f7c9d254640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-06-11T19:09:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.016+0000 7f7c9d254640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-06-11T19:09:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.016+0000 7f7c9d254640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.016+0000 7f7c9d254640 1 -- 172.21.15.98:0/3019621333 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7f7c980a53a0 con 0x7f7c98135b60 2024-06-11T19:09:33.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.719+0000 7f7c8f7fe640 1 -- 172.21.15.98:0/3019621333 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 173+0+0 (secure 0 0 0) 0x7f7c900510c0 con 0x7f7c98135b60 2024-06-11T19:09:33.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.719+0000 7f7c8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-06-11T19:09:33.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.719+0000 7f7c8f7fe640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-06-11T19:09:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rbd' on pool 'rbd' 2024-06-11T19:09:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 10 librados: watch_flush enter 2024-06-11T19:09:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 10 librados: watch_flush exit 2024-06-11T19:09:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 20 client.4252.objecter shutdown clearing up homeless session... 2024-06-11T19:09:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 10 client.4252.objecter successfully canceled tick 2024-06-11T19:09:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 1 -- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7c7803fdb0 msgr2=0x7f7c78042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7c7803fdb0 0x7f7c78042270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c880079e0 tx=0x7f7c88008040 comp rx=0 tx=0).stop 2024-06-11T19:09:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 10 monclient: shutdown 2024-06-11T19:09:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 1 -- 172.21.15.98:0/3019621333 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7c98135b60 msgr2=0x7f7c981bbde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:33.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7c98135b60 0x7f7c981bbde0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7c90007d90 tx=0x7f7c9000e4b0 comp rx=0 tx=0).stop 2024-06-11T19:09:33.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 1 -- 172.21.15.98:0/3019621333 shutdown_connections 2024-06-11T19:09:33.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7c7803fdb0 0x7f7c78042270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:33.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7c98135b60 0x7f7c981bbde0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:33.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c980a6350 0x7f7c98134dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:33.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 1 --2- 172.21.15.98:0/3019621333 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7c980a5a10 0x7f7c981cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:33.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 1 -- 172.21.15.98:0/3019621333 >> 172.21.15.98:0/3019621333 conn(0x7f7c980aba20 msgr2=0x7f7c981c0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:33.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 1 -- 172.21.15.98:0/3019621333 shutdown_connections 2024-06-11T19:09:33.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.723+0000 7f7c9d254640 1 -- 172.21.15.98:0/3019621333 wait complete. 2024-06-11T19:09:33.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.724+0000 7f7c9d254640 1 librados: shutdown 2024-06-11T19:09:33.737 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-06-11T19:09:33.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-06-11T19:09:33.738 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-06-11T19:09:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.842+0000 7f18e7cb9640 10 monclient: get_monmap_and_config 2024-06-11T19:09:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.842+0000 7f18e7cb9640 10 monclient: build_initial_monmap 2024-06-11T19:09:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.842+0000 7f18e7cb9640 10 monclient: monmap: 2024-06-11T19:09:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:33.843835+0000 2024-06-11T19:09:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:33.843835+0000 2024-06-11T19:09:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:33.844 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:33.844 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:33.844 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:33.844 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:33.844 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:33.844 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:33.844 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:33.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.843+0000 7f18e7cb9640 1 Processor -- start 2024-06-11T19:09:33.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.843+0000 7f18e7cb9640 1 -- start start 2024-06-11T19:09:33.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.843+0000 7f18e7cb9640 10 monclient: init 2024-06-11T19:09:33.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:33.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 10 monclient: _add_conns ranks=[1,10,6,8,4,0,9,5,7,2,3] 2024-06-11T19:09:33.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f18e00a4a70 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:33.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f18e00a2450 0x7f18e013dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 10 monclient(hunting): picked mon.noname-f con 0x7f18e00a2450 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 0x7f18e0140550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 10 monclient(hunting): picked mon.noname-d con 0x7f18e013e160 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18e0139820 con 0x7f18e00a4a70 2024-06-11T19:09:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f18e00a70b0 con 0x7f18e00a2450 2024-06-11T19:09:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f18e00a6e00 con 0x7f18e013e160 2024-06-11T19:09:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e7cb9640 10 monclient(hunting): authenticate will time out at 783.651089s 2024-06-11T19:09:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 0x7f18e0140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 0x7f18e0140550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49674/0 (socket says 172.21.15.98:49674) 2024-06-11T19:09:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 1 -- 172.21.15.98:0/1780205613 learned_addr learned my addr 172.21.15.98:0/1780205613 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 10 monclient(hunting): get_auth_request con 0x7f18e013e160 auth_method 0 2024-06-11T19:09:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 10 monclient(hunting): handle_auth_done global_id 4218 payload 274 2024-06-11T19:09:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 1 -- 172.21.15.98:0/1780205613 >> v2:172.21.15.117:3300/0 conn(0x7f18e00a2450 msgr2=0x7f18e013dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 1 --2- 172.21.15.98:0/1780205613 >> v2:172.21.15.117:3300/0 conn(0x7f18e00a2450 0x7f18e013dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 1 -- 172.21.15.98:0/1780205613 >> v1:172.21.15.117:6789/0 conn(0x7f18e00a4a70 legacy=0x7f18e00a4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 10 monclient: _finish_hunting 0 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 1 monclient: found mon.noname-d 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.844+0000 7f18e6256640 1 -- 172.21.15.98:0/1780205613 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e00a3030 con 0x7f18e013e160 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e6256640 1 --2- 172.21.15.98:0/1780205613 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 0x7f18e0140550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009e50 tx=0x7f18d802bdb0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=6e37af0e5aaebefe server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 1 -- 172.21.15.98:0/1780205613 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18d802cbc0 con 0x7f18e013e160 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 10 monclient: dump: 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 10 monclient: _finish_auth 0 2024-06-11T19:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 10 monclient: _check_auth_tickets 2024-06-11T19:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e7cb9640 5 monclient: authenticate success, global_id 4218 2024-06-11T19:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 1 -- 172.21.15.98:0/1780205613 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18d802cd80 con 0x7f18e013e160 2024-06-11T19:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e7cb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e7cb9640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e7cb9640 4 set_mon_vals no callback set 2024-06-11T19:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 1 -- 172.21.15.98:0/1780205613 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18d8034860 con 0x7f18e013e160 2024-06-11T19:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:09:33.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e4a53640 10 monclient: dump: 2024-06-11T19:09:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e7cb9640 10 monclient: shutdown 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e7cb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e7cb9640 1 -- 172.21.15.98:0/1780205613 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 msgr2=0x7f18e0140550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e7cb9640 1 --2- 172.21.15.98:0/1780205613 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 0x7f18e0140550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009e50 tx=0x7f18d802bdb0 comp rx=0 tx=0).stop 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e7cb9640 1 -- 172.21.15.98:0/1780205613 shutdown_connections 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e7cb9640 1 --2- 172.21.15.98:0/1780205613 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 0x7f18e0140550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e7cb9640 1 --2- 172.21.15.98:0/1780205613 >> v2:172.21.15.117:3300/0 conn(0x7f18e00a2450 0x7f18e013dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.845+0000 7f18e7cb9640 1 -- 172.21.15.98:0/1780205613 >> 172.21.15.98:0/1780205613 conn(0x7f18e00ab830 msgr2=0x7f18e00a79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.846+0000 7f18e7cb9640 1 -- 172.21.15.98:0/1780205613 shutdown_connections 2024-06-11T19:09:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.846+0000 7f18e7cb9640 1 -- 172.21.15.98:0/1780205613 wait complete. 2024-06-11T19:09:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.846+0000 7f18e7cb9640 10 monclient: build_initial_monmap 2024-06-11T19:09:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.846+0000 7f18e7cb9640 10 monclient: monmap: 2024-06-11T19:09:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.846+0000 7f18e7cb9640 1 librados: starting msgr at 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.846+0000 7f18e7cb9640 1 librados: starting objecter 2024-06-11T19:09:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.846+0000 7f18e7cb9640 1 Processor -- start 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.846+0000 7f18e7cb9640 1 -- start start 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.846+0000 7f18e7cb9640 1 librados: setting wanted keys 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.846+0000 7f18e7cb9640 1 librados: calling monclient init 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.846+0000 7f18e7cb9640 10 monclient: init 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 10 monclient: _add_conns ranks=[4,1,6,7,2,8,0,5,3] 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f18e00a2450 0x7f18e0135180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 10 monclient(hunting): picked mon.noname-c con 0x7f18e00a2450 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e00a4a70 0x7f18e009df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 10 monclient(hunting): picked mon.noname-f con 0x7f18e00a4a70 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 0x7f18e009e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 10 monclient(hunting): picked mon.noname-d con 0x7f18e013e160 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f18e00a8340 con 0x7f18e00a4a70 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f18e00a2d50 con 0x7f18e00a2450 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f18e00a84c0 con 0x7f18e013e160 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e7cb9640 10 monclient(hunting): authenticate will time out at 783.654317s 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 0x7f18e009e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5a55640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f18e00a2450 0x7f18e0135180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 0x7f18e009e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49686/0 (socket says 172.21.15.98:49686) 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 1 -- 172.21.15.98:0/586394012 learned_addr learned my addr 172.21.15.98:0/586394012 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5a55640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f18e00a2450 0x7f18e0135180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:40314/0 (socket says 172.21.15.98:40314) 2024-06-11T19:09:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 10 monclient(hunting): get_auth_request con 0x7f18e013e160 auth_method 0 2024-06-11T19:09:33.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:33.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:33.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:33.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5a55640 10 monclient(hunting): get_auth_request con 0x7f18e00a2450 auth_method 0 2024-06-11T19:09:33.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:33.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5a55640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5a55640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5a55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5a55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5a55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5254640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e00a4a70 0x7f18e009df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5254640 10 monclient(hunting): get_auth_request con 0x7f18e00a4a70 auth_method 0 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5254640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5254640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e5254640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 10 monclient(hunting): handle_auth_done global_id 4227 payload 824 2024-06-11T19:09:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 1 -- 172.21.15.98:0/586394012 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f18e00a2450 msgr2=0x7f18e0135180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f18e00a2450 0x7f18e0135180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 1 -- 172.21.15.98:0/586394012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e00a4a70 msgr2=0x7f18e009df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e00a4a70 0x7f18e009df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 10 monclient: _finish_hunting 0 2024-06-11T19:09:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 1 monclient: found mon.noname-d 2024-06-11T19:09:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:09:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 1 -- 172.21.15.98:0/586394012 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e0121890 con 0x7f18e013e160 2024-06-11T19:09:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18e6256640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 0x7f18e009e4c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009f40 tx=0x7f18d8004a40 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18d27fc640 10 client.?.objecter ms_handle_connect 0x7f18e013e160 2024-06-11T19:09:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.847+0000 7f18d27fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 1 -- 172.21.15.98:0/586394012 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18d8043030 con 0x7f18e013e160 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 10 monclient: dump: 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 1 -- 172.21.15.98:0/586394012 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18d80046c0 con 0x7f18e013e160 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18e7cb9640 5 monclient: authenticate success, global_id 4227 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18e7cb9640 10 monclient: _renew_subs 2024-06-11T19:09:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18e7cb9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18e7cb9640 1 -- 172.21.15.98:0/586394012 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18e00a6290 con 0x7f18e013e160 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 1 -- 172.21.15.98:0/586394012 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18d8009510 con 0x7f18e013e160 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 10 monclient: dump: 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18e7cb9640 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d3fff640 4 set_mon_vals no callback set 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18e7cb9640 10 monclient: _renew_subs 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18e7cb9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18e7cb9640 1 -- 172.21.15.98:0/586394012 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e00a1980 con 0x7f18e013e160 2024-06-11T19:09:33.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18e7cb9640 1 librados: init done 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.848+0000 7f18d27fc640 1 -- 172.21.15.98:0/586394012 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f18d8046070 con 0x7f18e013e160 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18e7cb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18e7cb9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18e7cb9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18e7cb9640 1 -- 172.21.15.98:0/586394012 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18e00754b0 con 0x7f18e013e160 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18d27fc640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f18c403fdb0 0x7f18c4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18d27fc640 1 -- 172.21.15.98:0/586394012 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(17..17 src has 1..17) ==== 4346+0+0 (secure 0 0 0) 0x7f18d8044070 con 0x7f18e013e160 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18d27fc640 10 client.4227.objecter ms_dispatch 0x7f18e0000b90 osd_map(17..17 src has 1..17) 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18d27fc640 3 client.4227.objecter handle_osd_map got epochs [17,17] > 0 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18d27fc640 3 client.4227.objecter handle_osd_map decoding full epoch 17 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18d27fc640 20 client.4227.objecter dump_active .. 0 homeless 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18e5a55640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f18c403fdb0 0x7f18c4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:33.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18e5a55640 10 monclient: get_auth_request con 0x7f18c403fdb0 auth_method 0 2024-06-11T19:09:33.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18e5a55640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f18c403fdb0 0x7f18c4042270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0017f0 tx=0x7f18dc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:33.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.849+0000 7f18d27fc640 10 client.4227.objecter ms_handle_connect 0x7f18c403fdb0 2024-06-11T19:09:33.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.854+0000 7f18d27fc640 1 -- 172.21.15.98:0/586394012 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f18e00754b0 con 0x7f18e013e160 2024-06-11T19:09:33.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.854+0000 7f18d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:33.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:33.854+0000 7f18d27fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.041+0000 7f18e7cb9640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-06-11T19:09:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.041+0000 7f18e7cb9640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-06-11T19:09:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.041+0000 7f18e7cb9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:09:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.041+0000 7f18e7cb9640 1 -- 172.21.15.98:0/586394012 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f18e00a1780 con 0x7f18e013e160 2024-06-11T19:09:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.042+0000 7f18d27fc640 1 -- 172.21.15.98:0/586394012 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v3) ==== 86+0+159 (secure 0 0 0) 0x7f18d804d020 con 0x7f18e013e160 2024-06-11T19:09:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.042+0000 7f18d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-06-11T19:09:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.042+0000 7f18d27fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:34.042 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:09:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 10 librados: watch_flush enter 2024-06-11T19:09:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 10 librados: watch_flush exit 2024-06-11T19:09:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 20 client.4227.objecter shutdown clearing up homeless session... 2024-06-11T19:09:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 10 client.4227.objecter successfully canceled tick 2024-06-11T19:09:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 -- 172.21.15.98:0/586394012 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f18c403fdb0 msgr2=0x7f18c4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f18c403fdb0 0x7f18c4042270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0017f0 tx=0x7f18dc009040 comp rx=0 tx=0).stop 2024-06-11T19:09:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 10 monclient: shutdown 2024-06-11T19:09:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 -- 172.21.15.98:0/586394012 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 msgr2=0x7f18e009e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 0x7f18e009e4c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009f40 tx=0x7f18d8004a40 comp rx=0 tx=0).stop 2024-06-11T19:09:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 -- 172.21.15.98:0/586394012 shutdown_connections 2024-06-11T19:09:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f18c403fdb0 0x7f18c4042270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f18e013e160 0x7f18e009e4c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e00a4a70 0x7f18e009df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 --2- 172.21.15.98:0/586394012 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f18e00a2450 0x7f18e0135180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 -- 172.21.15.98:0/586394012 >> 172.21.15.98:0/586394012 conn(0x7f18e00ab830 msgr2=0x7f18e0138190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 -- 172.21.15.98:0/586394012 shutdown_connections 2024-06-11T19:09:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 -- 172.21.15.98:0/586394012 wait complete. 2024-06-11T19:09:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.044+0000 7f18e7cb9640 1 librados: shutdown 2024-06-11T19:09:34.054 INFO:teuthology.orchestra.run.smithi098.stdout:[{"version":3,"timestamp":"2024-06-11T19:09:23.655590+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"6556.849993"}]}] 2024-06-11T19:09:34.054 INFO:tasks.ceph_manager:config epoch is 3 2024-06-11T19:09:34.054 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-11T19:09:34.054 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-06-11T19:09:34.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-06-11T19:09:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.153+0000 7f584f5fe640 10 monclient: get_monmap_and_config 2024-06-11T19:09:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.153+0000 7f584f5fe640 10 monclient: build_initial_monmap 2024-06-11T19:09:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.153+0000 7f584f5fe640 10 monclient: monmap: 2024-06-11T19:09:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:34.154974+0000 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:34.154974+0000 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:34.155 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.154+0000 7f584f5fe640 1 Processor -- start 2024-06-11T19:09:34.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.154+0000 7f584f5fe640 1 -- start start 2024-06-11T19:09:34.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.154+0000 7f584f5fe640 10 monclient: init 2024-06-11T19:09:34.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:34.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 10 monclient: _add_conns ranks=[7,6,2,4,0,3,1,5,10,9,8] 2024-06-11T19:09:34.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58480a2f40 0x7f58480a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 10 monclient(hunting): picked mon.noname-i con 0x7f58480a2f40 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:34.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58480a4df0 0x7f58480a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 10 monclient(hunting): picked mon.noname-d con 0x7f58480a4df0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:34.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58481366d0 0x7f5848136af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 10 monclient(hunting): picked mon.noname-b con 0x7f58481366d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:34.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f58480a5710 con 0x7f58481366d0 2024-06-11T19:09:34.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f58480a3320 con 0x7f58480a4df0 2024-06-11T19:09:34.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f58480a2460 con 0x7f58480a2f40 2024-06-11T19:09:34.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:34.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584f5fe640 10 monclient(hunting): authenticate will time out at 783.962323s 2024-06-11T19:09:34.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58480a4df0 0x7f58480a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584db9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58481366d0 0x7f5848136af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58480a4df0 0x7f58480a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49690/0 (socket says 172.21.15.98:49690) 2024-06-11T19:09:34.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 1 -- 172.21.15.98:0/485912509 learned_addr learned my addr 172.21.15.98:0/485912509 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:34.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584db9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58481366d0 0x7f5848136af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47848/0 (socket says 172.21.15.98:47848) 2024-06-11T19:09:34.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 10 monclient(hunting): get_auth_request con 0x7f58480a4df0 auth_method 0 2024-06-11T19:09:34.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584db9b640 10 monclient(hunting): get_auth_request con 0x7f58481366d0 auth_method 0 2024-06-11T19:09:34.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584db9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584db9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584db9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584db9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:34.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584db9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:34.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584db9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:34.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:34.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:34.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:34.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584d39a640 1 --2- 172.21.15.98:0/485912509 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58480a2f40 0x7f58480a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584d39a640 10 monclient(hunting): get_auth_request con 0x7f58480a2f40 auth_method 0 2024-06-11T19:09:34.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584d39a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584d39a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584d39a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 10 monclient(hunting): handle_auth_done global_id 4236 payload 274 2024-06-11T19:09:34.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 1 -- 172.21.15.98:0/485912509 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58480a2f40 msgr2=0x7f58480a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:34.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 1 --2- 172.21.15.98:0/485912509 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58480a2f40 0x7f58480a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 1 -- 172.21.15.98:0/485912509 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58481366d0 msgr2=0x7f5848136af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:34.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 1 --2- 172.21.15.98:0/485912509 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58481366d0 0x7f5848136af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 10 monclient: _finish_hunting 0 2024-06-11T19:09:34.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 1 monclient: found mon.noname-d 2024-06-11T19:09:34.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:34.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:09:34.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584cb99640 1 -- 172.21.15.98:0/485912509 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58480a0ed0 con 0x7f58480a4df0 2024-06-11T19:09:34.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.155+0000 7f584db9b640 1 --2- 172.21.15.98:0/485912509 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58481366d0 0x7f5848136af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:34.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f584cb99640 1 --2- 172.21.15.98:0/485912509 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58480a4df0 0x7f58480a51d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f58380098a0 tx=0x7f583802b7a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=f50b78843400714b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:34.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 1 -- 172.21.15.98:0/485912509 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5838038070 con 0x7f58480a4df0 2024-06-11T19:09:34.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:34.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:09:34.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 10 monclient: dump: 2024-06-11T19:09:34.171 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:34.171 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.171 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:34.173 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 10 monclient: _finish_auth 0 2024-06-11T19:09:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 1 -- 172.21.15.98:0/485912509 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5838031970 con 0x7f58480a4df0 2024-06-11T19:09:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f584f5fe640 5 monclient: authenticate success, global_id 4236 2024-06-11T19:09:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f584f5fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f584f5fe640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 1 -- 172.21.15.98:0/485912509 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5838032390 con 0x7f58480a4df0 2024-06-11T19:09:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f584f5fe640 4 set_mon_vals no callback set 2024-06-11T19:09:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:09:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.156+0000 7f583ffff640 10 monclient: dump: 2024-06-11T19:09:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:34.176 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 10 monclient: shutdown 2024-06-11T19:09:34.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:34.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 1 -- 172.21.15.98:0/485912509 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58480a4df0 msgr2=0x7f58480a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 1 --2- 172.21.15.98:0/485912509 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58480a4df0 0x7f58480a51d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f58380098a0 tx=0x7f583802b7a0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 1 -- 172.21.15.98:0/485912509 shutdown_connections 2024-06-11T19:09:34.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 1 --2- 172.21.15.98:0/485912509 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58481366d0 0x7f5848136af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 1 --2- 172.21.15.98:0/485912509 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58480a4df0 0x7f58480a51d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 1 --2- 172.21.15.98:0/485912509 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58480a2f40 0x7f58480a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 1 -- 172.21.15.98:0/485912509 >> 172.21.15.98:0/485912509 conn(0x7f58480a95b0 msgr2=0x7f58480a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:34.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 1 -- 172.21.15.98:0/485912509 shutdown_connections 2024-06-11T19:09:34.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 1 -- 172.21.15.98:0/485912509 wait complete. 2024-06-11T19:09:34.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 10 monclient: build_initial_monmap 2024-06-11T19:09:34.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.157+0000 7f584f5fe640 10 monclient: monmap: 2024-06-11T19:09:34.178 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:34.178 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:34.178 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:34.178 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:34.178 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:34.179 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:34.179 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:34.179 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:34.179 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:34.179 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:34.179 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:34.179 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:34.179 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 1 librados: starting msgr at 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 1 librados: starting objecter 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 1 Processor -- start 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 1 -- start start 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 1 librados: setting wanted keys 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 1 librados: calling monclient init 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 10 monclient: init 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 10 monclient: _add_conns ranks=[2,7,5,8,6,0,3,4,1] 2024-06-11T19:09:34.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58480a2f40 0x7f5848135560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 10 monclient(hunting): picked mon.noname-b con 0x7f58480a2f40 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58480a4df0 0x7f5848135aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 10 monclient(hunting): picked mon.noname-i con 0x7f58480a4df0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58481366d0 0x7f5848121040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 10 monclient(hunting): picked mon.noname-h con 0x7f58481366d0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f58480a1250 con 0x7f58480a2f40 2024-06-11T19:09:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f5848137030 con 0x7f58481366d0 2024-06-11T19:09:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f58481cc4d0 con 0x7f58480a4df0 2024-06-11T19:09:34.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584f5fe640 10 monclient(hunting): authenticate will time out at 783.965708s 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584d39a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58480a2f40 0x7f5848135560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584d39a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58480a2f40 0x7f5848135560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47862/0 (socket says 172.21.15.98:47862) 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584d39a640 1 -- 172.21.15.98:0/2980514146 learned_addr learned my addr 172.21.15.98:0/2980514146 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584d39a640 10 monclient(hunting): get_auth_request con 0x7f58480a2f40 auth_method 0 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584d39a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584d39a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584d39a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584d39a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584d39a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.158+0000 7f584d39a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584d39a640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-06-11T19:09:34.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584d39a640 1 -- 172.21.15.98:0/2980514146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58480a4df0 msgr2=0x7f5848135aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584d39a640 1 --2- 172.21.15.98:0/2980514146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58480a4df0 0x7f5848135aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584d39a640 1 -- 172.21.15.98:0/2980514146 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58481366d0 msgr2=0x7f5848121040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584d39a640 1 --2- 172.21.15.98:0/2980514146 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58481366d0 0x7f5848121040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584d39a640 10 monclient: _finish_hunting 0 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584d39a640 1 monclient: found mon.noname-b 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584d39a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584d39a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584d39a640 1 -- 172.21.15.98:0/2980514146 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5848124c60 con 0x7f58480a2f40 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584d39a640 1 --2- 172.21.15.98:0/2980514146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58480a2f40 0x7f5848135560 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f584400d800 tx=0x7f584400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 10 client.?.objecter ms_handle_connect 0x7f58480a2f40 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 1 -- 172.21.15.98:0/2980514146 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f584400bab0 con 0x7f58480a2f40 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:34.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 10 monclient: dump: 2024-06-11T19:09:34.184 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:34.184 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.184 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.184 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.184 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:34.184 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.184 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:34.184 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:34.184 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:34.184 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:34.184 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 1 -- 172.21.15.98:0/2980514146 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5844019070 con 0x7f58480a2f40 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584f5fe640 5 monclient: authenticate success, global_id 4250 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584f5fe640 10 monclient: _renew_subs 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584f5fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584f5fe640 1 -- 172.21.15.98:0/2980514146 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5848124f40 con 0x7f58480a2f40 2024-06-11T19:09:34.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 1 -- 172.21.15.98:0/2980514146 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f584400bc90 con 0x7f58480a2f40 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583dffb640 10 monclient: dump: 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:34.186 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f583f7fe640 4 set_mon_vals no callback set 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584f5fe640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584f5fe640 10 monclient: _renew_subs 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584f5fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584f5fe640 1 -- 172.21.15.98:0/2980514146 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5848136cb0 con 0x7f58480a2f40 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.159+0000 7f584f5fe640 1 librados: init done 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f583dffb640 1 -- 172.21.15.98:0/2980514146 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f584400b640 con 0x7f58480a2f40 2024-06-11T19:09:34.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f583dffb640 1 --2- 172.21.15.98:0/2980514146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f582c03fde0 0x7f582c0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f584f5fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f584f5fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f584f5fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f583dffb640 1 -- 172.21.15.98:0/2980514146 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4346+0+0 (secure 0 0 0) 0x7f584400f040 con 0x7f58480a2f40 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f583dffb640 10 client.4250.objecter ms_dispatch 0x7f5848000b90 osd_map(17..17 src has 1..17) 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f583dffb640 3 client.4250.objecter handle_osd_map got epochs [17,17] > 0 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f583dffb640 3 client.4250.objecter handle_osd_map decoding full epoch 17 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f584f5fe640 1 -- 172.21.15.98:0/2980514146 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58481373c0 con 0x7f58480a2f40 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f583dffb640 20 client.4250.objecter dump_active .. 0 homeless 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f584cb99640 1 --2- 172.21.15.98:0/2980514146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f582c03fde0 0x7f582c0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.160+0000 7f584cb99640 10 monclient: get_auth_request con 0x7f582c03fde0 auth_method 0 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.161+0000 7f583dffb640 10 client.4250.objecter ms_handle_connect 0x7f582c03fde0 2024-06-11T19:09:34.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.161+0000 7f584cb99640 1 --2- 172.21.15.98:0/2980514146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f582c03fde0 0x7f582c0422a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5838031fd0 tx=0x7f5838030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:34.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.165+0000 7f583dffb640 1 -- 172.21.15.98:0/2980514146 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5844022020 con 0x7f58480a2f40 2024-06-11T19:09:34.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.165+0000 7f583dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:34.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.165+0000 7f583dffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.390+0000 7f584f5fe640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-06-11T19:09:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.390+0000 7f584f5fe640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-06-11T19:09:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.390+0000 7f584f5fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.390+0000 7f584f5fe640 1 -- 172.21.15.98:0/2980514146 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f58481234c0 con 0x7f58480a2f40 2024-06-11T19:09:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.391+0000 7f583dffb640 1 -- 172.21.15.98:0/2980514146 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+95496 (secure 0 0 0) 0x7f5844013390 con 0x7f58480a2f40 2024-06-11T19:09:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.391+0000 7f583dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-06-11T19:09:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.391+0000 7f583dffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:34.393 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:09:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 10 librados: watch_flush enter 2024-06-11T19:09:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 10 librados: watch_flush exit 2024-06-11T19:09:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 20 client.4250.objecter shutdown clearing up homeless session... 2024-06-11T19:09:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 10 client.4250.objecter successfully canceled tick 2024-06-11T19:09:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 -- 172.21.15.98:0/2980514146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f582c03fde0 msgr2=0x7f582c0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 --2- 172.21.15.98:0/2980514146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f582c03fde0 0x7f582c0422a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5838031fd0 tx=0x7f5838030000 comp rx=0 tx=0).stop 2024-06-11T19:09:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 10 monclient: shutdown 2024-06-11T19:09:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 -- 172.21.15.98:0/2980514146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58480a2f40 msgr2=0x7f5848135560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 --2- 172.21.15.98:0/2980514146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58480a2f40 0x7f5848135560 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f584400d800 tx=0x7f584400dcc0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 -- 172.21.15.98:0/2980514146 shutdown_connections 2024-06-11T19:09:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 --2- 172.21.15.98:0/2980514146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f582c03fde0 0x7f582c0422a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 --2- 172.21.15.98:0/2980514146 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58481366d0 0x7f5848121040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 --2- 172.21.15.98:0/2980514146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58480a4df0 0x7f5848135aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 --2- 172.21.15.98:0/2980514146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58480a2f40 0x7f5848135560 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 -- 172.21.15.98:0/2980514146 >> 172.21.15.98:0/2980514146 conn(0x7f58480a95b0 msgr2=0x7f58480a4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 -- 172.21.15.98:0/2980514146 shutdown_connections 2024-06-11T19:09:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 -- 172.21.15.98:0/2980514146 wait complete. 2024-06-11T19:09:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.394+0000 7f584f5fe640 1 librados: shutdown 2024-06-11T19:09:34.404 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"flags":0,"active_gid":4111,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2398011028},{"type":"v1","addr":"172.21.15.117:6825","nonce":2398011028}]},"active_addr":"172.21.15.117:6825/2398011028","active_change":"2024-06-11T19:09:19.356632+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":3344709121}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":299377982}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":2210857100}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":367149216}]}]} 2024-06-11T19:09:34.408 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-06-11T19:09:34.408 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-06-11T19:09:34.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-11T19:09:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.506+0000 7fb5e5a69640 10 monclient: get_monmap_and_config 2024-06-11T19:09:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.506+0000 7fb5e5a69640 10 monclient: build_initial_monmap 2024-06-11T19:09:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.506+0000 7fb5e5a69640 10 monclient: monmap: 2024-06-11T19:09:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:34.507542+0000 2024-06-11T19:09:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:34.507542+0000 2024-06-11T19:09:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:34.508 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:34.508 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 1 Processor -- start 2024-06-11T19:09:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 1 -- start start 2024-06-11T19:09:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 10 monclient: init 2024-06-11T19:09:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 10 monclient: _add_conns ranks=[7,3,8,1,10,0,9,4,6,5,2] 2024-06-11T19:09:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb5e00a4920 0x7fb5e00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 10 monclient(hunting): picked mon.noname-i con 0x7fb5e00a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5e00a5240 0x7fb5e00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 10 monclient(hunting): picked mon.noname-g con 0x7fb5e00a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a2530 0x7fb5e00a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 10 monclient(hunting): picked mon.noname-e con 0x7fb5e00a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fb5e00a7140 con 0x7fb5e00a5240 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fb5e00a2b50 con 0x7fb5e00a4920 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb5e00a2e30 con 0x7fb5e00a2530 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5e5a69640 10 monclient(hunting): authenticate will time out at 784.314794s 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5dffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a2530 0x7fb5e00a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5dffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a2530 0x7fb5e00a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54436/0 (socket says 172.21.15.98:54436) 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5dffff640 1 -- 172.21.15.98:0/3972143792 learned_addr learned my addr 172.21.15.98:0/3972143792 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5dffff640 10 monclient(hunting): get_auth_request con 0x7fb5e00a2530 auth_method 0 2024-06-11T19:09:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5dffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.507+0000 7fb5dffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5df7fe640 1 --2- 172.21.15.98:0/3972143792 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb5e00a4920 0x7fb5e00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5deffd640 1 --2- 172.21.15.98:0/3972143792 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5e00a5240 0x7fb5e00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5df7fe640 10 monclient(hunting): get_auth_request con 0x7fb5e00a4920 auth_method 0 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5df7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5deffd640 10 monclient(hunting): get_auth_request con 0x7fb5e00a5240 auth_method 0 2024-06-11T19:09:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5deffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5deffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 1 -- 172.21.15.98:0/3972143792 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb5e00a4920 msgr2=0x7fb5e00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 1 --2- 172.21.15.98:0/3972143792 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb5e00a4920 0x7fb5e00a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 1 -- 172.21.15.98:0/3972143792 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5e00a5240 msgr2=0x7fb5e00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 1 --2- 172.21.15.98:0/3972143792 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5e00a5240 0x7fb5e00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 10 monclient: _finish_hunting 0 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 1 monclient: found mon.noname-e 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 1 -- 172.21.15.98:0/3972143792 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e00a3110 con 0x7fb5e00a2530 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5df7fe640 1 --2- 172.21.15.98:0/3972143792 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb5e00a4920 0x7fb5e00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.508+0000 7fb5dffff640 1 --2- 172.21.15.98:0/3972143792 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a2530 0x7fb5e00a5ee0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009e80 tx=0x7fb5d802bd70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=721caaaa27a0106f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 1 -- 172.21.15.98:0/3972143792 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5d802cb80 con 0x7fb5e00a2530 2024-06-11T19:09:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 10 monclient: dump: 2024-06-11T19:09:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 5 monclient: authenticate success, global_id 4247 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 1 -- 172.21.15.98:0/3972143792 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5d802cd40 con 0x7fb5e00a2530 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 4 set_mon_vals no callback set 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 1 -- 172.21.15.98:0/3972143792 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5d8034860 con 0x7fb5e00a2530 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5de7fc640 10 monclient: dump: 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 10 monclient: shutdown 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 1 -- 172.21.15.98:0/3972143792 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a2530 msgr2=0x7fb5e00a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 1 --2- 172.21.15.98:0/3972143792 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a2530 0x7fb5e00a5ee0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009e80 tx=0x7fb5d802bd70 comp rx=0 tx=0).stop 2024-06-11T19:09:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 1 -- 172.21.15.98:0/3972143792 shutdown_connections 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 1 --2- 172.21.15.98:0/3972143792 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a2530 0x7fb5e00a5ee0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 1 --2- 172.21.15.98:0/3972143792 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5e00a5240 0x7fb5e00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 1 --2- 172.21.15.98:0/3972143792 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb5e00a4920 0x7fb5e00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 1 -- 172.21.15.98:0/3972143792 >> 172.21.15.98:0/3972143792 conn(0x7fb5e00ab850 msgr2=0x7fb5e00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 1 -- 172.21.15.98:0/3972143792 shutdown_connections 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.509+0000 7fb5e5a69640 1 -- 172.21.15.98:0/3972143792 wait complete. 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.510+0000 7fb5e5a69640 10 monclient: build_initial_monmap 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.510+0000 7fb5e5a69640 10 monclient: monmap: 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.510+0000 7fb5e5a69640 1 librados: starting msgr at 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.510+0000 7fb5e5a69640 1 librados: starting objecter 2024-06-11T19:09:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.510+0000 7fb5e5a69640 1 Processor -- start 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.510+0000 7fb5e5a69640 1 -- start start 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.510+0000 7fb5e5a69640 1 librados: setting wanted keys 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.510+0000 7fb5e5a69640 1 librados: calling monclient init 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.510+0000 7fb5e5a69640 10 monclient: init 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 10 monclient: _add_conns ranks=[6,2,8,1,5,3,7,4,0] 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5e00a2530 0x7fb5e01cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 10 monclient(hunting): picked mon.noname-d con 0x7fb5e00a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5e00a4920 0x7fb5e01ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 10 monclient(hunting): picked mon.noname-b con 0x7fb5e00a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a5240 0x7fb5e0124df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 10 monclient(hunting): picked mon.noname-e con 0x7fb5e00a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb5e00a6da0 con 0x7fb5e00a4920 2024-06-11T19:09:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb5e0138e50 con 0x7fb5e00a2530 2024-06-11T19:09:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb5e00a6420 con 0x7fb5e00a5240 2024-06-11T19:09:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5e5a69640 10 monclient(hunting): authenticate will time out at 784.318252s 2024-06-11T19:09:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a5240 0x7fb5e0124df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5df7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5e00a2530 0x7fb5e01cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5deffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5e00a4920 0x7fb5e01ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a5240 0x7fb5e0124df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54444/0 (socket says 172.21.15.98:54444) 2024-06-11T19:09:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5df7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5e00a2530 0x7fb5e01cdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49704/0 (socket says 172.21.15.98:49704) 2024-06-11T19:09:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 1 -- 172.21.15.98:0/961004308 learned_addr learned my addr 172.21.15.98:0/961004308 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5deffd640 10 monclient(hunting): get_auth_request con 0x7fb5e00a4920 auth_method 0 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5deffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5deffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 10 monclient(hunting): get_auth_request con 0x7fb5e00a5240 auth_method 0 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5df7fe640 10 monclient(hunting): get_auth_request con 0x7fb5e00a2530 auth_method 0 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5df7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 1 -- 172.21.15.98:0/961004308 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5e00a2530 msgr2=0x7fb5e01cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5e00a2530 0x7fb5e01cdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 1 -- 172.21.15.98:0/961004308 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5e00a4920 msgr2=0x7fb5e01ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5e00a4920 0x7fb5e01ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 10 monclient: _finish_hunting 0 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 1 monclient: found mon.noname-e 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 1 -- 172.21.15.98:0/961004308 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e00a8e10 con 0x7fb5e00a5240 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5deffd640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5e00a4920 0x7fb5e01ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5df7fe640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5e00a2530 0x7fb5e01cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5dffff640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a5240 0x7fb5e0124df0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8036040 tx=0x7fb5d8004a40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5d7fff640 10 client.?.objecter ms_handle_connect 0x7fb5e00a5240 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.511+0000 7fb5d7fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 1 -- 172.21.15.98:0/961004308 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5d8042030 con 0x7fb5e00a5240 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 10 monclient: dump: 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 10 monclient: _finish_auth 0 2024-06-11T19:09:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5e5a69640 5 monclient: authenticate success, global_id 4256 2024-06-11T19:09:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 1 -- 172.21.15.98:0/961004308 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5d80046c0 con 0x7fb5e00a5240 2024-06-11T19:09:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5e5a69640 10 monclient: _renew_subs 2024-06-11T19:09:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5e5a69640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5e5a69640 1 -- 172.21.15.98:0/961004308 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5e0128d30 con 0x7fb5e00a5240 2024-06-11T19:09:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 1 -- 172.21.15.98:0/961004308 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5d803e8c0 con 0x7fb5e00a5240 2024-06-11T19:09:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5ddffb640 4 set_mon_vals no callback set 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 10 monclient: dump: 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5e5a69640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5e5a69640 10 monclient: _renew_subs 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5e5a69640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5e5a69640 1 -- 172.21.15.98:0/961004308 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5e01269d0 con 0x7fb5e00a5240 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5e5a69640 1 librados: init done 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.512+0000 7fb5d7fff640 1 -- 172.21.15.98:0/961004308 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fb5d8045070 con 0x7fb5e00a5240 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5e5a69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5e5a69640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5e5a69640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5e5a69640 1 -- 172.21.15.98:0/961004308 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5e00a6720 con 0x7fb5e00a5240 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5d7fff640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb5c003fdb0 0x7fb5c0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5d7fff640 1 -- 172.21.15.98:0/961004308 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(17..17 src has 1..17) ==== 4346+0+0 (secure 0 0 0) 0x7fb5d8043070 con 0x7fb5e00a5240 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5d7fff640 10 client.4256.objecter ms_dispatch 0x7fb5e0000b90 osd_map(17..17 src has 1..17) 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5d7fff640 3 client.4256.objecter handle_osd_map got epochs [17,17] > 0 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5d7fff640 3 client.4256.objecter handle_osd_map decoding full epoch 17 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5d7fff640 20 client.4256.objecter dump_active .. 0 homeless 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5df7fe640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb5c003fdb0 0x7fb5c0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5df7fe640 10 monclient: get_auth_request con 0x7fb5c003fdb0 auth_method 0 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5df7fe640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb5c003fdb0 0x7fb5c0042270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0006e80 tx=0x7fb5d0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.513+0000 7fb5d7fff640 10 client.4256.objecter ms_handle_connect 0x7fb5c003fdb0 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.517+0000 7fb5d7fff640 1 -- 172.21.15.98:0/961004308 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb5d804c020 con 0x7fb5e00a5240 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.517+0000 7fb5d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.517+0000 7fb5d7fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:34.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.701+0000 7fb5e5a69640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.702+0000 7fb5e5a69640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.702+0000 7fb5e5a69640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.702+0000 7fb5e5a69640 1 -- 172.21.15.98:0/961004308 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5e00aac60 con 0x7fb5e00a5240 2024-06-11T19:09:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.702+0000 7fb5d7fff640 1 -- 172.21.15.98:0/961004308 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12260 (secure 0 0 0) 0x7fb5e00a8e10 con 0x7fb5e00a5240 2024-06-11T19:09:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.702+0000 7fb5d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.702+0000 7fb5d7fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:34.703 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:09:34.703 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","created":"2024-06-11T19:09:15.016586+0000","modified":"2024-06-11T19:09:33.704003+0000","last_up_change":"2024-06-11T19:09:24.656699+0000","last_in_change":"2024-06-11T19:09:16.662340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-11T19:09:25.381456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-11T19:09:31.785910+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ef1075e3-3e1b-4904-b14b-90751df6c38c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6811","nonce":4087730304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6815","nonce":4087730304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6823","nonce":4087730304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6819","nonce":4087730304}]},"public_addr":"172.21.15.98:6811/4087730304","cluster_addr":"172.21.15.98:6815/4087730304","heartbeat_back_addr":"172.21.15.98:6823/4087730304","heartbeat_front_addr":"172.21.15.98:6819/4087730304","state":["exists","up"]},{"osd":1,"uuid":"985893e9-bd60-46fc-b913-7502ff73628d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6801","nonce":3886067876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6803","nonce":3886067876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6807","nonce":3886067876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6805","nonce":3886067876}]},"public_addr":"172.21.15.98:6801/3886067876","cluster_addr":"172.21.15.98:6803/3886067876","heartbeat_back_addr":"172.21.15.98:6807/3886067876","heartbeat_front_addr":"172.21.15.98:6805/3886067876","state":["exists","up"]},{"osd":2,"uuid":"2b0b4d24-8009-4cec-80e4-c0fa2d705879","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6810","nonce":3751501255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6814","nonce":3751501255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6822","nonce":3751501255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6818","nonce":3751501255}]},"public_addr":"172.21.15.98:6810/3751501255","cluster_addr":"172.21.15.98:6814/3751501255","heartbeat_back_addr":"172.21.15.98:6822/3751501255","heartbeat_front_addr":"172.21.15.98:6818/3751501255","state":["exists","up"]},{"osd":3,"uuid":"73793608-9e04-4837-bd2b-1b130a4e88d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6809","nonce":2561136926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6811","nonce":2561136926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6815","nonce":2561136926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6813","nonce":2561136926}]},"public_addr":"172.21.15.117:6809/2561136926","cluster_addr":"172.21.15.117:6811/2561136926","heartbeat_back_addr":"172.21.15.117:6815/2561136926","heartbeat_front_addr":"172.21.15.117:6813/2561136926","state":["exists","up"]},{"osd":4,"uuid":"60e657ae-e2a2-471e-b998-fba98d9c94e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6801","nonce":3877218228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6803","nonce":3877218228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6807","nonce":3877218228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6805","nonce":3877218228}]},"public_addr":"172.21.15.117:6801/3877218228","cluster_addr":"172.21.15.117:6803/3877218228","heartbeat_back_addr":"172.21.15.117:6807/3877218228","heartbeat_front_addr":"172.21.15.117:6805/3877218228","state":["exists","up"]},{"osd":5,"uuid":"dae66a65-2ca2-4f8a-98e3-52b190abee7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6817","nonce":987023138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6819","nonce":987023138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6823","nonce":987023138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6821","nonce":987023138}]},"public_addr":"172.21.15.117:6817/987023138","cluster_addr":"172.21.15.117:6819/987023138","heartbeat_back_addr":"172.21.15.117:6823/987023138","heartbeat_front_addr":"172.21.15.117:6821/987023138","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.744223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.800796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.792849+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.854633+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.844451+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.763441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-11T19:09:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 10 librados: watch_flush enter 2024-06-11T19:09:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 10 librados: watch_flush exit 2024-06-11T19:09:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 20 client.4256.objecter shutdown clearing up homeless session... 2024-06-11T19:09:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 10 client.4256.objecter successfully canceled tick 2024-06-11T19:09:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 1 -- 172.21.15.98:0/961004308 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb5c003fdb0 msgr2=0x7fb5c0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb5c003fdb0 0x7fb5c0042270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0006e80 tx=0x7fb5d0008040 comp rx=0 tx=0).stop 2024-06-11T19:09:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 10 monclient: shutdown 2024-06-11T19:09:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 1 -- 172.21.15.98:0/961004308 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a5240 msgr2=0x7fb5e0124df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a5240 0x7fb5e0124df0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8036040 tx=0x7fb5d8004a40 comp rx=0 tx=0).stop 2024-06-11T19:09:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 1 -- 172.21.15.98:0/961004308 shutdown_connections 2024-06-11T19:09:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb5c003fdb0 0x7fb5c0042270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5e00a5240 0x7fb5e0124df0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5e00a4920 0x7fb5e01ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 1 --2- 172.21.15.98:0/961004308 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5e00a2530 0x7fb5e01cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.704+0000 7fb5e5a69640 1 -- 172.21.15.98:0/961004308 >> 172.21.15.98:0/961004308 conn(0x7fb5e00ab850 msgr2=0x7fb5e009c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:34.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.705+0000 7fb5e5a69640 1 -- 172.21.15.98:0/961004308 shutdown_connections 2024-06-11T19:09:34.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.705+0000 7fb5e5a69640 1 -- 172.21.15.98:0/961004308 wait complete. 2024-06-11T19:09:34.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.705+0000 7fb5e5a69640 1 librados: shutdown 2024-06-11T19:09:34.717 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-06-11T19:09:34.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-11T19:09:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.816+0000 7fd3cc36d640 10 monclient: get_monmap_and_config 2024-06-11T19:09:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.816+0000 7fd3cc36d640 10 monclient: build_initial_monmap 2024-06-11T19:09:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.816+0000 7fd3cc36d640 10 monclient: monmap: 2024-06-11T19:09:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:34.817531+0000 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:34.817531+0000 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:34.828 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 1 Processor -- start 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 1 -- start start 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 10 monclient: init 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 10 monclient: _add_conns ranks=[6,8,7,3,10,2,9,0,4,5,1] 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a4920 0x7fd3c40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 10 monclient(hunting): picked mon.noname-d con 0x7fd3c40a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a5240 0x7fd3c40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 10 monclient(hunting): picked mon.noname-e con 0x7fd3c40a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd3c40a2530 0x7fd3c40a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 10 monclient(hunting): picked mon.noname-i con 0x7fd3c40a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd3c40a7140 con 0x7fd3c40a4920 2024-06-11T19:09:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd3c40a2b50 con 0x7fd3c40a2530 2024-06-11T19:09:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd3c40a2e30 con 0x7fd3c40a5240 2024-06-11T19:09:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3cc36d640 10 monclient(hunting): authenticate will time out at 784.624780s 2024-06-11T19:09:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3c9908640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a5240 0x7fd3c40a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3ca109640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a4920 0x7fd3c40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3ca109640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a4920 0x7fd3c40a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49710/0 (socket says 172.21.15.98:49710) 2024-06-11T19:09:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3ca109640 1 -- 172.21.15.98:0/3776630061 learned_addr learned my addr 172.21.15.98:0/3776630061 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3ca109640 10 monclient(hunting): get_auth_request con 0x7fd3c40a4920 auth_method 0 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3ca109640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3ca109640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3ca109640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3c9908640 10 monclient(hunting): get_auth_request con 0x7fd3c40a5240 auth_method 0 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3c9908640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3c9908640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.817+0000 7fd3c9908640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3c9908640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3c9908640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3c9908640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 10 monclient(hunting): handle_auth_done global_id 4254 payload 274 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 1 -- 172.21.15.98:0/3776630061 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a5240 msgr2=0x7fd3c40a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 1 --2- 172.21.15.98:0/3776630061 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a5240 0x7fd3c40a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 1 -- 172.21.15.98:0/3776630061 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd3c40a2530 msgr2=0x7fd3c40a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca90a640 1 --2- 172.21.15.98:0/3776630061 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd3c40a2530 0x7fd3c40a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 1 --2- 172.21.15.98:0/3776630061 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd3c40a2530 0x7fd3c40a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 10 monclient: _finish_hunting 0 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 1 monclient: found mon.noname-d 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 1 -- 172.21.15.98:0/3776630061 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c40a3110 con 0x7fd3c40a4920 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca90a640 10 monclient: get_auth_request con 0x7fd3c40a2530 auth_method 0 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3c9908640 1 --2- 172.21.15.98:0/3776630061 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a5240 0x7fd3c40a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca90a640 1 --2- 172.21.15.98:0/3776630061 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd3c40a2530 0x7fd3c40a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:34.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.818+0000 7fd3ca109640 1 --2- 172.21.15.98:0/3776630061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a4920 0x7fd3c40a4d00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40098a0 tx=0x7fd3b402b7a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c8b1dc450724594c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 1 -- 172.21.15.98:0/3776630061 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3b4038070 con 0x7fd3c40a4920 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 10 monclient: dump: 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:34.833 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:34.834 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 10 monclient: _finish_auth 0 2024-06-11T19:09:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 10 monclient: _check_auth_tickets 2024-06-11T19:09:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 1 -- 172.21.15.98:0/3776630061 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3b4031970 con 0x7fd3c40a4920 2024-06-11T19:09:34.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 5 monclient: authenticate success, global_id 4254 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 1 -- 172.21.15.98:0/3776630061 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3b4031ba0 con 0x7fd3c40a4920 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 4 set_mon_vals no callback set 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3c9107640 10 monclient: dump: 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:34.835 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 10 monclient: shutdown 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:34.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3776630061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a4920 msgr2=0x7fd3c40a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 1 --2- 172.21.15.98:0/3776630061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a4920 0x7fd3c40a4d00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40098a0 tx=0x7fd3b402b7a0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3776630061 shutdown_connections 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 1 --2- 172.21.15.98:0/3776630061 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a5240 0x7fd3c40a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 1 --2- 172.21.15.98:0/3776630061 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd3c40a2530 0x7fd3c40a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 1 --2- 172.21.15.98:0/3776630061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a4920 0x7fd3c40a4d00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3776630061 >> 172.21.15.98:0/3776630061 conn(0x7fd3c40ab850 msgr2=0x7fd3c40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3776630061 shutdown_connections 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.819+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3776630061 wait complete. 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.820+0000 7fd3cc36d640 10 monclient: build_initial_monmap 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.820+0000 7fd3cc36d640 10 monclient: monmap: 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:34.837 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.820+0000 7fd3cc36d640 1 librados: starting msgr at 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.820+0000 7fd3cc36d640 1 librados: starting objecter 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.820+0000 7fd3cc36d640 1 Processor -- start 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.820+0000 7fd3cc36d640 1 -- start start 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.820+0000 7fd3cc36d640 1 librados: setting wanted keys 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.820+0000 7fd3cc36d640 1 librados: calling monclient init 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.820+0000 7fd3cc36d640 10 monclient: init 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 10 monclient: _add_conns ranks=[8,7,6,4,0,5,3,1,2] 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a2530 0x7fd3c41cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 10 monclient(hunting): picked mon.noname-e con 0x7fd3c40a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd3c40a4920 0x7fd3c41cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 10 monclient(hunting): picked mon.noname-i con 0x7fd3c40a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:34.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a5240 0x7fd3c4125030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 10 monclient(hunting): picked mon.noname-d con 0x7fd3c40a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd3c40a6da0 con 0x7fd3c40a5240 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd3c4138e50 con 0x7fd3c40a4920 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd3c40a6420 con 0x7fd3c40a2530 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 10 monclient(hunting): authenticate will time out at 784.628235s 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a2530 0x7fd3c41cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca90a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a5240 0x7fd3c4125030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a2530 0x7fd3c41cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54456/0 (socket says 172.21.15.98:54456) 2024-06-11T19:09:34.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 1 -- 172.21.15.98:0/3836559052 learned_addr learned my addr 172.21.15.98:0/3836559052 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 10 monclient(hunting): get_auth_request con 0x7fd3c40a2530 auth_method 0 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca90a640 10 monclient(hunting): get_auth_request con 0x7fd3c40a5240 auth_method 0 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca90a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca90a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca90a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca90a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:34.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca90a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca90a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3c9908640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd3c40a4920 0x7fd3c41cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3c9908640 10 monclient(hunting): get_auth_request con 0x7fd3c40a4920 auth_method 0 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3c9908640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3c9908640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3c9908640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 1 -- 172.21.15.98:0/3836559052 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd3c40a4920 msgr2=0x7fd3c41cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd3c40a4920 0x7fd3c41cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 1 -- 172.21.15.98:0/3836559052 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a5240 msgr2=0x7fd3c4125030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a5240 0x7fd3c4125030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 10 monclient: _finish_hunting 0 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 1 monclient: found mon.noname-e 2024-06-11T19:09:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 1 -- 172.21.15.98:0/3836559052 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c4137e90 con 0x7fd3c40a2530 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3c9908640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd3c40a4920 0x7fd3c41cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca90a640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a5240 0x7fd3c4125030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3ca109640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a2530 0x7fd3c41cdaa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40097c0 tx=0x7fd3b40046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3baffd640 10 client.?.objecter ms_handle_connect 0x7fd3c40a2530 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3baffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3baffd640 1 -- 172.21.15.98:0/3836559052 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3b4042020 con 0x7fd3c40a2530 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3baffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3baffd640 10 monclient: dump: 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3baffd640 10 monclient: _finish_auth 0 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3baffd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 5 monclient: authenticate success, global_id 4274 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3baffd640 1 -- 172.21.15.98:0/3836559052 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3b4031ea0 con 0x7fd3c40a2530 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 10 monclient: _renew_subs 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.821+0000 7fd3cc36d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.822+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3836559052 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c4128c10 con 0x7fd3c40a2530 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.822+0000 7fd3baffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.822+0000 7fd3baffd640 1 -- 172.21.15.98:0/3836559052 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3b4038040 con 0x7fd3c40a2530 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.822+0000 7fd3baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.822+0000 7fd3baffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.822+0000 7fd3cc36d640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.822+0000 7fd3c8906640 4 set_mon_vals no callback set 2024-06-11T19:09:34.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.822+0000 7fd3baffd640 10 monclient: dump: 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.822+0000 7fd3cc36d640 10 monclient: _renew_subs 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.822+0000 7fd3cc36d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.822+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3836559052 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c4128ef0 con 0x7fd3c40a2530 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.822+0000 7fd3cc36d640 1 librados: init done 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3baffd640 1 -- 172.21.15.98:0/3836559052 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd3b4045020 con 0x7fd3c40a2530 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3cc36d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3cc36d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3cc36d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3836559052 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3c41387f0 con 0x7fd3c40a2530 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3baffd640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3a803fdb0 0x7fd3a8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3baffd640 1 -- 172.21.15.98:0/3836559052 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fd3b4072b10 con 0x7fd3c40a2530 2024-06-11T19:09:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3baffd640 10 client.4274.objecter ms_dispatch 0x7fd3c4000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3baffd640 3 client.4274.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3baffd640 3 client.4274.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3baffd640 20 client.4274.objecter dump_active .. 0 homeless 2024-06-11T19:09:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3c9908640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3a803fdb0 0x7fd3a8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.823+0000 7fd3c9908640 10 monclient: get_auth_request con 0x7fd3a803fdb0 auth_method 0 2024-06-11T19:09:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.824+0000 7fd3baffd640 10 client.4274.objecter ms_handle_connect 0x7fd3a803fdb0 2024-06-11T19:09:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.824+0000 7fd3c9908640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3a803fdb0 0x7fd3a8042270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40a5620 tx=0x7fd3c0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.827+0000 7fd3baffd640 1 -- 172.21.15.98:0/3836559052 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd3b404c030 con 0x7fd3c40a2530 2024-06-11T19:09:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.827+0000 7fd3baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:34.827+0000 7fd3baffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:35.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.010+0000 7fd3cc36d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:35.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.010+0000 7fd3cc36d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:35.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.010+0000 7fd3cc36d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:35.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.010+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3836559052 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3c40a9050 con 0x7fd3c40a2530 2024-06-11T19:09:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.011+0000 7fd3baffd640 1 -- 172.21.15.98:0/3836559052 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12247 (secure 0 0 0) 0x7fd3b4030070 con 0x7fd3c40a2530 2024-06-11T19:09:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.011+0000 7fd3baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.011+0000 7fd3baffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:35.011 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:09:35.011 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","created":"2024-06-11T19:09:15.016586+0000","modified":"2024-06-11T19:09:34.710947+0000","last_up_change":"2024-06-11T19:09:24.656699+0000","last_in_change":"2024-06-11T19:09:16.662340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-11T19:09:25.381456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-11T19:09:31.785910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ef1075e3-3e1b-4904-b14b-90751df6c38c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6811","nonce":4087730304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6815","nonce":4087730304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6823","nonce":4087730304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6819","nonce":4087730304}]},"public_addr":"172.21.15.98:6811/4087730304","cluster_addr":"172.21.15.98:6815/4087730304","heartbeat_back_addr":"172.21.15.98:6823/4087730304","heartbeat_front_addr":"172.21.15.98:6819/4087730304","state":["exists","up"]},{"osd":1,"uuid":"985893e9-bd60-46fc-b913-7502ff73628d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6801","nonce":3886067876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6803","nonce":3886067876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6807","nonce":3886067876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6805","nonce":3886067876}]},"public_addr":"172.21.15.98:6801/3886067876","cluster_addr":"172.21.15.98:6803/3886067876","heartbeat_back_addr":"172.21.15.98:6807/3886067876","heartbeat_front_addr":"172.21.15.98:6805/3886067876","state":["exists","up"]},{"osd":2,"uuid":"2b0b4d24-8009-4cec-80e4-c0fa2d705879","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6810","nonce":3751501255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6814","nonce":3751501255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6822","nonce":3751501255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6818","nonce":3751501255}]},"public_addr":"172.21.15.98:6810/3751501255","cluster_addr":"172.21.15.98:6814/3751501255","heartbeat_back_addr":"172.21.15.98:6822/3751501255","heartbeat_front_addr":"172.21.15.98:6818/3751501255","state":["exists","up"]},{"osd":3,"uuid":"73793608-9e04-4837-bd2b-1b130a4e88d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6809","nonce":2561136926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6811","nonce":2561136926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6815","nonce":2561136926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6813","nonce":2561136926}]},"public_addr":"172.21.15.117:6809/2561136926","cluster_addr":"172.21.15.117:6811/2561136926","heartbeat_back_addr":"172.21.15.117:6815/2561136926","heartbeat_front_addr":"172.21.15.117:6813/2561136926","state":["exists","up"]},{"osd":4,"uuid":"60e657ae-e2a2-471e-b998-fba98d9c94e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6801","nonce":3877218228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6803","nonce":3877218228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6807","nonce":3877218228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6805","nonce":3877218228}]},"public_addr":"172.21.15.117:6801/3877218228","cluster_addr":"172.21.15.117:6803/3877218228","heartbeat_back_addr":"172.21.15.117:6807/3877218228","heartbeat_front_addr":"172.21.15.117:6805/3877218228","state":["exists","up"]},{"osd":5,"uuid":"dae66a65-2ca2-4f8a-98e3-52b190abee7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6817","nonce":987023138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6819","nonce":987023138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6823","nonce":987023138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6821","nonce":987023138}]},"public_addr":"172.21.15.117:6817/987023138","cluster_addr":"172.21.15.117:6819/987023138","heartbeat_back_addr":"172.21.15.117:6823/987023138","heartbeat_front_addr":"172.21.15.117:6821/987023138","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.744223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.800796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.792849+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.854633+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.844451+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.763441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-11T19:09:35.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 10 librados: watch_flush enter 2024-06-11T19:09:35.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 10 librados: watch_flush exit 2024-06-11T19:09:35.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 20 client.4274.objecter shutdown clearing up homeless session... 2024-06-11T19:09:35.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 10 client.4274.objecter successfully canceled tick 2024-06-11T19:09:35.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3836559052 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3a803fdb0 msgr2=0x7fd3a8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3a803fdb0 0x7fd3a8042270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40a5620 tx=0x7fd3c0008040 comp rx=0 tx=0).stop 2024-06-11T19:09:35.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 10 monclient: shutdown 2024-06-11T19:09:35.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3836559052 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a2530 msgr2=0x7fd3c41cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a2530 0x7fd3c41cdaa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40097c0 tx=0x7fd3b40046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3836559052 shutdown_connections 2024-06-11T19:09:35.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3a803fdb0 0x7fd3a8042270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3c40a5240 0x7fd3c4125030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd3c40a4920 0x7fd3c41cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 --2- 172.21.15.98:0/3836559052 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3c40a2530 0x7fd3c41cdaa0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3836559052 >> 172.21.15.98:0/3836559052 conn(0x7fd3c40ab850 msgr2=0x7fd3c40a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3836559052 shutdown_connections 2024-06-11T19:09:35.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 -- 172.21.15.98:0/3836559052 wait complete. 2024-06-11T19:09:35.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.013+0000 7fd3cc36d640 1 librados: shutdown 2024-06-11T19:09:35.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-06-11T19:09:35.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-06-11T19:09:35.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-06-11T19:09:35.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-06-11T19:09:35.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-06-11T19:09:35.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-06-11T19:09:35.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.139+0000 7f2931a22640 10 monclient: get_monmap_and_config 2024-06-11T19:09:35.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.139+0000 7f2931a22640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.139+0000 7f2931a22640 10 monclient: monmap: 2024-06-11T19:09:35.139 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.139 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.139 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:35.140187+0000 2024-06-11T19:09:35.139 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:35.140187+0000 2024-06-11T19:09:35.139 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.139 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.139 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:35.139 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:35.140 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.140 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.140 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.140 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.140 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.140 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.140 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.140 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:35.140 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:35.140 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.139+0000 7f2931a22640 1 Processor -- start 2024-06-11T19:09:35.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.139+0000 7f2931a22640 1 -- start start 2024-06-11T19:09:35.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.139+0000 7f2931a22640 10 monclient: init 2024-06-11T19:09:35.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7fd1c78b2640 10 monclient: get_monmap_and_config 2024-06-11T19:09:35.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7fd1c78b2640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 10 monclient: _add_conns ranks=[4,8,3,2,6,5,9,7,10,1,0] 2024-06-11T19:09:35.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f292c131c60 0x7f292c0a1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7fd1c78b2640 10 monclient: monmap: 2024-06-11T19:09:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:35.141317+0000 2024-06-11T19:09:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:35.141317+0000 2024-06-11T19:09:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:35.147 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 10 monclient(hunting): picked mon.noname-c con 0x7f292c131c60 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:35.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f292c132040 0x7f292c132460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 10 monclient(hunting): picked mon.noname-e con 0x7f292c132040 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f292c134e30 0x7f292c1352b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f292affd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f292c131c60 0x7f292c0a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 10 monclient(hunting): picked mon.noname-g con 0x7f292c134e30 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f292c0a02a0 con 0x7f292c134e30 2024-06-11T19:09:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f292c0a0580 con 0x7f292c131c60 2024-06-11T19:09:35.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f292affd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f292c131c60 0x7f292c0a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:40330/0 (socket says 172.21.15.98:40330) 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f292affd640 1 -- 172.21.15.98:0/688814164 learned_addr learned my addr 172.21.15.98:0/688814164 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 1 -- 172.21.15.98:0/688814164 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f292c0a0860 con 0x7f292c132040 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f2931a22640 10 monclient(hunting): authenticate will time out at 784.947814s 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f292affd640 10 monclient(hunting): get_auth_request con 0x7f292c131c60 auth_method 0 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f292affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f292affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f292affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f292a7fc640 1 --2- 172.21.15.98:0/688814164 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f292c132040 0x7f292c132460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f292a7fc640 10 monclient(hunting): get_auth_request con 0x7f292c132040 auth_method 0 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f292a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f292a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.140+0000 7f292a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7fd1c78b2640 1 Processor -- start 2024-06-11T19:09:35.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 10 monclient(hunting): handle_auth_done global_id 4270 payload 274 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 1 -- 172.21.15.98:0/688814164 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f292c132040 msgr2=0x7f292c132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292b7fe640 1 --2- 172.21.15.98:0/688814164 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f292c134e30 0x7f292c1352b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 1 --2- 172.21.15.98:0/688814164 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f292c132040 0x7f292c132460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 1 -- 172.21.15.98:0/688814164 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f292c134e30 msgr2=0x7f292c1352b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 1 --2- 172.21.15.98:0/688814164 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f292c134e30 0x7f292c1352b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 1 monclient: found mon.noname-c 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292affd640 1 -- 172.21.15.98:0/688814164 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292c09f570 con 0x7f292c131c60 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292b7fe640 10 monclient: get_auth_request con 0x7f292c134e30 auth_method 0 2024-06-11T19:09:35.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292b7fe640 1 --2- 172.21.15.98:0/688814164 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f292c134e30 0x7f292c1352b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:35.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7f292a7fc640 1 --2- 172.21.15.98:0/688814164 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f292c132040 0x7f292c132460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7fd1c78b2640 1 -- start start 2024-06-11T19:09:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.141+0000 7fd1c78b2640 10 monclient: init 2024-06-11T19:09:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 10 monclient: _add_conns ranks=[1,4,9,5,7,2,8,0,6,3,10] 2024-06-11T19:09:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd1c00a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1c00a2530 0x7fd1c013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 10 monclient(hunting): picked mon.noname-c con 0x7fd1c00a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd1c013e1a0 0x7fd1c0140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 10 monclient(hunting): picked mon.noname-a con 0x7fd1c013e1a0 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1c00a7140 con 0x7fd1c00a4920 2024-06-11T19:09:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd1c00a2b50 con 0x7fd1c013e1a0 2024-06-11T19:09:35.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd1c00a2e30 con 0x7fd1c00a2530 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c78b2640 10 monclient(hunting): authenticate will time out at 784.949416s 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c564e640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd1c00a4920 0x7fd1c00a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:40510/0 (socket says 172.21.15.98:40510) 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c4e4d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1c00a2530 0x7fd1c013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c564e640 1 -- 172.21.15.98:0/2993629710 learned_addr learned my addr 172.21.15.98:0/2993629710 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c4e4d640 10 monclient(hunting): get_auth_request con 0x7fd1c00a2530 auth_method 0 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c4e4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c4e4d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c4e4d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c4e4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c4e4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c4e4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7f292affd640 1 --2- 172.21.15.98:0/688814164 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f292c131c60 0x7f292c0a1520 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f291c013180 tx=0x7f291c033fc0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=d50933acca1de205 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c5e4f640 1 --2- 172.21.15.98:0/2993629710 >> v2:172.21.15.98:3300/0 conn(0x7fd1c013e1a0 0x7fd1c0140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c5e4f640 10 monclient(hunting): get_auth_request con 0x7fd1c013e1a0 auth_method 0 2024-06-11T19:09:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c5e4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c5e4f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c5e4f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c4e4d640 10 monclient(hunting): handle_auth_done global_id 4279 payload 274 2024-06-11T19:09:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.142+0000 7fd1c4e4d640 1 -- 172.21.15.98:0/2993629710 >> v2:172.21.15.98:3300/0 conn(0x7fd1c013e1a0 msgr2=0x7fd1c0140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7fd1c4e4d640 1 --2- 172.21.15.98:0/2993629710 >> v2:172.21.15.98:3300/0 conn(0x7fd1c013e1a0 0x7fd1c0140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7fd1c4e4d640 1 -- 172.21.15.98:0/2993629710 >> v1:172.21.15.117:6789/0 conn(0x7fd1c00a4920 legacy=0x7fd1c00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7fd1c4e4d640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7fd1c4e4d640 1 monclient: found mon.noname-c 2024-06-11T19:09:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7fd1c4e4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7fd1c4e4d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7fd1c4e4d640 1 -- 172.21.15.98:0/2993629710 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c00a3110 con 0x7fd1c00a2530 2024-06-11T19:09:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 1 -- 172.21.15.98:0/688814164 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f291c0110d0 con 0x7f292c131c60 2024-06-11T19:09:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 10 monclient: dump: 2024-06-11T19:09:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2931a22640 5 monclient: authenticate success, global_id 4270 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 1 -- 172.21.15.98:0/688814164 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f291c034b70 con 0x7f292c131c60 2024-06-11T19:09:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2931a22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:35.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7fd1c5e4f640 1 --2- 172.21.15.98:0/2993629710 >> v2:172.21.15.98:3300/0 conn(0x7fd1c013e1a0 0x7fd1c0140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:35.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 1 -- 172.21.15.98:0/688814164 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f291c0110d0 con 0x7f292c131c60 2024-06-11T19:09:35.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:35.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2929ffb640 10 monclient: dump: 2024-06-11T19:09:35.165 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.166 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.166 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.166 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.166 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.166 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.166 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.166 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.166 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.166 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.166 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.166 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.166 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.167 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.167 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.167 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2931a22640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:35.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.143+0000 7f2931a22640 4 set_mon_vals no callback set 2024-06-11T19:09:35.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7f2931a22640 10 monclient: shutdown 2024-06-11T19:09:35.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1c4e4d640 1 --2- 172.21.15.98:0/2993629710 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1c00a2530 0x7fd1c013dc60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00097a0 tx=0x7fd1b002b790 comp rx=0 tx=0).ready entity=mon.4 client_cookie=435b718dc38b8717 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7f2931a22640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7f2931a22640 1 -- 172.21.15.98:0/688814164 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f292c131c60 msgr2=0x7f292c0a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7f2931a22640 1 --2- 172.21.15.98:0/688814164 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f292c131c60 0x7f292c0a1520 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f291c013180 tx=0x7f291c033fc0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7f2931a22640 1 -- 172.21.15.98:0/688814164 shutdown_connections 2024-06-11T19:09:35.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7f2931a22640 1 --2- 172.21.15.98:0/688814164 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f292c134e30 0x7f292c1352b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7f2931a22640 1 --2- 172.21.15.98:0/688814164 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f292c132040 0x7f292c132460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 1 -- 172.21.15.98:0/2993629710 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1b0039070 con 0x7fd1c00a2530 2024-06-11T19:09:35.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:35.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 10 monclient: dump: 2024-06-11T19:09:35.169 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.169 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.169 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.169 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.169 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.169 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 1 -- 172.21.15.98:0/2993629710 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b0031960 con 0x7fd1c00a2530 2024-06-11T19:09:35.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 1 -- 172.21.15.98:0/2993629710 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1b0032380 con 0x7fd1c00a2530 2024-06-11T19:09:35.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:35.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7f2931a22640 1 --2- 172.21.15.98:0/688814164 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f292c131c60 0x7f292c0a1520 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7f2931a22640 1 -- 172.21.15.98:0/688814164 >> 172.21.15.98:0/688814164 conn(0x7f292c0a7850 msgr2=0x7f292c0a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1b7fff640 10 monclient: dump: 2024-06-11T19:09:35.172 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.172 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.172 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.172 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.172 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.172 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.172 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.172 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.172 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.173 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1c78b2640 5 monclient: authenticate success, global_id 4279 2024-06-11T19:09:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1c78b2640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7fd1c78b2640 4 set_mon_vals no callback set 2024-06-11T19:09:35.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7f2931a22640 1 -- 172.21.15.98:0/688814164 shutdown_connections 2024-06-11T19:09:35.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.144+0000 7f2931a22640 1 -- 172.21.15.98:0/688814164 wait complete. 2024-06-11T19:09:35.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7fd1c78b2640 10 monclient: shutdown 2024-06-11T19:09:35.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7f2931a22640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7fd1c78b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7fd1c78b2640 1 -- 172.21.15.98:0/2993629710 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1c00a2530 msgr2=0x7fd1c013dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7fd1c78b2640 1 --2- 172.21.15.98:0/2993629710 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1c00a2530 0x7fd1c013dc60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00097a0 tx=0x7fd1b002b790 comp rx=0 tx=0).stop 2024-06-11T19:09:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7f2931a22640 10 monclient: monmap: 2024-06-11T19:09:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.176 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.177 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7fd1c78b2640 1 -- 172.21.15.98:0/2993629710 shutdown_connections 2024-06-11T19:09:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7fd1c78b2640 1 --2- 172.21.15.98:0/2993629710 >> v2:172.21.15.98:3300/0 conn(0x7fd1c013e1a0 0x7fd1c0140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7fd1c78b2640 1 --2- 172.21.15.98:0/2993629710 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1c00a2530 0x7fd1c013dc60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7fd1c78b2640 1 -- 172.21.15.98:0/2993629710 >> 172.21.15.98:0/2993629710 conn(0x7fd1c00ab850 msgr2=0x7fd1c00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7fd1c78b2640 1 -- 172.21.15.98:0/2993629710 shutdown_connections 2024-06-11T19:09:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7f2931a22640 1 librados: starting msgr at 2024-06-11T19:09:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7f2931a22640 1 librados: starting objecter 2024-06-11T19:09:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7fd1c78b2640 1 -- 172.21.15.98:0/2993629710 wait complete. 2024-06-11T19:09:35.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.145+0000 7f2931a22640 1 Processor -- start 2024-06-11T19:09:35.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7f2931a22640 1 -- start start 2024-06-11T19:09:35.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7f2931a22640 1 librados: setting wanted keys 2024-06-11T19:09:35.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7f2931a22640 1 librados: calling monclient init 2024-06-11T19:09:35.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7f2931a22640 10 monclient: init 2024-06-11T19:09:35.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 10 monclient: monmap: 2024-06-11T19:09:35.180 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.180 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:35.180 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.181 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.182 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.182 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.182 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 1 librados: starting msgr at 2024-06-11T19:09:35.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 1 librados: starting objecter 2024-06-11T19:09:35.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 1 Processor -- start 2024-06-11T19:09:35.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 1 -- start start 2024-06-11T19:09:35.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 1 librados: setting wanted keys 2024-06-11T19:09:35.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 1 librados: calling monclient init 2024-06-11T19:09:35.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 10 monclient: init 2024-06-11T19:09:35.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7f2931a22640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7f2931a22640 10 monclient: _add_conns ranks=[7,6,2,8,3,0,1,5,4] 2024-06-11T19:09:35.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7f2931a22640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f292c131c60 0x7f292c1c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7f2931a22640 10 monclient(hunting): picked mon.noname-i con 0x7f292c131c60 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:35.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7f2931a22640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f292c132040 0x7f292c1c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7f2931a22640 10 monclient(hunting): picked mon.noname-d con 0x7f292c132040 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:35.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 10 monclient: _add_conns ranks=[2,3,1,8,7,6,5,4,0] 2024-06-11T19:09:35.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1c00a2530 0x7fd1c009d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f292c132040 0x7f292c1c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292affd640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f292c131c60 0x7f292c1c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292affd640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f292c131c60 0x7f292c1c9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:52884/0 (socket says 172.21.15.98:52884) 2024-06-11T19:09:35.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.146+0000 7fd1c78b2640 10 monclient(hunting): picked mon.noname-b con 0x7fd1c00a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:35.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c78b2640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1c00a4920 0x7fd1c009d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c78b2640 10 monclient(hunting): picked mon.noname-g con 0x7fd1c00a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c78b2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1c013e1a0 0x7fd1c009db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c78b2640 10 monclient(hunting): picked mon.noname-f con 0x7fd1c013e1a0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c78b2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c78b2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c0138e50 con 0x7fd1c013e1a0 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c78b2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c78b2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c00a84b0 con 0x7fd1c00a2530 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c78b2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c78b2640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c00a6da0 con 0x7fd1c00a4920 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c78b2640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c78b2640 10 monclient(hunting): authenticate will time out at 784.954062s 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c564e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1c00a2530 0x7fd1c009d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1c00a4920 0x7fd1c009d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c564e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1c00a2530 0x7fd1c009d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47884/0 (socket says 172.21.15.98:47884) 2024-06-11T19:09:35.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c564e640 1 -- 172.21.15.98:0/3295856107 learned_addr learned my addr 172.21.15.98:0/3295856107 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f292c132040 0x7f292c1c9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49732/0 (socket says 172.21.15.98:49732) 2024-06-11T19:09:35.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c564e640 10 monclient(hunting): get_auth_request con 0x7fd1c00a2530 auth_method 0 2024-06-11T19:09:35.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c564e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c564e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c564e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 10 monclient(hunting): get_auth_request con 0x7fd1c00a4920 auth_method 0 2024-06-11T19:09:35.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c5e4f640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1c013e1a0 0x7fd1c009db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c5e4f640 10 monclient(hunting): get_auth_request con 0x7fd1c013e1a0 auth_method 0 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c5e4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c5e4f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c5e4f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c564e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c564e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c564e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c5e4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c5e4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c5e4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2024-06-11T19:09:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 1 -- 172.21.15.98:0/3295856107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1c00a2530 msgr2=0x7fd1c009d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1c00a2530 0x7fd1c009d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 1 -- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1c013e1a0 msgr2=0x7fd1c009db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1c013e1a0 0x7fd1c009db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 1 monclient: found mon.noname-g 2024-06-11T19:09:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:09:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c4e4d640 1 -- 172.21.15.98:0/3295856107 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c00a4290 con 0x7fd1c00a4920 2024-06-11T19:09:35.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7fd1c564e640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1c00a2530 0x7fd1c009d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f2931a22640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f292c134e30 0x7f292c1c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292affd640 1 -- 172.21.15.98:0/2217245093 learned_addr learned my addr 172.21.15.98:0/2217245093 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f2931a22640 10 monclient(hunting): picked mon.noname-b con 0x7f292c134e30 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:35.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f2931a22640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1c4e4d640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1c00a4920 0x7fd1c009d640 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0033000 tx=0x7fd1b00046c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 10 client.?.objecter ms_handle_connect 0x7fd1c00a4920 2024-06-11T19:09:35.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:35.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f2931a22640 1 -- 172.21.15.98:0/2217245093 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f292c135970 con 0x7f292c134e30 2024-06-11T19:09:35.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f2931a22640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f2931a22640 1 -- 172.21.15.98:0/2217245093 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f292c1357f0 con 0x7f292c132040 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f2931a22640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f2931a22640 1 -- 172.21.15.98:0/2217245093 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f292c135af0 con 0x7f292c131c60 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f2931a22640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f2931a22640 10 monclient(hunting): authenticate will time out at 784.954176s 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292affd640 10 monclient(hunting): get_auth_request con 0x7f292c131c60 auth_method 0 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 10 monclient(hunting): get_auth_request con 0x7f292c132040 auth_method 0 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 10 monclient(hunting): handle_auth_done global_id 4272 payload 824 2024-06-11T19:09:35.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292b7fe640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f292c134e30 0x7f292c1c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 1 -- 172.21.15.98:0/2217245093 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f292c131c60 msgr2=0x7f292c1c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f292c131c60 0x7f292c1c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 1 -- 172.21.15.98:0/2217245093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f292c134e30 msgr2=0x7f292c1c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f292c134e30 0x7f292c1c9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 1 monclient: found mon.noname-d 2024-06-11T19:09:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:09:35.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 1 -- 172.21.15.98:0/3295856107 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1b0047070 con 0x7fd1c00a4920 2024-06-11T19:09:35.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:09:35.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 10 monclient: dump: 2024-06-11T19:09:35.198 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.198 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.198 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.198 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.200 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 1 -- 172.21.15.98:0/3295856107 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b00044b0 con 0x7fd1c00a4920 2024-06-11T19:09:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292a7fc640 1 -- 172.21.15.98:0/2217245093 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292c1bcb60 con 0x7f292c132040 2024-06-11T19:09:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.147+0000 7f292b7fe640 10 monclient: get_auth_request con 0x7f292c134e30 auth_method 0 2024-06-11T19:09:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f292b7fe640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f292c134e30 0x7f292c1c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f292a7fc640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f292c132040 0x7f292c1c9760 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f292000ebe0 tx=0x7f292000c520 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 10 client.?.objecter ms_handle_connect 0x7f292c132040 2024-06-11T19:09:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 1 -- 172.21.15.98:0/2217245093 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2920005e00 con 0x7f292c132040 2024-06-11T19:09:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:09:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 10 monclient: dump: 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.204 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 1 -- 172.21.15.98:0/3295856107 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1b0039040 con 0x7fd1c00a4920 2024-06-11T19:09:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1c78b2640 5 monclient: authenticate success, global_id 4260 2024-06-11T19:09:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1c78b2640 10 monclient: _renew_subs 2024-06-11T19:09:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1c78b2640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:09:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1c78b2640 1 -- 172.21.15.98:0/3295856107 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c00a0ee0 con 0x7fd1c00a4920 2024-06-11T19:09:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:09:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b5ffb640 10 monclient: dump: 2024-06-11T19:09:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.207 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1b77fe640 4 set_mon_vals no callback set 2024-06-11T19:09:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1c78b2640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1c78b2640 10 monclient: _renew_subs 2024-06-11T19:09:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1c78b2640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:09:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1c78b2640 1 -- 172.21.15.98:0/3295856107 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c00a0920 con 0x7fd1c00a4920 2024-06-11T19:09:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7fd1c78b2640 1 librados: init done 2024-06-11T19:09:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2931a22640 5 monclient: authenticate success, global_id 4272 2024-06-11T19:09:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2931a22640 10 monclient: _renew_subs 2024-06-11T19:09:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2931a22640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:09:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2931a22640 1 -- 172.21.15.98:0/2217245093 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f292c0a1590 con 0x7f292c132040 2024-06-11T19:09:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 1 -- 172.21.15.98:0/2217245093 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f292000ed90 con 0x7f292c132040 2024-06-11T19:09:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 1 -- 172.21.15.98:0/2217245093 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2920020060 con 0x7f292c132040 2024-06-11T19:09:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:09:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2913fff640 10 monclient: dump: 2024-06-11T19:09:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.213 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2931a22640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.149+0000 7fd1c78b2640 10 client.4260.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2931a22640 10 monclient: _renew_subs 2024-06-11T19:09:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2931a22640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:09:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2931a22640 1 -- 172.21.15.98:0/2217245093 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f292c1352b0 con 0x7f292c132040 2024-06-11T19:09:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.149+0000 7fd1c78b2640 15 client.4260.objecter _session_command_op_assign -1 1 2024-06-11T19:09:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.149+0000 7fd1c78b2640 20 client.4260.objecter _get_session osd=-1 returning homeless 2024-06-11T19:09:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.149+0000 7fd1c78b2640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.149+0000 7fd1c78b2640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:09:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.149+0000 7fd1c78b2640 1 -- 172.21.15.98:0/3295856107 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd1c00a4020 con 0x7fd1c00a4920 2024-06-11T19:09:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.149+0000 7fd1b5ffb640 1 -- 172.21.15.98:0/3295856107 <== mon.3 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd1b0041c00 con 0x7fd1c00a4920 2024-06-11T19:09:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f2931a22640 1 librados: init done 2024-06-11T19:09:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.148+0000 7f29297fa640 4 set_mon_vals no callback set 2024-06-11T19:09:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd1a403fdb0 0x7fd1a4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 1 -- 172.21.15.98:0/3295856107 <== mon.3 v2:172.21.15.117:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fd1b007abf0 con 0x7fd1c00a4920 2024-06-11T19:09:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 10 client.4260.objecter ms_dispatch 0x7fd1c0000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 3 client.4260.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 3 client.4260.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 10 client.4260.objecter checking command 1 2024-06-11T19:09:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] conn(0x7fd1a4045c20 0x7fd1a4048040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 20 client.4260.objecter _get_session s=0x7fd1a4045550 osd=5 3 2024-06-11T19:09:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 20 client.4260.objecter put_session s=0x7fd1a4045550 osd=5 3 2024-06-11T19:09:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 15 client.4260.objecter _session_command_op_remove -1 1 2024-06-11T19:09:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 20 client.4260.objecter _get_session s=0x7fd1a4045550 osd=5 3 2024-06-11T19:09:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 20 client.4260.objecter get_session s=0x7fd1a4045550 osd=5 3 2024-06-11T19:09:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 15 client.4260.objecter _session_command_op_assign 5 1 2024-06-11T19:09:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 20 client.4260.objecter put_session s=0x7fd1a4045550 osd=5 4 2024-06-11T19:09:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 10 client.4260.objecter _send_command 1 2024-06-11T19:09:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 1 -- 172.21.15.98:0/3295856107 --> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd1b0043e50 con 0x7fd1a4045c20 2024-06-11T19:09:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 20 client.4260.objecter dump_active .. 0 homeless 2024-06-11T19:09:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 1 -- 172.21.15.98:0/3295856107 <== mon.3 v2:172.21.15.117:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fd1b0081020 con 0x7fd1c00a4920 2024-06-11T19:09:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1b5ffb640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-06-11T19:09:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7f2913fff640 1 -- 172.21.15.98:0/2217245093 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f292000cc30 con 0x7f292c132040 2024-06-11T19:09:35.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1c564e640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd1a403fdb0 0x7fd1a4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1c5e4f640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] conn(0x7fd1a4045c20 0x7fd1a4048040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7f2911ffb640 10 client.4272.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7f2911ffb640 15 client.4272.objecter _session_command_op_assign -1 1 2024-06-11T19:09:35.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7f2911ffb640 20 client.4272.objecter _get_session osd=-1 returning homeless 2024-06-11T19:09:35.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7f2911ffb640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7f2911ffb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:09:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1c5e4f640 10 monclient: get_auth_request con 0x7fd1a4045c20 auth_method 0 2024-06-11T19:09:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7fd1c564e640 10 monclient: get_auth_request con 0x7fd1a403fdb0 auth_method 0 2024-06-11T19:09:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.150+0000 7f2911ffb640 1 -- 172.21.15.98:0/2217245093 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2904002910 con 0x7f292c132040 2024-06-11T19:09:35.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f291803fde0 0x7f29180422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 1 -- 172.21.15.98:0/2217245093 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f292000f040 con 0x7f292c132040 2024-06-11T19:09:35.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7fd1c564e640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd1a403fdb0 0x7fd1a4042270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc000f60 tx=0x7fd1bc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 10 client.4272.objecter ms_dispatch 0x7f292c000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:35.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 3 client.4272.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:35.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 3 client.4272.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:35.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7fd1b5ffb640 10 client.4260.objecter ms_handle_connect 0x7fd1a403fdb0 2024-06-11T19:09:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 10 client.4272.objecter checking command 1 2024-06-11T19:09:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x7f2918045be0 0x7f2918048000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 20 client.4272.objecter _get_session s=0x7f2918045510 osd=2 3 2024-06-11T19:09:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 20 client.4272.objecter put_session s=0x7f2918045510 osd=2 3 2024-06-11T19:09:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 15 client.4272.objecter _session_command_op_remove -1 1 2024-06-11T19:09:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 20 client.4272.objecter _get_session s=0x7f2918045510 osd=2 3 2024-06-11T19:09:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 20 client.4272.objecter get_session s=0x7f2918045510 osd=2 3 2024-06-11T19:09:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 15 client.4272.objecter _session_command_op_assign 2 1 2024-06-11T19:09:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 20 client.4272.objecter put_session s=0x7f2918045510 osd=2 4 2024-06-11T19:09:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 10 client.4272.objecter _send_command 1 2024-06-11T19:09:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 1 -- 172.21.15.98:0/2217245093 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2920016e70 con 0x7f2918045be0 2024-06-11T19:09:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 20 client.4272.objecter dump_active .. 0 homeless 2024-06-11T19:09:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 1 -- 172.21.15.98:0/2217245093 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f2920058050 con 0x7f292c132040 2024-06-11T19:09:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-06-11T19:09:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7fd1b5ffb640 10 client.4260.objecter ms_handle_connect 0x7fd1a4045c20 2024-06-11T19:09:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f292affd640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f291803fde0 0x7f29180422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7fd1c5e4f640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] conn(0x7fd1a4045c20 0x7fd1a4048040 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f292affd640 10 monclient: get_auth_request con 0x7f291803fde0 auth_method 0 2024-06-11T19:09:35.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f292b7fe640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x7f2918045be0 0x7f2918048000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f292b7fe640 10 monclient: get_auth_request con 0x7f2918045be0 auth_method 0 2024-06-11T19:09:35.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f292b7fe640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x7f2918045be0 0x7f2918048000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.151+0000 7f2913fff640 10 client.4272.objecter ms_handle_connect 0x7f2918045be0 2024-06-11T19:09:35.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.152+0000 7f2913fff640 10 client.4272.objecter ms_handle_connect 0x7f291803fde0 2024-06-11T19:09:35.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.152+0000 7f292affd640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f291803fde0 0x7f29180422a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f291c013180 tx=0x7f291c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.152+0000 7fd1b5ffb640 1 -- 172.21.15.98:0/3295856107 <== osd.5 v2:172.21.15.117:6816/987023138 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd1b0043e50 con 0x7fd1a4045c20 2024-06-11T19:09:35.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.152+0000 7fd1b5ffb640 10 client.4260.objecter ms_dispatch 0x7fd1c0000b90 command_reply(tid 1: 0 ) 2024-06-11T19:09:35.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.152+0000 7fd1b5ffb640 10 client.4260.objecter _finish_command 1 = system:0 2024-06-11T19:09:35.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.152+0000 7fd1b5ffb640 20 client.4260.objecter put_session s=0x7fd1a4045550 osd=5 4 2024-06-11T19:09:35.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.152+0000 7fd1b5ffb640 15 client.4260.objecter _session_command_op_remove 5 1 2024-06-11T19:09:35.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.152+0000 7f2913fff640 1 -- 172.21.15.98:0/2217245093 <== osd.2 v2:172.21.15.98:6808/3751501255 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2920016e70 con 0x7f2918045be0 2024-06-11T19:09:35.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.152+0000 7f2913fff640 10 client.4272.objecter ms_dispatch 0x7f292c000b90 command_reply(tid 1: 0 ) 2024-06-11T19:09:35.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.152+0000 7f2913fff640 10 client.4272.objecter _finish_command 1 = system:0 2024-06-11T19:09:35.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.152+0000 7f2913fff640 20 client.4272.objecter put_session s=0x7f2918045510 osd=2 4 2024-06-11T19:09:35.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.152+0000 7f2913fff640 15 client.4272.objecter _session_command_op_remove 2 1 2024-06-11T19:09:35.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.166+0000 7fd1c78b2640 10 client.4260.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-11T19:09:35.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.166+0000 7fd1c78b2640 15 client.4260.objecter _session_command_op_assign -1 2 2024-06-11T19:09:35.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.166+0000 7fd1c78b2640 20 client.4260.objecter _get_session s=0x7fd1a4045550 osd=5 3 2024-06-11T19:09:35.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.166+0000 7fd1c78b2640 20 client.4260.objecter put_session s=0x7fd1a4045550 osd=5 3 2024-06-11T19:09:35.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.166+0000 7fd1c78b2640 20 client.4260.objecter _get_session s=0x7fd1a4045550 osd=5 3 2024-06-11T19:09:35.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.166+0000 7fd1c78b2640 15 client.4260.objecter _session_command_op_remove -1 2 2024-06-11T19:09:35.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.166+0000 7fd1c78b2640 20 client.4260.objecter get_session s=0x7fd1a4045550 osd=5 3 2024-06-11T19:09:35.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.166+0000 7fd1c78b2640 15 client.4260.objecter _session_command_op_assign 5 2 2024-06-11T19:09:35.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.166+0000 7fd1c78b2640 20 client.4260.objecter put_session s=0x7fd1a4045550 osd=5 4 2024-06-11T19:09:35.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.166+0000 7fd1c78b2640 10 client.4260.objecter _send_command 2 2024-06-11T19:09:35.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.166+0000 7fd1c78b2640 1 -- 172.21.15.98:0/3295856107 --> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd1c00aad80 con 0x7fd1a4045c20 2024-06-11T19:09:35.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 10 client.4272.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-11T19:09:35.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 15 client.4272.objecter _session_command_op_assign -1 2 2024-06-11T19:09:35.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 20 client.4272.objecter _get_session s=0x7f2918045510 osd=2 3 2024-06-11T19:09:35.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 20 client.4272.objecter put_session s=0x7f2918045510 osd=2 3 2024-06-11T19:09:35.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 20 client.4272.objecter _get_session s=0x7f2918045510 osd=2 3 2024-06-11T19:09:35.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 15 client.4272.objecter _session_command_op_remove -1 2 2024-06-11T19:09:35.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 20 client.4272.objecter get_session s=0x7f2918045510 osd=2 3 2024-06-11T19:09:35.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 15 client.4272.objecter _session_command_op_assign 2 2 2024-06-11T19:09:35.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 20 client.4272.objecter put_session s=0x7f2918045510 osd=2 4 2024-06-11T19:09:35.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 10 client.4272.objecter _send_command 2 2024-06-11T19:09:35.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 1 -- 172.21.15.98:0/2217245093 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2904001220 con 0x7f2918045be0 2024-06-11T19:09:35.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1b5ffb640 1 -- 172.21.15.98:0/3295856107 <== osd.5 v2:172.21.15.117:6816/987023138 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd1c00aad80 con 0x7fd1a4045c20 2024-06-11T19:09:35.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1b5ffb640 10 client.4260.objecter ms_dispatch 0x7fd1c0000b90 command_reply(tid 2: 0 ) 2024-06-11T19:09:35.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1b5ffb640 10 client.4260.objecter _finish_command 2 = system:0 2024-06-11T19:09:35.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1b5ffb640 20 client.4260.objecter put_session s=0x7fd1a4045550 osd=5 4 2024-06-11T19:09:35.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1b5ffb640 15 client.4260.objecter _session_command_op_remove 5 2 2024-06-11T19:09:35.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2913fff640 1 -- 172.21.15.98:0/2217245093 <== osd.2 v2:172.21.15.98:6808/3751501255 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2904001220 con 0x7f2918045be0 2024-06-11T19:09:35.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2913fff640 10 client.4272.objecter ms_dispatch 0x7f292c000b90 command_reply(tid 2: 0 ) 2024-06-11T19:09:35.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2913fff640 10 client.4272.objecter _finish_command 2 = system:0 2024-06-11T19:09:35.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2913fff640 20 client.4272.objecter put_session s=0x7f2918045510 osd=2 4 2024-06-11T19:09:35.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2913fff640 15 client.4272.objecter _session_command_op_remove 2 2 2024-06-11T19:09:35.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 10 librados: watch_flush enter 2024-06-11T19:09:35.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1c78b2640 10 librados: watch_flush enter 2024-06-11T19:09:35.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1c78b2640 10 librados: watch_flush exit 2024-06-11T19:09:35.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1c78b2640 10 client.4260.objecter close_session for osd.5 2024-06-11T19:09:35.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1c78b2640 1 -- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] conn(0x7fd1a4045c20 msgr2=0x7fd1a4048040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1c78b2640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] conn(0x7fd1a4045c20 0x7fd1a4048040 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1c78b2640 20 client.4260.objecter put_session s=0x7fd1a4045550 osd=5 1 2024-06-11T19:09:35.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1c78b2640 20 client.4260.objecter shutdown clearing up homeless session... 2024-06-11T19:09:35.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 10 librados: watch_flush exit 2024-06-11T19:09:35.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7fd1c78b2640 10 client.4260.objecter successfully canceled tick 2024-06-11T19:09:35.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 10 client.4272.objecter close_session for osd.2 2024-06-11T19:09:35.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 1 -- 172.21.15.98:0/2217245093 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x7f2918045be0 msgr2=0x7f2918048000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x7f2918045be0 0x7f2918048000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 20 client.4272.objecter put_session s=0x7f2918045510 osd=2 1 2024-06-11T19:09:35.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 20 client.4272.objecter shutdown clearing up homeless session... 2024-06-11T19:09:35.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.167+0000 7f2911ffb640 10 client.4272.objecter successfully canceled tick 2024-06-11T19:09:35.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7f2911ffb640 1 -- 172.21.15.98:0/2217245093 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f291803fde0 msgr2=0x7f29180422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7fd1c78b2640 1 -- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd1a403fdb0 msgr2=0x7fd1a4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7fd1c78b2640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd1a403fdb0 0x7fd1a4042270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc000f60 tx=0x7fd1bc009040 comp rx=0 tx=0).stop 2024-06-11T19:09:35.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7fd1c78b2640 10 monclient: shutdown 2024-06-11T19:09:35.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7fd1c78b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7fd1c78b2640 1 -- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1c00a4920 msgr2=0x7fd1c009d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7fd1c78b2640 1 --2- 172.21.15.98:0/3295856107 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1c00a4920 0x7fd1c009d640 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0033000 tx=0x7fd1b00046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7fd1c5e4f640 1 -- 172.21.15.98:0/3295856107 reap_dead start 2024-06-11T19:09:35.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7f2911ffb640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f291803fde0 0x7f29180422a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f291c013180 tx=0x7f291c008040 comp rx=0 tx=0).stop 2024-06-11T19:09:35.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7f2911ffb640 10 monclient: shutdown 2024-06-11T19:09:35.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7f2911ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7f2911ffb640 1 -- 172.21.15.98:0/2217245093 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f292c132040 msgr2=0x7f292c1c9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7f2911ffb640 1 --2- 172.21.15.98:0/2217245093 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f292c132040 0x7f292c1c9760 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f292000ebe0 tx=0x7f292000c520 comp rx=0 tx=0).stop 2024-06-11T19:09:35.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7f292b7fe640 1 -- 172.21.15.98:0/2217245093 reap_dead start 2024-06-11T19:09:35.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7fd1c78b2640 1 -- 172.21.15.98:0/3295856107 shutdown_connections 2024-06-11T19:09:35.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7fd1c78b2640 1 -- 172.21.15.98:0/3295856107 >> 172.21.15.98:0/3295856107 conn(0x7fd1c00ab850 msgr2=0x7fd1c00a9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7f2911ffb640 1 -- 172.21.15.98:0/2217245093 shutdown_connections 2024-06-11T19:09:35.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7f2911ffb640 1 -- 172.21.15.98:0/2217245093 >> 172.21.15.98:0/2217245093 conn(0x7f292c0a7850 msgr2=0x7f292c0a21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7fd1c78b2640 1 -- 172.21.15.98:0/3295856107 shutdown_connections 2024-06-11T19:09:35.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7fd1c78b2640 1 -- 172.21.15.98:0/3295856107 wait complete. 2024-06-11T19:09:35.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7f2911ffb640 1 -- 172.21.15.98:0/2217245093 shutdown_connections 2024-06-11T19:09:35.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7f2911ffb640 1 -- 172.21.15.98:0/2217245093 wait complete. 2024-06-11T19:09:35.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7fd1c78b2640 1 librados: shutdown 2024-06-11T19:09:35.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.168+0000 7f2911ffb640 1 librados: shutdown 2024-06-11T19:09:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.181+0000 7f37c2bc6640 10 monclient: get_monmap_and_config 2024-06-11T19:09:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.181+0000 7f37c2bc6640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.181+0000 7f37c2bc6640 10 monclient: monmap: 2024-06-11T19:09:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:35.182583+0000 2024-06-11T19:09:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:35.182583+0000 2024-06-11T19:09:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:35.245 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.246 INFO:teuthology.orchestra.run.smithi098.stdout:51539607556 2024-06-11T19:09:35.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-06-11T19:09:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.182+0000 7f37c2bc6640 1 Processor -- start 2024-06-11T19:09:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.182+0000 7f37c2bc6640 1 -- start start 2024-06-11T19:09:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.182+0000 7f65051c6640 10 monclient: get_monmap_and_config 2024-06-11T19:09:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.182+0000 7f65051c6640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.182+0000 7f65051c6640 10 monclient: monmap: 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:35.183581+0000 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:35.183581+0000 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.249 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.249 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.249 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:35.249 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:35.249 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.182+0000 7f37c2bc6640 10 monclient: init 2024-06-11T19:09:35.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 10 monclient: _add_conns ranks=[9,2,8,3,0,5,1,7,6,10,4] 2024-06-11T19:09:35.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f37bc0a4920 0x7f37bc0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.250 INFO:teuthology.orchestra.run.smithi098.stdout:47244640261 2024-06-11T19:09:35.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-06-11T19:09:35.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 10 monclient(hunting): picked mon.noname-a con 0x7f37bc0a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:35.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f37bc0a5240 0x7f37bc0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 10 monclient(hunting): picked mon.noname-b con 0x7f37bc0a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:35.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f37bc0a2530 0x7f37bc0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f65051c6640 1 Processor -- start 2024-06-11T19:09:35.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 10 monclient(hunting): picked mon.noname-e con 0x7f37bc0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:35.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f37bc0a7140 con 0x7f37bc0a4920 2024-06-11T19:09:35.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f37bc0a2b50 con 0x7f37bc0a5240 2024-06-11T19:09:35.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f37bc0a2e30 con 0x7f37bc0a2530 2024-06-11T19:09:35.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c2bc6640 10 monclient(hunting): authenticate will time out at 784.990246s 2024-06-11T19:09:35.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f65051c6640 1 -- start start 2024-06-11T19:09:35.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f65051c6640 10 monclient: init 2024-06-11T19:09:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c0962640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f37bc0a4920 0x7f37bc0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c1163640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f37bc0a2530 0x7f37bc0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f37bc0a5240 0x7f37bc0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c0962640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f37bc0a4920 0x7f37bc0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45664/0 (socket says 172.21.15.98:45664) 2024-06-11T19:09:35.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c0962640 1 -- 172.21.15.98:0/746193012 learned_addr learned my addr 172.21.15.98:0/746193012 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c0962640 10 monclient(hunting): get_auth_request con 0x7f37bc0a4920 auth_method 0 2024-06-11T19:09:35.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c0962640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c0962640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c0962640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c1163640 10 monclient(hunting): get_auth_request con 0x7f37bc0a2530 auth_method 0 2024-06-11T19:09:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c1163640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c1163640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c1163640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 10 monclient(hunting): get_auth_request con 0x7f37bc0a5240 auth_method 0 2024-06-11T19:09:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c1163640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c1163640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c1163640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c0962640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c0962640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c0962640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 1 -- 172.21.15.98:0/746193012 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f37bc0a2530 msgr2=0x7f37bc0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 1 --2- 172.21.15.98:0/746193012 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f37bc0a2530 0x7f37bc0a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 1 -- 172.21.15.98:0/746193012 >> v2:172.21.15.98:3300/0 conn(0x7f37bc0a4920 msgr2=0x7f37bc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 1 --2- 172.21.15.98:0/746193012 >> v2:172.21.15.98:3300/0 conn(0x7f37bc0a4920 0x7f37bc0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 1 monclient: found mon.noname-b 2024-06-11T19:09:35.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37bbfff640 1 -- 172.21.15.98:0/746193012 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37bc0a3110 con 0x7f37bc0a5240 2024-06-11T19:09:35.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f37c1163640 1 --2- 172.21.15.98:0/746193012 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f37bc0a2530 0x7f37bc0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f65051c6640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f65051c6640 10 monclient: _add_conns ranks=[8,3,0,6,5,9,2,1,4,10,7] 2024-06-11T19:09:35.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f65051c6640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f65000a0d10 0x7f65000a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f65051c6640 10 monclient(hunting): picked mon.noname-e con 0x7f65000a0d10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:35.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f65051c6640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6500138c80 0x7f6500139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.183+0000 7f65051c6640 10 monclient(hunting): picked mon.noname-g con 0x7f6500138c80 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:35.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f65051c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65000a22d0 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:35.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f65051c6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6500076a00 con 0x7f65000a22d0 2024-06-11T19:09:35.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f65051c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64ff59e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f65000a22d0 0x7f65000a26d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53438/0 (socket says 172.21.15.98:53438) 2024-06-11T19:09:35.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64ff59e640 1 -- 172.21.15.98:0/2475355878 learned_addr learned my addr 172.21.15.98:0/2475355878 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:09:35.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 1 --2- 172.21.15.98:0/2475355878 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f65000a0d10 0x7f65000a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f65051c6640 1 -- 172.21.15.98:0/2475355878 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f650009f010 con 0x7f6500138c80 2024-06-11T19:09:35.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f65051c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f65051c6640 1 -- 172.21.15.98:0/2475355878 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f650009ed30 con 0x7f65000a0d10 2024-06-11T19:09:35.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f65051c6640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f65051c6640 10 monclient(hunting): authenticate will time out at 784.991221s 2024-06-11T19:09:35.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 10 monclient(hunting): get_auth_request con 0x7f65000a0d10 auth_method 0 2024-06-11T19:09:35.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fe59c640 1 --2- 172.21.15.98:0/2475355878 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6500138c80 0x7f6500139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fe59c640 10 monclient(hunting): get_auth_request con 0x7f6500138c80 auth_method 0 2024-06-11T19:09:35.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fe59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fdd9b640 1 -- 172.21.15.98:0/2475355878 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1050173778 0 0) 0x7f6500076a00 con 0x7f65000a22d0 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fdd9b640 10 monclient(hunting): _negotiate 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fdd9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fdd9b640 10 monclient(hunting): my global_id is 4293 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fdd9b640 1 -- 172.21.15.98:0/2475355878 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64e8003520 con 0x7f65000a22d0 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 1 -- 172.21.15.98:0/2475355878 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6500138c80 msgr2=0x7f6500139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 1 --2- 172.21.15.98:0/2475355878 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6500138c80 0x7f6500139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 1 -- 172.21.15.98:0/2475355878 >> v1:172.21.15.98:6789/0 conn(0x7f65000a22d0 legacy=0x7f65000a26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 1 monclient: found mon.noname-e 2024-06-11T19:09:35.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:35.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f64fed9d640 1 -- 172.21.15.98:0/2475355878 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65000a1630 con 0x7f65000a0d10 2024-06-11T19:09:35.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.184+0000 7f37bbfff640 1 --2- 172.21.15.98:0/746193012 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f37bc0a5240 0x7f37bc0a59a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f37a80097a0 tx=0x7f37a802b6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a6d06926de44e7 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 1 -- 172.21.15.98:0/746193012 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f37a8037070 con 0x7f37bc0a5240 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 10 monclient: dump: 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.268 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37c2bc6640 5 monclient: authenticate success, global_id 4277 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 1 -- 172.21.15.98:0/746193012 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37a8030980 con 0x7f37bc0a5240 2024-06-11T19:09:35.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37c2bc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37c2bc6640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 1 -- 172.21.15.98:0/746193012 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f37a8030bb0 con 0x7f37bc0a5240 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37c2bc6640 4 set_mon_vals no callback set 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f37bb7fe640 10 monclient: dump: 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.270 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.271 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.271 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.271 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.271 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.271 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.271 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.271 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.271 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.271 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.271 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f64fed9d640 1 --2- 172.21.15.98:0/2475355878 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f65000a0d10 0x7f65000a10f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f64f00097d0 tx=0x7f64f002b790 comp rx=0 tx=0).ready entity=mon.8 client_cookie=c13d12dc9c519ff1 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f64fdd9b640 1 -- 172.21.15.98:0/2475355878 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f64f0039070 con 0x7f65000a0d10 2024-06-11T19:09:35.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f64fdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f64fdd9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:35.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f64fdd9b640 10 monclient: dump: 2024-06-11T19:09:35.272 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.273 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f64fdd9b640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f64fdd9b640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f64fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f65051c6640 5 monclient: authenticate success, global_id 4301 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.185+0000 7f65051c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f64fdd9b640 1 -- 172.21.15.98:0/2475355878 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64f00314f0 con 0x7f65000a0d10 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f64fdd9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f65051c6640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f65051c6640 4 set_mon_vals no callback set 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f64fdd9b640 1 -- 172.21.15.98:0/2475355878 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f64f0039070 con 0x7f65000a0d10 2024-06-11T19:09:35.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f64fdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 10 monclient: shutdown 2024-06-11T19:09:35.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f64fdd9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:35.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f64fdd9b640 10 monclient: dump: 2024-06-11T19:09:35.275 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.275 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.275 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.275 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.275 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.276 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.276 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.276 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.276 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.276 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.276 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.276 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.276 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.276 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.276 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.276 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 1 -- 172.21.15.98:0/746193012 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f37bc0a5240 msgr2=0x7f37bc0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 1 --2- 172.21.15.98:0/746193012 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f37bc0a5240 0x7f37bc0a59a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f37a80097a0 tx=0x7f37a802b6f0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f5915f5a640 10 monclient: get_monmap_and_config 2024-06-11T19:09:35.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f5915f5a640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 1 -- 172.21.15.98:0/746193012 shutdown_connections 2024-06-11T19:09:35.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 1 --2- 172.21.15.98:0/746193012 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f37bc0a2530 0x7f37bc0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 1 --2- 172.21.15.98:0/746193012 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f37bc0a5240 0x7f37bc0a59a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 1 --2- 172.21.15.98:0/746193012 >> v2:172.21.15.98:3300/0 conn(0x7f37bc0a4920 0x7f37bc0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f5915f5a640 10 monclient: monmap: 2024-06-11T19:09:35.278 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.278 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.278 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:35.187275+0000 2024-06-11T19:09:35.278 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:35.187275+0000 2024-06-11T19:09:35.278 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.278 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.278 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:35.278 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:35.279 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.279 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.279 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.279 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.279 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.279 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.279 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.279 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:35.279 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:35.279 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 1 -- 172.21.15.98:0/746193012 >> 172.21.15.98:0/746193012 conn(0x7f37bc0ab850 msgr2=0x7f37bc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 1 -- 172.21.15.98:0/746193012 shutdown_connections 2024-06-11T19:09:35.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 1 -- 172.21.15.98:0/746193012 wait complete. 2024-06-11T19:09:35.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f92b2e69640 10 monclient: get_monmap_and_config 2024-06-11T19:09:35.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f92b2e69640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f92b2e69640 10 monclient: monmap: 2024-06-11T19:09:35.280 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.280 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.280 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:35.187402+0000 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:35.187402+0000 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:35.282 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:35.282 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f65051c6640 10 monclient: shutdown 2024-06-11T19:09:35.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f65051c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f65051c6640 1 -- 172.21.15.98:0/2475355878 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f65000a0d10 msgr2=0x7f65000a10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f37c2bc6640 10 monclient: monmap: 2024-06-11T19:09:35.283 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.283 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:35.283 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:35.283 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:35.283 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.283 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:35.283 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:35.284 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:35.284 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.284 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.284 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.284 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.284 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.284 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.284 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.284 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.186+0000 7f65051c6640 1 --2- 172.21.15.98:0/2475355878 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f65000a0d10 0x7f65000a10f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f64f00097d0 tx=0x7f64f002b790 comp rx=0 tx=0).stop 2024-06-11T19:09:35.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 1 Processor -- start 2024-06-11T19:09:35.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f65051c6640 1 -- 172.21.15.98:0/2475355878 shutdown_connections 2024-06-11T19:09:35.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f65051c6640 1 --2- 172.21.15.98:0/2475355878 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6500138c80 0x7f6500139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 1 -- start start 2024-06-11T19:09:35.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 10 monclient: init 2024-06-11T19:09:35.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f65051c6640 1 --2- 172.21.15.98:0/2475355878 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f65000a0d10 0x7f65000a10f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f37c2bc6640 1 librados: starting msgr at 2024-06-11T19:09:35.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f37c2bc6640 1 librados: starting objecter 2024-06-11T19:09:35.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f65051c6640 1 -- 172.21.15.98:0/2475355878 >> 172.21.15.98:0/2475355878 conn(0x7f65000a7850 msgr2=0x7f65000a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f5915f5a640 1 Processor -- start 2024-06-11T19:09:35.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f65051c6640 1 -- 172.21.15.98:0/2475355878 shutdown_connections 2024-06-11T19:09:35.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f37c2bc6640 1 Processor -- start 2024-06-11T19:09:35.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f65051c6640 1 -- 172.21.15.98:0/2475355878 wait complete. 2024-06-11T19:09:35.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f5915f5a640 1 -- start start 2024-06-11T19:09:35.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f5915f5a640 10 monclient: init 2024-06-11T19:09:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f37c2bc6640 1 -- start start 2024-06-11T19:09:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f37c2bc6640 1 librados: setting wanted keys 2024-06-11T19:09:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f37c2bc6640 1 librados: calling monclient init 2024-06-11T19:09:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f37c2bc6640 10 monclient: init 2024-06-11T19:09:35.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 10 monclient: _add_conns ranks=[7,9,2,1,8,0,10,6,4,3,5] 2024-06-11T19:09:35.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f92ac0a52a0 0x7f92ac0a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 10 monclient(hunting): picked mon.noname-i con 0x7f92ac0a52a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:35.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92ac09d520 0x7f92ac0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 10 monclient(hunting): picked mon.noname-a con 0x7f92ac09d520 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:35.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92ac073830 0x7f92ac0a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 10 monclient(hunting): picked mon.noname-b con 0x7f92ac073830 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:35.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f92ac09daa0 con 0x7f92ac09d520 2024-06-11T19:09:35.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f92ac09d2e0 con 0x7f92ac073830 2024-06-11T19:09:35.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 10 monclient: _add_conns ranks=[0,4,7,6,8,3,5,2,1] 2024-06-11T19:09:35.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37bc0a2530 0x7f37bc1cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 10 monclient(hunting): picked mon.noname-a con 0x7f37bc0a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f37bc0a4920 0x7f37bc1cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f92ac0a5680 con 0x7f92ac0a52a0 2024-06-11T19:09:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b2e69640 10 monclient(hunting): authenticate will time out at 784.994776s 2024-06-11T19:09:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.187+0000 7f92b0c05640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f92ac0a52a0 0x7f92ac0a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92abfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92ac09d520 0x7f92ac0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f92ac0a52a0 0x7f92ac0a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:52900/0 (socket says 172.21.15.98:52900) 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 1 -- 172.21.15.98:0/3929789249 learned_addr learned my addr 172.21.15.98:0/3929789249 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 10 monclient(hunting): get_auth_request con 0x7f92ac0a52a0 auth_method 0 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92abfff640 10 monclient(hunting): get_auth_request con 0x7f92ac09d520 auth_method 0 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92abfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92abfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b1406640 1 --2- 172.21.15.98:0/3929789249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92ac073830 0x7f92ac0a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 10 monclient(hunting): picked mon.noname-c con 0x7f37bc0a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f37bc0a5240 0x7f37bc124f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 10 monclient(hunting): picked mon.noname-i con 0x7f37bc0a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc0a6da0 con 0x7f37bc0a2530 2024-06-11T19:09:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f37bc138e50 con 0x7f37bc0a4920 2024-06-11T19:09:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f37bc0a6420 con 0x7f37bc0a5240 2024-06-11T19:09:35.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c2bc6640 10 monclient(hunting): authenticate will time out at 784.995289s 2024-06-11T19:09:35.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b1406640 10 monclient(hunting): get_auth_request con 0x7f92ac073830 auth_method 0 2024-06-11T19:09:35.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b1406640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b1406640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b1406640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 10 monclient: _add_conns ranks=[1,3,4,9,2,10,6,0,5,7,8] 2024-06-11T19:09:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37bbfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f37bc0a4920 0x7f37bc1cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37bbfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f37bc0a4920 0x7f37bc1cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:40340/0 (socket says 172.21.15.98:40340) 2024-06-11T19:09:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37bbfff640 1 -- 172.21.15.98:0/1017726184 learned_addr learned my addr 172.21.15.98:0/1017726184 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37bbfff640 10 monclient(hunting): get_auth_request con 0x7f37bc0a4920 auth_method 0 2024-06-11T19:09:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37bbfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f59100733a0 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:35.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c0962640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37bc0a2530 0x7f37bc1cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5910135990 0x7f591013a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b1406640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b1406640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b1406640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 10 monclient(hunting): handle_auth_done global_id 4318 payload 274 2024-06-11T19:09:35.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 1 -- 172.21.15.98:0/3929789249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92ac073830 msgr2=0x7f92ac0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 1 --2- 172.21.15.98:0/3929789249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92ac073830 0x7f92ac0a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c0962640 10 monclient(hunting): get_auth_request con 0x7f37bc0a2530 auth_method 0 2024-06-11T19:09:35.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c0962640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c0962640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c0962640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 1 -- 172.21.15.98:0/3929789249 >> v2:172.21.15.98:3300/0 conn(0x7f92ac09d520 msgr2=0x7f92ac0a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c1163640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f37bc0a5240 0x7f37bc124f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 10 monclient(hunting): picked mon.noname-g con 0x7f5910135990 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:35.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c1163640 10 monclient(hunting): get_auth_request con 0x7f37bc0a5240 auth_method 0 2024-06-11T19:09:35.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c1163640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c1163640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c1163640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f65051c6640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f591013ad40 0x7f591013d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 1 --2- 172.21.15.98:0/3929789249 >> v2:172.21.15.98:3300/0 conn(0x7f92ac09d520 0x7f92ac0a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 1 monclient: found mon.noname-i 2024-06-11T19:09:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:09:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f92b0c05640 1 -- 172.21.15.98:0/3929789249 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92ac09e9d0 con 0x7f92ac0a52a0 2024-06-11T19:09:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 10 monclient(hunting): picked mon.noname-c con 0x7f591013ad40 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f65051c6640 10 monclient: monmap: 2024-06-11T19:09:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.304 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5914f58640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f59100733a0 0x7f5910073780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:40514/0 (socket says 172.21.15.98:40514) 2024-06-11T19:09:35.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5910073e90 con 0x7f59100733a0 2024-06-11T19:09:35.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5914f58640 1 -- 172.21.15.98:0/2075572368 learned_addr learned my addr 172.21.15.98:0/2075572368 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:09:35.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 1 -- 172.21.15.98:0/2075572368 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f59100705a0 con 0x7f5910135990 2024-06-11T19:09:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 1 -- 172.21.15.98:0/2075572368 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5910135db0 con 0x7f591013ad40 2024-06-11T19:09:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f590ffff640 1 --2- 172.21.15.98:0/2075572368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5910135990 0x7f591013a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c0962640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c0962640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c0962640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915759640 1 --2- 172.21.15.98:0/2075572368 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f591013ad40 0x7f591013d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915f5a640 10 monclient(hunting): authenticate will time out at 784.995748s 2024-06-11T19:09:35.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f590ffff640 10 monclient(hunting): get_auth_request con 0x7f5910135990 auth_method 0 2024-06-11T19:09:35.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f590ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c1163640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c1163640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c1163640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f37c0962640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-06-11T19:09:35.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f590ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f590ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f65051c6640 1 librados: starting msgr at 2024-06-11T19:09:35.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f65051c6640 1 librados: starting objecter 2024-06-11T19:09:35.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c0962640 1 -- 172.21.15.98:0/1017726184 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f37bc0a5240 msgr2=0x7f37bc124f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c0962640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f37bc0a5240 0x7f37bc124f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c0962640 1 -- 172.21.15.98:0/1017726184 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f37bc0a4920 msgr2=0x7f37bc1cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c0962640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f37bc0a4920 0x7f37bc1cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c0962640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c0962640 1 monclient: found mon.noname-a 2024-06-11T19:09:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c0962640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c0962640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c0962640 1 -- 172.21.15.98:0/1017726184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37bc128b40 con 0x7f37bc0a2530 2024-06-11T19:09:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915759640 10 monclient(hunting): get_auth_request con 0x7f591013ad40 auth_method 0 2024-06-11T19:09:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915759640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915759640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f5915759640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c0962640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37bc0a2530 0x7f37bc1cdaa0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f37b000e850 tx=0x7f37b000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 10 client.?.objecter ms_handle_connect 0x7f37bc0a2530 2024-06-11T19:09:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f590ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f65051c6640 1 Processor -- start 2024-06-11T19:09:35.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f590ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.188+0000 7f590ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 1 -- 172.21.15.98:0/1017726184 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f37b0017070 con 0x7f37bc0a2530 2024-06-11T19:09:35.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:35.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f65051c6640 1 -- start start 2024-06-11T19:09:35.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f65051c6640 1 librados: setting wanted keys 2024-06-11T19:09:35.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92b0c05640 1 --2- 172.21.15.98:0/3929789249 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f92ac0a52a0 0x7f92ac0a1910 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f92a00098a0 tx=0x7f92a002b7c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=a2867345c7dc967d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f65051c6640 1 librados: calling monclient init 2024-06-11T19:09:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 10 monclient: dump: 2024-06-11T19:09:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.314 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 1 -- 172.21.15.98:0/1017726184 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b000cdb0 con 0x7f37bc0a2530 2024-06-11T19:09:35.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f65051c6640 10 monclient: init 2024-06-11T19:09:35.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c2bc6640 5 monclient: authenticate success, global_id 4311 2024-06-11T19:09:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c2bc6640 10 monclient: _renew_subs 2024-06-11T19:09:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c2bc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c2bc6640 1 -- 172.21.15.98:0/1017726184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37bc128e20 con 0x7f37bc0a2530 2024-06-11T19:09:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 1 -- 172.21.15.98:0/1017726184 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f37b000eea0 con 0x7f37bc0a2530 2024-06-11T19:09:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37baffd640 4 set_mon_vals no callback set 2024-06-11T19:09:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c2bc6640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37b97fa640 10 monclient: dump: 2024-06-11T19:09:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c2bc6640 10 monclient: _renew_subs 2024-06-11T19:09:35.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c2bc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:35.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c2bc6640 1 -- 172.21.15.98:0/1017726184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37bc0a8da0 con 0x7f37bc0a2530 2024-06-11T19:09:35.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f37c2bc6640 1 librados: init done 2024-06-11T19:09:35.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590f7fe640 1 -- 172.21.15.98:0/2075572368 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3312636232 0 0) 0x7f5910073e90 con 0x7f59100733a0 2024-06-11T19:09:35.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 1 -- 172.21.15.98:0/3929789249 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92a0038070 con 0x7f92ac0a52a0 2024-06-11T19:09:35.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590f7fe640 10 monclient(hunting): _negotiate 2024-06-11T19:09:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:09:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 10 monclient: dump: 2024-06-11T19:09:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 1 -- 172.21.15.98:0/3929789249 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92a0031990 con 0x7f92ac0a52a0 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92b2e69640 5 monclient: authenticate success, global_id 4318 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 1 -- 172.21.15.98:0/3929789249 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92a00323b0 con 0x7f92ac0a52a0 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92b2e69640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92b2e69640 4 set_mon_vals no callback set 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f92ab7fe640 10 monclient: dump: 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.322 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590f7fe640 10 monclient(hunting): my global_id is 4276 2024-06-11T19:09:35.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590f7fe640 1 -- 172.21.15.98:0/2075572368 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58f8007930 con 0x7f59100733a0 2024-06-11T19:09:35.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590ffff640 10 monclient(hunting): handle_auth_done global_id 4278 payload 274 2024-06-11T19:09:35.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590ffff640 1 -- 172.21.15.98:0/2075572368 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f591013ad40 msgr2=0x7f591013d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590ffff640 1 --2- 172.21.15.98:0/2075572368 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f591013ad40 0x7f591013d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590ffff640 1 -- 172.21.15.98:0/2075572368 >> v1:172.21.15.117:6789/0 conn(0x7f59100733a0 legacy=0x7f5910073780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 10 monclient: _add_conns ranks=[1,4,2,0,7,3,5,6,8] 2024-06-11T19:09:35.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65000a0d10 0x7f65001cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 10 monclient(hunting): picked mon.noname-f con 0x7f65000a0d10 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:35.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f65000a22d0 0x7f65001ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590ffff640 1 monclient: found mon.noname-g 2024-06-11T19:09:35.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f37c2bc6640 10 client.4311.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f37c2bc6640 15 client.4311.objecter _session_command_op_assign -1 1 2024-06-11T19:09:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 10 monclient(hunting): picked mon.noname-c con 0x7f65000a22d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f37c2bc6640 20 client.4311.objecter _get_session osd=-1 returning homeless 2024-06-11T19:09:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f37c2bc6640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f37c2bc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6500138c80 0x7f6500130370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f37c2bc6640 1 -- 172.21.15.98:0/1017726184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f37bc126ac0 con 0x7f37bc0a2530 2024-06-11T19:09:35.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f92b2e69640 10 monclient: shutdown 2024-06-11T19:09:35.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f92b2e69640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f92b2e69640 1 -- 172.21.15.98:0/3929789249 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f92ac0a52a0 msgr2=0x7f92ac0a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f64fed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65000a0d10 0x7f65001cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f92b2e69640 1 --2- 172.21.15.98:0/3929789249 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f92ac0a52a0 0x7f92ac0a1910 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f92a00098a0 tx=0x7f92a002b7c0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 10 monclient(hunting): picked mon.noname-b con 0x7f6500138c80 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:35.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590ffff640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:09:35.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f65000a2de0 con 0x7f65000a0d10 2024-06-11T19:09:35.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f64ff59e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6500138c80 0x7f6500130370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f64fe59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f65000a22d0 0x7f65001ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f64ff59e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6500138c80 0x7f6500130370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47926/0 (socket says 172.21.15.98:47926) 2024-06-11T19:09:35.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f64ff59e640 1 -- 172.21.15.98:0/4023514578 learned_addr learned my addr 172.21.15.98:0/4023514578 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f92b2e69640 1 -- 172.21.15.98:0/3929789249 shutdown_connections 2024-06-11T19:09:35.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f65000a3d70 con 0x7f6500138c80 2024-06-11T19:09:35.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f92b2e69640 1 --2- 172.21.15.98:0/3929789249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92ac073830 0x7f92ac0a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f92b2e69640 1 --2- 172.21.15.98:0/3929789249 >> v2:172.21.15.98:3300/0 conn(0x7f92ac09d520 0x7f92ac0a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f92b2e69640 1 --2- 172.21.15.98:0/3929789249 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f92ac0a52a0 0x7f92ac0a1910 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f65051c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f92b2e69640 1 -- 172.21.15.98:0/3929789249 >> 172.21.15.98:0/3929789249 conn(0x7f92ac0a4780 msgr2=0x7f92ac0a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f65000a2f60 con 0x7f65000a22d0 2024-06-11T19:09:35.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f65051c6640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f92b2e69640 1 -- 172.21.15.98:0/3929789249 shutdown_connections 2024-06-11T19:09:35.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f65051c6640 10 monclient(hunting): authenticate will time out at 784.998065s 2024-06-11T19:09:35.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f92b2e69640 1 -- 172.21.15.98:0/3929789249 wait complete. 2024-06-11T19:09:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 10 monclient(hunting): get_auth_request con 0x7f6500138c80 auth_method 0 2024-06-11T19:09:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fe59c640 10 monclient(hunting): get_auth_request con 0x7f65000a22d0 auth_method 0 2024-06-11T19:09:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fe59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fed9d640 10 monclient(hunting): get_auth_request con 0x7f65000a0d10 auth_method 0 2024-06-11T19:09:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.189+0000 7f590ffff640 1 -- 172.21.15.98:0/2075572368 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59100a3810 con 0x7f5910135990 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590ffff640 1 --2- 172.21.15.98:0/2075572368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5910135990 0x7f591013a800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f590000bdc0 tx=0x7f5900033f90 comp rx=0 tx=0).ready entity=mon.3 client_cookie=f64dcbe001d45d88 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 1 -- 172.21.15.98:0/2075572368 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5900041070 con 0x7f5910135990 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 10 monclient: dump: 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.336 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.337 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.337 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.337 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.337 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.337 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.337 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.337 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.337 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f37b97fa640 1 -- 172.21.15.98:0/1017726184 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f37b001a070 con 0x7f37bc0a2530 2024-06-11T19:09:35.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 1 -- 172.21.15.98:0/2075572368 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5900034b40 con 0x7f5910135990 2024-06-11T19:09:35.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-06-11T19:09:35.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 1 -- 172.21.15.98:0/4023514578 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f65000a22d0 msgr2=0x7f65001ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 1 --2- 172.21.15.98:0/4023514578 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f65000a22d0 0x7f65001ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 1 -- 172.21.15.98:0/4023514578 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65000a0d10 msgr2=0x7f65001cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 1 --2- 172.21.15.98:0/4023514578 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65000a0d10 0x7f65001cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 1 monclient: found mon.noname-b 2024-06-11T19:09:35.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 1 -- 172.21.15.98:0/4023514578 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65001328f0 con 0x7f6500138c80 2024-06-11T19:09:35.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f92b2e69640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f5915f5a640 5 monclient: authenticate success, global_id 4278 2024-06-11T19:09:35.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f92b2e69640 10 monclient: monmap: 2024-06-11T19:09:35.341 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.341 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:35.341 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:35.341 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:35.341 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.341 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:35.341 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:35.341 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:35.342 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.342 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.342 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.342 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.342 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.342 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.342 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.342 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f5915f5a640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:35.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f5915f5a640 4 set_mon_vals no callback set 2024-06-11T19:09:35.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64ff59e640 1 --2- 172.21.15.98:0/4023514578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6500138c80 0x7f6500130370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f64f800cb40 tx=0x7f64f800a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64e77fe640 10 client.?.objecter ms_handle_connect 0x7f6500138c80 2024-06-11T19:09:35.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64e77fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 1 -- 172.21.15.98:0/2075572368 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5900041070 con 0x7f5910135990 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.190+0000 7f590f7fe640 10 monclient: dump: 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.344 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f5915f5a640 10 monclient: shutdown 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f5915f5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f5915f5a640 1 -- 172.21.15.98:0/2075572368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5910135990 msgr2=0x7f591013a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f5915f5a640 1 --2- 172.21.15.98:0/2075572368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5910135990 0x7f591013a800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f590000bdc0 tx=0x7f5900033f90 comp rx=0 tx=0).stop 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f5915f5a640 1 -- 172.21.15.98:0/2075572368 shutdown_connections 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f5915f5a640 1 --2- 172.21.15.98:0/2075572368 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f591013ad40 0x7f591013d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f5915f5a640 1 --2- 172.21.15.98:0/2075572368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5910135990 0x7f591013a800 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f5915f5a640 1 -- 172.21.15.98:0/2075572368 >> 172.21.15.98:0/2075572368 conn(0x7f591006c140 msgr2=0x7f591006c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f5915f5a640 1 -- 172.21.15.98:0/2075572368 shutdown_connections 2024-06-11T19:09:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f5915f5a640 1 -- 172.21.15.98:0/2075572368 wait complete. 2024-06-11T19:09:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f92b2e69640 1 librados: starting msgr at 2024-06-11T19:09:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f92b2e69640 1 librados: starting objecter 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64e77fe640 1 -- 172.21.15.98:0/4023514578 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f64f8015c90 con 0x7f6500138c80 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f64e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f64e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f64e77fe640 10 monclient: dump: 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.348 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.348 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.348 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.348 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.348 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.348 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.348 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f64e77fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f64e77fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f64e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f92b2e69640 1 Processor -- start 2024-06-11T19:09:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f64e77fe640 1 -- 172.21.15.98:0/4023514578 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64f8019070 con 0x7f6500138c80 2024-06-11T19:09:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f64e77fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f64e77fe640 1 -- 172.21.15.98:0/4023514578 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f64f801e470 con 0x7f6500138c80 2024-06-11T19:09:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f64e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f64e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f64e77fe640 10 monclient: dump: 2024-06-11T19:09:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.350 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f37b97fa640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f37a003fdb0 0x7f37a0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f37b97fa640 1 -- 172.21.15.98:0/1017726184 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f37b000f040 con 0x7f37bc0a2530 2024-06-11T19:09:35.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f37b97fa640 10 client.4311.objecter ms_dispatch 0x7f37bc000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:35.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f37b97fa640 3 client.4311.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:35.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.191+0000 7f37b97fa640 3 client.4311.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:35.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f65051c6640 5 monclient: authenticate success, global_id 4295 2024-06-11T19:09:35.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f65051c6640 10 monclient: _renew_subs 2024-06-11T19:09:35.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f64fd59a640 4 set_mon_vals no callback set 2024-06-11T19:09:35.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 10 client.4311.objecter checking command 1 2024-06-11T19:09:35.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f92b2e69640 1 -- start start 2024-06-11T19:09:35.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f92b2e69640 1 librados: setting wanted keys 2024-06-11T19:09:35.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f65051c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6500133e10 con 0x7f6500138c80 2024-06-11T19:09:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] conn(0x7f37a0045c20 0x7f37a0048040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37bbfff640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f37a003fdb0 0x7f37a0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 20 client.4311.objecter _get_session s=0x7f37a0045550 osd=1 3 2024-06-11T19:09:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 20 client.4311.objecter put_session s=0x7f37a0045550 osd=1 3 2024-06-11T19:09:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 15 client.4311.objecter _session_command_op_remove -1 1 2024-06-11T19:09:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 20 client.4311.objecter _get_session s=0x7f37a0045550 osd=1 3 2024-06-11T19:09:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 20 client.4311.objecter get_session s=0x7f37a0045550 osd=1 3 2024-06-11T19:09:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 15 client.4311.objecter _session_command_op_assign 1 1 2024-06-11T19:09:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 20 client.4311.objecter put_session s=0x7f37a0045550 osd=1 4 2024-06-11T19:09:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 10 client.4311.objecter _send_command 1 2024-06-11T19:09:35.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f92b2e69640 1 librados: calling monclient init 2024-06-11T19:09:35.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f92b2e69640 10 monclient: init 2024-06-11T19:09:35.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37bbfff640 10 monclient: get_auth_request con 0x7f37a003fdb0 auth_method 0 2024-06-11T19:09:35.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 1 -- 172.21.15.98:0/1017726184 --> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37a00486f0 con 0x7f37a0045c20 2024-06-11T19:09:35.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 20 client.4311.objecter dump_active .. 0 homeless 2024-06-11T19:09:35.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 1 -- 172.21.15.98:0/1017726184 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f37b0058050 con 0x7f37bc0a2530 2024-06-11T19:09:35.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-06-11T19:09:35.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37c1163640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] conn(0x7f37a0045c20 0x7f37a0048040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37c1163640 10 monclient: get_auth_request con 0x7f37a0045c20 auth_method 0 2024-06-11T19:09:35.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f5915f5a640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f5915f5a640 10 monclient: monmap: 2024-06-11T19:09:35.355 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.355 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.357 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f65051c6640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f65051c6640 10 monclient: _renew_subs 2024-06-11T19:09:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f65051c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6500133b30 con 0x7f6500138c80 2024-06-11T19:09:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f65051c6640 1 librados: init done 2024-06-11T19:09:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37bbfff640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f37a003fdb0 0x7f37a0042270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f37bc09d170 tx=0x7f37a8008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 10 client.4311.objecter ms_handle_connect 0x7f37a003fdb0 2024-06-11T19:09:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37b97fa640 10 client.4311.objecter ms_handle_connect 0x7f37a0045c20 2024-06-11T19:09:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f37c1163640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] conn(0x7f37a0045c20 0x7f37a0048040 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f5915f5a640 1 librados: starting msgr at 2024-06-11T19:09:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f5915f5a640 1 librados: starting objecter 2024-06-11T19:09:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f5915f5a640 1 Processor -- start 2024-06-11T19:09:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f5915f5a640 1 -- start start 2024-06-11T19:09:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f5915f5a640 1 librados: setting wanted keys 2024-06-11T19:09:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f5915f5a640 1 librados: calling monclient init 2024-06-11T19:09:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.192+0000 7f5915f5a640 10 monclient: init 2024-06-11T19:09:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 10 monclient: _add_conns ranks=[5,7,2,4,8,3,0,6,1] 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f92ac073830 0x7f92ac132410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 10 monclient(hunting): picked mon.noname-h con 0x7f92ac073830 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f92ac09d520 0x7f92ac132950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 10 monclient(hunting): picked mon.noname-i con 0x7f92ac09d520 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92ac0a52a0 0x7f92ac132e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 10 monclient(hunting): picked mon.noname-b con 0x7f92ac0a52a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f92ac0a1fd0 con 0x7f92ac0a52a0 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f92ac0a1e50 con 0x7f92ac073830 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f92ac0a2150 con 0x7f92ac09d520 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b2e69640 10 monclient(hunting): authenticate will time out at 785.000179s 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92ac0a52a0 0x7f92ac132e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92ac0a52a0 0x7f92ac132e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47936/0 (socket says 172.21.15.98:47936) 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 1 -- 172.21.15.98:0/3349081287 learned_addr learned my addr 172.21.15.98:0/3349081287 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 10 monclient(hunting): get_auth_request con 0x7f92ac0a52a0 auth_method 0 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b0c05640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f92ac073830 0x7f92ac132410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92abfff640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f92ac09d520 0x7f92ac132950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b0c05640 10 monclient(hunting): get_auth_request con 0x7f92ac073830 auth_method 0 2024-06-11T19:09:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b0c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b0c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b0c05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92abfff640 10 monclient(hunting): get_auth_request con 0x7f92ac09d520 auth_method 0 2024-06-11T19:09:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92abfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92abfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f65051c6640 10 client.4295.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-06-11T19:09:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f65051c6640 15 client.4295.objecter _session_command_op_assign -1 1 2024-06-11T19:09:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f65051c6640 20 client.4295.objecter _get_session osd=-1 returning homeless 2024-06-11T19:09:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f65051c6640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 1 -- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f92ac09d520 msgr2=0x7f92ac132950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f65051c6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f65000a10f0 con 0x7f6500138c80 2024-06-11T19:09:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f92ac09d520 0x7f92ac132950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 1 -- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f92ac073830 msgr2=0x7f92ac132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f92ac073830 0x7f92ac132410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 1 monclient: found mon.noname-b 2024-06-11T19:09:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 1 -- 172.21.15.98:0/3349081287 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92ac13b150 con 0x7f92ac0a52a0 2024-06-11T19:09:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92abfff640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f92ac09d520 0x7f92ac132950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 10 monclient: _add_conns ranks=[2,5,4,0,6,8,1,3,7] 2024-06-11T19:09:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59100733a0 0x7f59101c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 10 monclient(hunting): picked mon.noname-b con 0x7f59100733a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5910135990 0x7f5910132610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 10 monclient(hunting): picked mon.noname-h con 0x7f5910135990 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f591013ad40 0x7f591009dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 10 monclient(hunting): picked mon.noname-c con 0x7f591013ad40 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5910071e70 con 0x7f59100733a0 2024-06-11T19:09:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b0c05640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f92ac073830 0x7f92ac132410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5910071b90 con 0x7f591013ad40 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f59100a1cc0 con 0x7f5910135990 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5915f5a640 10 monclient(hunting): authenticate will time out at 785.000835s 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5914f58640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59100733a0 0x7f59101c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5914f58640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59100733a0 0x7f59101c91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47950/0 (socket says 172.21.15.98:47950) 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5914f58640 1 -- 172.21.15.98:0/4187791862 learned_addr learned my addr 172.21.15.98:0/4187791862 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5914f58640 10 monclient(hunting): get_auth_request con 0x7f59100733a0 auth_method 0 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5914f58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5914f58640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f5914f58640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f92b1406640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92ac0a52a0 0x7f92ac132e90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f92a400ef90 tx=0x7f92a400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.193+0000 7f590ffff640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5910135990 0x7f5910132610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f590ffff640 10 monclient(hunting): get_auth_request con 0x7f5910135990 auth_method 0 2024-06-11T19:09:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f590ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f590ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f590ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 10 client.?.objecter ms_handle_connect 0x7f92ac0a52a0 2024-06-11T19:09:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f37b97fa640 1 -- 172.21.15.98:0/1017726184 <== osd.1 v2:172.21.15.98:6800/3886067876 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37a00486f0 con 0x7f37a0045c20 2024-06-11T19:09:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f37b97fa640 10 client.4311.objecter ms_dispatch 0x7f37bc000b90 command_reply(tid 1: 0 ) 2024-06-11T19:09:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f37b97fa640 10 client.4311.objecter _finish_command 1 = system:0 2024-06-11T19:09:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f37b97fa640 20 client.4311.objecter put_session s=0x7f37a0045550 osd=1 4 2024-06-11T19:09:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f37b97fa640 15 client.4311.objecter _session_command_op_remove 1 1 2024-06-11T19:09:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 1 -- 172.21.15.98:0/3349081287 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92a4018020 con 0x7f92ac0a52a0 2024-06-11T19:09:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5915759640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f591013ad40 0x7f591009dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 10 monclient: dump: 2024-06-11T19:09:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.373 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f590ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f590ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f590ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5915759640 10 monclient(hunting): get_auth_request con 0x7f591013ad40 auth_method 0 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5915759640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5915759640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5915759640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5915759640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5915759640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5915759640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 1 -- 172.21.15.98:0/4187791862 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5910135990 msgr2=0x7f5910132610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5910135990 0x7f5910132610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 1 -- 172.21.15.98:0/4187791862 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f591013ad40 msgr2=0x7f591009dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 1 -- 172.21.15.98:0/4023514578 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f64f800a710 con 0x7f6500138c80 2024-06-11T19:09:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92b2e69640 5 monclient: authenticate success, global_id 4304 2024-06-11T19:09:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 1 -- 172.21.15.98:0/3349081287 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92a400cd30 con 0x7f92ac0a52a0 2024-06-11T19:09:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 1 -- 172.21.15.98:0/3349081287 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92a4014d60 con 0x7f92ac0a52a0 2024-06-11T19:09:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92a97fa640 10 monclient: dump: 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.378 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92aaffd640 4 set_mon_vals no callback set 2024-06-11T19:09:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92b2e69640 10 monclient: _renew_subs 2024-06-11T19:09:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92b2e69640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92b2e69640 1 -- 172.21.15.98:0/3349081287 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92ac13b430 con 0x7f92ac0a52a0 2024-06-11T19:09:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92b2e69640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92b2e69640 10 monclient: _renew_subs 2024-06-11T19:09:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92b2e69640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92b2e69640 1 -- 172.21.15.98:0/3349081287 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92ac133630 con 0x7f92ac0a52a0 2024-06-11T19:09:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f92b2e69640 1 librados: init done 2024-06-11T19:09:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 1 --2- 172.21.15.98:0/4023514578 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f64ec03fdb0 0x7f64ec042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 1 -- 172.21.15.98:0/4023514578 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f64f8059ba0 con 0x7f6500138c80 2024-06-11T19:09:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 10 client.4295.objecter ms_dispatch 0x7f6500000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 3 client.4295.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 3 client.4295.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f591013ad40 0x7f591009dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 1 monclient: found mon.noname-b 2024-06-11T19:09:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 10 client.4295.objecter checking command 1 2024-06-11T19:09:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 1 --2- 172.21.15.98:0/4023514578 >> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] conn(0x7f64ec045bb0 0x7f64ec047fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 20 client.4295.objecter _get_session s=0x7f64ec0454e0 osd=4 3 2024-06-11T19:09:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 20 client.4295.objecter put_session s=0x7f64ec0454e0 osd=4 3 2024-06-11T19:09:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 15 client.4295.objecter _session_command_op_remove -1 1 2024-06-11T19:09:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 20 client.4295.objecter _get_session s=0x7f64ec0454e0 osd=4 3 2024-06-11T19:09:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 20 client.4295.objecter get_session s=0x7f64ec0454e0 osd=4 3 2024-06-11T19:09:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 15 client.4295.objecter _session_command_op_assign 4 1 2024-06-11T19:09:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5914f58640 1 -- 172.21.15.98:0/4187791862 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59100a1480 con 0x7f59100733a0 2024-06-11T19:09:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f5915759640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f591013ad40 0x7f591009dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 20 client.4295.objecter put_session s=0x7f64ec0454e0 osd=4 4 2024-06-11T19:09:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64fed9d640 1 --2- 172.21.15.98:0/4023514578 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f64ec03fdb0 0x7f64ec042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 10 client.4295.objecter _send_command 1 2024-06-11T19:09:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 1 -- 172.21.15.98:0/4023514578 --> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64ec048680 con 0x7f64ec045bb0 2024-06-11T19:09:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 20 client.4295.objecter dump_active .. 0 homeless 2024-06-11T19:09:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 1 -- 172.21.15.98:0/4023514578 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f64f8060050 con 0x7f6500138c80 2024-06-11T19:09:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64e77fe640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-06-11T19:09:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.194+0000 7f64fed9d640 10 monclient: get_auth_request con 0x7f64ec03fdb0 auth_method 0 2024-06-11T19:09:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f5914f58640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59100733a0 0x7f59101c91b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5904001eb0 tx=0x7f590400c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 10 client.?.objecter ms_handle_connect 0x7f59100733a0 2024-06-11T19:09:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f64fe59c640 1 --2- 172.21.15.98:0/4023514578 >> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] conn(0x7f64ec045bb0 0x7f64ec047fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f64fe59c640 10 monclient: get_auth_request con 0x7f64ec045bb0 auth_method 0 2024-06-11T19:09:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 1 -- 172.21.15.98:0/4187791862 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f59040047e0 con 0x7f59100733a0 2024-06-11T19:09:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f64e77fe640 10 client.4295.objecter ms_handle_connect 0x7f64ec03fdb0 2024-06-11T19:09:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f64fed9d640 1 --2- 172.21.15.98:0/4023514578 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f64ec03fdb0 0x7f64ec042270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f64f0036000 tx=0x7f64f0008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 10 monclient: dump: 2024-06-11T19:09:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.385 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f92b2e69640 10 client.4304.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f92b2e69640 15 client.4304.objecter _session_command_op_assign -1 1 2024-06-11T19:09:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f92b2e69640 20 client.4304.objecter _get_session osd=-1 returning homeless 2024-06-11T19:09:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f92b2e69640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f92b2e69640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 1 -- 172.21.15.98:0/4187791862 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59040049c0 con 0x7f59100733a0 2024-06-11T19:09:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f92b2e69640 1 -- 172.21.15.98:0/3349081287 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f92ac09dc50 con 0x7f92ac0a52a0 2024-06-11T19:09:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f5915f5a640 5 monclient: authenticate success, global_id 4313 2024-06-11T19:09:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f5915f5a640 10 monclient: _renew_subs 2024-06-11T19:09:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f5915f5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f64e77fe640 10 client.4295.objecter ms_handle_connect 0x7f64ec045bb0 2024-06-11T19:09:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f64fe59c640 1 --2- 172.21.15.98:0/4023514578 >> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] conn(0x7f64ec045bb0 0x7f64ec047fd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f5915f5a640 1 -- 172.21.15.98:0/4187791862 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591009ffc0 con 0x7f59100733a0 2024-06-11T19:09:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 1 -- 172.21.15.98:0/4187791862 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f59040133c0 con 0x7f59100733a0 2024-06-11T19:09:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590d7fa640 10 monclient: dump: 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.391 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f590effd640 4 set_mon_vals no callback set 2024-06-11T19:09:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f5915f5a640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f5915f5a640 10 monclient: _renew_subs 2024-06-11T19:09:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f5915f5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f5915f5a640 1 -- 172.21.15.98:0/4187791862 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59100a0240 con 0x7f59100733a0 2024-06-11T19:09:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.195+0000 7f5915f5a640 1 librados: init done 2024-06-11T19:09:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 1 -- 172.21.15.98:0/3349081287 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f92a4005d10 con 0x7f92ac0a52a0 2024-06-11T19:09:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f929003fdb0 0x7f9290042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 1 -- 172.21.15.98:0/3349081287 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f92a40521d0 con 0x7f92ac0a52a0 2024-06-11T19:09:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 10 client.4304.objecter ms_dispatch 0x7f92ac000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 3 client.4304.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 3 client.4304.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 10 client.4304.objecter checking command 1 2024-06-11T19:09:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] conn(0x7f9290045c20 0x7f9290048040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 20 client.4304.objecter _get_session s=0x7f9290045550 osd=3 3 2024-06-11T19:09:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 20 client.4304.objecter put_session s=0x7f9290045550 osd=3 3 2024-06-11T19:09:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f5915f5a640 10 client.4313.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f5915f5a640 15 client.4313.objecter _session_command_op_assign -1 1 2024-06-11T19:09:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f5915f5a640 20 client.4313.objecter _get_session osd=-1 returning homeless 2024-06-11T19:09:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f5915f5a640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f5915f5a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f5915f5a640 1 -- 172.21.15.98:0/4187791862 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f59100a17c0 con 0x7f59100733a0 2024-06-11T19:09:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 15 client.4304.objecter _session_command_op_remove -1 1 2024-06-11T19:09:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 20 client.4304.objecter _get_session s=0x7f9290045550 osd=3 3 2024-06-11T19:09:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 20 client.4304.objecter get_session s=0x7f9290045550 osd=3 3 2024-06-11T19:09:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 15 client.4304.objecter _session_command_op_assign 3 1 2024-06-11T19:09:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 20 client.4304.objecter put_session s=0x7f9290045550 osd=3 4 2024-06-11T19:09:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 10 client.4304.objecter _send_command 1 2024-06-11T19:09:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 1 -- 172.21.15.98:0/3349081287 --> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f92900486f0 con 0x7f9290045c20 2024-06-11T19:09:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 20 client.4304.objecter dump_active .. 0 homeless 2024-06-11T19:09:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f64e77fe640 1 -- 172.21.15.98:0/4023514578 <== osd.4 v2:172.21.15.117:6800/3877218228 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f64ec048680 con 0x7f64ec045bb0 2024-06-11T19:09:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 1 -- 172.21.15.98:0/3349081287 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f92a4059050 con 0x7f92ac0a52a0 2024-06-11T19:09:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92a97fa640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-06-11T19:09:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92b0c05640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f929003fdb0 0x7f9290042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92b0c05640 10 monclient: get_auth_request con 0x7f929003fdb0 auth_method 0 2024-06-11T19:09:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f64e77fe640 10 client.4295.objecter ms_dispatch 0x7f6500000b90 command_reply(tid 1: 0 ) 2024-06-11T19:09:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f64e77fe640 10 client.4295.objecter _finish_command 1 = system:0 2024-06-11T19:09:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f64e77fe640 20 client.4295.objecter put_session s=0x7f64ec0454e0 osd=4 4 2024-06-11T19:09:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f64e77fe640 15 client.4295.objecter _session_command_op_remove 4 1 2024-06-11T19:09:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92abfff640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] conn(0x7f9290045c20 0x7f9290048040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f92abfff640 10 monclient: get_auth_request con 0x7f9290045c20 auth_method 0 2024-06-11T19:09:35.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.196+0000 7f590d7fa640 1 -- 172.21.15.98:0/4187791862 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f5904011070 con 0x7f59100733a0 2024-06-11T19:09:35.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f58fc03fdb0 0x7f58fc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f92a97fa640 10 client.4304.objecter ms_handle_connect 0x7f929003fdb0 2024-06-11T19:09:35.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f92b0c05640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f929003fdb0 0x7f9290042270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f92a0035000 tx=0x7f92a0030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 1 -- 172.21.15.98:0/4187791862 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f5904050ee0 con 0x7f59100733a0 2024-06-11T19:09:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 10 client.4313.objecter ms_dispatch 0x7f5910000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 3 client.4313.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 3 client.4313.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 10 client.4313.objecter checking command 1 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] conn(0x7f58fc045c20 0x7f58fc048040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 20 client.4313.objecter _get_session s=0x7f58fc045550 osd=0 3 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 20 client.4313.objecter put_session s=0x7f58fc045550 osd=0 3 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 15 client.4313.objecter _session_command_op_remove -1 1 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 20 client.4313.objecter _get_session s=0x7f58fc045550 osd=0 3 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 20 client.4313.objecter get_session s=0x7f58fc045550 osd=0 3 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 15 client.4313.objecter _session_command_op_assign 0 1 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 20 client.4313.objecter put_session s=0x7f58fc045550 osd=0 4 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 10 client.4313.objecter _send_command 1 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 1 -- 172.21.15.98:0/4187791862 --> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5904011e70 con 0x7f58fc045c20 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 20 client.4313.objecter dump_active .. 0 homeless 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 1 -- 172.21.15.98:0/4187791862 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f59040534c0 con 0x7f59100733a0 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-06-11T19:09:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590ffff640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f58fc03fdb0 0x7f58fc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590ffff640 10 monclient: get_auth_request con 0x7f58fc03fdb0 auth_method 0 2024-06-11T19:09:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f5915759640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] conn(0x7f58fc045c20 0x7f58fc048040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f5915759640 10 monclient: get_auth_request con 0x7f58fc045c20 auth_method 0 2024-06-11T19:09:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f92a97fa640 10 client.4304.objecter ms_handle_connect 0x7f9290045c20 2024-06-11T19:09:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f92abfff640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] conn(0x7f9290045c20 0x7f9290048040 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f590d7fa640 10 client.4313.objecter ms_handle_connect 0x7f58fc045c20 2024-06-11T19:09:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.197+0000 7f5915759640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] conn(0x7f58fc045c20 0x7f58fc048040 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.198+0000 7f590d7fa640 10 client.4313.objecter ms_handle_connect 0x7f58fc03fdb0 2024-06-11T19:09:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.198+0000 7f590ffff640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f58fc03fdb0 0x7f58fc042270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f590003e000 tx=0x7f5900038000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.198+0000 7f92a97fa640 1 -- 172.21.15.98:0/3349081287 <== osd.3 v2:172.21.15.117:6808/2561136926 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f92900486f0 con 0x7f9290045c20 2024-06-11T19:09:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.198+0000 7f92a97fa640 10 client.4304.objecter ms_dispatch 0x7f92ac000b90 command_reply(tid 1: 0 ) 2024-06-11T19:09:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.198+0000 7f92a97fa640 10 client.4304.objecter _finish_command 1 = system:0 2024-06-11T19:09:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.198+0000 7f92a97fa640 20 client.4304.objecter put_session s=0x7f9290045550 osd=3 4 2024-06-11T19:09:35.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.198+0000 7f92a97fa640 15 client.4304.objecter _session_command_op_remove 3 1 2024-06-11T19:09:35.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.198+0000 7f590d7fa640 1 -- 172.21.15.98:0/4187791862 <== osd.0 v2:172.21.15.98:6809/4087730304 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5904011e70 con 0x7f58fc045c20 2024-06-11T19:09:35.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.198+0000 7f590d7fa640 10 client.4313.objecter ms_dispatch 0x7f5910000b90 command_reply(tid 1: 0 ) 2024-06-11T19:09:35.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.198+0000 7f590d7fa640 10 client.4313.objecter _finish_command 1 = system:0 2024-06-11T19:09:35.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.198+0000 7f590d7fa640 20 client.4313.objecter put_session s=0x7f58fc045550 osd=0 4 2024-06-11T19:09:35.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.198+0000 7f590d7fa640 15 client.4313.objecter _session_command_op_remove 0 1 2024-06-11T19:09:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.212+0000 7f65051c6640 10 client.4295.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-11T19:09:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.212+0000 7f65051c6640 15 client.4295.objecter _session_command_op_assign -1 2 2024-06-11T19:09:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.212+0000 7f65051c6640 20 client.4295.objecter _get_session s=0x7f64ec0454e0 osd=4 3 2024-06-11T19:09:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.212+0000 7f65051c6640 20 client.4295.objecter put_session s=0x7f64ec0454e0 osd=4 3 2024-06-11T19:09:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.212+0000 7f65051c6640 20 client.4295.objecter _get_session s=0x7f64ec0454e0 osd=4 3 2024-06-11T19:09:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.212+0000 7f65051c6640 15 client.4295.objecter _session_command_op_remove -1 2 2024-06-11T19:09:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.212+0000 7f65051c6640 20 client.4295.objecter get_session s=0x7f64ec0454e0 osd=4 3 2024-06-11T19:09:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.212+0000 7f65051c6640 15 client.4295.objecter _session_command_op_assign 4 2 2024-06-11T19:09:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.212+0000 7f65051c6640 20 client.4295.objecter put_session s=0x7f64ec0454e0 osd=4 4 2024-06-11T19:09:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.212+0000 7f65051c6640 10 client.4295.objecter _send_command 2 2024-06-11T19:09:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.212+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 --> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6500073920 con 0x7f64ec045bb0 2024-06-11T19:09:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f64e77fe640 1 -- 172.21.15.98:0/4023514578 <== osd.4 v2:172.21.15.117:6800/3877218228 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6500073920 con 0x7f64ec045bb0 2024-06-11T19:09:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f64e77fe640 10 client.4295.objecter ms_dispatch 0x7f6500000b90 command_reply(tid 2: 0 ) 2024-06-11T19:09:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f64e77fe640 10 client.4295.objecter _finish_command 2 = system:0 2024-06-11T19:09:35.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f64e77fe640 20 client.4295.objecter put_session s=0x7f64ec0454e0 osd=4 4 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f64e77fe640 15 client.4295.objecter _session_command_op_remove 4 2 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 10 librados: watch_flush enter 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 10 librados: watch_flush exit 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 10 client.4295.objecter close_session for osd.4 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 >> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] conn(0x7f64ec045bb0 msgr2=0x7f64ec047fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 1 --2- 172.21.15.98:0/4023514578 >> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] conn(0x7f64ec045bb0 0x7f64ec047fd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 20 client.4295.objecter put_session s=0x7f64ec0454e0 osd=4 1 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 20 client.4295.objecter shutdown clearing up homeless session... 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 10 client.4295.objecter successfully canceled tick 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f64ec03fdb0 msgr2=0x7f64ec042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 1 --2- 172.21.15.98:0/4023514578 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f64ec03fdb0 0x7f64ec042270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f64f0036000 tx=0x7f64f0008030 comp rx=0 tx=0).stop 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 10 monclient: shutdown 2024-06-11T19:09:35.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6500138c80 msgr2=0x7f6500130370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 1 --2- 172.21.15.98:0/4023514578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6500138c80 0x7f6500130370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f64f800cb40 tx=0x7f64f800a060 comp rx=0 tx=0).stop 2024-06-11T19:09:35.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f64ff59e640 1 -- 172.21.15.98:0/4023514578 reap_dead start 2024-06-11T19:09:35.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 shutdown_connections 2024-06-11T19:09:35.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 >> 172.21.15.98:0/4023514578 conn(0x7f65000a7850 msgr2=0x7f65000a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 shutdown_connections 2024-06-11T19:09:35.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 1 -- 172.21.15.98:0/4023514578 wait complete. 2024-06-11T19:09:35.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.213+0000 7f65051c6640 1 librados: shutdown 2024-06-11T19:09:35.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 10 client.4313.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-11T19:09:35.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 15 client.4313.objecter _session_command_op_assign -1 2 2024-06-11T19:09:35.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 20 client.4313.objecter _get_session s=0x7f58fc045550 osd=0 3 2024-06-11T19:09:35.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 20 client.4313.objecter put_session s=0x7f58fc045550 osd=0 3 2024-06-11T19:09:35.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 20 client.4313.objecter _get_session s=0x7f58fc045550 osd=0 3 2024-06-11T19:09:35.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 15 client.4313.objecter _session_command_op_remove -1 2 2024-06-11T19:09:35.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 20 client.4313.objecter get_session s=0x7f58fc045550 osd=0 3 2024-06-11T19:09:35.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 15 client.4313.objecter _session_command_op_assign 0 2 2024-06-11T19:09:35.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 20 client.4313.objecter put_session s=0x7f58fc045550 osd=0 4 2024-06-11T19:09:35.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 10 client.4313.objecter _send_command 2 2024-06-11T19:09:35.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 1 -- 172.21.15.98:0/4187791862 --> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f591013a920 con 0x7f58fc045c20 2024-06-11T19:09:35.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f590d7fa640 1 -- 172.21.15.98:0/4187791862 <== osd.0 v2:172.21.15.98:6809/4087730304 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f591013a920 con 0x7f58fc045c20 2024-06-11T19:09:35.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f590d7fa640 10 client.4313.objecter ms_dispatch 0x7f5910000b90 command_reply(tid 2: 0 ) 2024-06-11T19:09:35.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f590d7fa640 10 client.4313.objecter _finish_command 2 = system:0 2024-06-11T19:09:35.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f590d7fa640 20 client.4313.objecter put_session s=0x7f58fc045550 osd=0 4 2024-06-11T19:09:35.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f590d7fa640 15 client.4313.objecter _session_command_op_remove 0 2 2024-06-11T19:09:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 10 librados: watch_flush enter 2024-06-11T19:09:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 10 librados: watch_flush exit 2024-06-11T19:09:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 10 client.4313.objecter close_session for osd.0 2024-06-11T19:09:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 1 -- 172.21.15.98:0/4187791862 >> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] conn(0x7f58fc045c20 msgr2=0x7f58fc048040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] conn(0x7f58fc045c20 0x7f58fc048040 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 20 client.4313.objecter put_session s=0x7f58fc045550 osd=0 1 2024-06-11T19:09:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 20 client.4313.objecter shutdown clearing up homeless session... 2024-06-11T19:09:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 10 client.4313.objecter successfully canceled tick 2024-06-11T19:09:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 1 -- 172.21.15.98:0/4187791862 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f58fc03fdb0 msgr2=0x7f58fc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f58fc03fdb0 0x7f58fc042270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f590003e000 tx=0x7f5900038000 comp rx=0 tx=0).stop 2024-06-11T19:09:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 10 monclient: shutdown 2024-06-11T19:09:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.214+0000 7f5915f5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.215+0000 7f5915f5a640 1 -- 172.21.15.98:0/4187791862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59100733a0 msgr2=0x7f59101c91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.215+0000 7f5915f5a640 1 --2- 172.21.15.98:0/4187791862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f59100733a0 0x7f59101c91b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5904001eb0 tx=0x7f590400c960 comp rx=0 tx=0).stop 2024-06-11T19:09:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.215+0000 7f5915759640 1 -- 172.21.15.98:0/4187791862 reap_dead start 2024-06-11T19:09:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.215+0000 7f5915f5a640 1 -- 172.21.15.98:0/4187791862 shutdown_connections 2024-06-11T19:09:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.215+0000 7f5915f5a640 1 -- 172.21.15.98:0/4187791862 >> 172.21.15.98:0/4187791862 conn(0x7f591006c140 msgr2=0x7f59100a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.215+0000 7f5915f5a640 1 -- 172.21.15.98:0/4187791862 shutdown_connections 2024-06-11T19:09:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.215+0000 7f5915f5a640 1 -- 172.21.15.98:0/4187791862 wait complete. 2024-06-11T19:09:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.215+0000 7f5915f5a640 1 librados: shutdown 2024-06-11T19:09:35.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37c2bc6640 10 client.4311.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-11T19:09:35.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37c2bc6640 15 client.4311.objecter _session_command_op_assign -1 2 2024-06-11T19:09:35.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37c2bc6640 20 client.4311.objecter _get_session s=0x7f37a0045550 osd=1 3 2024-06-11T19:09:35.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37c2bc6640 20 client.4311.objecter put_session s=0x7f37a0045550 osd=1 3 2024-06-11T19:09:35.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37c2bc6640 20 client.4311.objecter _get_session s=0x7f37a0045550 osd=1 3 2024-06-11T19:09:35.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37c2bc6640 15 client.4311.objecter _session_command_op_remove -1 2 2024-06-11T19:09:35.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37c2bc6640 20 client.4311.objecter get_session s=0x7f37a0045550 osd=1 3 2024-06-11T19:09:35.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37c2bc6640 15 client.4311.objecter _session_command_op_assign 1 2 2024-06-11T19:09:35.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37c2bc6640 20 client.4311.objecter put_session s=0x7f37a0045550 osd=1 4 2024-06-11T19:09:35.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37c2bc6640 10 client.4311.objecter _send_command 2 2024-06-11T19:09:35.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37c2bc6640 1 -- 172.21.15.98:0/1017726184 --> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f37bc1285e0 con 0x7f37a0045c20 2024-06-11T19:09:35.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37b97fa640 1 -- 172.21.15.98:0/1017726184 <== osd.1 v2:172.21.15.98:6800/3886067876 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f37b4003990 con 0x7f37a0045c20 2024-06-11T19:09:35.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37b97fa640 10 client.4311.objecter ms_dispatch 0x7f37bc000b90 command_reply(tid 2: 0 ) 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37b97fa640 10 client.4311.objecter _finish_command 2 = system:0 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37b97fa640 20 client.4311.objecter put_session s=0x7f37a0045550 osd=1 4 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.223+0000 7f37b97fa640 15 client.4311.objecter _session_command_op_remove 1 2 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 10 librados: watch_flush enter 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 10 librados: watch_flush exit 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 10 client.4311.objecter close_session for osd.1 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 1 -- 172.21.15.98:0/1017726184 >> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] conn(0x7f37a0045c20 msgr2=0x7f37a0048040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] conn(0x7f37a0045c20 0x7f37a0048040 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 20 client.4311.objecter put_session s=0x7f37a0045550 osd=1 1 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 20 client.4311.objecter shutdown clearing up homeless session... 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 10 client.4311.objecter successfully canceled tick 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 1 -- 172.21.15.98:0/1017726184 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f37a003fdb0 msgr2=0x7f37a0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f37a003fdb0 0x7f37a0042270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f37bc09d170 tx=0x7f37a8008030 comp rx=0 tx=0).stop 2024-06-11T19:09:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 10 monclient: shutdown 2024-06-11T19:09:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 1 -- 172.21.15.98:0/1017726184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37bc0a2530 msgr2=0x7f37bc1cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 1 --2- 172.21.15.98:0/1017726184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37bc0a2530 0x7f37bc1cdaa0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f37b000e850 tx=0x7f37b000ec50 comp rx=0 tx=0).stop 2024-06-11T19:09:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c1163640 1 -- 172.21.15.98:0/1017726184 reap_dead start 2024-06-11T19:09:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 1 -- 172.21.15.98:0/1017726184 shutdown_connections 2024-06-11T19:09:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 1 -- 172.21.15.98:0/1017726184 >> 172.21.15.98:0/1017726184 conn(0x7f37bc0ab850 msgr2=0x7f37bc09c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 1 -- 172.21.15.98:0/1017726184 shutdown_connections 2024-06-11T19:09:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.224+0000 7f37c2bc6640 1 -- 172.21.15.98:0/1017726184 wait complete. 2024-06-11T19:09:35.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.225+0000 7f37c2bc6640 1 librados: shutdown 2024-06-11T19:09:35.419 INFO:teuthology.orchestra.run.smithi098.stdout:47244640261 2024-06-11T19:09:35.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-11T19:09:35.420 INFO:teuthology.orchestra.run.smithi098.stdout:47244640262 2024-06-11T19:09:35.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-06-11T19:09:35.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92b2e69640 10 client.4304.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-11T19:09:35.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92b2e69640 15 client.4304.objecter _session_command_op_assign -1 2 2024-06-11T19:09:35.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92b2e69640 20 client.4304.objecter _get_session s=0x7f9290045550 osd=3 3 2024-06-11T19:09:35.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92b2e69640 20 client.4304.objecter put_session s=0x7f9290045550 osd=3 3 2024-06-11T19:09:35.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92b2e69640 20 client.4304.objecter _get_session s=0x7f9290045550 osd=3 3 2024-06-11T19:09:35.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92b2e69640 15 client.4304.objecter _session_command_op_remove -1 2 2024-06-11T19:09:35.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92b2e69640 20 client.4304.objecter get_session s=0x7f9290045550 osd=3 3 2024-06-11T19:09:35.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92b2e69640 15 client.4304.objecter _session_command_op_assign 3 2 2024-06-11T19:09:35.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92b2e69640 20 client.4304.objecter put_session s=0x7f9290045550 osd=3 4 2024-06-11T19:09:35.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92b2e69640 10 client.4304.objecter _send_command 2 2024-06-11T19:09:35.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92b2e69640 1 -- 172.21.15.98:0/3349081287 --> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f92ac0a1910 con 0x7f9290045c20 2024-06-11T19:09:35.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92a97fa640 1 -- 172.21.15.98:0/3349081287 <== osd.3 v2:172.21.15.117:6808/2561136926 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f92ac0a1910 con 0x7f9290045c20 2024-06-11T19:09:35.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.228+0000 7f92a97fa640 10 client.4304.objecter ms_dispatch 0x7f92ac000b90 command_reply(tid 2: 0 ) 2024-06-11T19:09:35.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92a97fa640 10 client.4304.objecter _finish_command 2 = system:0 2024-06-11T19:09:35.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92a97fa640 20 client.4304.objecter put_session s=0x7f9290045550 osd=3 4 2024-06-11T19:09:35.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92a97fa640 15 client.4304.objecter _session_command_op_remove 3 2 2024-06-11T19:09:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 10 librados: watch_flush enter 2024-06-11T19:09:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 10 librados: watch_flush exit 2024-06-11T19:09:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 10 client.4304.objecter close_session for osd.3 2024-06-11T19:09:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 1 -- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] conn(0x7f9290045c20 msgr2=0x7f9290048040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] conn(0x7f9290045c20 0x7f9290048040 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 20 client.4304.objecter put_session s=0x7f9290045550 osd=3 1 2024-06-11T19:09:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 20 client.4304.objecter shutdown clearing up homeless session... 2024-06-11T19:09:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 10 client.4304.objecter successfully canceled tick 2024-06-11T19:09:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 1 -- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f929003fdb0 msgr2=0x7f9290042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f929003fdb0 0x7f9290042270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f92a0035000 tx=0x7f92a0030000 comp rx=0 tx=0).stop 2024-06-11T19:09:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 10 monclient: shutdown 2024-06-11T19:09:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 1 -- 172.21.15.98:0/3349081287 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92ac0a52a0 msgr2=0x7f92ac132e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 1 --2- 172.21.15.98:0/3349081287 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f92ac0a52a0 0x7f92ac132e90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f92a400ef90 tx=0x7f92a400c490 comp rx=0 tx=0).stop 2024-06-11T19:09:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b1406640 1 -- 172.21.15.98:0/3349081287 reap_dead start 2024-06-11T19:09:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 1 -- 172.21.15.98:0/3349081287 shutdown_connections 2024-06-11T19:09:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 1 -- 172.21.15.98:0/3349081287 >> 172.21.15.98:0/3349081287 conn(0x7f92ac0a4780 msgr2=0x7f92ac0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 1 -- 172.21.15.98:0/3349081287 shutdown_connections 2024-06-11T19:09:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 1 -- 172.21.15.98:0/3349081287 wait complete. 2024-06-11T19:09:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.229+0000 7f92b2e69640 1 librados: shutdown 2024-06-11T19:09:35.425 INFO:teuthology.orchestra.run.smithi098.stdout:51539607558 2024-06-11T19:09:35.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-06-11T19:09:35.427 INFO:teuthology.orchestra.run.smithi098.stdout:47244640261 2024-06-11T19:09:35.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-06-11T19:09:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.571+0000 7f42234d1640 10 monclient: get_monmap_and_config 2024-06-11T19:09:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.571+0000 7f42234d1640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.571+0000 7f42234d1640 10 monclient: monmap: 2024-06-11T19:09:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:35.572279+0000 2024-06-11T19:09:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:35.572279+0000 2024-06-11T19:09:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:35.572 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.571+0000 7f42234d1640 1 Processor -- start 2024-06-11T19:09:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 1 -- start start 2024-06-11T19:09:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 10 monclient: init 2024-06-11T19:09:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 10 monclient: _add_conns ranks=[8,1,6,0,9,2,7,10,3,4,5] 2024-06-11T19:09:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 0x7f421c0a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 10 monclient(hunting): picked mon.noname-e con 0x7f421c0a2f40 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f421c0a4df0 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f421c0a51d0 0x7f421c136de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 10 monclient(hunting): picked mon.noname-d con 0x7f421c0a51d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f421c0a0ed0 con 0x7f421c0a4df0 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f421c0a3320 con 0x7f421c0a51d0 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f421c0a2460 con 0x7f421c0a2f40 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f42234d1640 10 monclient(hunting): authenticate will time out at 785.379915s 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.572+0000 7f422126d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 0x7f421c0a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 0x7f421c0a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54486/0 (socket says 172.21.15.98:54486) 2024-06-11T19:09:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 1 -- 172.21.15.98:0/271552147 learned_addr learned my addr 172.21.15.98:0/271552147 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 10 monclient(hunting): get_auth_request con 0x7f421c0a2f40 auth_method 0 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f4221a6e640 1 --2- 172.21.15.98:0/271552147 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f421c0a51d0 0x7f421c136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 1 -- 172.21.15.98:0/271552147 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f421c0a51d0 msgr2=0x7f421c136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 1 --2- 172.21.15.98:0/271552147 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f421c0a51d0 0x7f421c136de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f4213fff640 1 -- 172.21.15.98:0/271552147 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2447352415 0 0) 0x7f421c0a0ed0 con 0x7f421c0a4df0 2024-06-11T19:09:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 1 -- 172.21.15.98:0/271552147 >> v1:172.21.15.117:6789/0 conn(0x7f421c0a4df0 legacy=0x7f421c1366d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 1 monclient: found mon.noname-e 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f422126d640 1 -- 172.21.15.98:0/271552147 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421c0a2180 con 0x7f421c0a2f40 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.573+0000 7f4213fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.574+0000 7f422126d640 1 --2- 172.21.15.98:0/271552147 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 0x7f421c0a48b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f42180098d0 tx=0x7f421802b840 comp rx=0 tx=0).ready entity=mon.8 client_cookie=47d8552ff821802b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.574+0000 7f4213fff640 1 -- 172.21.15.98:0/271552147 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4218031a10 con 0x7f421c0a2f40 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.574+0000 7f4213fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.574+0000 7f4213fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.574+0000 7f4213fff640 10 monclient: dump: 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f4213fff640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f4213fff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f4213fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f4213fff640 1 -- 172.21.15.98:0/271552147 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4218037070 con 0x7f421c0a2f40 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f4213fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f4213fff640 1 -- 172.21.15.98:0/271552147 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4218031a10 con 0x7f421c0a2f40 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f42234d1640 5 monclient: authenticate success, global_id 4310 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f4213fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f4213fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f4213fff640 10 monclient: dump: 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.583 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f42234d1640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f42234d1640 4 set_mon_vals no callback set 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f42234d1640 10 monclient: shutdown 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f42234d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f42234d1640 1 -- 172.21.15.98:0/271552147 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 msgr2=0x7f421c0a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.575+0000 7f42234d1640 1 --2- 172.21.15.98:0/271552147 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 0x7f421c0a48b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f42180098d0 tx=0x7f421802b840 comp rx=0 tx=0).stop 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.576+0000 7f42234d1640 1 -- 172.21.15.98:0/271552147 shutdown_connections 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.576+0000 7f42234d1640 1 --2- 172.21.15.98:0/271552147 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f421c0a51d0 0x7f421c136de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.576+0000 7f42234d1640 1 --2- 172.21.15.98:0/271552147 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 0x7f421c0a48b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.576+0000 7f42234d1640 1 -- 172.21.15.98:0/271552147 >> 172.21.15.98:0/271552147 conn(0x7f421c0a95b0 msgr2=0x7f421c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.576+0000 7f42234d1640 1 -- 172.21.15.98:0/271552147 shutdown_connections 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.576+0000 7f42234d1640 1 -- 172.21.15.98:0/271552147 wait complete. 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.577+0000 7f42234d1640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.577+0000 7f42234d1640 10 monclient: monmap: 2024-06-11T19:09:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.586 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.587 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.587 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.577+0000 7f42234d1640 1 librados: starting msgr at 2024-06-11T19:09:35.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.577+0000 7f42234d1640 1 librados: starting objecter 2024-06-11T19:09:35.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.577+0000 7f42234d1640 1 Processor -- start 2024-06-11T19:09:35.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.577+0000 7f42234d1640 1 -- start start 2024-06-11T19:09:35.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.577+0000 7f42234d1640 1 librados: setting wanted keys 2024-06-11T19:09:35.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.577+0000 7f42234d1640 1 librados: calling monclient init 2024-06-11T19:09:35.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.577+0000 7f42234d1640 10 monclient: init 2024-06-11T19:09:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.577+0000 7fb0bba2e640 10 monclient: get_monmap_and_config 2024-06-11T19:09:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.577+0000 7fb0bba2e640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.577+0000 7fb0bba2e640 10 monclient: monmap: 2024-06-11T19:09:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:35.578776+0000 2024-06-11T19:09:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:35.578776+0000 2024-06-11T19:09:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:35.588 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:35.589 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.589 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.589 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.589 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.589 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.589 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.589 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.589 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:35.589 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:35.589 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 10 monclient: _add_conns ranks=[8,4,5,7,6,0,3,1,2] 2024-06-11T19:09:35.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 0x7f421c135b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 10 monclient(hunting): picked mon.noname-e con 0x7f421c0a2f40 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f421c0a4df0 0x7f421c1360c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 10 monclient(hunting): picked mon.noname-c con 0x7f421c0a4df0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f422126d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 0x7f421c135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f421c0a51d0 0x7f421c125700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f421c0a4df0 0x7f421c1360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 10 monclient(hunting): picked mon.noname-h con 0x7f421c0a51d0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f421c0a1250 con 0x7f421c0a4df0 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f422126d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 0x7f421c135b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54488/0 (socket says 172.21.15.98:54488) 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f421c0a4df0 0x7f421c1360c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:40392/0 (socket says 172.21.15.98:40392) 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f422126d640 1 -- 172.21.15.98:0/4231906019 learned_addr learned my addr 172.21.15.98:0/4231906019 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 1 -- 172.21.15.98:0/4231906019 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f421c137320 con 0x7f421c0a51d0 2024-06-11T19:09:35.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 1 -- 172.21.15.98:0/4231906019 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f421c127e10 con 0x7f421c0a2f40 2024-06-11T19:09:35.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f42234d1640 10 monclient(hunting): authenticate will time out at 785.385488s 2024-06-11T19:09:35.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 10 monclient(hunting): get_auth_request con 0x7f421c0a4df0 auth_method 0 2024-06-11T19:09:35.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7fb0bba2e640 1 Processor -- start 2024-06-11T19:09:35.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f422126d640 10 monclient(hunting): get_auth_request con 0x7f421c0a2f40 auth_method 0 2024-06-11T19:09:35.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f422126d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f422126d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f422126d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-06-11T19:09:35.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 1 -- 172.21.15.98:0/4231906019 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 msgr2=0x7f421c135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 0x7f421c135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 1 -- 172.21.15.98:0/4231906019 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f421c0a51d0 msgr2=0x7f421c125700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f421c0a51d0 0x7f421c125700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 1 monclient: found mon.noname-c 2024-06-11T19:09:35.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:35.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f4220a6c640 1 -- 172.21.15.98:0/4231906019 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421c1292c0 con 0x7f421c0a4df0 2024-06-11T19:09:35.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7f422126d640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 0x7f421c135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:35.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7fb0bba2e640 1 -- start start 2024-06-11T19:09:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.578+0000 7fb0bba2e640 10 monclient: init 2024-06-11T19:09:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4220a6c640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f421c0a4df0 0x7f421c1360c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f420c00c390 tx=0x7f420c00c850 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 10 client.?.objecter ms_handle_connect 0x7f421c0a4df0 2024-06-11T19:09:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 1 -- 172.21.15.98:0/4231906019 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f420c017070 con 0x7f421c0a4df0 2024-06-11T19:09:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:35.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 10 monclient: _add_conns ranks=[5,10,4,6,1,2,0,9,8,7,3] 2024-06-11T19:09:35.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb0b409f450 0x7fb0b40a1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 10 monclient(hunting): picked mon.noname-h con 0x7fb0b409f450 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 10 monclient: dump: 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f42234d1640 5 monclient: authenticate success, global_id 4324 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 1 -- 172.21.15.98:0/4231906019 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f420c0052e0 con 0x7f421c0a4df0 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 1 -- 172.21.15.98:0/4231906019 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f420c005490 con 0x7f421c0a4df0 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f4211ffb640 10 monclient: dump: 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.600 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f42137fe640 4 set_mon_vals no callback set 2024-06-11T19:09:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb0b40766a0 0x7fb0b4134e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 10 monclient(hunting): picked mon.noname-f con 0x7fb0b40766a0 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0b4076a80 0x7fb0b4135370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 10 monclient(hunting): picked mon.noname-c con 0x7fb0b4076a80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fb0b40a02a0 con 0x7fb0b40766a0 2024-06-11T19:09:35.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb0b40a0580 con 0x7fb0b4076a80 2024-06-11T19:09:35.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fb0b40a0860 con 0x7fb0b409f450 2024-06-11T19:09:35.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0bba2e640 10 monclient(hunting): authenticate will time out at 785.386533s 2024-06-11T19:09:35.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f42234d1640 10 monclient: _renew_subs 2024-06-11T19:09:35.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f42234d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:35.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f42234d1640 1 -- 172.21.15.98:0/4231906019 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f421c0a88f0 con 0x7f421c0a4df0 2024-06-11T19:09:35.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0b97ca640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb0b409f450 0x7fb0b40a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f42234d1640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f42234d1640 10 monclient: _renew_subs 2024-06-11T19:09:35.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f42234d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:35.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f42234d1640 1 -- 172.21.15.98:0/4231906019 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f421c128fe0 con 0x7f421c0a4df0 2024-06-11T19:09:35.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7f42234d1640 1 librados: init done 2024-06-11T19:09:35.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0b97ca640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb0b409f450 0x7fb0b40a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:40254/0 (socket says 172.21.15.98:40254) 2024-06-11T19:09:35.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0b97ca640 1 -- 172.21.15.98:0/802588981 learned_addr learned my addr 172.21.15.98:0/802588981 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0b97ca640 10 monclient(hunting): get_auth_request con 0x7fb0b409f450 auth_method 0 2024-06-11T19:09:35.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0b97ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0b97ca640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.579+0000 7fb0b97ca640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b9fcb640 1 --2- 172.21.15.98:0/802588981 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0b4076a80 0x7fb0b4135370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b9fcb640 10 monclient(hunting): get_auth_request con 0x7fb0b4076a80 auth_method 0 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b9fcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b9fcb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b9fcb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b8fc9640 1 --2- 172.21.15.98:0/802588981 >> v2:172.21.15.117:3300/0 conn(0x7fb0b40766a0 0x7fb0b4134e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b8fc9640 10 monclient(hunting): get_auth_request con 0x7fb0b40766a0 auth_method 0 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b8fc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b8fc9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b8fc9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b8fc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b8fc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b8fc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b9fcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b9fcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b9fcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 1 -- 172.21.15.98:0/802588981 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0b4076a80 msgr2=0x7fb0b4135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 1 --2- 172.21.15.98:0/802588981 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0b4076a80 0x7fb0b4135370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 1 -- 172.21.15.98:0/802588981 >> v2:172.21.15.117:3300/0 conn(0x7fb0b40766a0 msgr2=0x7fb0b4134e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 1 --2- 172.21.15.98:0/802588981 >> v2:172.21.15.117:3300/0 conn(0x7fb0b40766a0 0x7fb0b4134e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 1 monclient: found mon.noname-h 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:09:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b97ca640 1 -- 172.21.15.98:0/802588981 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b41358b0 con 0x7fb0b409f450 2024-06-11T19:09:35.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7fb0b9fcb640 1 --2- 172.21.15.98:0/802588981 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0b4076a80 0x7fb0b4135370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7fb0b8fc9640 1 --2- 172.21.15.98:0/802588981 >> v2:172.21.15.117:3300/0 conn(0x7fb0b40766a0 0x7fb0b4134e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.580+0000 7f4211ffb640 1 -- 172.21.15.98:0/4231906019 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f420c0035f0 con 0x7f421c0a4df0 2024-06-11T19:09:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7f4211ffb640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f420003fdb0 0x7f4200042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7f4211ffb640 1 -- 172.21.15.98:0/4231906019 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f420c050690 con 0x7f421c0a4df0 2024-06-11T19:09:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7f4211ffb640 10 client.4324.objecter ms_dispatch 0x7f421c000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7f4211ffb640 3 client.4324.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7f4211ffb640 3 client.4324.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7f4211ffb640 20 client.4324.objecter dump_active .. 0 homeless 2024-06-11T19:09:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7f422126d640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f420003fdb0 0x7f4200042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7f422126d640 10 monclient: get_auth_request con 0x7f420003fdb0 auth_method 0 2024-06-11T19:09:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7f42234d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7f42234d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7f42234d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.581+0000 7f42234d1640 1 -- 172.21.15.98:0/4231906019 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f421c0a48b0 con 0x7f421c0a4df0 2024-06-11T19:09:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0b97ca640 1 --2- 172.21.15.98:0/802588981 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb0b409f450 0x7fb0b40a1520 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009870 tx=0x7fb0b002b7c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=1c00ac87707b940b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7f422126d640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f420003fdb0 0x7f4200042270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f421802bd80 tx=0x7f4218008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7f4211ffb640 10 client.4324.objecter ms_handle_connect 0x7f420003fdb0 2024-06-11T19:09:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 1 -- 172.21.15.98:0/802588981 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0b0038070 con 0x7fb0b409f450 2024-06-11T19:09:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:09:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 10 monclient: dump: 2024-06-11T19:09:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 1 -- 172.21.15.98:0/802588981 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b0031990 con 0x7fb0b409f450 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 1 -- 172.21.15.98:0/802588981 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0b00323b0 con 0x7fb0b409f450 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0abfff640 10 monclient: dump: 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.620 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.621 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.621 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.621 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.621 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.621 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.621 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.621 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.621 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0bba2e640 5 monclient: authenticate success, global_id 4235 2024-06-11T19:09:35.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0bba2e640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:35.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.582+0000 7fb0bba2e640 4 set_mon_vals no callback set 2024-06-11T19:09:35.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.583+0000 7fb0bba2e640 10 monclient: shutdown 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.583+0000 7fb0bba2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.583+0000 7fb0bba2e640 1 -- 172.21.15.98:0/802588981 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb0b409f450 msgr2=0x7fb0b40a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.583+0000 7fb0bba2e640 1 --2- 172.21.15.98:0/802588981 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb0b409f450 0x7fb0b40a1520 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009870 tx=0x7fb0b002b7c0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.583+0000 7fb0bba2e640 1 -- 172.21.15.98:0/802588981 shutdown_connections 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.583+0000 7fb0bba2e640 1 --2- 172.21.15.98:0/802588981 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0b4076a80 0x7fb0b4135370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.583+0000 7fb0bba2e640 1 --2- 172.21.15.98:0/802588981 >> v2:172.21.15.117:3300/0 conn(0x7fb0b40766a0 0x7fb0b4134e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.583+0000 7fb0bba2e640 1 --2- 172.21.15.98:0/802588981 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb0b409f450 0x7fb0b40a1520 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.583+0000 7fb0bba2e640 1 -- 172.21.15.98:0/802588981 >> 172.21.15.98:0/802588981 conn(0x7fb0b40a7850 msgr2=0x7fb0b40a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.583+0000 7fb0bba2e640 1 -- 172.21.15.98:0/802588981 shutdown_connections 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.583+0000 7fb0bba2e640 1 -- 172.21.15.98:0/802588981 wait complete. 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.584+0000 7fb0bba2e640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.584+0000 7fb0bba2e640 10 monclient: monmap: 2024-06-11T19:09:35.622 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.623 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.584+0000 7fb0bba2e640 1 librados: starting msgr at 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.584+0000 7fb0bba2e640 1 librados: starting objecter 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.584+0000 7fb0bba2e640 1 Processor -- start 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.584+0000 7fb0bba2e640 1 -- start start 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.584+0000 7fb0bba2e640 1 librados: setting wanted keys 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.584+0000 7fb0bba2e640 1 librados: calling monclient init 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.584+0000 7fb0bba2e640 10 monclient: init 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 10 monclient: _add_conns ranks=[1,3,6,4,7,5,8,2,0] 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0b40766a0 0x7fb0b409a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 10 monclient(hunting): picked mon.noname-f con 0x7fb0b40766a0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0b4076a80 0x7fb0b409aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 10 monclient(hunting): picked mon.noname-g con 0x7fb0b4076a80 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:35.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0b409f450 0x7fb0b41c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 10 monclient(hunting): picked mon.noname-d con 0x7fb0b409f450 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b4131ab0 con 0x7fb0b40766a0 2024-06-11T19:09:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b4131930 con 0x7fb0b4076a80 2024-06-11T19:09:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb0b4131c30 con 0x7fb0b409f450 2024-06-11T19:09:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0bba2e640 10 monclient(hunting): authenticate will time out at 785.392199s 2024-06-11T19:09:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b9fcb640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0b409f450 0x7fb0b41c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0b40766a0 0x7fb0b409a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0b40766a0 0x7fb0b409a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:39620/0 (socket says 172.21.15.98:39620) 2024-06-11T19:09:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 1 -- 172.21.15.98:0/3818742412 learned_addr learned my addr 172.21.15.98:0/3818742412 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b9fcb640 10 monclient(hunting): get_auth_request con 0x7fb0b409f450 auth_method 0 2024-06-11T19:09:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b9fcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b9fcb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b9fcb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 10 monclient(hunting): get_auth_request con 0x7fb0b40766a0 auth_method 0 2024-06-11T19:09:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b8fc9640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0b4076a80 0x7fb0b409aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b8fc9640 10 monclient(hunting): get_auth_request con 0x7fb0b4076a80 auth_method 0 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b8fc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b8fc9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b8fc9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b9fcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b9fcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b9fcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b8fc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b8fc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b8fc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 10 monclient(hunting): handle_auth_done global_id 4312 payload 824 2024-06-11T19:09:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 1 -- 172.21.15.98:0/3818742412 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0b409f450 msgr2=0x7fb0b41c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0b409f450 0x7fb0b41c93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 1 -- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0b4076a80 msgr2=0x7fb0b409aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0b4076a80 0x7fb0b409aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 1 monclient: found mon.noname-f 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b97ca640 1 -- 172.21.15.98:0/3818742412 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b412e710 con 0x7fb0b40766a0 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b8fc9640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0b4076a80 0x7fb0b409aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.585+0000 7fb0b9fcb640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0b409f450 0x7fb0b41c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0b97ca640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0b40766a0 0x7fb0b409a4f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00090b0 tx=0x7fb0b00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0a9ffb640 10 client.?.objecter ms_handle_connect 0x7fb0b40766a0 2024-06-11T19:09:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0a9ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0a9ffb640 1 -- 172.21.15.98:0/3818742412 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0b0038050 con 0x7fb0b40766a0 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0a9ffb640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0a9ffb640 10 monclient: dump: 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0a9ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0a9ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0a9ffb640 1 -- 172.21.15.98:0/3818742412 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b0044070 con 0x7fb0b40766a0 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0a9ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0ab7fe640 4 set_mon_vals no callback set 2024-06-11T19:09:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0bba2e640 5 monclient: authenticate success, global_id 4312 2024-06-11T19:09:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0bba2e640 10 monclient: _renew_subs 2024-06-11T19:09:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0bba2e640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:09:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0bba2e640 1 -- 172.21.15.98:0/3818742412 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b4130180 con 0x7fb0b40766a0 2024-06-11T19:09:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0bba2e640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0bba2e640 10 monclient: _renew_subs 2024-06-11T19:09:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0bba2e640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:09:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0bba2e640 1 -- 172.21.15.98:0/3818742412 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b4134e70 con 0x7fb0b40766a0 2024-06-11T19:09:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.586+0000 7fb0bba2e640 1 librados: init done 2024-06-11T19:09:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.587+0000 7fb0a9ffb640 1 -- 172.21.15.98:0/3818742412 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0b0004350 con 0x7fb0b40766a0 2024-06-11T19:09:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.587+0000 7f4211ffb640 1 -- 172.21.15.98:0/4231906019 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f420c011c10 con 0x7f421c0a4df0 2024-06-11T19:09:35.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.587+0000 7f4211ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.587+0000 7f4211ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:35.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.587+0000 7fb0a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.587+0000 7fb0a9ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:09:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.587+0000 7fb0a9ffb640 10 monclient: dump: 2024-06-11T19:09:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.635 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.587+0000 7fb0a9ffb640 1 -- 172.21.15.98:0/3818742412 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fb0b0003610 con 0x7fb0b40766a0 2024-06-11T19:09:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.588+0000 7fb0a9ffb640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb09803fde0 0x7fb0980422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.588+0000 7fb0bba2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.588+0000 7fb0bba2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.588+0000 7fb0bba2e640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:09:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.588+0000 7fb0a9ffb640 1 -- 172.21.15.98:0/3818742412 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fb0b0030070 con 0x7fb0b40766a0 2024-06-11T19:09:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.588+0000 7fb0a9ffb640 10 client.4312.objecter ms_dispatch 0x7fb0b4000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.588+0000 7fb0a9ffb640 3 client.4312.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.588+0000 7fb0a9ffb640 3 client.4312.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.588+0000 7fb0bba2e640 1 -- 172.21.15.98:0/3818742412 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b4069520 con 0x7fb0b40766a0 2024-06-11T19:09:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.588+0000 7fb0a9ffb640 20 client.4312.objecter dump_active .. 0 homeless 2024-06-11T19:09:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.588+0000 7fb0b8fc9640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb09803fde0 0x7fb0980422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.588+0000 7fb0b8fc9640 10 monclient: get_auth_request con 0x7fb09803fde0 auth_method 0 2024-06-11T19:09:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.589+0000 7fb0b8fc9640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb09803fde0 0x7fb0980422a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb0b409e050 tx=0x7fb0ac012040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.589+0000 7fb0a9ffb640 10 client.4312.objecter ms_handle_connect 0x7fb09803fde0 2024-06-11T19:09:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.593+0000 7fb0a9ffb640 1 -- 172.21.15.98:0/3818742412 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb0b4069520 con 0x7fb0b40766a0 2024-06-11T19:09:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.593+0000 7fb0a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.593+0000 7fb0a9ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:35.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.620+0000 7fcc5c5c4640 10 monclient: get_monmap_and_config 2024-06-11T19:09:35.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.620+0000 7fcc5c5c4640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.620+0000 7fe465e03640 10 monclient: get_monmap_and_config 2024-06-11T19:09:35.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.620+0000 7fcc5c5c4640 10 monclient: monmap: 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:35.621337+0000 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:35.621337+0000 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:35.639 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.620+0000 7fe465e03640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.620+0000 7fe465e03640 10 monclient: monmap: 2024-06-11T19:09:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:35.621533+0000 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:35.621533+0000 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:35.641 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.620+0000 7f1bcf069640 10 monclient: get_monmap_and_config 2024-06-11T19:09:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.620+0000 7f1bcf069640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.620+0000 7f1bcf069640 10 monclient: monmap: 2024-06-11T19:09:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:35.621966+0000 2024-06-11T19:09:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:35.621966+0000 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:35.643 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.621+0000 7fcc5c5c4640 1 Processor -- start 2024-06-11T19:09:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.621+0000 7fcc5c5c4640 1 -- start start 2024-06-11T19:09:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.621+0000 7fcc5c5c4640 10 monclient: init 2024-06-11T19:09:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.621+0000 7fe465e03640 1 Processor -- start 2024-06-11T19:09:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.621+0000 7fe465e03640 1 -- start start 2024-06-11T19:09:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.621+0000 7fe465e03640 10 monclient: init 2024-06-11T19:09:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.621+0000 7f1bcf069640 1 Processor -- start 2024-06-11T19:09:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.621+0000 7f1bcf069640 1 -- start start 2024-06-11T19:09:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.621+0000 7f1bcf069640 10 monclient: init 2024-06-11T19:09:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 10 monclient: _add_conns ranks=[4,10,7,9,6,1,3,2,5,8,0] 2024-06-11T19:09:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc80a0cd0 0x7f1bc80a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 10 monclient(hunting): picked mon.noname-c con 0x7f1bc80a0cd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1bc8138c80 0x7f1bc8139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 10 monclient(hunting): picked mon.noname-f con 0x7f1bc8138c80 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1bc80a2250 0x7f1bc80a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 10 monclient: _add_conns ranks=[5,3,0,10,4,7,9,8,6,1,2] 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 0x7fcc540a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 10 monclient(hunting): picked mon.noname-h con 0x7fcc540a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcc540a5240 0x7fcc540a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 10 monclient(hunting): picked mon.noname-g con 0x7fcc540a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc540a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc540a7140 con 0x7fcc540a2530 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fcc540a2b50 con 0x7fcc540a5240 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fcc540a2e30 con 0x7fcc540a4920 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5c5c4640 10 monclient(hunting): authenticate will time out at 785.429484s 2024-06-11T19:09:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 10 monclient(hunting): picked mon.noname-i con 0x7f1bc80a2250 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcce05640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc80a0cd0 0x7f1bc80a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f1bc80739b0 con 0x7f1bc8138c80 2024-06-11T19:09:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcce05640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc80a0cd0 0x7f1bc80a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:40420/0 (socket says 172.21.15.98:40420) 2024-06-11T19:09:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcce05640 1 -- 172.21.15.98:0/1818967053 learned_addr learned my addr 172.21.15.98:0/1818967053 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 10 monclient: _add_conns ranks=[8,9,7,3,5,4,0,1,6,2,10] 2024-06-11T19:09:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4600a4920 0x7fe4600a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 10 monclient(hunting): picked mon.noname-e con 0x7fe4600a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe4600a5240 0x7fe4600a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bc7fff640 1 --2- 172.21.15.98:0/1818967053 >> v2:172.21.15.117:3300/0 conn(0x7f1bc8138c80 0x7f1bc8139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 10 monclient(hunting): picked mon.noname-a con 0x7fe4600a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4600a2530 0x7fe4600a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 10 monclient(hunting): picked mon.noname-i con 0x7fe4600a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe4600a7140 con 0x7fe4600a5240 2024-06-11T19:09:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fe4600a2b50 con 0x7fe4600a2530 2024-06-11T19:09:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe4600a2e30 con 0x7fe4600a4920 2024-06-11T19:09:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe465e03640 10 monclient(hunting): authenticate will time out at 785.429502s 2024-06-11T19:09:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe45effd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe4600a5240 0x7fe4600a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe45effd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe4600a5240 0x7fe4600a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45698/0 (socket says 172.21.15.98:45698) 2024-06-11T19:09:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe45effd640 1 -- 172.21.15.98:0/1692361617 learned_addr learned my addr 172.21.15.98:0/1692361617 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe45effd640 10 monclient(hunting): get_auth_request con 0x7fe4600a5240 auth_method 0 2024-06-11T19:09:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe45effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe45effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe45effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcd606640 1 --2- 172.21.15.98:0/1818967053 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1bc80a2250 0x7f1bc80a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe45ffff640 1 --2- 172.21.15.98:0/1692361617 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4600a2530 0x7fe4600a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe45ffff640 10 monclient(hunting): get_auth_request con 0x7fe4600a2530 auth_method 0 2024-06-11T19:09:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe45ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe45ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fe45ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5ab61640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcc540a2530 0x7fcc540a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53448/0 (socket says 172.21.15.98:53448) 2024-06-11T19:09:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7fcc5ab61640 1 -- 172.21.15.98:0/2129343388 learned_addr learned my addr 172.21.15.98:0/2129343388 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:09:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 1 --2- 172.21.15.98:0/2129343388 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 0x7fcc540a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 10 monclient(hunting): get_auth_request con 0x7fcc540a4920 auth_method 0 2024-06-11T19:09:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 1 -- 172.21.15.98:0/1818967053 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1bc809ecf0 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 1 -- 172.21.15.98:0/1818967053 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f1bc809efd0 con 0x7f1bc80a2250 2024-06-11T19:09:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcf069640 10 monclient(hunting): authenticate will time out at 785.429984s 2024-06-11T19:09:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcce05640 10 monclient(hunting): get_auth_request con 0x7f1bc80a0cd0 auth_method 0 2024-06-11T19:09:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.622+0000 7f1bcce05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bcce05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bcce05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 1 --2- 172.21.15.98:0/1692361617 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4600a4920 0x7fe4600a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 10 monclient(hunting): get_auth_request con 0x7fe4600a4920 auth_method 0 2024-06-11T19:09:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 10 monclient(hunting): get_auth_request con 0x7f1bc8138c80 auth_method 0 2024-06-11T19:09:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bcd606640 10 monclient(hunting): get_auth_request con 0x7f1bc80a2250 auth_method 0 2024-06-11T19:09:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bcd606640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bcd606640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bcd606640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 10 monclient(hunting): handle_auth_done global_id 4328 payload 274 2024-06-11T19:09:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f8f14e35640 10 monclient: get_monmap_and_config 2024-06-11T19:09:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f8f14e35640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 1 -- 172.21.15.98:0/1692361617 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4600a2530 msgr2=0x7fe4600a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-06-11T19:09:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 1 --2- 172.21.15.98:0/1692361617 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4600a2530 0x7fe4600a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 1 -- 172.21.15.98:0/2129343388 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcc540a5240 msgr2=0x7fcc540a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-11T19:09:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 1 --2- 172.21.15.98:0/2129343388 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcc540a5240 0x7fcc540a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 1 -- 172.21.15.98:0/2129343388 >> v1:172.21.15.98:6789/0 conn(0x7fcc540a2530 legacy=0x7fcc540a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 1 -- 172.21.15.98:0/1692361617 >> v2:172.21.15.98:3300/0 conn(0x7fe4600a5240 msgr2=0x7fe4600a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 1 monclient: found mon.noname-h 2024-06-11T19:09:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:09:35.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 1 --2- 172.21.15.98:0/1692361617 >> v2:172.21.15.98:3300/0 conn(0x7fe4600a5240 0x7fe4600a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f8f14e35640 10 monclient: monmap: 2024-06-11T19:09:35.662 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.662 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.662 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:35.624501+0000 2024-06-11T19:09:35.662 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:35.624501+0000 2024-06-11T19:09:35.662 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:35.663 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fcc5a360640 1 -- 172.21.15.98:0/2129343388 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc540a3110 con 0x7fcc540a4920 2024-06-11T19:09:35.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 1 monclient: found mon.noname-e 2024-06-11T19:09:35.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:35.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7fe45f7fe640 1 -- 172.21.15.98:0/1692361617 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4600a3110 con 0x7fe4600a4920 2024-06-11T19:09:35.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2024-06-11T19:09:35.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 1 -- 172.21.15.98:0/1818967053 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1bc80a2250 msgr2=0x7f1bc80a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 1 --2- 172.21.15.98:0/1818967053 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1bc80a2250 0x7f1bc80a26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 1 -- 172.21.15.98:0/1818967053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc80a0cd0 msgr2=0x7f1bc80a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 1 --2- 172.21.15.98:0/1818967053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc80a0cd0 0x7f1bc80a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 1 monclient: found mon.noname-f 2024-06-11T19:09:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:09:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.623+0000 7f1bc7fff640 1 -- 172.21.15.98:0/1818967053 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc809f2b0 con 0x7f1bc8138c80 2024-06-11T19:09:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7f1bcce05640 1 --2- 172.21.15.98:0/1818967053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc80a0cd0 0x7f1bc80a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7f8f14e35640 1 Processor -- start 2024-06-11T19:09:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7f8f14e35640 1 -- start start 2024-06-11T19:09:35.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7f8f14e35640 10 monclient: init 2024-06-11T19:09:35.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5a360640 1 --2- 172.21.15.98:0/2129343388 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 0x7fcc540a4d00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcc500097d0 tx=0x7fcc5002b790 comp rx=0 tx=0).ready entity=mon.5 client_cookie=6c4da4e409d40d3e server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 1 -- 172.21.15.98:0/2129343388 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcc50039070 con 0x7fcc540a4920 2024-06-11T19:09:35.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:09:35.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 10 monclient: dump: 2024-06-11T19:09:35.669 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.669 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.669 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.670 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 1 -- 172.21.15.98:0/2129343388 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc50031960 con 0x7fcc540a4920 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 1 -- 172.21.15.98:0/2129343388 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcc50032380 con 0x7fcc540a4920 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5935e640 10 monclient: dump: 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5c5c4640 5 monclient: authenticate success, global_id 4244 2024-06-11T19:09:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5c5c4640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fcc5c5c4640 4 set_mon_vals no callback set 2024-06-11T19:09:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.624+0000 7fe45f7fe640 1 --2- 172.21.15.98:0/1692361617 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4600a4920 0x7fe4600a4d00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe4500097a0 tx=0x7fe45002b6f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=e03c5b78b444d27a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 1 -- 172.21.15.98:0/1692361617 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe450037070 con 0x7fe4600a4920 2024-06-11T19:09:35.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 10 monclient: _add_conns ranks=[3,7,5,4,8,9,2,0,1,10,6] 2024-06-11T19:09:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 0x7f8f100a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f1bc7fff640 1 --2- 172.21.15.98:0/1818967053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bc8138c80 0x7f1bc8139060 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80097c0 tx=0x7f1bb802b6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cb75d63a1cc909c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 10 monclient(hunting): picked mon.noname-g con 0x7f8f100a52a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f1009d520 0x7f8f100a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 10 monclient(hunting): picked mon.noname-i con 0x7f8f1009d520 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fcc5c5c4640 10 monclient: shutdown 2024-06-11T19:09:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f10073830 0x7f8f100a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fcc5c5c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2129343388 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 msgr2=0x7fcc540a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fcc5c5c4640 1 --2- 172.21.15.98:0/2129343388 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 0x7fcc540a4d00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcc500097d0 tx=0x7fcc5002b790 comp rx=0 tx=0).stop 2024-06-11T19:09:35.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 10 monclient(hunting): picked mon.noname-h con 0x7f8f10073830 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:35.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f1bc77fe640 1 -- 172.21.15.98:0/1818967053 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1bb8037070 con 0x7f1bc8138c80 2024-06-11T19:09:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f1bc77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f0e59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 0x7f8f100a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f0dd9b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f1009d520 0x7f8f100a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f0e59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 0x7f8f100a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:53730/0 (socket says 172.21.15.98:53730) 2024-06-11T19:09:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f0dd9b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f1009d520 0x7f8f100a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:52948/0 (socket says 172.21.15.98:52948) 2024-06-11T19:09:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f0e59c640 1 -- 172.21.15.98:0/2178964740 learned_addr learned my addr 172.21.15.98:0/2178964740 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f1bc77fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:09:35.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f1bc77fe640 10 monclient: dump: 2024-06-11T19:09:35.678 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.678 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.678 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.678 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.678 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.679 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 1 -- 172.21.15.98:0/2178964740 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f8f1009daa0 con 0x7f8f100a52a0 2024-06-11T19:09:35.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f1bc77fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f1bc77fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f1bc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f0ed9d640 1 --2- 172.21.15.98:0/2178964740 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f10073830 0x7f8f100a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 1 -- 172.21.15.98:0/2178964740 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f8f1009d2e0 con 0x7f8f10073830 2024-06-11T19:09:35.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:35.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 10 monclient: dump: 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.682 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.683 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.683 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.683 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 1 -- 172.21.15.98:0/1692361617 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe450030980 con 0x7fe4600a4920 2024-06-11T19:09:35.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 1 -- 172.21.15.98:0/2178964740 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8f100a5680 con 0x7f8f1009d520 2024-06-11T19:09:35.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 1 -- 172.21.15.98:0/1692361617 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe450030bb0 con 0x7fe4600a4920 2024-06-11T19:09:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe45e7fc640 10 monclient: dump: 2024-06-11T19:09:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.684 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.685 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.685 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.685 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.685 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.685 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.685 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.685 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.685 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe465e03640 5 monclient: authenticate success, global_id 4328 2024-06-11T19:09:35.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe465e03640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:35.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7fe465e03640 4 set_mon_vals no callback set 2024-06-11T19:09:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f8f14e35640 10 monclient(hunting): authenticate will time out at 785.432994s 2024-06-11T19:09:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f1bc77fe640 1 -- 172.21.15.98:0/1818967053 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bb8030980 con 0x7f1bc8138c80 2024-06-11T19:09:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f1bcf069640 5 monclient: authenticate success, global_id 4321 2024-06-11T19:09:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.625+0000 7f1bcf069640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f1bc77fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request con 0x7f8f100a52a0 auth_method 0 2024-06-11T19:09:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2129343388 shutdown_connections 2024-06-11T19:09:35.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fcc5c5c4640 1 --2- 172.21.15.98:0/2129343388 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcc540a5240 0x7fcc540a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fcc5c5c4640 1 --2- 172.21.15.98:0/2129343388 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 0x7fcc540a4d00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f1bcf069640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:35.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f1bc77fe640 1 -- 172.21.15.98:0/1818967053 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1bb80313a0 con 0x7f1bc8138c80 2024-06-11T19:09:35.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f1bcf069640 4 set_mon_vals no callback set 2024-06-11T19:09:35.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f1bc77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f1bc77fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f1bc77fe640 10 monclient: dump: 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.689 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2129343388 >> 172.21.15.98:0/2129343388 conn(0x7fcc540ab850 msgr2=0x7fcc540a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0dd9b640 10 monclient(hunting): get_auth_request con 0x7f8f1009d520 auth_method 0 2024-06-11T19:09:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0dd9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2129343388 shutdown_connections 2024-06-11T19:09:35.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2129343388 wait complete. 2024-06-11T19:09:35.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0ed9d640 10 monclient(hunting): get_auth_request con 0x7f8f10073830 auth_method 0 2024-06-11T19:09:35.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0ed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fe465e03640 10 monclient: shutdown 2024-06-11T19:09:35.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fe465e03640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fe465e03640 1 -- 172.21.15.98:0/1692361617 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4600a4920 msgr2=0x7fe4600a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fe465e03640 1 --2- 172.21.15.98:0/1692361617 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4600a4920 0x7fe4600a4d00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe4500097a0 tx=0x7fe45002b6f0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-06-11T19:09:35.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 1 -- 172.21.15.98:0/2178964740 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f1009d520 msgr2=0x7f8f100a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 1 --2- 172.21.15.98:0/2178964740 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f1009d520 0x7f8f100a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 1 -- 172.21.15.98:0/2178964740 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f10073830 msgr2=0x7f8f100a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:35.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 1 --2- 172.21.15.98:0/2178964740 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f10073830 0x7f8f100a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 1 monclient: found mon.noname-g 2024-06-11T19:09:35.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:09:35.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0e59c640 1 -- 172.21.15.98:0/2178964740 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1009e9d0 con 0x7f8f100a52a0 2024-06-11T19:09:35.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f8f0dd9b640 1 --2- 172.21.15.98:0/2178964740 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f1009d520 0x7f8f100a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fe465e03640 1 -- 172.21.15.98:0/1692361617 shutdown_connections 2024-06-11T19:09:35.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fe465e03640 1 --2- 172.21.15.98:0/1692361617 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4600a2530 0x7fe4600a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fe465e03640 1 --2- 172.21.15.98:0/1692361617 >> v2:172.21.15.98:3300/0 conn(0x7fe4600a5240 0x7fe4600a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fe465e03640 1 --2- 172.21.15.98:0/1692361617 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4600a4920 0x7fe4600a4d00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fe465e03640 1 -- 172.21.15.98:0/1692361617 >> 172.21.15.98:0/1692361617 conn(0x7fe4600ab850 msgr2=0x7fe4600a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fe465e03640 1 -- 172.21.15.98:0/1692361617 shutdown_connections 2024-06-11T19:09:35.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fe465e03640 1 -- 172.21.15.98:0/1692361617 wait complete. 2024-06-11T19:09:35.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f1bcf069640 10 monclient: shutdown 2024-06-11T19:09:35.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f1bcf069640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f1bcf069640 1 -- 172.21.15.98:0/1818967053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bc8138c80 msgr2=0x7f1bc8139060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7f1bcf069640 1 --2- 172.21.15.98:0/1818967053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bc8138c80 0x7f1bc8139060 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80097c0 tx=0x7f1bb802b6d0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fcc5c5c4640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.626+0000 7fcc5c5c4640 10 monclient: monmap: 2024-06-11T19:09:35.696 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.696 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:35.696 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:35.696 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:35.696 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.696 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:35.696 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.697 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7f1bcf069640 1 -- 172.21.15.98:0/1818967053 shutdown_connections 2024-06-11T19:09:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7f1bcf069640 1 --2- 172.21.15.98:0/1818967053 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1bc80a2250 0x7f1bc80a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7f1bcf069640 1 --2- 172.21.15.98:0/1818967053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bc8138c80 0x7f1bc8139060 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7f1bcf069640 1 --2- 172.21.15.98:0/1818967053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc80a0cd0 0x7f1bc80a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7f1bcf069640 1 -- 172.21.15.98:0/1818967053 >> 172.21.15.98:0/1818967053 conn(0x7f1bc80a7850 msgr2=0x7f1bc80a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7f1bcf069640 1 -- 172.21.15.98:0/1818967053 shutdown_connections 2024-06-11T19:09:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7f1bcf069640 1 -- 172.21.15.98:0/1818967053 wait complete. 2024-06-11T19:09:35.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7fcc5c5c4640 1 librados: starting msgr at 2024-06-11T19:09:35.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7fcc5c5c4640 1 librados: starting objecter 2024-06-11T19:09:35.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7fe465e03640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7fcc5c5c4640 1 Processor -- start 2024-06-11T19:09:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7fe465e03640 10 monclient: monmap: 2024-06-11T19:09:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.701 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7fcc5c5c4640 1 -- start start 2024-06-11T19:09:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7fcc5c5c4640 1 librados: setting wanted keys 2024-06-11T19:09:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7fcc5c5c4640 1 librados: calling monclient init 2024-06-11T19:09:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7fcc5c5c4640 10 monclient: init 2024-06-11T19:09:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7f8f0e59c640 1 --2- 172.21.15.98:0/2178964740 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 0x7f8f100a1910 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8f000098d0 tx=0x7f8f0002fbc0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=268b84dc8951e813 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7f1bcf069640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7fe465e03640 1 librados: starting msgr at 2024-06-11T19:09:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7f1bcf069640 10 monclient: monmap: 2024-06-11T19:09:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.704 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7fe465e03640 1 librados: starting objecter 2024-06-11T19:09:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7f8f0d59a640 1 -- 172.21.15.98:0/2178964740 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f0003d070 con 0x7f8f100a52a0 2024-06-11T19:09:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f0d59a640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:09:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f0d59a640 10 monclient: dump: 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.707 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f0d59a640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f0d59a640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f0d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f1bcf069640 1 librados: starting msgr at 2024-06-11T19:09:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f1bcf069640 1 librados: starting objecter 2024-06-11T19:09:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f0d59a640 1 -- 172.21.15.98:0/2178964740 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f00030be0 con 0x7f8f100a52a0 2024-06-11T19:09:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f14e35640 5 monclient: authenticate success, global_id 4296 2024-06-11T19:09:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f14e35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f0d59a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f0d59a640 1 -- 172.21.15.98:0/2178964740 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f00030e10 con 0x7f8f100a52a0 2024-06-11T19:09:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f1bcf069640 1 Processor -- start 2024-06-11T19:09:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f14e35640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f8f14e35640 4 set_mon_vals no callback set 2024-06-11T19:09:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f1bcf069640 1 -- start start 2024-06-11T19:09:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f1bcf069640 1 librados: setting wanted keys 2024-06-11T19:09:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f1bcf069640 1 librados: calling monclient init 2024-06-11T19:09:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7f1bcf069640 10 monclient: init 2024-06-11T19:09:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 10 monclient: _add_conns ranks=[6,5,2,4,1,0,3,8,7] 2024-06-11T19:09:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcc540a2530 0x7fcc541cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 10 monclient(hunting): picked mon.noname-d con 0x7fcc540a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 0x7fcc541ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 10 monclient(hunting): picked mon.noname-h con 0x7fcc540a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc540a5240 0x7fcc54121e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 10 monclient(hunting): picked mon.noname-b con 0x7fcc540a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcc540a6da0 con 0x7fcc540a5240 2024-06-11T19:09:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fcc54138e50 con 0x7fcc540a4920 2024-06-11T19:09:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f14e35640 10 monclient: shutdown 2024-06-11T19:09:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f0d59a640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:09:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 10 monclient: _add_conns ranks=[6,7,4,0,1,3,2,8,5] 2024-06-11T19:09:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1bc80a0cd0 0x7f1bc81d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f0d59a640 10 monclient: dump: 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.715 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f14e35640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f14e35640 1 -- 172.21.15.98:0/2178964740 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 msgr2=0x7f8f100a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f14e35640 1 --2- 172.21.15.98:0/2178964740 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 0x7f8f100a1910 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8f000098d0 tx=0x7f8f0002fbc0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fcc540a65f0 con 0x7fcc540a2530 2024-06-11T19:09:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5c5c4640 10 monclient(hunting): authenticate will time out at 785.435148s 2024-06-11T19:09:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 0x7fcc541ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 0x7fcc541ce120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:40276/0 (socket says 172.21.15.98:40276) 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 1 -- 172.21.15.98:0/2571719973 learned_addr learned my addr 172.21.15.98:0/2571719973 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 10 monclient(hunting): get_auth_request con 0x7fcc540a4920 auth_method 0 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5a360640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcc540a2530 0x7fcc541cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5a360640 10 monclient(hunting): get_auth_request con 0x7fcc540a2530 auth_method 0 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5a360640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5a360640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5a360640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5ab61640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc540a5240 0x7fcc54121e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5a360640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5a360640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5a360640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 10 monclient(hunting): picked mon.noname-d con 0x7f1bc80a0cd0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1bc80a2250 0x7f1bc81d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5ab61640 10 monclient(hunting): get_auth_request con 0x7fcc540a5240 auth_method 0 2024-06-11T19:09:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5ab61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5ab61640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5ab61640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5ab61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5ab61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5ab61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 10 monclient(hunting): handle_auth_done global_id 4262 payload 824 2024-06-11T19:09:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 1 -- 172.21.15.98:0/2571719973 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcc540a2530 msgr2=0x7fcc541cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcc540a2530 0x7fcc541cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 1 -- 172.21.15.98:0/2571719973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc540a5240 msgr2=0x7fcc54121e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc540a5240 0x7fcc54121e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 1 monclient: found mon.noname-h 2024-06-11T19:09:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:09:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc59b5f640 1 -- 172.21.15.98:0/2571719973 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc54125b60 con 0x7fcc540a4920 2024-06-11T19:09:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fcc5a360640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcc540a2530 0x7fcc541cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc5ab61640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc540a5240 0x7fcc54121e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc59b5f640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 0x7fcc541ce120 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009fd0 tx=0x7fcc4401c880 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 10 client.?.objecter ms_handle_connect 0x7fcc540a4920 2024-06-11T19:09:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 10 monclient(hunting): picked mon.noname-i con 0x7f1bc80a2250 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcce05640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1bc80a0cd0 0x7f1bc81d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc8138c80 0x7f1bc806f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcce05640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1bc80a0cd0 0x7f1bc81d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49766/0 (socket says 172.21.15.98:49766) 2024-06-11T19:09:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 10 monclient(hunting): picked mon.noname-c con 0x7f1bc8138c80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.627+0000 7fe465e03640 1 Processor -- start 2024-06-11T19:09:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 1 -- start start 2024-06-11T19:09:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 1 librados: setting wanted keys 2024-06-11T19:09:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 1 librados: calling monclient init 2024-06-11T19:09:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 10 monclient: init 2024-06-11T19:09:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1bc80a1770 con 0x7f1bc8138c80 2024-06-11T19:09:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1bc80a15f0 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 10 monclient: _add_conns ranks=[1,2,0,3,8,6,4,7,5] 2024-06-11T19:09:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4600a2530 0x7fe460136d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 10 monclient(hunting): picked mon.noname-f con 0x7fe4600a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4600a5240 0x7fe4601372c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 10 monclient(hunting): picked mon.noname-b con 0x7fe4600a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f1bc80a2c20 con 0x7f1bc80a2250 2024-06-11T19:09:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcf069640 10 monclient(hunting): authenticate will time out at 785.436512s 2024-06-11T19:09:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcce05640 1 -- 172.21.15.98:0/2235947377 learned_addr learned my addr 172.21.15.98:0/2235947377 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcce05640 10 monclient(hunting): get_auth_request con 0x7f1bc80a0cd0 auth_method 0 2024-06-11T19:09:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcce05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4601227a0 0x7fe460122c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 10 monclient(hunting): picked mon.noname-a con 0x7fe4601227a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe4600a6da0 con 0x7fe4601227a0 2024-06-11T19:09:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe460138e50 con 0x7fe4600a2530 2024-06-11T19:09:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe4600a6420 con 0x7fe4600a5240 2024-06-11T19:09:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.628+0000 7fe465e03640 10 monclient(hunting): authenticate will time out at 785.435999s 2024-06-11T19:09:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4600a5240 0x7fe4601372c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcce05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcce05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4600a5240 0x7fe4601372c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47970/0 (socket says 172.21.15.98:47970) 2024-06-11T19:09:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 1 -- 172.21.15.98:0/1281138208 learned_addr learned my addr 172.21.15.98:0/1281138208 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 10 monclient(hunting): get_auth_request con 0x7fe4600a5240 auth_method 0 2024-06-11T19:09:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f14e35640 1 -- 172.21.15.98:0/2178964740 shutdown_connections 2024-06-11T19:09:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f14e35640 1 --2- 172.21.15.98:0/2178964740 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f10073830 0x7f8f100a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcd606640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc8138c80 0x7f1bc806f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcd606640 10 monclient(hunting): get_auth_request con 0x7f1bc8138c80 auth_method 0 2024-06-11T19:09:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f14e35640 1 --2- 172.21.15.98:0/2178964740 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f1009d520 0x7f8f100a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bc7fff640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1bc80a2250 0x7f1bc81d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcd606640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcd606640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcd606640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bc7fff640 10 monclient(hunting): get_auth_request con 0x7f1bc80a2250 auth_method 0 2024-06-11T19:09:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bc7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f14e35640 1 --2- 172.21.15.98:0/2178964740 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 0x7f8f100a1910 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 1 -- 172.21.15.98:0/2571719973 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcc44022ab0 con 0x7fcc540a4920 2024-06-11T19:09:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f14e35640 1 -- 172.21.15.98:0/2178964740 >> 172.21.15.98:0/2178964740 conn(0x7f8f100a4780 msgr2=0x7f8f100a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:09:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 10 monclient: dump: 2024-06-11T19:09:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.731 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcd606640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcd606640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcd606640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 1 -- 172.21.15.98:0/2571719973 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc44027070 con 0x7fcc540a4920 2024-06-11T19:09:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 1 -- 172.21.15.98:0/2571719973 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcc44022c70 con 0x7fcc540a4920 2024-06-11T19:09:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcce05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcce05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f1bcce05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc58b5d640 4 set_mon_vals no callback set 2024-06-11T19:09:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc4affd640 10 monclient: dump: 2024-06-11T19:09:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.734 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2024-06-11T19:09:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc5c5c4640 5 monclient: authenticate success, global_id 4262 2024-06-11T19:09:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc5c5c4640 10 monclient: _renew_subs 2024-06-11T19:09:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc5c5c4640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2571719973 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc54137710 con 0x7fcc540a4920 2024-06-11T19:09:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc5c5c4640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 1 -- 172.21.15.98:0/1281138208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4600a2530 msgr2=0x7fe460136d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:09:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4600a2530 0x7fe460136d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc5c5c4640 10 monclient: _renew_subs 2024-06-11T19:09:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc5c5c4640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2571719973 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc541255a0 con 0x7fcc540a4920 2024-06-11T19:09:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fcc5c5c4640 1 librados: init done 2024-06-11T19:09:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45effd640 1 -- 172.21.15.98:0/1281138208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4601227a0 msgr2=0x7fe460122c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:09:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7fe45ffff640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4601227a0 0x7fe460122c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bcce05640 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2024-06-11T19:09:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bcce05640 1 -- 172.21.15.98:0/2235947377 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1bc80a2250 msgr2=0x7f1bc81d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bcce05640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1bc80a2250 0x7f1bc81d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bcce05640 1 -- 172.21.15.98:0/2235947377 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc8138c80 msgr2=0x7f1bc806f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bcce05640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc8138c80 0x7f1bc806f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bcce05640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bcce05640 1 monclient: found mon.noname-d 2024-06-11T19:09:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bcce05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bcce05640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:09:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bcce05640 1 -- 172.21.15.98:0/2235947377 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc8073000 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bc7fff640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1bc80a2250 0x7f1bc81d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe45effd640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4601227a0 0x7fe460122c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe45effd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe45effd640 1 monclient: found mon.noname-b 2024-06-11T19:09:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe45effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe45effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe45effd640 1 -- 172.21.15.98:0/1281138208 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe460126760 con 0x7fe4600a5240 2024-06-11T19:09:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.629+0000 7f8f14e35640 1 -- 172.21.15.98:0/2178964740 shutdown_connections 2024-06-11T19:09:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f8f14e35640 1 -- 172.21.15.98:0/2178964740 wait complete. 2024-06-11T19:09:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe45ffff640 10 monclient: get_auth_request con 0x7fe4601227a0 auth_method 0 2024-06-11T19:09:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe45ffff640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4601227a0 0x7fe460122c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe45effd640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4600a5240 0x7fe4601372c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe45400e9a0 tx=0x7fe45400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 10 client.?.objecter ms_handle_connect 0x7fe4600a5240 2024-06-11T19:09:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bcce05640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1bc80a0cd0 0x7f1bc81d1d70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00d750 tx=0x7f1bbc00dc10 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bc57fa640 10 client.?.objecter ms_handle_connect 0x7f1bc80a0cd0 2024-06-11T19:09:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bc57fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7f1bcd606640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc8138c80 0x7f1bc806f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 1 -- 172.21.15.98:0/1281138208 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe454017070 con 0x7fe4600a5240 2024-06-11T19:09:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 10 monclient: dump: 2024-06-11T19:09:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 1 -- 172.21.15.98:0/1281138208 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe45400cc60 con 0x7fe4600a5240 2024-06-11T19:09:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 1 -- 172.21.15.98:0/1281138208 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe45400ce50 con 0x7fe4600a5240 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe447fff640 10 monclient: dump: 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.744 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.630+0000 7fe45dffb640 4 set_mon_vals no callback set 2024-06-11T19:09:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f1bc57fa640 1 -- 172.21.15.98:0/2235947377 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1bbc00de60 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f1bc57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fe465e03640 5 monclient: authenticate success, global_id 4331 2024-06-11T19:09:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f1bc57fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:09:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f1bc57fa640 10 monclient: dump: 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.747 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f1bc57fa640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f1bc57fa640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fcc5c5c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f1bc57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fcc5c5c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f1bc57fa640 1 -- 172.21.15.98:0/2235947377 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bbc0044a0 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fcc5c5c4640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f1bcf069640 5 monclient: authenticate success, global_id 4299 2024-06-11T19:09:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f8f14e35640 10 monclient: build_initial_monmap 2024-06-11T19:09:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f1bcf069640 10 monclient: _renew_subs 2024-06-11T19:09:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f1bcf069640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:09:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f1bcf069640 1 -- 172.21.15.98:0/2235947377 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bc8139060 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2571719973 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc540a4d00 con 0x7fcc540a4920 2024-06-11T19:09:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f8f14e35640 10 monclient: monmap: 2024-06-11T19:09:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:35.751 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fe465e03640 10 monclient: _renew_subs 2024-06-11T19:09:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fe465e03640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fe465e03640 1 -- 172.21.15.98:0/1281138208 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4600a94d0 con 0x7fe4600a5240 2024-06-11T19:09:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fe465e03640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fe465e03640 10 monclient: _renew_subs 2024-06-11T19:09:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fe465e03640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fe465e03640 1 -- 172.21.15.98:0/1281138208 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe460124d20 con 0x7fe4600a5240 2024-06-11T19:09:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7fe465e03640 1 librados: init done 2024-06-11T19:09:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f8f14e35640 1 librados: starting msgr at 2024-06-11T19:09:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f8f14e35640 1 librados: starting objecter 2024-06-11T19:09:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.631+0000 7f8f14e35640 1 Processor -- start 2024-06-11T19:09:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f1bcf069640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f1bcf069640 10 monclient: _renew_subs 2024-06-11T19:09:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f1bcf069640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:09:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f1bcf069640 1 -- 172.21.15.98:0/2235947377 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bc8070ff0 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f1bcf069640 1 librados: init done 2024-06-11T19:09:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 1 -- start start 2024-06-11T19:09:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 1 librados: setting wanted keys 2024-06-11T19:09:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 1 librados: calling monclient init 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 10 monclient: init 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 10 monclient: _add_conns ranks=[8,0,3,2,5,4,6,7,1] 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f10073830 0x7f8f101d1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 10 monclient(hunting): picked mon.noname-e con 0x7f8f10073830 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f1009d520 0x7f8f10132800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 10 monclient(hunting): picked mon.noname-a con 0x7f8f1009d520 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 0x7f8f10132d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 10 monclient(hunting): picked mon.noname-g con 0x7f8f100a52a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f100a1fd0 con 0x7f8f1009d520 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f8f100a1e50 con 0x7f8f100a52a0 2024-06-11T19:09:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8f100a2150 con 0x7f8f10073830 2024-06-11T19:09:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f8f14e35640 10 monclient(hunting): authenticate will time out at 785.439894s 2024-06-11T19:09:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f1bc57fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7f1bc6ffd640 4 set_mon_vals no callback set 2024-06-11T19:09:35.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7fe465e03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7fe465e03640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7fe465e03640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7fcc4affd640 1 -- 172.21.15.98:0/2571719973 <== mon.5 v2:172.21.15.117:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fcc44022640 con 0x7fcc540a4920 2024-06-11T19:09:35.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7fe465e03640 1 -- 172.21.15.98:0/1281138208 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4600a6750 con 0x7fe4600a5240 2024-06-11T19:09:35.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f10073830 0x7f8f101d1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f10073830 0x7f8f101d1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54516/0 (socket says 172.21.15.98:54516) 2024-06-11T19:09:35.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 1 -- 172.21.15.98:0/3050944562 learned_addr learned my addr 172.21.15.98:0/3050944562 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:35.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request con 0x7f8f10073830 auth_method 0 2024-06-11T19:09:35.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.632+0000 7fcc4affd640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcc3803fdb0 0x7fcc38042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7fcc4affd640 1 -- 172.21.15.98:0/2571719973 <== mon.5 v2:172.21.15.117:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fcc4401f070 con 0x7fcc540a4920 2024-06-11T19:09:35.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7fcc4affd640 10 client.4262.objecter ms_dispatch 0x7fcc54000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:35.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:35.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:35.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f1bc57fa640 1 -- 172.21.15.98:0/2235947377 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1bbc00bce0 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f1bc57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f1bc57fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f1bc57fa640 10 monclient: dump: 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.761 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7fcc4affd640 3 client.4262.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7fcc4affd640 3 client.4262.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7fcc4affd640 20 client.4262.objecter dump_active .. 0 homeless 2024-06-11T19:09:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f1bc57fa640 1 -- 172.21.15.98:0/2235947377 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f1bbc004030 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7fcc5a360640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcc3803fdb0 0x7fcc38042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7fcc5a360640 10 monclient: get_auth_request con 0x7fcc3803fdb0 auth_method 0 2024-06-11T19:09:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0dd9b640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f1009d520 0x7f8f10132800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0dd9b640 10 monclient(hunting): get_auth_request con 0x7f8f1009d520 auth_method 0 2024-06-11T19:09:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0dd9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f1bcf069640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f1bcf069640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f1bcf069640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:09:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f1bcf069640 1 -- 172.21.15.98:0/2235947377 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bc80a8520 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-06-11T19:09:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0ed9d640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 0x7f8f10132d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 1 -- 172.21.15.98:0/3050944562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 msgr2=0x7f8f10132d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f1bc57fa640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1bb403fe00 0x7f1bb40422c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 0x7f8f10132d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 1 -- 172.21.15.98:0/3050944562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f1009d520 msgr2=0x7f8f10132800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f1009d520 0x7f8f10132800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 10 monclient: _finish_hunting 0 2024-06-11T19:09:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 1 monclient: found mon.noname-e 2024-06-11T19:09:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0e59c640 1 -- 172.21.15.98:0/3050944562 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f10128a60 con 0x7f8f10073830 2024-06-11T19:09:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.633+0000 7f8f0dd9b640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f1009d520 0x7f8f10132800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8f0ed9d640 10 monclient: get_auth_request con 0x7f8f100a52a0 auth_method 0 2024-06-11T19:09:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f1bc7fff640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1bb403fe00 0x7f1bb40422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f1bc7fff640 10 monclient: get_auth_request con 0x7f1bb403fe00 auth_method 0 2024-06-11T19:09:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7fcc5a360640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcc3803fdb0 0x7fcc38042270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc50036000 tx=0x7fcc50008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7fcc4affd640 10 client.4262.objecter ms_handle_connect 0x7fcc3803fdb0 2024-06-11T19:09:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8f0ed9d640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 0x7f8f10132d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f1bc57fa640 1 -- 172.21.15.98:0/2235947377 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f1bbc00f040 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f1bc57fa640 10 client.4299.objecter ms_dispatch 0x7f1bc8000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8f0e59c640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f10073830 0x7f8f101d1fe0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009870 tx=0x7f8f000046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 10 client.?.objecter ms_handle_connect 0x7f8f10073830 2024-06-11T19:09:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 1 -- 172.21.15.98:0/3050944562 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f00046070 con 0x7f8f10073830 2024-06-11T19:09:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f1bc57fa640 3 client.4299.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f1bc57fa640 3 client.4299.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f1bc57fa640 20 client.4299.objecter dump_active .. 0 homeless 2024-06-11T19:09:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7fe447fff640 1 -- 172.21.15.98:0/1281138208 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fe45401a070 con 0x7fe4600a5240 2024-06-11T19:09:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7fe447fff640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe44c03ff80 0x7fe44c042440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7fe447fff640 1 -- 172.21.15.98:0/1281138208 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fe45400f040 con 0x7fe4600a5240 2024-06-11T19:09:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7fe447fff640 10 client.4331.objecter ms_dispatch 0x7fe460000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7fe447fff640 3 client.4331.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7fe447fff640 3 client.4331.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 10 monclient: dump: 2024-06-11T19:09:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.770 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7fe447fff640 20 client.4331.objecter dump_active .. 0 homeless 2024-06-11T19:09:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 10 monclient: _finish_auth 0 2024-06-11T19:09:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8f14e35640 5 monclient: authenticate success, global_id 4337 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 1 -- 172.21.15.98:0/3050944562 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f00003690 con 0x7f8f10073830 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8f14e35640 10 monclient: _renew_subs 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8f14e35640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8f14e35640 1 -- 172.21.15.98:0/3050944562 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f10133280 con 0x7f8f10073830 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 1 -- 172.21.15.98:0/3050944562 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f0003d030 con 0x7f8f10073830 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8f0cd99640 4 set_mon_vals no callback set 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f8efeffd640 10 monclient: dump: 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:35.773 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f1bc7fff640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1bb403fe00 0x7f1bb40422c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80097c0 tx=0x7f1bb8008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.634+0000 7f1bc57fa640 10 client.4299.objecter ms_handle_connect 0x7f1bb403fe00 2024-06-11T19:09:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.635+0000 7f8f14e35640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.635+0000 7f8f14e35640 10 monclient: _renew_subs 2024-06-11T19:09:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.635+0000 7f8f14e35640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.635+0000 7f8f14e35640 1 -- 172.21.15.98:0/3050944562 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f101299c0 con 0x7f8f10073830 2024-06-11T19:09:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.635+0000 7f8f14e35640 1 librados: init done 2024-06-11T19:09:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.635+0000 7fe45f7fe640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe44c03ff80 0x7fe44c042440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.635+0000 7fe45f7fe640 10 monclient: get_auth_request con 0x7fe44c03ff80 auth_method 0 2024-06-11T19:09:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.636+0000 7fe447fff640 10 client.4331.objecter ms_handle_connect 0x7fe44c03ff80 2024-06-11T19:09:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.636+0000 7fe45f7fe640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe44c03ff80 0x7fe44c042440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe450034000 tx=0x7fe450008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.636+0000 7f8f14e35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.636+0000 7f8f14e35640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.636+0000 7f8f14e35640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.636+0000 7f8f14e35640 1 -- 172.21.15.98:0/3050944562 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f10129c00 con 0x7f8f10073830 2024-06-11T19:09:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.637+0000 7f8efeffd640 1 -- 172.21.15.98:0/3050944562 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8f00012300 con 0x7f8f10073830 2024-06-11T19:09:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.637+0000 7f8efeffd640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8ef003fdb0 0x7f8ef0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.637+0000 7f8efeffd640 1 -- 172.21.15.98:0/3050944562 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f8f0003e070 con 0x7f8f10073830 2024-06-11T19:09:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.637+0000 7f8efeffd640 10 client.4337.objecter ms_dispatch 0x7f8f10000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.637+0000 7f8efeffd640 3 client.4337.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:35.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.637+0000 7f8efeffd640 3 client.4337.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.637+0000 7f8efeffd640 20 client.4337.objecter dump_active .. 0 homeless 2024-06-11T19:09:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.637+0000 7f8f0dd9b640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8ef003fdb0 0x7f8ef0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.637+0000 7f8f0dd9b640 10 monclient: get_auth_request con 0x7f8ef003fdb0 auth_method 0 2024-06-11T19:09:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.637+0000 7f8f0dd9b640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8ef003fdb0 0x7f8ef0042270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f10072fb0 tx=0x7f8f04008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.638+0000 7f8efeffd640 10 client.4337.objecter ms_handle_connect 0x7f8ef003fdb0 2024-06-11T19:09:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.637+0000 7fcc4affd640 1 -- 172.21.15.98:0/2571719973 <== mon.5 v2:172.21.15.117:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcc44022e10 con 0x7fcc540a4920 2024-06-11T19:09:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.638+0000 7fcc4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.638+0000 7fcc4affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.639+0000 7f1bc57fa640 1 -- 172.21.15.98:0/2235947377 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1bbc012a40 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.639+0000 7f1bc57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.639+0000 7f1bc57fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.643+0000 7fe447fff640 1 -- 172.21.15.98:0/1281138208 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe454012e20 con 0x7fe4600a5240 2024-06-11T19:09:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.643+0000 7fe447fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.643+0000 7fe447fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.646+0000 7f8efeffd640 1 -- 172.21.15.98:0/3050944562 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8f0000fcd0 con 0x7f8f10073830 2024-06-11T19:09:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.646+0000 7f8efeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.646+0000 7f8efeffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:35.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.841+0000 7fb0bba2e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-06-11T19:09:35.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.842+0000 7fb0bba2e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-06-11T19:09:35.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.842+0000 7fb0bba2e640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:09:35.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.842+0000 7fb0bba2e640 1 -- 172.21.15.98:0/3818742412 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb0b412dfe0 con 0x7fb0b40766a0 2024-06-11T19:09:35.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.842+0000 7fb0a9ffb640 1 -- 172.21.15.98:0/3818742412 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb0b412dfe0 con 0x7fb0b40766a0 2024-06-11T19:09:35.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.842+0000 7fb0a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-06-11T19:09:35.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.842+0000 7fb0a9ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:35.842 INFO:teuthology.orchestra.run.smithi098.stdout:51539607558 2024-06-11T19:09:35.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 10 librados: watch_flush enter 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 10 librados: watch_flush exit 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 20 client.4312.objecter shutdown clearing up homeless session... 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 10 client.4312.objecter successfully canceled tick 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 1 -- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb09803fde0 msgr2=0x7fb0980422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb09803fde0 0x7fb0980422a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb0b409e050 tx=0x7fb0ac012040 comp rx=0 tx=0).stop 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 10 monclient: shutdown 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 1 -- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0b40766a0 msgr2=0x7fb0b409a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0b40766a0 0x7fb0b409a4f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00090b0 tx=0x7fb0b00046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 1 -- 172.21.15.98:0/3818742412 shutdown_connections 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb09803fde0 0x7fb0980422a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0b409f450 0x7fb0b41c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0b4076a80 0x7fb0b409aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 1 --2- 172.21.15.98:0/3818742412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0b40766a0 0x7fb0b409a4f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.844+0000 7fb0bba2e640 1 -- 172.21.15.98:0/3818742412 >> 172.21.15.98:0/3818742412 conn(0x7fb0b40a7850 msgr2=0x7fb0b40a21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.845+0000 7fb0bba2e640 1 -- 172.21.15.98:0/3818742412 shutdown_connections 2024-06-11T19:09:35.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.845+0000 7fb0bba2e640 1 -- 172.21.15.98:0/3818742412 wait complete. 2024-06-11T19:09:35.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.845+0000 7fb0bba2e640 1 librados: shutdown 2024-06-11T19:09:35.855 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.1 2024-06-11T19:09:35.855 DEBUG:teuthology.parallel:result is None 2024-06-11T19:09:35.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.871+0000 7fe465e03640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-06-11T19:09:35.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.871+0000 7fe465e03640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-06-11T19:09:35.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.871+0000 7fe465e03640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:35.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.871+0000 7fe465e03640 1 -- 172.21.15.98:0/1281138208 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fe4600008d0 con 0x7fe4600a5240 2024-06-11T19:09:35.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.871+0000 7fe447fff640 1 -- 172.21.15.98:0/1281138208 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe45401ac60 con 0x7fe4600a5240 2024-06-11T19:09:35.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.871+0000 7fe447fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-06-11T19:09:35.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.871+0000 7fe447fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:35.873 INFO:teuthology.orchestra.run.smithi098.stdout:51539607556 2024-06-11T19:09:35.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 10 librados: watch_flush enter 2024-06-11T19:09:35.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 10 librados: watch_flush exit 2024-06-11T19:09:35.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 20 client.4331.objecter shutdown clearing up homeless session... 2024-06-11T19:09:35.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 10 client.4331.objecter successfully canceled tick 2024-06-11T19:09:35.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 1 -- 172.21.15.98:0/1281138208 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe44c03ff80 msgr2=0x7fe44c042440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe44c03ff80 0x7fe44c042440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe450034000 tx=0x7fe450008030 comp rx=0 tx=0).stop 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 10 monclient: shutdown 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 1 -- 172.21.15.98:0/1281138208 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4600a5240 msgr2=0x7fe4601372c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4600a5240 0x7fe4601372c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe45400e9a0 tx=0x7fe45400ee60 comp rx=0 tx=0).stop 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 1 -- 172.21.15.98:0/1281138208 shutdown_connections 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe44c03ff80 0x7fe44c042440 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4601227a0 0x7fe460122c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4600a5240 0x7fe4601372c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 1 --2- 172.21.15.98:0/1281138208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4600a2530 0x7fe460136d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 1 -- 172.21.15.98:0/1281138208 >> 172.21.15.98:0/1281138208 conn(0x7fe4600ab850 msgr2=0x7fe4600726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.873+0000 7fe465e03640 1 -- 172.21.15.98:0/1281138208 shutdown_connections 2024-06-11T19:09:35.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.874+0000 7fe465e03640 1 -- 172.21.15.98:0/1281138208 wait complete. 2024-06-11T19:09:35.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.874+0000 7fe465e03640 1 librados: shutdown 2024-06-11T19:09:35.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.2 2024-06-11T19:09:35.884 DEBUG:teuthology.parallel:result is None 2024-06-11T19:09:35.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.982+0000 7f42234d1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-06-11T19:09:35.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.982+0000 7f42234d1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-06-11T19:09:35.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.982+0000 7f42234d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:35.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.982+0000 7f42234d1640 1 -- 172.21.15.98:0/4231906019 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f421c127b60 con 0x7f421c0a4df0 2024-06-11T19:09:35.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.983+0000 7f4211ffb640 1 -- 172.21.15.98:0/4231906019 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f420c011df0 con 0x7f421c0a4df0 2024-06-11T19:09:35.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.983+0000 7f4211ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-06-11T19:09:35.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.983+0000 7f4211ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:35.983 INFO:teuthology.orchestra.run.smithi098.stdout:47244640261 2024-06-11T19:09:35.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.985+0000 7fcc5c5c4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-06-11T19:09:35.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.985+0000 7fcc5c5c4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-06-11T19:09:35.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.985+0000 7fcc5c5c4640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:35.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.985+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2571719973 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fcc54124400 con 0x7fcc540a4920 2024-06-11T19:09:35.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.985+0000 7fcc4affd640 1 -- 172.21.15.98:0/2571719973 <== mon.5 v2:172.21.15.117:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcc4402a360 con 0x7fcc540a4920 2024-06-11T19:09:35.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.985+0000 7fcc4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-06-11T19:09:35.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.985+0000 7fcc4affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:35.985 INFO:teuthology.orchestra.run.smithi098.stdout:47244640262 2024-06-11T19:09:35.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.985+0000 7f42234d1640 10 librados: watch_flush enter 2024-06-11T19:09:35.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 10 librados: watch_flush exit 2024-06-11T19:09:35.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 20 client.4324.objecter shutdown clearing up homeless session... 2024-06-11T19:09:35.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 10 client.4324.objecter successfully canceled tick 2024-06-11T19:09:35.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 -- 172.21.15.98:0/4231906019 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f420003fdb0 msgr2=0x7f4200042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f420003fdb0 0x7f4200042270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f421802bd80 tx=0x7f4218008030 comp rx=0 tx=0).stop 2024-06-11T19:09:35.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 10 monclient: shutdown 2024-06-11T19:09:35.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 -- 172.21.15.98:0/4231906019 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f421c0a4df0 msgr2=0x7f421c1360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f421c0a4df0 0x7f421c1360c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f420c00c390 tx=0x7f420c00c850 comp rx=0 tx=0).stop 2024-06-11T19:09:35.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 -- 172.21.15.98:0/4231906019 shutdown_connections 2024-06-11T19:09:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f420003fdb0 0x7f4200042270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f421c0a4df0 0x7f421c1360c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f421c0a51d0 0x7f421c125700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 --2- 172.21.15.98:0/4231906019 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f421c0a2f40 0x7f421c135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 -- 172.21.15.98:0/4231906019 >> 172.21.15.98:0/4231906019 conn(0x7f421c0a95b0 msgr2=0x7f421c09def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 -- 172.21.15.98:0/4231906019 shutdown_connections 2024-06-11T19:09:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 -- 172.21.15.98:0/4231906019 wait complete. 2024-06-11T19:09:35.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.986+0000 7f42234d1640 1 librados: shutdown 2024-06-11T19:09:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.987+0000 7fcc5c5c4640 10 librados: watch_flush enter 2024-06-11T19:09:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.987+0000 7fcc5c5c4640 10 librados: watch_flush exit 2024-06-11T19:09:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.987+0000 7fcc5c5c4640 20 client.4262.objecter shutdown clearing up homeless session... 2024-06-11T19:09:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.987+0000 7fcc5c5c4640 10 client.4262.objecter successfully canceled tick 2024-06-11T19:09:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2571719973 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcc3803fdb0 msgr2=0x7fcc38042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcc3803fdb0 0x7fcc38042270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc50036000 tx=0x7fcc50008030 comp rx=0 tx=0).stop 2024-06-11T19:09:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 10 monclient: shutdown 2024-06-11T19:09:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2571719973 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 msgr2=0x7fcc541ce120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 0x7fcc541ce120 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009fd0 tx=0x7fcc4401c880 comp rx=0 tx=0).stop 2024-06-11T19:09:35.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2571719973 shutdown_connections 2024-06-11T19:09:35.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcc3803fdb0 0x7fcc38042270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcc540a5240 0x7fcc54121e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcc540a4920 0x7fcc541ce120 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 1 --2- 172.21.15.98:0/2571719973 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcc540a2530 0x7fcc541cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2571719973 >> 172.21.15.98:0/2571719973 conn(0x7fcc540ab850 msgr2=0x7fcc54137960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2571719973 shutdown_connections 2024-06-11T19:09:35.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.988+0000 7fcc5c5c4640 1 -- 172.21.15.98:0/2571719973 wait complete. 2024-06-11T19:09:35.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.989+0000 7fcc5c5c4640 1 librados: shutdown 2024-06-11T19:09:35.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.989+0000 7f1bcf069640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-06-11T19:09:35.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.989+0000 7f1bcf069640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-06-11T19:09:35.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.989+0000 7f1bcf069640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:09:35.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.989+0000 7f1bcf069640 1 -- 172.21.15.98:0/2235947377 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f1bc8068c60 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.989+0000 7f1bc57fa640 1 -- 172.21.15.98:0/2235947377 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1bbc0523c0 con 0x7f1bc80a0cd0 2024-06-11T19:09:35.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.989+0000 7f1bc57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-06-11T19:09:35.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.989+0000 7f1bc57fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:35.992 INFO:teuthology.orchestra.run.smithi098.stdout:47244640261 2024-06-11T19:09:35.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 10 librados: watch_flush enter 2024-06-11T19:09:35.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 10 librados: watch_flush exit 2024-06-11T19:09:35.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 20 client.4299.objecter shutdown clearing up homeless session... 2024-06-11T19:09:35.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 10 client.4299.objecter successfully canceled tick 2024-06-11T19:09:35.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 1 -- 172.21.15.98:0/2235947377 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1bb403fe00 msgr2=0x7f1bb40422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1bb403fe00 0x7f1bb40422c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80097c0 tx=0x7f1bb8008030 comp rx=0 tx=0).stop 2024-06-11T19:09:35.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 10 monclient: shutdown 2024-06-11T19:09:35.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:35.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 1 -- 172.21.15.98:0/2235947377 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1bc80a0cd0 msgr2=0x7f1bc81d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:35.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1bc80a0cd0 0x7f1bc81d1d70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00d750 tx=0x7f1bbc00dc10 comp rx=0 tx=0).stop 2024-06-11T19:09:35.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 1 -- 172.21.15.98:0/2235947377 shutdown_connections 2024-06-11T19:09:35.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1bb403fe00 0x7f1bb40422c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1bc8138c80 0x7f1bc806f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1bc80a2250 0x7f1bc81d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 1 --2- 172.21.15.98:0/2235947377 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1bc80a0cd0 0x7f1bc81d1d70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:35.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.993+0000 7f1bcf069640 1 -- 172.21.15.98:0/2235947377 >> 172.21.15.98:0/2235947377 conn(0x7f1bc80a7850 msgr2=0x7f1bc80a3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:35.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.994+0000 7f1bcf069640 1 -- 172.21.15.98:0/2235947377 shutdown_connections 2024-06-11T19:09:35.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.994+0000 7f1bcf069640 1 -- 172.21.15.98:0/2235947377 wait complete. 2024-06-11T19:09:35.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:35.994+0000 7f1bcf069640 1 librados: shutdown 2024-06-11T19:09:36.001 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.0 2024-06-11T19:09:36.001 DEBUG:teuthology.parallel:result is None 2024-06-11T19:09:36.001 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.5 2024-06-11T19:09:36.002 DEBUG:teuthology.parallel:result is None 2024-06-11T19:09:36.005 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.3 2024-06-11T19:09:36.006 DEBUG:teuthology.parallel:result is None 2024-06-11T19:09:36.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.027+0000 7f8f14e35640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-06-11T19:09:36.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.027+0000 7f8f14e35640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-06-11T19:09:36.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.027+0000 7f8f14e35640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:36.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.027+0000 7f8f14e35640 1 -- 172.21.15.98:0/3050944562 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f8f10129ee0 con 0x7f8f10073830 2024-06-11T19:09:36.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.027+0000 7f8efeffd640 1 -- 172.21.15.98:0/3050944562 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8f00003b20 con 0x7f8f10073830 2024-06-11T19:09:36.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.027+0000 7f8efeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-06-11T19:09:36.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.027+0000 7f8efeffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:36.028 INFO:teuthology.orchestra.run.smithi098.stdout:47244640261 2024-06-11T19:09:36.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 10 librados: watch_flush enter 2024-06-11T19:09:36.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 10 librados: watch_flush exit 2024-06-11T19:09:36.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 20 client.4337.objecter shutdown clearing up homeless session... 2024-06-11T19:09:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 10 client.4337.objecter successfully canceled tick 2024-06-11T19:09:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 -- 172.21.15.98:0/3050944562 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8ef003fdb0 msgr2=0x7f8ef0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8ef003fdb0 0x7f8ef0042270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f10072fb0 tx=0x7f8f04008040 comp rx=0 tx=0).stop 2024-06-11T19:09:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 10 monclient: shutdown 2024-06-11T19:09:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 -- 172.21.15.98:0/3050944562 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f10073830 msgr2=0x7f8f101d1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f10073830 0x7f8f101d1fe0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009870 tx=0x7f8f000046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 -- 172.21.15.98:0/3050944562 shutdown_connections 2024-06-11T19:09:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8ef003fdb0 0x7f8ef0042270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f100a52a0 0x7f8f10132d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f1009d520 0x7f8f10132800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 --2- 172.21.15.98:0/3050944562 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f10073830 0x7f8f101d1fe0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 -- 172.21.15.98:0/3050944562 >> 172.21.15.98:0/3050944562 conn(0x7f8f100a4780 msgr2=0x7f8f101bb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:36.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 -- 172.21.15.98:0/3050944562 shutdown_connections 2024-06-11T19:09:36.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 -- 172.21.15.98:0/3050944562 wait complete. 2024-06-11T19:09:36.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.029+0000 7f8f14e35640 1 librados: shutdown 2024-06-11T19:09:36.040 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.4 2024-06-11T19:09:36.040 DEBUG:teuthology.parallel:result is None 2024-06-11T19:09:36.040 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-06-11T19:09:36.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-11T19:09:36.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.179+0000 7f5ddce42640 10 monclient: get_monmap_and_config 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.179+0000 7f5ddce42640 10 monclient: build_initial_monmap 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.180+0000 7f5ddce42640 10 monclient: monmap: 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:36.180999+0000 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:36.180999+0000 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:36.180 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:36.181 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:36.181 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:36.181 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:36.181 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:36.181 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:36.181 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:36.181 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.180+0000 7f5ddce42640 1 Processor -- start 2024-06-11T19:09:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.180+0000 7f5ddce42640 1 -- start start 2024-06-11T19:09:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.180+0000 7f5ddce42640 10 monclient: init 2024-06-11T19:09:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 10 monclient: _add_conns ranks=[5,7,9,8,1,0,6,4,2,3,10] 2024-06-11T19:09:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5dd80a4920 0x7f5dd80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 10 monclient(hunting): picked mon.noname-h con 0x7f5dd80a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5dd80a5240 0x7f5dd80a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 10 monclient(hunting): picked mon.noname-i con 0x7f5dd80a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:36.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5dd80a2530 0x7f5dd80a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 10 monclient(hunting): picked mon.noname-a con 0x7f5dd80a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5dd80a7140 con 0x7f5dd80a2530 2024-06-11T19:09:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f5dd80a2b50 con 0x7f5dd80a4920 2024-06-11T19:09:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f5dd80a2e30 con 0x7f5dd80a5240 2024-06-11T19:09:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5ddce42640 10 monclient(hunting): authenticate will time out at 785.988324s 2024-06-11T19:09:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5dd80a2530 0x7f5dd80a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5dd80a2530 0x7f5dd80a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45724/0 (socket says 172.21.15.98:45724) 2024-06-11T19:09:36.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 1 -- 172.21.15.98:0/3788506952 learned_addr learned my addr 172.21.15.98:0/3788506952 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 10 monclient(hunting): get_auth_request con 0x7f5dd80a2530 auth_method 0 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd659c640 1 --2- 172.21.15.98:0/3788506952 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5dd80a4920 0x7f5dd80a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd659c640 10 monclient(hunting): get_auth_request con 0x7f5dd80a4920 auth_method 0 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd5d9b640 1 --2- 172.21.15.98:0/3788506952 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5dd80a5240 0x7f5dd80a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 1 -- 172.21.15.98:0/3788506952 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5dd80a5240 msgr2=0x7f5dd80a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 1 --2- 172.21.15.98:0/3788506952 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5dd80a5240 0x7f5dd80a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 1 -- 172.21.15.98:0/3788506952 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5dd80a4920 msgr2=0x7f5dd80a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 1 --2- 172.21.15.98:0/3788506952 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5dd80a4920 0x7f5dd80a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 1 monclient: found mon.noname-a 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd6d9d640 1 -- 172.21.15.98:0/3788506952 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd80a3110 con 0x7f5dd80a2530 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd5d9b640 10 monclient: get_auth_request con 0x7f5dd80a5240 auth_method 0 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.181+0000 7f5dd5d9b640 1 --2- 172.21.15.98:0/3788506952 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5dd80a5240 0x7f5dd80a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd6d9d640 1 --2- 172.21.15.98:0/3788506952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dd80a2530 0x7f5dd80a5ee0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009e50 tx=0x7f5dd002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442ecc30a805240e server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:36.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 1 -- 172.21.15.98:0/3788506952 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5dd002cbc0 con 0x7f5dd80a2530 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 10 monclient: dump: 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:36.194 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:36.195 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 10 monclient: _finish_auth 0 2024-06-11T19:09:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 10 monclient: _check_auth_tickets 2024-06-11T19:09:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 1 -- 172.21.15.98:0/3788506952 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd002cd80 con 0x7f5dd80a2530 2024-06-11T19:09:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5ddce42640 5 monclient: authenticate success, global_id 4347 2024-06-11T19:09:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5ddce42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 1 -- 172.21.15.98:0/3788506952 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5dd0034860 con 0x7f5dd80a2530 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5ddce42640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5ddce42640 4 set_mon_vals no callback set 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.182+0000 7f5dd559a640 10 monclient: dump: 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.183+0000 7f5ddce42640 10 monclient: shutdown 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.183+0000 7f5ddce42640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.183+0000 7f5ddce42640 1 -- 172.21.15.98:0/3788506952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dd80a2530 msgr2=0x7f5dd80a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.183+0000 7f5ddce42640 1 --2- 172.21.15.98:0/3788506952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dd80a2530 0x7f5dd80a5ee0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009e50 tx=0x7f5dd002bdb0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.183+0000 7f5ddce42640 1 -- 172.21.15.98:0/3788506952 shutdown_connections 2024-06-11T19:09:36.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.183+0000 7f5ddce42640 1 --2- 172.21.15.98:0/3788506952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5dd80a2530 0x7f5dd80a5ee0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.183+0000 7f5ddce42640 1 --2- 172.21.15.98:0/3788506952 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5dd80a5240 0x7f5dd80a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.183+0000 7f5ddce42640 1 --2- 172.21.15.98:0/3788506952 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5dd80a4920 0x7f5dd80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.183+0000 7f5ddce42640 1 -- 172.21.15.98:0/3788506952 >> 172.21.15.98:0/3788506952 conn(0x7f5dd80ab850 msgr2=0x7f5dd80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.183+0000 7f5ddce42640 1 -- 172.21.15.98:0/3788506952 shutdown_connections 2024-06-11T19:09:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.183+0000 7f5ddce42640 1 -- 172.21.15.98:0/3788506952 wait complete. 2024-06-11T19:09:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 10 monclient: build_initial_monmap 2024-06-11T19:09:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 10 monclient: monmap: 2024-06-11T19:09:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 1 librados: starting msgr at 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 1 librados: starting objecter 2024-06-11T19:09:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 1 Processor -- start 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 1 -- start start 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 1 librados: setting wanted keys 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 1 librados: calling monclient init 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 10 monclient: init 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 10 monclient: _add_conns ranks=[7,4,5,3,2,6,8,1,0] 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5dd80a2530 0x7f5dd8137db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 10 monclient(hunting): picked mon.noname-i con 0x7f5dd80a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5dd80a4920 0x7f5dd8121890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.184+0000 7f5ddce42640 10 monclient(hunting): picked mon.noname-c con 0x7f5dd80a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5dd80a5240 0x7f5dd8121dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 10 monclient(hunting): picked mon.noname-h con 0x7f5dd80a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5dd80a6da0 con 0x7f5dd80a4920 2024-06-11T19:09:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f5dd8138e50 con 0x7f5dd80a5240 2024-06-11T19:09:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f5dd80a6420 con 0x7f5dd80a2530 2024-06-11T19:09:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 10 monclient(hunting): authenticate will time out at 785.992033s 2024-06-11T19:09:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5dd80a4920 0x7f5dd8121890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd659c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5dd80a2530 0x7f5dd8137db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5dd80a4920 0x7f5dd8121890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:40438/0 (socket says 172.21.15.98:40438) 2024-06-11T19:09:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 1 -- 172.21.15.98:0/1701719169 learned_addr learned my addr 172.21.15.98:0/1701719169 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 10 monclient(hunting): get_auth_request con 0x7f5dd80a4920 auth_method 0 2024-06-11T19:09:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd659c640 10 monclient(hunting): get_auth_request con 0x7f5dd80a2530 auth_method 0 2024-06-11T19:09:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd6d9d640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5dd80a5240 0x7f5dd8121dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd6d9d640 10 monclient(hunting): get_auth_request con 0x7f5dd80a5240 auth_method 0 2024-06-11T19:09:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd6d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 1 -- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5dd80a2530 msgr2=0x7f5dd8137db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5dd80a2530 0x7f5dd8137db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 1 -- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5dd80a5240 msgr2=0x7f5dd8121dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5dd80a5240 0x7f5dd8121dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 1 monclient: found mon.noname-c 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 1 -- 172.21.15.98:0/1701719169 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd81387f0 con 0x7f5dd80a4920 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd5d9b640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5dd80a4920 0x7f5dd8121890 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00c7f0 tx=0x7f5dcc00ccb0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 10 client.?.objecter ms_handle_connect 0x7f5dd80a4920 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 1 -- 172.21.15.98:0/1701719169 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5dcc015070 con 0x7f5dd80a4920 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 10 monclient: dump: 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:36.206 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 10 monclient: _finish_auth 0 2024-06-11T19:09:36.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:36.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:36.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 1 -- 172.21.15.98:0/1701719169 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dcc007a90 con 0x7f5dd80a4920 2024-06-11T19:09:36.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 5 monclient: authenticate success, global_id 4360 2024-06-11T19:09:36.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 10 monclient: _renew_subs 2024-06-11T19:09:36.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:36.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 1 -- 172.21.15.98:0/1701719169 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd8125770 con 0x7f5dd80a4920 2024-06-11T19:09:36.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:36.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 1 -- 172.21.15.98:0/1701719169 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5dcc007c80 con 0x7f5dd80a4920 2024-06-11T19:09:36.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dd4d99640 4 set_mon_vals no callback set 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5dc6ffd640 10 monclient: dump: 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 10 monclient: _renew_subs 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 1 -- 172.21.15.98:0/1701719169 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd8125490 con 0x7f5dd80a4920 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.185+0000 7f5ddce42640 1 librados: init done 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.186+0000 7f5dc6ffd640 1 -- 172.21.15.98:0/1701719169 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f5dcc018070 con 0x7f5dd80a4920 2024-06-11T19:09:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.186+0000 7f5dc6ffd640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5db803fdb0 0x7f5db8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.186+0000 7f5dc6ffd640 1 -- 172.21.15.98:0/1701719169 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f5dcc0512c0 con 0x7f5dd80a4920 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.186+0000 7f5dc6ffd640 10 client.4360.objecter ms_dispatch 0x7f5dd8000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.186+0000 7f5dc6ffd640 3 client.4360.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.186+0000 7f5dc6ffd640 3 client.4360.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.186+0000 7f5ddce42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.186+0000 7f5ddce42640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.186+0000 7f5ddce42640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.186+0000 7f5ddce42640 1 -- 172.21.15.98:0/1701719169 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd80a4d00 con 0x7f5dd80a4920 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.186+0000 7f5dc6ffd640 20 client.4360.objecter dump_active .. 0 homeless 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.187+0000 7f5dd659c640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5db803fdb0 0x7f5db8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.187+0000 7f5dd659c640 10 monclient: get_auth_request con 0x7f5db803fdb0 auth_method 0 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.187+0000 7f5dc6ffd640 10 client.4360.objecter ms_handle_connect 0x7f5db803fdb0 2024-06-11T19:09:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.187+0000 7f5dd659c640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5db803fdb0 0x7f5db8042270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8006fd0 tx=0x7f5dc8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:36.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.191+0000 7f5dc6ffd640 1 -- 172.21.15.98:0/1701719169 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5dcc007e20 con 0x7f5dd80a4920 2024-06-11T19:09:36.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.191+0000 7f5dc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:36.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.191+0000 7f5dc6ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:36.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.366+0000 7f5ddce42640 1 -- 172.21.15.98:0/1701719169 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5dd81ce3a0 con 0x7f5db803fdb0 2024-06-11T19:09:36.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.367+0000 7f5dc6ffd640 1 -- 172.21.15.98:0/1701719169 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33795 (secure 0 0 0) 0x7f5dc8010990 con 0x7f5db803fdb0 2024-06-11T19:09:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:09:36.368 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-06-11T19:09:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.369+0000 7f5ddce42640 10 librados: watch_flush enter 2024-06-11T19:09:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.369+0000 7f5ddce42640 10 librados: watch_flush exit 2024-06-11T19:09:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.369+0000 7f5ddce42640 20 client.4360.objecter shutdown clearing up homeless session... 2024-06-11T19:09:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.369+0000 7f5ddce42640 10 client.4360.objecter successfully canceled tick 2024-06-11T19:09:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.369+0000 7f5ddce42640 1 -- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5db803fdb0 msgr2=0x7f5db8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.369+0000 7f5ddce42640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5db803fdb0 0x7f5db8042270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8006fd0 tx=0x7f5dc8008040 comp rx=0 tx=0).stop 2024-06-11T19:09:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.369+0000 7f5ddce42640 10 monclient: shutdown 2024-06-11T19:09:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.369+0000 7f5ddce42640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.369+0000 7f5ddce42640 1 -- 172.21.15.98:0/1701719169 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5dd80a4920 msgr2=0x7f5dd8121890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.369+0000 7f5ddce42640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5dd80a4920 0x7f5dd8121890 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00c7f0 tx=0x7f5dcc00ccb0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.369+0000 7f5ddce42640 1 -- 172.21.15.98:0/1701719169 shutdown_connections 2024-06-11T19:09:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.370+0000 7f5ddce42640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5db803fdb0 0x7f5db8042270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.370+0000 7f5ddce42640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5dd80a5240 0x7f5dd8121dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.370+0000 7f5ddce42640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5dd80a4920 0x7f5dd8121890 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.370+0000 7f5ddce42640 1 --2- 172.21.15.98:0/1701719169 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5dd80a2530 0x7f5dd8137db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.370+0000 7f5ddce42640 1 -- 172.21.15.98:0/1701719169 >> 172.21.15.98:0/1701719169 conn(0x7f5dd80ab850 msgr2=0x7f5dd80a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.370+0000 7f5ddce42640 1 -- 172.21.15.98:0/1701719169 shutdown_connections 2024-06-11T19:09:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.370+0000 7f5ddce42640 1 -- 172.21.15.98:0/1701719169 wait complete. 2024-06-11T19:09:36.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.370+0000 7f5ddce42640 1 librados: shutdown 2024-06-11T19:09:36.380 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-06-11T19:09:35.367311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164148,"kb_used_data":2000,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066892,"statfs":{"total":579820584960,"available":579652497408,"internally_reserved":0,"allocated":2048000,"data_stored":1378592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000759"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716208+0000","last_change":"2024-06-11T19:09:33.725085+0000","last_active":"2024-06-11T19:09:34.716208+0000","last_peered":"2024-06-11T19:09:34.716208+0000","last_clean":"2024-06-11T19:09:34.716208+0000","last_became_active":"2024-06-11T19:09:33.724778+0000","last_became_peered":"2024-06-11T19:09:33.724778+0000","last_unstale":"2024-06-11T19:09:34.716208+0000","last_undegraded":"2024-06-11T19:09:34.716208+0000","last_fullsized":"2024-06-11T19:09:34.716208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:57:46.868411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.717260+0000","last_change":"2024-06-11T19:09:33.725197+0000","last_active":"2024-06-11T19:09:34.717260+0000","last_peered":"2024-06-11T19:09:34.717260+0000","last_clean":"2024-06-11T19:09:34.717260+0000","last_became_active":"2024-06-11T19:09:33.724912+0000","last_became_peered":"2024-06-11T19:09:33.724912+0000","last_unstale":"2024-06-11T19:09:34.717260+0000","last_undegraded":"2024-06-11T19:09:34.717260+0000","last_fullsized":"2024-06-11T19:09:34.717260+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:36:12.941264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716160+0000","last_change":"2024-06-11T19:09:33.720915+0000","last_active":"2024-06-11T19:09:34.716160+0000","last_peered":"2024-06-11T19:09:34.716160+0000","last_clean":"2024-06-11T19:09:34.716160+0000","last_became_active":"2024-06-11T19:09:33.720599+0000","last_became_peered":"2024-06-11T19:09:33.720599+0000","last_unstale":"2024-06-11T19:09:34.716160+0000","last_undegraded":"2024-06-11T19:09:34.716160+0000","last_fullsized":"2024-06-11T19:09:34.716160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:28:26.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-06-11T19:09:33.736653+0000","last_change":"2024-06-11T19:09:33.729223+0000","last_active":"2024-06-11T19:09:33.736653+0000","last_peered":"2024-06-11T19:09:33.736653+0000","last_clean":"2024-06-11T19:09:33.736653+0000","last_became_active":"2024-06-11T19:09:33.728897+0000","last_became_peered":"2024-06-11T19:09:33.728897+0000","last_unstale":"2024-06-11T19:09:33.736653+0000","last_undegraded":"2024-06-11T19:09:33.736653+0000","last_fullsized":"2024-06-11T19:09:33.736653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:31:28.135169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.720622+0000","last_change":"2024-06-11T19:09:33.722503+0000","last_active":"2024-06-11T19:09:34.720622+0000","last_peered":"2024-06-11T19:09:34.720622+0000","last_clean":"2024-06-11T19:09:34.720622+0000","last_became_active":"2024-06-11T19:09:33.722180+0000","last_became_peered":"2024-06-11T19:09:33.722180+0000","last_unstale":"2024-06-11T19:09:34.720622+0000","last_undegraded":"2024-06-11T19:09:34.720622+0000","last_fullsized":"2024-06-11T19:09:34.720622+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:28:15.836943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716628+0000","last_change":"2024-06-11T19:09:33.729275+0000","last_active":"2024-06-11T19:09:34.716628+0000","last_peered":"2024-06-11T19:09:34.716628+0000","last_clean":"2024-06-11T19:09:34.716628+0000","last_became_active":"2024-06-11T19:09:33.728948+0000","last_became_peered":"2024-06-11T19:09:33.728948+0000","last_unstale":"2024-06-11T19:09:34.716628+0000","last_undegraded":"2024-06-11T19:09:34.716628+0000","last_fullsized":"2024-06-11T19:09:34.716628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:29:28.908169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.720640+0000","last_change":"2024-06-11T19:09:33.729213+0000","last_active":"2024-06-11T19:09:34.720640+0000","last_peered":"2024-06-11T19:09:34.720640+0000","last_clean":"2024-06-11T19:09:34.720640+0000","last_became_active":"2024-06-11T19:09:33.728930+0000","last_became_peered":"2024-06-11T19:09:33.728930+0000","last_unstale":"2024-06-11T19:09:34.720640+0000","last_undegraded":"2024-06-11T19:09:34.720640+0000","last_fullsized":"2024-06-11T19:09:34.720640+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:26:38.808169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716204+0000","last_change":"2024-06-11T19:09:33.725261+0000","last_active":"2024-06-11T19:09:34.716204+0000","last_peered":"2024-06-11T19:09:34.716204+0000","last_clean":"2024-06-11T19:09:34.716204+0000","last_became_active":"2024-06-11T19:09:33.724870+0000","last_became_peered":"2024-06-11T19:09:33.724870+0000","last_unstale":"2024-06-11T19:09:34.716204+0000","last_undegraded":"2024-06-11T19:09:34.716204+0000","last_fullsized":"2024-06-11T19:09:34.716204+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:32:51.020718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":108,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716046+0000","last_change":"2024-06-11T19:09:26.673425+0000","last_active":"2024-06-11T19:09:34.716046+0000","last_peered":"2024-06-11T19:09:34.716046+0000","last_clean":"2024-06-11T19:09:34.716046+0000","last_became_active":"2024-06-11T19:09:26.673120+0000","last_became_peered":"2024-06-11T19:09:26.673120+0000","last_unstale":"2024-06-11T19:09:34.716046+0000","last_undegraded":"2024-06-11T19:09:34.716046+0000","last_fullsized":"2024-06-11T19:09:34.716046+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_clean_scrub_stamp":"2024-06-11T19:09:25.660869+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:57:43.705570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":688128,"data_stored":579128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":688128,"data_stored":579128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-11T19:09:36.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-11T19:09:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.478+0000 7ff74e5c6640 10 monclient: get_monmap_and_config 2024-06-11T19:09:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.478+0000 7ff74e5c6640 10 monclient: build_initial_monmap 2024-06-11T19:09:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.478+0000 7ff74e5c6640 10 monclient: monmap: 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:36.479860+0000 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:36.479860+0000 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:36.480 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:36.480 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:36.480 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:36.480 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.479+0000 7ff74e5c6640 1 Processor -- start 2024-06-11T19:09:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.479+0000 7ff74e5c6640 1 -- start start 2024-06-11T19:09:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.479+0000 7ff74e5c6640 10 monclient: init 2024-06-11T19:09:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 10 monclient: _add_conns ranks=[2,7,4,10,1,6,3,0,9,5,8] 2024-06-11T19:09:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7480a4920 0x7ff7480a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 10 monclient(hunting): picked mon.noname-b con 0x7ff7480a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7480a5240 0x7ff7480a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 10 monclient(hunting): picked mon.noname-i con 0x7ff7480a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7480a2530 0x7ff7480a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 10 monclient(hunting): picked mon.noname-c con 0x7ff7480a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff7480a7140 con 0x7ff7480a4920 2024-06-11T19:09:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff7480a2b50 con 0x7ff7480a2530 2024-06-11T19:09:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff7480a2e30 con 0x7ff7480a5240 2024-06-11T19:09:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74e5c6640 10 monclient(hunting): authenticate will time out at 786.287161s 2024-06-11T19:09:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7480a4920 0x7ff7480a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74cb63640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7480a2530 0x7ff7480a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7480a4920 0x7ff7480a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47976/0 (socket says 172.21.15.98:47976) 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 1 -- 172.21.15.98:0/1001923067 learned_addr learned my addr 172.21.15.98:0/1001923067 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 10 monclient(hunting): get_auth_request con 0x7ff7480a4920 auth_method 0 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74cb63640 10 monclient(hunting): get_auth_request con 0x7ff7480a2530 auth_method 0 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74cb63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74cb63640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74cb63640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff7477fe640 1 --2- 172.21.15.98:0/1001923067 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7480a5240 0x7ff7480a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74cb63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74cb63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74cb63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff7477fe640 10 monclient(hunting): get_auth_request con 0x7ff7480a5240 auth_method 0 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff7477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff7477fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff7477fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 1 -- 172.21.15.98:0/1001923067 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7480a5240 msgr2=0x7ff7480a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 1 --2- 172.21.15.98:0/1001923067 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7480a5240 0x7ff7480a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 1 -- 172.21.15.98:0/1001923067 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7480a2530 msgr2=0x7ff7480a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 1 --2- 172.21.15.98:0/1001923067 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7480a2530 0x7ff7480a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 10 monclient: _finish_hunting 0 2024-06-11T19:09:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 1 monclient: found mon.noname-b 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff747fff640 1 -- 172.21.15.98:0/1001923067 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7480a3110 con 0x7ff7480a4920 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff74cb63640 1 --2- 172.21.15.98:0/1001923067 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7480a2530 0x7ff7480a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.480+0000 7ff7477fe640 1 --2- 172.21.15.98:0/1001923067 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7480a5240 0x7ff7480a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff747fff640 1 --2- 172.21.15.98:0/1001923067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7480a4920 0x7ff7480a4d00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff7380098a0 tx=0x7ff73802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c61fff7ba4ff265 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 1 -- 172.21.15.98:0/1001923067 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff738038070 con 0x7ff7480a4920 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 10 monclient: dump: 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:36.494 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 10 monclient: _finish_auth 0 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 1 -- 172.21.15.98:0/1001923067 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff738031970 con 0x7ff7480a4920 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff74e5c6640 5 monclient: authenticate success, global_id 4340 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff74e5c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff74e5c6640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 1 -- 172.21.15.98:0/1001923067 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff738031ba0 con 0x7ff7480a4920 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff74e5c6640 4 set_mon_vals no callback set 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff746ffd640 10 monclient: dump: 2024-06-11T19:09:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff74e5c6640 10 monclient: shutdown 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff74e5c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff74e5c6640 1 -- 172.21.15.98:0/1001923067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7480a4920 msgr2=0x7ff7480a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.481+0000 7ff74e5c6640 1 --2- 172.21.15.98:0/1001923067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7480a4920 0x7ff7480a4d00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff7380098a0 tx=0x7ff73802b7a0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.482+0000 7ff74e5c6640 1 -- 172.21.15.98:0/1001923067 shutdown_connections 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.482+0000 7ff74e5c6640 1 --2- 172.21.15.98:0/1001923067 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7480a2530 0x7ff7480a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.482+0000 7ff74e5c6640 1 --2- 172.21.15.98:0/1001923067 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7480a5240 0x7ff7480a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.482+0000 7ff74e5c6640 1 --2- 172.21.15.98:0/1001923067 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7480a4920 0x7ff7480a4d00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.482+0000 7ff74e5c6640 1 -- 172.21.15.98:0/1001923067 >> 172.21.15.98:0/1001923067 conn(0x7ff7480ab850 msgr2=0x7ff7480a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.482+0000 7ff74e5c6640 1 -- 172.21.15.98:0/1001923067 shutdown_connections 2024-06-11T19:09:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.482+0000 7ff74e5c6640 1 -- 172.21.15.98:0/1001923067 wait complete. 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.482+0000 7ff74e5c6640 10 monclient: build_initial_monmap 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.482+0000 7ff74e5c6640 10 monclient: monmap: 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.482+0000 7ff74e5c6640 1 librados: starting msgr at 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.482+0000 7ff74e5c6640 1 librados: starting objecter 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.482+0000 7ff74e5c6640 1 Processor -- start 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 1 -- start start 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 1 librados: setting wanted keys 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 1 librados: calling monclient init 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 10 monclient: init 2024-06-11T19:09:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 10 monclient: _add_conns ranks=[0,4,6,3,7,8,5,1,2] 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7480a2530 0x7ff7481cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 10 monclient(hunting): picked mon.noname-a con 0x7ff7480a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7480a4920 0x7ff7481ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 10 monclient(hunting): picked mon.noname-c con 0x7ff7480a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff7480a5240 0x7ff748122ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 10 monclient(hunting): picked mon.noname-d con 0x7ff7480a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7480a6da0 con 0x7ff7480a2530 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff748138e50 con 0x7ff7480a4920 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff7480a6420 con 0x7ff7480a5240 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74e5c6640 10 monclient(hunting): authenticate will time out at 786.290693s 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff747fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7480a2530 0x7ff7481cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff7477fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7480a4920 0x7ff7481ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff747fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7480a2530 0x7ff7481cdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45726/0 (socket says 172.21.15.98:45726) 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff747fff640 1 -- 172.21.15.98:0/2789833902 learned_addr learned my addr 172.21.15.98:0/2789833902 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff747fff640 10 monclient(hunting): get_auth_request con 0x7ff7480a2530 auth_method 0 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff747fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff747fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff7477fe640 10 monclient(hunting): get_auth_request con 0x7ff7480a4920 auth_method 0 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff7477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff7477fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff7477fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74cb63640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff7480a5240 0x7ff748122ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff7477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff7477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff7477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74cb63640 10 monclient(hunting): get_auth_request con 0x7ff7480a5240 auth_method 0 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74cb63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74cb63640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74cb63640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74cb63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74cb63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.483+0000 7ff74cb63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff747fff640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff747fff640 1 -- 172.21.15.98:0/2789833902 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff7480a5240 msgr2=0x7ff748122ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff747fff640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff7480a5240 0x7ff748122ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff747fff640 1 -- 172.21.15.98:0/2789833902 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7480a4920 msgr2=0x7ff7481ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff747fff640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7480a4920 0x7ff7481ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff747fff640 10 monclient: _finish_hunting 0 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff747fff640 1 monclient: found mon.noname-a 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff747fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff747fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff747fff640 1 -- 172.21.15.98:0/2789833902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7480a9080 con 0x7ff7480a2530 2024-06-11T19:09:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff7477fe640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7480a4920 0x7ff7481ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff747fff640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7480a2530 0x7ff7481cdc80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff738000c00 tx=0x7ff7380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 10 client.?.objecter ms_handle_connect 0x7ff7480a2530 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 1 -- 172.21.15.98:0/2789833902 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff738042020 con 0x7ff7480a2530 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 10 monclient: dump: 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 10 monclient: _finish_auth 0 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 1 -- 172.21.15.98:0/2789833902 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff738031ea0 con 0x7ff7480a2530 2024-06-11T19:09:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff74e5c6640 5 monclient: authenticate success, global_id 4356 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 1 -- 172.21.15.98:0/2789833902 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff738038040 con 0x7ff7480a2530 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff74e5c6640 10 monclient: _renew_subs 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff74e5c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff74e5c6640 1 -- 172.21.15.98:0/2789833902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff748125700 con 0x7ff7480a2530 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff7467fc640 4 set_mon_vals no callback set 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff744ff9640 10 monclient: dump: 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff74e5c6640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff74e5c6640 10 monclient: _renew_subs 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff74e5c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff74e5c6640 1 -- 172.21.15.98:0/2789833902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff748126bb0 con 0x7ff7480a2530 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.484+0000 7ff74e5c6640 1 librados: init done 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff744ff9640 1 -- 172.21.15.98:0/2789833902 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff738045020 con 0x7ff7480a2530 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff74e5c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff74e5c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff74e5c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff74e5c6640 1 -- 172.21.15.98:0/2789833902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7480a66f0 con 0x7ff7480a2530 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff744ff9640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff73403fdb0 0x7ff734042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff744ff9640 1 -- 172.21.15.98:0/2789833902 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7ff738072ab0 con 0x7ff7480a2530 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff744ff9640 10 client.4356.objecter ms_dispatch 0x7ff748000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff744ff9640 3 client.4356.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff744ff9640 3 client.4356.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff744ff9640 20 client.4356.objecter dump_active .. 0 homeless 2024-06-11T19:09:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff7477fe640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff73403fdb0 0x7ff734042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.485+0000 7ff7477fe640 10 monclient: get_auth_request con 0x7ff73403fdb0 auth_method 0 2024-06-11T19:09:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.486+0000 7ff744ff9640 10 client.4356.objecter ms_handle_connect 0x7ff73403fdb0 2024-06-11T19:09:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.486+0000 7ff7477fe640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff73403fdb0 0x7ff734042270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff7481249d0 tx=0x7ff73c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.490+0000 7ff744ff9640 1 -- 172.21.15.98:0/2789833902 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff73804c030 con 0x7ff7480a2530 2024-06-11T19:09:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.490+0000 7ff744ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.490+0000 7ff744ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.667+0000 7ff74e5c6640 1 -- 172.21.15.98:0/2789833902 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7480a61d0 con 0x7ff73403fdb0 2024-06-11T19:09:36.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.668+0000 7ff744ff9640 1 -- 172.21.15.98:0/2789833902 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33795 (secure 0 0 0) 0x7ff7480a61d0 con 0x7ff73403fdb0 2024-06-11T19:09:36.668 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:09:36.669 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-06-11T19:09:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 10 librados: watch_flush enter 2024-06-11T19:09:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 10 librados: watch_flush exit 2024-06-11T19:09:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 20 client.4356.objecter shutdown clearing up homeless session... 2024-06-11T19:09:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 10 client.4356.objecter successfully canceled tick 2024-06-11T19:09:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 -- 172.21.15.98:0/2789833902 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff73403fdb0 msgr2=0x7ff734042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff73403fdb0 0x7ff734042270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff7481249d0 tx=0x7ff73c008040 comp rx=0 tx=0).stop 2024-06-11T19:09:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 10 monclient: shutdown 2024-06-11T19:09:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 -- 172.21.15.98:0/2789833902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7480a2530 msgr2=0x7ff7481cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7480a2530 0x7ff7481cdc80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff738000c00 tx=0x7ff7380046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 -- 172.21.15.98:0/2789833902 shutdown_connections 2024-06-11T19:09:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff73403fdb0 0x7ff734042270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff7480a5240 0x7ff748122ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7480a4920 0x7ff7481ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 --2- 172.21.15.98:0/2789833902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7480a2530 0x7ff7481cdc80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 -- 172.21.15.98:0/2789833902 >> 172.21.15.98:0/2789833902 conn(0x7ff7480ab850 msgr2=0x7ff7481bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 -- 172.21.15.98:0/2789833902 shutdown_connections 2024-06-11T19:09:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 -- 172.21.15.98:0/2789833902 wait complete. 2024-06-11T19:09:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.670+0000 7ff74e5c6640 1 librados: shutdown 2024-06-11T19:09:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-06-11T19:09:35.367311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164148,"kb_used_data":2000,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066892,"statfs":{"total":579820584960,"available":579652497408,"internally_reserved":0,"allocated":2048000,"data_stored":1378592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000759"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716208+0000","last_change":"2024-06-11T19:09:33.725085+0000","last_active":"2024-06-11T19:09:34.716208+0000","last_peered":"2024-06-11T19:09:34.716208+0000","last_clean":"2024-06-11T19:09:34.716208+0000","last_became_active":"2024-06-11T19:09:33.724778+0000","last_became_peered":"2024-06-11T19:09:33.724778+0000","last_unstale":"2024-06-11T19:09:34.716208+0000","last_undegraded":"2024-06-11T19:09:34.716208+0000","last_fullsized":"2024-06-11T19:09:34.716208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:57:46.868411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.717260+0000","last_change":"2024-06-11T19:09:33.725197+0000","last_active":"2024-06-11T19:09:34.717260+0000","last_peered":"2024-06-11T19:09:34.717260+0000","last_clean":"2024-06-11T19:09:34.717260+0000","last_became_active":"2024-06-11T19:09:33.724912+0000","last_became_peered":"2024-06-11T19:09:33.724912+0000","last_unstale":"2024-06-11T19:09:34.717260+0000","last_undegraded":"2024-06-11T19:09:34.717260+0000","last_fullsized":"2024-06-11T19:09:34.717260+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:36:12.941264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716160+0000","last_change":"2024-06-11T19:09:33.720915+0000","last_active":"2024-06-11T19:09:34.716160+0000","last_peered":"2024-06-11T19:09:34.716160+0000","last_clean":"2024-06-11T19:09:34.716160+0000","last_became_active":"2024-06-11T19:09:33.720599+0000","last_became_peered":"2024-06-11T19:09:33.720599+0000","last_unstale":"2024-06-11T19:09:34.716160+0000","last_undegraded":"2024-06-11T19:09:34.716160+0000","last_fullsized":"2024-06-11T19:09:34.716160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:28:26.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-06-11T19:09:33.736653+0000","last_change":"2024-06-11T19:09:33.729223+0000","last_active":"2024-06-11T19:09:33.736653+0000","last_peered":"2024-06-11T19:09:33.736653+0000","last_clean":"2024-06-11T19:09:33.736653+0000","last_became_active":"2024-06-11T19:09:33.728897+0000","last_became_peered":"2024-06-11T19:09:33.728897+0000","last_unstale":"2024-06-11T19:09:33.736653+0000","last_undegraded":"2024-06-11T19:09:33.736653+0000","last_fullsized":"2024-06-11T19:09:33.736653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:31:28.135169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.720622+0000","last_change":"2024-06-11T19:09:33.722503+0000","last_active":"2024-06-11T19:09:34.720622+0000","last_peered":"2024-06-11T19:09:34.720622+0000","last_clean":"2024-06-11T19:09:34.720622+0000","last_became_active":"2024-06-11T19:09:33.722180+0000","last_became_peered":"2024-06-11T19:09:33.722180+0000","last_unstale":"2024-06-11T19:09:34.720622+0000","last_undegraded":"2024-06-11T19:09:34.720622+0000","last_fullsized":"2024-06-11T19:09:34.720622+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:28:15.836943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716628+0000","last_change":"2024-06-11T19:09:33.729275+0000","last_active":"2024-06-11T19:09:34.716628+0000","last_peered":"2024-06-11T19:09:34.716628+0000","last_clean":"2024-06-11T19:09:34.716628+0000","last_became_active":"2024-06-11T19:09:33.728948+0000","last_became_peered":"2024-06-11T19:09:33.728948+0000","last_unstale":"2024-06-11T19:09:34.716628+0000","last_undegraded":"2024-06-11T19:09:34.716628+0000","last_fullsized":"2024-06-11T19:09:34.716628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:29:28.908169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.720640+0000","last_change":"2024-06-11T19:09:33.729213+0000","last_active":"2024-06-11T19:09:34.720640+0000","last_peered":"2024-06-11T19:09:34.720640+0000","last_clean":"2024-06-11T19:09:34.720640+0000","last_became_active":"2024-06-11T19:09:33.728930+0000","last_became_peered":"2024-06-11T19:09:33.728930+0000","last_unstale":"2024-06-11T19:09:34.720640+0000","last_undegraded":"2024-06-11T19:09:34.720640+0000","last_fullsized":"2024-06-11T19:09:34.720640+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:26:38.808169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716204+0000","last_change":"2024-06-11T19:09:33.725261+0000","last_active":"2024-06-11T19:09:34.716204+0000","last_peered":"2024-06-11T19:09:34.716204+0000","last_clean":"2024-06-11T19:09:34.716204+0000","last_became_active":"2024-06-11T19:09:33.724870+0000","last_became_peered":"2024-06-11T19:09:33.724870+0000","last_unstale":"2024-06-11T19:09:34.716204+0000","last_undegraded":"2024-06-11T19:09:34.716204+0000","last_fullsized":"2024-06-11T19:09:34.716204+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:32:51.020718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":108,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716046+0000","last_change":"2024-06-11T19:09:26.673425+0000","last_active":"2024-06-11T19:09:34.716046+0000","last_peered":"2024-06-11T19:09:34.716046+0000","last_clean":"2024-06-11T19:09:34.716046+0000","last_became_active":"2024-06-11T19:09:26.673120+0000","last_became_peered":"2024-06-11T19:09:26.673120+0000","last_unstale":"2024-06-11T19:09:34.716046+0000","last_undegraded":"2024-06-11T19:09:34.716046+0000","last_fullsized":"2024-06-11T19:09:34.716046+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_clean_scrub_stamp":"2024-06-11T19:09:25.660869+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:57:43.705570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":688128,"data_stored":579128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":688128,"data_stored":579128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-11T19:09:36.682 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-06-11T19:09:36.682 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-11T19:09:36.682 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-06-11T19:09:36.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-06-11T19:09:36.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.782+0000 7fa7b3c7a640 10 monclient: get_monmap_and_config 2024-06-11T19:09:36.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.782+0000 7fa7b3c7a640 10 monclient: build_initial_monmap 2024-06-11T19:09:36.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.782+0000 7fa7b3c7a640 10 monclient: monmap: 2024-06-11T19:09:36.782 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:36.782 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.782 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:36.783665+0000 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:36.783665+0000 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:36.784 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:36.784 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 1 Processor -- start 2024-06-11T19:09:36.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 1 -- start start 2024-06-11T19:09:36.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 10 monclient: init 2024-06-11T19:09:36.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:36.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 10 monclient: _add_conns ranks=[3,1,10,5,4,7,8,2,6,9,0] 2024-06-11T19:09:36.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7ac0a4920 0x7fa7ac0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 10 monclient(hunting): picked mon.noname-g con 0x7fa7ac0a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:36.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa7ac0a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:36.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa7ac0a2530 0x7fa7ac0a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 10 monclient(hunting): picked mon.noname-f con 0x7fa7ac0a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:36.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7ac0a7140 con 0x7fa7ac0a5240 2024-06-11T19:09:36.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fa7ac0a2b50 con 0x7fa7ac0a2530 2024-06-11T19:09:36.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fa7ac0a2e30 con 0x7fa7ac0a4920 2024-06-11T19:09:36.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:36.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.783+0000 7fa7b3c7a640 10 monclient(hunting): authenticate will time out at 786.590922s 2024-06-11T19:09:36.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b1a16640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7ac0a4920 0x7fa7ac0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa7ac0a2530 0x7fa7ac0a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b1a16640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7ac0a4920 0x7fa7ac0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:53750/0 (socket says 172.21.15.98:53750) 2024-06-11T19:09:36.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b1a16640 1 -- 172.21.15.98:0/491206071 learned_addr learned my addr 172.21.15.98:0/491206071 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:36.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b1a16640 10 monclient(hunting): get_auth_request con 0x7fa7ac0a4920 auth_method 0 2024-06-11T19:09:36.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa7ac0a2530 0x7fa7ac0a60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:39626/0 (socket says 172.21.15.98:39626) 2024-06-11T19:09:36.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b1a16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b1a16640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b1a16640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 10 monclient(hunting): get_auth_request con 0x7fa7ac0a2530 auth_method 0 2024-06-11T19:09:36.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b1a16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b1a16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b1a16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 1 -- 172.21.15.98:0/491206071 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7ac0a4920 msgr2=0x7fa7ac0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 1 --2- 172.21.15.98:0/491206071 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7ac0a4920 0x7fa7ac0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 1 -- 172.21.15.98:0/491206071 >> v1:172.21.15.117:6789/0 conn(0x7fa7ac0a5240 legacy=0x7fa7ac0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 10 monclient: _finish_hunting 0 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 1 monclient: found mon.noname-f 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:36.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.784+0000 7fa7b2217640 1 -- 172.21.15.98:0/491206071 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ac0a3110 con 0x7fa7ac0a2530 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b2217640 1 --2- 172.21.15.98:0/491206071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7ac0a2530 0x7fa7ac0a60b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40097a0 tx=0x7fa7a402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2dfd8acc097edae server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 1 -- 172.21.15.98:0/491206071 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7a4038070 con 0x7fa7ac0a2530 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 10 monclient: dump: 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:36.797 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 10 monclient: _finish_auth 0 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 10 monclient: _check_auth_tickets 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 1 -- 172.21.15.98:0/491206071 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a4031970 con 0x7fa7ac0a2530 2024-06-11T19:09:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 1 -- 172.21.15.98:0/491206071 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7a4031ba0 con 0x7fa7ac0a2530 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b0a14640 10 monclient: dump: 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b3c7a640 5 monclient: authenticate success, global_id 4330 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b3c7a640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.785+0000 7fa7b3c7a640 4 set_mon_vals no callback set 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 10 monclient: shutdown 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/491206071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7ac0a2530 msgr2=0x7fa7ac0a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 1 --2- 172.21.15.98:0/491206071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7ac0a2530 0x7fa7ac0a60b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40097a0 tx=0x7fa7a402b7a0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/491206071 shutdown_connections 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 1 --2- 172.21.15.98:0/491206071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7ac0a2530 0x7fa7ac0a60b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 1 --2- 172.21.15.98:0/491206071 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7ac0a4920 0x7fa7ac0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/491206071 >> 172.21.15.98:0/491206071 conn(0x7fa7ac0ab850 msgr2=0x7fa7ac0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/491206071 shutdown_connections 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/491206071 wait complete. 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 10 monclient: build_initial_monmap 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 10 monclient: monmap: 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:36.802 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 1 librados: starting msgr at 2024-06-11T19:09:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.786+0000 7fa7b3c7a640 1 librados: starting objecter 2024-06-11T19:09:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 1 Processor -- start 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 1 -- start start 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 1 librados: setting wanted keys 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 1 librados: calling monclient init 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 10 monclient: init 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 10 monclient: _add_conns ranks=[8,3,2,5,4,7,0,6,1] 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ac0a2530 0x7fa7ac1376f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 10 monclient(hunting): picked mon.noname-e con 0x7fa7ac0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7ac0a4920 0x7fa7ac137c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 10 monclient(hunting): picked mon.noname-g con 0x7fa7ac0a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7ac0a5240 0x7fa7ac123110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 10 monclient(hunting): picked mon.noname-b con 0x7fa7ac0a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa7ac0a6da0 con 0x7fa7ac0a5240 2024-06-11T19:09:36.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fa7ac138e50 con 0x7fa7ac0a4920 2024-06-11T19:09:36.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:36.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa7ac0a65f0 con 0x7fa7ac0a2530 2024-06-11T19:09:36.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:36.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b3c7a640 10 monclient(hunting): authenticate will time out at 786.594562s 2024-06-11T19:09:36.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1a16640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ac0a2530 0x7fa7ac1376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1a16640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ac0a2530 0x7fa7ac1376f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54528/0 (socket says 172.21.15.98:54528) 2024-06-11T19:09:36.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1a16640 1 -- 172.21.15.98:0/4110259254 learned_addr learned my addr 172.21.15.98:0/4110259254 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:36.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1a16640 10 monclient(hunting): get_auth_request con 0x7fa7ac0a2530 auth_method 0 2024-06-11T19:09:36.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1a16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1a16640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1a16640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7ac0a5240 0x7fa7ac123110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 10 monclient(hunting): get_auth_request con 0x7fa7ac0a5240 auth_method 0 2024-06-11T19:09:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1215640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7ac0a4920 0x7fa7ac137c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1a16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1a16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1a16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1215640 10 monclient(hunting): get_auth_request con 0x7fa7ac0a4920 auth_method 0 2024-06-11T19:09:36.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1215640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:36.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1215640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:36.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1215640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:36.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1215640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:36.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1215640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:36.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1215640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:36.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 10 monclient(hunting): handle_auth_done global_id 4349 payload 824 2024-06-11T19:09:36.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 1 -- 172.21.15.98:0/4110259254 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ac0a2530 msgr2=0x7fa7ac1376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ac0a2530 0x7fa7ac1376f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 1 -- 172.21.15.98:0/4110259254 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7ac0a4920 msgr2=0x7fa7ac137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:36.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7ac0a4920 0x7fa7ac137c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:36.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 10 monclient: _finish_hunting 0 2024-06-11T19:09:36.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 1 monclient: found mon.noname-b 2024-06-11T19:09:36.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:36.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:36.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b2217640 1 -- 172.21.15.98:0/4110259254 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ac1c0bf0 con 0x7fa7ac0a5240 2024-06-11T19:09:36.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.787+0000 7fa7b1a16640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ac0a2530 0x7fa7ac1376f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:36.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7b2217640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7ac0a5240 0x7fa7ac123110 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4033000 tx=0x7fa7a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:36.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 10 client.?.objecter ms_handle_connect 0x7fa7ac0a5240 2024-06-11T19:09:36.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 1 -- 172.21.15.98:0/4110259254 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7a4038050 con 0x7fa7ac0a5240 2024-06-11T19:09:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 10 monclient: dump: 2024-06-11T19:09:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:36.814 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7b3c7a640 5 monclient: authenticate success, global_id 4349 2024-06-11T19:09:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 1 -- 172.21.15.98:0/4110259254 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a4044070 con 0x7fa7ac0a5240 2024-06-11T19:09:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7b3c7a640 10 monclient: _renew_subs 2024-06-11T19:09:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7b3c7a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/4110259254 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7ac126df0 con 0x7fa7ac0a5240 2024-06-11T19:09:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 1 -- 172.21.15.98:0/4110259254 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7a4031ea0 con 0x7fa7ac0a5240 2024-06-11T19:09:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7b3c7a640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a27fc640 10 monclient: dump: 2024-06-11T19:09:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:36.819 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7a3fff640 4 set_mon_vals no callback set 2024-06-11T19:09:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7b3c7a640 10 monclient: _renew_subs 2024-06-11T19:09:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7b3c7a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/4110259254 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7ac0a92d0 con 0x7fa7ac0a5240 2024-06-11T19:09:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.788+0000 7fa7b3c7a640 1 librados: init done 2024-06-11T19:09:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7a27fc640 1 -- 172.21.15.98:0/4110259254 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fa7a4003610 con 0x7fa7ac0a5240 2024-06-11T19:09:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7b3c7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7b3c7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7b3c7a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/4110259254 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7ac0a4d00 con 0x7fa7ac0a5240 2024-06-11T19:09:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7a27fc640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa79003fdb0 0x7fa790042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7a27fc640 1 -- 172.21.15.98:0/4110259254 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fa7a4073040 con 0x7fa7ac0a5240 2024-06-11T19:09:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7a27fc640 10 client.4349.objecter ms_dispatch 0x7fa7ac000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7a27fc640 3 client.4349.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7a27fc640 3 client.4349.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7a27fc640 20 client.4349.objecter dump_active .. 0 homeless 2024-06-11T19:09:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7b1a16640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa79003fdb0 0x7fa790042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.789+0000 7fa7b1a16640 10 monclient: get_auth_request con 0x7fa79003fdb0 auth_method 0 2024-06-11T19:09:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.790+0000 7fa7a27fc640 10 client.4349.objecter ms_handle_connect 0x7fa79003fdb0 2024-06-11T19:09:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.790+0000 7fa7b1a16640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa79003fdb0 0x7fa790042270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8009360 tx=0x7fa7a800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.794+0000 7fa7a27fc640 1 -- 172.21.15.98:0/4110259254 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa7ac1c0bf0 con 0x7fa7ac0a5240 2024-06-11T19:09:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.794+0000 7fa7a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:36.794+0000 7fa7a27fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:37.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.011+0000 7fa7b3c7a640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-06-11T19:09:37.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.011+0000 7fa7b3c7a640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-06-11T19:09:37.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.011+0000 7fa7b3c7a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:37.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.011+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/4110259254 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa7ac125690 con 0x7fa7ac0a5240 2024-06-11T19:09:37.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.012+0000 7fa7a27fc640 1 -- 172.21.15.98:0/4110259254 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fa7a4030070 con 0x7fa7ac0a5240 2024-06-11T19:09:37.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.012+0000 7fa7a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-06-11T19:09:37.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.012+0000 7fa7a27fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:37.012 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:09:37.012 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-11T19:09:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 10 librados: watch_flush enter 2024-06-11T19:09:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 10 librados: watch_flush exit 2024-06-11T19:09:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 20 client.4349.objecter shutdown clearing up homeless session... 2024-06-11T19:09:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 10 client.4349.objecter successfully canceled tick 2024-06-11T19:09:37.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/4110259254 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa79003fdb0 msgr2=0x7fa790042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa79003fdb0 0x7fa790042270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8009360 tx=0x7fa7a800b040 comp rx=0 tx=0).stop 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 10 monclient: shutdown 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/4110259254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7ac0a5240 msgr2=0x7fa7ac123110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7ac0a5240 0x7fa7ac123110 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4033000 tx=0x7fa7a40046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/4110259254 shutdown_connections 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa79003fdb0 0x7fa790042270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7ac0a5240 0x7fa7ac123110 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7ac0a4920 0x7fa7ac137c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 --2- 172.21.15.98:0/4110259254 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ac0a2530 0x7fa7ac1376f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/4110259254 >> 172.21.15.98:0/4110259254 conn(0x7fa7ac0ab850 msgr2=0x7fa7ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/4110259254 shutdown_connections 2024-06-11T19:09:37.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 -- 172.21.15.98:0/4110259254 wait complete. 2024-06-11T19:09:37.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.014+0000 7fa7b3c7a640 1 librados: shutdown 2024-06-11T19:09:37.024 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-06-11T19:09:37.025 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-06-11T19:09:37.038 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-06-11T19:09:37.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-11T19:09:37.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.136+0000 7f89a793f640 10 monclient: get_monmap_and_config 2024-06-11T19:09:37.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.136+0000 7f89a793f640 10 monclient: build_initial_monmap 2024-06-11T19:09:37.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.136+0000 7f89a793f640 10 monclient: monmap: 2024-06-11T19:09:37.136 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:37.136 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.136 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:37.137059+0000 2024-06-11T19:09:37.136 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:37.137059+0000 2024-06-11T19:09:37.136 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:37.136 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.136 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:37.136 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:37.136 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:37.137 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:37.137 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:37.137 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:37.137 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:37.137 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:37.137 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:37.137 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:37.137 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:37.137 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.136+0000 7f89a793f640 1 Processor -- start 2024-06-11T19:09:37.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.136+0000 7f89a793f640 1 -- start start 2024-06-11T19:09:37.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.136+0000 7f89a793f640 10 monclient: init 2024-06-11T19:09:37.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:37.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 10 monclient: _add_conns ranks=[8,2,3,9,5,1,10,0,4,6,7] 2024-06-11T19:09:37.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89a00a4920 0x7f89a00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 10 monclient(hunting): picked mon.noname-e con 0x7f89a00a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:37.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89a00a5240 0x7f89a00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 10 monclient(hunting): picked mon.noname-b con 0x7f89a00a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:37.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89a00a2530 0x7f89a00a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 10 monclient(hunting): picked mon.noname-g con 0x7f89a00a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:37.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f89a00a7140 con 0x7f89a00a5240 2024-06-11T19:09:37.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f89a00a2b50 con 0x7f89a00a2530 2024-06-11T19:09:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f89a00a2e30 con 0x7f89a00a4920 2024-06-11T19:09:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a793f640 10 monclient(hunting): authenticate will time out at 786.944272s 2024-06-11T19:09:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a4eda640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89a00a5240 0x7f89a00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89a00a4920 0x7f89a00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89a00a4920 0x7f89a00a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54534/0 (socket says 172.21.15.98:54534) 2024-06-11T19:09:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a4eda640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89a00a5240 0x7f89a00a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:47994/0 (socket says 172.21.15.98:47994) 2024-06-11T19:09:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 1 -- 172.21.15.98:0/915984325 learned_addr learned my addr 172.21.15.98:0/915984325 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 10 monclient(hunting): get_auth_request con 0x7f89a00a4920 auth_method 0 2024-06-11T19:09:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a4eda640 10 monclient(hunting): get_auth_request con 0x7f89a00a5240 auth_method 0 2024-06-11T19:09:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a4eda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a4eda640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a4eda640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a4eda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a4eda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a4eda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a5edc640 1 --2- 172.21.15.98:0/915984325 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89a00a2530 0x7f89a00a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a5edc640 10 monclient(hunting): get_auth_request con 0x7f89a00a2530 auth_method 0 2024-06-11T19:09:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a5edc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a5edc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a5edc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 10 monclient(hunting): handle_auth_done global_id 4355 payload 274 2024-06-11T19:09:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 1 -- 172.21.15.98:0/915984325 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89a00a2530 msgr2=0x7f89a00a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 1 --2- 172.21.15.98:0/915984325 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89a00a2530 0x7f89a00a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 1 -- 172.21.15.98:0/915984325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89a00a5240 msgr2=0x7f89a00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 1 --2- 172.21.15.98:0/915984325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89a00a5240 0x7f89a00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 10 monclient: _finish_hunting 0 2024-06-11T19:09:37.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 1 monclient: found mon.noname-e 2024-06-11T19:09:37.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:37.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:37.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a56db640 1 -- 172.21.15.98:0/915984325 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a00a3110 con 0x7f89a00a4920 2024-06-11T19:09:37.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.137+0000 7f89a4eda640 1 --2- 172.21.15.98:0/915984325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89a00a5240 0x7f89a00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:37.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f89a56db640 1 --2- 172.21.15.98:0/915984325 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89a00a4920 0x7f89a00a4d00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f899c0098a0 tx=0x7f899c02b7a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=f9324d90c63f70aa server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:37.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 1 -- 172.21.15.98:0/915984325 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f899c038070 con 0x7f89a00a4920 2024-06-11T19:09:37.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:37.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:37.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 10 monclient: dump: 2024-06-11T19:09:37.152 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:37.153 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.153 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.153 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.153 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:37.153 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.153 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:37.153 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:37.153 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:37.153 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:37.153 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:37.153 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:37.154 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 10 monclient: _finish_auth 0 2024-06-11T19:09:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 1 -- 172.21.15.98:0/915984325 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f899c031970 con 0x7f89a00a4920 2024-06-11T19:09:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f89a793f640 5 monclient: authenticate success, global_id 4355 2024-06-11T19:09:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f89a793f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:37.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f89a793f640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 1 -- 172.21.15.98:0/915984325 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f899c031ba0 con 0x7f89a00a4920 2024-06-11T19:09:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f89a793f640 4 set_mon_vals no callback set 2024-06-11T19:09:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.138+0000 7f8997fff640 10 monclient: dump: 2024-06-11T19:09:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.155 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:37.156 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 10 monclient: shutdown 2024-06-11T19:09:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 1 -- 172.21.15.98:0/915984325 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89a00a4920 msgr2=0x7f89a00a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 1 --2- 172.21.15.98:0/915984325 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89a00a4920 0x7f89a00a4d00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f899c0098a0 tx=0x7f899c02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 1 -- 172.21.15.98:0/915984325 shutdown_connections 2024-06-11T19:09:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 1 --2- 172.21.15.98:0/915984325 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89a00a2530 0x7f89a00a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 1 --2- 172.21.15.98:0/915984325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89a00a5240 0x7f89a00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 1 --2- 172.21.15.98:0/915984325 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89a00a4920 0x7f89a00a4d00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 1 -- 172.21.15.98:0/915984325 >> 172.21.15.98:0/915984325 conn(0x7f89a00ab850 msgr2=0x7f89a00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 1 -- 172.21.15.98:0/915984325 shutdown_connections 2024-06-11T19:09:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 1 -- 172.21.15.98:0/915984325 wait complete. 2024-06-11T19:09:37.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 10 monclient: build_initial_monmap 2024-06-11T19:09:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 10 monclient: monmap: 2024-06-11T19:09:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:37.159 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 1 librados: starting msgr at 2024-06-11T19:09:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.139+0000 7f89a793f640 1 librados: starting objecter 2024-06-11T19:09:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 1 Processor -- start 2024-06-11T19:09:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 1 -- start start 2024-06-11T19:09:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 1 librados: setting wanted keys 2024-06-11T19:09:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 1 librados: calling monclient init 2024-06-11T19:09:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 10 monclient: init 2024-06-11T19:09:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 10 monclient: _add_conns ranks=[5,3,7,2,6,8,4,0,1] 2024-06-11T19:09:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f89a00a2530 0x7f89a0137740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 10 monclient(hunting): picked mon.noname-h con 0x7f89a00a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89a00a4920 0x7f89a0137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 10 monclient(hunting): picked mon.noname-g con 0x7f89a00a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f89a00a5240 0x7f89a01232a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 10 monclient(hunting): picked mon.noname-i con 0x7f89a00a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f89a00a6da0 con 0x7f89a00a4920 2024-06-11T19:09:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f89a0138e50 con 0x7f89a00a2530 2024-06-11T19:09:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f89a00a6420 con 0x7f89a00a5240 2024-06-11T19:09:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a793f640 10 monclient(hunting): authenticate will time out at 786.947547s 2024-06-11T19:09:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a4eda640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89a00a4920 0x7f89a0137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a56db640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f89a00a2530 0x7f89a0137740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a5edc640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f89a00a5240 0x7f89a01232a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a4eda640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89a00a4920 0x7f89a0137c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:53790/0 (socket says 172.21.15.98:53790) 2024-06-11T19:09:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a4eda640 1 -- 172.21.15.98:0/1038508274 learned_addr learned my addr 172.21.15.98:0/1038508274 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a4eda640 10 monclient(hunting): get_auth_request con 0x7f89a00a4920 auth_method 0 2024-06-11T19:09:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a4eda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a4eda640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a4eda640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a56db640 10 monclient(hunting): get_auth_request con 0x7f89a00a2530 auth_method 0 2024-06-11T19:09:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a56db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a56db640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a56db640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a5edc640 10 monclient(hunting): get_auth_request con 0x7f89a00a5240 auth_method 0 2024-06-11T19:09:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a5edc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a5edc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.140+0000 7f89a5edc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a5edc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a5edc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a5edc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a56db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a56db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a56db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2024-06-11T19:09:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 1 -- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f89a00a5240 msgr2=0x7f89a01232a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f89a00a5240 0x7f89a01232a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 1 -- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f89a00a2530 msgr2=0x7f89a0137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f89a00a2530 0x7f89a0137740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 10 monclient: _finish_hunting 0 2024-06-11T19:09:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 1 monclient: found mon.noname-g 2024-06-11T19:09:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:09:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 1 -- 172.21.15.98:0/1038508274 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a01c0d10 con 0x7f89a00a4920 2024-06-11T19:09:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a56db640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f89a00a2530 0x7f89a0137740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a5edc640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f89a00a5240 0x7f89a01232a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a4eda640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89a00a4920 0x7f89a0137c80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f899000e8d0 tx=0x7f899000ed90 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 10 client.?.objecter ms_handle_connect 0x7f89a00a4920 2024-06-11T19:09:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 1 -- 172.21.15.98:0/1038508274 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8990017070 con 0x7f89a00a4920 2024-06-11T19:09:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:09:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 10 monclient: dump: 2024-06-11T19:09:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:37.166 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:37.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 1 -- 172.21.15.98:0/1038508274 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f899000cdb0 con 0x7f89a00a4920 2024-06-11T19:09:37.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a793f640 5 monclient: authenticate success, global_id 4332 2024-06-11T19:09:37.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a793f640 10 monclient: _renew_subs 2024-06-11T19:09:37.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a793f640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:09:37.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a793f640 1 -- 172.21.15.98:0/1038508274 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a0126b80 con 0x7f89a00a4920 2024-06-11T19:09:37.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:37.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 1 -- 172.21.15.98:0/1038508274 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f89900138e0 con 0x7f89a00a4920 2024-06-11T19:09:37.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:37.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:09:37.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f8995ffb640 10 monclient: dump: 2024-06-11T19:09:37.167 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:37.168 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:37.169 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a793f640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89977fe640 4 set_mon_vals no callback set 2024-06-11T19:09:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a793f640 10 monclient: _renew_subs 2024-06-11T19:09:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a793f640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:09:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a793f640 1 -- 172.21.15.98:0/1038508274 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a00a92f0 con 0x7f89a00a4920 2024-06-11T19:09:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.141+0000 7f89a793f640 1 librados: init done 2024-06-11T19:09:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.142+0000 7f8995ffb640 1 -- 172.21.15.98:0/1038508274 <== mon.3 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f899001a070 con 0x7f89a00a4920 2024-06-11T19:09:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.142+0000 7f89a793f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.142+0000 7f89a793f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.142+0000 7f89a793f640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:09:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.142+0000 7f89a793f640 1 -- 172.21.15.98:0/1038508274 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a0073550 con 0x7f89a00a4920 2024-06-11T19:09:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.143+0000 7f8995ffb640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f898403fdb0 0x7f8984042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.143+0000 7f8995ffb640 1 -- 172.21.15.98:0/1038508274 <== mon.3 v2:172.21.15.117:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f899000f040 con 0x7f89a00a4920 2024-06-11T19:09:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.143+0000 7f8995ffb640 10 client.4332.objecter ms_dispatch 0x7f89a0000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.143+0000 7f8995ffb640 3 client.4332.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.143+0000 7f8995ffb640 3 client.4332.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.143+0000 7f8995ffb640 20 client.4332.objecter dump_active .. 0 homeless 2024-06-11T19:09:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.143+0000 7f89a56db640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f898403fdb0 0x7f8984042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.143+0000 7f89a56db640 10 monclient: get_auth_request con 0x7f898403fdb0 auth_method 0 2024-06-11T19:09:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.143+0000 7f8995ffb640 10 client.4332.objecter ms_handle_connect 0x7f898403fdb0 2024-06-11T19:09:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.143+0000 7f89a56db640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f898403fdb0 0x7f8984042270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f899c035000 tx=0x7f899c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.148+0000 7f8995ffb640 1 -- 172.21.15.98:0/1038508274 <== mon.3 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f89a0073550 con 0x7f89a00a4920 2024-06-11T19:09:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.148+0000 7f8995ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.148+0000 7f8995ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.329+0000 7f89a793f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.329+0000 7f89a793f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.329+0000 7f89a793f640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:09:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.329+0000 7f89a793f640 1 -- 172.21.15.98:0/1038508274 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89a00a66f0 con 0x7f89a00a4920 2024-06-11T19:09:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.330+0000 7f8995ffb640 1 -- 172.21.15.98:0/1038508274 <== mon.3 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12247 (secure 0 0 0) 0x7f8990021070 con 0x7f89a00a4920 2024-06-11T19:09:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.330+0000 7f8995ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:09:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.330+0000 7f8995ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:09:37.331 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","created":"2024-06-11T19:09:15.016586+0000","modified":"2024-06-11T19:09:34.710947+0000","last_up_change":"2024-06-11T19:09:24.656699+0000","last_in_change":"2024-06-11T19:09:16.662340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-11T19:09:25.381456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-11T19:09:31.785910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ef1075e3-3e1b-4904-b14b-90751df6c38c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6811","nonce":4087730304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6815","nonce":4087730304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6823","nonce":4087730304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6819","nonce":4087730304}]},"public_addr":"172.21.15.98:6811/4087730304","cluster_addr":"172.21.15.98:6815/4087730304","heartbeat_back_addr":"172.21.15.98:6823/4087730304","heartbeat_front_addr":"172.21.15.98:6819/4087730304","state":["exists","up"]},{"osd":1,"uuid":"985893e9-bd60-46fc-b913-7502ff73628d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6801","nonce":3886067876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6803","nonce":3886067876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6807","nonce":3886067876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6805","nonce":3886067876}]},"public_addr":"172.21.15.98:6801/3886067876","cluster_addr":"172.21.15.98:6803/3886067876","heartbeat_back_addr":"172.21.15.98:6807/3886067876","heartbeat_front_addr":"172.21.15.98:6805/3886067876","state":["exists","up"]},{"osd":2,"uuid":"2b0b4d24-8009-4cec-80e4-c0fa2d705879","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6810","nonce":3751501255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6814","nonce":3751501255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6822","nonce":3751501255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6818","nonce":3751501255}]},"public_addr":"172.21.15.98:6810/3751501255","cluster_addr":"172.21.15.98:6814/3751501255","heartbeat_back_addr":"172.21.15.98:6822/3751501255","heartbeat_front_addr":"172.21.15.98:6818/3751501255","state":["exists","up"]},{"osd":3,"uuid":"73793608-9e04-4837-bd2b-1b130a4e88d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6809","nonce":2561136926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6811","nonce":2561136926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6815","nonce":2561136926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6813","nonce":2561136926}]},"public_addr":"172.21.15.117:6809/2561136926","cluster_addr":"172.21.15.117:6811/2561136926","heartbeat_back_addr":"172.21.15.117:6815/2561136926","heartbeat_front_addr":"172.21.15.117:6813/2561136926","state":["exists","up"]},{"osd":4,"uuid":"60e657ae-e2a2-471e-b998-fba98d9c94e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6801","nonce":3877218228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6803","nonce":3877218228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6807","nonce":3877218228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6805","nonce":3877218228}]},"public_addr":"172.21.15.117:6801/3877218228","cluster_addr":"172.21.15.117:6803/3877218228","heartbeat_back_addr":"172.21.15.117:6807/3877218228","heartbeat_front_addr":"172.21.15.117:6805/3877218228","state":["exists","up"]},{"osd":5,"uuid":"dae66a65-2ca2-4f8a-98e3-52b190abee7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6817","nonce":987023138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6819","nonce":987023138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6823","nonce":987023138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6821","nonce":987023138}]},"public_addr":"172.21.15.117:6817/987023138","cluster_addr":"172.21.15.117:6819/987023138","heartbeat_back_addr":"172.21.15.117:6823/987023138","heartbeat_front_addr":"172.21.15.117:6821/987023138","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.744223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.800796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.792849+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.854633+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.844451+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.763441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-11T19:09:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 10 librados: watch_flush enter 2024-06-11T19:09:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 10 librados: watch_flush exit 2024-06-11T19:09:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 20 client.4332.objecter shutdown clearing up homeless session... 2024-06-11T19:09:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 10 client.4332.objecter successfully canceled tick 2024-06-11T19:09:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 1 -- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f898403fdb0 msgr2=0x7f8984042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f898403fdb0 0x7f8984042270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f899c035000 tx=0x7f899c008030 comp rx=0 tx=0).stop 2024-06-11T19:09:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 10 monclient: shutdown 2024-06-11T19:09:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 1 -- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89a00a4920 msgr2=0x7f89a0137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89a00a4920 0x7f89a0137c80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f899000e8d0 tx=0x7f899000ed90 comp rx=0 tx=0).stop 2024-06-11T19:09:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 1 -- 172.21.15.98:0/1038508274 shutdown_connections 2024-06-11T19:09:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f898403fdb0 0x7f8984042270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f89a00a5240 0x7f89a01232a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89a00a4920 0x7f89a0137c80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 1 --2- 172.21.15.98:0/1038508274 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f89a00a2530 0x7f89a0137740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.332+0000 7f89a793f640 1 -- 172.21.15.98:0/1038508274 >> 172.21.15.98:0/1038508274 conn(0x7f89a00ab850 msgr2=0x7f89a00a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.333+0000 7f89a793f640 1 -- 172.21.15.98:0/1038508274 shutdown_connections 2024-06-11T19:09:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.333+0000 7f89a793f640 1 -- 172.21.15.98:0/1038508274 wait complete. 2024-06-11T19:09:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.333+0000 7f89a793f640 1 librados: shutdown 2024-06-11T19:09:37.343 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-11T19:09:25.381456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-06-11T19:09:31.785910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-11T19:09:37.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-11T19:09:37.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.446+0000 7f939735f640 10 monclient: get_monmap_and_config 2024-06-11T19:09:37.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.447+0000 7f939735f640 10 monclient: build_initial_monmap 2024-06-11T19:09:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.447+0000 7f939735f640 10 monclient: monmap: 2024-06-11T19:09:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:37.448048+0000 2024-06-11T19:09:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:37.448048+0000 2024-06-11T19:09:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:37.447 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:37.448 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:37.448 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.447+0000 7f939735f640 1 Processor -- start 2024-06-11T19:09:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.447+0000 7f939735f640 1 -- start start 2024-06-11T19:09:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.447+0000 7f939735f640 10 monclient: init 2024-06-11T19:09:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 10 monclient: _add_conns ranks=[4,8,2,0,1,9,6,10,5,3,7] 2024-06-11T19:09:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9390067ce0 0x7f939009e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 10 monclient(hunting): picked mon.noname-c con 0x7f9390067ce0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93900680c0 0x7f93900684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 10 monclient(hunting): picked mon.noname-e con 0x7f93900680c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9390132410 0x7f9390132870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 10 monclient(hunting): picked mon.noname-b con 0x7f9390132410 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f939009dad0 con 0x7f9390132410 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f939009d480 con 0x7f9390067ce0 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f939009d760 con 0x7f93900680c0 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f939735f640 10 monclient(hunting): authenticate will time out at 787.255428s 2024-06-11T19:09:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93900680c0 0x7f93900684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93950fb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9390067ce0 0x7f939009e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93958fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9390132410 0x7f9390132870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93900680c0 0x7f93900684c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54542/0 (socket says 172.21.15.98:54542) 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93950fb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9390067ce0 0x7f939009e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:40454/0 (socket says 172.21.15.98:40454) 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 1 -- 172.21.15.98:0/4133074818 learned_addr learned my addr 172.21.15.98:0/4133074818 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93958fc640 10 monclient(hunting): get_auth_request con 0x7f9390132410 auth_method 0 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93958fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93958fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93958fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 10 monclient(hunting): get_auth_request con 0x7f93900680c0 auth_method 0 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93950fb640 10 monclient(hunting): get_auth_request con 0x7f9390067ce0 auth_method 0 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93950fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93950fb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93950fb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93950fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93950fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93950fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93958fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93958fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93958fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 10 monclient(hunting): handle_auth_done global_id 4364 payload 274 2024-06-11T19:09:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 1 -- 172.21.15.98:0/4133074818 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9390067ce0 msgr2=0x7f939009e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 1 --2- 172.21.15.98:0/4133074818 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9390067ce0 0x7f939009e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 1 -- 172.21.15.98:0/4133074818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9390132410 msgr2=0x7f9390132870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 1 --2- 172.21.15.98:0/4133074818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9390132410 0x7f9390132870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 10 monclient: _finish_hunting 0 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 1 monclient: found mon.noname-e 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93948fa640 1 -- 172.21.15.98:0/4133074818 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939009e760 con 0x7f93900680c0 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93950fb640 1 --2- 172.21.15.98:0/4133074818 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9390067ce0 0x7f939009e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.448+0000 7f93958fc640 1 --2- 172.21.15.98:0/4133074818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9390132410 0x7f9390132870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f93948fa640 1 --2- 172.21.15.98:0/4133074818 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93900680c0 0x7f93900684c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f938c0098a0 tx=0x7f938c02b7a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=c5e2fd1edb2721c9 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 1 -- 172.21.15.98:0/4133074818 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f938c038070 con 0x7f93900680c0 2024-06-11T19:09:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 10 monclient: dump: 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 10 monclient: _finish_auth 0 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 1 -- 172.21.15.98:0/4133074818 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f938c031970 con 0x7f93900680c0 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f939735f640 5 monclient: authenticate success, global_id 4364 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f939735f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f939735f640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 1 -- 172.21.15.98:0/4133074818 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f938c031ba0 con 0x7f93900680c0 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f939735f640 4 set_mon_vals no callback set 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.449+0000 7f9383fff640 10 monclient: dump: 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.450+0000 7f939735f640 10 monclient: shutdown 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.450+0000 7f939735f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.450+0000 7f939735f640 1 -- 172.21.15.98:0/4133074818 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93900680c0 msgr2=0x7f93900684c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.450+0000 7f939735f640 1 --2- 172.21.15.98:0/4133074818 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93900680c0 0x7f93900684c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f938c0098a0 tx=0x7f938c02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.450+0000 7f939735f640 1 -- 172.21.15.98:0/4133074818 shutdown_connections 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.450+0000 7f939735f640 1 --2- 172.21.15.98:0/4133074818 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9390132410 0x7f9390132870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.450+0000 7f939735f640 1 --2- 172.21.15.98:0/4133074818 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93900680c0 0x7f93900684c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.450+0000 7f939735f640 1 --2- 172.21.15.98:0/4133074818 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9390067ce0 0x7f939009e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.450+0000 7f939735f640 1 -- 172.21.15.98:0/4133074818 >> 172.21.15.98:0/4133074818 conn(0x7f93900a4ba0 msgr2=0x7f93900a4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.450+0000 7f939735f640 1 -- 172.21.15.98:0/4133074818 shutdown_connections 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.450+0000 7f939735f640 1 -- 172.21.15.98:0/4133074818 wait complete. 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient: build_initial_monmap 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient: monmap: 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 1 librados: starting msgr at 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 1 librados: starting objecter 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 1 Processor -- start 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 1 -- start start 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 1 librados: setting wanted keys 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 1 librados: calling monclient init 2024-06-11T19:09:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient: init 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient: _add_conns ranks=[0,6,5,2,8,3,7,4,1] 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9390067ce0 0x7f939011ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient(hunting): picked mon.noname-a con 0x7f9390067ce0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f93900680c0 0x7f939011f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient(hunting): picked mon.noname-d con 0x7f93900680c0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9390132410 0x7f939011f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient(hunting): picked mon.noname-h con 0x7f9390132410 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9390068a00 con 0x7f9390067ce0 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f9390073bd0 con 0x7f9390132410 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f9390132db0 con 0x7f93900680c0 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.451+0000 7f939735f640 10 monclient(hunting): authenticate will time out at 787.258987s 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9390067ce0 0x7f939011ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9390067ce0 0x7f939011ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45738/0 (socket says 172.21.15.98:45738) 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 1 -- 172.21.15.98:0/2623862228 learned_addr learned my addr 172.21.15.98:0/2623862228 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 10 monclient(hunting): get_auth_request con 0x7f9390067ce0 auth_method 0 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93948fa640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f93900680c0 0x7f939011f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93948fa640 10 monclient(hunting): get_auth_request con 0x7f93900680c0 auth_method 0 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93948fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93948fa640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93948fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93948fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93948fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93948fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 1 -- 172.21.15.98:0/2623862228 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f93900680c0 msgr2=0x7f939011f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f93900680c0 0x7f939011f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 1 -- 172.21.15.98:0/2623862228 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9390132410 msgr2=0x7f939011f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93958fc640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9390132410 0x7f939011f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9390132410 0x7f939011f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 10 monclient: _finish_hunting 0 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 1 monclient: found mon.noname-a 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 1 -- 172.21.15.98:0/2623862228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9390139b00 con 0x7f9390067ce0 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93950fb640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9390067ce0 0x7f939011ece0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f938400ec10 tx=0x7f938400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 10 client.?.objecter ms_handle_connect 0x7f9390067ce0 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 1 -- 172.21.15.98:0/2623862228 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9384017070 con 0x7f9390067ce0 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 10 monclient: dump: 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:37.472 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f939735f640 5 monclient: authenticate success, global_id 4365 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 1 -- 172.21.15.98:0/2623862228 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f938400cc30 con 0x7f9390067ce0 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f939735f640 10 monclient: _renew_subs 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f939735f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f939735f640 1 -- 172.21.15.98:0/2623862228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93900686a0 con 0x7f9390067ce0 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 1 -- 172.21.15.98:0/2623862228 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f938400cde0 con 0x7f9390067ce0 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f9381ffb640 10 monclient: dump: 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f93837fe640 4 set_mon_vals no callback set 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.452+0000 7f939735f640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.453+0000 7f939735f640 10 monclient: _renew_subs 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.453+0000 7f939735f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.453+0000 7f939735f640 1 -- 172.21.15.98:0/2623862228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9390122840 con 0x7f9390067ce0 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.453+0000 7f939735f640 1 librados: init done 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.453+0000 7f9381ffb640 1 -- 172.21.15.98:0/2623862228 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f938401a070 con 0x7f9390067ce0 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.453+0000 7f939735f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.453+0000 7f939735f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.453+0000 7f939735f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.453+0000 7f939735f640 1 -- 172.21.15.98:0/2623862228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93901330e0 con 0x7f9390067ce0 2024-06-11T19:09:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.454+0000 7f9381ffb640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f937403fdb0 0x7f9374042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.454+0000 7f9381ffb640 1 -- 172.21.15.98:0/2623862228 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f938400f040 con 0x7f9390067ce0 2024-06-11T19:09:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.454+0000 7f9381ffb640 10 client.4365.objecter ms_dispatch 0x7f9390000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.454+0000 7f9381ffb640 3 client.4365.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.454+0000 7f9381ffb640 3 client.4365.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.454+0000 7f9381ffb640 20 client.4365.objecter dump_active .. 0 homeless 2024-06-11T19:09:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.454+0000 7f93948fa640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f937403fdb0 0x7f9374042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.454+0000 7f93948fa640 10 monclient: get_auth_request con 0x7f937403fdb0 auth_method 0 2024-06-11T19:09:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.454+0000 7f9381ffb640 10 client.4365.objecter ms_handle_connect 0x7f937403fdb0 2024-06-11T19:09:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.454+0000 7f93948fa640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f937403fdb0 0x7f9374042270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f938c035000 tx=0x7f938c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.458+0000 7f9381ffb640 1 -- 172.21.15.98:0/2623862228 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9384021070 con 0x7f9390067ce0 2024-06-11T19:09:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.458+0000 7f9381ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.458+0000 7f9381ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.643+0000 7f939735f640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-06-11T19:09:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.643+0000 7f939735f640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-06-11T19:09:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.643+0000 7f939735f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.643+0000 7f939735f640 1 -- 172.21.15.98:0/2623862228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f939006d540 con 0x7f9390067ce0 2024-06-11T19:09:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.644+0000 7f9381ffb640 1 -- 172.21.15.98:0/2623862228 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) ==== 93+0+10 (secure 0 0 0) 0x7f93840510c0 con 0x7f9390067ce0 2024-06-11T19:09:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.644+0000 7f9381ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-06-11T19:09:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.644+0000 7f9381ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:37.644 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-06-11T19:09:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 10 librados: watch_flush enter 2024-06-11T19:09:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 10 librados: watch_flush exit 2024-06-11T19:09:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 20 client.4365.objecter shutdown clearing up homeless session... 2024-06-11T19:09:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 10 client.4365.objecter successfully canceled tick 2024-06-11T19:09:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 -- 172.21.15.98:0/2623862228 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f937403fdb0 msgr2=0x7f9374042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f937403fdb0 0x7f9374042270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f938c035000 tx=0x7f938c008030 comp rx=0 tx=0).stop 2024-06-11T19:09:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 10 monclient: shutdown 2024-06-11T19:09:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 -- 172.21.15.98:0/2623862228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9390067ce0 msgr2=0x7f939011ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9390067ce0 0x7f939011ece0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f938400ec10 tx=0x7f938400c520 comp rx=0 tx=0).stop 2024-06-11T19:09:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 -- 172.21.15.98:0/2623862228 shutdown_connections 2024-06-11T19:09:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f937403fdb0 0x7f9374042270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9390132410 0x7f939011f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f93900680c0 0x7f939011f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 --2- 172.21.15.98:0/2623862228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9390067ce0 0x7f939011ece0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 -- 172.21.15.98:0/2623862228 >> 172.21.15.98:0/2623862228 conn(0x7f93900a4ba0 msgr2=0x7f9390134e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 -- 172.21.15.98:0/2623862228 shutdown_connections 2024-06-11T19:09:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 -- 172.21.15.98:0/2623862228 wait complete. 2024-06-11T19:09:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.646+0000 7f939735f640 1 librados: shutdown 2024-06-11T19:09:37.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-06-11T19:09:37.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.767+0000 7fe4b3566640 10 monclient: get_monmap_and_config 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.767+0000 7fe4b3566640 10 monclient: build_initial_monmap 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.767+0000 7fe4b3566640 10 monclient: monmap: 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:37.768104+0000 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:37.768104+0000 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:37.768 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:37.768 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:37.768 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:37.768 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:37.768 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:37.768 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:37.768 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.767+0000 7fe4b3566640 1 Processor -- start 2024-06-11T19:09:37.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.767+0000 7fe4b3566640 1 -- start start 2024-06-11T19:09:37.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.767+0000 7fe4b3566640 10 monclient: init 2024-06-11T19:09:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 10 monclient: _add_conns ranks=[6,9,5,1,4,7,10,0,8,3,2] 2024-06-11T19:09:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4ac067ce0 0x7fe4ac09e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 10 monclient(hunting): picked mon.noname-d con 0x7fe4ac067ce0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe4ac0680c0 0x7fe4ac0684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 10 monclient(hunting): picked mon.noname-a con 0x7fe4ac0680c0 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4ac132410 0x7fe4ac132870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 10 monclient(hunting): picked mon.noname-h con 0x7fe4ac132410 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe4ac09dad0 con 0x7fe4ac0680c0 2024-06-11T19:09:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fe4ac09d480 con 0x7fe4ac132410 2024-06-11T19:09:37.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe4ac09d760 con 0x7fe4ac067ce0 2024-06-11T19:09:37.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:37.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b3566640 10 monclient(hunting): authenticate will time out at 787.575473s 2024-06-11T19:09:37.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1302640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4ac067ce0 0x7fe4ac09e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe4ac0680c0 0x7fe4ac0684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1302640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4ac067ce0 0x7fe4ac09e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49782/0 (socket says 172.21.15.98:49782) 2024-06-11T19:09:37.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 10 monclient(hunting): get_auth_request con 0x7fe4ac0680c0 auth_method 0 2024-06-11T19:09:37.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1302640 1 -- 172.21.15.98:0/371381837 learned_addr learned my addr 172.21.15.98:0/371381837 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:37.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1302640 10 monclient(hunting): get_auth_request con 0x7fe4ac067ce0 auth_method 0 2024-06-11T19:09:37.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1302640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1302640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1302640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1302640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1302640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1b03640 1 --2- 172.21.15.98:0/371381837 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4ac132410 0x7fe4ac132870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1b03640 10 monclient(hunting): get_auth_request con 0x7fe4ac132410 auth_method 0 2024-06-11T19:09:37.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1b03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1b03640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1b03640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 10 monclient(hunting): handle_auth_done global_id 4374 payload 274 2024-06-11T19:09:37.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 1 -- 172.21.15.98:0/371381837 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4ac067ce0 msgr2=0x7fe4ac09e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:37.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 1 --2- 172.21.15.98:0/371381837 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4ac067ce0 0x7fe4ac09e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 1 -- 172.21.15.98:0/371381837 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4ac132410 msgr2=0x7fe4ac132870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:37.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 1 --2- 172.21.15.98:0/371381837 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4ac132410 0x7fe4ac132870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 10 monclient: _finish_hunting 0 2024-06-11T19:09:37.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 1 monclient: found mon.noname-a 2024-06-11T19:09:37.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:37.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:37.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b0b01640 1 -- 172.21.15.98:0/371381837 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4ac09e760 con 0x7fe4ac0680c0 2024-06-11T19:09:37.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.768+0000 7fe4b1302640 1 --2- 172.21.15.98:0/371381837 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4ac067ce0 0x7fe4ac09e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:37.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4b0b01640 1 --2- 172.21.15.98:0/371381837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4ac0680c0 0x7fe4ac0684c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009d90 tx=0x7fe4a002bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752ac643c1d3dc16 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:37.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4abfff640 1 -- 172.21.15.98:0/371381837 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a0039070 con 0x7fe4ac0680c0 2024-06-11T19:09:37.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:37.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:37.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4abfff640 10 monclient: dump: 2024-06-11T19:09:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:37.785 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4abfff640 10 monclient: _finish_auth 0 2024-06-11T19:09:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4abfff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4b3566640 5 monclient: authenticate success, global_id 4374 2024-06-11T19:09:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4abfff640 1 -- 172.21.15.98:0/371381837 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4a002cd00 con 0x7fe4ac0680c0 2024-06-11T19:09:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4b3566640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4abfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4b3566640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4b3566640 4 set_mon_vals no callback set 2024-06-11T19:09:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4abfff640 1 -- 172.21.15.98:0/371381837 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a00327b0 con 0x7fe4ac0680c0 2024-06-11T19:09:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.769+0000 7fe4abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4abfff640 10 monclient: dump: 2024-06-11T19:09:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4b3566640 10 monclient: shutdown 2024-06-11T19:09:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4b3566640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4b3566640 1 -- 172.21.15.98:0/371381837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4ac0680c0 msgr2=0x7fe4ac0684c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4b3566640 1 --2- 172.21.15.98:0/371381837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4ac0680c0 0x7fe4ac0684c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009d90 tx=0x7fe4a002bce0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4b3566640 1 -- 172.21.15.98:0/371381837 shutdown_connections 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4b3566640 1 --2- 172.21.15.98:0/371381837 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4ac132410 0x7fe4ac132870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4b3566640 1 --2- 172.21.15.98:0/371381837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4ac0680c0 0x7fe4ac0684c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4b3566640 1 --2- 172.21.15.98:0/371381837 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4ac067ce0 0x7fe4ac09e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4b3566640 1 -- 172.21.15.98:0/371381837 >> 172.21.15.98:0/371381837 conn(0x7fe4ac0a4ba0 msgr2=0x7fe4ac0a4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4b3566640 1 -- 172.21.15.98:0/371381837 shutdown_connections 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.770+0000 7fe4b3566640 1 -- 172.21.15.98:0/371381837 wait complete. 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.771+0000 7fe4b3566640 10 monclient: build_initial_monmap 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.771+0000 7fe4b3566640 10 monclient: monmap: 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.771+0000 7fe4b3566640 1 librados: starting msgr at 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.771+0000 7fe4b3566640 1 librados: starting objecter 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.771+0000 7fe4b3566640 1 Processor -- start 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.771+0000 7fe4b3566640 1 -- start start 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.771+0000 7fe4b3566640 1 librados: setting wanted keys 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.771+0000 7fe4b3566640 1 librados: calling monclient init 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.771+0000 7fe4b3566640 10 monclient: init 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 10 monclient: _add_conns ranks=[3,4,5,6,2,0,1,8,7] 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4ac067ce0 0x7fe4ac1cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 10 monclient(hunting): picked mon.noname-g con 0x7fe4ac067ce0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0680c0 0x7fe4ac1ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 10 monclient(hunting): picked mon.noname-c con 0x7fe4ac0680c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4ac132410 0x7fe4ac1ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 10 monclient(hunting): picked mon.noname-h con 0x7fe4ac132410 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fe4ac068a00 con 0x7fe4ac067ce0 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe4ac073bd0 con 0x7fe4ac0680c0 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fe4ac132db0 con 0x7fe4ac132410 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 10 monclient(hunting): authenticate will time out at 787.579244s 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0680c0 0x7fe4ac1ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0680c0 0x7fe4ac1ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:40456/0 (socket says 172.21.15.98:40456) 2024-06-11T19:09:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 1 -- 172.21.15.98:0/1857155043 learned_addr learned my addr 172.21.15.98:0/1857155043 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 10 monclient(hunting): get_auth_request con 0x7fe4ac0680c0 auth_method 0 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1302640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4ac067ce0 0x7fe4ac1cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1302640 10 monclient(hunting): get_auth_request con 0x7fe4ac067ce0 auth_method 0 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1302640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1302640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1b03640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4ac132410 0x7fe4ac1ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1b03640 10 monclient(hunting): get_auth_request con 0x7fe4ac132410 auth_method 0 2024-06-11T19:09:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1b03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1b03640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1b03640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1302640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1302640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1302640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-06-11T19:09:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 1 -- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4ac132410 msgr2=0x7fe4ac1ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4ac132410 0x7fe4ac1ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 1 -- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4ac067ce0 msgr2=0x7fe4ac1cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4ac067ce0 0x7fe4ac1cddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 10 monclient: _finish_hunting 0 2024-06-11T19:09:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 1 monclient: found mon.noname-c 2024-06-11T19:09:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 1 -- 172.21.15.98:0/1857155043 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4ac0a5230 con 0x7fe4ac0680c0 2024-06-11T19:09:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b1b03640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4ac132410 0x7fe4ac1ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b0b01640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0680c0 0x7fe4ac1ce300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0000f80 tx=0x7fe4a0004a40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 10 client.?.objecter ms_handle_connect 0x7fe4ac0680c0 2024-06-11T19:09:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1857155043 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a0039050 con 0x7fe4ac0680c0 2024-06-11T19:09:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 10 monclient: dump: 2024-06-11T19:09:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1857155043 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4a0045070 con 0x7fe4ac0680c0 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 5 monclient: authenticate success, global_id 4396 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 10 monclient: _renew_subs 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4b3566640 1 -- 172.21.15.98:0/1857155043 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4ac132b00 con 0x7fe4ac0680c0 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1857155043 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a00046c0 con 0x7fe4ac0680c0 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4a9ffb640 10 monclient: dump: 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.772+0000 7fe4ab7fe640 4 set_mon_vals no callback set 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.773+0000 7fe4b3566640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.773+0000 7fe4b3566640 10 monclient: _renew_subs 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.773+0000 7fe4b3566640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.773+0000 7fe4b3566640 1 -- 172.21.15.98:0/1857155043 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4ac122220 con 0x7fe4ac0680c0 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.773+0000 7fe4b3566640 1 librados: init done 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.773+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1857155043 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fe4a0004250 con 0x7fe4ac0680c0 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4a9ffb640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe49803fdb0 0x7fe498042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1857155043 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fe4a0073b90 con 0x7fe4ac0680c0 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4a9ffb640 10 client.4396.objecter ms_dispatch 0x7fe4ac000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4a9ffb640 3 client.4396.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4a9ffb640 3 client.4396.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4b3566640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4b3566640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4b3566640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4b3566640 1 -- 172.21.15.98:0/1857155043 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4ac122020 con 0x7fe4ac0680c0 2024-06-11T19:09:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4a9ffb640 20 client.4396.objecter dump_active .. 0 homeless 2024-06-11T19:09:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4b1302640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe49803fdb0 0x7fe498042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4b1302640 10 monclient: get_auth_request con 0x7fe49803fdb0 auth_method 0 2024-06-11T19:09:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4b1302640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe49803fdb0 0x7fe498042270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4006ca0 tx=0x7fe4a4009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.774+0000 7fe4a9ffb640 10 client.4396.objecter ms_handle_connect 0x7fe49803fdb0 2024-06-11T19:09:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.778+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1857155043 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe4a002cd00 con 0x7fe4ac0680c0 2024-06-11T19:09:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.778+0000 7fe4a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.778+0000 7fe4a9ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.965+0000 7fe4b3566640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-06-11T19:09:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.965+0000 7fe4b3566640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-06-11T19:09:37.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.965+0000 7fe4b3566640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:37.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.965+0000 7fe4b3566640 1 -- 172.21.15.98:0/1857155043 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7fe4ac1204f0 con 0x7fe4ac0680c0 2024-06-11T19:09:37.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.966+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1857155043 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v18) ==== 92+0+10 (secure 0 0 0) 0x7fe4a004a460 con 0x7fe4ac0680c0 2024-06-11T19:09:37.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.966+0000 7fe4a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-06-11T19:09:37.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.966+0000 7fe4a9ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:37.966 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2024-06-11T19:09:37.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 10 librados: watch_flush enter 2024-06-11T19:09:37.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 10 librados: watch_flush exit 2024-06-11T19:09:37.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 20 client.4396.objecter shutdown clearing up homeless session... 2024-06-11T19:09:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 10 client.4396.objecter successfully canceled tick 2024-06-11T19:09:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 -- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe49803fdb0 msgr2=0x7fe498042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe49803fdb0 0x7fe498042270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4006ca0 tx=0x7fe4a4009240 comp rx=0 tx=0).stop 2024-06-11T19:09:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 10 monclient: shutdown 2024-06-11T19:09:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 -- 172.21.15.98:0/1857155043 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0680c0 msgr2=0x7fe4ac1ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0680c0 0x7fe4ac1ce300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0000f80 tx=0x7fe4a0004a40 comp rx=0 tx=0).stop 2024-06-11T19:09:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 -- 172.21.15.98:0/1857155043 shutdown_connections 2024-06-11T19:09:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe49803fdb0 0x7fe498042270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4ac132410 0x7fe4ac1ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0680c0 0x7fe4ac1ce300 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 --2- 172.21.15.98:0/1857155043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4ac067ce0 0x7fe4ac1cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:37.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 -- 172.21.15.98:0/1857155043 >> 172.21.15.98:0/1857155043 conn(0x7fe4ac0a4ba0 msgr2=0x7fe4ac134db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:37.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 -- 172.21.15.98:0/1857155043 shutdown_connections 2024-06-11T19:09:37.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 -- 172.21.15.98:0/1857155043 wait complete. 2024-06-11T19:09:37.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:37.968+0000 7fe4b3566640 1 librados: shutdown 2024-06-11T19:09:37.978 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-06-11T19:09:37.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-11T19:09:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.080+0000 7f9994fa8640 10 monclient: get_monmap_and_config 2024-06-11T19:09:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.080+0000 7f9994fa8640 10 monclient: build_initial_monmap 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.080+0000 7f9994fa8640 10 monclient: monmap: 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:38.081444+0000 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:38.081444+0000 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:38.082 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:38.082 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:38.082 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:38.082 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:38.082 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:38.082 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.080+0000 7f9994fa8640 1 Processor -- start 2024-06-11T19:09:38.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 1 -- start start 2024-06-11T19:09:38.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 10 monclient: init 2024-06-11T19:09:38.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:38.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 10 monclient: _add_conns ranks=[6,8,10,1,7,0,4,9,3,2,5] 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f99900a4880 0x7f99900a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 10 monclient(hunting): picked mon.noname-d con 0x7f99900a4880 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f99900a5e40 0x7f9990138860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 10 monclient(hunting): picked mon.noname-e con 0x7f99900a5e40 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f99900a51a0 0x7f9990138da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 10 monclient(hunting): picked mon.noname-f con 0x7f99900a51a0 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f99900a6e10 con 0x7f99900a51a0 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f99900a74a0 con 0x7f99900a4880 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f99900a6fe0 con 0x7f99900a5e40 2024-06-11T19:09:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f9994fa8640 10 monclient(hunting): authenticate will time out at 787.888680s 2024-06-11T19:09:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998e59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f99900a4880 0x7f99900a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998dd9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f99900a5e40 0x7f9990138860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998e59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f99900a4880 0x7f99900a4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49794/0 (socket says 172.21.15.98:49794) 2024-06-11T19:09:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998e59c640 1 -- 172.21.15.98:0/447521589 learned_addr learned my addr 172.21.15.98:0/447521589 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998e59c640 10 monclient(hunting): get_auth_request con 0x7f99900a4880 auth_method 0 2024-06-11T19:09:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998dd9b640 10 monclient(hunting): get_auth_request con 0x7f99900a5e40 auth_method 0 2024-06-11T19:09:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998dd9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.081+0000 7f998dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998e59c640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998ed9d640 1 --2- 172.21.15.98:0/447521589 >> v2:172.21.15.117:3300/0 conn(0x7f99900a51a0 0x7f9990138da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998e59c640 1 -- 172.21.15.98:0/447521589 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f99900a5e40 msgr2=0x7f9990138860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998e59c640 1 --2- 172.21.15.98:0/447521589 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f99900a5e40 0x7f9990138860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998e59c640 1 -- 172.21.15.98:0/447521589 >> v2:172.21.15.117:3300/0 conn(0x7f99900a51a0 msgr2=0x7f9990138da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998e59c640 1 --2- 172.21.15.98:0/447521589 >> v2:172.21.15.117:3300/0 conn(0x7f99900a51a0 0x7f9990138da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998e59c640 10 monclient: _finish_hunting 0 2024-06-11T19:09:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998e59c640 1 monclient: found mon.noname-d 2024-06-11T19:09:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998e59c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:09:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998e59c640 1 -- 172.21.15.98:0/447521589 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99900a2ab0 con 0x7f99900a4880 2024-06-11T19:09:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998ed9d640 10 monclient: get_auth_request con 0x7f99900a51a0 auth_method 0 2024-06-11T19:09:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998ed9d640 1 --2- 172.21.15.98:0/447521589 >> v2:172.21.15.117:3300/0 conn(0x7f99900a51a0 0x7f9990138da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998e59c640 1 --2- 172.21.15.98:0/447521589 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f99900a4880 0x7f99900a4c60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f99800098a0 tx=0x7f998002b7b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=7233c2017f3e9652 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998d59a640 1 -- 172.21.15.98:0/447521589 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9980038070 con 0x7f99900a4880 2024-06-11T19:09:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.082+0000 7f998d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f998d59a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f998d59a640 10 monclient: dump: 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:38.096 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f998d59a640 10 monclient: _finish_auth 0 2024-06-11T19:09:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f998d59a640 10 monclient: _check_auth_tickets 2024-06-11T19:09:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f998d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f998d59a640 1 -- 172.21.15.98:0/447521589 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9980031980 con 0x7f99900a4880 2024-06-11T19:09:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 5 monclient: authenticate success, global_id 4335 2024-06-11T19:09:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f998d59a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f998d59a640 1 -- 172.21.15.98:0/447521589 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9980031bb0 con 0x7f99900a4880 2024-06-11T19:09:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f998d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 4 set_mon_vals no callback set 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f998d59a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f998d59a640 10 monclient: dump: 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:38.097 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 10 monclient: shutdown 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 1 -- 172.21.15.98:0/447521589 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f99900a4880 msgr2=0x7f99900a4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 1 --2- 172.21.15.98:0/447521589 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f99900a4880 0x7f99900a4c60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f99800098a0 tx=0x7f998002b7b0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 1 -- 172.21.15.98:0/447521589 shutdown_connections 2024-06-11T19:09:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 1 --2- 172.21.15.98:0/447521589 >> v2:172.21.15.117:3300/0 conn(0x7f99900a51a0 0x7f9990138da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 1 --2- 172.21.15.98:0/447521589 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f99900a5e40 0x7f9990138860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 1 --2- 172.21.15.98:0/447521589 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f99900a4880 0x7f99900a4c60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 1 -- 172.21.15.98:0/447521589 >> 172.21.15.98:0/447521589 conn(0x7f99900ab840 msgr2=0x7f99900a7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 1 -- 172.21.15.98:0/447521589 shutdown_connections 2024-06-11T19:09:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.083+0000 7f9994fa8640 1 -- 172.21.15.98:0/447521589 wait complete. 2024-06-11T19:09:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.084+0000 7f9994fa8640 10 monclient: build_initial_monmap 2024-06-11T19:09:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.084+0000 7f9994fa8640 10 monclient: monmap: 2024-06-11T19:09:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.084+0000 7f9994fa8640 1 librados: starting msgr at 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.084+0000 7f9994fa8640 1 librados: starting objecter 2024-06-11T19:09:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.084+0000 7f9994fa8640 1 Processor -- start 2024-06-11T19:09:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.084+0000 7f9994fa8640 1 -- start start 2024-06-11T19:09:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.084+0000 7f9994fa8640 1 librados: setting wanted keys 2024-06-11T19:09:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.084+0000 7f9994fa8640 1 librados: calling monclient init 2024-06-11T19:09:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.084+0000 7f9994fa8640 10 monclient: init 2024-06-11T19:09:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 10 monclient: _add_conns ranks=[0,3,7,1,4,2,8,6,5] 2024-06-11T19:09:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99900a4880 0x7f99901cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 10 monclient(hunting): picked mon.noname-a con 0x7f99900a4880 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99900a51a0 0x7f9990125230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 10 monclient(hunting): picked mon.noname-g con 0x7f99900a51a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f99900a5e40 0x7f9990125770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 10 monclient(hunting): picked mon.noname-i con 0x7f99900a5e40 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f99900a25b0 con 0x7f99900a4880 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f99900a2430 con 0x7f99900a51a0 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f99901392e0 con 0x7f99900a5e40 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f9994fa8640 10 monclient(hunting): authenticate will time out at 787.892417s 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99900a4880 0x7f99901cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99900a4880 0x7f99901cdd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45754/0 (socket says 172.21.15.98:45754) 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 1 -- 172.21.15.98:0/1385406852 learned_addr learned my addr 172.21.15.98:0/1385406852 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 10 monclient(hunting): get_auth_request con 0x7f99900a4880 auth_method 0 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 1 -- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f99900a5e40 msgr2=0x7f9990125770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f99900a5e40 0x7f9990125770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 1 -- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99900a51a0 msgr2=0x7f9990125230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998dd9b640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99900a51a0 0x7f9990125230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99900a51a0 0x7f9990125230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 10 monclient: _finish_hunting 0 2024-06-11T19:09:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 1 monclient: found mon.noname-a 2024-06-11T19:09:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 1 -- 172.21.15.98:0/1385406852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9990128f10 con 0x7f99900a4880 2024-06-11T19:09:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f998e59c640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99900a4880 0x7f99901cdd10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f99800097c0 tx=0x7f99800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f997effd640 10 client.?.objecter ms_handle_connect 0x7f99900a4880 2024-06-11T19:09:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.085+0000 7f997effd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 1 -- 172.21.15.98:0/1385406852 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9980042020 con 0x7f99900a4880 2024-06-11T19:09:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 10 monclient: dump: 2024-06-11T19:09:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:38.106 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 10 monclient: _finish_auth 0 2024-06-11T19:09:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 1 -- 172.21.15.98:0/1385406852 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9980031eb0 con 0x7f99900a4880 2024-06-11T19:09:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f9994fa8640 5 monclient: authenticate success, global_id 4383 2024-06-11T19:09:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 1 -- 172.21.15.98:0/1385406852 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9980038040 con 0x7f99900a4880 2024-06-11T19:09:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f9994fa8640 10 monclient: _renew_subs 2024-06-11T19:09:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f9994fa8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f9994fa8640 1 -- 172.21.15.98:0/1385406852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99900a1620 con 0x7f99900a4880 2024-06-11T19:09:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f998cd99640 4 set_mon_vals no callback set 2024-06-11T19:09:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f9994fa8640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f997effd640 10 monclient: dump: 2024-06-11T19:09:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f9994fa8640 10 monclient: _renew_subs 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f9994fa8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f9994fa8640 1 -- 172.21.15.98:0/1385406852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9990128c30 con 0x7f99900a4880 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.086+0000 7f9994fa8640 1 librados: init done 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f997effd640 1 -- 172.21.15.98:0/1385406852 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f9980045020 con 0x7f99900a4880 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f997effd640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f997003fdb0 0x7f9970042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f997effd640 1 -- 172.21.15.98:0/1385406852 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f99800728d0 con 0x7f99900a4880 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f997effd640 10 client.4383.objecter ms_dispatch 0x7f9990000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f997effd640 3 client.4383.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f997effd640 3 client.4383.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f997effd640 20 client.4383.objecter dump_active .. 0 homeless 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f9994fa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f9994fa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f9994fa8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f9994fa8640 1 -- 172.21.15.98:0/1385406852 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9990139640 con 0x7f99900a4880 2024-06-11T19:09:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f998dd9b640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f997003fdb0 0x7f9970042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f998dd9b640 10 monclient: get_auth_request con 0x7f997003fdb0 auth_method 0 2024-06-11T19:09:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f997effd640 10 client.4383.objecter ms_handle_connect 0x7f997003fdb0 2024-06-11T19:09:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.087+0000 7f998dd9b640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f997003fdb0 0x7f9970042270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f99840098f0 tx=0x7f9984008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.091+0000 7f997effd640 1 -- 172.21.15.98:0/1385406852 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f998004c070 con 0x7f99900a4880 2024-06-11T19:09:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.092+0000 7f997effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.092+0000 7f997effd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.275+0000 7f9994fa8640 1 -- 172.21.15.98:0/1385406852 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9990127a60 con 0x7f997003fdb0 2024-06-11T19:09:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.277+0000 7f997effd640 1 -- 172.21.15.98:0/1385406852 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33771 (secure 0 0 0) 0x7f9990127a60 con 0x7f997003fdb0 2024-06-11T19:09:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:09:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-06-11T19:09:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.278+0000 7f9994fa8640 10 librados: watch_flush enter 2024-06-11T19:09:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.278+0000 7f9994fa8640 10 librados: watch_flush exit 2024-06-11T19:09:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.278+0000 7f9994fa8640 20 client.4383.objecter shutdown clearing up homeless session... 2024-06-11T19:09:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.278+0000 7f9994fa8640 10 client.4383.objecter successfully canceled tick 2024-06-11T19:09:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 -- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f997003fdb0 msgr2=0x7f9970042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f997003fdb0 0x7f9970042270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f99840098f0 tx=0x7f9984008040 comp rx=0 tx=0).stop 2024-06-11T19:09:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 10 monclient: shutdown 2024-06-11T19:09:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 -- 172.21.15.98:0/1385406852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99900a4880 msgr2=0x7f99901cdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99900a4880 0x7f99901cdd10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f99800097c0 tx=0x7f99800046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 -- 172.21.15.98:0/1385406852 shutdown_connections 2024-06-11T19:09:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f997003fdb0 0x7f9970042270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f99900a5e40 0x7f9990125770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99900a51a0 0x7f9990125230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 --2- 172.21.15.98:0/1385406852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99900a4880 0x7f99901cdd10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 -- 172.21.15.98:0/1385406852 >> 172.21.15.98:0/1385406852 conn(0x7f99900ab840 msgr2=0x7f99900a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 -- 172.21.15.98:0/1385406852 shutdown_connections 2024-06-11T19:09:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 -- 172.21.15.98:0/1385406852 wait complete. 2024-06-11T19:09:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.279+0000 7f9994fa8640 1 librados: shutdown 2024-06-11T19:09:38.289 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-06-11T19:09:37.367711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164148,"kb_used_data":2000,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066892,"statfs":{"total":579820584960,"available":579652497408,"internally_reserved":0,"allocated":2048000,"data_stored":1378592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001158"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716208+0000","last_change":"2024-06-11T19:09:33.725085+0000","last_active":"2024-06-11T19:09:34.716208+0000","last_peered":"2024-06-11T19:09:34.716208+0000","last_clean":"2024-06-11T19:09:34.716208+0000","last_became_active":"2024-06-11T19:09:33.724778+0000","last_became_peered":"2024-06-11T19:09:33.724778+0000","last_unstale":"2024-06-11T19:09:34.716208+0000","last_undegraded":"2024-06-11T19:09:34.716208+0000","last_fullsized":"2024-06-11T19:09:34.716208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:57:46.868411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.717260+0000","last_change":"2024-06-11T19:09:33.725197+0000","last_active":"2024-06-11T19:09:34.717260+0000","last_peered":"2024-06-11T19:09:34.717260+0000","last_clean":"2024-06-11T19:09:34.717260+0000","last_became_active":"2024-06-11T19:09:33.724912+0000","last_became_peered":"2024-06-11T19:09:33.724912+0000","last_unstale":"2024-06-11T19:09:34.717260+0000","last_undegraded":"2024-06-11T19:09:34.717260+0000","last_fullsized":"2024-06-11T19:09:34.717260+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:36:12.941264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716160+0000","last_change":"2024-06-11T19:09:33.720915+0000","last_active":"2024-06-11T19:09:34.716160+0000","last_peered":"2024-06-11T19:09:34.716160+0000","last_clean":"2024-06-11T19:09:34.716160+0000","last_became_active":"2024-06-11T19:09:33.720599+0000","last_became_peered":"2024-06-11T19:09:33.720599+0000","last_unstale":"2024-06-11T19:09:34.716160+0000","last_undegraded":"2024-06-11T19:09:34.716160+0000","last_fullsized":"2024-06-11T19:09:34.716160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:28:26.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-06-11T19:09:33.736653+0000","last_change":"2024-06-11T19:09:33.729223+0000","last_active":"2024-06-11T19:09:33.736653+0000","last_peered":"2024-06-11T19:09:33.736653+0000","last_clean":"2024-06-11T19:09:33.736653+0000","last_became_active":"2024-06-11T19:09:33.728897+0000","last_became_peered":"2024-06-11T19:09:33.728897+0000","last_unstale":"2024-06-11T19:09:33.736653+0000","last_undegraded":"2024-06-11T19:09:33.736653+0000","last_fullsized":"2024-06-11T19:09:33.736653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:31:28.135169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.720622+0000","last_change":"2024-06-11T19:09:33.722503+0000","last_active":"2024-06-11T19:09:34.720622+0000","last_peered":"2024-06-11T19:09:34.720622+0000","last_clean":"2024-06-11T19:09:34.720622+0000","last_became_active":"2024-06-11T19:09:33.722180+0000","last_became_peered":"2024-06-11T19:09:33.722180+0000","last_unstale":"2024-06-11T19:09:34.720622+0000","last_undegraded":"2024-06-11T19:09:34.720622+0000","last_fullsized":"2024-06-11T19:09:34.720622+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:28:15.836943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716628+0000","last_change":"2024-06-11T19:09:33.729275+0000","last_active":"2024-06-11T19:09:34.716628+0000","last_peered":"2024-06-11T19:09:34.716628+0000","last_clean":"2024-06-11T19:09:34.716628+0000","last_became_active":"2024-06-11T19:09:33.728948+0000","last_became_peered":"2024-06-11T19:09:33.728948+0000","last_unstale":"2024-06-11T19:09:34.716628+0000","last_undegraded":"2024-06-11T19:09:34.716628+0000","last_fullsized":"2024-06-11T19:09:34.716628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:29:28.908169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.720640+0000","last_change":"2024-06-11T19:09:33.729213+0000","last_active":"2024-06-11T19:09:34.720640+0000","last_peered":"2024-06-11T19:09:34.720640+0000","last_clean":"2024-06-11T19:09:34.720640+0000","last_became_active":"2024-06-11T19:09:33.728930+0000","last_became_peered":"2024-06-11T19:09:33.728930+0000","last_unstale":"2024-06-11T19:09:34.720640+0000","last_undegraded":"2024-06-11T19:09:34.720640+0000","last_fullsized":"2024-06-11T19:09:34.720640+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:26:38.808169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716204+0000","last_change":"2024-06-11T19:09:33.725261+0000","last_active":"2024-06-11T19:09:34.716204+0000","last_peered":"2024-06-11T19:09:34.716204+0000","last_clean":"2024-06-11T19:09:34.716204+0000","last_became_active":"2024-06-11T19:09:33.724870+0000","last_became_peered":"2024-06-11T19:09:33.724870+0000","last_unstale":"2024-06-11T19:09:34.716204+0000","last_undegraded":"2024-06-11T19:09:34.716204+0000","last_fullsized":"2024-06-11T19:09:34.716204+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:32:51.020718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":108,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716046+0000","last_change":"2024-06-11T19:09:26.673425+0000","last_active":"2024-06-11T19:09:34.716046+0000","last_peered":"2024-06-11T19:09:34.716046+0000","last_clean":"2024-06-11T19:09:34.716046+0000","last_became_active":"2024-06-11T19:09:26.673120+0000","last_became_peered":"2024-06-11T19:09:26.673120+0000","last_unstale":"2024-06-11T19:09:34.716046+0000","last_undegraded":"2024-06-11T19:09:34.716046+0000","last_fullsized":"2024-06-11T19:09:34.716046+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_clean_scrub_stamp":"2024-06-11T19:09:25.660869+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:57:43.705570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607559,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":688128,"data_stored":579128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":688128,"data_stored":579128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-11T19:09:38.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-11T19:09:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.389+0000 7f8e08d41640 10 monclient: get_monmap_and_config 2024-06-11T19:09:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.389+0000 7f8e08d41640 10 monclient: build_initial_monmap 2024-06-11T19:09:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.389+0000 7f8e08d41640 10 monclient: monmap: 2024-06-11T19:09:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:38.390266+0000 2024-06-11T19:09:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:38.390266+0000 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:38.391 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.389+0000 7f8e08d41640 1 Processor -- start 2024-06-11T19:09:38.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.389+0000 7f8e08d41640 1 -- start start 2024-06-11T19:09:38.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.389+0000 7f8e08d41640 10 monclient: init 2024-06-11T19:09:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 10 monclient: _add_conns ranks=[7,3,8,1,5,9,2,4,6,10,0] 2024-06-11T19:09:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8e0409d330 0x7f8e0409d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 10 monclient(hunting): picked mon.noname-i con 0x7f8e0409d330 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8e0409dc50 0x7f8e04136770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 10 monclient(hunting): picked mon.noname-g con 0x7f8e0409dc50 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8e040a2850 0x7f8e04136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 10 monclient(hunting): picked mon.noname-e con 0x7f8e040a2850 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f8e0409bb20 con 0x7f8e0409dc50 2024-06-11T19:09:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8e0409b560 con 0x7f8e0409d330 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8e0409b840 con 0x7f8e040a2850 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e08d41640 10 monclient(hunting): authenticate will time out at 788.197562s 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8e040a2850 0x7f8e04136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8e040a2850 0x7f8e04136cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54550/0 (socket says 172.21.15.98:54550) 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 1 -- 172.21.15.98:0/3065015433 learned_addr learned my addr 172.21.15.98:0/3065015433 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 10 monclient(hunting): get_auth_request con 0x7f8e040a2850 auth_method 0 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:38.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e01d9b640 1 --2- 172.21.15.98:0/3065015433 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8e0409dc50 0x7f8e04136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 10 monclient(hunting): handle_auth_done global_id 4382 payload 274 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 1 -- 172.21.15.98:0/3065015433 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8e0409d330 msgr2=0x7f8e0409d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 1 --2- 172.21.15.98:0/3065015433 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8e0409d330 0x7f8e0409d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 1 -- 172.21.15.98:0/3065015433 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8e0409dc50 msgr2=0x7f8e04136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 1 --2- 172.21.15.98:0/3065015433 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8e0409dc50 0x7f8e04136770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 1 monclient: found mon.noname-e 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.390+0000 7f8e02d9d640 1 -- 172.21.15.98:0/3065015433 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e040769a0 con 0x7f8e040a2850 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e02d9d640 1 --2- 172.21.15.98:0/3065015433 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8e040a2850 0x7f8e04136cb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009e80 tx=0x7f8dfc02bd70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=1bfe058d9445a36a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 1 -- 172.21.15.98:0/3065015433 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8dfc02cb80 con 0x7f8e040a2850 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 10 monclient: dump: 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 10 monclient: _finish_auth 0 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 10 monclient: _check_auth_tickets 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 1 -- 172.21.15.98:0/3065015433 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dfc02cd40 con 0x7f8e040a2850 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e08d41640 5 monclient: authenticate success, global_id 4382 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e08d41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 1 -- 172.21.15.98:0/3065015433 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8dfc034860 con 0x7f8e040a2850 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e0159a640 10 monclient: dump: 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:38.404 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e08d41640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.391+0000 7f8e08d41640 4 set_mon_vals no callback set 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.392+0000 7f8e08d41640 10 monclient: shutdown 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.392+0000 7f8e08d41640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.392+0000 7f8e08d41640 1 -- 172.21.15.98:0/3065015433 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8e040a2850 msgr2=0x7f8e04136cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.392+0000 7f8e08d41640 1 --2- 172.21.15.98:0/3065015433 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8e040a2850 0x7f8e04136cb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009e80 tx=0x7f8dfc02bd70 comp rx=0 tx=0).stop 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.392+0000 7f8e08d41640 1 -- 172.21.15.98:0/3065015433 shutdown_connections 2024-06-11T19:09:38.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.392+0000 7f8e08d41640 1 --2- 172.21.15.98:0/3065015433 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8e040a2850 0x7f8e04136cb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.392+0000 7f8e08d41640 1 --2- 172.21.15.98:0/3065015433 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8e0409dc50 0x7f8e04136770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.392+0000 7f8e08d41640 1 --2- 172.21.15.98:0/3065015433 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8e0409d330 0x7f8e0409d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.392+0000 7f8e08d41640 1 -- 172.21.15.98:0/3065015433 >> 172.21.15.98:0/3065015433 conn(0x7f8e040a7850 msgr2=0x7f8e040a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.392+0000 7f8e08d41640 1 -- 172.21.15.98:0/3065015433 shutdown_connections 2024-06-11T19:09:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.392+0000 7f8e08d41640 1 -- 172.21.15.98:0/3065015433 wait complete. 2024-06-11T19:09:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.393+0000 7f8e08d41640 10 monclient: build_initial_monmap 2024-06-11T19:09:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.393+0000 7f8e08d41640 10 monclient: monmap: 2024-06-11T19:09:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.393+0000 7f8e08d41640 1 librados: starting msgr at 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.393+0000 7f8e08d41640 1 librados: starting objecter 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.393+0000 7f8e08d41640 1 Processor -- start 2024-06-11T19:09:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.393+0000 7f8e08d41640 1 -- start start 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.393+0000 7f8e08d41640 1 librados: setting wanted keys 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.393+0000 7f8e08d41640 1 librados: calling monclient init 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.393+0000 7f8e08d41640 10 monclient: init 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 10 monclient: _add_conns ranks=[2,5,0,4,6,3,1,8,7] 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e0409d330 0x7f8e0409a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 10 monclient(hunting): picked mon.noname-b con 0x7f8e0409d330 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8e040a2850 0x7f8e0409a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 10 monclient(hunting): picked mon.noname-h con 0x7f8e040a2850 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e04134b30 0x7f8e04134f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 10 monclient(hunting): picked mon.noname-a con 0x7f8e04134b30 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e041371f0 con 0x7f8e04134b30 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8e0409e270 con 0x7f8e0409d330 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f8e04137370 con 0x7f8e040a2850 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 10 monclient(hunting): authenticate will time out at 788.201246s 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e0409d330 0x7f8e0409a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e02d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e04134b30 0x7f8e04134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e0409d330 0x7f8e0409a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48012/0 (socket says 172.21.15.98:48012) 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 1 -- 172.21.15.98:0/4023450390 learned_addr learned my addr 172.21.15.98:0/4023450390 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e02d9d640 10 monclient(hunting): get_auth_request con 0x7f8e04134b30 auth_method 0 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e02d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 10 monclient(hunting): get_auth_request con 0x7f8e0409d330 auth_method 0 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e01d9b640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8e040a2850 0x7f8e0409a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 10 monclient(hunting): handle_auth_done global_id 4376 payload 824 2024-06-11T19:09:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 1 -- 172.21.15.98:0/4023450390 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8e040a2850 msgr2=0x7f8e0409a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8e040a2850 0x7f8e0409a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 1 -- 172.21.15.98:0/4023450390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e04134b30 msgr2=0x7f8e04134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e04134b30 0x7f8e04134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 10 monclient: _finish_hunting 0 2024-06-11T19:09:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 1 monclient: found mon.noname-b 2024-06-11T19:09:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 1 -- 172.21.15.98:0/4023450390 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e04073fa0 con 0x7f8e0409d330 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e01d9b640 10 monclient: get_auth_request con 0x7f8e040a2850 auth_method 0 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e02d9d640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e04134b30 0x7f8e04134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e01d9b640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8e040a2850 0x7f8e0409a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e0259c640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e0409d330 0x7f8e0409a310 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8df400b4a0 tx=0x7f8df400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8df2ffd640 10 client.?.objecter ms_handle_connect 0x7f8e0409d330 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8df2ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8df2ffd640 1 -- 172.21.15.98:0/4023450390 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8df4014070 con 0x7f8e0409d330 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8df2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8df2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8df2ffd640 10 monclient: dump: 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.412 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8df2ffd640 10 monclient: _finish_auth 0 2024-06-11T19:09:38.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8df2ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8df2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8df2ffd640 1 -- 172.21.15.98:0/4023450390 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8df4010a80 con 0x7f8e0409d330 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.394+0000 7f8e08d41640 5 monclient: authenticate success, global_id 4376 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8e08d41640 10 monclient: _renew_subs 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8e08d41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8e08d41640 1 -- 172.21.15.98:0/4023450390 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e040a4900 con 0x7f8e0409d330 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8df2ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8df2ffd640 1 -- 172.21.15.98:0/4023450390 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8df4010c30 con 0x7f8e0409d330 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8df2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8df2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8e00d99640 4 set_mon_vals no callback set 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8e08d41640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:38.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8df2ffd640 10 monclient: dump: 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:38.415 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8e08d41640 10 monclient: _renew_subs 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8e08d41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8e08d41640 1 -- 172.21.15.98:0/4023450390 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e040a02e0 con 0x7f8e0409d330 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.395+0000 7f8e08d41640 1 librados: init done 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8e08d41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8e08d41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8e08d41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8e08d41640 1 -- 172.21.15.98:0/4023450390 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e0409d710 con 0x7f8e0409d330 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8df2ffd640 1 -- 172.21.15.98:0/4023450390 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8df4017070 con 0x7f8e0409d330 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8df2ffd640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8de403fdb0 0x7f8de4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8df2ffd640 1 -- 172.21.15.98:0/4023450390 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f8df400d040 con 0x7f8e0409d330 2024-06-11T19:09:38.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8df2ffd640 10 client.4376.objecter ms_dispatch 0x7f8e04000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:38.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8df2ffd640 3 client.4376.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:38.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8df2ffd640 3 client.4376.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:38.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8df2ffd640 20 client.4376.objecter dump_active .. 0 homeless 2024-06-11T19:09:38.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8e01d9b640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8de403fdb0 0x7f8de4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.396+0000 7f8e01d9b640 10 monclient: get_auth_request con 0x7f8de403fdb0 auth_method 0 2024-06-11T19:09:38.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.397+0000 7f8e01d9b640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8de403fdb0 0x7f8de4042270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e04070620 tx=0x7f8df8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:38.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.397+0000 7f8df2ffd640 10 client.4376.objecter ms_handle_connect 0x7f8de403fdb0 2024-06-11T19:09:38.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.400+0000 7f8df2ffd640 1 -- 172.21.15.98:0/4023450390 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8e040a4900 con 0x7f8e0409d330 2024-06-11T19:09:38.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.400+0000 7f8df2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:38.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.400+0000 7f8df2ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:38.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.584+0000 7f8e08d41640 1 -- 172.21.15.98:0/4023450390 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8e041ce5d0 con 0x7f8de403fdb0 2024-06-11T19:09:38.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.585+0000 7f8df2ffd640 1 -- 172.21.15.98:0/4023450390 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33771 (secure 0 0 0) 0x7f8e041ce5d0 con 0x7f8de403fdb0 2024-06-11T19:09:38.585 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:09:38.586 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-06-11T19:09:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 10 librados: watch_flush enter 2024-06-11T19:09:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 10 librados: watch_flush exit 2024-06-11T19:09:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 20 client.4376.objecter shutdown clearing up homeless session... 2024-06-11T19:09:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 10 client.4376.objecter successfully canceled tick 2024-06-11T19:09:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 -- 172.21.15.98:0/4023450390 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8de403fdb0 msgr2=0x7f8de4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8de403fdb0 0x7f8de4042270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e04070620 tx=0x7f8df8008040 comp rx=0 tx=0).stop 2024-06-11T19:09:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 10 monclient: shutdown 2024-06-11T19:09:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 -- 172.21.15.98:0/4023450390 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e0409d330 msgr2=0x7f8e0409a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e0409d330 0x7f8e0409a310 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8df400b4a0 tx=0x7f8df400b960 comp rx=0 tx=0).stop 2024-06-11T19:09:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 -- 172.21.15.98:0/4023450390 shutdown_connections 2024-06-11T19:09:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8de403fdb0 0x7f8de4042270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e04134b30 0x7f8e04134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8e040a2850 0x7f8e0409a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 --2- 172.21.15.98:0/4023450390 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8e0409d330 0x7f8e0409a310 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 -- 172.21.15.98:0/4023450390 >> 172.21.15.98:0/4023450390 conn(0x7f8e040a7850 msgr2=0x7f8e040a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:38.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 -- 172.21.15.98:0/4023450390 shutdown_connections 2024-06-11T19:09:38.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 -- 172.21.15.98:0/4023450390 wait complete. 2024-06-11T19:09:38.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.587+0000 7f8e08d41640 1 librados: shutdown 2024-06-11T19:09:38.597 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-06-11T19:09:37.367711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164148,"kb_used_data":2000,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066892,"statfs":{"total":579820584960,"available":579652497408,"internally_reserved":0,"allocated":2048000,"data_stored":1378592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001158"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716208+0000","last_change":"2024-06-11T19:09:33.725085+0000","last_active":"2024-06-11T19:09:34.716208+0000","last_peered":"2024-06-11T19:09:34.716208+0000","last_clean":"2024-06-11T19:09:34.716208+0000","last_became_active":"2024-06-11T19:09:33.724778+0000","last_became_peered":"2024-06-11T19:09:33.724778+0000","last_unstale":"2024-06-11T19:09:34.716208+0000","last_undegraded":"2024-06-11T19:09:34.716208+0000","last_fullsized":"2024-06-11T19:09:34.716208+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:57:46.868411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.717260+0000","last_change":"2024-06-11T19:09:33.725197+0000","last_active":"2024-06-11T19:09:34.717260+0000","last_peered":"2024-06-11T19:09:34.717260+0000","last_clean":"2024-06-11T19:09:34.717260+0000","last_became_active":"2024-06-11T19:09:33.724912+0000","last_became_peered":"2024-06-11T19:09:33.724912+0000","last_unstale":"2024-06-11T19:09:34.717260+0000","last_undegraded":"2024-06-11T19:09:34.717260+0000","last_fullsized":"2024-06-11T19:09:34.717260+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:36:12.941264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716160+0000","last_change":"2024-06-11T19:09:33.720915+0000","last_active":"2024-06-11T19:09:34.716160+0000","last_peered":"2024-06-11T19:09:34.716160+0000","last_clean":"2024-06-11T19:09:34.716160+0000","last_became_active":"2024-06-11T19:09:33.720599+0000","last_became_peered":"2024-06-11T19:09:33.720599+0000","last_unstale":"2024-06-11T19:09:34.716160+0000","last_undegraded":"2024-06-11T19:09:34.716160+0000","last_fullsized":"2024-06-11T19:09:34.716160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:28:26.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-06-11T19:09:33.736653+0000","last_change":"2024-06-11T19:09:33.729223+0000","last_active":"2024-06-11T19:09:33.736653+0000","last_peered":"2024-06-11T19:09:33.736653+0000","last_clean":"2024-06-11T19:09:33.736653+0000","last_became_active":"2024-06-11T19:09:33.728897+0000","last_became_peered":"2024-06-11T19:09:33.728897+0000","last_unstale":"2024-06-11T19:09:33.736653+0000","last_undegraded":"2024-06-11T19:09:33.736653+0000","last_fullsized":"2024-06-11T19:09:33.736653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:31:28.135169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.720622+0000","last_change":"2024-06-11T19:09:33.722503+0000","last_active":"2024-06-11T19:09:34.720622+0000","last_peered":"2024-06-11T19:09:34.720622+0000","last_clean":"2024-06-11T19:09:34.720622+0000","last_became_active":"2024-06-11T19:09:33.722180+0000","last_became_peered":"2024-06-11T19:09:33.722180+0000","last_unstale":"2024-06-11T19:09:34.720622+0000","last_undegraded":"2024-06-11T19:09:34.720622+0000","last_fullsized":"2024-06-11T19:09:34.720622+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:28:15.836943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716628+0000","last_change":"2024-06-11T19:09:33.729275+0000","last_active":"2024-06-11T19:09:34.716628+0000","last_peered":"2024-06-11T19:09:34.716628+0000","last_clean":"2024-06-11T19:09:34.716628+0000","last_became_active":"2024-06-11T19:09:33.728948+0000","last_became_peered":"2024-06-11T19:09:33.728948+0000","last_unstale":"2024-06-11T19:09:34.716628+0000","last_undegraded":"2024-06-11T19:09:34.716628+0000","last_fullsized":"2024-06-11T19:09:34.716628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:29:28.908169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.720640+0000","last_change":"2024-06-11T19:09:33.729213+0000","last_active":"2024-06-11T19:09:34.720640+0000","last_peered":"2024-06-11T19:09:34.720640+0000","last_clean":"2024-06-11T19:09:34.720640+0000","last_became_active":"2024-06-11T19:09:33.728930+0000","last_became_peered":"2024-06-11T19:09:33.728930+0000","last_unstale":"2024-06-11T19:09:34.720640+0000","last_undegraded":"2024-06-11T19:09:34.720640+0000","last_fullsized":"2024-06-11T19:09:34.720640+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:26:38.808169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716204+0000","last_change":"2024-06-11T19:09:33.725261+0000","last_active":"2024-06-11T19:09:34.716204+0000","last_peered":"2024-06-11T19:09:34.716204+0000","last_clean":"2024-06-11T19:09:34.716204+0000","last_became_active":"2024-06-11T19:09:33.724870+0000","last_became_peered":"2024-06-11T19:09:33.724870+0000","last_unstale":"2024-06-11T19:09:34.716204+0000","last_undegraded":"2024-06-11T19:09:34.716204+0000","last_fullsized":"2024-06-11T19:09:34.716204+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:32:51.020718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":108,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-06-11T19:09:34.716046+0000","last_change":"2024-06-11T19:09:26.673425+0000","last_active":"2024-06-11T19:09:34.716046+0000","last_peered":"2024-06-11T19:09:34.716046+0000","last_clean":"2024-06-11T19:09:34.716046+0000","last_became_active":"2024-06-11T19:09:26.673120+0000","last_became_peered":"2024-06-11T19:09:26.673120+0000","last_unstale":"2024-06-11T19:09:34.716046+0000","last_undegraded":"2024-06-11T19:09:34.716046+0000","last_fullsized":"2024-06-11T19:09:34.716046+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_clean_scrub_stamp":"2024-06-11T19:09:25.660869+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:57:43.705570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607559,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":688128,"data_stored":579128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":688128,"data_stored":579128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":55084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-11T19:09:38.598 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-06-11T19:09:38.599 DEBUG:tasks.mon_thrash:Yielding 2024-06-11T19:09:38.599 INFO:teuthology.run_tasks:Running task workunit... 2024-06-11T19:09:38.613 INFO:tasks.workunit:Pulling workunits from ref 37bbe1d8d788b7c9a71280e7d510c295a5853f0b 2024-06-11T19:09:38.614 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-06-11T19:09:38.614 DEBUG:teuthology.orchestra.run.smithi117:> stat -- /home/ubuntu/cephtest/mnt.0 2024-06-11T19:09:38.614 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-06-11T19:09:38.615 INFO:tasks.mon_thrash.mon_thrasher:seed: 1718132978, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: True, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-06-11T19:09:38.615 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:09:38.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:09:38.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T19:09:38.634 INFO:teuthology.orchestra.run.smithi117.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-06-11T19:09:38.634 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-06-11T19:09:38.692 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-06-11T19:09:38.693 DEBUG:teuthology.orchestra.run.smithi117:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-06-11T19:09:38.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.712+0000 7f6d0c15f640 10 monclient: get_monmap_and_config 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.712+0000 7f6d0c15f640 10 monclient: build_initial_monmap 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.712+0000 7f6d0c15f640 10 monclient: monmap: 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:38.713344+0000 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:38.713344+0000 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:38.713 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:38.713 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:38.713 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:38.713 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:38.713 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:38.713 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:38.713 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.712+0000 7f6d0c15f640 1 Processor -- start 2024-06-11T19:09:38.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 1 -- start start 2024-06-11T19:09:38.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 10 monclient: init 2024-06-11T19:09:38.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:38.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 10 monclient: _add_conns ranks=[8,9,0,5,4,7,2,1,10,3,6] 2024-06-11T19:09:38.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6d040a0cd0 0x7f6d040a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 10 monclient(hunting): picked mon.noname-e con 0x7f6d040a0cd0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:38.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d04138c80 0x7f6d04139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 10 monclient(hunting): picked mon.noname-a con 0x7f6d04138c80 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d040a2250 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0409efd0 con 0x7f6d040a2250 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6d040739b0 con 0x7f6d04138c80 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6d0409ecf0 con 0x7f6d040a0cd0 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0c15f640 10 monclient(hunting): authenticate will time out at 788.520577s 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d09efb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6d040a0cd0 0x7f6d040a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d096fa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d04138c80 0x7f6d04139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0a6fc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6d040a2250 0x7f6d040a26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53452/0 (socket says 172.21.15.98:53452) 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d0a6fc640 1 -- 172.21.15.98:0/2948192238 learned_addr learned my addr 172.21.15.98:0/2948192238 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:09:38.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d096fa640 10 monclient(hunting): get_auth_request con 0x7f6d04138c80 auth_method 0 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d096fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d096fa640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d096fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d09efb640 10 monclient(hunting): get_auth_request con 0x7f6d040a0cd0 auth_method 0 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d09efb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d09efb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d09efb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d096fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d096fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d096fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d09efb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d09efb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d09efb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:38.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.713+0000 7f6d096fa640 10 monclient(hunting): handle_auth_done global_id 4401 payload 274 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d096fa640 1 -- 172.21.15.98:0/2948192238 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6d040a0cd0 msgr2=0x7f6d040a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d096fa640 1 --2- 172.21.15.98:0/2948192238 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6d040a0cd0 0x7f6d040a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d096fa640 1 -- 172.21.15.98:0/2948192238 >> v1:172.21.15.98:6789/0 conn(0x7f6d040a2250 legacy=0x7f6d040a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d096fa640 10 monclient: _finish_hunting 0 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d096fa640 1 monclient: found mon.noname-a 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d096fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d096fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d08ef9640 1 -- 172.21.15.98:0/2948192238 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3864482532 0 0) 0x7f6cfc001490 con 0x7f6d040a2250 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d096fa640 1 -- 172.21.15.98:0/2948192238 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0409f2b0 con 0x7f6d04138c80 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d08ef9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d09efb640 1 --2- 172.21.15.98:0/2948192238 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6d040a0cd0 0x7f6d040a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d096fa640 1 --2- 172.21.15.98:0/2948192238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d04138c80 0x7f6d04139060 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009790 tx=0x7f6cf402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab9a56c5953132d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d08ef9640 1 -- 172.21.15.98:0/2948192238 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6cf4039070 con 0x7f6d04138c80 2024-06-11T19:09:38.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d08ef9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d08ef9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.714+0000 7f6d08ef9640 10 monclient: dump: 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:38.725 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d08ef9640 10 monclient: _finish_auth 0 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d08ef9640 10 monclient: _check_auth_tickets 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d08ef9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d08ef9640 1 -- 172.21.15.98:0/2948192238 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf4031960 con 0x7f6d04138c80 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 5 monclient: authenticate success, global_id 4401 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d08ef9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d08ef9640 1 -- 172.21.15.98:0/2948192238 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6cf4032380 con 0x7f6d04138c80 2024-06-11T19:09:38.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d08ef9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 4 set_mon_vals no callback set 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d08ef9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d08ef9640 10 monclient: dump: 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:38.727 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:38.730 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:38.730 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:38.730 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:38.730 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:38.730 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 10 monclient: shutdown 2024-06-11T19:09:38.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:38.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 1 -- 172.21.15.98:0/2948192238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d04138c80 msgr2=0x7f6d04139060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 1 --2- 172.21.15.98:0/2948192238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d04138c80 0x7f6d04139060 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009790 tx=0x7f6cf402b790 comp rx=0 tx=0).stop 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 1 -- 172.21.15.98:0/2948192238 shutdown_connections 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 1 --2- 172.21.15.98:0/2948192238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d04138c80 0x7f6d04139060 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 1 --2- 172.21.15.98:0/2948192238 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6d040a0cd0 0x7f6d040a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 1 -- 172.21.15.98:0/2948192238 >> 172.21.15.98:0/2948192238 conn(0x7f6d040a7850 msgr2=0x7f6d040a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 1 -- 172.21.15.98:0/2948192238 shutdown_connections 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.715+0000 7f6d0c15f640 1 -- 172.21.15.98:0/2948192238 wait complete. 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient: build_initial_monmap 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient: monmap: 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:38.731 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 1 librados: starting msgr at 2024-06-11T19:09:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 1 librados: starting objecter 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 1 Processor -- start 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 1 -- start start 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 1 librados: setting wanted keys 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 1 librados: calling monclient init 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient: init 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient: _add_conns ranks=[3,0,6,5,8,4,7,1,2] 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d040a0cd0 0x7f6d041cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient(hunting): picked mon.noname-g con 0x7f6d040a0cd0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d040a2250 0x7f6d041358f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient(hunting): picked mon.noname-a con 0x7f6d040a2250 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6d04138c80 0x7f6d04135e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient(hunting): picked mon.noname-d con 0x7f6d04138c80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d040a1770 con 0x7f6d040a2250 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f6d040a15f0 con 0x7f6d040a0cd0 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f6d040a2df0 con 0x7f6d04138c80 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.716+0000 7f6d0c15f640 10 monclient(hunting): authenticate will time out at 788.523993s 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d040a2250 0x7f6d041358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d040a2250 0x7f6d041358f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45788/0 (socket says 172.21.15.98:45788) 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 1 -- 172.21.15.98:0/1331590436 learned_addr learned my addr 172.21.15.98:0/1331590436 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 10 monclient(hunting): get_auth_request con 0x7f6d040a2250 auth_method 0 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:38.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0a6fc640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6d04138c80 0x7f6d04135e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0a6fc640 10 monclient(hunting): get_auth_request con 0x7f6d04138c80 auth_method 0 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0a6fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0a6fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0a6fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0a6fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0a6fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0a6fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:38.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d09efb640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d040a0cd0 0x7f6d041cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 10 monclient(hunting): handle_auth_done global_id 4419 payload 824 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 1 -- 172.21.15.98:0/1331590436 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6d04138c80 msgr2=0x7f6d04135e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6d04138c80 0x7f6d04135e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 1 -- 172.21.15.98:0/1331590436 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d040a0cd0 msgr2=0x7f6d041cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d040a0cd0 0x7f6d041cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 10 monclient: _finish_hunting 0 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 1 monclient: found mon.noname-a 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 1 -- 172.21.15.98:0/1331590436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d041320c0 con 0x7f6d040a2250 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d09efb640 10 monclient: get_auth_request con 0x7f6d040a0cd0 auth_method 0 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d09efb640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d040a0cd0 0x7f6d041cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d096fa640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d040a2250 0x7f6d041358f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40096e0 tx=0x7f6cf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:38.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 10 client.?.objecter ms_handle_connect 0x7f6d040a2250 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 1 -- 172.21.15.98:0/1331590436 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6cf4039040 con 0x7f6d040a2250 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 10 monclient: dump: 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:38.738 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:38.738 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:38.738 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:38.738 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:38.738 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:38.738 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:38.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:38.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:38.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 1 -- 172.21.15.98:0/1331590436 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf40044b0 con 0x7f6d040a2250 2024-06-11T19:09:38.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0c15f640 5 monclient: authenticate success, global_id 4419 2024-06-11T19:09:38.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 1 -- 172.21.15.98:0/1331590436 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6cf4005e00 con 0x7f6d040a2250 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0c15f640 10 monclient: _renew_subs 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0c15f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0c15f640 1 -- 172.21.15.98:0/1331590436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d04136bd0 con 0x7f6d040a2250 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfbfff640 4 set_mon_vals no callback set 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6cfa7fc640 10 monclient: dump: 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0c15f640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0c15f640 10 monclient: _renew_subs 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0c15f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0c15f640 1 -- 172.21.15.98:0/1331590436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d041ce290 con 0x7f6d040a2250 2024-06-11T19:09:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.717+0000 7f6d0c15f640 1 librados: init done 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.718+0000 7f6cfa7fc640 1 -- 172.21.15.98:0/1331590436 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f6cf4005990 con 0x7f6d040a2250 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.718+0000 7f6d0c15f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.718+0000 7f6d0c15f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.718+0000 7f6d0c15f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.718+0000 7f6d0c15f640 1 -- 172.21.15.98:0/1331590436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d041bf960 con 0x7f6d040a2250 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.718+0000 7f6cfa7fc640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6ce803fdb0 0x7f6ce8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.718+0000 7f6cfa7fc640 1 -- 172.21.15.98:0/1331590436 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f6cf4072fc0 con 0x7f6d040a2250 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.718+0000 7f6cfa7fc640 10 client.4419.objecter ms_dispatch 0x7f6d04000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.718+0000 7f6cfa7fc640 3 client.4419.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.718+0000 7f6cfa7fc640 3 client.4419.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.718+0000 7f6cfa7fc640 20 client.4419.objecter dump_active .. 0 homeless 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.719+0000 7f6d09efb640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6ce803fdb0 0x7f6ce8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.719+0000 7f6d09efb640 10 monclient: get_auth_request con 0x7f6ce803fdb0 auth_method 0 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.719+0000 7f6cfa7fc640 10 client.4419.objecter ms_handle_connect 0x7f6ce803fdb0 2024-06-11T19:09:38.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.719+0000 7f6d09efb640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6ce803fdb0 0x7f6ce8042270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009ac0 tx=0x7f6d00008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:38.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.723+0000 7f6cfa7fc640 1 -- 172.21.15.98:0/1331590436 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6d041bf960 con 0x7f6d040a2250 2024-06-11T19:09:38.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.723+0000 7f6cfa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:38.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.723+0000 7f6cfa7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:38.759 INFO:tasks.workunit:timeout=3h 2024-06-11T19:09:38.759 INFO:tasks.workunit:cleanup=True 2024-06-11T19:09:38.759 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 37bbe1d8d788b7c9a71280e7d510c295a5853f0b 2024-06-11T19:09:38.817 INFO:tasks.workunit.client.0.smithi117.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-06-11T19:09:38.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.975+0000 7f6d0c15f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:09:38.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.975+0000 7f6d0c15f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:09:38.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.975+0000 7f6d0c15f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:38.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.975+0000 7f6d0c15f640 1 -- 172.21.15.98:0/1331590436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6d040a10b0 con 0x7f6d040a2250 2024-06-11T19:09:38.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.976+0000 7f6cfa7fc640 1 -- 172.21.15.98:0/1331590436 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f6cf403f420 con 0x7f6d040a2250 2024-06-11T19:09:38.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.976+0000 7f6cfa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:09:38.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.976+0000 7f6cfa7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:38.976 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":8,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:09:38.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.977+0000 7f6d0c15f640 10 librados: watch_flush enter 2024-06-11T19:09:38.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.977+0000 7f6d0c15f640 10 librados: watch_flush exit 2024-06-11T19:09:38.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.977+0000 7f6d0c15f640 20 client.4419.objecter shutdown clearing up homeless session... 2024-06-11T19:09:38.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.977+0000 7f6d0c15f640 10 client.4419.objecter successfully canceled tick 2024-06-11T19:09:38.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 -- 172.21.15.98:0/1331590436 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6ce803fdb0 msgr2=0x7f6ce8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6ce803fdb0 0x7f6ce8042270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009ac0 tx=0x7f6d00008040 comp rx=0 tx=0).stop 2024-06-11T19:09:38.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 10 monclient: shutdown 2024-06-11T19:09:38.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:38.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 -- 172.21.15.98:0/1331590436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d040a2250 msgr2=0x7f6d041358f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d040a2250 0x7f6d041358f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40096e0 tx=0x7f6cf40046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 -- 172.21.15.98:0/1331590436 shutdown_connections 2024-06-11T19:09:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6ce803fdb0 0x7f6ce8042270 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6d04138c80 0x7f6d04135e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d040a2250 0x7f6d041358f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 --2- 172.21.15.98:0/1331590436 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d040a0cd0 0x7f6d041cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 -- 172.21.15.98:0/1331590436 >> 172.21.15.98:0/1331590436 conn(0x7f6d040a7850 msgr2=0x7f6d041be700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 -- 172.21.15.98:0/1331590436 shutdown_connections 2024-06-11T19:09:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 -- 172.21.15.98:0/1331590436 wait complete. 2024-06-11T19:09:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:38.978+0000 7f6d0c15f640 1 librados: shutdown 2024-06-11T19:09:38.988 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:09:38.988 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-06-11T19:09:38.988 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:09:38.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:09:39.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.089+0000 7ff5299a1640 10 monclient: get_monmap_and_config 2024-06-11T19:09:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.089+0000 7ff5299a1640 10 monclient: build_initial_monmap 2024-06-11T19:09:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.089+0000 7ff5299a1640 10 monclient: monmap: 2024-06-11T19:09:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:39.090163+0000 2024-06-11T19:09:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:39.090163+0000 2024-06-11T19:09:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:39.089 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:39.090 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.089+0000 7ff5299a1640 1 Processor -- start 2024-06-11T19:09:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.089+0000 7ff5299a1640 1 -- start start 2024-06-11T19:09:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.089+0000 7ff5299a1640 10 monclient: init 2024-06-11T19:09:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 10 monclient: _add_conns ranks=[9,5,3,1,0,6,2,10,7,4,8] 2024-06-11T19:09:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff5240a0c70 0x7ff5240a28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 10 monclient(hunting): picked mon.noname-a con 0x7ff5240a0c70 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff52406c960 0x7ff5240a4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 10 monclient(hunting): picked mon.noname-h con 0x7ff52406c960 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff5240a2e30 0x7ff5240a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 10 monclient(hunting): picked mon.noname-g con 0x7ff5240a2e30 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff52406c790 con 0x7ff5240a0c70 2024-06-11T19:09:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ff5240a1050 con 0x7ff5240a2e30 2024-06-11T19:09:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ff5240a0750 con 0x7ff52406c960 2024-06-11T19:09:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5299a1640 10 monclient(hunting): authenticate will time out at 788.897475s 2024-06-11T19:09:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff5240a0c70 0x7ff5240a28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff5240a0c70 0x7ff5240a28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45800/0 (socket says 172.21.15.98:45800) 2024-06-11T19:09:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 1 -- 172.21.15.98:0/3133266697 learned_addr learned my addr 172.21.15.98:0/3133266697 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 10 monclient(hunting): get_auth_request con 0x7ff5240a0c70 auth_method 0 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff523fff640 1 --2- 172.21.15.98:0/3133266697 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff52406c960 0x7ff5240a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5291a0640 1 --2- 172.21.15.98:0/3133266697 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff5240a2e30 0x7ff5240a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff523fff640 10 monclient(hunting): get_auth_request con 0x7ff52406c960 auth_method 0 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff523fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff523fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 10 monclient(hunting): handle_auth_done global_id 4428 payload 274 2024-06-11T19:09:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 1 -- 172.21.15.98:0/3133266697 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff52406c960 msgr2=0x7ff5240a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 1 --2- 172.21.15.98:0/3133266697 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff52406c960 0x7ff5240a4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 1 -- 172.21.15.98:0/3133266697 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff5240a2e30 msgr2=0x7ff5240a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 1 --2- 172.21.15.98:0/3133266697 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff5240a2e30 0x7ff5240a4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 10 monclient: _finish_hunting 0 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 1 monclient: found mon.noname-a 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff52899f640 1 -- 172.21.15.98:0/3133266697 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5240a0470 con 0x7ff5240a0c70 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5291a0640 10 monclient: get_auth_request con 0x7ff5240a2e30 auth_method 0 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.090+0000 7ff5291a0640 1 --2- 172.21.15.98:0/3133266697 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff5240a2e30 0x7ff5240a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff52899f640 1 --2- 172.21.15.98:0/3133266697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5240a0c70 0x7ff5240a28f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff5140098a0 tx=0x7ff51402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec857bd82cf9192f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 1 -- 172.21.15.98:0/3133266697 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff514031a10 con 0x7ff5240a0c70 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:39.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 10 monclient: dump: 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 1 -- 172.21.15.98:0/3133266697 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff514037070 con 0x7ff5240a0c70 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5299a1640 5 monclient: authenticate success, global_id 4428 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5299a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5299a1640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5299a1640 4 set_mon_vals no callback set 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 1 -- 172.21.15.98:0/3133266697 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff514031c10 con 0x7ff5240a0c70 2024-06-11T19:09:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.091+0000 7ff5237fe640 10 monclient: dump: 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 10 monclient: shutdown 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 1 -- 172.21.15.98:0/3133266697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5240a0c70 msgr2=0x7ff5240a28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 1 --2- 172.21.15.98:0/3133266697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5240a0c70 0x7ff5240a28f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff5140098a0 tx=0x7ff51402b840 comp rx=0 tx=0).stop 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 1 -- 172.21.15.98:0/3133266697 shutdown_connections 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 1 --2- 172.21.15.98:0/3133266697 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff52406c960 0x7ff5240a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 1 --2- 172.21.15.98:0/3133266697 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff5240a2e30 0x7ff5240a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 1 --2- 172.21.15.98:0/3133266697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5240a0c70 0x7ff5240a28f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 1 -- 172.21.15.98:0/3133266697 >> 172.21.15.98:0/3133266697 conn(0x7ff5240a9440 msgr2=0x7ff5240a5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 1 -- 172.21.15.98:0/3133266697 shutdown_connections 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 1 -- 172.21.15.98:0/3133266697 wait complete. 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 10 monclient: build_initial_monmap 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.092+0000 7ff5299a1640 10 monclient: monmap: 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 1 librados: starting msgr at 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 1 librados: starting objecter 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 1 Processor -- start 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 1 -- start start 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 1 librados: setting wanted keys 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 1 librados: calling monclient init 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 10 monclient: init 2024-06-11T19:09:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 10 monclient: _add_conns ranks=[4,6,2,0,1,5,7,8,3] 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff52406c960 0x7ff5241cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 10 monclient(hunting): picked mon.noname-c con 0x7ff52406c960 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff5240a0c70 0x7ff5241ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 10 monclient(hunting): picked mon.noname-d con 0x7ff5240a0c70 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.093+0000 7ff5299a1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5240a2e30 0x7ff5241a8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 10 monclient(hunting): picked mon.noname-b con 0x7ff5240a2e30 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff5240a5230 con 0x7ff5240a2e30 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff5240a50b0 con 0x7ff52406c960 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff5240a53b0 con 0x7ff5240a0c70 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 10 monclient(hunting): authenticate will time out at 788.901046s 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff52406c960 0x7ff5241cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff52406c960 0x7ff5241cdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:40460/0 (socket says 172.21.15.98:40460) 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 1 -- 172.21.15.98:0/3336329157 learned_addr learned my addr 172.21.15.98:0/3336329157 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 10 monclient(hunting): get_auth_request con 0x7ff52406c960 auth_method 0 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff523fff640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff5240a0c70 0x7ff5241ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5291a0640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5240a2e30 0x7ff5241a8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff523fff640 10 monclient(hunting): get_auth_request con 0x7ff5240a0c70 auth_method 0 2024-06-11T19:09:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff523fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff523fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5291a0640 10 monclient(hunting): get_auth_request con 0x7ff5240a2e30 auth_method 0 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5291a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5291a0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5291a0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff523fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff523fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff523fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5291a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5291a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5291a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2024-06-11T19:09:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 1 -- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff5240a0c70 msgr2=0x7ff5241ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff5240a0c70 0x7ff5241ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 1 -- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5240a2e30 msgr2=0x7ff5241a8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5240a2e30 0x7ff5241a8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 10 monclient: _finish_hunting 0 2024-06-11T19:09:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 1 monclient: found mon.noname-c 2024-06-11T19:09:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:39.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 1 -- 172.21.15.98:0/3336329157 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5241acd40 con 0x7ff52406c960 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5291a0640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff5240a2e30 0x7ff5241a8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff52899f640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff52406c960 0x7ff5241cdb70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff5140098a0 tx=0x7ff5140046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 10 client.?.objecter ms_handle_connect 0x7ff52406c960 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff514041070 con 0x7ff52406c960 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 10 monclient: dump: 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.115 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.116 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 10 monclient: _finish_auth 0 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 10 monclient: _check_auth_tickets 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5140315a0 con 0x7ff52406c960 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 5 monclient: authenticate success, global_id 4405 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 10 monclient: _renew_subs 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 1 -- 172.21.15.98:0/3336329157 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5240a1ed0 con 0x7ff52406c960 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff514031bc0 con 0x7ff52406c960 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5217fa640 10 monclient: dump: 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.119 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.119 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.119 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.119 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:39.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 10 monclient: _renew_subs 2024-06-11T19:09:39.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:39.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 1 -- 172.21.15.98:0/3336329157 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5241aca60 con 0x7ff52406c960 2024-06-11T19:09:39.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff5299a1640 1 librados: init done 2024-06-11T19:09:39.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.094+0000 7ff522ffd640 4 set_mon_vals no callback set 2024-06-11T19:09:39.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff51404c060 con 0x7ff52406c960 2024-06-11T19:09:39.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff5299a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff5299a1640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241ad710 0x7ff5241a92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff5299a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff5299a1640 1 -- 172.21.15.98:0/3336329157 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff5240a2100 con 0x7ff5241ad710 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff523fff640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241ad710 0x7ff5241a92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff523fff640 10 monclient: get_auth_request con 0x7ff5241ad710 auth_method 0 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff523fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff523fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff523fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff523fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff523fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.095+0000 7ff5217fa640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff51003fdb0 0x7ff510042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7ff51402f050 con 0x7ff52406c960 2024-06-11T19:09:39.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 10 client.4405.objecter ms_dispatch 0x7ff524000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:39.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 3 client.4405.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:39.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 3 client.4405.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:39.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff523fff640 10 monclient(hunting): handle_auth_done global_id 4437 payload 824 2024-06-11T19:09:39.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff523fff640 1 -- 172.21.15.98:0/3336329157 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5240a28f0 con 0x7ff5241ad710 2024-06-11T19:09:39.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 20 client.4405.objecter dump_active .. 0 homeless 2024-06-11T19:09:39.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5291a0640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff51003fdb0 0x7ff510042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5291a0640 10 monclient: get_auth_request con 0x7ff51003fdb0 auth_method 0 2024-06-11T19:09:39.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff523fff640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241ad710 0x7ff5241a92c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff5180045e0 tx=0x7ff518007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 10 client.4405.objecter ms_handle_connect 0x7ff5241ad710 2024-06-11T19:09:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 10 client.4405.objecter resend_mon_ops 2024-06-11T19:09:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff518015890 con 0x7ff5241ad710 2024-06-11T19:09:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 10 monclient: dump: 2024-06-11T19:09:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.122 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 10 client.4405.objecter ms_handle_connect 0x7ff51003fdb0 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5291a0640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff51003fdb0 0x7ff510042270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff5241be960 tx=0x7ff51c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff51801a070 con 0x7ff5241ad710 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:39.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241ad710 msgr2=0x7ff5241a92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241ad710 0x7ff5241a92c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff5180045e0 tx=0x7ff518007700 comp rx=0 tx=0).stop 2024-06-11T19:09:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241ad710 msgr2=0x7ff5241a92c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:39.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.096+0000 7ff5217fa640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241ad710 0x7ff5241a92c0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff5180045e0 tx=0x7ff518007700 comp rx=0 tx=0).stop 2024-06-11T19:09:39.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.102+0000 7ff5299a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:39.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.102+0000 7ff5299a1640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241a9800 0x7ff5241a9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.102+0000 7ff5299a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.102+0000 7ff5299a1640 1 -- 172.21.15.98:0/3336329157 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff5241ac630 con 0x7ff5241a9800 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff523fff640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241a9800 0x7ff5241a9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff523fff640 10 monclient: get_auth_request con 0x7ff5241a9800 auth_method 0 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff523fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff523fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff523fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff523fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff523fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff523fff640 10 monclient(hunting): handle_auth_done global_id 4446 payload 824 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff523fff640 1 -- 172.21.15.98:0/3336329157 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff524063be0 con 0x7ff5241a9800 2024-06-11T19:09:39.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 10 client.4405.objecter ms_handle_connect 0x7ff5241a9800 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff523fff640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241a9800 0x7ff5241a9be0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff5180045e0 tx=0x7ff518004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 10 client.4405.objecter resend_mon_ops 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff518015aa0 con 0x7ff5241a9800 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 10 monclient: dump: 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.127 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9442 (secure 0 0 0) 0x7ff518015630 con 0x7ff5241a9800 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241a9800 msgr2=0x7ff5241a9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241a9800 0x7ff5241a9be0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff5180045e0 tx=0x7ff518004a40 comp rx=0 tx=0).stop 2024-06-11T19:09:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 1 -- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241a9800 msgr2=0x7ff5241a9be0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:39.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.103+0000 7ff5217fa640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5241a9800 0x7ff5241a9be0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff5180045e0 tx=0x7ff518004a40 comp rx=0 tx=0).stop 2024-06-11T19:09:39.129 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:39.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:39.129 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:39.129 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:39.129 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 8, 2024-06-11T19:09:39.129 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 29340, 2024-06-11T19:09:39.129 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:39.129 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:39.129 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:39.130 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:39.130 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:39.130 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:39.130 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:39.130 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:09:39.130 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:39.130 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:39.130 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.130 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 24, 2024-06-11T19:09:39.130 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:39.130 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.131 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.133 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:39.133 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:39.133 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:39.133 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:39.133 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:39.133 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:39.133 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:39.133 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:39.133 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:39.133 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:39.133 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:39.133 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.135 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.135 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.135 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.135 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.135 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:39.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.135 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:39.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.135 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:39.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:39.135 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:39.137 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.138 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.139 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:39.141 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:39.142 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.142 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.142 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.142 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:39.142 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:39.142 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.143 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:39.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:39.148 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.150 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:39.150 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.150 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.150 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.150 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:39.150 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:39.150 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.150 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.150 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.151 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:39.151 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:39.151 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.151 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.151 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:39.151 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.152 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.153 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:39.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.154 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.154 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.154 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.154 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.154 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:39.154 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:39.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 10 librados: watch_flush enter 2024-06-11T19:09:39.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 10 librados: watch_flush exit 2024-06-11T19:09:39.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 20 client.4405.objecter shutdown clearing up homeless session... 2024-06-11T19:09:39.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 10 client.4405.objecter successfully canceled tick 2024-06-11T19:09:39.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 1 -- 172.21.15.98:0/3336329157 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff51003fdb0 msgr2=0x7ff510042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff51003fdb0 0x7ff510042270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff5241be960 tx=0x7ff51c008040 comp rx=0 tx=0).stop 2024-06-11T19:09:39.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 10 monclient: shutdown 2024-06-11T19:09:39.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:39.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 1 -- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff52406c960 msgr2=0x7ff5241cdb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 1 --2- 172.21.15.98:0/3336329157 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff52406c960 0x7ff5241cdb70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff5140098a0 tx=0x7ff5140046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5291a0640 1 -- 172.21.15.98:0/3336329157 reap_dead start 2024-06-11T19:09:39.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5291a0640 1 -- 172.21.15.98:0/3336329157 reap_dead start 2024-06-11T19:09:39.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 1 -- 172.21.15.98:0/3336329157 shutdown_connections 2024-06-11T19:09:39.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 1 -- 172.21.15.98:0/3336329157 >> 172.21.15.98:0/3336329157 conn(0x7ff5240a9440 msgr2=0x7ff5241bfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:39.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 1 -- 172.21.15.98:0/3336329157 shutdown_connections 2024-06-11T19:09:39.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 1 -- 172.21.15.98:0/3336329157 wait complete. 2024-06-11T19:09:39.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.104+0000 7ff5299a1640 1 librados: shutdown 2024-06-11T19:09:39.157 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:09:39.157 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:09:39.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:09:39.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.257+0000 7fac2bd31640 10 monclient: get_monmap_and_config 2024-06-11T19:09:39.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.257+0000 7fac2bd31640 10 monclient: build_initial_monmap 2024-06-11T19:09:39.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.257+0000 7fac2bd31640 10 monclient: monmap: 2024-06-11T19:09:39.257 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:39.257 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.257 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:39.258209+0000 2024-06-11T19:09:39.257 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:39.258209+0000 2024-06-11T19:09:39.257 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:39.257 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.257 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:39.258 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.257+0000 7fac2bd31640 1 Processor -- start 2024-06-11T19:09:39.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 1 -- start start 2024-06-11T19:09:39.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 10 monclient: init 2024-06-11T19:09:39.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:39.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 10 monclient: _add_conns ranks=[8,3,1,4,10,2,5,9,0,7,6] 2024-06-11T19:09:39.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a4920 0x7fac240a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 10 monclient(hunting): picked mon.noname-e con 0x7fac240a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:39.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac240a5240 0x7fac240a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 10 monclient(hunting): picked mon.noname-g con 0x7fac240a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:39.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fac240a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:39.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac240a7140 con 0x7fac240a2530 2024-06-11T19:09:39.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fac240a2b50 con 0x7fac240a5240 2024-06-11T19:09:39.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fac240a2e30 con 0x7fac240a4920 2024-06-11T19:09:39.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:39.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac2bd31640 10 monclient(hunting): authenticate will time out at 789.065519s 2024-06-11T19:09:39.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a4920 0x7fac240a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a4920 0x7fac240a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54558/0 (socket says 172.21.15.98:54558) 2024-06-11T19:09:39.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 1 -- 172.21.15.98:0/1995527082 learned_addr learned my addr 172.21.15.98:0/1995527082 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:39.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 10 monclient(hunting): get_auth_request con 0x7fac240a4920 auth_method 0 2024-06-11T19:09:39.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac292cc640 1 --2- 172.21.15.98:0/1995527082 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac240a5240 0x7fac240a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac292cc640 10 monclient(hunting): get_auth_request con 0x7fac240a5240 auth_method 0 2024-06-11T19:09:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac292cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac292cc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac292cc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac292cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac292cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac292cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 10 monclient(hunting): handle_auth_done global_id 4400 payload 274 2024-06-11T19:09:39.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 1 -- 172.21.15.98:0/1995527082 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac240a5240 msgr2=0x7fac240a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:39.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 1 --2- 172.21.15.98:0/1995527082 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac240a5240 0x7fac240a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 1 -- 172.21.15.98:0/1995527082 >> v1:172.21.15.117:6789/0 conn(0x7fac240a2530 legacy=0x7fac240a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:39.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 1 monclient: found mon.noname-e 2024-06-11T19:09:39.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:39.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:39.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.258+0000 7fac29acd640 1 -- 172.21.15.98:0/1995527082 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac240a3110 con 0x7fac240a4920 2024-06-11T19:09:39.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac29acd640 1 --2- 172.21.15.98:0/1995527082 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a4920 0x7fac240a4d00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fac20009e80 tx=0x7fac2002be10 comp rx=0 tx=0).ready entity=mon.8 client_cookie=202412a13ff669ce server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 1 -- 172.21.15.98:0/1995527082 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac2002cc20 con 0x7fac240a4920 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 10 monclient: dump: 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.272 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 10 monclient: _finish_auth 0 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 1 -- 172.21.15.98:0/1995527082 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac2002cde0 con 0x7fac240a4920 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac2bd31640 5 monclient: authenticate success, global_id 4400 2024-06-11T19:09:39.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac2bd31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac2bd31640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 1 -- 172.21.15.98:0/1995527082 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac20034860 con 0x7fac240a4920 2024-06-11T19:09:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac2bd31640 4 set_mon_vals no callback set 2024-06-11T19:09:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.259+0000 7fac28acb640 10 monclient: dump: 2024-06-11T19:09:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.260+0000 7fac2bd31640 10 monclient: shutdown 2024-06-11T19:09:39.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.260+0000 7fac2bd31640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.260+0000 7fac2bd31640 1 -- 172.21.15.98:0/1995527082 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a4920 msgr2=0x7fac240a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.260+0000 7fac2bd31640 1 --2- 172.21.15.98:0/1995527082 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a4920 0x7fac240a4d00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fac20009e80 tx=0x7fac2002be10 comp rx=0 tx=0).stop 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.260+0000 7fac2bd31640 1 -- 172.21.15.98:0/1995527082 shutdown_connections 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.260+0000 7fac2bd31640 1 --2- 172.21.15.98:0/1995527082 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac240a5240 0x7fac240a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.260+0000 7fac2bd31640 1 --2- 172.21.15.98:0/1995527082 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a4920 0x7fac240a4d00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.260+0000 7fac2bd31640 1 -- 172.21.15.98:0/1995527082 >> 172.21.15.98:0/1995527082 conn(0x7fac240ab850 msgr2=0x7fac240a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.260+0000 7fac2bd31640 1 -- 172.21.15.98:0/1995527082 shutdown_connections 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.260+0000 7fac2bd31640 1 -- 172.21.15.98:0/1995527082 wait complete. 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.260+0000 7fac2bd31640 10 monclient: build_initial_monmap 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 10 monclient: monmap: 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 1 librados: starting msgr at 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 1 librados: starting objecter 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 1 Processor -- start 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 1 -- start start 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 1 librados: setting wanted keys 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 1 librados: calling monclient init 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 10 monclient: init 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 10 monclient: _add_conns ranks=[8,3,1,7,2,4,0,5,6] 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a2530 0x7fac241cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 10 monclient(hunting): picked mon.noname-e con 0x7fac240a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac240a4920 0x7fac241ce330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 10 monclient(hunting): picked mon.noname-g con 0x7fac240a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac240a5240 0x7fac24125170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 10 monclient(hunting): picked mon.noname-f con 0x7fac240a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fac240a6da0 con 0x7fac240a5240 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fac24138e50 con 0x7fac240a4920 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fac240a65f0 con 0x7fac240a2530 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2bd31640 10 monclient(hunting): authenticate will time out at 789.068830s 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac29acd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a2530 0x7fac241cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac29acd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a2530 0x7fac241cddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54570/0 (socket says 172.21.15.98:54570) 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac29acd640 1 -- 172.21.15.98:0/3171207417 learned_addr learned my addr 172.21.15.98:0/3171207417 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac29acd640 10 monclient(hunting): get_auth_request con 0x7fac240a2530 auth_method 0 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac29acd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac29acd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac29acd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac29acd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac29acd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac292cc640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac240a4920 0x7fac241ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac29acd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.261+0000 7fac2a2ce640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac240a5240 0x7fac24125170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac292cc640 10 monclient(hunting): get_auth_request con 0x7fac240a4920 auth_method 0 2024-06-11T19:09:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac292cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac292cc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac292cc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2a2ce640 10 monclient(hunting): get_auth_request con 0x7fac240a5240 auth_method 0 2024-06-11T19:09:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2a2ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2a2ce640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2a2ce640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac292cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac292cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac292cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2a2ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2a2ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2a2ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac29acd640 10 monclient(hunting): handle_auth_done global_id 4409 payload 824 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac29acd640 1 -- 172.21.15.98:0/3171207417 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac240a4920 msgr2=0x7fac241ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac29acd640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac240a4920 0x7fac241ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac29acd640 1 -- 172.21.15.98:0/3171207417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac240a5240 msgr2=0x7fac24125170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac29acd640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac240a5240 0x7fac24125170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac29acd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac29acd640 1 monclient: found mon.noname-e 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac29acd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac29acd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac29acd640 1 -- 172.21.15.98:0/3171207417 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac241272d0 con 0x7fac240a2530 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac29acd640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a2530 0x7fac241cddf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fac2002c420 tx=0x7fac20004ab0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 10 client.?.objecter ms_handle_connect 0x7fac240a2530 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac20043020 con 0x7fac240a2530 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 10 monclient: dump: 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac20007d30 con 0x7fac240a2530 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2bd31640 5 monclient: authenticate success, global_id 4409 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2bd31640 10 monclient: _renew_subs 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2bd31640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2bd31640 1 -- 172.21.15.98:0/3171207417 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac24128b30 con 0x7fac240a2530 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac2003ed30 con 0x7fac240a2530 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2bd31640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1a7fc640 10 monclient: dump: 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.285 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.286 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.286 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.286 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.286 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.286 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac1bfff640 4 set_mon_vals no callback set 2024-06-11T19:09:39.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2bd31640 10 monclient: _renew_subs 2024-06-11T19:09:39.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2bd31640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:39.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2bd31640 1 -- 172.21.15.98:0/3171207417 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fac240a9030 con 0x7fac240a2530 2024-06-11T19:09:39.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.262+0000 7fac2bd31640 1 librados: init done 2024-06-11T19:09:39.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fac20046020 con 0x7fac240a2530 2024-06-11T19:09:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac2bd31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac2bd31640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac241256b0 0x7fac24125a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac2bd31640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac2bd31640 1 -- 172.21.15.98:0/3171207417 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fac24050830 con 0x7fac241256b0 2024-06-11T19:09:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac292cc640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac241256b0 0x7fac24125a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac292cc640 10 monclient: get_auth_request con 0x7fac241256b0 auth_method 0 2024-06-11T19:09:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac292cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac292cc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac292cc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac1a7fc640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fac0803fde0 0x7fac080422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac292cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac292cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac292cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fac2003a040 con 0x7fac240a2530 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac1a7fc640 10 client.4409.objecter ms_dispatch 0x7fac24000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac1a7fc640 3 client.4409.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac1a7fc640 3 client.4409.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac1a7fc640 20 client.4409.objecter dump_active .. 0 homeless 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac292cc640 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac2a2ce640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fac0803fde0 0x7fac080422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac292cc640 1 -- 172.21.15.98:0/3171207417 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac240aacf0 con 0x7fac241256b0 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac2a2ce640 10 monclient: get_auth_request con 0x7fac0803fde0 auth_method 0 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac1a7fc640 10 client.4409.objecter ms_handle_connect 0x7fac241256b0 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac1a7fc640 10 client.4409.objecter resend_mon_ops 2024-06-11T19:09:39.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.263+0000 7fac292cc640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac241256b0 0x7fac24125a90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fac240a5620 tx=0x7fac14011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac14019070 con 0x7fac241256b0 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac1a7fc640 10 monclient: dump: 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.289 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac1a7fc640 10 client.4409.objecter ms_handle_connect 0x7fac0803fde0 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac2a2ce640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fac0803fde0 0x7fac080422a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fac1c000f60 tx=0x7fac1c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fac14004240 con 0x7fac241256b0 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac1a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac1a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac241256b0 msgr2=0x7fac24125a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac1a7fc640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac241256b0 0x7fac24125a90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fac240a5620 tx=0x7fac14011cf0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac241256b0 msgr2=0x7fac24125a90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:39.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.264+0000 7fac1a7fc640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac241256b0 0x7fac24125a90 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fac240a5620 tx=0x7fac14011cf0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.270+0000 7fac2bd31640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:39.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.270+0000 7fac2bd31640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac24125fd0 0x7fac241215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.270+0000 7fac2bd31640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.270+0000 7fac2bd31640 1 -- 172.21.15.98:0/3171207417 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fac240a4d00 con 0x7fac24125fd0 2024-06-11T19:09:39.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.270+0000 7fac292cc640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac24125fd0 0x7fac241215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.270+0000 7fac292cc640 10 monclient: get_auth_request con 0x7fac24125fd0 auth_method 0 2024-06-11T19:09:39.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.270+0000 7fac292cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.270+0000 7fac292cc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.270+0000 7fac292cc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.270+0000 7fac292cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.270+0000 7fac292cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.270+0000 7fac292cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac292cc640 10 monclient(hunting): handle_auth_done global_id 4403 payload 824 2024-06-11T19:09:39.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac292cc640 1 -- 172.21.15.98:0/3171207417 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac2406c300 con 0x7fac24125fd0 2024-06-11T19:09:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac292cc640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac24125fd0 0x7fac241215f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac240a5620 tx=0x7fac140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 10 client.4409.objecter ms_handle_connect 0x7fac24125fd0 2024-06-11T19:09:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 10 client.4409.objecter resend_mon_ops 2024-06-11T19:09:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac14019040 con 0x7fac24125fd0 2024-06-11T19:09:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 10 monclient: dump: 2024-06-11T19:09:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.294 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9444 (secure 0 0 0) 0x7fac14003c30 con 0x7fac24125fd0 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac24125fd0 msgr2=0x7fac241215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac24125fd0 0x7fac241215f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac240a5620 tx=0x7fac140046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 1 -- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac24125fd0 msgr2=0x7fac241215f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:39.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac1a7fc640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fac24125fd0 0x7fac241215f0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac240a5620 tx=0x7fac140046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.296 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:39.296 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:39.296 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:39.296 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:09:39.296 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 8, 2024-06-11T19:09:39.296 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 29504, 2024-06-11T19:09:39.296 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 24, 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:39.297 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.298 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:39.300 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.301 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.303 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.307 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:39.309 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:39.311 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:39.312 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.315 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.318 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.318 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.318 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:39.318 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac2bd31640 10 librados: watch_flush enter 2024-06-11T19:09:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac2bd31640 10 librados: watch_flush exit 2024-06-11T19:09:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac2bd31640 20 client.4409.objecter shutdown clearing up homeless session... 2024-06-11T19:09:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac2bd31640 10 client.4409.objecter successfully canceled tick 2024-06-11T19:09:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac2bd31640 1 -- 172.21.15.98:0/3171207417 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fac0803fde0 msgr2=0x7fac080422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac2bd31640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fac0803fde0 0x7fac080422a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fac1c000f60 tx=0x7fac1c009040 comp rx=0 tx=0).stop 2024-06-11T19:09:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac2bd31640 10 monclient: shutdown 2024-06-11T19:09:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac2bd31640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac2bd31640 1 -- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a2530 msgr2=0x7fac241cddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac2bd31640 1 --2- 172.21.15.98:0/3171207417 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac240a2530 0x7fac241cddf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fac2002c420 tx=0x7fac20004ab0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.271+0000 7fac2a2ce640 1 -- 172.21.15.98:0/3171207417 reap_dead start 2024-06-11T19:09:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.272+0000 7fac2a2ce640 1 -- 172.21.15.98:0/3171207417 reap_dead start 2024-06-11T19:09:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.272+0000 7fac2bd31640 1 -- 172.21.15.98:0/3171207417 shutdown_connections 2024-06-11T19:09:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.272+0000 7fac2bd31640 1 -- 172.21.15.98:0/3171207417 >> 172.21.15.98:0/3171207417 conn(0x7fac240ab850 msgr2=0x7fac2409c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.272+0000 7fac2bd31640 1 -- 172.21.15.98:0/3171207417 shutdown_connections 2024-06-11T19:09:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.272+0000 7fac2bd31640 1 -- 172.21.15.98:0/3171207417 wait complete. 2024-06-11T19:09:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.272+0000 7fac2bd31640 1 librados: shutdown 2024-06-11T19:09:39.321 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:09:39.321 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:09:39.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:09:39.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.421+0000 7f0cb7f3f640 10 monclient: get_monmap_and_config 2024-06-11T19:09:39.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.421+0000 7f0cb7f3f640 10 monclient: build_initial_monmap 2024-06-11T19:09:39.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.421+0000 7f0cb7f3f640 10 monclient: monmap: 2024-06-11T19:09:39.421 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:39.421 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.421 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:39.422670+0000 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:39.422670+0000 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:39.422 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:39.423 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 1 Processor -- start 2024-06-11T19:09:39.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 1 -- start start 2024-06-11T19:09:39.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 10 monclient: init 2024-06-11T19:09:39.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:39.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 10 monclient: _add_conns ranks=[8,10,7,2,1,3,5,0,6,9,4] 2024-06-11T19:09:39.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cb00a4920 0x7f0cb00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 10 monclient(hunting): picked mon.noname-e con 0x7f0cb00a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0cb00a5240 0x7f0cb00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 10 monclient(hunting): picked mon.noname-f con 0x7f0cb00a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0cb00a2530 0x7f0cb00a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 10 monclient(hunting): picked mon.noname-i con 0x7f0cb00a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.422+0000 7f0cb7f3f640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f0cb00a7140 con 0x7f0cb00a5240 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb7f3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb7f3f640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f0cb00a2b50 con 0x7f0cb00a2530 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb7f3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb7f3f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0cb00a2e30 con 0x7f0cb00a4920 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb7f3f640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb7f3f640 10 monclient(hunting): authenticate will time out at 789.230020s 2024-06-11T19:09:39.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cb00a4920 0x7f0cb00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cb00a4920 0x7f0cb00a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54572/0 (socket says 172.21.15.98:54572) 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 1 -- 172.21.15.98:0/469597212 learned_addr learned my addr 172.21.15.98:0/469597212 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 10 monclient(hunting): get_auth_request con 0x7f0cb00a4920 auth_method 0 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb64dc640 1 --2- 172.21.15.98:0/469597212 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0cb00a2530 0x7f0cb00a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb64dc640 10 monclient(hunting): get_auth_request con 0x7f0cb00a2530 auth_method 0 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb54da640 1 --2- 172.21.15.98:0/469597212 >> v2:172.21.15.117:3300/0 conn(0x7f0cb00a5240 0x7f0cb00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb64dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb64dc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb64dc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 10 monclient(hunting): handle_auth_done global_id 4418 payload 274 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 1 -- 172.21.15.98:0/469597212 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0cb00a2530 msgr2=0x7f0cb00a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 1 --2- 172.21.15.98:0/469597212 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0cb00a2530 0x7f0cb00a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 1 -- 172.21.15.98:0/469597212 >> v2:172.21.15.117:3300/0 conn(0x7f0cb00a5240 msgr2=0x7f0cb00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 1 --2- 172.21.15.98:0/469597212 >> v2:172.21.15.117:3300/0 conn(0x7f0cb00a5240 0x7f0cb00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 10 monclient: _finish_hunting 0 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 1 monclient: found mon.noname-e 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb5cdb640 1 -- 172.21.15.98:0/469597212 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb00a3110 con 0x7f0cb00a4920 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb54da640 10 monclient: get_auth_request con 0x7f0cb00a5240 auth_method 0 2024-06-11T19:09:39.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.423+0000 7f0cb54da640 1 --2- 172.21.15.98:0/469597212 >> v2:172.21.15.117:3300/0 conn(0x7f0cb00a5240 0x7f0cb00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb5cdb640 1 --2- 172.21.15.98:0/469597212 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cb00a4920 0x7f0cb00a4d00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009870 tx=0x7f0cac02b7c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=21759f0e5c64c9aa server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 1 -- 172.21.15.98:0/469597212 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0cac038070 con 0x7f0cb00a4920 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 10 monclient: dump: 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.436 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 10 monclient: _finish_auth 0 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 10 monclient: _check_auth_tickets 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 1 -- 172.21.15.98:0/469597212 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cac031990 con 0x7f0cb00a4920 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 5 monclient: authenticate success, global_id 4418 2024-06-11T19:09:39.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 1 -- 172.21.15.98:0/469597212 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0cac0323b0 con 0x7f0cb00a4920 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 4 set_mon_vals no callback set 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb4cd9640 10 monclient: dump: 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.438 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 10 monclient: shutdown 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/469597212 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cb00a4920 msgr2=0x7f0cb00a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 1 --2- 172.21.15.98:0/469597212 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cb00a4920 0x7f0cb00a4d00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009870 tx=0x7f0cac02b7c0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/469597212 shutdown_connections 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 1 --2- 172.21.15.98:0/469597212 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0cb00a2530 0x7f0cb00a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 1 --2- 172.21.15.98:0/469597212 >> v2:172.21.15.117:3300/0 conn(0x7f0cb00a5240 0x7f0cb00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 1 --2- 172.21.15.98:0/469597212 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cb00a4920 0x7f0cb00a4d00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/469597212 >> 172.21.15.98:0/469597212 conn(0x7f0cb00ab850 msgr2=0x7f0cb00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/469597212 shutdown_connections 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.424+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/469597212 wait complete. 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.425+0000 7f0cb7f3f640 10 monclient: build_initial_monmap 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.425+0000 7f0cb7f3f640 10 monclient: monmap: 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:39.440 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.425+0000 7f0cb7f3f640 1 librados: starting msgr at 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.425+0000 7f0cb7f3f640 1 librados: starting objecter 2024-06-11T19:09:39.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.425+0000 7f0cb7f3f640 1 Processor -- start 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.425+0000 7f0cb7f3f640 1 -- start start 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.425+0000 7f0cb7f3f640 1 librados: setting wanted keys 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.425+0000 7f0cb7f3f640 1 librados: calling monclient init 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.425+0000 7f0cb7f3f640 10 monclient: init 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 10 monclient: _add_conns ranks=[2,5,1,0,4,3,6,8,7] 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb00a2530 0x7f0cb01cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 10 monclient(hunting): picked mon.noname-b con 0x7f0cb00a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0cb00a4920 0x7f0cb01cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 10 monclient(hunting): picked mon.noname-h con 0x7f0cb00a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cb00a5240 0x7f0cb0122e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 10 monclient(hunting): picked mon.noname-f con 0x7f0cb00a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:39.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb00a6da0 con 0x7f0cb00a5240 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0cb0138e50 con 0x7f0cb00a2530 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f0cb00a6420 con 0x7f0cb00a4920 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 10 monclient(hunting): authenticate will time out at 789.233212s 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb00a2530 0x7f0cb01cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb00a2530 0x7f0cb01cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48050/0 (socket says 172.21.15.98:48050) 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 1 -- 172.21.15.98:0/2976223514 learned_addr learned my addr 172.21.15.98:0/2976223514 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 10 monclient(hunting): get_auth_request con 0x7f0cb00a2530 auth_method 0 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb64dc640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cb00a5240 0x7f0cb0122e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb54da640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0cb00a4920 0x7f0cb01cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 10 monclient(hunting): handle_auth_done global_id 4412 payload 824 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 1 -- 172.21.15.98:0/2976223514 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0cb00a4920 msgr2=0x7f0cb01cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0cb00a4920 0x7f0cb01cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 1 -- 172.21.15.98:0/2976223514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cb00a5240 msgr2=0x7f0cb0122e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cb00a5240 0x7f0cb0122e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 10 monclient: _finish_hunting 0 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 1 monclient: found mon.noname-b 2024-06-11T19:09:39.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 1 -- 172.21.15.98:0/2976223514 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb0126ab0 con 0x7f0cb00a2530 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb64dc640 10 monclient: get_auth_request con 0x7f0cb00a5240 auth_method 0 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb64dc640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cb00a5240 0x7f0cb0122e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb5cdb640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb00a2530 0x7f0cb01cda70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0cac0090b0 tx=0x7f0cac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 10 client.?.objecter ms_handle_connect 0x7f0cb00a2530 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0cac042020 con 0x7f0cb00a2530 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 10 monclient: dump: 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.445 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.446 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cac004470 con 0x7f0cb00a2530 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 5 monclient: authenticate success, global_id 4412 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 10 monclient: _renew_subs 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/2976223514 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb0125600 con 0x7f0cb00a2530 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0cac038040 con 0x7f0cb00a2530 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0ca67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0ca67fc640 10 monclient: dump: 2024-06-11T19:09:39.447 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.448 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.448 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.448 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.448 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.448 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.448 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.448 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.448 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.448 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.448 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.426+0000 7f0ca7fff640 4 set_mon_vals no callback set 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb7f3f640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb7f3f640 10 monclient: _renew_subs 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb7f3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/2976223514 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb00a9240 con 0x7f0cb00a2530 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb7f3f640 1 librados: init done 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb7f3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb7f3f640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb01233d0 0x7f0cb01237b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb7f3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/2976223514 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0cb0125820 con 0x7f0cb01233d0 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f0cac045020 con 0x7f0cb00a2530 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb54da640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb01233d0 0x7f0cb01237b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb54da640 10 monclient: get_auth_request con 0x7f0cb01233d0 auth_method 0 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb54da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb54da640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.427+0000 7f0cb54da640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0cb54da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0cb54da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0cb54da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0c9403fdb0 0x7f0c94042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0cb54da640 10 monclient(hunting): handle_auth_done global_id 4414 payload 824 2024-06-11T19:09:39.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f0cac030070 con 0x7f0cb00a2530 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0cb54da640 1 -- 172.21.15.98:0/2976223514 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0cb00a6750 con 0x7f0cb01233d0 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 10 client.4412.objecter ms_dispatch 0x7f0cb0000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 3 client.4412.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 3 client.4412.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0cb64dc640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0c9403fdb0 0x7f0c94042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 20 client.4412.objecter dump_active .. 0 homeless 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0cb64dc640 10 monclient: get_auth_request con 0x7f0c9403fdb0 auth_method 0 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 10 client.4412.objecter ms_handle_connect 0x7f0cb01233d0 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 10 client.4412.objecter resend_mon_ops 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0cb54da640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb01233d0 0x7f0cb01237b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00a5620 tx=0x7f0ca000d910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ca0016070 con 0x7f0cb01233d0 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:39.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 10 monclient: dump: 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.452 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.453 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 10 client.4412.objecter ms_handle_connect 0x7f0c9403fdb0 2024-06-11T19:09:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0cb64dc640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0c9403fdb0 0x7f0c94042270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009f70 tx=0x7f0ca800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0ca0012670 con 0x7f0cb01233d0 2024-06-11T19:09:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb01233d0 msgr2=0x7f0cb01237b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb01233d0 0x7f0cb01237b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00a5620 tx=0x7f0ca000d910 comp rx=0 tx=0).stop 2024-06-11T19:09:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb01233d0 msgr2=0x7f0cb01237b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:39.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.428+0000 7f0ca67fc640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb01233d0 0x7f0cb01237b0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00a5620 tx=0x7f0ca000d910 comp rx=0 tx=0).stop 2024-06-11T19:09:39.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb7f3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:39.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb7f3f640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb0123cf0 0x7f0cb009e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb7f3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/2976223514 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0cb0050830 con 0x7f0cb0123cf0 2024-06-11T19:09:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb54da640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb0123cf0 0x7f0cb009e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb54da640 10 monclient: get_auth_request con 0x7f0cb0123cf0 auth_method 0 2024-06-11T19:09:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb54da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb54da640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb54da640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb54da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb54da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb54da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb54da640 10 monclient(hunting): handle_auth_done global_id 4423 payload 824 2024-06-11T19:09:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb54da640 1 -- 172.21.15.98:0/2976223514 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0cb00a3480 con 0x7f0cb0123cf0 2024-06-11T19:09:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0cb54da640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb0123cf0 0x7f0cb009e9f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00a5620 tx=0x7f0ca0004730 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0ca67fc640 10 client.4412.objecter ms_handle_connect 0x7f0cb0123cf0 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0ca67fc640 10 client.4412.objecter resend_mon_ops 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ca001bd30 con 0x7f0cb0123cf0 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0ca67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0ca67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.435+0000 7f0ca67fc640 10 monclient: dump: 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.457 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.458 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f0ca00113c0 con 0x7f0cb0123cf0 2024-06-11T19:09:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0ca67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0ca67fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb0123cf0 msgr2=0x7f0cb009e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0ca67fc640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb0123cf0 0x7f0cb009e9f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00a5620 tx=0x7f0ca0004730 comp rx=0 tx=0).stop 2024-06-11T19:09:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0ca67fc640 1 -- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb0123cf0 msgr2=0x7f0cb009e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0ca67fc640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cb0123cf0 0x7f0cb009e9f0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00a5620 tx=0x7f0ca0004730 comp rx=0 tx=0).stop 2024-06-11T19:09:39.459 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:39.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:39.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:39.459 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:39.459 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 8, 2024-06-11T19:09:39.460 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 29653, 2024-06-11T19:09:39.460 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:39.460 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:39.460 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:39.460 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:39.460 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:39.460 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:39.460 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:39.460 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:09:39.460 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:39.460 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:39.460 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.461 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 24, 2024-06-11T19:09:39.461 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:39.461 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:39.461 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:39.461 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.461 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.461 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.461 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.461 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.461 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.461 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.462 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:39.463 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:39.464 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:39.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:39.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:39.464 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:39.464 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:39.464 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:39.464 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:39.464 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:39.464 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:39.464 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:39.464 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.464 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.465 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:39.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.466 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:39.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:39.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:39.466 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.467 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.468 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:39.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:39.474 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:39.474 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.474 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.474 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.474 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:39.474 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:39.474 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.475 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.477 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:39.478 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.480 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.481 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:39.482 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:39.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 10 librados: watch_flush enter 2024-06-11T19:09:39.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 10 librados: watch_flush exit 2024-06-11T19:09:39.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 20 client.4412.objecter shutdown clearing up homeless session... 2024-06-11T19:09:39.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 10 client.4412.objecter successfully canceled tick 2024-06-11T19:09:39.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/2976223514 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0c9403fdb0 msgr2=0x7f0c94042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0c9403fdb0 0x7f0c94042270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009f70 tx=0x7f0ca800b040 comp rx=0 tx=0).stop 2024-06-11T19:09:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 10 monclient: shutdown 2024-06-11T19:09:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb00a2530 msgr2=0x7f0cb01cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 1 --2- 172.21.15.98:0/2976223514 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cb00a2530 0x7f0cb01cda70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0cac0090b0 tx=0x7f0cac0046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb64dc640 1 -- 172.21.15.98:0/2976223514 reap_dead start 2024-06-11T19:09:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb64dc640 1 -- 172.21.15.98:0/2976223514 reap_dead start 2024-06-11T19:09:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/2976223514 shutdown_connections 2024-06-11T19:09:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/2976223514 >> 172.21.15.98:0/2976223514 conn(0x7f0cb00ab850 msgr2=0x7f0cb01bffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/2976223514 shutdown_connections 2024-06-11T19:09:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 1 -- 172.21.15.98:0/2976223514 wait complete. 2024-06-11T19:09:39.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.436+0000 7f0cb7f3f640 1 librados: shutdown 2024-06-11T19:09:39.485 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:09:39.485 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:09:39.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:09:39.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.584+0000 7f714239c640 10 monclient: get_monmap_and_config 2024-06-11T19:09:39.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.584+0000 7f714239c640 10 monclient: build_initial_monmap 2024-06-11T19:09:39.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.584+0000 7f714239c640 10 monclient: monmap: 2024-06-11T19:09:39.584 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:39.584 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:39.585659+0000 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:39.585659+0000 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:39.585 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:39.586 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:39.586 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:39.586 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.585+0000 7f714239c640 1 Processor -- start 2024-06-11T19:09:39.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.585+0000 7f714239c640 1 -- start start 2024-06-11T19:09:39.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.585+0000 7f714239c640 10 monclient: init 2024-06-11T19:09:39.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.585+0000 7f714239c640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.585+0000 7f714239c640 10 monclient: _add_conns ranks=[1,3,8,7,4,2,10,5,9,0,6] 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f714239c640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f713c0a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f714239c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f713c0a2530 0x7f713c13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f714239c640 10 monclient(hunting): picked mon.noname-g con 0x7f713c0a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f714239c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c13e1a0 0x7f713c140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f714239c640 10 monclient(hunting): picked mon.noname-e con 0x7f713c13e1a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f714239c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f713c0a7140 con 0x7f713c0a4920 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f714239c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f714239c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f713c0a2b50 con 0x7f713c0a2530 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f714239c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f714239c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f713c0a2e30 con 0x7f713c13e1a0 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f714239c640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:39.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f714239c640 10 monclient(hunting): authenticate will time out at 789.393108s 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c13e1a0 0x7f713c140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c13e1a0 0x7f713c140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54588/0 (socket says 172.21.15.98:54588) 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f713bfff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f713c0a4920 0x7f713c0a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:40536/0 (socket says 172.21.15.98:40536) 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 1 -- 172.21.15.98:0/3106978807 learned_addr learned my addr 172.21.15.98:0/3106978807 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 10 monclient(hunting): get_auth_request con 0x7f713c13e1a0 auth_method 0 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f713b7fe640 1 --2- 172.21.15.98:0/3106978807 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f713c0a2530 0x7f713c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f713b7fe640 10 monclient(hunting): get_auth_request con 0x7f713c0a2530 auth_method 0 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f713b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f713b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f713b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f713b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f713b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f713b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 10 monclient(hunting): handle_auth_done global_id 4427 payload 274 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f713affd640 1 -- 172.21.15.98:0/3106978807 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1172487969 0 0) 0x7f713c0a7140 con 0x7f713c0a4920 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 1 -- 172.21.15.98:0/3106978807 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f713c0a2530 msgr2=0x7f713c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 1 --2- 172.21.15.98:0/3106978807 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f713c0a2530 0x7f713c13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 1 -- 172.21.15.98:0/3106978807 >> v1:172.21.15.117:6789/0 conn(0x7f713c0a4920 legacy=0x7f713c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 10 monclient: _finish_hunting 0 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 1 monclient: found mon.noname-e 2024-06-11T19:09:39.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f7140939640 1 -- 172.21.15.98:0/3106978807 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713c0a3110 con 0x7f713c13e1a0 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.586+0000 7f713affd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f7140939640 1 --2- 172.21.15.98:0/3106978807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c13e1a0 0x7f713c140590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7134009e80 tx=0x7f713402be10 comp rx=0 tx=0).ready entity=mon.8 client_cookie=14023a57ec803b9d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 1 -- 172.21.15.98:0/3106978807 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f713402cc20 con 0x7f713c13e1a0 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 10 monclient: dump: 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.599 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 10 monclient: _finish_auth 0 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:39.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 1 -- 172.21.15.98:0/3106978807 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f713402cde0 con 0x7f713c13e1a0 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 1 -- 172.21.15.98:0/3106978807 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7134034860 con 0x7f713c13e1a0 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f714239c640 5 monclient: authenticate success, global_id 4427 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f714239c640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f714239c640 4 set_mon_vals no callback set 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f713affd640 10 monclient: dump: 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.601 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f714239c640 10 monclient: shutdown 2024-06-11T19:09:39.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f714239c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:39.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f714239c640 1 -- 172.21.15.98:0/3106978807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c13e1a0 msgr2=0x7f713c140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f714239c640 1 --2- 172.21.15.98:0/3106978807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c13e1a0 0x7f713c140590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7134009e80 tx=0x7f713402be10 comp rx=0 tx=0).stop 2024-06-11T19:09:39.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f714239c640 1 -- 172.21.15.98:0/3106978807 shutdown_connections 2024-06-11T19:09:39.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f714239c640 1 --2- 172.21.15.98:0/3106978807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c13e1a0 0x7f713c140590 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f714239c640 1 --2- 172.21.15.98:0/3106978807 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f713c0a2530 0x7f713c13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.587+0000 7f714239c640 1 -- 172.21.15.98:0/3106978807 >> 172.21.15.98:0/3106978807 conn(0x7f713c0ab850 msgr2=0x7f713c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:39.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.588+0000 7f714239c640 1 -- 172.21.15.98:0/3106978807 shutdown_connections 2024-06-11T19:09:39.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.588+0000 7f714239c640 1 -- 172.21.15.98:0/3106978807 wait complete. 2024-06-11T19:09:39.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.588+0000 7f714239c640 10 monclient: build_initial_monmap 2024-06-11T19:09:39.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.588+0000 7f714239c640 10 monclient: monmap: 2024-06-11T19:09:39.603 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:39.603 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:39.604 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.588+0000 7f714239c640 1 librados: starting msgr at 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.588+0000 7f714239c640 1 librados: starting objecter 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.588+0000 7f714239c640 1 Processor -- start 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 1 -- start start 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 1 librados: setting wanted keys 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 1 librados: calling monclient init 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 10 monclient: init 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 10 monclient: _add_conns ranks=[1,8,7,0,3,6,2,5,4] 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713c0a2530 0x7f713c1cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 10 monclient(hunting): picked mon.noname-f con 0x7f713c0a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c0a4920 0x7f713c136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 10 monclient(hunting): picked mon.noname-e con 0x7f713c0a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f713c13e1a0 0x7f713c136e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 10 monclient(hunting): picked mon.noname-i con 0x7f713c13e1a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f713c138e50 con 0x7f713c0a2530 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f713c0a84b0 con 0x7f713c13e1a0 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f713c0a6da0 con 0x7f713c0a4920 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f714239c640 10 monclient(hunting): authenticate will time out at 789.396495s 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c0a4920 0x7f713c136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c0a4920 0x7f713c136950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54600/0 (socket says 172.21.15.98:54600) 2024-06-11T19:09:39.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 1 -- 172.21.15.98:0/3063793030 learned_addr learned my addr 172.21.15.98:0/3063793030 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 10 monclient(hunting): get_auth_request con 0x7f713c0a4920 auth_method 0 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 10 monclient(hunting): handle_auth_done global_id 4436 payload 824 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f7140939640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f713c13e1a0 0x7f713c136e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 1 -- 172.21.15.98:0/3063793030 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f713c13e1a0 msgr2=0x7f713c136e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713bfff640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713c0a2530 0x7f713c1cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f713c13e1a0 0x7f713c136e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 1 -- 172.21.15.98:0/3063793030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713c0a2530 msgr2=0x7f713c1cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713c0a2530 0x7f713c1cd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 1 monclient: found mon.noname-e 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.589+0000 7f713b7fe640 1 -- 172.21.15.98:0/3063793030 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713c123300 con 0x7f713c0a4920 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f713b7fe640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c0a4920 0x7f713c136950 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f713000e8b0 tx=0x7f713000ed70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 10 client.?.objecter ms_handle_connect 0x7f713c0a4920 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7130017070 con 0x7f713c0a4920 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 10 monclient: dump: 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 10 monclient: _finish_auth 0 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f713000cdb0 con 0x7f713c0a4920 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f714239c640 5 monclient: authenticate success, global_id 4436 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71300138e0 con 0x7f713c0a4920 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f7138ff9640 10 monclient: dump: 2024-06-11T19:09:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.612 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f713a7fc640 4 set_mon_vals no callback set 2024-06-11T19:09:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f714239c640 10 monclient: _renew_subs 2024-06-11T19:09:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f714239c640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f714239c640 1 -- 172.21.15.98:0/3063793030 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f713c0a41f0 con 0x7f713c0a4920 2024-06-11T19:09:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f714239c640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f714239c640 10 monclient: _renew_subs 2024-06-11T19:09:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f714239c640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f714239c640 1 -- 172.21.15.98:0/3063793030 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f713c0a1600 con 0x7f713c0a4920 2024-06-11T19:09:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.590+0000 7f714239c640 1 librados: init done 2024-06-11T19:09:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f713001a070 con 0x7f713c0a4920 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f714239c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f714239c640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c09f970 0x7f713c09fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f714239c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f714239c640 1 -- 172.21.15.98:0/3063793030 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f713c1ce470 con 0x7f713c09f970 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f712803fdb0 0x7f7128042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f713000f040 con 0x7f713c0a4920 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f713bfff640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c09f970 0x7f713c09fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 10 client.4436.objecter ms_dispatch 0x7f713c000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 3 client.4436.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 3 client.4436.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f713bfff640 10 monclient: get_auth_request con 0x7f713c09f970 auth_method 0 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f713bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f713bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f713bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f713bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f713bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f713bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 20 client.4436.objecter dump_active .. 0 homeless 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f713bfff640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f713bfff640 1 -- 172.21.15.98:0/3063793030 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f713c13d510 con 0x7f713c09f970 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7140939640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f712803fdb0 0x7f7128042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7140939640 10 monclient: get_auth_request con 0x7f712803fdb0 auth_method 0 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 10 client.4436.objecter ms_handle_connect 0x7f713c09f970 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 10 client.4436.objecter resend_mon_ops 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f713bfff640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c09f970 0x7f713c09fd50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f712c000f60 tx=0x7f712c00d880 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f712c0047e0 con 0x7f713c09f970 2024-06-11T19:09:39.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 10 monclient: dump: 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7138ff9640 10 client.4436.objecter ms_handle_connect 0x7f712803fdb0 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.591+0000 7f7140939640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f712803fdb0 0x7f7128042270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7134035000 tx=0x7f7134030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.592+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f712c0072e0 con 0x7f713c09f970 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.592+0000 7f7138ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.592+0000 7f7138ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.592+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c09f970 msgr2=0x7f713c09fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.592+0000 7f7138ff9640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c09f970 0x7f713c09fd50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f712c000f60 tx=0x7f712c00d880 comp rx=0 tx=0).stop 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.592+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c09f970 msgr2=0x7f713c09fd50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.592+0000 7f7138ff9640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c09f970 0x7f713c09fd50 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f712c000f60 tx=0x7f712c00d880 comp rx=0 tx=0).stop 2024-06-11T19:09:39.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f714239c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:39.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f714239c640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c0a0290 0x7f713c13baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f714239c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f714239c640 1 -- 172.21.15.98:0/3063793030 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f713c050830 con 0x7f713c0a0290 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f713bfff640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c0a0290 0x7f713c13baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f713bfff640 10 monclient: get_auth_request con 0x7f713c0a0290 auth_method 0 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f713bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f713bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f713bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f713bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f713bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f713bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f713bfff640 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f713bfff640 1 -- 172.21.15.98:0/3063793030 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f713c0a4d00 con 0x7f713c0a0290 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f713bfff640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c0a0290 0x7f713c13baf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f712c000f60 tx=0x7f712c000ed0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f7138ff9640 10 client.4436.objecter ms_handle_connect 0x7f713c0a0290 2024-06-11T19:09:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.598+0000 7f7138ff9640 10 client.4436.objecter resend_mon_ops 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f712c01adb0 con 0x7f713c0a0290 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7138ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7138ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7138ff9640 10 monclient: dump: 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.621 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.621 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.621 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.621 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.621 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9442 (secure 0 0 0) 0x7f712c01a940 con 0x7f713c0a0290 2024-06-11T19:09:39.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7138ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:39.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7138ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:39.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c0a0290 msgr2=0x7f713c13baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7138ff9640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c0a0290 0x7f713c13baf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f712c000f60 tx=0x7f712c000ed0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7138ff9640 1 -- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c0a0290 msgr2=0x7f713c13baf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:39.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7138ff9640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f713c0a0290 0x7f713c13baf0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.622 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:39.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:39.622 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:39.622 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:39.622 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 8, 2024-06-11T19:09:39.622 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 29821, 2024-06-11T19:09:39.622 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:39.622 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:39.622 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:39.622 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:39.622 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 24, 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.623 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.624 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:39.625 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:39.626 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:39.626 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:39.626 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:39.626 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:39.626 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:39.626 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:39.626 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:39.626 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:39.626 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:39.626 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:39.626 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.627 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:39.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.645 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.645 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.645 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.645 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:39.645 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 10 librados: watch_flush enter 2024-06-11T19:09:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 10 librados: watch_flush exit 2024-06-11T19:09:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 20 client.4436.objecter shutdown clearing up homeless session... 2024-06-11T19:09:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 10 client.4436.objecter successfully canceled tick 2024-06-11T19:09:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 1 -- 172.21.15.98:0/3063793030 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f712803fdb0 msgr2=0x7f7128042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f712803fdb0 0x7f7128042270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7134035000 tx=0x7f7134030000 comp rx=0 tx=0).stop 2024-06-11T19:09:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 10 monclient: shutdown 2024-06-11T19:09:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 1 -- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c0a4920 msgr2=0x7f713c136950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 1 --2- 172.21.15.98:0/3063793030 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f713c0a4920 0x7f713c136950 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f713000e8b0 tx=0x7f713000ed70 comp rx=0 tx=0).stop 2024-06-11T19:09:39.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7140939640 1 -- 172.21.15.98:0/3063793030 reap_dead start 2024-06-11T19:09:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f7140939640 1 -- 172.21.15.98:0/3063793030 reap_dead start 2024-06-11T19:09:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 1 -- 172.21.15.98:0/3063793030 shutdown_connections 2024-06-11T19:09:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 1 -- 172.21.15.98:0/3063793030 >> 172.21.15.98:0/3063793030 conn(0x7f713c0ab850 msgr2=0x7f713c0a9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 1 -- 172.21.15.98:0/3063793030 shutdown_connections 2024-06-11T19:09:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 1 -- 172.21.15.98:0/3063793030 wait complete. 2024-06-11T19:09:39.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.599+0000 7f714239c640 1 librados: shutdown 2024-06-11T19:09:39.648 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:09:39.648 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:09:39.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:09:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.750+0000 7f30682d8640 10 monclient: get_monmap_and_config 2024-06-11T19:09:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.750+0000 7f30682d8640 10 monclient: build_initial_monmap 2024-06-11T19:09:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.750+0000 7f30682d8640 10 monclient: monmap: 2024-06-11T19:09:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:39.751972+0000 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:39.751972+0000 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:39.752 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:39.752 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:39.752 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.751+0000 7f30682d8640 1 Processor -- start 2024-06-11T19:09:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.751+0000 7f30682d8640 1 -- start start 2024-06-11T19:09:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.751+0000 7f30682d8640 10 monclient: init 2024-06-11T19:09:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 10 monclient: _add_conns ranks=[2,9,4,1,0,8,3,10,6,7,5] 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30600a2f40 0x7f30600a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 10 monclient(hunting): picked mon.noname-b con 0x7f30600a2f40 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f30600a4df0 0x7f30600a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 10 monclient(hunting): picked mon.noname-a con 0x7f30600a4df0 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f30601366d0 0x7f3060136b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 10 monclient(hunting): picked mon.noname-c con 0x7f30601366d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f30600a5710 con 0x7f30600a4df0 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f30600a3320 con 0x7f30600a2f40 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f30600a2460 con 0x7f30601366d0 2024-06-11T19:09:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f30682d8640 10 monclient(hunting): authenticate will time out at 789.559280s 2024-06-11T19:09:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30600a2f40 0x7f30600a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3065873640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f30600a4df0 0x7f30600a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066875640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f30601366d0 0x7f3060136b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30600a2f40 0x7f30600a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48062/0 (socket says 172.21.15.98:48062) 2024-06-11T19:09:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 1 -- 172.21.15.98:0/2085070754 learned_addr learned my addr 172.21.15.98:0/2085070754 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 10 monclient(hunting): get_auth_request con 0x7f30600a2f40 auth_method 0 2024-06-11T19:09:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3065873640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f30600a4df0 0x7f30600a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45844/0 (socket says 172.21.15.98:45844) 2024-06-11T19:09:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3065873640 10 monclient(hunting): get_auth_request con 0x7f30600a4df0 auth_method 0 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3065873640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3065873640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3065873640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066875640 10 monclient(hunting): get_auth_request con 0x7f30601366d0 auth_method 0 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066875640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066875640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066875640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3065873640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3065873640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3065873640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066875640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066875640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066875640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 10 monclient(hunting): handle_auth_done global_id 4421 payload 274 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 1 -- 172.21.15.98:0/2085070754 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f30601366d0 msgr2=0x7f3060136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 1 --2- 172.21.15.98:0/2085070754 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f30601366d0 0x7f3060136b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 1 -- 172.21.15.98:0/2085070754 >> v2:172.21.15.98:3300/0 conn(0x7f30600a4df0 msgr2=0x7f30600a51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 1 --2- 172.21.15.98:0/2085070754 >> v2:172.21.15.98:3300/0 conn(0x7f30600a4df0 0x7f30600a51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 10 monclient: _finish_hunting 0 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 1 monclient: found mon.noname-b 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3066074640 1 -- 172.21.15.98:0/2085070754 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30600a0ed0 con 0x7f30600a2f40 2024-06-11T19:09:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.752+0000 7f3065873640 1 --2- 172.21.15.98:0/2085070754 >> v2:172.21.15.98:3300/0 conn(0x7f30600a4df0 0x7f30600a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3066074640 1 --2- 172.21.15.98:0/2085070754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30600a2f40 0x7f30600a48b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f305c0098a0 tx=0x7f305c02b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c0d877b483f23b1 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 1 -- 172.21.15.98:0/2085070754 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f305c038070 con 0x7f30600a2f40 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 10 monclient: dump: 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.766 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 10 monclient: _finish_auth 0 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 10 monclient: _check_auth_tickets 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 1 -- 172.21.15.98:0/2085070754 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f305c031990 con 0x7f30600a2f40 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f30682d8640 5 monclient: authenticate success, global_id 4421 2024-06-11T19:09:39.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f30682d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f30682d8640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 1 -- 172.21.15.98:0/2085070754 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f305c0323b0 con 0x7f30600a2f40 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f30682d8640 4 set_mon_vals no callback set 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.753+0000 7f3065072640 10 monclient: dump: 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 10 monclient: shutdown 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 1 -- 172.21.15.98:0/2085070754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30600a2f40 msgr2=0x7f30600a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 1 --2- 172.21.15.98:0/2085070754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30600a2f40 0x7f30600a48b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f305c0098a0 tx=0x7f305c02b7c0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 1 -- 172.21.15.98:0/2085070754 shutdown_connections 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 1 --2- 172.21.15.98:0/2085070754 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f30601366d0 0x7f3060136b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 1 --2- 172.21.15.98:0/2085070754 >> v2:172.21.15.98:3300/0 conn(0x7f30600a4df0 0x7f30600a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 1 --2- 172.21.15.98:0/2085070754 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30600a2f40 0x7f30600a48b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 1 -- 172.21.15.98:0/2085070754 >> 172.21.15.98:0/2085070754 conn(0x7f30600a95b0 msgr2=0x7f30600a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 1 -- 172.21.15.98:0/2085070754 shutdown_connections 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 1 -- 172.21.15.98:0/2085070754 wait complete. 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 10 monclient: build_initial_monmap 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.754+0000 7f30682d8640 10 monclient: monmap: 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 1 librados: starting msgr at 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 1 librados: starting objecter 2024-06-11T19:09:39.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 1 Processor -- start 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 1 -- start start 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 1 librados: setting wanted keys 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 1 librados: calling monclient init 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 10 monclient: init 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 10 monclient: _add_conns ranks=[4,7,0,5,6,2,1,8,3] 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f30600a2f40 0x7f30601cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 10 monclient(hunting): picked mon.noname-c con 0x7f30600a2f40 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f30600a4df0 0x7f30601cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 10 monclient(hunting): picked mon.noname-i con 0x7f30600a4df0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30601366d0 0x7f306006bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 10 monclient(hunting): picked mon.noname-a con 0x7f30601366d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30600a1250 con 0x7f30601366d0 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3060137060 con 0x7f30600a2f40 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f3060126d00 con 0x7f30600a4df0 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f30682d8640 10 monclient(hunting): authenticate will time out at 789.562642s 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f30600a2f40 0x7f30601cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066875640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30601366d0 0x7f306006bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f30600a2f40 0x7f30601cd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:40504/0 (socket says 172.21.15.98:40504) 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 1 -- 172.21.15.98:0/192879070 learned_addr learned my addr 172.21.15.98:0/192879070 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 10 monclient(hunting): get_auth_request con 0x7f30600a2f40 auth_method 0 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066875640 10 monclient(hunting): get_auth_request con 0x7f30601366d0 auth_method 0 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066875640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066875640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066875640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066875640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066875640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066875640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 10 monclient(hunting): handle_auth_done global_id 4441 payload 824 2024-06-11T19:09:39.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 1 -- 172.21.15.98:0/192879070 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f30600a4df0 msgr2=0x7f30601cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f30600a4df0 0x7f30601cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 1 -- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30601366d0 msgr2=0x7f306006bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30601366d0 0x7f306006bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 10 monclient: _finish_hunting 0 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 1 monclient: found mon.noname-c 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066074640 1 -- 172.21.15.98:0/192879070 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3060126740 con 0x7f30600a2f40 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.755+0000 7f3066875640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30601366d0 0x7f306006bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3066074640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f30600a2f40 0x7f30601cd820 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f305c0090b0 tx=0x7f305c0046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 10 client.?.objecter ms_handle_connect 0x7f30600a2f40 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f305c042020 con 0x7f30600a2f40 2024-06-11T19:09:39.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 10 monclient: dump: 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 10 monclient: _finish_auth 0 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f305c003610 con 0x7f30600a2f40 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f30682d8640 5 monclient: authenticate success, global_id 4441 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f30682d8640 10 monclient: _renew_subs 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f30682d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f30682d8640 1 -- 172.21.15.98:0/192879070 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3060126a20 con 0x7f30600a2f40 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f305c038040 con 0x7f30600a2f40 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3056ffd640 10 monclient: dump: 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f30682d8640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f30682d8640 10 monclient: _renew_subs 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f30682d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f30682d8640 1 -- 172.21.15.98:0/192879070 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f30600a7f40 con 0x7f30600a2f40 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f30682d8640 1 librados: init done 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.756+0000 7f3064871640 4 set_mon_vals no callback set 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f30682d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:39.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f30682d8640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006c4a0 0x7f306006c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f30682d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f30682d8640 1 -- 172.21.15.98:0/192879070 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f30600a13d0 con 0x7f306006c4a0 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f305c045020 con 0x7f30600a2f40 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3065873640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006c4a0 0x7f306006c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3065873640 10 monclient: get_auth_request con 0x7f306006c4a0 auth_method 0 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3065873640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3065873640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3065873640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3065873640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3065873640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3065873640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f304403fdb0 0x7f3044042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f305c030070 con 0x7f30600a2f40 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 10 client.4441.objecter ms_dispatch 0x7f3060000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 3 client.4441.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 3 client.4441.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3065873640 10 monclient(hunting): handle_auth_done global_id 4445 payload 824 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3065873640 1 -- 172.21.15.98:0/192879070 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30600a5360 con 0x7f306006c4a0 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 20 client.4441.objecter dump_active .. 0 homeless 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3066875640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f304403fdb0 0x7f3044042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 10 client.4441.objecter ms_handle_connect 0x7f306006c4a0 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3065873640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006c4a0 0x7f306006c880 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f306006d680 tx=0x7f305000ec00 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 10 client.4441.objecter resend_mon_ops 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3066875640 10 monclient: get_auth_request con 0x7f304403fdb0 auth_method 0 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f30500044a0 con 0x7f306006c4a0 2024-06-11T19:09:39.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.757+0000 7f3056ffd640 10 monclient: dump: 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.782 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.758+0000 7f3056ffd640 10 client.4441.objecter ms_handle_connect 0x7f304403fdb0 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.758+0000 7f3066875640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f304403fdb0 0x7f3044042270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3058009fd0 tx=0x7f305800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.758+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f305000ee50 con 0x7f306006c4a0 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.758+0000 7f3056ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.758+0000 7f3056ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.758+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006c4a0 msgr2=0x7f306006c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.758+0000 7f3056ffd640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006c4a0 0x7f306006c880 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f306006d680 tx=0x7f305000ec00 comp rx=0 tx=0).stop 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.758+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006c4a0 msgr2=0x7f306006c880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:39.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.758+0000 7f3056ffd640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006c4a0 0x7f306006c880 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f306006d680 tx=0x7f305000ec00 comp rx=0 tx=0).stop 2024-06-11T19:09:39.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f30682d8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:39.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f30682d8640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006cdc0 0x7f30600705e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f30682d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f30682d8640 1 -- 172.21.15.98:0/192879070 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3060050830 con 0x7f306006cdc0 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f3065873640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006cdc0 0x7f30600705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f3065873640 10 monclient: get_auth_request con 0x7f306006cdc0 auth_method 0 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f3065873640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f3065873640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f3065873640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f3065873640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f3065873640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f3065873640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f3065873640 10 monclient(hunting): handle_auth_done global_id 4454 payload 824 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.764+0000 7f3065873640 1 -- 172.21.15.98:0/192879070 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f30601373f0 con 0x7f306006cdc0 2024-06-11T19:09:39.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 10 client.4441.objecter ms_handle_connect 0x7f306006cdc0 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 10 client.4441.objecter resend_mon_ops 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3065873640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006cdc0 0x7f30600705e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3050018000 tx=0x7f30500046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3050005c60 con 0x7f306006cdc0 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 10 monclient: dump: 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.787 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f3050005c60 con 0x7f306006cdc0 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006cdc0 msgr2=0x7f30600705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006cdc0 0x7f30600705e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3050018000 tx=0x7f30500046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 1 -- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006cdc0 msgr2=0x7f30600705e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3056ffd640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f306006cdc0 0x7f30600705e0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3050018000 tx=0x7f30500046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:39.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:39.789 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:39.789 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:39.789 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 8, 2024-06-11T19:09:39.789 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 30002, 2024-06-11T19:09:39.789 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:39.789 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:39.789 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 24, 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:39.790 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:39.791 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:39.793 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.794 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:39.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:39.798 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:39.809 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.810 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.810 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:39.810 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:39.810 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.810 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.810 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.810 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.810 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.810 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.812 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:39.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 10 librados: watch_flush enter 2024-06-11T19:09:39.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 10 librados: watch_flush exit 2024-06-11T19:09:39.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 20 client.4441.objecter shutdown clearing up homeless session... 2024-06-11T19:09:39.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 10 client.4441.objecter successfully canceled tick 2024-06-11T19:09:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 1 -- 172.21.15.98:0/192879070 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f304403fdb0 msgr2=0x7f3044042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f304403fdb0 0x7f3044042270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3058009fd0 tx=0x7f305800b040 comp rx=0 tx=0).stop 2024-06-11T19:09:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 10 monclient: shutdown 2024-06-11T19:09:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 1 -- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f30600a2f40 msgr2=0x7f30601cd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 1 --2- 172.21.15.98:0/192879070 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f30600a2f40 0x7f30601cd820 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f305c0090b0 tx=0x7f305c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3066875640 1 -- 172.21.15.98:0/192879070 reap_dead start 2024-06-11T19:09:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f3066875640 1 -- 172.21.15.98:0/192879070 reap_dead start 2024-06-11T19:09:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 1 -- 172.21.15.98:0/192879070 shutdown_connections 2024-06-11T19:09:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 1 -- 172.21.15.98:0/192879070 >> 172.21.15.98:0/192879070 conn(0x7f30600a95b0 msgr2=0x7f30601bd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 1 -- 172.21.15.98:0/192879070 shutdown_connections 2024-06-11T19:09:39.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 1 -- 172.21.15.98:0/192879070 wait complete. 2024-06-11T19:09:39.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.765+0000 7f30682d8640 1 librados: shutdown 2024-06-11T19:09:39.815 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:09:39.815 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:09:39.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:09:39.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.914+0000 7fb94de63640 10 monclient: get_monmap_and_config 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.914+0000 7fb94de63640 10 monclient: build_initial_monmap 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.914+0000 7fb94de63640 10 monclient: monmap: 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:39.915599+0000 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:39.915599+0000 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:39.916 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 1 Processor -- start 2024-06-11T19:09:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 1 -- start start 2024-06-11T19:09:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 10 monclient: init 2024-06-11T19:09:39.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:39.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 10 monclient: _add_conns ranks=[4,10,1,9,5,2,6,8,7,3,0] 2024-06-11T19:09:39.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb9480a4920 0x7fb9480a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 10 monclient(hunting): picked mon.noname-c con 0x7fb9480a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:39.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb9480a5240 0x7fb9480a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 10 monclient(hunting): picked mon.noname-f con 0x7fb9480a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:39.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fb9480a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:39.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9480a7140 con 0x7fb9480a2530 2024-06-11T19:09:39.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fb9480a2b50 con 0x7fb9480a5240 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb9480a2e30 con 0x7fb9480a4920 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.915+0000 7fb94de63640 10 monclient(hunting): authenticate will time out at 789.722932s 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb9480a4920 0x7fb9480a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb9480a4920 0x7fb9480a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:40518/0 (socket says 172.21.15.98:40518) 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 1 -- 172.21.15.98:0/2233058786 learned_addr learned my addr 172.21.15.98:0/2233058786 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 10 monclient(hunting): get_auth_request con 0x7fb9480a4920 auth_method 0 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb946ffd640 1 --2- 172.21.15.98:0/2233058786 >> v2:172.21.15.117:3300/0 conn(0x7fb9480a5240 0x7fb9480a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb946ffd640 10 monclient(hunting): get_auth_request con 0x7fb9480a5240 auth_method 0 2024-06-11T19:09:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb946ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb946ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb946ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb946ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb946ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 10 monclient(hunting): handle_auth_done global_id 4450 payload 274 2024-06-11T19:09:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 1 -- 172.21.15.98:0/2233058786 >> v2:172.21.15.117:3300/0 conn(0x7fb9480a5240 msgr2=0x7fb9480a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:39.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 1 --2- 172.21.15.98:0/2233058786 >> v2:172.21.15.117:3300/0 conn(0x7fb9480a5240 0x7fb9480a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 1 -- 172.21.15.98:0/2233058786 >> v1:172.21.15.117:6789/0 conn(0x7fb9480a2530 legacy=0x7fb9480a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 10 monclient: _finish_hunting 0 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 1 monclient: found mon.noname-c 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.916+0000 7fb9477fe640 1 -- 172.21.15.98:0/2233058786 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9480a3110 con 0x7fb9480a4920 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9477fe640 1 --2- 172.21.15.98:0/2233058786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb9480a4920 0x7fb9480a4d00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb938009870 tx=0x7fb93802b860 comp rx=0 tx=0).ready entity=mon.4 client_cookie=4e4c814e4849e31b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 1 -- 172.21.15.98:0/2233058786 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb938031a30 con 0x7fb9480a4920 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 10 monclient: dump: 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.929 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.930 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 1 -- 172.21.15.98:0/2233058786 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb938037070 con 0x7fb9480a4920 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 5 monclient: authenticate success, global_id 4450 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 1 -- 172.21.15.98:0/2233058786 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb938031c30 con 0x7fb9480a4920 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 4 set_mon_vals no callback set 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb9467fc640 10 monclient: dump: 2024-06-11T19:09:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.932 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 10 monclient: shutdown 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 1 -- 172.21.15.98:0/2233058786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb9480a4920 msgr2=0x7fb9480a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 1 --2- 172.21.15.98:0/2233058786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb9480a4920 0x7fb9480a4d00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb938009870 tx=0x7fb93802b860 comp rx=0 tx=0).stop 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 1 -- 172.21.15.98:0/2233058786 shutdown_connections 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 1 --2- 172.21.15.98:0/2233058786 >> v2:172.21.15.117:3300/0 conn(0x7fb9480a5240 0x7fb9480a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 1 --2- 172.21.15.98:0/2233058786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb9480a4920 0x7fb9480a4d00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 1 -- 172.21.15.98:0/2233058786 >> 172.21.15.98:0/2233058786 conn(0x7fb9480ab850 msgr2=0x7fb9480a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 1 -- 172.21.15.98:0/2233058786 shutdown_connections 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.917+0000 7fb94de63640 1 -- 172.21.15.98:0/2233058786 wait complete. 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.918+0000 7fb94de63640 10 monclient: build_initial_monmap 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.918+0000 7fb94de63640 10 monclient: monmap: 2024-06-11T19:09:39.933 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:39.934 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.918+0000 7fb94de63640 1 librados: starting msgr at 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.918+0000 7fb94de63640 1 librados: starting objecter 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.918+0000 7fb94de63640 1 Processor -- start 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.918+0000 7fb94de63640 1 -- start start 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.918+0000 7fb94de63640 1 librados: setting wanted keys 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.918+0000 7fb94de63640 1 librados: calling monclient init 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.918+0000 7fb94de63640 10 monclient: init 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 10 monclient: _add_conns ranks=[0,2,1,6,7,3,8,5,4] 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9480a2530 0x7fb948137670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 10 monclient(hunting): picked mon.noname-a con 0x7fb9480a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:39.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9480a4920 0x7fb948137bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 10 monclient(hunting): picked mon.noname-b con 0x7fb9480a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9480a5240 0x7fb9481231a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 10 monclient(hunting): picked mon.noname-f con 0x7fb9480a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9480a6da0 con 0x7fb9480a2530 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb948138e50 con 0x7fb9480a5240 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb9480a65f0 con 0x7fb9480a4920 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb94de63640 10 monclient(hunting): authenticate will time out at 789.726414s 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9480a2530 0x7fb948137670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb946ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9480a4920 0x7fb948137bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9480a2530 0x7fb948137670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45854/0 (socket says 172.21.15.98:45854) 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 1 -- 172.21.15.98:0/1927377835 learned_addr learned my addr 172.21.15.98:0/1927377835 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 10 monclient(hunting): get_auth_request con 0x7fb9480a2530 auth_method 0 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb946ffd640 10 monclient(hunting): get_auth_request con 0x7fb9480a4920 auth_method 0 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb946ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb946ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb946ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb946ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb947fff640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9480a5240 0x7fb9481231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb946ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb947fff640 10 monclient(hunting): get_auth_request con 0x7fb9480a5240 auth_method 0 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb947fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb947fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 10 monclient(hunting): handle_auth_done global_id 4473 payload 824 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 1 -- 172.21.15.98:0/1927377835 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9480a4920 msgr2=0x7fb948137bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9480a4920 0x7fb948137bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 1 -- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9480a5240 msgr2=0x7fb9481231a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9480a5240 0x7fb9481231a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 10 monclient: _finish_hunting 0 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 1 monclient: found mon.noname-a 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb9477fe640 1 -- 172.21.15.98:0/1927377835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb948126a90 con 0x7fb9480a2530 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.919+0000 7fb946ffd640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb9480a4920 0x7fb948137bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb9477fe640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9480a2530 0x7fb948137670 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb9380090b0 tx=0x7fb9380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 10 client.?.objecter ms_handle_connect 0x7fb9480a2530 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb938041070 con 0x7fb9480a2530 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:39.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 10 monclient: dump: 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.940 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 10 monclient: _finish_auth 0 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb938003e10 con 0x7fb9480a2530 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb94de63640 5 monclient: authenticate success, global_id 4473 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb94de63640 10 monclient: _renew_subs 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb94de63640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb94de63640 1 -- 172.21.15.98:0/1927377835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb948127050 con 0x7fb9480a2530 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb938031a30 con 0x7fb9480a2530 2024-06-11T19:09:39.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 10 monclient: dump: 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.942 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb94de63640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb94de63640 10 monclient: _renew_subs 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb94de63640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb94de63640 1 -- 172.21.15.98:0/1927377835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9481ce800 con 0x7fb9480a2530 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb94de63640 1 librados: init done 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb945ffb640 4 set_mon_vals no callback set 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.920+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fb93804c060 con 0x7fb9480a2530 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb92ffff640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb93403fdb0 0x7fb934042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb94de63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb94de63640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9481236e0 0x7fb948123ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fb9380725f0 con 0x7fb9480a2530 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb94de63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb92ffff640 10 client.4473.objecter ms_dispatch 0x7fb948000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb94de63640 1 -- 172.21.15.98:0/1927377835 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb9481258e0 con 0x7fb9481236e0 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb92ffff640 3 client.4473.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb92ffff640 3 client.4473.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb92ffff640 20 client.4473.objecter dump_active .. 0 homeless 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb946ffd640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb93403fdb0 0x7fb934042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb946ffd640 10 monclient: get_auth_request con 0x7fb93403fdb0 auth_method 0 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb947fff640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9481236e0 0x7fb948123ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb947fff640 10 monclient: get_auth_request con 0x7fb9481236e0 auth_method 0 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb947fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb947fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb947fff640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb947fff640 1 -- 172.21.15.98:0/1927377835 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9480a4d00 con 0x7fb9481236e0 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb92ffff640 10 client.4473.objecter ms_handle_connect 0x7fb93403fdb0 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb946ffd640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb93403fdb0 0x7fb934042270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb9480a5620 tx=0x7fb93c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb92ffff640 10 client.4473.objecter ms_handle_connect 0x7fb9481236e0 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb92ffff640 10 client.4473.objecter resend_mon_ops 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.921+0000 7fb947fff640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9481236e0 0x7fb948123ac0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb948126ca0 tx=0x7fb94000fd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.922+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb9400050d0 con 0x7fb9481236e0 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.922+0000 7fb92ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.922+0000 7fb92ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:09:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.922+0000 7fb92ffff640 10 monclient: dump: 2024-06-11T19:09:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.946 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.947 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.922+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb940018070 con 0x7fb9481236e0 2024-06-11T19:09:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.922+0000 7fb92ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.922+0000 7fb92ffff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.922+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9481236e0 msgr2=0x7fb948123ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.922+0000 7fb92ffff640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9481236e0 0x7fb948123ac0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb948126ca0 tx=0x7fb94000fd60 comp rx=0 tx=0).stop 2024-06-11T19:09:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.922+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9481236e0 msgr2=0x7fb948123ac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.922+0000 7fb92ffff640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9481236e0 0x7fb948123ac0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.928+0000 7fb94de63640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.928+0000 7fb94de63640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb948124000 0x7fb94809e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.928+0000 7fb94de63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.928+0000 7fb94de63640 1 -- 172.21.15.98:0/1927377835 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb948050830 con 0x7fb948124000 2024-06-11T19:09:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb947fff640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb948124000 0x7fb94809e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb947fff640 10 monclient: get_auth_request con 0x7fb948124000 auth_method 0 2024-06-11T19:09:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb947fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb947fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb947fff640 10 monclient(hunting): handle_auth_done global_id 4411 payload 824 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb947fff640 1 -- 172.21.15.98:0/1927377835 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb948124cf0 con 0x7fb948124000 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 10 client.4473.objecter ms_handle_connect 0x7fb948124000 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 10 client.4473.objecter resend_mon_ops 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb947fff640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb948124000 0x7fb94809e9f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb940017000 tx=0x7fb9400014a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb9400055b0 con 0x7fb948124000 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 10 monclient: dump: 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9442 (secure 0 0 0) 0x7fb9400055b0 con 0x7fb948124000 2024-06-11T19:09:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:39.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:39.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb948124000 msgr2=0x7fb94809e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb948124000 0x7fb94809e9f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb940017000 tx=0x7fb9400014a0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 1 -- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb948124000 msgr2=0x7fb94809e9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:39.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.929+0000 7fb92ffff640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb948124000 0x7fb94809e9f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.953 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:39.953 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:39.953 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:39.953 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:39.953 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 8, 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 30147, 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.954 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 24, 2024-06-11T19:09:39.955 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:39.955 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:39.955 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:39.955 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.955 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.955 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.955 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.955 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.955 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.955 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.955 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.955 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:39.957 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:39.958 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:39.960 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:39.962 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:39.963 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.965 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:39.967 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.967 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:39.967 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:39.967 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.967 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.967 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.967 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.968 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.970 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:39.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.974 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:39.975 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:09:39.976 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:39.976 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:39.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:39.976 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:39.976 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:39.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 10 librados: watch_flush enter 2024-06-11T19:09:39.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 10 librados: watch_flush exit 2024-06-11T19:09:39.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 20 client.4473.objecter shutdown clearing up homeless session... 2024-06-11T19:09:39.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 10 client.4473.objecter successfully canceled tick 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 1 -- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb93403fdb0 msgr2=0x7fb934042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb93403fdb0 0x7fb934042270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb9480a5620 tx=0x7fb93c009190 comp rx=0 tx=0).stop 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 10 monclient: shutdown 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 1 -- 172.21.15.98:0/1927377835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9480a2530 msgr2=0x7fb948137670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 1 --2- 172.21.15.98:0/1927377835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9480a2530 0x7fb948137670 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb9380090b0 tx=0x7fb9380046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb947fff640 1 -- 172.21.15.98:0/1927377835 reap_dead start 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb947fff640 1 -- 172.21.15.98:0/1927377835 reap_dead start 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 1 -- 172.21.15.98:0/1927377835 shutdown_connections 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 1 -- 172.21.15.98:0/1927377835 >> 172.21.15.98:0/1927377835 conn(0x7fb9480ab850 msgr2=0x7fb9481be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 1 -- 172.21.15.98:0/1927377835 shutdown_connections 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 1 -- 172.21.15.98:0/1927377835 wait complete. 2024-06-11T19:09:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:39.930+0000 7fb94de63640 1 librados: shutdown 2024-06-11T19:09:39.979 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:09:39.979 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:09:39.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:09:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.079+0000 7f45c87a1640 10 monclient: get_monmap_and_config 2024-06-11T19:09:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.079+0000 7f45c87a1640 10 monclient: build_initial_monmap 2024-06-11T19:09:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.079+0000 7f45c87a1640 10 monclient: monmap: 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:40.080886+0000 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:40.080886+0000 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:40.081 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.080+0000 7f45c87a1640 1 Processor -- start 2024-06-11T19:09:40.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.080+0000 7f45c87a1640 1 -- start start 2024-06-11T19:09:40.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.080+0000 7f45c87a1640 10 monclient: init 2024-06-11T19:09:40.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:40.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 10 monclient: _add_conns ranks=[7,8,0,9,5,2,6,3,4,1,10] 2024-06-11T19:09:40.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45c00a4920 0x7f45c00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 10 monclient(hunting): picked mon.noname-i con 0x7f45c00a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:40.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c00a5240 0x7f45c00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 10 monclient(hunting): picked mon.noname-e con 0x7f45c00a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:40.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45c00a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:40.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45c00a7140 con 0x7f45c00a2530 2024-06-11T19:09:40.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f45c00a2b50 con 0x7f45c00a4920 2024-06-11T19:09:40.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f45c00a2e30 con 0x7f45c00a5240 2024-06-11T19:09:40.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:40.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c87a1640 10 monclient(hunting): authenticate will time out at 789.888162s 2024-06-11T19:09:40.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c00a5240 0x7f45c00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c6d3e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f45c00a2530 0x7f45c00a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53458/0 (socket says 172.21.15.98:53458) 2024-06-11T19:09:40.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c6d3e640 1 -- 172.21.15.98:0/2360862420 learned_addr learned my addr 172.21.15.98:0/2360862420 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:09:40.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 10 monclient(hunting): get_auth_request con 0x7f45c00a5240 auth_method 0 2024-06-11T19:09:40.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c653d640 1 --2- 172.21.15.98:0/2360862420 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45c00a4920 0x7f45c00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c653d640 10 monclient(hunting): get_auth_request con 0x7f45c00a4920 auth_method 0 2024-06-11T19:09:40.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c653d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c653d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c653d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 10 monclient(hunting): handle_auth_done global_id 4463 payload 274 2024-06-11T19:09:40.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 1 -- 172.21.15.98:0/2360862420 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45c00a4920 msgr2=0x7f45c00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:40.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 1 --2- 172.21.15.98:0/2360862420 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45c00a4920 0x7f45c00a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 1 -- 172.21.15.98:0/2360862420 >> v1:172.21.15.98:6789/0 conn(0x7f45c00a2530 legacy=0x7f45c00a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 10 monclient: _finish_hunting 0 2024-06-11T19:09:40.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 1 monclient: found mon.noname-e 2024-06-11T19:09:40.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c5d3c640 1 -- 172.21.15.98:0/2360862420 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c00a3110 con 0x7f45c00a5240 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c553b640 1 -- 172.21.15.98:0/2360862420 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 929477465 0 0) 0x7f45c00a7140 con 0x7f45c00a2530 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.081+0000 7f45c553b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c5d3c640 1 --2- 172.21.15.98:0/2360862420 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c00a5240 0x7f45c00a59a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f45b00097d0 tx=0x7f45b002b770 comp rx=0 tx=0).ready entity=mon.8 client_cookie=ad52ffe092f0975f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 1 -- 172.21.15.98:0/2360862420 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45b0039070 con 0x7f45c00a5240 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 10 monclient: dump: 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.095 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 10 monclient: _finish_auth 0 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 10 monclient: _check_auth_tickets 2024-06-11T19:09:40.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c87a1640 5 monclient: authenticate success, global_id 4463 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c87a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 1 -- 172.21.15.98:0/2360862420 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b0030070 con 0x7f45c00a5240 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 1 -- 172.21.15.98:0/2360862420 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45b0031db0 con 0x7f45c00a5240 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c87a1640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c87a1640 4 set_mon_vals no callback set 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c553b640 10 monclient: dump: 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.097 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.098 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c87a1640 10 monclient: shutdown 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c87a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c87a1640 1 -- 172.21.15.98:0/2360862420 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c00a5240 msgr2=0x7f45c00a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.082+0000 7f45c87a1640 1 --2- 172.21.15.98:0/2360862420 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c00a5240 0x7f45c00a59a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f45b00097d0 tx=0x7f45b002b770 comp rx=0 tx=0).stop 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 1 -- 172.21.15.98:0/2360862420 shutdown_connections 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 1 --2- 172.21.15.98:0/2360862420 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c00a5240 0x7f45c00a59a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 1 --2- 172.21.15.98:0/2360862420 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45c00a4920 0x7f45c00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 1 -- 172.21.15.98:0/2360862420 >> 172.21.15.98:0/2360862420 conn(0x7f45c00ab850 msgr2=0x7f45c00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 1 -- 172.21.15.98:0/2360862420 shutdown_connections 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 1 -- 172.21.15.98:0/2360862420 wait complete. 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 10 monclient: build_initial_monmap 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 10 monclient: monmap: 2024-06-11T19:09:40.099 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:40.100 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 1 librados: starting msgr at 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 1 librados: starting objecter 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 1 Processor -- start 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 1 -- start start 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 1 librados: setting wanted keys 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 1 librados: calling monclient init 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.083+0000 7f45c87a1640 10 monclient: init 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 10 monclient: _add_conns ranks=[3,2,6,8,7,0,4,1,5] 2024-06-11T19:09:40.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c00a2530 0x7f45c01cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 10 monclient(hunting): picked mon.noname-g con 0x7f45c00a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c00a4920 0x7f45c01cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 10 monclient(hunting): picked mon.noname-b con 0x7f45c00a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f45c00a5240 0x7f45c0122da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 10 monclient(hunting): picked mon.noname-d con 0x7f45c00a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f45c00a6da0 con 0x7f45c00a4920 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f45c0138e50 con 0x7f45c00a2530 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f45c00a65f0 con 0x7f45c00a5240 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:40.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c87a1640 10 monclient(hunting): authenticate will time out at 789.891380s 2024-06-11T19:09:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c00a4920 0x7f45c01cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c6d3e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f45c00a5240 0x7f45c0122da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c00a4920 0x7f45c01cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48076/0 (socket says 172.21.15.98:48076) 2024-06-11T19:09:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 1 -- 172.21.15.98:0/4222366545 learned_addr learned my addr 172.21.15.98:0/4222366545 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 10 monclient(hunting): get_auth_request con 0x7f45c00a4920 auth_method 0 2024-06-11T19:09:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c6d3e640 10 monclient(hunting): get_auth_request con 0x7f45c00a5240 auth_method 0 2024-06-11T19:09:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c6d3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c6d3e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c6d3e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c653d640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c00a2530 0x7f45c01cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c6d3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c6d3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c6d3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c653d640 10 monclient(hunting): get_auth_request con 0x7f45c00a2530 auth_method 0 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c653d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c653d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c653d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c653d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c653d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c653d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 10 monclient(hunting): handle_auth_done global_id 4439 payload 824 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 1 -- 172.21.15.98:0/4222366545 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f45c00a5240 msgr2=0x7f45c0122da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f45c00a5240 0x7f45c0122da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 1 -- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c00a2530 msgr2=0x7f45c01cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c00a2530 0x7f45c01cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 10 monclient: _finish_hunting 0 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 1 monclient: found mon.noname-b 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 1 -- 172.21.15.98:0/4222366545 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c0126c40 con 0x7f45c00a4920 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c6d3e640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f45c00a5240 0x7f45c0122da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45c5d3c640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c00a4920 0x7f45c01cdf90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f45b00096f0 tx=0x7f45b00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.084+0000 7f45b6ffd640 10 client.?.objecter ms_handle_connect 0x7f45c00a4920 2024-06-11T19:09:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45b0043020 con 0x7f45c00a4920 2024-06-11T19:09:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 10 monclient: dump: 2024-06-11T19:09:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.106 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 10 monclient: _finish_auth 0 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b0003d40 con 0x7f45c00a4920 2024-06-11T19:09:40.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45c87a1640 5 monclient: authenticate success, global_id 4439 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45c87a1640 10 monclient: _renew_subs 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45c87a1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45c87a1640 1 -- 172.21.15.98:0/4222366545 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c00a8fa0 con 0x7f45c00a4920 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45b0039040 con 0x7f45c00a4920 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45c4d3a640 4 set_mon_vals no callback set 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45b6ffd640 10 monclient: dump: 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45c87a1640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45c87a1640 10 monclient: _renew_subs 2024-06-11T19:09:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45c87a1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45c87a1640 1 -- 172.21.15.98:0/4222366545 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c0126960 con 0x7f45c00a4920 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.085+0000 7f45c87a1640 1 librados: init done 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f45b0046020 con 0x7f45c00a4920 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c87a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c87a1640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c01232e0 0x7f45c01236c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c87a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c87a1640 1 -- 172.21.15.98:0/4222366545 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f45c0050830 con 0x7f45c01232e0 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f45a403fdb0 0x7f45a4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f45b0072c50 con 0x7f45c00a4920 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 10 client.4439.objecter ms_dispatch 0x7f45c0000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 3 client.4439.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 3 client.4439.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:40.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c653d640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c01232e0 0x7f45c01236c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c653d640 10 monclient: get_auth_request con 0x7f45c01232e0 auth_method 0 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c653d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c653d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c653d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 20 client.4439.objecter dump_active .. 0 homeless 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c6d3e640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f45a403fdb0 0x7f45a4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c653d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c653d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c653d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c6d3e640 10 monclient: get_auth_request con 0x7f45a403fdb0 auth_method 0 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c653d640 10 monclient(hunting): handle_auth_done global_id 4386 payload 824 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c653d640 1 -- 172.21.15.98:0/4222366545 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45c00692c0 con 0x7f45c01232e0 2024-06-11T19:09:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 10 client.4439.objecter ms_handle_connect 0x7f45c01232e0 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c653d640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c01232e0 0x7f45c01236c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f45bc004680 tx=0x7f45bc007700 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 10 client.4439.objecter resend_mon_ops 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 10 client.4439.objecter ms_handle_connect 0x7f45a403fdb0 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45c6d3e640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f45a403fdb0 0x7f45a4042270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f45b8007a40 tx=0x7f45b8007990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45bc015890 con 0x7f45c01232e0 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.086+0000 7f45b6ffd640 10 monclient: dump: 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.087+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f45bc01a070 con 0x7f45c01232e0 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.087+0000 7f45b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:40.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.087+0000 7f45b6ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:40.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.087+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c01232e0 msgr2=0x7f45c01236c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.087+0000 7f45b6ffd640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c01232e0 0x7f45c01236c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f45bc004680 tx=0x7f45bc007700 comp rx=0 tx=0).stop 2024-06-11T19:09:40.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.087+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c01232e0 msgr2=0x7f45c01236c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:40.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.087+0000 7f45b6ffd640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c01232e0 0x7f45c01236c0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f45bc004680 tx=0x7f45bc007700 comp rx=0 tx=0).stop 2024-06-11T19:09:40.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.093+0000 7f45c87a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:40.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.093+0000 7f45c87a1640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c0123c00 0x7f45c009e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.093+0000 7f45c87a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.093+0000 7f45c87a1640 1 -- 172.21.15.98:0/4222366545 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f45c006c300 con 0x7f45c0123c00 2024-06-11T19:09:40.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.093+0000 7f45c653d640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c0123c00 0x7f45c009e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.093+0000 7f45c653d640 10 monclient: get_auth_request con 0x7f45c0123c00 auth_method 0 2024-06-11T19:09:40.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.093+0000 7f45c653d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.093+0000 7f45c653d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.093+0000 7f45c653d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.093+0000 7f45c653d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.093+0000 7f45c653d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.093+0000 7f45c653d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45c653d640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2024-06-11T19:09:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45c653d640 1 -- 172.21.15.98:0/4222366545 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f45c0125790 con 0x7f45c0123c00 2024-06-11T19:09:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45c653d640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c0123c00 0x7f45c009e9f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00afd0 tx=0x7f45bc004a40 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 10 client.4439.objecter ms_handle_connect 0x7f45c0123c00 2024-06-11T19:09:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 10 client.4439.objecter resend_mon_ops 2024-06-11T19:09:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45bc015c40 con 0x7f45c0123c00 2024-06-11T19:09:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 10 monclient: dump: 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.118 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.119 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f45bc0157d0 con 0x7f45c0123c00 2024-06-11T19:09:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c0123c00 msgr2=0x7f45c009e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c0123c00 0x7f45c009e9f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00afd0 tx=0x7f45bc004a40 comp rx=0 tx=0).stop 2024-06-11T19:09:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 1 -- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c0123c00 msgr2=0x7f45c009e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:40.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45b6ffd640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45c0123c00 0x7f45c009e9f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.119 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 8, 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 30311, 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:40.120 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 25, 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:40.121 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:40.122 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:40.123 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:40.124 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.126 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:40.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:40.126 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:40.126 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:40.128 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:40.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.130 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.130 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:40.130 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:40.130 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.130 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.130 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:40.132 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:40.133 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:40.133 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:40.133 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.133 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:40.136 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:40.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.140 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.141 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.142 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:40.143 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:40.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.094+0000 7f45c87a1640 10 librados: watch_flush enter 2024-06-11T19:09:40.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 10 librados: watch_flush exit 2024-06-11T19:09:40.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 20 client.4439.objecter shutdown clearing up homeless session... 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 10 client.4439.objecter successfully canceled tick 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 1 -- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f45a403fdb0 msgr2=0x7f45a4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f45a403fdb0 0x7f45a4042270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f45b8007a40 tx=0x7f45b8007990 comp rx=0 tx=0).stop 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 10 monclient: shutdown 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 1 -- 172.21.15.98:0/4222366545 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c00a4920 msgr2=0x7f45c01cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 1 --2- 172.21.15.98:0/4222366545 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c00a4920 0x7f45c01cdf90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f45b00096f0 tx=0x7f45b00046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c6d3e640 1 -- 172.21.15.98:0/4222366545 reap_dead start 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c6d3e640 1 -- 172.21.15.98:0/4222366545 reap_dead start 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 1 -- 172.21.15.98:0/4222366545 shutdown_connections 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 1 -- 172.21.15.98:0/4222366545 >> 172.21.15.98:0/4222366545 conn(0x7f45c00ab850 msgr2=0x7f45c00a9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 1 -- 172.21.15.98:0/4222366545 shutdown_connections 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 1 -- 172.21.15.98:0/4222366545 wait complete. 2024-06-11T19:09:40.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.095+0000 7f45c87a1640 1 librados: shutdown 2024-06-11T19:09:40.146 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:09:40.146 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:09:40.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:09:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.245+0000 7f88cbfe3640 10 monclient: get_monmap_and_config 2024-06-11T19:09:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.245+0000 7f88cbfe3640 10 monclient: build_initial_monmap 2024-06-11T19:09:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.245+0000 7f88cbfe3640 10 monclient: monmap: 2024-06-11T19:09:40.245 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:40.246804+0000 2024-06-11T19:09:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:40.246804+0000 2024-06-11T19:09:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:40.246 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:40.247 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:40.247 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.246+0000 7f88cbfe3640 1 Processor -- start 2024-06-11T19:09:40.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.246+0000 7f88cbfe3640 1 -- start start 2024-06-11T19:09:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.246+0000 7f88cbfe3640 10 monclient: init 2024-06-11T19:09:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.246+0000 7f88cbfe3640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 10 monclient: _add_conns ranks=[5,2,10,0,1,4,9,6,7,8,3] 2024-06-11T19:09:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c40a4920 0x7f88c40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 10 monclient(hunting): picked mon.noname-h con 0x7f88c40a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c40a5240 0x7f88c40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 10 monclient(hunting): picked mon.noname-b con 0x7f88c40a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f88c40a2530 0x7f88c40a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 10 monclient(hunting): picked mon.noname-f con 0x7f88c40a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f88c40a7140 con 0x7f88c40a2530 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f88c40a2b50 con 0x7f88c40a5240 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f88c40a2e30 con 0x7f88c40a4920 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88cbfe3640 10 monclient(hunting): authenticate will time out at 790.054078s 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c40a5240 0x7f88c40a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c40a5240 0x7f88c40a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48084/0 (socket says 172.21.15.98:48084) 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 1 -- 172.21.15.98:0/3443860008 learned_addr learned my addr 172.21.15.98:0/3443860008 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 10 monclient(hunting): get_auth_request con 0x7f88c40a5240 auth_method 0 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c9d7f640 1 --2- 172.21.15.98:0/3443860008 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c40a4920 0x7f88c40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88ca580640 1 --2- 172.21.15.98:0/3443860008 >> v2:172.21.15.117:3300/0 conn(0x7f88c40a2530 0x7f88c40a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c9d7f640 10 monclient(hunting): get_auth_request con 0x7f88c40a4920 auth_method 0 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c9d7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c9d7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c9d7f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88ca580640 10 monclient(hunting): get_auth_request con 0x7f88c40a2530 auth_method 0 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88ca580640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88ca580640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88ca580640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 10 monclient(hunting): handle_auth_done global_id 4448 payload 274 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 1 -- 172.21.15.98:0/3443860008 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c40a4920 msgr2=0x7f88c40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 1 --2- 172.21.15.98:0/3443860008 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c40a4920 0x7f88c40a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 1 -- 172.21.15.98:0/3443860008 >> v2:172.21.15.117:3300/0 conn(0x7f88c40a2530 msgr2=0x7f88c40a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 1 --2- 172.21.15.98:0/3443860008 >> v2:172.21.15.117:3300/0 conn(0x7f88c40a2530 0x7f88c40a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 10 monclient: _finish_hunting 0 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 1 monclient: found mon.noname-b 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.247+0000 7f88c957e640 1 -- 172.21.15.98:0/3443860008 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88c40a3110 con 0x7f88c40a5240 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c957e640 1 --2- 172.21.15.98:0/3443860008 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c40a5240 0x7f88c40a59a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009870 tx=0x7f88b402b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c63817f5007ca7a2 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 1 -- 172.21.15.98:0/3443860008 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88b4038070 con 0x7f88c40a5240 2024-06-11T19:09:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 10 monclient: dump: 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.262 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 10 monclient: _finish_auth 0 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 10 monclient: _check_auth_tickets 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 1 -- 172.21.15.98:0/3443860008 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88b4031990 con 0x7f88c40a5240 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 5 monclient: authenticate success, global_id 4448 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 1 -- 172.21.15.98:0/3443860008 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88b40323b0 con 0x7f88c40a5240 2024-06-11T19:09:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 4 set_mon_vals no callback set 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88c8d7d640 10 monclient: dump: 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 10 monclient: shutdown 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3443860008 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c40a5240 msgr2=0x7f88c40a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 1 --2- 172.21.15.98:0/3443860008 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c40a5240 0x7f88c40a59a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009870 tx=0x7f88b402b7c0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3443860008 shutdown_connections 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 1 --2- 172.21.15.98:0/3443860008 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88c40a5240 0x7f88c40a59a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 1 --2- 172.21.15.98:0/3443860008 >> v2:172.21.15.117:3300/0 conn(0x7f88c40a2530 0x7f88c40a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 1 --2- 172.21.15.98:0/3443860008 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c40a4920 0x7f88c40a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.248+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3443860008 >> 172.21.15.98:0/3443860008 conn(0x7f88c40ab850 msgr2=0x7f88c40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.249+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3443860008 shutdown_connections 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.249+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3443860008 wait complete. 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.249+0000 7f88cbfe3640 10 monclient: build_initial_monmap 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.249+0000 7f88cbfe3640 10 monclient: monmap: 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.249+0000 7f88cbfe3640 1 librados: starting msgr at 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.249+0000 7f88cbfe3640 1 librados: starting objecter 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.249+0000 7f88cbfe3640 1 Processor -- start 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.249+0000 7f88cbfe3640 1 -- start start 2024-06-11T19:09:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.249+0000 7f88cbfe3640 1 librados: setting wanted keys 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.249+0000 7f88cbfe3640 1 librados: calling monclient init 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.249+0000 7f88cbfe3640 10 monclient: init 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 10 monclient: _add_conns ranks=[1,4,7,5,3,0,8,2,6] 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88c40a2530 0x7f88c41cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 10 monclient(hunting): picked mon.noname-f con 0x7f88c40a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88c40a4920 0x7f88c41ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 10 monclient(hunting): picked mon.noname-c con 0x7f88c40a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f88c40a5240 0x7f88c4124fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 10 monclient(hunting): picked mon.noname-i con 0x7f88c40a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88c40a6da0 con 0x7f88c40a2530 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f88c4138e50 con 0x7f88c40a4920 2024-06-11T19:09:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f88c40a6420 con 0x7f88c40a5240 2024-06-11T19:09:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88cbfe3640 10 monclient(hunting): authenticate will time out at 790.057391s 2024-06-11T19:09:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88c40a4920 0x7f88c41ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88c40a4920 0x7f88c41ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:40534/0 (socket says 172.21.15.98:40534) 2024-06-11T19:09:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 1 -- 172.21.15.98:0/3495611956 learned_addr learned my addr 172.21.15.98:0/3495611956 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 10 monclient(hunting): get_auth_request con 0x7f88c40a4920 auth_method 0 2024-06-11T19:09:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c9d7f640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88c40a2530 0x7f88c41cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88ca580640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f88c40a5240 0x7f88c4124fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c9d7f640 10 monclient(hunting): get_auth_request con 0x7f88c40a2530 auth_method 0 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c9d7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c9d7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c9d7f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88ca580640 10 monclient(hunting): get_auth_request con 0x7f88c40a5240 auth_method 0 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88ca580640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88ca580640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88ca580640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 10 monclient(hunting): handle_auth_done global_id 4459 payload 824 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 1 -- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f88c40a5240 msgr2=0x7f88c4124fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f88c40a5240 0x7f88c4124fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 1 -- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88c40a2530 msgr2=0x7f88c41cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88c40a2530 0x7f88c41cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 10 monclient: _finish_hunting 0 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 1 monclient: found mon.noname-c 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 1 -- 172.21.15.98:0/3495611956 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88c40a9010 con 0x7f88c40a4920 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88ca580640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f88c40a5240 0x7f88c4124fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c9d7f640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88c40a2530 0x7f88c41cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88c957e640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88c40a4920 0x7f88c41ce010 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009870 tx=0x7f88b40046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88ba7fc640 10 client.?.objecter ms_handle_connect 0x7f88c40a4920 2024-06-11T19:09:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.250+0000 7f88ba7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88b4041020 con 0x7f88c40a4920 2024-06-11T19:09:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 10 monclient: dump: 2024-06-11T19:09:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88b4039040 con 0x7f88c40a4920 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 5 monclient: authenticate success, global_id 4459 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88b4038070 con 0x7f88c40a4920 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 10 monclient: _renew_subs 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3495611956 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88c41387f0 con 0x7f88c40a4920 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88bbfff640 4 set_mon_vals no callback set 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 10 monclient: dump: 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.275 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 10 monclient: _renew_subs 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3495611956 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f88c41289e0 con 0x7f88c40a4920 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 1 librados: init done 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125520 0x7f88c4125900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3495611956 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f88c4127880 con 0x7f88c4125520 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.251+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f88b4044020 con 0x7f88c40a4920 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88c9d7f640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125520 0x7f88c4125900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88c9d7f640 10 monclient: get_auth_request con 0x7f88c4125520 auth_method 0 2024-06-11T19:09:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88c9d7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88c9d7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88c9d7f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f88a803fdb0 0x7f88a8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88c9d7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88c9d7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f88b4030070 con 0x7f88c40a4920 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 10 client.4459.objecter ms_dispatch 0x7f88c4000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 3 client.4459.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 3 client.4459.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88c9d7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 20 client.4459.objecter dump_active .. 0 homeless 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ca580640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f88a803fdb0 0x7f88a8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ca580640 10 monclient: get_auth_request con 0x7f88a803fdb0 auth_method 0 2024-06-11T19:09:40.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88c9d7f640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88c9d7f640 1 -- 172.21.15.98:0/3495611956 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88c40a6720 con 0x7f88c4125520 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 10 client.4459.objecter ms_handle_connect 0x7f88c4125520 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 10 client.4459.objecter resend_mon_ops 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88c9d7f640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125520 0x7f88c4125900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f88bc00f550 tx=0x7f88bc00fa10 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 10 client.4459.objecter ms_handle_connect 0x7f88a803fdb0 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ca580640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f88a803fdb0 0x7f88a8042270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f88c4126740 tx=0x7f88c000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88bc019020 con 0x7f88c4125520 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.252+0000 7f88ba7fc640 10 monclient: dump: 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.278 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.253+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f88bc014670 con 0x7f88c4125520 2024-06-11T19:09:40.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.253+0000 7f88ba7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:40.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.253+0000 7f88ba7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:40.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.253+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125520 msgr2=0x7f88c4125900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.253+0000 7f88ba7fc640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125520 0x7f88c4125900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f88bc00f550 tx=0x7f88bc00fa10 comp rx=0 tx=0).stop 2024-06-11T19:09:40.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.253+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125520 msgr2=0x7f88c4125900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:40.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.253+0000 7f88ba7fc640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125520 0x7f88c4125900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.259+0000 7f88cbfe3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:40.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.259+0000 7f88cbfe3640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125e40 0x7f88c409f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.259+0000 7f88cbfe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.259+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3495611956 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f88c4050830 con 0x7f88c4125e40 2024-06-11T19:09:40.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.259+0000 7f88c9d7f640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125e40 0x7f88c409f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.259+0000 7f88c9d7f640 10 monclient: get_auth_request con 0x7f88c4125e40 auth_method 0 2024-06-11T19:09:40.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.259+0000 7f88c9d7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.259+0000 7f88c9d7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.259+0000 7f88c9d7f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.259+0000 7f88c9d7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.259+0000 7f88c9d7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.259+0000 7f88c9d7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88c9d7f640 10 monclient(hunting): handle_auth_done global_id 4343 payload 824 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88c9d7f640 1 -- 172.21.15.98:0/3495611956 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f88c406c300 con 0x7f88c4125e40 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 10 client.4459.objecter ms_handle_connect 0x7f88c4125e40 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88c9d7f640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125e40 0x7f88c409f3d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f88bc01a000 tx=0x7f88bc0046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 10 client.4459.objecter resend_mon_ops 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88bc019040 con 0x7f88c4125e40 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 10 monclient: dump: 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.283 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9612 (secure 0 0 0) 0x7f88bc01e420 con 0x7f88c4125e40 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125e40 msgr2=0x7f88c409f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125e40 0x7f88c409f3d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f88bc01a000 tx=0x7f88bc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 1 -- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125e40 msgr2=0x7f88c409f3d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:40.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88ba7fc640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88c4125e40 0x7f88c409f3d0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f88bc01a000 tx=0x7f88bc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.285 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:40.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:40.285 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:40.285 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:40.285 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 8, 2024-06-11T19:09:40.285 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 30454, 2024-06-11T19:09:40.285 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:40.285 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:40.285 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 25, 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:40.286 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:40.287 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:40.288 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:40.289 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:40.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:40.291 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:40.293 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:40.294 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.294 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.294 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.294 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:40.294 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:40.294 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.295 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.297 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.298 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.302 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:40.304 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.304 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:40.304 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:40.304 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.304 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.304 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.305 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:40.305 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:40.305 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:40.305 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:40.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:40.308 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:40.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.309 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.309 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:40.309 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:40.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88cbfe3640 10 librados: watch_flush enter 2024-06-11T19:09:40.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88cbfe3640 10 librados: watch_flush exit 2024-06-11T19:09:40.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88cbfe3640 20 client.4459.objecter shutdown clearing up homeless session... 2024-06-11T19:09:40.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.260+0000 7f88cbfe3640 10 client.4459.objecter successfully canceled tick 2024-06-11T19:09:40.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f88a803fdb0 msgr2=0x7f88a8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88cbfe3640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f88a803fdb0 0x7f88a8042270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f88c4126740 tx=0x7f88c000b040 comp rx=0 tx=0).stop 2024-06-11T19:09:40.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88cbfe3640 10 monclient: shutdown 2024-06-11T19:09:40.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88cbfe3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:40.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3495611956 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88c40a4920 msgr2=0x7f88c41ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88cbfe3640 1 --2- 172.21.15.98:0/3495611956 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88c40a4920 0x7f88c41ce010 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009870 tx=0x7f88b40046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88ca580640 1 -- 172.21.15.98:0/3495611956 reap_dead start 2024-06-11T19:09:40.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88ca580640 1 -- 172.21.15.98:0/3495611956 reap_dead start 2024-06-11T19:09:40.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3495611956 shutdown_connections 2024-06-11T19:09:40.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3495611956 >> 172.21.15.98:0/3495611956 conn(0x7f88c40ab850 msgr2=0x7f88c409c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:40.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3495611956 shutdown_connections 2024-06-11T19:09:40.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88cbfe3640 1 -- 172.21.15.98:0/3495611956 wait complete. 2024-06-11T19:09:40.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.261+0000 7f88cbfe3640 1 librados: shutdown 2024-06-11T19:09:40.312 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:09:40.312 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:09:40.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:09:40.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.412+0000 7fd8e7be3640 10 monclient: get_monmap_and_config 2024-06-11T19:09:40.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.412+0000 7fd8e7be3640 10 monclient: build_initial_monmap 2024-06-11T19:09:40.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.412+0000 7fd8e7be3640 10 monclient: monmap: 2024-06-11T19:09:40.412 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:40.412 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.412 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:40.413654+0000 2024-06-11T19:09:40.412 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:40.413654+0000 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:40.414 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 1 Processor -- start 2024-06-11T19:09:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 1 -- start start 2024-06-11T19:09:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 10 monclient: init 2024-06-11T19:09:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 10 monclient: _add_conns ranks=[2,9,8,7,5,4,10,3,1,0,6] 2024-06-11T19:09:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e00a4920 0x7fd8e00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 10 monclient(hunting): picked mon.noname-b con 0x7fd8e00a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8e00a5240 0x7fd8e00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 10 monclient(hunting): picked mon.noname-a con 0x7fd8e00a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 0x7fd8e00a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 10 monclient(hunting): picked mon.noname-e con 0x7fd8e00a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd8e00a7140 con 0x7fd8e00a5240 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd8e00a2b50 con 0x7fd8e00a4920 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd8e00a2e30 con 0x7fd8e00a2530 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e7be3640 10 monclient(hunting): authenticate will time out at 790.220875s 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e597f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e00a4920 0x7fd8e00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e6180640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 0x7fd8e00a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.413+0000 7fd8e517e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8e00a5240 0x7fd8e00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e597f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e00a4920 0x7fd8e00a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48086/0 (socket says 172.21.15.98:48086) 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 0x7fd8e00a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54640/0 (socket says 172.21.15.98:54640) 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e597f640 1 -- 172.21.15.98:0/607783185 learned_addr learned my addr 172.21.15.98:0/607783185 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e517e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8e00a5240 0x7fd8e00a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45864/0 (socket says 172.21.15.98:45864) 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e517e640 10 monclient(hunting): get_auth_request con 0x7fd8e00a5240 auth_method 0 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e517e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e517e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e517e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 10 monclient(hunting): get_auth_request con 0x7fd8e00a2530 auth_method 0 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e597f640 10 monclient(hunting): get_auth_request con 0x7fd8e00a4920 auth_method 0 2024-06-11T19:09:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e597f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e597f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e597f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e517e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e517e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e517e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e597f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e597f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e597f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 10 monclient(hunting): handle_auth_done global_id 4472 payload 274 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 1 -- 172.21.15.98:0/607783185 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e00a4920 msgr2=0x7fd8e00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 1 --2- 172.21.15.98:0/607783185 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e00a4920 0x7fd8e00a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 1 -- 172.21.15.98:0/607783185 >> v2:172.21.15.98:3300/0 conn(0x7fd8e00a5240 msgr2=0x7fd8e00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 1 --2- 172.21.15.98:0/607783185 >> v2:172.21.15.98:3300/0 conn(0x7fd8e00a5240 0x7fd8e00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 10 monclient: _finish_hunting 0 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 1 monclient: found mon.noname-e 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e6180640 1 -- 172.21.15.98:0/607783185 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e00a3110 con 0x7fd8e00a2530 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e517e640 1 --2- 172.21.15.98:0/607783185 >> v2:172.21.15.98:3300/0 conn(0x7fd8e00a5240 0x7fd8e00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.414+0000 7fd8e597f640 1 --2- 172.21.15.98:0/607783185 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e00a4920 0x7fd8e00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e6180640 1 --2- 172.21.15.98:0/607783185 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 0x7fd8e00a5ee0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009d90 tx=0x7fd8d802bce0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=fa80c1824c418a50 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 1 -- 172.21.15.98:0/607783185 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8d8039070 con 0x7fd8e00a2530 2024-06-11T19:09:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 10 monclient: dump: 2024-06-11T19:09:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 10 monclient: _finish_auth 0 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 10 monclient: _check_auth_tickets 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 1 -- 172.21.15.98:0/607783185 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8d802cd00 con 0x7fd8e00a2530 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 5 monclient: authenticate success, global_id 4472 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 1 -- 172.21.15.98:0/607783185 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8d80327b0 con 0x7fd8e00a2530 2024-06-11T19:09:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 4 set_mon_vals no callback set 2024-06-11T19:09:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e497d640 10 monclient: dump: 2024-06-11T19:09:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 10 monclient: shutdown 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 1 -- 172.21.15.98:0/607783185 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 msgr2=0x7fd8e00a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 1 --2- 172.21.15.98:0/607783185 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 0x7fd8e00a5ee0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009d90 tx=0x7fd8d802bce0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 1 -- 172.21.15.98:0/607783185 shutdown_connections 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 1 --2- 172.21.15.98:0/607783185 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 0x7fd8e00a5ee0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 1 --2- 172.21.15.98:0/607783185 >> v2:172.21.15.98:3300/0 conn(0x7fd8e00a5240 0x7fd8e00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 1 --2- 172.21.15.98:0/607783185 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd8e00a4920 0x7fd8e00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 1 -- 172.21.15.98:0/607783185 >> 172.21.15.98:0/607783185 conn(0x7fd8e00ab850 msgr2=0x7fd8e00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 1 -- 172.21.15.98:0/607783185 shutdown_connections 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.415+0000 7fd8e7be3640 1 -- 172.21.15.98:0/607783185 wait complete. 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.416+0000 7fd8e7be3640 10 monclient: build_initial_monmap 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.416+0000 7fd8e7be3640 10 monclient: monmap: 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.416+0000 7fd8e7be3640 1 librados: starting msgr at 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.416+0000 7fd8e7be3640 1 librados: starting objecter 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.416+0000 7fd8e7be3640 1 Processor -- start 2024-06-11T19:09:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.416+0000 7fd8e7be3640 1 -- start start 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.416+0000 7fd8e7be3640 1 librados: setting wanted keys 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.416+0000 7fd8e7be3640 1 librados: calling monclient init 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.416+0000 7fd8e7be3640 10 monclient: init 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient: _add_conns ranks=[8,5,1,2,3,6,4,0,7] 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 0x7fd8e01cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient(hunting): picked mon.noname-e con 0x7fd8e00a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd8e00a4920 0x7fd8e01ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient(hunting): picked mon.noname-h con 0x7fd8e00a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8e00a5240 0x7fd8e0122ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient(hunting): picked mon.noname-f con 0x7fd8e00a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e00a6da0 con 0x7fd8e00a5240 2024-06-11T19:09:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd8e0138e50 con 0x7fd8e00a4920 2024-06-11T19:09:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd8e00a6420 con 0x7fd8e00a2530 2024-06-11T19:09:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient(hunting): authenticate will time out at 790.224172s 2024-06-11T19:09:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 0x7fd8e01cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 0x7fd8e01cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54648/0 (socket says 172.21.15.98:54648) 2024-06-11T19:09:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 1 -- 172.21.15.98:0/3850473734 learned_addr learned my addr 172.21.15.98:0/3850473734 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 10 monclient(hunting): get_auth_request con 0x7fd8e00a2530 auth_method 0 2024-06-11T19:09:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 10 monclient(hunting): handle_auth_done global_id 4481 payload 824 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 1 -- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd8e00a4920 msgr2=0x7fd8e01ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd8e00a4920 0x7fd8e01ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 1 -- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8e00a5240 msgr2=0x7fd8e0122ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8e00a5240 0x7fd8e0122ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 10 monclient: _finish_hunting 0 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 1 monclient: found mon.noname-e 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 1 -- 172.21.15.98:0/3850473734 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e0126af0 con 0x7fd8e00a2530 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e597f640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 0x7fd8e01cdad0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00e9d0 tx=0x7fd8dc00ee90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 10 client.?.objecter ms_handle_connect 0x7fd8e00a2530 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8dc017070 con 0x7fd8e00a2530 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 10 monclient: dump: 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8dc013470 con 0x7fd8e00a2530 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 5 monclient: authenticate success, global_id 4481 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8dc013a90 con 0x7fd8e00a2530 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d7fff640 4 set_mon_vals no callback set 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8d67fc640 10 monclient: dump: 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient: _renew_subs 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 1 -- 172.21.15.98:0/3850473734 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e0126dd0 con 0x7fd8e00a2530 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient: _renew_subs 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 1 -- 172.21.15.98:0/3850473734 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e0126810 con 0x7fd8e00a2530 2024-06-11T19:09:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.417+0000 7fd8e7be3640 1 librados: init done 2024-06-11T19:09:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.418+0000 7fd8e7be3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.418+0000 7fd8e7be3640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e01233e0 0x7fd8e01237c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.418+0000 7fd8e7be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.418+0000 7fd8e7be3640 1 -- 172.21.15.98:0/3850473734 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd8e0050830 con 0x7fd8e01233e0 2024-06-11T19:09:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.418+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd8dc01a070 con 0x7fd8e00a2530 2024-06-11T19:09:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.418+0000 7fd8d67fc640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd8c403fde0 0x7fd8c40422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.418+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fd8dc00f040 con 0x7fd8e00a2530 2024-06-11T19:09:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.418+0000 7fd8d67fc640 10 client.4481.objecter ms_dispatch 0x7fd8e0000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.418+0000 7fd8d67fc640 3 client.4481.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.418+0000 7fd8d67fc640 3 client.4481.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e517e640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e01233e0 0x7fd8e01237c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8d67fc640 20 client.4481.objecter dump_active .. 0 homeless 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e517e640 10 monclient: get_auth_request con 0x7fd8e01233e0 auth_method 0 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e517e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e517e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e517e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e517e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e517e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e6180640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd8c403fde0 0x7fd8c40422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e517e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e6180640 10 monclient: get_auth_request con 0x7fd8c403fde0 auth_method 0 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e517e640 10 monclient(hunting): handle_auth_done global_id 4435 payload 824 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e517e640 1 -- 172.21.15.98:0/3850473734 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd8e00a9130 con 0x7fd8e01233e0 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8d67fc640 10 client.4481.objecter ms_handle_connect 0x7fd8c403fde0 2024-06-11T19:09:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e6180640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd8c403fde0 0x7fd8c40422a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8036000 tx=0x7fd8d802e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8d67fc640 10 client.4481.objecter ms_handle_connect 0x7fd8e01233e0 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8d67fc640 10 client.4481.objecter resend_mon_ops 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8e517e640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e01233e0 0x7fd8e01237c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0124910 tx=0x7fd8d000ec00 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8d0017070 con 0x7fd8e01233e0 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8d67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.419+0000 7fd8d67fc640 10 monclient: dump: 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.420+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd8d000c6f0 con 0x7fd8e01233e0 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.420+0000 7fd8d67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.420+0000 7fd8d67fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.420+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e01233e0 msgr2=0x7fd8e01237c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.420+0000 7fd8d67fc640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e01233e0 0x7fd8e01237c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0124910 tx=0x7fd8d000ec00 comp rx=0 tx=0).stop 2024-06-11T19:09:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.420+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e01233e0 msgr2=0x7fd8e01237c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.420+0000 7fd8d67fc640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e01233e0 0x7fd8e01237c0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0124910 tx=0x7fd8d000ec00 comp rx=0 tx=0).stop 2024-06-11T19:09:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.426+0000 7fd8e7be3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.426+0000 7fd8e7be3640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e0123d00 0x7fd8e009e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.426+0000 7fd8e7be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.426+0000 7fd8e7be3640 1 -- 172.21.15.98:0/3850473734 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd8e0069740 con 0x7fd8e0123d00 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.426+0000 7fd8e517e640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e0123d00 0x7fd8e009e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.426+0000 7fd8e517e640 10 monclient: get_auth_request con 0x7fd8e0123d00 auth_method 0 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.426+0000 7fd8e517e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.426+0000 7fd8e517e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.426+0000 7fd8e517e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8e517e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8e517e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8e517e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8e517e640 10 monclient(hunting): handle_auth_done global_id 4444 payload 824 2024-06-11T19:09:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8e517e640 1 -- 172.21.15.98:0/3850473734 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd8e00a6750 con 0x7fd8e0123d00 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 10 client.4481.objecter ms_handle_connect 0x7fd8e0123d00 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 10 client.4481.objecter resend_mon_ops 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8e517e640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e0123d00 0x7fd8e009e9f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0019000 tx=0x7fd8d00046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8d0017070 con 0x7fd8e0123d00 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 10 monclient: dump: 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fd8d0006250 con 0x7fd8e0123d00 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e0123d00 msgr2=0x7fd8e009e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e0123d00 0x7fd8e009e9f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0019000 tx=0x7fd8d00046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 1 -- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e0123d00 msgr2=0x7fd8e009e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:40.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.427+0000 7fd8d67fc640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8e0123d00 0x7fd8e009e9f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.450 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:40.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:40.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:40.450 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:40.450 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 8, 2024-06-11T19:09:40.450 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 30643, 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 25, 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:40.451 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.453 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:40.454 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:40.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:40.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:40.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.462 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:40.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.467 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:40.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:40.471 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:40.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.472 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:40.472 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:40.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:40.472 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:40.472 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:40.472 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:40.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:40.472 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:40.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:40.472 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:40.472 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:40.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 10 librados: watch_flush enter 2024-06-11T19:09:40.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 10 librados: watch_flush exit 2024-06-11T19:09:40.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 20 client.4481.objecter shutdown clearing up homeless session... 2024-06-11T19:09:40.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 10 client.4481.objecter successfully canceled tick 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 1 -- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd8c403fde0 msgr2=0x7fd8c40422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd8c403fde0 0x7fd8c40422a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8036000 tx=0x7fd8d802e000 comp rx=0 tx=0).stop 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 10 monclient: shutdown 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 1 -- 172.21.15.98:0/3850473734 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 msgr2=0x7fd8e01cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 1 --2- 172.21.15.98:0/3850473734 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8e00a2530 0x7fd8e01cdad0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00e9d0 tx=0x7fd8dc00ee90 comp rx=0 tx=0).stop 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e6180640 1 -- 172.21.15.98:0/3850473734 reap_dead start 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e6180640 1 -- 172.21.15.98:0/3850473734 reap_dead start 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 1 -- 172.21.15.98:0/3850473734 shutdown_connections 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 1 -- 172.21.15.98:0/3850473734 >> 172.21.15.98:0/3850473734 conn(0x7fd8e00ab850 msgr2=0x7fd8e01be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 1 -- 172.21.15.98:0/3850473734 shutdown_connections 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 1 -- 172.21.15.98:0/3850473734 wait complete. 2024-06-11T19:09:40.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.428+0000 7fd8e7be3640 1 librados: shutdown 2024-06-11T19:09:40.476 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:09:40.476 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['d'] 2024-06-11T19:09:40.476 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-06-11T19:09:40.476 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-06-11T19:09:40.476 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d store 2024-06-11T19:09:40.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d sync_force --yes-i-really-mean-it 2024-06-11T19:09:40.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.576+0000 7f0bf6e8b640 10 monclient: get_monmap_and_config 2024-06-11T19:09:40.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.576+0000 7f0bf6e8b640 10 monclient: build_initial_monmap 2024-06-11T19:09:40.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.576+0000 7f0bf6e8b640 10 monclient: monmap: 2024-06-11T19:09:40.576 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:40.576 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.576 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:40.577687+0000 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:40.577687+0000 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:40.577 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:40.578 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 1 Processor -- start 2024-06-11T19:09:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 1 -- start start 2024-06-11T19:09:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 10 monclient: init 2024-06-11T19:09:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 10 monclient: _add_conns ranks=[10,8,3,1,6,4,5,0,7,9,2] 2024-06-11T19:09:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0bf00a4920 0x7f0bf00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 10 monclient(hunting): picked mon.noname-f con 0x7f0bf00a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0bf00a5240 0x7f0bf00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 10 monclient(hunting): picked mon.noname-e con 0x7f0bf00a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bf00a2530 0x7f0bf00a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 10 monclient(hunting): picked mon.noname-g con 0x7f0bf00a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f0bf00a7140 con 0x7f0bf00a4920 2024-06-11T19:09:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf00a2b50 con 0x7f0bf00a2530 2024-06-11T19:09:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0bf00a2e30 con 0x7f0bf00a5240 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.577+0000 7f0bf6e8b640 10 monclient(hunting): authenticate will time out at 790.384917s 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0bf00a5240 0x7f0bf00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0bf00a5240 0x7f0bf00a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54660/0 (socket says 172.21.15.98:54660) 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 1 -- 172.21.15.98:0/3167878972 learned_addr learned my addr 172.21.15.98:0/3167878972 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 10 monclient(hunting): get_auth_request con 0x7f0bf00a5240 auth_method 0 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf4c27640 1 --2- 172.21.15.98:0/3167878972 >> v2:172.21.15.117:3300/0 conn(0x7f0bf00a4920 0x7f0bf00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf5428640 1 --2- 172.21.15.98:0/3167878972 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bf00a2530 0x7f0bf00a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf4c27640 10 monclient(hunting): get_auth_request con 0x7f0bf00a4920 auth_method 0 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf4c27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf4c27640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf4c27640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf5428640 10 monclient(hunting): get_auth_request con 0x7f0bf00a2530 auth_method 0 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf5428640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf5428640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf5428640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf4c27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf4c27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf4c27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf5428640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf5428640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0bf5428640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 10 monclient(hunting): handle_auth_done global_id 4490 payload 274 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 1 -- 172.21.15.98:0/3167878972 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bf00a2530 msgr2=0x7f0bf00a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 1 --2- 172.21.15.98:0/3167878972 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bf00a2530 0x7f0bf00a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 1 -- 172.21.15.98:0/3167878972 >> v2:172.21.15.117:3300/0 conn(0x7f0bf00a4920 msgr2=0x7f0bf00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 1 --2- 172.21.15.98:0/3167878972 >> v2:172.21.15.117:3300/0 conn(0x7f0bf00a4920 0x7f0bf00a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 10 monclient: _finish_hunting 0 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 1 monclient: found mon.noname-e 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.578+0000 7f0beffff640 1 -- 172.21.15.98:0/3167878972 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf00a3110 con 0x7f0bf00a5240 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0beffff640 1 --2- 172.21.15.98:0/3167878972 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0bf00a5240 0x7f0bf00a59a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc009870 tx=0x7f0bdc02b7c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=11371b76acfbe710 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 1 -- 172.21.15.98:0/3167878972 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0bdc038070 con 0x7f0bf00a5240 2024-06-11T19:09:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 10 monclient: dump: 2024-06-11T19:09:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 1 -- 172.21.15.98:0/3167878972 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bdc031990 con 0x7f0bf00a5240 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 5 monclient: authenticate success, global_id 4490 2024-06-11T19:09:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 1 -- 172.21.15.98:0/3167878972 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0bdc0323b0 con 0x7f0bf00a5240 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 4 set_mon_vals no callback set 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bef7fe640 10 monclient: dump: 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.595 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 10 monclient: shutdown 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/3167878972 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0bf00a5240 msgr2=0x7f0bf00a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 1 --2- 172.21.15.98:0/3167878972 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0bf00a5240 0x7f0bf00a59a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc009870 tx=0x7f0bdc02b7c0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/3167878972 shutdown_connections 2024-06-11T19:09:40.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 1 --2- 172.21.15.98:0/3167878972 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bf00a2530 0x7f0bf00a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 1 --2- 172.21.15.98:0/3167878972 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0bf00a5240 0x7f0bf00a59a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 1 --2- 172.21.15.98:0/3167878972 >> v2:172.21.15.117:3300/0 conn(0x7f0bf00a4920 0x7f0bf00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/3167878972 >> 172.21.15.98:0/3167878972 conn(0x7f0bf00ab850 msgr2=0x7f0bf00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/3167878972 shutdown_connections 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.579+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/3167878972 wait complete. 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.580+0000 7f0bf6e8b640 10 monclient: build_initial_monmap 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.580+0000 7f0bf6e8b640 10 monclient: monmap: 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:40.597 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.580+0000 7f0bf6e8b640 1 librados: starting msgr at 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.580+0000 7f0bf6e8b640 1 librados: starting objecter 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.580+0000 7f0bf6e8b640 1 Processor -- start 2024-06-11T19:09:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.580+0000 7f0bf6e8b640 1 -- start start 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.580+0000 7f0bf6e8b640 1 librados: setting wanted keys 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.580+0000 7f0bf6e8b640 1 librados: calling monclient init 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.580+0000 7f0bf6e8b640 10 monclient: init 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 10 monclient: _add_conns ranks=[5,3,7,8,6,4,0,1,2] 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0bf00a2530 0x7f0bf0137700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 10 monclient(hunting): picked mon.noname-h con 0x7f0bf00a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bf00a4920 0x7f0bf0137c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 10 monclient(hunting): picked mon.noname-g con 0x7f0bf00a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0bf00a5240 0x7f0bf0123160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 10 monclient(hunting): picked mon.noname-i con 0x7f0bf00a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf00a6da0 con 0x7f0bf00a4920 2024-06-11T19:09:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f0bf0138e50 con 0x7f0bf00a2530 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f0bf00a6420 con 0x7f0bf00a5240 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf6e8b640 10 monclient(hunting): authenticate will time out at 790.388455s 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf5428640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0bf00a5240 0x7f0bf0123160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0beffff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bf00a4920 0x7f0bf0137c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf4c27640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0bf00a2530 0x7f0bf0137700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0beffff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bf00a4920 0x7f0bf0137c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:53890/0 (socket says 172.21.15.98:53890) 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf5428640 10 monclient(hunting): get_auth_request con 0x7f0bf00a5240 auth_method 0 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf5428640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf5428640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0beffff640 1 -- 172.21.15.98:0/2102066123 learned_addr learned my addr 172.21.15.98:0/2102066123 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf5428640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0beffff640 10 monclient(hunting): get_auth_request con 0x7f0bf00a4920 auth_method 0 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0beffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf4c27640 10 monclient(hunting): get_auth_request con 0x7f0bf00a2530 auth_method 0 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf4c27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf4c27640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf4c27640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf4c27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf4c27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf4c27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf5428640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf5428640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.581+0000 7f0bf5428640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf4c27640 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf4c27640 1 -- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0bf00a5240 msgr2=0x7f0bf0123160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf4c27640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0bf00a5240 0x7f0bf0123160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf4c27640 1 -- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bf00a4920 msgr2=0x7f0bf0137c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf4c27640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bf00a4920 0x7f0bf0137c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf4c27640 10 monclient: _finish_hunting 0 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf4c27640 1 monclient: found mon.noname-h 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf4c27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf4c27640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:09:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf4c27640 1 -- 172.21.15.98:0/2102066123 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf0126b60 con 0x7f0bf00a2530 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0beffff640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bf00a4920 0x7f0bf0137c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf4c27640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0bf00a2530 0x7f0bf0137700 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0be400e880 tx=0x7f0be400ed40 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 10 client.?.objecter ms_handle_connect 0x7f0bf00a2530 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0be4017070 con 0x7f0bf00a2530 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 10 monclient: dump: 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 10 monclient: _finish_auth 0 2024-06-11T19:09:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0be400cdb0 con 0x7f0bf00a2530 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf6e8b640 5 monclient: authenticate success, global_id 4352 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf6e8b640 10 monclient: _renew_subs 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf6e8b640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/2102066123 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf00a90a0 con 0x7f0bf00a2530 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0be40138e0 con 0x7f0bf00a2530 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bed7fa640 10 monclient: dump: 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0beeffd640 4 set_mon_vals no callback set 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf6e8b640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf6e8b640 10 monclient: _renew_subs 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf6e8b640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/2102066123 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf0124e30 con 0x7f0bf00a2530 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.582+0000 7f0bf6e8b640 1 librados: init done 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bf6e8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bf6e8b640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0126da0 0x7f0bf01236a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bf6e8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/2102066123 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0bf0050830 con 0x7f0bf0126da0 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0beffff640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0126da0 0x7f0bf01236a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0beffff640 10 monclient: get_auth_request con 0x7f0bf0126da0 auth_method 0 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0beffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 <== mon.5 v2:172.21.15.117:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f0be401a070 con 0x7f0bf00a2530 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0beffff640 10 monclient(hunting): handle_auth_done global_id 4389 payload 824 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0beffff640 1 -- 172.21.15.98:0/2102066123 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0bf00a6750 con 0x7f0bf0126da0 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bed7fa640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0bd403fdb0 0x7f0bd4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 <== mon.5 v2:172.21.15.117:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f0be400f040 con 0x7f0bf00a2530 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0beffff640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0126da0 0x7f0bf01236a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc036040 tx=0x7f0bdc0046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bed7fa640 10 client.4352.objecter ms_dispatch 0x7f0bf0000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bed7fa640 3 client.4352.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bed7fa640 3 client.4352.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bf5428640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0bd403fdb0 0x7f0bd4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bf5428640 10 monclient: get_auth_request con 0x7f0bd403fdb0 auth_method 0 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bed7fa640 20 client.4352.objecter dump_active .. 0 homeless 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bed7fa640 10 client.4352.objecter ms_handle_connect 0x7f0bf0126da0 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.583+0000 7f0bed7fa640 10 client.4352.objecter resend_mon_ops 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0bdc044070 con 0x7f0bf0126da0 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bed7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bed7fa640 10 monclient: dump: 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.609 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.610 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0bdc039040 con 0x7f0bf0126da0 2024-06-11T19:09:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bed7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bed7fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0126da0 msgr2=0x7f0bf01236a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bed7fa640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0126da0 0x7f0bf01236a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc036040 tx=0x7f0bdc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0126da0 msgr2=0x7f0bf01236a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bed7fa640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0126da0 0x7f0bf01236a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bed7fa640 10 client.4352.objecter ms_handle_connect 0x7f0bd403fdb0 2024-06-11T19:09:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.584+0000 7f0bf5428640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0bd403fdb0 0x7f0bd4042270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bf01248c0 tx=0x7f0be800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0bf6e8b640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:09:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0bf6e8b640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0123be0 0x7f0bf0123fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0bf6e8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/2102066123 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0bf0124500 con 0x7f0bf0123be0 2024-06-11T19:09:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0beffff640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0123be0 0x7f0bf0123fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0beffff640 10 monclient: get_auth_request con 0x7f0bf0123be0 auth_method 0 2024-06-11T19:09:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0beffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0beffff640 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2024-06-11T19:09:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.591+0000 7f0beffff640 1 -- 172.21.15.98:0/2102066123 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f0bf00a61a0 con 0x7f0bf0123be0 2024-06-11T19:09:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0beffff640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0123be0 0x7f0bf0123fc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc0097c0 tx=0x7f0bdc008030 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 10 client.4352.objecter ms_handle_connect 0x7f0bf0123be0 2024-06-11T19:09:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 10 client.4352.objecter resend_mon_ops 2024-06-11T19:09:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0bdc059080 con 0x7f0bf0123be0 2024-06-11T19:09:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:09:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 10 monclient: dump: 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.616 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f0bdc0495e0 con 0x7f0bf0123be0 2024-06-11T19:09:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:09:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0123be0 msgr2=0x7f0bf0123fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0123be0 0x7f0bf0123fc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc0097c0 tx=0x7f0bdc008030 comp rx=0 tx=0).stop 2024-06-11T19:09:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 1 -- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0123be0 msgr2=0x7f0bf0123fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bed7fa640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0bf0123be0 0x7f0bf0123fc0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc0097c0 tx=0x7f0bdc008030 comp rx=0 tx=0).stop 2024-06-11T19:09:40.616 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:40.616 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-06-11T19:09:40.616 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-06-11T19:09:40.617 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:40.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bf6e8b640 10 librados: watch_flush enter 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bf6e8b640 10 librados: watch_flush exit 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bf6e8b640 20 client.4352.objecter shutdown clearing up homeless session... 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.592+0000 7f0bf6e8b640 10 client.4352.objecter successfully canceled tick 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0bd403fdb0 msgr2=0x7f0bd4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf6e8b640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0bd403fdb0 0x7f0bd4042270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bf01248c0 tx=0x7f0be800b040 comp rx=0 tx=0).stop 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf6e8b640 10 monclient: shutdown 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf6e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0bf00a2530 msgr2=0x7f0bf0137700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf5428640 1 -- 172.21.15.98:0/2102066123 reap_dead start 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf6e8b640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0bf00a2530 0x7f0bf0137700 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0be400e880 tx=0x7f0be400ed40 comp rx=0 tx=0).stop 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/2102066123 shutdown_connections 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf6e8b640 1 --2- 172.21.15.98:0/2102066123 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0bf00a2530 0x7f0bf0137700 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/2102066123 >> 172.21.15.98:0/2102066123 conn(0x7f0bf00ab850 msgr2=0x7f0bf01bff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:40.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/2102066123 shutdown_connections 2024-06-11T19:09:40.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf6e8b640 1 -- 172.21.15.98:0/2102066123 wait complete. 2024-06-11T19:09:40.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.593+0000 7f0bf6e8b640 1 librados: shutdown 2024-06-11T19:09:40.620 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-06-11T19:09:40.620 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-06-11T19:09:40.620 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:09:40.630 INFO:tasks.ceph.mon.d:Stopped 2024-06-11T19:09:40.630 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-06-11T19:09:40.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:09:40.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.768+0000 7f7d8452a640 10 monclient: get_monmap_and_config 2024-06-11T19:09:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.768+0000 7f7d8452a640 10 monclient: build_initial_monmap 2024-06-11T19:09:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.768+0000 7f7d8452a640 10 monclient: monmap: 2024-06-11T19:09:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:40.769907+0000 2024-06-11T19:09:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:40.769907+0000 2024-06-11T19:09:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:40.770 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.769+0000 7f7d8452a640 1 Processor -- start 2024-06-11T19:09:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.769+0000 7f7d8452a640 1 -- start start 2024-06-11T19:09:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.769+0000 7f7d8452a640 10 monclient: init 2024-06-11T19:09:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 10 monclient: _add_conns ranks=[1,3,10,6,7,4,0,2,9,5,8] 2024-06-11T19:09:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f7d7c0a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d7c0a2530 0x7f7d7c13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 10 monclient(hunting): picked mon.noname-g con 0x7f7d7c0a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7d7c13e1a0 0x7f7d7c140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 10 monclient(hunting): picked mon.noname-f con 0x7f7d7c13e1a0 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d7c0a7140 con 0x7f7d7c0a4920 2024-06-11T19:09:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f7d7c0a2b50 con 0x7f7d7c13e1a0 2024-06-11T19:09:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7c0a2e30 con 0x7f7d7c0a2530 2024-06-11T19:09:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d8452a640 10 monclient(hunting): authenticate will time out at 790.577139s 2024-06-11T19:09:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d82ac7640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7d7c13e1a0 0x7f7d7c140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d7c0a2530 0x7f7d7c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d82ac7640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7d7c13e1a0 0x7f7d7c140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:39754/0 (socket says 172.21.15.98:39754) 2024-06-11T19:09:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d822c6640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f7d7c0a4920 0x7f7d7c0a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:40564/0 (socket says 172.21.15.98:40564) 2024-06-11T19:09:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d82ac7640 1 -- 172.21.15.98:0/3992793534 learned_addr learned my addr 172.21.15.98:0/3992793534 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 10 monclient(hunting): get_auth_request con 0x7f7d7c0a2530 auth_method 0 2024-06-11T19:09:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d82ac7640 10 monclient(hunting): get_auth_request con 0x7f7d7c13e1a0 auth_method 0 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d82ac7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d82ac7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d82ac7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d82ac7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d82ac7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d82ac7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 10 monclient(hunting): handle_auth_done global_id 4422 payload 274 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 1 -- 172.21.15.98:0/3992793534 >> v2:172.21.15.117:3300/0 conn(0x7f7d7c13e1a0 msgr2=0x7f7d7c140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 1 --2- 172.21.15.98:0/3992793534 >> v2:172.21.15.117:3300/0 conn(0x7f7d7c13e1a0 0x7f7d7c140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 1 -- 172.21.15.98:0/3992793534 >> v1:172.21.15.117:6789/0 conn(0x7f7d7c0a4920 legacy=0x7f7d7c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 10 monclient: _finish_hunting 0 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 1 monclient: found mon.noname-g 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d81ac5640 1 -- 172.21.15.98:0/3992793534 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7c0a3110 con 0x7f7d7c0a2530 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.770+0000 7f7d82ac7640 1 --2- 172.21.15.98:0/3992793534 >> v2:172.21.15.117:3300/0 conn(0x7f7d7c13e1a0 0x7f7d7c140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d81ac5640 1 --2- 172.21.15.98:0/3992793534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d7c0a2530 0x7f7d7c13dc60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009d90 tx=0x7f7d7002bd80 comp rx=0 tx=0).ready entity=mon.3 client_cookie=b8ec221b49b8b953 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 1 -- 172.21.15.98:0/3992793534 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d70039070 con 0x7f7d7c0a2530 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 10 monclient: dump: 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.786 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 10 monclient: _finish_auth 0 2024-06-11T19:09:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 10 monclient: _check_auth_tickets 2024-06-11T19:09:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 1 -- 172.21.15.98:0/3992793534 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d7002cda0 con 0x7f7d7c0a2530 2024-06-11T19:09:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 1 -- 172.21.15.98:0/3992793534 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d700327b0 con 0x7f7d7c0a2530 2024-06-11T19:09:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d812c4640 10 monclient: dump: 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d8452a640 5 monclient: authenticate success, global_id 4422 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d8452a640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.771+0000 7f7d8452a640 4 set_mon_vals no callback set 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.772+0000 7f7d8452a640 10 monclient: shutdown 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.772+0000 7f7d8452a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.772+0000 7f7d8452a640 1 -- 172.21.15.98:0/3992793534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d7c0a2530 msgr2=0x7f7d7c13dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.772+0000 7f7d8452a640 1 --2- 172.21.15.98:0/3992793534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d7c0a2530 0x7f7d7c13dc60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009d90 tx=0x7f7d7002bd80 comp rx=0 tx=0).stop 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.772+0000 7f7d8452a640 1 -- 172.21.15.98:0/3992793534 shutdown_connections 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.772+0000 7f7d8452a640 1 --2- 172.21.15.98:0/3992793534 >> v2:172.21.15.117:3300/0 conn(0x7f7d7c13e1a0 0x7f7d7c140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.772+0000 7f7d8452a640 1 --2- 172.21.15.98:0/3992793534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d7c0a2530 0x7f7d7c13dc60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.772+0000 7f7d8452a640 1 -- 172.21.15.98:0/3992793534 >> 172.21.15.98:0/3992793534 conn(0x7f7d7c0ab850 msgr2=0x7f7d7c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.772+0000 7f7d8452a640 1 -- 172.21.15.98:0/3992793534 shutdown_connections 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.772+0000 7f7d8452a640 1 -- 172.21.15.98:0/3992793534 wait complete. 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.772+0000 7f7d8452a640 10 monclient: build_initial_monmap 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.772+0000 7f7d8452a640 10 monclient: monmap: 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 1 librados: starting msgr at 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 1 librados: starting objecter 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 1 Processor -- start 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 1 -- start start 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 1 librados: setting wanted keys 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 1 librados: calling monclient init 2024-06-11T19:09:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 10 monclient: init 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 10 monclient: _add_conns ranks=[8,4,2,3,6,0,5,7,1] 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d7c0a2530 0x7f7d7c1cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 10 monclient(hunting): picked mon.noname-e con 0x7f7d7c0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7d7c0a4920 0x7f7d7c1cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 10 monclient(hunting): picked mon.noname-c con 0x7f7d7c0a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d7c13e1a0 0x7f7d7c09e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 10 monclient(hunting): picked mon.noname-b con 0x7f7d7c13e1a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7c138e50 con 0x7f7d7c13e1a0 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7d7c0a84b0 con 0x7f7d7c0a4920 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f7d7c0a6da0 con 0x7f7d7c0a2530 2024-06-11T19:09:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d8452a640 10 monclient(hunting): authenticate will time out at 790.580783s 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d822c6640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d7c0a2530 0x7f7d7c1cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d82ac7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d7c13e1a0 0x7f7d7c09e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d81ac5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7d7c0a4920 0x7f7d7c1cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d822c6640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d7c0a2530 0x7f7d7c1cd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54686/0 (socket says 172.21.15.98:54686) 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d822c6640 1 -- 172.21.15.98:0/116726114 learned_addr learned my addr 172.21.15.98:0/116726114 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d822c6640 10 monclient(hunting): get_auth_request con 0x7f7d7c0a2530 auth_method 0 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d822c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d822c6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d822c6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d81ac5640 10 monclient(hunting): get_auth_request con 0x7f7d7c0a4920 auth_method 0 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d81ac5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d81ac5640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d81ac5640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d82ac7640 10 monclient(hunting): get_auth_request con 0x7f7d7c13e1a0 auth_method 0 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d82ac7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d82ac7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.773+0000 7f7d82ac7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d822c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d822c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d822c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d82ac7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d82ac7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:40.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d82ac7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 10 monclient(hunting): handle_auth_done global_id 4468 payload 824 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 1 -- 172.21.15.98:0/116726114 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d7c0a2530 msgr2=0x7f7d7c1cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d7c0a2530 0x7f7d7c1cd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 1 -- 172.21.15.98:0/116726114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d7c13e1a0 msgr2=0x7f7d7c09e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d7c13e1a0 0x7f7d7c09e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 10 monclient: _finish_hunting 0 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 1 monclient: found mon.noname-c 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 1 -- 172.21.15.98:0/116726114 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7c122350 con 0x7f7d7c0a4920 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d822c6640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d7c0a2530 0x7f7d7c1cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d82ac7640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d7c13e1a0 0x7f7d7c09e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d81ac5640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7d7c0a4920 0x7f7d7c1cdc20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7d70033000 tx=0x7f7d70004a40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 10 client.?.objecter ms_handle_connect 0x7f7d7c0a4920 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 1 -- 172.21.15.98:0/116726114 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d70039050 con 0x7f7d7c0a4920 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 10 monclient: dump: 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 10 monclient: _finish_auth 0 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d8452a640 5 monclient: authenticate success, global_id 4468 2024-06-11T19:09:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 1 -- 172.21.15.98:0/116726114 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d70045070 con 0x7f7d7c0a4920 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d8452a640 10 monclient: _renew_subs 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d8452a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d8452a640 1 -- 172.21.15.98:0/116726114 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d7c0a0650 con 0x7f7d7c0a4920 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 1 -- 172.21.15.98:0/116726114 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d70004780 con 0x7f7d7c0a4920 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d8452a640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d80ac3640 4 set_mon_vals no callback set 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d7affd640 10 monclient: dump: 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:40.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d8452a640 10 monclient: _renew_subs 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d8452a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d8452a640 1 -- 172.21.15.98:0/116726114 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d7c121680 con 0x7f7d7c0a4920 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.774+0000 7f7d8452a640 1 librados: init done 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.775+0000 7f7d7affd640 1 -- 172.21.15.98:0/116726114 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f7d70004310 con 0x7f7d7c0a4920 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.775+0000 7f7d8452a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.775+0000 7f7d8452a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.775+0000 7f7d8452a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.775+0000 7f7d8452a640 1 -- 172.21.15.98:0/116726114 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d7c0a4240 con 0x7f7d7c0a4920 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.775+0000 7f7d7affd640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7d6803fdb0 0x7f7d68042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.775+0000 7f7d7affd640 1 -- 172.21.15.98:0/116726114 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f7d70073570 con 0x7f7d7c0a4920 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.775+0000 7f7d7affd640 10 client.4468.objecter ms_dispatch 0x7f7d7c000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.775+0000 7f7d7affd640 3 client.4468.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.775+0000 7f7d7affd640 3 client.4468.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:40.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.775+0000 7f7d7affd640 20 client.4468.objecter dump_active .. 0 homeless 2024-06-11T19:09:40.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.776+0000 7f7d822c6640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7d6803fdb0 0x7f7d68042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:40.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.776+0000 7f7d822c6640 10 monclient: get_auth_request con 0x7f7d6803fdb0 auth_method 0 2024-06-11T19:09:40.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.776+0000 7f7d7affd640 10 client.4468.objecter ms_handle_connect 0x7f7d6803fdb0 2024-06-11T19:09:40.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.776+0000 7f7d822c6640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7d6803fdb0 0x7f7d68042270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7d74000f60 tx=0x7f7d74007f00 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:40.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.780+0000 7f7d7affd640 1 -- 172.21.15.98:0/116726114 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7d7004e020 con 0x7f7d7c0a4920 2024-06-11T19:09:40.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.780+0000 7f7d7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:40.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:40.780+0000 7f7d7affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.035+0000 7f7d8452a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:09:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.035+0000 7f7d8452a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:09:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.035+0000 7f7d8452a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.035+0000 7f7d8452a640 1 -- 172.21.15.98:0/116726114 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7d7c136030 con 0x7f7d7c0a4920 2024-06-11T19:09:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.035+0000 7f7d7affd640 1 -- 172.21.15.98:0/116726114 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f7d70040d90 con 0x7f7d7c0a4920 2024-06-11T19:09:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.035+0000 7f7d7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:09:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.035+0000 7f7d7affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:41.036 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":8,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":26,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:09:41.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 10 librados: watch_flush enter 2024-06-11T19:09:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 10 librados: watch_flush exit 2024-06-11T19:09:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 20 client.4468.objecter shutdown clearing up homeless session... 2024-06-11T19:09:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 10 client.4468.objecter successfully canceled tick 2024-06-11T19:09:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 -- 172.21.15.98:0/116726114 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7d6803fdb0 msgr2=0x7f7d68042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7d6803fdb0 0x7f7d68042270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7d74000f60 tx=0x7f7d74007f00 comp rx=0 tx=0).stop 2024-06-11T19:09:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 10 monclient: shutdown 2024-06-11T19:09:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 -- 172.21.15.98:0/116726114 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7d7c0a4920 msgr2=0x7f7d7c1cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7d7c0a4920 0x7f7d7c1cdc20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7d70033000 tx=0x7f7d70004a40 comp rx=0 tx=0).stop 2024-06-11T19:09:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 -- 172.21.15.98:0/116726114 shutdown_connections 2024-06-11T19:09:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7d6803fdb0 0x7f7d68042270 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d7c13e1a0 0x7f7d7c09e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7d7c0a4920 0x7f7d7c1cdc20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 --2- 172.21.15.98:0/116726114 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d7c0a2530 0x7f7d7c1cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 -- 172.21.15.98:0/116726114 >> 172.21.15.98:0/116726114 conn(0x7f7d7c0ab850 msgr2=0x7f7d7c0a61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 -- 172.21.15.98:0/116726114 shutdown_connections 2024-06-11T19:09:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 -- 172.21.15.98:0/116726114 wait complete. 2024-06-11T19:09:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:41.037+0000 7f7d8452a640 1 librados: shutdown 2024-06-11T19:09:44.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:09:44.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.158+0000 7f1a80b0b640 10 monclient: get_monmap_and_config 2024-06-11T19:09:44.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.158+0000 7f1a80b0b640 10 monclient: build_initial_monmap 2024-06-11T19:09:44.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.158+0000 7f1a80b0b640 10 monclient: monmap: 2024-06-11T19:09:44.158 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:44.158 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:44.158 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:44.159231+0000 2024-06-11T19:09:44.158 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:44.159231+0000 2024-06-11T19:09:44.158 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.158+0000 7f1a80b0b640 1 Processor -- start 2024-06-11T19:09:44.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.158+0000 7f1a80b0b640 1 -- start start 2024-06-11T19:09:44.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.158+0000 7f1a80b0b640 10 monclient: init 2024-06-11T19:09:44.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:44.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 10 monclient: _add_conns ranks=[5,1,8,2,3,0,9,4,7,6,10] 2024-06-11T19:09:44.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1a7c09eee0 0x7f1a7c09e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:44.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 10 monclient(hunting): picked mon.noname-h con 0x7f1a7c09eee0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:44.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f1a7c09f2c0 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:44.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c09f6a0 0x7f1a7c135d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 10 monclient(hunting): picked mon.noname-e con 0x7f1a7c09f6a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a7c0a0de0 con 0x7f1a7c09f2c0 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f1a7c09db60 con 0x7f1a7c09eee0 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f1a7c076a00 con 0x7f1a7c09f6a0 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a80b0b640 10 monclient(hunting): authenticate will time out at 793.966548s 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c09f6a0 0x7f1a7c135d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c09f6a0 0x7f1a7c135d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:43934/0 (socket says 172.21.15.98:43934) 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 1 -- 172.21.15.98:0/946291119 learned_addr learned my addr 172.21.15.98:0/946291119 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 10 monclient(hunting): get_auth_request con 0x7f1a7c09f6a0 auth_method 0 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:44.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7a59c640 1 --2- 172.21.15.98:0/946291119 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1a7c09eee0 0x7f1a7c09e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7a59c640 10 monclient(hunting): get_auth_request con 0x7f1a7c09eee0 auth_method 0 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 10 monclient(hunting): handle_auth_done global_id 4508 payload 274 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 1 -- 172.21.15.98:0/946291119 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1a7c09eee0 msgr2=0x7f1a7c09e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 1 --2- 172.21.15.98:0/946291119 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1a7c09eee0 0x7f1a7c09e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:44.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 1 -- 172.21.15.98:0/946291119 >> v1:172.21.15.117:6789/0 conn(0x7f1a7c09f2c0 legacy=0x7f1a7c1355f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 1 monclient: found mon.noname-e 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.159+0000 7f1a7ad9d640 1 -- 172.21.15.98:0/946291119 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a7c0a0b00 con 0x7f1a7c09f6a0 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7ad9d640 1 --2- 172.21.15.98:0/946291119 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c09f6a0 0x7f1a7c135d00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009e80 tx=0x7f1a7402bd70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=4ce39b63b2502aba server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 1 -- 172.21.15.98:0/946291119 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1a7402cb80 con 0x7f1a7c09f6a0 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 10 monclient: dump: 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:44.171 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 10 monclient: _finish_auth 0 2024-06-11T19:09:44.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 10 monclient: _check_auth_tickets 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a80b0b640 5 monclient: authenticate success, global_id 4508 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a80b0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 1 -- 172.21.15.98:0/946291119 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a7402cd40 con 0x7f1a7c09f6a0 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a80b0b640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a80b0b640 4 set_mon_vals no callback set 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 1 -- 172.21.15.98:0/946291119 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1a74034860 con 0x7f1a7c09f6a0 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.160+0000 7f1a7959a640 10 monclient: dump: 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:44.173 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:44.174 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.161+0000 7f1a80b0b640 10 monclient: shutdown 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.161+0000 7f1a80b0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.161+0000 7f1a80b0b640 1 -- 172.21.15.98:0/946291119 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c09f6a0 msgr2=0x7f1a7c135d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.161+0000 7f1a80b0b640 1 --2- 172.21.15.98:0/946291119 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c09f6a0 0x7f1a7c135d00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009e80 tx=0x7f1a7402bd70 comp rx=0 tx=0).stop 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.161+0000 7f1a80b0b640 1 -- 172.21.15.98:0/946291119 shutdown_connections 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.161+0000 7f1a80b0b640 1 --2- 172.21.15.98:0/946291119 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c09f6a0 0x7f1a7c135d00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.161+0000 7f1a80b0b640 1 --2- 172.21.15.98:0/946291119 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1a7c09eee0 0x7f1a7c09e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.161+0000 7f1a80b0b640 1 -- 172.21.15.98:0/946291119 >> 172.21.15.98:0/946291119 conn(0x7f1a7c0a7850 msgr2=0x7f1a7c0a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.161+0000 7f1a80b0b640 1 -- 172.21.15.98:0/946291119 shutdown_connections 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.161+0000 7f1a80b0b640 1 -- 172.21.15.98:0/946291119 wait complete. 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.162+0000 7f1a80b0b640 10 monclient: build_initial_monmap 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.162+0000 7f1a80b0b640 10 monclient: monmap: 2024-06-11T19:09:44.175 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:44.176 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.162+0000 7f1a80b0b640 1 librados: starting msgr at 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.162+0000 7f1a80b0b640 1 librados: starting objecter 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.162+0000 7f1a80b0b640 1 Processor -- start 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.162+0000 7f1a80b0b640 1 -- start start 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.162+0000 7f1a80b0b640 1 librados: setting wanted keys 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.162+0000 7f1a80b0b640 1 librados: calling monclient init 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.162+0000 7f1a80b0b640 10 monclient: init 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 10 monclient: _add_conns ranks=[1,4,8,7,5,0,3,2,6] 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a7c09eee0 0x7f1a7c1cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 10 monclient(hunting): picked mon.noname-f con 0x7f1a7c09eee0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a7c09f2c0 0x7f1a7c1cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 10 monclient(hunting): picked mon.noname-c con 0x7f1a7c09f2c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:44.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c13a6d0 0x7f1a7c13ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 10 monclient(hunting): picked mon.noname-e con 0x7f1a7c13a6d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7c0a1940 con 0x7f1a7c09eee0 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1a7c0a17c0 con 0x7f1a7c09f2c0 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f1a7c0a01d0 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a80b0b640 10 monclient(hunting): authenticate will time out at 793.970327s 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c13a6d0 0x7f1a7c13ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a79d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a7c09f2c0 0x7f1a7c1cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c13a6d0 0x7f1a7c13ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:43948/0 (socket says 172.21.15.98:43948) 2024-06-11T19:09:44.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 1 -- 172.21.15.98:0/657646053 learned_addr learned my addr 172.21.15.98:0/657646053 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:44.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 10 monclient(hunting): get_auth_request con 0x7f1a7c13a6d0 auth_method 0 2024-06-11T19:09:44.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:44.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:44.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:44.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a79d9b640 10 monclient(hunting): get_auth_request con 0x7f1a7c09f2c0 auth_method 0 2024-06-11T19:09:44.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:44.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a79d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:44.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:44.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:44.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:44.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:44.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a79d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a79d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a79d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7a59c640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a7c09eee0 0x7f1a7c1cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 10 monclient(hunting): handle_auth_done global_id 4517 payload 824 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 1 -- 172.21.15.98:0/657646053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a7c09f2c0 msgr2=0x7f1a7c1cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a7c09f2c0 0x7f1a7c1cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 1 -- 172.21.15.98:0/657646053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a7c09eee0 msgr2=0x7f1a7c1cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a7c09eee0 0x7f1a7c1cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 1 monclient: found mon.noname-e 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 1 -- 172.21.15.98:0/657646053 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a7c1c4850 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7a59c640 10 monclient: get_auth_request con 0x7f1a7c09eee0 auth_method 0 2024-06-11T19:09:44.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7a59c640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a7c09eee0 0x7f1a7c1cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a79d9b640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a7c09f2c0 0x7f1a7c1cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a7ad9d640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c13a6d0 0x7f1a7c13ab50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009fa0 tx=0x7f1a74004a40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a6affd640 10 client.?.objecter ms_handle_connect 0x7f1a7c13a6d0 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.163+0000 7f1a6affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 1 -- 172.21.15.98:0/657646053 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1a74042070 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 10 monclient: dump: 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:44.181 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 10 monclient: _finish_auth 0 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:44.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a80b0b640 5 monclient: authenticate success, global_id 4517 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 1 -- 172.21.15.98:0/657646053 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a740046c0 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a80b0b640 10 monclient: _renew_subs 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a80b0b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a80b0b640 1 -- 172.21.15.98:0/657646053 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a7c0a46b0 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 1 -- 172.21.15.98:0/657646053 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1a74009510 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a78d99640 4 set_mon_vals no callback set 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a80b0b640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a6affd640 10 monclient: dump: 2024-06-11T19:09:44.183 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a80b0b640 10 monclient: _renew_subs 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a80b0b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a80b0b640 1 -- 172.21.15.98:0/657646053 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a7c136de0 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.164+0000 7f1a80b0b640 1 librados: init done 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a6affd640 1 -- 172.21.15.98:0/657646053 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f1a74045070 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a80b0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a80b0b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a80b0b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a80b0b640 1 -- 172.21.15.98:0/657646053 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a7c09e930 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a6affd640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1a5c03fdb0 0x7f1a5c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a6affd640 1 -- 172.21.15.98:0/657646053 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f1a74073880 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a6affd640 10 client.4517.objecter ms_dispatch 0x7f1a7c000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:44.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a6affd640 3 client.4517.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a6affd640 3 client.4517.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a6affd640 20 client.4517.objecter dump_active .. 0 homeless 2024-06-11T19:09:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a7a59c640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1a5c03fdb0 0x7f1a5c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.165+0000 7f1a7a59c640 10 monclient: get_auth_request con 0x7f1a5c03fdb0 auth_method 0 2024-06-11T19:09:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.166+0000 7f1a6affd640 10 client.4517.objecter ms_handle_connect 0x7f1a5c03fdb0 2024-06-11T19:09:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.166+0000 7f1a7a59c640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1a5c03fdb0 0x7f1a5c042270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1a70006fd0 tx=0x7f1a70008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.170+0000 7f1a6affd640 1 -- 172.21.15.98:0/657646053 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1a7c1c4850 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.170+0000 7f1a6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.170+0000 7f1a6affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:44.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.423+0000 7f1a80b0b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:09:44.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.423+0000 7f1a80b0b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:09:44.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.423+0000 7f1a80b0b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:44.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.423+0000 7f1a80b0b640 1 -- 172.21.15.98:0/657646053 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1a7c068c60 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.423+0000 7f1a6affd640 1 -- 172.21.15.98:0/657646053 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f1a7c0a0de0 con 0x7f1a7c13a6d0 2024-06-11T19:09:44.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.423+0000 7f1a6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:09:44.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.423+0000 7f1a6affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:44.424 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":8,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":29,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:09:44.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 10 librados: watch_flush enter 2024-06-11T19:09:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 10 librados: watch_flush exit 2024-06-11T19:09:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 20 client.4517.objecter shutdown clearing up homeless session... 2024-06-11T19:09:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 10 client.4517.objecter successfully canceled tick 2024-06-11T19:09:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 -- 172.21.15.98:0/657646053 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1a5c03fdb0 msgr2=0x7f1a5c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1a5c03fdb0 0x7f1a5c042270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1a70006fd0 tx=0x7f1a70008040 comp rx=0 tx=0).stop 2024-06-11T19:09:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 10 monclient: shutdown 2024-06-11T19:09:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 -- 172.21.15.98:0/657646053 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c13a6d0 msgr2=0x7f1a7c13ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c13a6d0 0x7f1a7c13ab50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009fa0 tx=0x7f1a74004a40 comp rx=0 tx=0).stop 2024-06-11T19:09:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 -- 172.21.15.98:0/657646053 shutdown_connections 2024-06-11T19:09:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1a5c03fdb0 0x7f1a5c042270 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:44.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a7c13a6d0 0x7f1a7c13ab50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:44.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a7c09f2c0 0x7f1a7c1cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:44.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 --2- 172.21.15.98:0/657646053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a7c09eee0 0x7f1a7c1cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:44.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 -- 172.21.15.98:0/657646053 >> 172.21.15.98:0/657646053 conn(0x7f1a7c0a7850 msgr2=0x7f1a7c09be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:44.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 -- 172.21.15.98:0/657646053 shutdown_connections 2024-06-11T19:09:44.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 -- 172.21.15.98:0/657646053 wait complete. 2024-06-11T19:09:44.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:44.425+0000 7f1a80b0b640 1 librados: shutdown 2024-06-11T19:09:47.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:09:47.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.546+0000 7fad17fe3640 10 monclient: get_monmap_and_config 2024-06-11T19:09:47.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.546+0000 7fad17fe3640 10 monclient: build_initial_monmap 2024-06-11T19:09:47.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.546+0000 7fad17fe3640 10 monclient: monmap: 2024-06-11T19:09:47.546 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:47.546 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:47.546 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:47.547460+0000 2024-06-11T19:09:47.546 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:47.547460+0000 2024-06-11T19:09:47.546 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:47.546 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:47.547 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:47.547 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:47.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 1 Processor -- start 2024-06-11T19:09:47.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 1 -- start start 2024-06-11T19:09:47.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 10 monclient: init 2024-06-11T19:09:47.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:47.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 10 monclient: _add_conns ranks=[8,2,0,1,4,6,9,10,3,7,5] 2024-06-11T19:09:47.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a4920 0x7fad100a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:47.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 10 monclient(hunting): picked mon.noname-e con 0x7fad100a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:47.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad100a5240 0x7fad100a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:47.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 10 monclient(hunting): picked mon.noname-b con 0x7fad100a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:47.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad100a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:47.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad100a7140 con 0x7fad100a2530 2024-06-11T19:09:47.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:47.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fad100a2b50 con 0x7fad100a5240 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fad100a2e30 con 0x7fad100a4920 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad17fe3640 10 monclient(hunting): authenticate will time out at 797.354756s 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad15d7f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a4920 0x7fad100a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad15d7f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a4920 0x7fad100a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:43964/0 (socket says 172.21.15.98:43964) 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad15d7f640 1 -- 172.21.15.98:0/971208241 learned_addr learned my addr 172.21.15.98:0/971208241 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad15d7f640 10 monclient(hunting): get_auth_request con 0x7fad100a4920 auth_method 0 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad15d7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad15d7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad15d7f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad1557e640 1 --2- 172.21.15.98:0/971208241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad100a5240 0x7fad100a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:47.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad16580640 1 --1- 172.21.15.98:0/971208241 >> v1:172.21.15.98:6789/0 conn(0x7fad100a2530 0x7fad100a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43178/0 (socket says 172.21.15.98:43178) 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad1557e640 10 monclient(hunting): get_auth_request con 0x7fad100a5240 auth_method 0 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad1557e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad1557e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.547+0000 7fad1557e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad15d7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad15d7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad15d7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad1557e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad1557e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad1557e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad15d7f640 10 monclient(hunting): handle_auth_done global_id 4526 payload 274 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad15d7f640 1 -- 172.21.15.98:0/971208241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad100a5240 msgr2=0x7fad100a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:47.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad15d7f640 1 --2- 172.21.15.98:0/971208241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad100a5240 0x7fad100a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad15d7f640 1 -- 172.21.15.98:0/971208241 >> v1:172.21.15.98:6789/0 conn(0x7fad100a2530 legacy=0x7fad100a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad15d7f640 10 monclient: _finish_hunting 0 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad15d7f640 1 monclient: found mon.noname-e 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad15d7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad15d7f640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad15d7f640 1 -- 172.21.15.98:0/971208241 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad100a3110 con 0x7fad100a4920 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.548+0000 7fad1557e640 1 --2- 172.21.15.98:0/971208241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad100a5240 0x7fad100a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad15d7f640 1 --2- 172.21.15.98:0/971208241 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a4920 0x7fad100a4d00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0098a0 tx=0x7fad0c02b840 comp rx=0 tx=0).ready entity=mon.8 client_cookie=3b5fb759247e8c93 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 1 -- 172.21.15.98:0/971208241 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad0c031a10 con 0x7fad100a4920 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 10 monclient: dump: 2024-06-11T19:09:47.560 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:47.561 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 10 monclient: _finish_auth 0 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 10 monclient: _check_auth_tickets 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 1 -- 172.21.15.98:0/971208241 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0c037070 con 0x7fad100a4920 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 5 monclient: authenticate success, global_id 4526 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 1 -- 172.21.15.98:0/971208241 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad0c031c10 con 0x7fad100a4920 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 4 set_mon_vals no callback set 2024-06-11T19:09:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad14d7d640 10 monclient: dump: 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 10 monclient: shutdown 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 1 -- 172.21.15.98:0/971208241 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a4920 msgr2=0x7fad100a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 1 --2- 172.21.15.98:0/971208241 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a4920 0x7fad100a4d00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0098a0 tx=0x7fad0c02b840 comp rx=0 tx=0).stop 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 1 -- 172.21.15.98:0/971208241 shutdown_connections 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 1 --2- 172.21.15.98:0/971208241 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fad100a5240 0x7fad100a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 1 --2- 172.21.15.98:0/971208241 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a4920 0x7fad100a4d00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 1 -- 172.21.15.98:0/971208241 >> 172.21.15.98:0/971208241 conn(0x7fad100ab850 msgr2=0x7fad100a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 1 -- 172.21.15.98:0/971208241 shutdown_connections 2024-06-11T19:09:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.549+0000 7fad17fe3640 1 -- 172.21.15.98:0/971208241 wait complete. 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.550+0000 7fad17fe3640 10 monclient: build_initial_monmap 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.550+0000 7fad17fe3640 10 monclient: monmap: 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:47.565 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.550+0000 7fad17fe3640 1 librados: starting msgr at 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.550+0000 7fad17fe3640 1 librados: starting objecter 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.550+0000 7fad17fe3640 1 Processor -- start 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.550+0000 7fad17fe3640 1 -- start start 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.550+0000 7fad17fe3640 1 librados: setting wanted keys 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.550+0000 7fad17fe3640 1 librados: calling monclient init 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.550+0000 7fad17fe3640 10 monclient: init 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 10 monclient: _add_conns ranks=[0,1,8,6,4,5,3,2,7] 2024-06-11T19:09:47.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad100a2530 0x7fad10137590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 10 monclient(hunting): picked mon.noname-a con 0x7fad100a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad100a4920 0x7fad10137ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 10 monclient(hunting): picked mon.noname-f con 0x7fad100a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a5240 0x7fad10123140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 10 monclient(hunting): picked mon.noname-e con 0x7fad100a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad100a6da0 con 0x7fad100a2530 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fad10138e50 con 0x7fad100a4920 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fad100a65f0 con 0x7fad100a5240 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad17fe3640 10 monclient(hunting): authenticate will time out at 797.358439s 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad15d7f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad100a2530 0x7fad10137590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:47.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad15d7f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad100a2530 0x7fad10137590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42014/0 (socket says 172.21.15.98:42014) 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad15d7f640 1 -- 172.21.15.98:0/603017650 learned_addr learned my addr 172.21.15.98:0/603017650 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad15d7f640 10 monclient(hunting): get_auth_request con 0x7fad100a2530 auth_method 0 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad15d7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad15d7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad15d7f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a5240 0x7fad10123140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 10 monclient(hunting): get_auth_request con 0x7fad100a5240 auth_method 0 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad15d7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad15d7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad15d7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 10 monclient(hunting): handle_auth_done global_id 4535 payload 824 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad1557e640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad100a4920 0x7fad10137ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 1 -- 172.21.15.98:0/603017650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad100a4920 msgr2=0x7fad10137ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad100a4920 0x7fad10137ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 1 -- 172.21.15.98:0/603017650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad100a2530 msgr2=0x7fad10137590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad100a2530 0x7fad10137590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 10 monclient: _finish_hunting 0 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 1 monclient: found mon.noname-e 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:47.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad16580640 1 -- 172.21.15.98:0/603017650 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad101c0c20 con 0x7fad100a5240 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.551+0000 7fad15d7f640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad100a2530 0x7fad10137590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad16580640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a5240 0x7fad10123140 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fad0800c6e0 tx=0x7fad0800cba0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 10 client.?.objecter ms_handle_connect 0x7fad100a5240 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 1 -- 172.21.15.98:0/603017650 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad08018070 con 0x7fad100a5240 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 10 monclient: dump: 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:47.570 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:47.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad17fe3640 5 monclient: authenticate success, global_id 4535 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 1 -- 172.21.15.98:0/603017650 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0800a640 con 0x7fad100a5240 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad17fe3640 10 monclient: _renew_subs 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad17fe3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad17fe3640 1 -- 172.21.15.98:0/603017650 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad10126cb0 con 0x7fad100a5240 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 1 -- 172.21.15.98:0/603017650 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad0800aca0 con 0x7fad100a5240 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad17fe3640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad07fff640 4 set_mon_vals no callback set 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad067fc640 10 monclient: dump: 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:47.572 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:47.574 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad17fe3640 10 monclient: _renew_subs 2024-06-11T19:09:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad17fe3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad17fe3640 1 -- 172.21.15.98:0/603017650 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fad100a8da0 con 0x7fad100a5240 2024-06-11T19:09:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.552+0000 7fad17fe3640 1 librados: init done 2024-06-11T19:09:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad067fc640 1 -- 172.21.15.98:0/603017650 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fad0800a830 con 0x7fad100a5240 2024-06-11T19:09:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad17fe3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad17fe3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad17fe3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad17fe3640 1 -- 172.21.15.98:0/603017650 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad100a4d00 con 0x7fad100a5240 2024-06-11T19:09:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad067fc640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7facf403fdb0 0x7facf4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad067fc640 1 -- 172.21.15.98:0/603017650 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fad08050650 con 0x7fad100a5240 2024-06-11T19:09:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad067fc640 10 client.4535.objecter ms_dispatch 0x7fad10000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad067fc640 3 client.4535.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad067fc640 3 client.4535.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad067fc640 20 client.4535.objecter dump_active .. 0 homeless 2024-06-11T19:09:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad15d7f640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7facf403fdb0 0x7facf4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.553+0000 7fad15d7f640 10 monclient: get_auth_request con 0x7facf403fdb0 auth_method 0 2024-06-11T19:09:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.554+0000 7fad067fc640 10 client.4535.objecter ms_handle_connect 0x7facf403fdb0 2024-06-11T19:09:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.554+0000 7fad15d7f640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7facf403fdb0 0x7facf4042270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fad0c000c00 tx=0x7fad0c03d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.557+0000 7fad067fc640 1 -- 172.21.15.98:0/603017650 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fad101c0c20 con 0x7fad100a5240 2024-06-11T19:09:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.557+0000 7fad067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.557+0000 7fad067fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:47.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.812+0000 7fad17fe3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:09:47.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.812+0000 7fad17fe3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:09:47.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.812+0000 7fad17fe3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:47.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.812+0000 7fad17fe3640 1 -- 172.21.15.98:0/603017650 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fad10124c30 con 0x7fad100a5240 2024-06-11T19:09:47.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.812+0000 7fad067fc640 1 -- 172.21.15.98:0/603017650 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fad0801bdd0 con 0x7fad100a5240 2024-06-11T19:09:47.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.812+0000 7fad067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:09:47.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.812+0000 7fad067fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:47.812 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":8,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":32,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:09:47.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 10 librados: watch_flush enter 2024-06-11T19:09:47.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 10 librados: watch_flush exit 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 20 client.4535.objecter shutdown clearing up homeless session... 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 10 client.4535.objecter successfully canceled tick 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 -- 172.21.15.98:0/603017650 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7facf403fdb0 msgr2=0x7facf4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7facf403fdb0 0x7facf4042270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fad0c000c00 tx=0x7fad0c03d040 comp rx=0 tx=0).stop 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 10 monclient: shutdown 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 -- 172.21.15.98:0/603017650 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a5240 msgr2=0x7fad10123140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a5240 0x7fad10123140 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fad0800c6e0 tx=0x7fad0800cba0 comp rx=0 tx=0).stop 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 -- 172.21.15.98:0/603017650 shutdown_connections 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7facf403fdb0 0x7facf4042270 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad100a5240 0x7fad10123140 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad100a4920 0x7fad10137ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 --2- 172.21.15.98:0/603017650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad100a2530 0x7fad10137590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:47.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 -- 172.21.15.98:0/603017650 >> 172.21.15.98:0/603017650 conn(0x7fad100ab850 msgr2=0x7fad101bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:47.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 -- 172.21.15.98:0/603017650 shutdown_connections 2024-06-11T19:09:47.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 -- 172.21.15.98:0/603017650 wait complete. 2024-06-11T19:09:47.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:47.814+0000 7fad17fe3640 1 librados: shutdown 2024-06-11T19:09:50.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:09:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.935+0000 7fca703f3640 10 monclient: get_monmap_and_config 2024-06-11T19:09:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.935+0000 7fca703f3640 10 monclient: build_initial_monmap 2024-06-11T19:09:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.935+0000 7fca703f3640 10 monclient: monmap: 2024-06-11T19:09:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:50.936793+0000 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:50.936793+0000 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:50.937 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.936+0000 7fca703f3640 1 Processor -- start 2024-06-11T19:09:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.936+0000 7fca703f3640 1 -- start start 2024-06-11T19:09:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.936+0000 7fca703f3640 10 monclient: init 2024-06-11T19:09:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 10 monclient: _add_conns ranks=[0,6,4,5,1,3,8,2,9,7,10] 2024-06-11T19:09:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca680a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca680a2530 0x7fca6813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 10 monclient(hunting): picked mon.noname-d con 0x7fca680a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 10 monclient(hunting): picked mon.noname-c con 0x7fca6813e1a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca680a7140 con 0x7fca680a4920 2024-06-11T19:09:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fca680a2b50 con 0x7fca6813e1a0 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fca680a2e30 con 0x7fca680a2530 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca703f3640 10 monclient(hunting): authenticate will time out at 800.744226s 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6d98e640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca680a2530 msgr2=0x7fca6813dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6d98e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca680a2530 0x7fca6813dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6e18f640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fca680a4920 0x7fca680a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43180/0 (socket says 172.21.15.98:43180) 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6e18f640 1 -- 172.21.15.98:0/1616374611 learned_addr learned my addr 172.21.15.98:0/1616374611 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6e990640 10 monclient(hunting): get_auth_request con 0x7fca6813e1a0 auth_method 0 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6e990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6e990640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6e990640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 msgr2=0x7fca68140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:09:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 msgr2=0x7fca68140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:50.937+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.138+0000 7fca6d98e640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca680a2530 msgr2=0x7fca6813dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:09:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.138+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca680a2530 0x7fca6813dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:09:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.138+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.138+0000 7fca6e990640 10 monclient(hunting): get_auth_request con 0x7fca6813e1a0 auth_method 0 2024-06-11T19:09:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.138+0000 7fca6e990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.138+0000 7fca6e990640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.138+0000 7fca6e990640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.138+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 msgr2=0x7fca68140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:09:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.138+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 msgr2=0x7fca68140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.138+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.138+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:09:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.539+0000 7fca6d98e640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca680a2530 msgr2=0x7fca6813dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:09:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.539+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca680a2530 0x7fca6813dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:09:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.539+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.539+0000 7fca6e990640 10 monclient(hunting): get_auth_request con 0x7fca6813e1a0 auth_method 0 2024-06-11T19:09:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.539+0000 7fca6e990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.539+0000 7fca6e990640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.539+0000 7fca6e990640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:51.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.539+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 msgr2=0x7fca68140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.539+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 msgr2=0x7fca68140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.539+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:51.539+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:09:52.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:52.340+0000 7fca6d98e640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca680a2530 msgr2=0x7fca6813dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:09:52.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:52.340+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca680a2530 0x7fca6813dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:09:52.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:52.340+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:52.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:52.340+0000 7fca6e990640 10 monclient(hunting): get_auth_request con 0x7fca6813e1a0 auth_method 0 2024-06-11T19:09:52.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:52.340+0000 7fca6e990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:52.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:52.340+0000 7fca6e990640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:52.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:52.340+0000 7fca6e990640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:52.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:52.340+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 msgr2=0x7fca68140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:52.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:52.340+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 msgr2=0x7fca68140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:52.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:52.340+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:52.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:52.340+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:09:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 10 monclient(hunting): tick 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 1 monclient(hunting): continuing hunt 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca680a2530 msgr2=0x7fca6813dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca680a2530 0x7fca6813dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 msgr2=0x7fca68140590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca6813e1a0 0x7fca68140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 1 -- 172.21.15.98:0/1616374611 >> v1:172.21.15.98:6789/0 conn(0x7fca680a4920 legacy=0x7fca680a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 10 monclient: _add_conns ranks=[8,9,2,3,1,7,5,10] 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 10 monclient(hunting): picked mon.noname-e con 0x7fca4c003840 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fca4c006270 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 0x7fca4c00b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fca4c008db0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 1 -- 172.21.15.98:0/1616374611 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fca680a2b50 con 0x7fca4c006270 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 1 -- 172.21.15.98:0/1616374611 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fca680a2e30 con 0x7fca4c008db0 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 1 -- 172.21.15.98:0/1616374611 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fca4c00ba10 con 0x7fca4c003840 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca5bfff640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6d98e640 10 monclient(hunting): get_auth_request con 0x7fca4c006270 auth_method 0 2024-06-11T19:09:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6d98e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6d98e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6d98e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e18f640 10 monclient(hunting): get_auth_request con 0x7fca4c003840 auth_method 0 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e18f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e18f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e18f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 0x7fca4c00b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e990640 10 monclient(hunting): get_auth_request con 0x7fca4c008db0 auth_method 0 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e990640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e990640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6d98e640 1 -- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 msgr2=0x7fca4c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6d98e640 1 -- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 msgr2=0x7fca4c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e18f640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 msgr2=0x7fca4c005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:09:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e18f640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 msgr2=0x7fca4c005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 msgr2=0x7fca4c00b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:09:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 msgr2=0x7fca4c00b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.937+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 0x7fca4c00b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:53.938+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 0x7fca4c00b250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:54.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:54.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:54.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6d98e640 10 monclient(hunting): get_auth_request con 0x7fca4c006270 auth_method 0 2024-06-11T19:09:54.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6d98e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:54.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6d98e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:54.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6d98e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:54.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6e18f640 10 monclient(hunting): get_auth_request con 0x7fca4c003840 auth_method 0 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6e18f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6e18f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6e18f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6d98e640 1 -- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 msgr2=0x7fca4c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6d98e640 1 -- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 msgr2=0x7fca4c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6e18f640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 msgr2=0x7fca4c005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6e18f640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 msgr2=0x7fca4c005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.138+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.139+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 0x7fca4c00b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.139+0000 7fca6e990640 10 monclient(hunting): get_auth_request con 0x7fca4c008db0 auth_method 0 2024-06-11T19:09:54.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.139+0000 7fca6e990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:54.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.139+0000 7fca6e990640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:54.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.139+0000 7fca6e990640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:54.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.139+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 msgr2=0x7fca4c00b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:09:54.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.139+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 msgr2=0x7fca4c00b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:54.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.139+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 0x7fca4c00b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:54.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.139+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 0x7fca4c00b250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:09:54.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:54.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6d98e640 10 monclient(hunting): get_auth_request con 0x7fca4c006270 auth_method 0 2024-06-11T19:09:54.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6d98e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:54.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6d98e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:54.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6d98e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6e18f640 10 monclient(hunting): get_auth_request con 0x7fca4c003840 auth_method 0 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6e18f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6e18f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6e18f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6d98e640 1 -- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 msgr2=0x7fca4c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6d98e640 1 -- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 msgr2=0x7fca4c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6e18f640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 msgr2=0x7fca4c005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6e18f640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 msgr2=0x7fca4c005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.539+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:09:54.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.540+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 0x7fca4c00b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:54.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.540+0000 7fca6e990640 10 monclient(hunting): get_auth_request con 0x7fca4c008db0 auth_method 0 2024-06-11T19:09:54.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.540+0000 7fca6e990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:54.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.540+0000 7fca6e990640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:54.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.540+0000 7fca6e990640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:54.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.540+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 msgr2=0x7fca4c00b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:09:54.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.540+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 msgr2=0x7fca4c00b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:09:54.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.540+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 0x7fca4c00b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:54.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:54.540+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 0x7fca4c00b250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:09:55.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.340+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.340+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.340+0000 7fca6d98e640 10 monclient(hunting): get_auth_request con 0x7fca4c006270 auth_method 0 2024-06-11T19:09:55.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.340+0000 7fca6d98e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.340+0000 7fca6d98e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.340+0000 7fca6d98e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.340+0000 7fca6e18f640 10 monclient(hunting): get_auth_request con 0x7fca4c003840 auth_method 0 2024-06-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.340+0000 7fca6e18f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.340+0000 7fca6e18f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.340+0000 7fca6e18f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.340+0000 7fca6d98e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.340+0000 7fca6d98e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6d98e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e990640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 0x7fca4c00b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e990640 10 monclient(hunting): get_auth_request con 0x7fca4c008db0 auth_method 0 2024-06-11T19:09:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e990640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e990640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 10 monclient(hunting): handle_auth_done global_id 4544 payload 274 2024-06-11T19:09:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 msgr2=0x7fca4c00b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca4c008db0 0x7fca4c00b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 1 -- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 msgr2=0x7fca4c008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 10 monclient: _finish_hunting 0 2024-06-11T19:09:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 1 monclient: found mon.noname-e 2024-06-11T19:09:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e18f640 1 -- 172.21.15.98:0/1616374611 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca680a3110 con 0x7fca4c003840 2024-06-11T19:09:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6d98e640 1 --2- 172.21.15.98:0/1616374611 >> v2:172.21.15.98:3300/0 conn(0x7fca4c006270 0x7fca4c008700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.341+0000 7fca6e990640 1 -- 172.21.15.98:0/1616374611 reap_dead start 2024-06-11T19:09:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6e18f640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fca64007d60 tx=0x7fca6402be50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=e1a886a977a32174 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 1 -- 172.21.15.98:0/1616374611 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca64032070 con 0x7fca4c003840 2024-06-11T19:09:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 10 monclient: dump: 2024-06-11T19:09:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.348 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.349 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 10 monclient: _finish_auth 0 2024-06-11T19:09:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 10 monclient: _check_auth_tickets 2024-06-11T19:09:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 1 -- 172.21.15.98:0/1616374611 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca6402cc60 con 0x7fca4c003840 2024-06-11T19:09:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca703f3640 5 monclient: authenticate success, global_id 4544 2024-06-11T19:09:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca703f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca703f3640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:55.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca703f3640 4 set_mon_vals no callback set 2024-06-11T19:09:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 1 -- 172.21.15.98:0/1616374611 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca6402ce50 con 0x7fca4c003840 2024-06-11T19:09:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.342+0000 7fca6d18d640 10 monclient: dump: 2024-06-11T19:09:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.350 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.351 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.343+0000 7fca703f3640 10 monclient: shutdown 2024-06-11T19:09:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.343+0000 7fca703f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:55.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.343+0000 7fca703f3640 1 -- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 msgr2=0x7fca4c005c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.343+0000 7fca703f3640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fca64007d60 tx=0x7fca6402be50 comp rx=0 tx=0).stop 2024-06-11T19:09:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.343+0000 7fca703f3640 1 -- 172.21.15.98:0/1616374611 shutdown_connections 2024-06-11T19:09:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.343+0000 7fca703f3640 1 --2- 172.21.15.98:0/1616374611 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca4c003840 0x7fca4c005c90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.343+0000 7fca703f3640 1 -- 172.21.15.98:0/1616374611 >> 172.21.15.98:0/1616374611 conn(0x7fca680ab850 msgr2=0x7fca680a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.343+0000 7fca703f3640 1 -- 172.21.15.98:0/1616374611 shutdown_connections 2024-06-11T19:09:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.343+0000 7fca703f3640 1 -- 172.21.15.98:0/1616374611 wait complete. 2024-06-11T19:09:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.344+0000 7fca703f3640 10 monclient: build_initial_monmap 2024-06-11T19:09:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.344+0000 7fca703f3640 10 monclient: monmap: 2024-06-11T19:09:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:55.352 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:55.353 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.344+0000 7fca703f3640 1 librados: starting msgr at 2024-06-11T19:09:55.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.344+0000 7fca703f3640 1 librados: starting objecter 2024-06-11T19:09:55.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.344+0000 7fca703f3640 1 Processor -- start 2024-06-11T19:09:55.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.344+0000 7fca703f3640 1 -- start start 2024-06-11T19:09:55.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.344+0000 7fca703f3640 1 librados: setting wanted keys 2024-06-11T19:09:55.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.344+0000 7fca703f3640 1 librados: calling monclient init 2024-06-11T19:09:55.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.344+0000 7fca703f3640 10 monclient: init 2024-06-11T19:09:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 10 monclient: _add_conns ranks=[4,8,2,1,5,0,7,3,6] 2024-06-11T19:09:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca4c003840 0x7fca680a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 10 monclient(hunting): picked mon.noname-c con 0x7fca4c003840 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca6809cfc0 0x7fca6809d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 10 monclient(hunting): picked mon.noname-e con 0x7fca6809cfc0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:55.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca6809d900 0x7fca681d4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 10 monclient(hunting): picked mon.noname-b con 0x7fca6809d900 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fca680a6da0 con 0x7fca6809d900 2024-06-11T19:09:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fca68138e50 con 0x7fca4c003840 2024-06-11T19:09:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fca680a4d00 con 0x7fca6809cfc0 2024-06-11T19:09:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca703f3640 10 monclient(hunting): authenticate will time out at 805.152304s 2024-06-11T19:09:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e990640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca6809d900 0x7fca681d4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca6809cfc0 0x7fca6809d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e18f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca4c003840 0x7fca680a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e990640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca6809d900 0x7fca681d4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50674/0 (socket says 172.21.15.98:50674) 2024-06-11T19:09:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 10 monclient(hunting): get_auth_request con 0x7fca6809cfc0 auth_method 0 2024-06-11T19:09:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e990640 1 -- 172.21.15.98:0/2276558578 learned_addr learned my addr 172.21.15.98:0/2276558578 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e990640 10 monclient(hunting): get_auth_request con 0x7fca6809d900 auth_method 0 2024-06-11T19:09:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e990640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e990640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e18f640 10 monclient(hunting): get_auth_request con 0x7fca4c003840 auth_method 0 2024-06-11T19:09:55.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e18f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e18f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e18f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e18f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e18f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e18f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e990640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e990640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6e990640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 10 monclient(hunting): handle_auth_done global_id 4553 payload 824 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 1 -- 172.21.15.98:0/2276558578 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca4c003840 msgr2=0x7fca680a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 1 --2- 172.21.15.98:0/2276558578 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca4c003840 0x7fca680a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 1 -- 172.21.15.98:0/2276558578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca6809d900 msgr2=0x7fca681d4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 1 --2- 172.21.15.98:0/2276558578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca6809d900 0x7fca681d4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 10 monclient: _finish_hunting 0 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 1 monclient: found mon.noname-e 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.345+0000 7fca6d98e640 1 -- 172.21.15.98:0/2276558578 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca680a0980 con 0x7fca6809cfc0 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca6d98e640 1 --2- 172.21.15.98:0/2276558578 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca6809cfc0 0x7fca6809d3c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca4c008d00 tx=0x7fca5c005480 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:55.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 10 client.?.objecter ms_handle_connect 0x7fca6809cfc0 2024-06-11T19:09:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 1 -- 172.21.15.98:0/2276558578 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca5c005b60 con 0x7fca6809cfc0 2024-06-11T19:09:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 10 monclient: dump: 2024-06-11T19:09:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.364 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.365 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 10 monclient: _finish_auth 0 2024-06-11T19:09:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 1 -- 172.21.15.98:0/2276558578 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca5c01c070 con 0x7fca6809cfc0 2024-06-11T19:09:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca703f3640 5 monclient: authenticate success, global_id 4553 2024-06-11T19:09:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 1 -- 172.21.15.98:0/2276558578 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca5c017cd0 con 0x7fca6809cfc0 2024-06-11T19:09:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca5affd640 10 monclient: dump: 2024-06-11T19:09:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.366 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.367 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.368 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca703f3640 10 monclient: _renew_subs 2024-06-11T19:09:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca6c98c640 4 set_mon_vals no callback set 2024-06-11T19:09:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca703f3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca703f3640 1 -- 172.21.15.98:0/2276558578 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca680aa860 con 0x7fca6809cfc0 2024-06-11T19:09:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca703f3640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca703f3640 10 monclient: _renew_subs 2024-06-11T19:09:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca703f3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca703f3640 1 -- 172.21.15.98:0/2276558578 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fca680a06a0 con 0x7fca6809cfc0 2024-06-11T19:09:55.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.346+0000 7fca703f3640 1 librados: init done 2024-06-11T19:09:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca5affd640 1 -- 172.21.15.98:0/2276558578 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fca5c0173f0 con 0x7fca6809cfc0 2024-06-11T19:09:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca5affd640 1 --2- 172.21.15.98:0/2276558578 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca4c041110 0x7fca4c0435d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca5affd640 1 -- 172.21.15.98:0/2276558578 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fca5c014040 con 0x7fca6809cfc0 2024-06-11T19:09:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca5affd640 10 client.4553.objecter ms_dispatch 0x7fca68000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca5affd640 3 client.4553.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca5affd640 3 client.4553.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca703f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca703f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca703f3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca703f3640 1 -- 172.21.15.98:0/2276558578 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca68075070 con 0x7fca6809cfc0 2024-06-11T19:09:55.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca5affd640 20 client.4553.objecter dump_active .. 0 homeless 2024-06-11T19:09:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca6e18f640 1 --2- 172.21.15.98:0/2276558578 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca4c041110 0x7fca4c0435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.347+0000 7fca6e18f640 10 monclient: get_auth_request con 0x7fca4c041110 auth_method 0 2024-06-11T19:09:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.348+0000 7fca6e18f640 1 --2- 172.21.15.98:0/2276558578 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca4c041110 0x7fca4c0435d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fca64037000 tx=0x7fca64010040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.348+0000 7fca5affd640 10 client.4553.objecter ms_handle_connect 0x7fca4c041110 2024-06-11T19:09:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.352+0000 7fca5affd640 1 -- 172.21.15.98:0/2276558578 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fca68075070 con 0x7fca6809cfc0 2024-06-11T19:09:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.352+0000 7fca5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:55.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.352+0000 7fca5affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:55.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.608+0000 7fca703f3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:09:55.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.608+0000 7fca703f3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:09:55.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.608+0000 7fca703f3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:55.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.608+0000 7fca703f3640 1 -- 172.21.15.98:0/2276558578 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fca680a5000 con 0x7fca6809cfc0 2024-06-11T19:09:55.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.609+0000 7fca5affd640 1 -- 172.21.15.98:0/2276558578 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7fca5c01fc20 con 0x7fca6809cfc0 2024-06-11T19:09:55.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.609+0000 7fca5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:09:55.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.609+0000 7fca5affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:55.609 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":10,"quorum":[0,1,2,3,4,5,7,8],"quorum_names":["a","f","b","g","c","h","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:09:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 10 librados: watch_flush enter 2024-06-11T19:09:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 10 librados: watch_flush exit 2024-06-11T19:09:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 20 client.4553.objecter shutdown clearing up homeless session... 2024-06-11T19:09:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 10 client.4553.objecter successfully canceled tick 2024-06-11T19:09:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 -- 172.21.15.98:0/2276558578 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca4c041110 msgr2=0x7fca4c0435d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 --2- 172.21.15.98:0/2276558578 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca4c041110 0x7fca4c0435d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fca64037000 tx=0x7fca64010040 comp rx=0 tx=0).stop 2024-06-11T19:09:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 10 monclient: shutdown 2024-06-11T19:09:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:55.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 -- 172.21.15.98:0/2276558578 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca6809cfc0 msgr2=0x7fca6809d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 --2- 172.21.15.98:0/2276558578 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca6809cfc0 0x7fca6809d3c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca4c008d00 tx=0x7fca5c005480 comp rx=0 tx=0).stop 2024-06-11T19:09:55.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 -- 172.21.15.98:0/2276558578 shutdown_connections 2024-06-11T19:09:55.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 --2- 172.21.15.98:0/2276558578 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca4c041110 0x7fca4c0435d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 --2- 172.21.15.98:0/2276558578 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca6809d900 0x7fca681d4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 --2- 172.21.15.98:0/2276558578 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca6809cfc0 0x7fca6809d3c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 --2- 172.21.15.98:0/2276558578 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca4c003840 0x7fca680a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 -- 172.21.15.98:0/2276558578 >> 172.21.15.98:0/2276558578 conn(0x7fca680ab850 msgr2=0x7fca68137c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:55.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 -- 172.21.15.98:0/2276558578 shutdown_connections 2024-06-11T19:09:55.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 -- 172.21.15.98:0/2276558578 wait complete. 2024-06-11T19:09:55.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.611+0000 7fca703f3640 1 librados: shutdown 2024-06-11T19:09:55.621 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-06-11T19:09:55.622 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-06-11T19:09:55.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:09:55.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.721+0000 7f4bd97f3640 10 monclient: get_monmap_and_config 2024-06-11T19:09:55.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.721+0000 7f4bd97f3640 10 monclient: build_initial_monmap 2024-06-11T19:09:55.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.721+0000 7f4bd97f3640 10 monclient: monmap: 2024-06-11T19:09:55.721 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:55.721 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.721 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:55.722155+0000 2024-06-11T19:09:55.721 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:55.722155+0000 2024-06-11T19:09:55.721 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:55.721 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.721 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:55.721 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:55.721 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:55.722 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:55.722 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:55.722 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:55.722 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:55.722 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:55.722 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:55.722 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:55.722 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:55.725 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.721+0000 7f4bd97f3640 1 Processor -- start 2024-06-11T19:09:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.721+0000 7f4bd97f3640 1 -- start start 2024-06-11T19:09:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.721+0000 7f4bd97f3640 10 monclient: init 2024-06-11T19:09:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 10 monclient: _add_conns ranks=[5,0,10,8,4,6,3,9,1,7,2] 2024-06-11T19:09:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd409d330 0x7f4bd4136c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 10 monclient(hunting): picked mon.noname-h con 0x7f4bd409d330 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bd409d710 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4bd409db10 0x7f4bd40a2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 10 monclient(hunting): picked mon.noname-f con 0x7f4bd409db10 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bd409b5d0 con 0x7f4bd409d710 2024-06-11T19:09:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f4bd41376a0 con 0x7f4bd409db10 2024-06-11T19:09:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f4bd409b840 con 0x7f4bd409d330 2024-06-11T19:09:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd97f3640 10 monclient(hunting): authenticate will time out at 805.529249s 2024-06-11T19:09:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd27fc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4bd409d710 0x7f4bd40a2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39858/0 (socket says 172.21.15.98:39858) 2024-06-11T19:09:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd27fc640 1 -- 172.21.15.98:0/2629768190 learned_addr learned my addr 172.21.15.98:0/2629768190 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:09:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd37fe640 1 --2- 172.21.15.98:0/2629768190 >> v2:172.21.15.117:3300/0 conn(0x7f4bd409db10 0x7f4bd40a2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 1 --2- 172.21.15.98:0/2629768190 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd409d330 0x7f4bd4136c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f4bd409d330 auth_method 0 2024-06-11T19:09:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd37fe640 10 monclient(hunting): get_auth_request con 0x7f4bd409db10 auth_method 0 2024-06-11T19:09:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd1ffb640 1 -- 172.21.15.98:0/2629768190 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1577923440 0 0) 0x7f4bd409b5d0 con 0x7f4bd409d710 2024-06-11T19:09:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd1ffb640 10 monclient(hunting): _negotiate 2024-06-11T19:09:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd1ffb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd1ffb640 10 monclient(hunting): my global_id is 4518 2024-06-11T19:09:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd1ffb640 1 -- 172.21.15.98:0/2629768190 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bc0003520 con 0x7f4bd409d710 2024-06-11T19:09:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 10 monclient(hunting): handle_auth_done global_id 4370 payload 274 2024-06-11T19:09:55.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 1 -- 172.21.15.98:0/2629768190 >> v2:172.21.15.117:3300/0 conn(0x7f4bd409db10 msgr2=0x7f4bd40a2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:55.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 1 --2- 172.21.15.98:0/2629768190 >> v2:172.21.15.117:3300/0 conn(0x7f4bd409db10 0x7f4bd40a2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 1 -- 172.21.15.98:0/2629768190 >> v1:172.21.15.98:6789/0 conn(0x7f4bd409d710 legacy=0x7f4bd40a2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:55.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 1 monclient: found mon.noname-h 2024-06-11T19:09:55.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:55.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:09:55.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.722+0000 7f4bd2ffd640 1 -- 172.21.15.98:0/2629768190 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd409bb20 con 0x7f4bd409d330 2024-06-11T19:09:55.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd2ffd640 1 --2- 172.21.15.98:0/2629768190 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd409d330 0x7f4bd4136c80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0097d0 tx=0x7f4bcc02b790 comp rx=0 tx=0).ready entity=mon.5 client_cookie=eb9648738087cfad server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:55.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 1 -- 172.21.15.98:0/2629768190 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bcc039070 con 0x7f4bd409d330 2024-06-11T19:09:55.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:09:55.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 10 monclient: dump: 2024-06-11T19:09:55.744 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.744 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.744 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.744 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.744 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.744 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.744 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.744 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.744 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.745 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.745 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.745 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.745 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.745 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.745 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.745 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:55.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:55.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:55.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 1 -- 172.21.15.98:0/2629768190 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bcc031960 con 0x7f4bd409d330 2024-06-11T19:09:55.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd97f3640 5 monclient: authenticate success, global_id 4370 2024-06-11T19:09:55.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd97f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:55.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:55.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd97f3640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:55.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd97f3640 4 set_mon_vals no callback set 2024-06-11T19:09:55.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 1 -- 172.21.15.98:0/2629768190 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bcc031b10 con 0x7f4bd409d330 2024-06-11T19:09:55.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:55.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.723+0000 7f4bd1ffb640 10 monclient: dump: 2024-06-11T19:09:55.746 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.747 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.747 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.747 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.747 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.747 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.747 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.747 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.747 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.747 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.747 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.747 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.747 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.748 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.748 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.748 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.724+0000 7f4bd97f3640 10 monclient: shutdown 2024-06-11T19:09:55.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.724+0000 7f4bd97f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:55.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.724+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2629768190 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd409d330 msgr2=0x7f4bd4136c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.724+0000 7f4bd97f3640 1 --2- 172.21.15.98:0/2629768190 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd409d330 0x7f4bd4136c80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0097d0 tx=0x7f4bcc02b790 comp rx=0 tx=0).stop 2024-06-11T19:09:55.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.724+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2629768190 shutdown_connections 2024-06-11T19:09:55.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.724+0000 7f4bd97f3640 1 --2- 172.21.15.98:0/2629768190 >> v2:172.21.15.117:3300/0 conn(0x7f4bd409db10 0x7f4bd40a2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.724+0000 7f4bd97f3640 1 --2- 172.21.15.98:0/2629768190 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd409d330 0x7f4bd4136c80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.724+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2629768190 >> 172.21.15.98:0/2629768190 conn(0x7f4bd40a7850 msgr2=0x7f4bd40a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.724+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2629768190 shutdown_connections 2024-06-11T19:09:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.724+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2629768190 wait complete. 2024-06-11T19:09:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.724+0000 7f4bd97f3640 10 monclient: build_initial_monmap 2024-06-11T19:09:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.724+0000 7f4bd97f3640 10 monclient: monmap: 2024-06-11T19:09:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:55.749 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:55.750 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:55.750 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:55.750 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:55.750 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:55.750 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:55.750 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:55.750 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:55.750 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:55.750 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 1 librados: starting msgr at 2024-06-11T19:09:55.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 1 librados: starting objecter 2024-06-11T19:09:55.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 1 Processor -- start 2024-06-11T19:09:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 1 -- start start 2024-06-11T19:09:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 1 librados: setting wanted keys 2024-06-11T19:09:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 1 librados: calling monclient init 2024-06-11T19:09:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 10 monclient: init 2024-06-11T19:09:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 10 monclient: _add_conns ranks=[4,7,5,1,2,8,0,3,6] 2024-06-11T19:09:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4bd409d330 0x7f4bd41cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 10 monclient(hunting): picked mon.noname-c con 0x7f4bd409d330 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4bd409d710 0x7f4bd409a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 10 monclient(hunting): picked mon.noname-i con 0x7f4bd409d710 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd409db10 0x7f4bd409a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 10 monclient(hunting): picked mon.noname-h con 0x7f4bd409db10 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f4bd409e180 con 0x7f4bd409d330 2024-06-11T19:09:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f4bd4076a00 con 0x7f4bd409db10 2024-06-11T19:09:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f4bd409e300 con 0x7f4bd409d710 2024-06-11T19:09:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd97f3640 10 monclient(hunting): authenticate will time out at 805.532790s 2024-06-11T19:09:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd2ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4bd409d330 0x7f4bd41cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd2ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4bd409d330 0x7f4bd41cd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:48190/0 (socket says 172.21.15.98:48190) 2024-06-11T19:09:55.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd2ffd640 1 -- 172.21.15.98:0/2558814421 learned_addr learned my addr 172.21.15.98:0/2558814421 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f4bd409d330 auth_method 0 2024-06-11T19:09:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd37fe640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd409db10 0x7f4bd409a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd27fc640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4bd409d710 0x7f4bd409a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd37fe640 10 monclient(hunting): get_auth_request con 0x7f4bd409db10 auth_method 0 2024-06-11T19:09:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd27fc640 10 monclient(hunting): get_auth_request con 0x7f4bd409d710 auth_method 0 2024-06-11T19:09:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.725+0000 7f4bd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 10 monclient(hunting): handle_auth_done global_id 4495 payload 824 2024-06-11T19:09:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 1 -- 172.21.15.98:0/2558814421 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4bd409d710 msgr2=0x7f4bd409a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4bd409d710 0x7f4bd409a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 1 -- 172.21.15.98:0/2558814421 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd409db10 msgr2=0x7f4bd409a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd409db10 0x7f4bd409a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 1 monclient: found mon.noname-c 2024-06-11T19:09:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 1 -- 172.21.15.98:0/2558814421 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd40a55f0 con 0x7f4bd409d330 2024-06-11T19:09:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd37fe640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4bd409db10 0x7f4bd409a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd2ffd640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4bd409d330 0x7f4bd41cd6f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc037040 tx=0x7f4bcc0046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 10 client.?.objecter ms_handle_connect 0x7f4bd409d330 2024-06-11T19:09:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bcc043020 con 0x7f4bd409d330 2024-06-11T19:09:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 10 monclient: dump: 2024-06-11T19:09:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.758 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bcc0044b0 con 0x7f4bd409d330 2024-06-11T19:09:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd97f3640 5 monclient: authenticate success, global_id 4495 2024-06-11T19:09:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd97f3640 10 monclient: _renew_subs 2024-06-11T19:09:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd97f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2558814421 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd40a7030 con 0x7f4bd409d330 2024-06-11T19:09:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bcc039040 con 0x7f4bd409d330 2024-06-11T19:09:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bcb7fe640 10 monclient: dump: 2024-06-11T19:09:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.760 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd97f3640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd97f3640 10 monclient: _renew_subs 2024-06-11T19:09:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd97f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2558814421 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd40a0600 con 0x7f4bd409d330 2024-06-11T19:09:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd97f3640 1 librados: init done 2024-06-11T19:09:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.726+0000 7f4bd17fa640 4 set_mon_vals no callback set 2024-06-11T19:09:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd97f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd97f3640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd41349d0 0x7f4bd409e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd97f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2558814421 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd4050830 con 0x7f4bd41349d0 2024-06-11T19:09:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f4bcc046020 con 0x7f4bd409d330 2024-06-11T19:09:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd27fc640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd41349d0 0x7f4bd409e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd27fc640 10 monclient: get_auth_request con 0x7f4bd41349d0 auth_method 0 2024-06-11T19:09:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd27fc640 10 monclient(hunting): handle_auth_done global_id 4527 payload 824 2024-06-11T19:09:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bcb7fe640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4bb403fde0 0x7f4bb40422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd27fc640 1 -- 172.21.15.98:0/2558814421 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4bd40a2e50 con 0x7f4bd41349d0 2024-06-11T19:09:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f4bcc072bd0 con 0x7f4bd409d330 2024-06-11T19:09:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bcb7fe640 10 client.4495.objecter ms_dispatch 0x7f4bd4000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bcb7fe640 3 client.4495.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:55.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bcb7fe640 3 client.4495.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd27fc640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd41349d0 0x7f4bd409e930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0010e0 tx=0x7f4bbc00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.727+0000 7f4bd37fe640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4bb403fde0 0x7f4bb40422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bd37fe640 10 monclient: get_auth_request con 0x7f4bb403fde0 auth_method 0 2024-06-11T19:09:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 20 client.4495.objecter dump_active .. 0 homeless 2024-06-11T19:09:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 10 client.4495.objecter ms_handle_connect 0x7f4bd41349d0 2024-06-11T19:09:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 10 client.4495.objecter resend_mon_ops 2024-06-11T19:09:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bbc007540 con 0x7f4bd41349d0 2024-06-11T19:09:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 10 monclient: dump: 2024-06-11T19:09:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.765 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4bbc018070 con 0x7f4bd41349d0 2024-06-11T19:09:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bd37fe640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4bb403fde0 0x7f4bb40422a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009fd0 tx=0x7f4bc400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd41349d0 msgr2=0x7f4bd409e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd41349d0 0x7f4bd409e930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0010e0 tx=0x7f4bbc00fc00 comp rx=0 tx=0).stop 2024-06-11T19:09:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd41349d0 msgr2=0x7f4bd409e930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd41349d0 0x7f4bd409e930 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0010e0 tx=0x7f4bbc00fc00 comp rx=0 tx=0).stop 2024-06-11T19:09:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.728+0000 7f4bcb7fe640 10 client.4495.objecter ms_handle_connect 0x7f4bb403fde0 2024-06-11T19:09:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.734+0000 7f4bd97f3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.734+0000 7f4bd97f3640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd409ee70 0x7f4bd409f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.734+0000 7f4bd97f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.734+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2558814421 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd41cdc30 con 0x7f4bd409ee70 2024-06-11T19:09:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.734+0000 7f4bd27fc640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd409ee70 0x7f4bd409f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.734+0000 7f4bd27fc640 10 monclient: get_auth_request con 0x7f4bd409ee70 auth_method 0 2024-06-11T19:09:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.734+0000 7f4bd27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.734+0000 7f4bd27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.734+0000 7f4bd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.734+0000 7f4bd27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.734+0000 7f4bd27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bd27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bd27fc640 10 monclient(hunting): handle_auth_done global_id 4536 payload 824 2024-06-11T19:09:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bd27fc640 1 -- 172.21.15.98:0/2558814421 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4bd409f7c0 con 0x7f4bd409ee70 2024-06-11T19:09:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 10 client.4495.objecter ms_handle_connect 0x7f4bd409ee70 2024-06-11T19:09:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bd27fc640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd409ee70 0x7f4bd409f280 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0010e0 tx=0x7f4bbc001050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 10 client.4495.objecter resend_mon_ops 2024-06-11T19:09:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bbc005e20 con 0x7f4bd409ee70 2024-06-11T19:09:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 10 monclient: dump: 2024-06-11T19:09:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.768 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.769 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9433 (secure 0 0 0) 0x7f4bbc0059b0 con 0x7f4bd409ee70 2024-06-11T19:09:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd409ee70 msgr2=0x7f4bd409f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd409ee70 0x7f4bd409f280 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0010e0 tx=0x7f4bbc001050 comp rx=0 tx=0).stop 2024-06-11T19:09:55.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 1 -- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd409ee70 msgr2=0x7f4bd409f280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:55.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bcb7fe640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd409ee70 0x7f4bd409f280 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0010e0 tx=0x7f4bbc001050 comp rx=0 tx=0).stop 2024-06-11T19:09:55.770 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:09:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 10, 2024-06-11T19:09:55.770 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 45972, 2024-06-11T19:09:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:55.771 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:09:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:55.772 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:55.773 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:55.776 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:55.778 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:55.779 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:55.780 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:55.782 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:55.784 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:55.786 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:55.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:55.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:55.792 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:55.794 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.795 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:09:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:55.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:55.797 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:55.797 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:55.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bd97f3640 10 librados: watch_flush enter 2024-06-11T19:09:55.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bd97f3640 10 librados: watch_flush exit 2024-06-11T19:09:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bd97f3640 20 client.4495.objecter shutdown clearing up homeless session... 2024-06-11T19:09:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bd97f3640 10 client.4495.objecter successfully canceled tick 2024-06-11T19:09:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2558814421 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4bb403fde0 msgr2=0x7f4bb40422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bd97f3640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4bb403fde0 0x7f4bb40422a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009fd0 tx=0x7f4bc400b040 comp rx=0 tx=0).stop 2024-06-11T19:09:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.735+0000 7f4bd97f3640 10 monclient: shutdown 2024-06-11T19:09:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.736+0000 7f4bd97f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.736+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4bd409d330 msgr2=0x7f4bd41cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.736+0000 7f4bd97f3640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4bd409d330 0x7f4bd41cd6f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc037040 tx=0x7f4bcc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.736+0000 7f4bd37fe640 1 -- 172.21.15.98:0/2558814421 reap_dead start 2024-06-11T19:09:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.736+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2558814421 shutdown_connections 2024-06-11T19:09:55.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.736+0000 7f4bd97f3640 1 --2- 172.21.15.98:0/2558814421 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4bd409d330 0x7f4bd41cd6f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.736+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2558814421 >> 172.21.15.98:0/2558814421 conn(0x7f4bd40a7850 msgr2=0x7f4bd4071470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.736+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2558814421 shutdown_connections 2024-06-11T19:09:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.736+0000 7f4bd97f3640 1 -- 172.21.15.98:0/2558814421 wait complete. 2024-06-11T19:09:55.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.736+0000 7f4bd97f3640 1 librados: shutdown 2024-06-11T19:09:55.801 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:09:55.801 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-06-11T19:09:55.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:09:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.901+0000 7fe65210b640 10 monclient: get_monmap_and_config 2024-06-11T19:09:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.901+0000 7fe65210b640 10 monclient: build_initial_monmap 2024-06-11T19:09:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.901+0000 7fe65210b640 10 monclient: monmap: 2024-06-11T19:09:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:55.902552+0000 2024-06-11T19:09:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:55.902552+0000 2024-06-11T19:09:55.901 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:55.902 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.902 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:55.902 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:55.903 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:55.903 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:55.903 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:55.903 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:55.903 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:55.903 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:55.903 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:55.903 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:55.903 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:55.903 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 1 Processor -- start 2024-06-11T19:09:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 1 -- start start 2024-06-11T19:09:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 10 monclient: init 2024-06-11T19:09:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 10 monclient: _add_conns ranks=[1,8,7,4,5,6,3,10,2,0,9] 2024-06-11T19:09:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe64c0a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe64c0a2530 0x7fe64c13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 10 monclient(hunting): picked mon.noname-e con 0x7fe64c0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe64c13e1a0 0x7fe64c140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 10 monclient(hunting): picked mon.noname-i con 0x7fe64c13e1a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64c0a7140 con 0x7fe64c0a4920 2024-06-11T19:09:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fe64c0a2b50 con 0x7fe64c13e1a0 2024-06-11T19:09:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe64c0a2e30 con 0x7fe64c0a2530 2024-06-11T19:09:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe65210b640 10 monclient(hunting): authenticate will time out at 805.709858s 2024-06-11T19:09:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.902+0000 7fe64affd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe64c0a2530 0x7fe64c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe64c0a2530 0x7fe64c13dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:53152/0 (socket says 172.21.15.98:53152) 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 1 -- 172.21.15.98:0/1546268811 learned_addr learned my addr 172.21.15.98:0/1546268811 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 10 monclient(hunting): get_auth_request con 0x7fe64c0a2530 auth_method 0 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64bfff640 1 --2- 172.21.15.98:0/1546268811 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe64c13e1a0 0x7fe64c140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64bfff640 10 monclient(hunting): get_auth_request con 0x7fe64c13e1a0 auth_method 0 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 10 monclient(hunting): handle_auth_done global_id 4562 payload 274 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 1 -- 172.21.15.98:0/1546268811 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe64c13e1a0 msgr2=0x7fe64c140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 1 --2- 172.21.15.98:0/1546268811 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe64c13e1a0 0x7fe64c140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 1 -- 172.21.15.98:0/1546268811 >> v1:172.21.15.117:6789/0 conn(0x7fe64c0a4920 legacy=0x7fe64c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 1 monclient: found mon.noname-e 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 1 -- 172.21.15.98:0/1546268811 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64c0a3110 con 0x7fe64c0a2530 2024-06-11T19:09:55.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.903+0000 7fe64affd640 1 --2- 172.21.15.98:0/1546268811 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe64c0a2530 0x7fe64c13dc60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe6400098a0 tx=0x7fe64002b840 comp rx=0 tx=0).ready entity=mon.8 client_cookie=c99f73f66072080f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 1 -- 172.21.15.98:0/1546268811 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe640031a10 con 0x7fe64c0a2530 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 10 monclient: dump: 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.921 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.922 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.922 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.922 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.922 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.922 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.922 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:55.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:55.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:55.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 5 monclient: authenticate success, global_id 4562 2024-06-11T19:09:55.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 1 -- 172.21.15.98:0/1546268811 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe640037070 con 0x7fe64c0a2530 2024-06-11T19:09:55.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 4 set_mon_vals no callback set 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 1 -- 172.21.15.98:0/1546268811 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe640031c10 con 0x7fe64c0a2530 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe64a7fc640 10 monclient: dump: 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.923 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.924 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.924 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.924 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.924 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.924 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.924 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.924 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.924 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.924 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 10 monclient: shutdown 2024-06-11T19:09:55.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:55.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 1 -- 172.21.15.98:0/1546268811 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe64c0a2530 msgr2=0x7fe64c13dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 1 --2- 172.21.15.98:0/1546268811 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe64c0a2530 0x7fe64c13dc60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe6400098a0 tx=0x7fe64002b840 comp rx=0 tx=0).stop 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 1 -- 172.21.15.98:0/1546268811 shutdown_connections 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 1 --2- 172.21.15.98:0/1546268811 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe64c13e1a0 0x7fe64c140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 1 --2- 172.21.15.98:0/1546268811 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe64c0a2530 0x7fe64c13dc60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 1 -- 172.21.15.98:0/1546268811 >> 172.21.15.98:0/1546268811 conn(0x7fe64c0ab850 msgr2=0x7fe64c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 1 -- 172.21.15.98:0/1546268811 shutdown_connections 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.904+0000 7fe65210b640 1 -- 172.21.15.98:0/1546268811 wait complete. 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.905+0000 7fe65210b640 10 monclient: build_initial_monmap 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.905+0000 7fe65210b640 10 monclient: monmap: 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:55.925 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.905+0000 7fe65210b640 1 librados: starting msgr at 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.905+0000 7fe65210b640 1 librados: starting objecter 2024-06-11T19:09:55.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.905+0000 7fe65210b640 1 Processor -- start 2024-06-11T19:09:55.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.905+0000 7fe65210b640 1 -- start start 2024-06-11T19:09:55.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.905+0000 7fe65210b640 1 librados: setting wanted keys 2024-06-11T19:09:55.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.905+0000 7fe65210b640 1 librados: calling monclient init 2024-06-11T19:09:55.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.905+0000 7fe65210b640 10 monclient: init 2024-06-11T19:09:55.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:55.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 10 monclient: _add_conns ranks=[4,7,3,2,8,1,0,6,5] 2024-06-11T19:09:55.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe64c0a2530 0x7fe64c1cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 10 monclient(hunting): picked mon.noname-c con 0x7fe64c0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:55.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe64c0a4920 0x7fe64c136b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 10 monclient(hunting): picked mon.noname-i con 0x7fe64c0a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe64c13e1a0 0x7fe64c137080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 10 monclient(hunting): picked mon.noname-g con 0x7fe64c13e1a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fe64c138e50 con 0x7fe64c13e1a0 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe64c0a84b0 con 0x7fe64c0a2530 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fe64c0a6da0 con 0x7fe64c0a4920 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 10 monclient(hunting): authenticate will time out at 805.713236s 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe64c0a2530 0x7fe64c1cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe64c0a2530 0x7fe64c1cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:48198/0 (socket says 172.21.15.98:48198) 2024-06-11T19:09:55.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 1 -- 172.21.15.98:0/3488748889 learned_addr learned my addr 172.21.15.98:0/3488748889 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:55.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 10 monclient(hunting): get_auth_request con 0x7fe64c0a2530 auth_method 0 2024-06-11T19:09:55.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64affd640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe64c0a4920 0x7fe64c136b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64affd640 10 monclient(hunting): get_auth_request con 0x7fe64c0a4920 auth_method 0 2024-06-11T19:09:55.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64bfff640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe64c13e1a0 0x7fe64c137080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64bfff640 10 monclient(hunting): get_auth_request con 0x7fe64c13e1a0 auth_method 0 2024-06-11T19:09:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 10 monclient(hunting): handle_auth_done global_id 4504 payload 824 2024-06-11T19:09:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 1 -- 172.21.15.98:0/3488748889 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe64c0a4920 msgr2=0x7fe64c136b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe64c0a4920 0x7fe64c136b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 1 -- 172.21.15.98:0/3488748889 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe64c13e1a0 msgr2=0x7fe64c137080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe64c13e1a0 0x7fe64c137080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:09:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 1 monclient: found mon.noname-c 2024-06-11T19:09:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 1 -- 172.21.15.98:0/3488748889 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64c0a1820 con 0x7fe64c0a2530 2024-06-11T19:09:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64affd640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe64c0a4920 0x7fe64c136b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe64b7fe640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe64c0a2530 0x7fe64c1cd6b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00c500 tx=0x7fe63c00c9c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 10 client.?.objecter ms_handle_connect 0x7fe64c0a2530 2024-06-11T19:09:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe63c012630 con 0x7fe64c0a2530 2024-06-11T19:09:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 10 monclient: dump: 2024-06-11T19:09:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 10 monclient: _finish_auth 0 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:55.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 5 monclient: authenticate success, global_id 4504 2024-06-11T19:09:55.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe63c012830 con 0x7fe64c0a2530 2024-06-11T19:09:55.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:55.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe63c012e50 con 0x7fe64c0a2530 2024-06-11T19:09:55.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 10 monclient: _renew_subs 2024-06-11T19:09:55.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:55.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe65210b640 1 -- 172.21.15.98:0/3488748889 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe64c1234f0 con 0x7fe64c0a2530 2024-06-11T19:09:55.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:55.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe633fff640 10 monclient: dump: 2024-06-11T19:09:55.936 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.936 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.936 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.937 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.937 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.937 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.937 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.937 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.937 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.937 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.937 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.937 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.937 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.937 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.938 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.938 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.906+0000 7fe649ffb640 4 set_mon_vals no callback set 2024-06-11T19:09:55.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe65210b640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:55.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe65210b640 10 monclient: _renew_subs 2024-06-11T19:09:55.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe65210b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:55.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe65210b640 1 -- 172.21.15.98:0/3488748889 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fe64c0a40d0 con 0x7fe64c0a2530 2024-06-11T19:09:55.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe65210b640 1 librados: init done 2024-06-11T19:09:55.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe65210b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:55.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe65210b640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c09fba0 0x7fe64c09ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe65210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe65210b640 1 -- 172.21.15.98:0/3488748889 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe64c050830 con 0x7fe64c09fba0 2024-06-11T19:09:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fe63c0129e0 con 0x7fe64c0a2530 2024-06-11T19:09:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe64affd640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c09fba0 0x7fe64c09ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe64affd640 10 monclient: get_auth_request con 0x7fe64c09fba0 auth_method 0 2024-06-11T19:09:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe64affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe64affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe64affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.907+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe64affd640 10 monclient(hunting): handle_auth_done global_id 4502 payload 824 2024-06-11T19:09:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe64affd640 1 -- 172.21.15.98:0/3488748889 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe64c0aa8d0 con 0x7fe64c09fba0 2024-06-11T19:09:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe63803ff50 0x7fe638042410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fe63c050890 con 0x7fe64c0a2530 2024-06-11T19:09:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 10 client.4504.objecter ms_dispatch 0x7fe64c000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 3 client.4504.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 3 client.4504.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe64affd640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c09fba0 0x7fe64c09ff80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe64c073de0 tx=0x7fe640004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe64bfff640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe63803ff50 0x7fe638042410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 20 client.4504.objecter dump_active .. 0 homeless 2024-06-11T19:09:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 10 client.4504.objecter ms_handle_connect 0x7fe64c09fba0 2024-06-11T19:09:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 10 client.4504.objecter resend_mon_ops 2024-06-11T19:09:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe64bfff640 10 monclient: get_auth_request con 0x7fe63803ff50 auth_method 0 2024-06-11T19:09:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe64004a070 con 0x7fe64c09fba0 2024-06-11T19:09:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 10 monclient: dump: 2024-06-11T19:09:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:55.942 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 10 client.4504.objecter ms_handle_connect 0x7fe63803ff50 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe64bfff640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe63803ff50 0x7fe638042410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe64c122450 tx=0x7fe64400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe640037040 con 0x7fe64c09fba0 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c09fba0 msgr2=0x7fe64c09ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c09fba0 0x7fe64c09ff80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe64c073de0 tx=0x7fe640004730 comp rx=0 tx=0).stop 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c09fba0 msgr2=0x7fe64c09ff80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.908+0000 7fe633fff640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c09fba0 0x7fe64c09ff80 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe64c073de0 tx=0x7fe640004730 comp rx=0 tx=0).stop 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe65210b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe65210b640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c0a04c0 0x7fe64c13bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe65210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe65210b640 1 -- 172.21.15.98:0/3488748889 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe64c0a4d00 con 0x7fe64c0a04c0 2024-06-11T19:09:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe64affd640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c0a04c0 0x7fe64c13bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe64affd640 10 monclient: get_auth_request con 0x7fe64c0a04c0 auth_method 0 2024-06-11T19:09:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe64affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe64affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe64affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe64affd640 1 -- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c0a04c0 msgr2=0x7fe64c13bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-06-11T19:09:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe64affd640 1 -- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c0a04c0 msgr2=0x7fe64c13bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:09:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe64affd640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c0a04c0 0x7fe64c13bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:09:55.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:55.915+0000 7fe64affd640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c0a04c0 0x7fe64c13bd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:56.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe64affd640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c0a04c0 0x7fe64c13bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe64affd640 10 monclient: get_auth_request con 0x7fe64c0a04c0 auth_method 0 2024-06-11T19:09:56.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe64affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe64affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe64affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:09:56.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe64affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe64affd640 10 monclient(hunting): handle_auth_done global_id 4520 payload 824 2024-06-11T19:09:56.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe64affd640 1 -- 172.21.15.98:0/3488748889 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe64c06c300 con 0x7fe64c0a04c0 2024-06-11T19:09:56.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe64affd640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c0a04c0 0x7fe64c13bd50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe64002b880 tx=0x7fe6400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 10 client.4504.objecter ms_handle_connect 0x7fe64c0a04c0 2024-06-11T19:09:56.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 10 client.4504.objecter resend_mon_ops 2024-06-11T19:09:56.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe640052070 con 0x7fe64c0a04c0 2024-06-11T19:09:56.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:56.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 10 monclient: dump: 2024-06-11T19:09:56.119 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.119 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.119 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.119 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.119 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.119 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.121 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9431 (secure 0 0 0) 0x7fe640003e70 con 0x7fe64c0a04c0 2024-06-11T19:09:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c0a04c0 msgr2=0x7fe64c13bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c0a04c0 0x7fe64c13bd50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe64002b880 tx=0x7fe6400046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 1 -- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c0a04c0 msgr2=0x7fe64c13bd50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe633fff640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe64c0a04c0 0x7fe64c13bd50 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe64002b880 tx=0x7fe6400046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.121 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 10, 2024-06-11T19:09:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 46349, 2024-06-11T19:09:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:56.149 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:56.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.116+0000 7fe65210b640 10 librados: watch_flush enter 2024-06-11T19:09:56.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 10 librados: watch_flush exit 2024-06-11T19:09:56.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 20 client.4504.objecter shutdown clearing up homeless session... 2024-06-11T19:09:56.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 10 client.4504.objecter successfully canceled tick 2024-06-11T19:09:56.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 1 -- 172.21.15.98:0/3488748889 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe63803ff50 msgr2=0x7fe638042410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe63803ff50 0x7fe638042410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe64c122450 tx=0x7fe64400b040 comp rx=0 tx=0).stop 2024-06-11T19:09:56.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 10 monclient: shutdown 2024-06-11T19:09:56.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:56.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 1 -- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe64c0a2530 msgr2=0x7fe64c1cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 1 --2- 172.21.15.98:0/3488748889 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe64c0a2530 0x7fe64c1cd6b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00c500 tx=0x7fe63c00c9c0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe64bfff640 1 -- 172.21.15.98:0/3488748889 reap_dead start 2024-06-11T19:09:56.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe64bfff640 1 -- 172.21.15.98:0/3488748889 reap_dead start 2024-06-11T19:09:56.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 1 -- 172.21.15.98:0/3488748889 shutdown_connections 2024-06-11T19:09:56.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 1 -- 172.21.15.98:0/3488748889 >> 172.21.15.98:0/3488748889 conn(0x7fe64c0ab850 msgr2=0x7fe64c0a8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:56.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 1 -- 172.21.15.98:0/3488748889 shutdown_connections 2024-06-11T19:09:56.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 1 -- 172.21.15.98:0/3488748889 wait complete. 2024-06-11T19:09:56.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.117+0000 7fe65210b640 1 librados: shutdown 2024-06-11T19:09:56.152 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:09:56.153 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-06-11T19:09:56.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:09:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.252+0000 7fca3db20640 10 monclient: get_monmap_and_config 2024-06-11T19:09:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.252+0000 7fca3db20640 10 monclient: build_initial_monmap 2024-06-11T19:09:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.252+0000 7fca3db20640 10 monclient: monmap: 2024-06-11T19:09:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:56.253357+0000 2024-06-11T19:09:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:56.253357+0000 2024-06-11T19:09:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:56.253 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:56.253 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:56.253 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:56.253 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:56.253 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:56.253 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:56.253 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:56.253 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:56.253 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:56.253 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:56.253 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 1 Processor -- start 2024-06-11T19:09:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 1 -- start start 2024-06-11T19:09:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 10 monclient: init 2024-06-11T19:09:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 10 monclient: _add_conns ranks=[9,7,8,3,5,2,6,1,10,0,4] 2024-06-11T19:09:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fca380a4920 0x7fca380a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 10 monclient(hunting): picked mon.noname-a con 0x7fca380a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fca380a5240 0x7fca380a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 10 monclient(hunting): picked mon.noname-i con 0x7fca380a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca380a2530 0x7fca380a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 10 monclient(hunting): picked mon.noname-e con 0x7fca380a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fca380a7140 con 0x7fca380a4920 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fca380a2b50 con 0x7fca380a5240 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fca380a2e30 con 0x7fca380a2530 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca3db20640 10 monclient(hunting): authenticate will time out at 806.060685s 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca377fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fca380a4920 0x7fca380a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca377fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fca380a4920 0x7fca380a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55134/0 (socket says 172.21.15.98:55134) 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca377fe640 1 -- 172.21.15.98:0/563889096 learned_addr learned my addr 172.21.15.98:0/563889096 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca37fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca380a2530 0x7fca380a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca377fe640 10 monclient(hunting): get_auth_request con 0x7fca380a4920 auth_method 0 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca37fff640 10 monclient(hunting): get_auth_request con 0x7fca380a2530 auth_method 0 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca37fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca37fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca36ffd640 1 --2- 172.21.15.98:0/563889096 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fca380a5240 0x7fca380a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca36ffd640 10 monclient(hunting): get_auth_request con 0x7fca380a5240 auth_method 0 2024-06-11T19:09:56.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca36ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.253+0000 7fca36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca377fe640 10 monclient(hunting): handle_auth_done global_id 4545 payload 274 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca377fe640 1 -- 172.21.15.98:0/563889096 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca380a2530 msgr2=0x7fca380a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca377fe640 1 --2- 172.21.15.98:0/563889096 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca380a2530 0x7fca380a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca377fe640 1 -- 172.21.15.98:0/563889096 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fca380a5240 msgr2=0x7fca380a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca377fe640 1 --2- 172.21.15.98:0/563889096 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fca380a5240 0x7fca380a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca377fe640 10 monclient: _finish_hunting 0 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca377fe640 1 monclient: found mon.noname-a 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca377fe640 1 -- 172.21.15.98:0/563889096 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca380a3110 con 0x7fca380a4920 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca377fe640 1 --2- 172.21.15.98:0/563889096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca380a4920 0x7fca380a4d00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca28009870 tx=0x7fca2802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea1b1da58c202cc server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 1 -- 172.21.15.98:0/563889096 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca28038070 con 0x7fca380a4920 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:56.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 10 monclient: dump: 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.268 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 1 -- 172.21.15.98:0/563889096 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca28031990 con 0x7fca380a4920 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca3db20640 5 monclient: authenticate success, global_id 4545 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca3db20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca3db20640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 1 -- 172.21.15.98:0/563889096 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca280323b0 con 0x7fca380a4920 2024-06-11T19:09:56.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca3db20640 4 set_mon_vals no callback set 2024-06-11T19:09:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.254+0000 7fca367fc640 10 monclient: dump: 2024-06-11T19:09:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.270 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.255+0000 7fca3db20640 10 monclient: shutdown 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.255+0000 7fca3db20640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.255+0000 7fca3db20640 1 -- 172.21.15.98:0/563889096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca380a4920 msgr2=0x7fca380a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.255+0000 7fca3db20640 1 --2- 172.21.15.98:0/563889096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca380a4920 0x7fca380a4d00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca28009870 tx=0x7fca2802b7c0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.255+0000 7fca3db20640 1 -- 172.21.15.98:0/563889096 shutdown_connections 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.255+0000 7fca3db20640 1 --2- 172.21.15.98:0/563889096 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca380a2530 0x7fca380a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.255+0000 7fca3db20640 1 --2- 172.21.15.98:0/563889096 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fca380a5240 0x7fca380a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.255+0000 7fca3db20640 1 --2- 172.21.15.98:0/563889096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca380a4920 0x7fca380a4d00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.255+0000 7fca3db20640 1 -- 172.21.15.98:0/563889096 >> 172.21.15.98:0/563889096 conn(0x7fca380ab850 msgr2=0x7fca380a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.255+0000 7fca3db20640 1 -- 172.21.15.98:0/563889096 shutdown_connections 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.255+0000 7fca3db20640 1 -- 172.21.15.98:0/563889096 wait complete. 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 10 monclient: build_initial_monmap 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 10 monclient: monmap: 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:56.272 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 1 librados: starting msgr at 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 1 librados: starting objecter 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 1 Processor -- start 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 1 -- start start 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 1 librados: setting wanted keys 2024-06-11T19:09:56.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 1 librados: calling monclient init 2024-06-11T19:09:56.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 10 monclient: init 2024-06-11T19:09:56.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:56.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 10 monclient: _add_conns ranks=[6,4,5,3,0,8,1,7,2] 2024-06-11T19:09:56.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca380a2530 0x7fca381cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.256+0000 7fca3db20640 10 monclient(hunting): picked mon.noname-d con 0x7fca380a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:56.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca380a4920 0x7fca381ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 10 monclient(hunting): picked mon.noname-c con 0x7fca380a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:56.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fca380a5240 0x7fca38123180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 10 monclient(hunting): picked mon.noname-h con 0x7fca380a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:56.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fca380a6da0 con 0x7fca380a4920 2024-06-11T19:09:56.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fca38138e50 con 0x7fca380a5240 2024-06-11T19:09:56.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca377fe640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca380a2530 msgr2=0x7fca381cdcf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:09:56.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fca380a6420 con 0x7fca380a2530 2024-06-11T19:09:56.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:56.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 10 monclient(hunting): authenticate will time out at 806.064059s 2024-06-11T19:09:56.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca377fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca380a2530 0x7fca381cdcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:56.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca380a4920 0x7fca381ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca380a4920 0x7fca381ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:48202/0 (socket says 172.21.15.98:48202) 2024-06-11T19:09:56.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 1 -- 172.21.15.98:0/1473521809 learned_addr learned my addr 172.21.15.98:0/1473521809 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:56.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 10 monclient(hunting): get_auth_request con 0x7fca380a4920 auth_method 0 2024-06-11T19:09:56.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca37fff640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fca380a5240 0x7fca38123180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca37fff640 10 monclient(hunting): get_auth_request con 0x7fca380a5240 auth_method 0 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca37fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca37fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 10 monclient(hunting): handle_auth_done global_id 4513 payload 824 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 1 -- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca380a2530 msgr2=0x7fca381cdcf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca380a2530 0x7fca381cdcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 1 -- 172.21.15.98:0/1473521809 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fca380a5240 msgr2=0x7fca38123180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fca380a5240 0x7fca38123180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 1 monclient: found mon.noname-c 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 1 -- 172.21.15.98:0/1473521809 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca380a9290 con 0x7fca380a4920 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca36ffd640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca380a4920 0x7fca381ce230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00e8a0 tx=0x7fca2c00ed60 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 10 client.?.objecter ms_handle_connect 0x7fca380a4920 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:56.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca2c017070 con 0x7fca380a4920 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 10 monclient: dump: 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.278 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 10 monclient: _finish_auth 0 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 5 monclient: authenticate success, global_id 4513 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca2c00ee90 con 0x7fca380a4920 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 10 monclient: _renew_subs 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 1 -- 172.21.15.98:0/1473521809 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca38124bd0 con 0x7fca380a4920 2024-06-11T19:09:56.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca2c00cdb0 con 0x7fca380a4920 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca1ffff640 10 monclient: dump: 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.280 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 10 monclient: _renew_subs 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 1 -- 172.21.15.98:0/1473521809 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fca38126c50 con 0x7fca380a4920 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca35ffb640 4 set_mon_vals no callback set 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.257+0000 7fca3db20640 1 librados: init done 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fca2c01a070 con 0x7fca380a4920 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca3db20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca3db20640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca381236c0 0x7fca38123aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca3db20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca3db20640 1 -- 172.21.15.98:0/1473521809 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fca38050830 con 0x7fca381236c0 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca1ffff640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca2403fdb0 0x7fca24042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca377fe640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca381236c0 0x7fca38123aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca377fe640 10 monclient: get_auth_request con 0x7fca381236c0 auth_method 0 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fca2c00f040 con 0x7fca380a4920 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca1ffff640 10 client.4513.objecter ms_dispatch 0x7fca38000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca1ffff640 3 client.4513.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca1ffff640 3 client.4513.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.258+0000 7fca377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca37fff640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca2403fdb0 0x7fca24042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 20 client.4513.objecter dump_active .. 0 homeless 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca37fff640 10 monclient: get_auth_request con 0x7fca2403fdb0 auth_method 0 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca377fe640 10 monclient(hunting): handle_auth_done global_id 4522 payload 824 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca377fe640 1 -- 172.21.15.98:0/1473521809 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca380a6750 con 0x7fca381236c0 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 10 client.4513.objecter ms_handle_connect 0x7fca381236c0 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 10 client.4513.objecter resend_mon_ops 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca377fe640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca381236c0 0x7fca38123aa0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fca38073de0 tx=0x7fca280046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca28041030 con 0x7fca381236c0 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 10 monclient: dump: 2024-06-11T19:09:56.283 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.284 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.285 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca37fff640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca2403fdb0 0x7fca24042270 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fca381248e0 tx=0x7fca3000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 10 client.4513.objecter ms_handle_connect 0x7fca2403fdb0 2024-06-11T19:09:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fca28039040 con 0x7fca381236c0 2024-06-11T19:09:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca381236c0 msgr2=0x7fca38123aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca381236c0 0x7fca38123aa0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fca38073de0 tx=0x7fca280046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca381236c0 msgr2=0x7fca38123aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:56.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.259+0000 7fca1ffff640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca381236c0 0x7fca38123aa0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca3db20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:56.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca3db20640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca38123fe0 0x7fca3809e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca3db20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca3db20640 1 -- 172.21.15.98:0/1473521809 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fca38126690 con 0x7fca38123fe0 2024-06-11T19:09:56.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca377fe640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca38123fe0 0x7fca3809e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca377fe640 10 monclient: get_auth_request con 0x7fca38123fe0 auth_method 0 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca377fe640 10 monclient(hunting): handle_auth_done global_id 4531 payload 824 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca377fe640 1 -- 172.21.15.98:0/1473521809 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fca38125490 con 0x7fca38123fe0 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 10 client.4513.objecter ms_handle_connect 0x7fca38123fe0 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca377fe640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca38123fe0 0x7fca3809e9f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fca280097c0 tx=0x7fca28009240 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 10 client.4513.objecter resend_mon_ops 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca28041040 con 0x7fca38123fe0 2024-06-11T19:09:56.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 10 monclient: dump: 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.289 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.290 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.290 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.290 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.290 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.290 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9261 (secure 0 0 0) 0x7fca28051430 con 0x7fca38123fe0 2024-06-11T19:09:56.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:56.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:56.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca38123fe0 msgr2=0x7fca3809e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca38123fe0 0x7fca3809e9f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fca280097c0 tx=0x7fca28009240 comp rx=0 tx=0).stop 2024-06-11T19:09:56.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 1 -- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca38123fe0 msgr2=0x7fca3809e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:56.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.266+0000 7fca1ffff640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca38123fe0 0x7fca3809e9f0 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fca280097c0 tx=0x7fca28009240 comp rx=0 tx=0).stop 2024-06-11T19:09:56.291 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 10, 2024-06-11T19:09:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 46484, 2024-06-11T19:09:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:56.314 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:56.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 10 librados: watch_flush enter 2024-06-11T19:09:56.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 10 librados: watch_flush exit 2024-06-11T19:09:56.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 20 client.4513.objecter shutdown clearing up homeless session... 2024-06-11T19:09:56.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 10 client.4513.objecter successfully canceled tick 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 1 -- 172.21.15.98:0/1473521809 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca2403fdb0 msgr2=0x7fca24042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca2403fdb0 0x7fca24042270 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fca381248e0 tx=0x7fca3000b040 comp rx=0 tx=0).stop 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 10 monclient: shutdown 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 1 -- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca380a4920 msgr2=0x7fca381ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 1 --2- 172.21.15.98:0/1473521809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fca380a4920 0x7fca381ce230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00e8a0 tx=0x7fca2c00ed60 comp rx=0 tx=0).stop 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca37fff640 1 -- 172.21.15.98:0/1473521809 reap_dead start 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca37fff640 1 -- 172.21.15.98:0/1473521809 reap_dead start 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 1 -- 172.21.15.98:0/1473521809 shutdown_connections 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 1 -- 172.21.15.98:0/1473521809 >> 172.21.15.98:0/1473521809 conn(0x7fca380ab850 msgr2=0x7fca381bf2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 1 -- 172.21.15.98:0/1473521809 shutdown_connections 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 1 -- 172.21.15.98:0/1473521809 wait complete. 2024-06-11T19:09:56.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.267+0000 7fca3db20640 1 librados: shutdown 2024-06-11T19:09:56.318 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:09:56.318 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-06-11T19:09:56.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:09:56.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.417+0000 7f271f985640 10 monclient: get_monmap_and_config 2024-06-11T19:09:56.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.417+0000 7f271f985640 10 monclient: build_initial_monmap 2024-06-11T19:09:56.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.417+0000 7f271f985640 10 monclient: monmap: 2024-06-11T19:09:56.418 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:56.418 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.418 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:56.418676+0000 2024-06-11T19:09:56.418 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:56.418676+0000 2024-06-11T19:09:56.418 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:56.418 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.418 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:56.418 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:56.418 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:56.419 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 1 Processor -- start 2024-06-11T19:09:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 1 -- start start 2024-06-11T19:09:56.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 10 monclient: init 2024-06-11T19:09:56.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 10 monclient: _add_conns ranks=[6,4,1,0,9,3,10,7,5,2,8] 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f27180a4920 0x7f27180a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 10 monclient(hunting): picked mon.noname-d con 0x7f27180a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a5240 0x7f27180a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 10 monclient(hunting): picked mon.noname-c con 0x7f27180a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f27180a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27180a7140 con 0x7f27180a2530 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f27180a2b50 con 0x7f27180a5240 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f27180a2e30 con 0x7f27180a4920 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:56.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271f985640 10 monclient(hunting): authenticate will time out at 806.225921s 2024-06-11T19:09:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271d721640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f27180a4920 msgr2=0x7f27180a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:09:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.418+0000 7f271d721640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f27180a4920 0x7f27180a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a5240 0x7f27180a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a5240 0x7f27180a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:48232/0 (socket says 172.21.15.98:48232) 2024-06-11T19:09:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 1 -- 172.21.15.98:0/1154856735 learned_addr learned my addr 172.21.15.98:0/1154856735 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 10 monclient(hunting): get_auth_request con 0x7f27180a5240 auth_method 0 2024-06-11T19:09:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 10 monclient(hunting): handle_auth_done global_id 4540 payload 274 2024-06-11T19:09:56.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 1 -- 172.21.15.98:0/1154856735 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f27180a4920 msgr2=0x7f27180a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:09:56.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 1 --2- 172.21.15.98:0/1154856735 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f27180a4920 0x7f27180a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 1 -- 172.21.15.98:0/1154856735 >> v1:172.21.15.117:6789/0 conn(0x7f27180a2530 legacy=0x7f27180a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 10 monclient: _finish_hunting 0 2024-06-11T19:09:56.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 1 monclient: found mon.noname-c 2024-06-11T19:09:56.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:56.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:56.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.419+0000 7f271cf20640 1 -- 172.21.15.98:0/1154856735 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27180a3110 con 0x7f27180a5240 2024-06-11T19:09:56.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271cf20640 1 --2- 172.21.15.98:0/1154856735 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a5240 0x7f27180a59a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f27080098a0 tx=0x7f270802b860 comp rx=0 tx=0).ready entity=mon.4 client_cookie=2c7d1ee962fd1cee server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 1 -- 172.21.15.98:0/1154856735 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2708031a30 con 0x7f27180a5240 2024-06-11T19:09:56.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 10 monclient: dump: 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.432 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 10 monclient: _finish_auth 0 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 5 monclient: authenticate success, global_id 4540 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 1 -- 172.21.15.98:0/1154856735 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2708037070 con 0x7f27180a5240 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:56.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 1 -- 172.21.15.98:0/1154856735 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2708031c30 con 0x7f27180a5240 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 4 set_mon_vals no callback set 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f270ffff640 10 monclient: dump: 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.434 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 10 monclient: shutdown 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 1 -- 172.21.15.98:0/1154856735 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a5240 msgr2=0x7f27180a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 1 --2- 172.21.15.98:0/1154856735 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a5240 0x7f27180a59a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f27080098a0 tx=0x7f270802b860 comp rx=0 tx=0).stop 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 1 -- 172.21.15.98:0/1154856735 shutdown_connections 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 1 --2- 172.21.15.98:0/1154856735 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a5240 0x7f27180a59a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 1 --2- 172.21.15.98:0/1154856735 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f27180a4920 0x7f27180a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 1 -- 172.21.15.98:0/1154856735 >> 172.21.15.98:0/1154856735 conn(0x7f27180ab850 msgr2=0x7f27180a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 1 -- 172.21.15.98:0/1154856735 shutdown_connections 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.420+0000 7f271f985640 1 -- 172.21.15.98:0/1154856735 wait complete. 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.421+0000 7f271f985640 10 monclient: build_initial_monmap 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.421+0000 7f271f985640 10 monclient: monmap: 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:56.436 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.421+0000 7f271f985640 1 librados: starting msgr at 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.421+0000 7f271f985640 1 librados: starting objecter 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.421+0000 7f271f985640 1 Processor -- start 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.421+0000 7f271f985640 1 -- start start 2024-06-11T19:09:56.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.421+0000 7f271f985640 1 librados: setting wanted keys 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.421+0000 7f271f985640 1 librados: calling monclient init 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.421+0000 7f271f985640 10 monclient: init 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 10 monclient: _add_conns ranks=[4,0,2,5,7,8,3,6,1] 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a2530 0x7f27181cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 10 monclient(hunting): picked mon.noname-c con 0x7f27180a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27180a4920 0x7f27181ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 10 monclient(hunting): picked mon.noname-a con 0x7f27180a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27180a5240 0x7f2718122ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 10 monclient(hunting): picked mon.noname-b con 0x7f27180a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27180a6da0 con 0x7f27180a4920 2024-06-11T19:09:56.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2718138e50 con 0x7f27180a5240 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f27180a65f0 con 0x7f27180a2530 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271f985640 10 monclient(hunting): authenticate will time out at 806.229291s 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a2530 0x7f27181cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271cf20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27180a4920 0x7f27181ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a2530 0x7f27181cddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:48240/0 (socket says 172.21.15.98:48240) 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 1 -- 172.21.15.98:0/3303541125 learned_addr learned my addr 172.21.15.98:0/3303541125 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 10 monclient(hunting): get_auth_request con 0x7f27180a2530 auth_method 0 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271df22640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27180a5240 0x7f2718122ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271df22640 10 monclient(hunting): get_auth_request con 0x7f27180a5240 auth_method 0 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271df22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271df22640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271df22640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271cf20640 10 monclient(hunting): get_auth_request con 0x7f27180a4920 auth_method 0 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271cf20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271cf20640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271cf20640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271df22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271df22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271df22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271cf20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271cf20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271cf20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 10 monclient(hunting): handle_auth_done global_id 4549 payload 824 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 1 -- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27180a5240 msgr2=0x7f2718122ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f27180a5240 0x7f2718122ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 1 -- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27180a4920 msgr2=0x7f27181ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27180a4920 0x7f27181ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 10 monclient: _finish_hunting 0 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 1 monclient: found mon.noname-c 2024-06-11T19:09:56.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 1 -- 172.21.15.98:0/3303541125 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2718126e60 con 0x7f27180a2530 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f271d721640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a2530 0x7f27181cddc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f271400ba10 tx=0x7f271400bed0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f270dffb640 10 client.?.objecter ms_handle_connect 0x7f27180a2530 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f270dffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f271400cde0 con 0x7f27180a2530 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.422+0000 7f270dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270dffb640 10 monclient: dump: 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.442 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.443 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.443 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270dffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270dffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2714010070 con 0x7f27180a2530 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f271f985640 5 monclient: authenticate success, global_id 4549 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270dffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f27140158e0 con 0x7f27180a2530 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270dffb640 10 monclient: dump: 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.444 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270f7fe640 4 set_mon_vals no callback set 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f271f985640 10 monclient: _renew_subs 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f271f985640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:56.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f271f985640 1 -- 172.21.15.98:0/3303541125 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27180aad70 con 0x7f27180a2530 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f271f985640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f271f985640 10 monclient: _renew_subs 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f271f985640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f271f985640 1 -- 172.21.15.98:0/3303541125 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f2718125420 con 0x7f27180a2530 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f271f985640 1 librados: init done 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.423+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f2714015470 con 0x7f27180a2530 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271f985640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271f985640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2718127830 0x7f2718063180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271f985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271f985640 1 -- 172.21.15.98:0/3303541125 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2718069750 con 0x7f2718127830 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f26fc03fdb0 0x7f26fc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f2714051440 con 0x7f27180a2530 2024-06-11T19:09:56.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 10 client.4549.objecter ms_dispatch 0x7f2718000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271cf20640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2718127830 0x7f2718063180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 3 client.4549.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 3 client.4549.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271cf20640 10 monclient: get_auth_request con 0x7f2718127830 auth_method 0 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271cf20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271cf20640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271cf20640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 20 client.4549.objecter dump_active .. 0 homeless 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271cf20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271cf20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271cf20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271df22640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f26fc03fdb0 0x7f26fc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271df22640 10 monclient: get_auth_request con 0x7f26fc03fdb0 auth_method 0 2024-06-11T19:09:56.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271cf20640 10 monclient(hunting): handle_auth_done global_id 4580 payload 824 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271cf20640 1 -- 172.21.15.98:0/3303541125 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27180a90a0 con 0x7f2718127830 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 10 client.4549.objecter ms_handle_connect 0x7f2718127830 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 10 client.4549.objecter resend_mon_ops 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271cf20640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2718127830 0x7f2718063180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f27180abf80 tx=0x7f27080046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 10 client.4549.objecter ms_handle_connect 0x7f26fc03fdb0 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f27080315c0 con 0x7f2718127830 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f271df22640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f26fc03fdb0 0x7f26fc042270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2710000f60 tx=0x7f2710009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.424+0000 7f270dffb640 10 monclient: dump: 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.448 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.425+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2708037040 con 0x7f2718127830 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.425+0000 7f270dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.425+0000 7f270dffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:56.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.425+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2718127830 msgr2=0x7f2718063180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.425+0000 7f270dffb640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2718127830 0x7f2718063180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f27180abf80 tx=0x7f27080046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.425+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2718127830 msgr2=0x7f2718063180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:56.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.425+0000 7f270dffb640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2718127830 0x7f2718063180 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271f985640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:56.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271f985640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f27180636c0 0x7f271802cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271f985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271f985640 1 -- 172.21.15.98:0/3303541125 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2718050830 con 0x7f27180636c0 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271cf20640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f27180636c0 0x7f271802cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271cf20640 10 monclient: get_auth_request con 0x7f27180636c0 auth_method 0 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271cf20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271cf20640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271cf20640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271cf20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271cf20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271cf20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271cf20640 10 monclient(hunting): handle_auth_done global_id 4589 payload 824 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271cf20640 1 -- 172.21.15.98:0/3303541125 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2718126750 con 0x7f27180636c0 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f271cf20640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f27180636c0 0x7f271802cb40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f27080097c0 tx=0x7f2708031000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f270dffb640 10 client.4549.objecter ms_handle_connect 0x7f27180636c0 2024-06-11T19:09:56.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.431+0000 7f270dffb640 10 client.4549.objecter resend_mon_ops 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f270804c2b0 con 0x7f27180636c0 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f270dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f270dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f270dffb640 10 monclient: dump: 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.453 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.454 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9431 (secure 0 0 0) 0x7f270804c2b0 con 0x7f27180636c0 2024-06-11T19:09:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f270dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f270dffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f27180636c0 msgr2=0x7f271802cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f270dffb640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f27180636c0 0x7f271802cb40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f27080097c0 tx=0x7f2708031000 comp rx=0 tx=0).stop 2024-06-11T19:09:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f270dffb640 1 -- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f27180636c0 msgr2=0x7f271802cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f270dffb640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f27180636c0 0x7f271802cb40 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f27080097c0 tx=0x7f2708031000 comp rx=0 tx=0).stop 2024-06-11T19:09:56.455 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 10, 2024-06-11T19:09:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 46669, 2024-06-11T19:09:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.462 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.465 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.467 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.468 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.470 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:56.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:56.473 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:56.475 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.476 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.478 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:56.478 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 10 librados: watch_flush enter 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 10 librados: watch_flush exit 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 20 client.4549.objecter shutdown clearing up homeless session... 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 10 client.4549.objecter successfully canceled tick 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 1 -- 172.21.15.98:0/3303541125 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f26fc03fdb0 msgr2=0x7f26fc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f26fc03fdb0 0x7f26fc042270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2710000f60 tx=0x7f2710009040 comp rx=0 tx=0).stop 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 10 monclient: shutdown 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 1 -- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a2530 msgr2=0x7f27181cddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a2530 0x7f27181cddc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f271400ba10 tx=0x7f271400bed0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271df22640 1 -- 172.21.15.98:0/3303541125 reap_dead start 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 1 -- 172.21.15.98:0/3303541125 shutdown_connections 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 1 --2- 172.21.15.98:0/3303541125 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f27180a2530 0x7f27181cddc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 1 -- 172.21.15.98:0/3303541125 >> 172.21.15.98:0/3303541125 conn(0x7f27180ab850 msgr2=0x7f27181bef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 1 -- 172.21.15.98:0/3303541125 shutdown_connections 2024-06-11T19:09:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 1 -- 172.21.15.98:0/3303541125 wait complete. 2024-06-11T19:09:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.432+0000 7f271f985640 1 librados: shutdown 2024-06-11T19:09:56.481 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:09:56.481 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-06-11T19:09:56.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:09:56.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.582+0000 7f3e92f5f640 10 monclient: get_monmap_and_config 2024-06-11T19:09:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.582+0000 7f3e92f5f640 10 monclient: build_initial_monmap 2024-06-11T19:09:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.582+0000 7f3e92f5f640 10 monclient: monmap: 2024-06-11T19:09:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:56.583068+0000 2024-06-11T19:09:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:56.583068+0000 2024-06-11T19:09:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:56.582 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:56.583 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.582+0000 7f3e92f5f640 1 Processor -- start 2024-06-11T19:09:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.582+0000 7f3e92f5f640 1 -- start start 2024-06-11T19:09:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.582+0000 7f3e92f5f640 10 monclient: init 2024-06-11T19:09:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 10 monclient: _add_conns ranks=[0,5,6,4,1,7,10,3,2,8,9] 2024-06-11T19:09:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e8c0a2f40 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3e8c0a4fc0 0x7f3e8c13b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 10 monclient(hunting): picked mon.noname-h con 0x7f3e8c0a4fc0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e8c0a53a0 0x7f3e8c13df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 10 monclient(hunting): picked mon.noname-d con 0x7f3e8c0a53a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e8c0a0ed0 con 0x7f3e8c0a2f40 2024-06-11T19:09:56.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f3e8c0a3320 con 0x7f3e8c0a4fc0 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3e8c0a2460 con 0x7f3e8c0a53a0 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e92f5f640 10 monclient(hunting): authenticate will time out at 806.390386s 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e914fc640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e8c0a53a0 msgr2=0x7f3e8c13df40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e914fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e8c0a53a0 0x7f3e8c13df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e90cfb640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3e8c0a2f40 0x7f3e8c0a48b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39872/0 (socket says 172.21.15.98:39872) 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e90cfb640 1 -- 172.21.15.98:0/1939384200 learned_addr learned my addr 172.21.15.98:0/1939384200 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 1 --2- 172.21.15.98:0/1939384200 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3e8c0a4fc0 0x7f3e8c13b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 10 monclient(hunting): get_auth_request con 0x7f3e8c0a4fc0 auth_method 0 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8b7fe640 1 -- 172.21.15.98:0/1939384200 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3378825789 0 0) 0x7f3e8c0a0ed0 con 0x7f3e8c0a2f40 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8b7fe640 10 monclient(hunting): _negotiate 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8b7fe640 10 monclient(hunting): my global_id is 4563 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8b7fe640 1 -- 172.21.15.98:0/1939384200 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e74003520 con 0x7f3e8c0a2f40 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 10 monclient(hunting): handle_auth_done global_id 4397 payload 274 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 1 -- 172.21.15.98:0/1939384200 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e8c0a53a0 msgr2=0x7f3e8c13df40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 1 --2- 172.21.15.98:0/1939384200 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e8c0a53a0 0x7f3e8c13df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 1 -- 172.21.15.98:0/1939384200 >> v1:172.21.15.98:6789/0 conn(0x7f3e8c0a2f40 legacy=0x7f3e8c0a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 1 monclient: found mon.noname-h 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.583+0000 7f3e8bfff640 1 -- 172.21.15.98:0/1939384200 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8c0a2180 con 0x7f3e8c0a4fc0 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8bfff640 1 --2- 172.21.15.98:0/1939384200 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3e8c0a4fc0 0x7f3e8c13b9f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009800 tx=0x7f3e8002b790 comp rx=0 tx=0).ready entity=mon.5 client_cookie=e6d777ddfca2ff18 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 1 -- 172.21.15.98:0/1939384200 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e80039070 con 0x7f3e8c0a4fc0 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 10 monclient: dump: 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:09:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e92f5f640 5 monclient: authenticate success, global_id 4397 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e92f5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 1 -- 172.21.15.98:0/1939384200 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e80031960 con 0x7f3e8c0a4fc0 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e92f5f640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e92f5f640 4 set_mon_vals no callback set 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 1 -- 172.21.15.98:0/1939384200 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e80031b10 con 0x7f3e8c0a4fc0 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.584+0000 7f3e8b7fe640 10 monclient: dump: 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.600 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.585+0000 7f3e92f5f640 10 monclient: shutdown 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.585+0000 7f3e92f5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.585+0000 7f3e92f5f640 1 -- 172.21.15.98:0/1939384200 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3e8c0a4fc0 msgr2=0x7f3e8c13b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.585+0000 7f3e92f5f640 1 --2- 172.21.15.98:0/1939384200 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3e8c0a4fc0 0x7f3e8c13b9f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009800 tx=0x7f3e8002b790 comp rx=0 tx=0).stop 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.585+0000 7f3e92f5f640 1 -- 172.21.15.98:0/1939384200 shutdown_connections 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.585+0000 7f3e92f5f640 1 --2- 172.21.15.98:0/1939384200 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e8c0a53a0 0x7f3e8c13df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.585+0000 7f3e92f5f640 1 --2- 172.21.15.98:0/1939384200 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3e8c0a4fc0 0x7f3e8c13b9f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.585+0000 7f3e92f5f640 1 -- 172.21.15.98:0/1939384200 >> 172.21.15.98:0/1939384200 conn(0x7f3e8c0a95b0 msgr2=0x7f3e8c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.585+0000 7f3e92f5f640 1 -- 172.21.15.98:0/1939384200 shutdown_connections 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.585+0000 7f3e92f5f640 1 -- 172.21.15.98:0/1939384200 wait complete. 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.585+0000 7f3e92f5f640 10 monclient: build_initial_monmap 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.585+0000 7f3e92f5f640 10 monclient: monmap: 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:56.601 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:56.602 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 1 librados: starting msgr at 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 1 librados: starting objecter 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 1 Processor -- start 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 1 -- start start 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 1 librados: setting wanted keys 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 1 librados: calling monclient init 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 10 monclient: init 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 10 monclient: _add_conns ranks=[7,4,8,0,5,1,6,3,2] 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3e8c0a2f40 0x7f3e8c136760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 10 monclient(hunting): picked mon.noname-i con 0x7f3e8c0a2f40 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e8c0a4fc0 0x7f3e8c06bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 10 monclient(hunting): picked mon.noname-c con 0x7f3e8c0a4fc0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e8c0a53a0 0x7f3e8c06c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 10 monclient(hunting): picked mon.noname-e con 0x7f3e8c0a53a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3e8c13e600 con 0x7f3e8c0a4fc0 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f3e8c13e480 con 0x7f3e8c0a2f40 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3e8c13e780 con 0x7f3e8c0a53a0 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e92f5f640 10 monclient(hunting): authenticate will time out at 806.393903s 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.586+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e8c0a4fc0 0x7f3e8c06bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e8c0a53a0 0x7f3e8c06c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e8c0a4fc0 0x7f3e8c06bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:48250/0 (socket says 172.21.15.98:48250) 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e8bfff640 1 -- 172.21.15.98:0/3222318240 learned_addr learned my addr 172.21.15.98:0/3222318240 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e8bfff640 10 monclient(hunting): get_auth_request con 0x7f3e8c0a4fc0 auth_method 0 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e8bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e90cfb640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3e8c0a2f40 0x7f3e8c136760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 10 monclient(hunting): get_auth_request con 0x7f3e8c0a53a0 auth_method 0 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e90cfb640 10 monclient(hunting): get_auth_request con 0x7f3e8c0a2f40 auth_method 0 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e90cfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e90cfb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e90cfb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e90cfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e90cfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e90cfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 10 monclient(hunting): handle_auth_done global_id 4598 payload 824 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 1 -- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3e8c0a2f40 msgr2=0x7f3e8c136760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3e8c0a2f40 0x7f3e8c136760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 1 -- 172.21.15.98:0/3222318240 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e8c0a4fc0 msgr2=0x7f3e8c06bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e8c0a4fc0 0x7f3e8c06bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 10 monclient: _finish_hunting 0 2024-06-11T19:09:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 1 monclient: found mon.noname-e 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 1 -- 172.21.15.98:0/3222318240 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8c13d210 con 0x7f3e8c0a53a0 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e8bfff640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e8c0a4fc0 0x7f3e8c06bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e914fc640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e8c0a53a0 0x7f3e8c06c4a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3e84003d00 tx=0x7f3e84009ed0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 10 client.?.objecter ms_handle_connect 0x7f3e8c0a53a0 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e8400cbf0 con 0x7f3e8c0a53a0 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 10 monclient: dump: 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 10 monclient: _finish_auth 0 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 10 monclient: _check_auth_tickets 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e84010030 con 0x7f3e8c0a53a0 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e92f5f640 5 monclient: authenticate success, global_id 4598 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e92f5f640 10 monclient: _renew_subs 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e92f5f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e92f5f640 1 -- 172.21.15.98:0/3222318240 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8c13cdf0 con 0x7f3e8c0a53a0 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e8400cdb0 con 0x7f3e8c0a53a0 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e897fa640 10 monclient: dump: 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.609 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.610 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.611 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e92f5f640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:56.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e92f5f640 10 monclient: _renew_subs 2024-06-11T19:09:56.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e92f5f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:56.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e8affd640 4 set_mon_vals no callback set 2024-06-11T19:09:56.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e92f5f640 1 -- 172.21.15.98:0/3222318240 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e8c09f400 con 0x7f3e8c0a53a0 2024-06-11T19:09:56.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.587+0000 7f3e92f5f640 1 librados: init done 2024-06-11T19:09:56.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e92f5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:56.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e92f5f640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c136ca0 0x7f3e8c06c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e92f5f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e92f5f640 1 -- 172.21.15.98:0/3222318240 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c050830 con 0x7f3e8c136ca0 2024-06-11T19:09:56.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3e840153e0 con 0x7f3e8c0a53a0 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e90cfb640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c136ca0 0x7f3e8c06c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e90cfb640 10 monclient: get_auth_request con 0x7f3e8c136ca0 auth_method 0 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e90cfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e90cfb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e90cfb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e90cfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e90cfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e90cfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e897fa640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3e7803fdb0 0x7f3e78042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f3e84051670 con 0x7f3e8c0a53a0 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e897fa640 10 client.4598.objecter ms_dispatch 0x7f3e8c000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e897fa640 3 client.4598.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e897fa640 3 client.4598.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e897fa640 20 client.4598.objecter dump_active .. 0 homeless 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e90cfb640 10 monclient(hunting): handle_auth_done global_id 4483 payload 824 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e8bfff640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3e7803fdb0 0x7f3e78042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e90cfb640 1 -- 172.21.15.98:0/3222318240 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e8c0a8890 con 0x7f3e8c136ca0 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.588+0000 7f3e8bfff640 10 monclient: get_auth_request con 0x7f3e7803fdb0 auth_method 0 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e90cfb640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c136ca0 0x7f3e8c06c9e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c001540 tx=0x7f3e7c001a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 10 client.4598.objecter ms_handle_connect 0x7f3e8c136ca0 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 10 client.4598.objecter resend_mon_ops 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e7c017030 con 0x7f3e8c136ca0 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e8bfff640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3e7803fdb0 0x7f3e78042270 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3e8002b7d0 tx=0x7f3e80008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 10 monclient: dump: 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.615 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 10 client.4598.objecter ms_handle_connect 0x7f3e7803fdb0 2024-06-11T19:09:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3e7c00f6b0 con 0x7f3e8c136ca0 2024-06-11T19:09:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c136ca0 msgr2=0x7f3e8c06c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c136ca0 0x7f3e8c06c9e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c001540 tx=0x7f3e7c001a00 comp rx=0 tx=0).stop 2024-06-11T19:09:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c136ca0 msgr2=0x7f3e8c06c9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.589+0000 7f3e897fa640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c136ca0 0x7f3e8c06c9e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.595+0000 7f3e92f5f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.595+0000 7f3e92f5f640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c06cf20 0x7f3e8c0a6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.595+0000 7f3e92f5f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.595+0000 7f3e92f5f640 1 -- 172.21.15.98:0/3222318240 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c0692c0 con 0x7f3e8c06cf20 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e90cfb640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c06cf20 0x7f3e8c0a6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e90cfb640 10 monclient: get_auth_request con 0x7f3e8c06cf20 auth_method 0 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e90cfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e90cfb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e90cfb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e90cfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e90cfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e90cfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e90cfb640 10 monclient(hunting): handle_auth_done global_id 4492 payload 824 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e90cfb640 1 -- 172.21.15.98:0/3222318240 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3e8c06d330 con 0x7f3e8c06cf20 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e90cfb640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c06cf20 0x7f3e8c0a6ca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c001540 tx=0x7f3e7c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 10 client.4598.objecter ms_handle_connect 0x7f3e8c06cf20 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 10 client.4598.objecter resend_mon_ops 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e7c01dc80 con 0x7f3e8c06cf20 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 10 monclient: dump: 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9431 (secure 0 0 0) 0x7f3e7c01c660 con 0x7f3e8c06cf20 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c06cf20 msgr2=0x7f3e8c0a6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c06cf20 0x7f3e8c0a6ca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c001540 tx=0x7f3e7c0057f0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 1 -- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c06cf20 msgr2=0x7f3e8c0a6ca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.596+0000 7f3e897fa640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e8c06cf20 0x7f3e8c0a6ca0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c001540 tx=0x7f3e7c0057f0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.620 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 10, 2024-06-11T19:09:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 46815, 2024-06-11T19:09:56.620 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:56.621 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.622 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:56.624 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.625 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:56.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.634 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:56.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:56.639 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.641 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.642 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:09:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.643 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:56.643 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:56.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 10 librados: watch_flush enter 2024-06-11T19:09:56.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 10 librados: watch_flush exit 2024-06-11T19:09:56.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 20 client.4598.objecter shutdown clearing up homeless session... 2024-06-11T19:09:56.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 10 client.4598.objecter successfully canceled tick 2024-06-11T19:09:56.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 1 -- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3e7803fdb0 msgr2=0x7f3e78042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3e7803fdb0 0x7f3e78042270 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3e8002b7d0 tx=0x7f3e80008030 comp rx=0 tx=0).stop 2024-06-11T19:09:56.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 10 monclient: shutdown 2024-06-11T19:09:56.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:56.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 1 -- 172.21.15.98:0/3222318240 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e8c0a53a0 msgr2=0x7f3e8c06c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 1 --2- 172.21.15.98:0/3222318240 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e8c0a53a0 0x7f3e8c06c4a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3e84003d00 tx=0x7f3e84009ed0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e914fc640 1 -- 172.21.15.98:0/3222318240 reap_dead start 2024-06-11T19:09:56.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e914fc640 1 -- 172.21.15.98:0/3222318240 reap_dead start 2024-06-11T19:09:56.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 1 -- 172.21.15.98:0/3222318240 shutdown_connections 2024-06-11T19:09:56.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 1 -- 172.21.15.98:0/3222318240 >> 172.21.15.98:0/3222318240 conn(0x7f3e8c0a95b0 msgr2=0x7f3e8c09dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:56.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 1 -- 172.21.15.98:0/3222318240 shutdown_connections 2024-06-11T19:09:56.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 1 -- 172.21.15.98:0/3222318240 wait complete. 2024-06-11T19:09:56.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.597+0000 7f3e92f5f640 1 librados: shutdown 2024-06-11T19:09:56.647 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:09:56.647 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-06-11T19:09:56.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:09:56.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.747+0000 7fd6ef47a640 10 monclient: get_monmap_and_config 2024-06-11T19:09:56.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.747+0000 7fd6ef47a640 10 monclient: build_initial_monmap 2024-06-11T19:09:56.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.747+0000 7fd6ef47a640 10 monclient: monmap: 2024-06-11T19:09:56.747 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:56.747 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:56.748656+0000 2024-06-11T19:09:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:56.748656+0000 2024-06-11T19:09:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:56.749 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 1 Processor -- start 2024-06-11T19:09:56.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 1 -- start start 2024-06-11T19:09:56.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 10 monclient: init 2024-06-11T19:09:56.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:56.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 10 monclient: _add_conns ranks=[2,7,3,8,5,4,1,0,10,9,6] 2024-06-11T19:09:56.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e80a52a0 0x7fd6e80a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 10 monclient(hunting): picked mon.noname-b con 0x7fd6e80a52a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:56.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd6e809d520 0x7fd6e80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 10 monclient(hunting): picked mon.noname-i con 0x7fd6e809d520 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:56.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e8073830 0x7fd6e80a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 10 monclient(hunting): picked mon.noname-g con 0x7fd6e8073830 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd6e809daa0 con 0x7fd6e80a52a0 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd6e809d2e0 con 0x7fd6e8073830 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd6e80a5680 con 0x7fd6e809d520 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.748+0000 7fd6ef47a640 10 monclient(hunting): authenticate will time out at 806.555931s 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e80a52a0 0x7fd6e80a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e80a52a0 0x7fd6e80a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50724/0 (socket says 172.21.15.98:50724) 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 1 -- 172.21.15.98:0/1383687288 learned_addr learned my addr 172.21.15.98:0/1383687288 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 10 monclient(hunting): get_auth_request con 0x7fd6e80a52a0 auth_method 0 2024-06-11T19:09:56.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eda17640 1 --2- 172.21.15.98:0/1383687288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e8073830 0x7fd6e80a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eca15640 1 --2- 172.21.15.98:0/1383687288 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd6e809d520 0x7fd6e80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eda17640 10 monclient(hunting): get_auth_request con 0x7fd6e8073830 auth_method 0 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eda17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eda17640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eda17640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eca15640 10 monclient(hunting): get_auth_request con 0x7fd6e809d520 auth_method 0 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eca15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eca15640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eca15640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eda17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eda17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eda17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eca15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eca15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6eca15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 10 monclient(hunting): handle_auth_done global_id 4538 payload 274 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 1 -- 172.21.15.98:0/1383687288 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd6e809d520 msgr2=0x7fd6e80a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 1 --2- 172.21.15.98:0/1383687288 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd6e809d520 0x7fd6e80a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 1 -- 172.21.15.98:0/1383687288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e8073830 msgr2=0x7fd6e80a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 1 --2- 172.21.15.98:0/1383687288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e8073830 0x7fd6e80a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 10 monclient: _finish_hunting 0 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 1 monclient: found mon.noname-b 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.749+0000 7fd6ed216640 1 -- 172.21.15.98:0/1383687288 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e809e9d0 con 0x7fd6e80a52a0 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ed216640 1 --2- 172.21.15.98:0/1383687288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e80a52a0 0x7fd6e80a1910 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40098d0 tx=0x7fd6e402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c454f25438a86ff server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 1 -- 172.21.15.98:0/1383687288 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6e4039070 con 0x7fd6e80a52a0 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 10 monclient: dump: 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.768 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 10 monclient: _finish_auth 0 2024-06-11T19:09:56.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 5 monclient: authenticate success, global_id 4538 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 1 -- 172.21.15.98:0/1383687288 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e4031960 con 0x7fd6e80a52a0 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 4 set_mon_vals no callback set 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 1 -- 172.21.15.98:0/1383687288 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6e4032380 con 0x7fd6e80a52a0 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6dffff640 10 monclient: dump: 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.770 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 10 monclient: shutdown 2024-06-11T19:09:56.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 1 -- 172.21.15.98:0/1383687288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e80a52a0 msgr2=0x7fd6e80a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 1 --2- 172.21.15.98:0/1383687288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e80a52a0 0x7fd6e80a1910 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40098d0 tx=0x7fd6e402b790 comp rx=0 tx=0).stop 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 1 -- 172.21.15.98:0/1383687288 shutdown_connections 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 1 --2- 172.21.15.98:0/1383687288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e8073830 0x7fd6e80a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 1 --2- 172.21.15.98:0/1383687288 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd6e809d520 0x7fd6e80a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 1 --2- 172.21.15.98:0/1383687288 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6e80a52a0 0x7fd6e80a1910 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.750+0000 7fd6ef47a640 1 -- 172.21.15.98:0/1383687288 >> 172.21.15.98:0/1383687288 conn(0x7fd6e80a4780 msgr2=0x7fd6e80a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.751+0000 7fd6ef47a640 1 -- 172.21.15.98:0/1383687288 shutdown_connections 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.751+0000 7fd6ef47a640 1 -- 172.21.15.98:0/1383687288 wait complete. 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.751+0000 7fd6ef47a640 10 monclient: build_initial_monmap 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.751+0000 7fd6ef47a640 10 monclient: monmap: 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:56.772 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.751+0000 7fd6ef47a640 1 librados: starting msgr at 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.751+0000 7fd6ef47a640 1 librados: starting objecter 2024-06-11T19:09:56.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.751+0000 7fd6ef47a640 1 Processor -- start 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.751+0000 7fd6ef47a640 1 -- start start 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.751+0000 7fd6ef47a640 1 librados: setting wanted keys 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.751+0000 7fd6ef47a640 1 librados: calling monclient init 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.751+0000 7fd6ef47a640 10 monclient: init 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 10 monclient: _add_conns ranks=[4,8,1,7,5,3,6,2,0] 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6e8073830 0x7fd6e81cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 10 monclient(hunting): picked mon.noname-c con 0x7fd6e8073830 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd6e809d520 0x7fd6e8134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 10 monclient(hunting): picked mon.noname-e con 0x7fd6e809d520 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6e80a52a0 0x7fd6e8134aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 10 monclient(hunting): picked mon.noname-f con 0x7fd6e80a52a0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:09:56.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e80a1fd0 con 0x7fd6e80a52a0 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd6e80a1e50 con 0x7fd6e8073830 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd6e80a2150 con 0x7fd6e809d520 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ef47a640 10 monclient(hunting): authenticate will time out at 806.559463s 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6e8073830 0x7fd6e81cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eca15640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd6e809d520 0x7fd6e8134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6e8073830 0x7fd6e81cdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:48256/0 (socket says 172.21.15.98:48256) 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 1 -- 172.21.15.98:0/571152984 learned_addr learned my addr 172.21.15.98:0/571152984 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 10 monclient(hunting): get_auth_request con 0x7fd6e8073830 auth_method 0 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eca15640 10 monclient(hunting): get_auth_request con 0x7fd6e809d520 auth_method 0 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eca15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eca15640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eca15640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eca15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eca15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eda17640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6e80a52a0 0x7fd6e8134aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eca15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eda17640 10 monclient(hunting): get_auth_request con 0x7fd6e80a52a0 auth_method 0 2024-06-11T19:09:56.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eda17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eda17640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eda17640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eda17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eda17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6eda17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 10 monclient(hunting): handle_auth_done global_id 4567 payload 824 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 1 -- 172.21.15.98:0/571152984 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd6e809d520 msgr2=0x7fd6e8134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd6e809d520 0x7fd6e8134560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 1 -- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6e80a52a0 msgr2=0x7fd6e8134aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6e80a52a0 0x7fd6e8134aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 10 monclient: _finish_hunting 0 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 1 monclient: found mon.noname-c 2024-06-11T19:09:56.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 1 -- 172.21.15.98:0/571152984 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e8134fe0 con 0x7fd6e8073830 2024-06-11T19:09:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ed216640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6e8073830 0x7fd6e81cdbb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40098d0 tx=0x7fd6e40046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ddffb640 10 client.?.objecter ms_handle_connect 0x7fd6e8073830 2024-06-11T19:09:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.752+0000 7fd6ddffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6e4043020 con 0x7fd6e8073830 2024-06-11T19:09:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 10 monclient: dump: 2024-06-11T19:09:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.778 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.779 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 10 monclient: _finish_auth 0 2024-06-11T19:09:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 10 monclient: _check_auth_tickets 2024-06-11T19:09:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e40044b0 con 0x7fd6e8073830 2024-06-11T19:09:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ef47a640 5 monclient: authenticate success, global_id 4567 2024-06-11T19:09:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ef47a640 10 monclient: _renew_subs 2024-06-11T19:09:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ef47a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ef47a640 1 -- 172.21.15.98:0/571152984 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e81256c0 con 0x7fd6e8073830 2024-06-11T19:09:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6e4039040 con 0x7fd6e8073830 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ddffb640 10 monclient: dump: 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6df7fe640 4 set_mon_vals no callback set 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ef47a640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ef47a640 10 monclient: _renew_subs 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ef47a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ef47a640 1 -- 172.21.15.98:0/571152984 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e8124940 con 0x7fd6e8073830 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.753+0000 7fd6ef47a640 1 librados: init done 2024-06-11T19:09:56.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd6e4046020 con 0x7fd6e8073830 2024-06-11T19:09:56.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ef47a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:56.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ef47a640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806f510 0x7fd6e806f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ef47a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ef47a640 1 -- 172.21.15.98:0/571152984 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd6e8050830 con 0x7fd6e806f510 2024-06-11T19:09:56.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eca15640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806f510 0x7fd6e806f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eca15640 10 monclient: get_auth_request con 0x7fd6e806f510 auth_method 0 2024-06-11T19:09:56.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ddffb640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6cc03fdb0 0x7fd6cc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eca15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eca15640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eca15640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7fd6e4072a40 con 0x7fd6e8073830 2024-06-11T19:09:56.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ddffb640 10 client.4567.objecter ms_dispatch 0x7fd6e8000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ddffb640 3 client.4567.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ddffb640 3 client.4567.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eca15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eca15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eca15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ddffb640 20 client.4567.objecter dump_active .. 0 homeless 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eda17640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6cc03fdb0 0x7fd6cc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eda17640 10 monclient: get_auth_request con 0x7fd6cc03fdb0 auth_method 0 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eca15640 10 monclient(hunting): handle_auth_done global_id 4449 payload 824 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eca15640 1 -- 172.21.15.98:0/571152984 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6e80710c0 con 0x7fd6e806f510 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ddffb640 10 client.4567.objecter ms_handle_connect 0x7fd6e806f510 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6eca15640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806f510 0x7fd6e806f8f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80a8d40 tx=0x7fd6d8011cf0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ddffb640 10 client.4567.objecter resend_mon_ops 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6d8004030 con 0x7fd6e806f510 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.754+0000 7fd6ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.755+0000 7fd6ddffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.755+0000 7fd6ddffb640 10 monclient: dump: 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.785 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.755+0000 7fd6eda17640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6cc03fdb0 0x7fd6cc042270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8073c50 tx=0x7fd6e000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.755+0000 7fd6ddffb640 10 client.4567.objecter ms_handle_connect 0x7fd6cc03fdb0 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.755+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd6d800fa70 con 0x7fd6e806f510 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.755+0000 7fd6ddffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.755+0000 7fd6ddffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.755+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806f510 msgr2=0x7fd6e806f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.755+0000 7fd6ddffb640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806f510 0x7fd6e806f8f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80a8d40 tx=0x7fd6d8011cf0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.755+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806f510 msgr2=0x7fd6e806f8f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.755+0000 7fd6ddffb640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806f510 0x7fd6e806f8f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.761+0000 7fd6ef47a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.761+0000 7fd6ef47a640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806fe30 0x7fd6e81bb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.761+0000 7fd6ef47a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.761+0000 7fd6ef47a640 1 -- 172.21.15.98:0/571152984 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd6e80a1650 con 0x7fd6e806fe30 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.761+0000 7fd6eca15640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806fe30 0x7fd6e81bb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.761+0000 7fd6eca15640 10 monclient: get_auth_request con 0x7fd6e806fe30 auth_method 0 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.761+0000 7fd6eca15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.761+0000 7fd6eca15640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.761+0000 7fd6eca15640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.761+0000 7fd6eca15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.761+0000 7fd6eca15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.761+0000 7fd6eca15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6eca15640 10 monclient(hunting): handle_auth_done global_id 4458 payload 824 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6eca15640 1 -- 172.21.15.98:0/571152984 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6e80692c0 con 0x7fd6e806fe30 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6eca15640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806fe30 0x7fd6e81bb4c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8006a90 tx=0x7fd6d80046c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 10 client.4567.objecter ms_handle_connect 0x7fd6e806fe30 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 10 client.4567.objecter resend_mon_ops 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6d80148b0 con 0x7fd6e806fe30 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 10 monclient: dump: 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9261 (secure 0 0 0) 0x7fd6d80273a0 con 0x7fd6e806fe30 2024-06-11T19:09:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806fe30 msgr2=0x7fd6e81bb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806fe30 0x7fd6e81bb4c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8006a90 tx=0x7fd6d80046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 1 -- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806fe30 msgr2=0x7fd6e81bb4c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ddffb640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6e806fe30 0x7fd6e81bb4c0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8006a90 tx=0x7fd6d80046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.790 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 10, 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 46980, 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-06-11T19:09:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:56.806 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:56.811 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.812 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.813 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:56.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.813 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.813 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.813 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:09:56.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.813 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.813 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:56.813 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:56.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ef47a640 10 librados: watch_flush enter 2024-06-11T19:09:56.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.762+0000 7fd6ef47a640 10 librados: watch_flush exit 2024-06-11T19:09:56.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 20 client.4567.objecter shutdown clearing up homeless session... 2024-06-11T19:09:56.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 10 client.4567.objecter successfully canceled tick 2024-06-11T19:09:56.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 1 -- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6cc03fdb0 msgr2=0x7fd6cc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6cc03fdb0 0x7fd6cc042270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8073c50 tx=0x7fd6e000b040 comp rx=0 tx=0).stop 2024-06-11T19:09:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 10 monclient: shutdown 2024-06-11T19:09:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 1 -- 172.21.15.98:0/571152984 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6e8073830 msgr2=0x7fd6e81cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6e8073830 0x7fd6e81cdbb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40098d0 tx=0x7fd6e40046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6eda17640 1 -- 172.21.15.98:0/571152984 reap_dead start 2024-06-11T19:09:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 1 -- 172.21.15.98:0/571152984 shutdown_connections 2024-06-11T19:09:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 1 --2- 172.21.15.98:0/571152984 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6e8073830 0x7fd6e81cdbb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 1 -- 172.21.15.98:0/571152984 >> 172.21.15.98:0/571152984 conn(0x7fd6e80a4780 msgr2=0x7fd6e80a67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 1 -- 172.21.15.98:0/571152984 shutdown_connections 2024-06-11T19:09:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 1 -- 172.21.15.98:0/571152984 wait complete. 2024-06-11T19:09:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.763+0000 7fd6ef47a640 1 librados: shutdown 2024-06-11T19:09:56.817 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:09:56.817 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-06-11T19:09:56.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:09:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.917+0000 7f5f88b9c640 10 monclient: get_monmap_and_config 2024-06-11T19:09:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.917+0000 7f5f88b9c640 10 monclient: build_initial_monmap 2024-06-11T19:09:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.917+0000 7f5f88b9c640 10 monclient: monmap: 2024-06-11T19:09:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:56.918536+0000 2024-06-11T19:09:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:56.918536+0000 2024-06-11T19:09:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:56.918 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 1 Processor -- start 2024-06-11T19:09:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 1 -- start start 2024-06-11T19:09:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 10 monclient: init 2024-06-11T19:09:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 10 monclient: _add_conns ranks=[6,10,9,7,8,0,1,2,4,5,3] 2024-06-11T19:09:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5f840a4920 0x7f5f840a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 10 monclient(hunting): picked mon.noname-d con 0x7f5f840a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5f840a5240 0x7f5f840a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 10 monclient(hunting): picked mon.noname-f con 0x7f5f840a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:09:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5f840a2530 0x7f5f840a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 10 monclient(hunting): picked mon.noname-a con 0x7f5f840a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:09:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5f840a7140 con 0x7f5f840a2530 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f5f840a2b50 con 0x7f5f840a5240 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5f840a2e30 con 0x7f5f840a4920 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f88b9c640 10 monclient(hunting): authenticate will time out at 806.725842s 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f8259c640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5f840a4920 msgr2=0x7f5f840a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f8259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5f840a4920 0x7f5f840a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.918+0000 7f5f82d9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5f840a2530 0x7f5f840a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5f840a2530 0x7f5f840a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55146/0 (socket says 172.21.15.98:55146) 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 1 -- 172.21.15.98:0/940724562 learned_addr learned my addr 172.21.15.98:0/940724562 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 10 monclient(hunting): get_auth_request con 0x7f5f840a2530 auth_method 0 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f81d9b640 1 --2- 172.21.15.98:0/940724562 >> v2:172.21.15.117:3300/0 conn(0x7f5f840a5240 0x7f5f840a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f81d9b640 10 monclient(hunting): get_auth_request con 0x7f5f840a5240 auth_method 0 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f81d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f81d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f81d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f81d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f81d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 10 monclient(hunting): handle_auth_done global_id 4572 payload 274 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 1 -- 172.21.15.98:0/940724562 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5f840a4920 msgr2=0x7f5f840a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 1 --2- 172.21.15.98:0/940724562 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5f840a4920 0x7f5f840a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 1 -- 172.21.15.98:0/940724562 >> v2:172.21.15.117:3300/0 conn(0x7f5f840a5240 msgr2=0x7f5f840a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 1 --2- 172.21.15.98:0/940724562 >> v2:172.21.15.117:3300/0 conn(0x7f5f840a5240 0x7f5f840a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 1 monclient: found mon.noname-a 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.919+0000 7f5f82d9d640 1 -- 172.21.15.98:0/940724562 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f840a3110 con 0x7f5f840a2530 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f82d9d640 1 --2- 172.21.15.98:0/940724562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f840a2530 0x7f5f840a5ee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5f840a8420 tx=0x7f5f7c02bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8477243d44891e server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 1 -- 172.21.15.98:0/940724562 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5f7c02cbf0 con 0x7f5f840a2530 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:09:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 10 monclient: dump: 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 10 monclient: _finish_auth 0 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 10 monclient: _check_auth_tickets 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 1 -- 172.21.15.98:0/940724562 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f7c02cdb0 con 0x7f5f840a2530 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 5 monclient: authenticate success, global_id 4572 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 1 -- 172.21.15.98:0/940724562 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5f7c034860 con 0x7f5f840a2530 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f8159a640 10 monclient: dump: 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 4 set_mon_vals no callback set 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 10 monclient: shutdown 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 1 -- 172.21.15.98:0/940724562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f840a2530 msgr2=0x7f5f840a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 1 --2- 172.21.15.98:0/940724562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f840a2530 0x7f5f840a5ee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5f840a8420 tx=0x7f5f7c02bde0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 1 -- 172.21.15.98:0/940724562 shutdown_connections 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 1 --2- 172.21.15.98:0/940724562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f840a2530 0x7f5f840a5ee0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 1 --2- 172.21.15.98:0/940724562 >> v2:172.21.15.117:3300/0 conn(0x7f5f840a5240 0x7f5f840a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 1 --2- 172.21.15.98:0/940724562 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5f840a4920 0x7f5f840a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 1 -- 172.21.15.98:0/940724562 >> 172.21.15.98:0/940724562 conn(0x7f5f840ab850 msgr2=0x7f5f840a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 1 -- 172.21.15.98:0/940724562 shutdown_connections 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.920+0000 7f5f88b9c640 1 -- 172.21.15.98:0/940724562 wait complete. 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.921+0000 7f5f88b9c640 10 monclient: build_initial_monmap 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.921+0000 7f5f88b9c640 10 monclient: monmap: 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.921+0000 7f5f88b9c640 1 librados: starting msgr at 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.921+0000 7f5f88b9c640 1 librados: starting objecter 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.921+0000 7f5f88b9c640 1 Processor -- start 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 1 -- start start 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 1 librados: setting wanted keys 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 1 librados: calling monclient init 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 10 monclient: init 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 10 monclient: _add_conns ranks=[6,8,7,3,4,1,0,2,5] 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5f840a2530 0x7f5f841cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 10 monclient(hunting): picked mon.noname-d con 0x7f5f840a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f840a4920 0x7f5f841ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 10 monclient(hunting): picked mon.noname-e con 0x7f5f840a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5f840a5240 0x7f5f841252e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f8259c640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5f840a2530 msgr2=0x7f5f841cdb00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 10 monclient(hunting): picked mon.noname-i con 0x7f5f840a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5f840a6da0 con 0x7f5f840a2530 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f8259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5f840a2530 0x7f5f841cdb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f5f84138e50 con 0x7f5f840a5240 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5f840a6420 con 0x7f5f840a4920 2024-06-11T19:09:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f88b9c640 10 monclient(hunting): authenticate will time out at 806.729524s 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f840a4920 0x7f5f841ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f840a4920 0x7f5f841ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:53210/0 (socket says 172.21.15.98:53210) 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 1 -- 172.21.15.98:0/4270840112 learned_addr learned my addr 172.21.15.98:0/4270840112 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 10 monclient(hunting): get_auth_request con 0x7f5f840a4920 auth_method 0 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f82d9d640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5f840a5240 0x7f5f841252e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f82d9d640 10 monclient(hunting): get_auth_request con 0x7f5f840a5240 auth_method 0 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f82d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 10 monclient(hunting): handle_auth_done global_id 4616 payload 824 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 1 -- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5f840a5240 msgr2=0x7f5f841252e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5f840a5240 0x7f5f841252e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 1 -- 172.21.15.98:0/4270840112 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5f840a2530 msgr2=0x7f5f841cdb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5f840a2530 0x7f5f841cdb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 1 monclient: found mon.noname-e 2024-06-11T19:09:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 1 -- 172.21.15.98:0/4270840112 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f841291c0 con 0x7f5f840a4920 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.922+0000 7f5f81d9b640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f840a4920 0x7f5f841ce040 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00e870 tx=0x7f5f6c00ed30 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 10 client.?.objecter ms_handle_connect 0x7f5f840a4920 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5f6c017070 con 0x7f5f840a4920 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 10 monclient: dump: 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 10 monclient: _finish_auth 0 2024-06-11T19:09:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 10 monclient: _check_auth_tickets 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f6c00ee60 con 0x7f5f840a4920 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f88b9c640 5 monclient: authenticate success, global_id 4616 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f88b9c640 10 monclient: _renew_subs 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f88b9c640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f88b9c640 1 -- 172.21.15.98:0/4270840112 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f84127380 con 0x7f5f840a4920 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5f6c00cdb0 con 0x7f5f840a4920 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f80d99640 4 set_mon_vals no callback set 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f88b9c640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f7affd640 10 monclient: dump: 2024-06-11T19:09:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f88b9c640 10 monclient: _renew_subs 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f88b9c640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f88b9c640 1 -- 172.21.15.98:0/4270840112 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f840a4d90 con 0x7f5f840a4920 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.923+0000 7f5f88b9c640 1 librados: init done 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f88b9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f88b9c640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84125820 0x7f5f84125c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f88b9c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f88b9c640 1 -- 172.21.15.98:0/4270840112 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f5f84050830 con 0x7f5f84125820 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f5f6c01a070 con 0x7f5f840a4920 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f8259c640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84125820 0x7f5f84125c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f8259c640 10 monclient: get_auth_request con 0x7f5f84125820 auth_method 0 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f8259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f8259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f7affd640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5f6403fdb0 0x7f5f64042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f8259c640 10 monclient(hunting): handle_auth_done global_id 4406 payload 824 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f8259c640 1 -- 172.21.15.98:0/4270840112 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f84126ed0 con 0x7f5f84125820 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f5f6c00f040 con 0x7f5f840a4920 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f7affd640 10 client.4616.objecter ms_dispatch 0x7f5f84000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f7affd640 3 client.4616.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f7affd640 3 client.4616.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f7affd640 20 client.4616.objecter dump_active .. 0 homeless 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f82d9d640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5f6403fdb0 0x7f5f64042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f82d9d640 10 monclient: get_auth_request con 0x7f5f6403fdb0 auth_method 0 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f7affd640 10 client.4616.objecter ms_handle_connect 0x7f5f84125820 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f7affd640 10 client.4616.objecter resend_mon_ops 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.924+0000 7f5f8259c640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84125820 0x7f5f84125c00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5f74003980 tx=0x7f5f7400b830 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5f74016070 con 0x7f5f84125820 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f7affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f7affd640 10 monclient: dump: 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f7affd640 10 client.4616.objecter ms_handle_connect 0x7f5f6403fdb0 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f82d9d640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5f6403fdb0 0x7f5f64042270 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c035000 tx=0x7f5f7c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5f74003bc0 con 0x7f5f84125820 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f7affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f7affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84125820 msgr2=0x7f5f84125c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f7affd640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84125820 0x7f5f84125c00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5f74003980 tx=0x7f5f7400b830 comp rx=0 tx=0).stop 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84125820 msgr2=0x7f5f84125c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.925+0000 7f5f7affd640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84125820 0x7f5f84125c00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.931+0000 7f5f88b9c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.931+0000 7f5f88b9c640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84126140 0x7f5f841215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.931+0000 7f5f88b9c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.931+0000 7f5f88b9c640 1 -- 172.21.15.98:0/4270840112 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f5f840a9140 con 0x7f5f84126140 2024-06-11T19:09:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.931+0000 7f5f8259c640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84126140 0x7f5f841215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.931+0000 7f5f8259c640 10 monclient: get_auth_request con 0x7f5f84126140 auth_method 0 2024-06-11T19:09:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.931+0000 7f5f8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.931+0000 7f5f8259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.931+0000 7f5f8259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.931+0000 7f5f8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.931+0000 7f5f8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.931+0000 7f5f8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f8259c640 10 monclient(hunting): handle_auth_done global_id 4415 payload 824 2024-06-11T19:09:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f8259c640 1 -- 172.21.15.98:0/4270840112 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5f840a66f0 con 0x7f5f84126140 2024-06-11T19:09:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f8259c640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84126140 0x7f5f841215f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5f74003980 tx=0x7f5f740037f0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 10 client.4616.objecter ms_handle_connect 0x7f5f84126140 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 10 client.4616.objecter resend_mon_ops 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5f74016040 con 0x7f5f84126140 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 10 monclient: dump: 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9601 (secure 0 0 0) 0x7f5f740188d0 con 0x7f5f84126140 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84126140 msgr2=0x7f5f841215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84126140 0x7f5f841215f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5f74003980 tx=0x7f5f740037f0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 1 -- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84126140 msgr2=0x7f5f841215f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f7affd640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5f84126140 0x7f5f841215f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:56.955 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 10, 2024-06-11T19:09:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 47127, 2024-06-11T19:09:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.957 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:56.958 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:56.959 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:56.962 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:56.962 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.962 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.962 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.962 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:56.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:56.962 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:56.963 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:56.965 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.968 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:56.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:56.974 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.975 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.976 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:09:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:56.977 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f88b9c640 10 librados: watch_flush enter 2024-06-11T19:09:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f88b9c640 10 librados: watch_flush exit 2024-06-11T19:09:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f88b9c640 20 client.4616.objecter shutdown clearing up homeless session... 2024-06-11T19:09:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f88b9c640 10 client.4616.objecter successfully canceled tick 2024-06-11T19:09:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f88b9c640 1 -- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5f6403fdb0 msgr2=0x7f5f64042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f88b9c640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5f6403fdb0 0x7f5f64042270 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c035000 tx=0x7f5f7c030000 comp rx=0 tx=0).stop 2024-06-11T19:09:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f88b9c640 10 monclient: shutdown 2024-06-11T19:09:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f88b9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f88b9c640 1 -- 172.21.15.98:0/4270840112 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f840a4920 msgr2=0x7f5f841ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f88b9c640 1 --2- 172.21.15.98:0/4270840112 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f840a4920 0x7f5f841ce040 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00e870 tx=0x7f5f6c00ed30 comp rx=0 tx=0).stop 2024-06-11T19:09:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f82d9d640 1 -- 172.21.15.98:0/4270840112 reap_dead start 2024-06-11T19:09:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f82d9d640 1 -- 172.21.15.98:0/4270840112 reap_dead start 2024-06-11T19:09:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f88b9c640 1 -- 172.21.15.98:0/4270840112 shutdown_connections 2024-06-11T19:09:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.932+0000 7f5f88b9c640 1 -- 172.21.15.98:0/4270840112 >> 172.21.15.98:0/4270840112 conn(0x7f5f840ab850 msgr2=0x7f5f8409c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.933+0000 7f5f88b9c640 1 -- 172.21.15.98:0/4270840112 shutdown_connections 2024-06-11T19:09:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.933+0000 7f5f88b9c640 1 -- 172.21.15.98:0/4270840112 wait complete. 2024-06-11T19:09:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:56.933+0000 7f5f88b9c640 1 librados: shutdown 2024-06-11T19:09:56.980 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:09:56.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-06-11T19:09:56.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:09:57.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.081+0000 7f16f5385640 10 monclient: get_monmap_and_config 2024-06-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.081+0000 7f16f5385640 10 monclient: build_initial_monmap 2024-06-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.081+0000 7f16f5385640 10 monclient: monmap: 2024-06-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:09:57.082308+0000 2024-06-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:09:57.082308+0000 2024-06-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:09:57.081 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:09:57.082 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:57.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.081+0000 7f16f5385640 1 Processor -- start 2024-06-11T19:09:57.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.081+0000 7f16f5385640 1 -- start start 2024-06-11T19:09:57.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.081+0000 7f16f5385640 10 monclient: init 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 10 monclient: _add_conns ranks=[2,3,0,8,10,1,4,5,6,9,7] 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f00a52a0 0x7f16f00a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 10 monclient(hunting): picked mon.noname-b con 0x7f16f00a52a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f16f009d520 0x7f16f00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 10 monclient(hunting): picked mon.noname-g con 0x7f16f009d520 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16f0073830 addr v1:172.21.15.98:6789/0 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16f009d2e0 con 0x7f16f0073830 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f16f009daa0 con 0x7f16f00a52a0 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f16f00a5680 con 0x7f16f009d520 2024-06-11T19:09:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16f5385640 10 monclient(hunting): authenticate will time out at 806.889411s 2024-06-11T19:09:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16ef7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f16f0073830 0x7f16f00a3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39888/0 (socket says 172.21.15.98:39888) 2024-06-11T19:09:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16ef7fe640 1 -- 172.21.15.98:0/3661556933 learned_addr learned my addr 172.21.15.98:0/3661556933 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:09:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 1 --2- 172.21.15.98:0/3661556933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f00a52a0 0x7f16f00a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 10 monclient(hunting): get_auth_request con 0x7f16f00a52a0 auth_method 0 2024-06-11T19:09:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16e67fc640 1 --2- 172.21.15.98:0/3661556933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f16f009d520 0x7f16f00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16e67fc640 10 monclient(hunting): get_auth_request con 0x7f16f009d520 auth_method 0 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16e67fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16ee7fc640 1 -- 172.21.15.98:0/3661556933 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 534940279 0 0) 0x7f16f009d2e0 con 0x7f16f0073830 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16ee7fc640 10 monclient(hunting): _negotiate 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16ee7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16ee7fc640 10 monclient(hunting): my global_id is 4581 2024-06-11T19:09:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16ee7fc640 1 -- 172.21.15.98:0/3661556933 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16dc003520 con 0x7f16f0073830 2024-06-11T19:09:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 10 monclient(hunting): handle_auth_done global_id 4547 payload 274 2024-06-11T19:09:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 1 -- 172.21.15.98:0/3661556933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f16f009d520 msgr2=0x7f16f00a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:09:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 1 --2- 172.21.15.98:0/3661556933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f16f009d520 0x7f16f00a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 1 -- 172.21.15.98:0/3661556933 >> v1:172.21.15.98:6789/0 conn(0x7f16f0073830 legacy=0x7f16f00a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 1 monclient: found mon.noname-b 2024-06-11T19:09:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:09:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.082+0000 7f16eeffd640 1 -- 172.21.15.98:0/3661556933 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f009e9d0 con 0x7f16f00a52a0 2024-06-11T19:09:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16eeffd640 1 --2- 172.21.15.98:0/3661556933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f00a52a0 0x7f16f00a1910 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009800 tx=0x7f16e002b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=426082ab12c4911d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 1 -- 172.21.15.98:0/3661556933 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16e0037070 con 0x7f16f00a52a0 2024-06-11T19:09:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 10 monclient: dump: 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 10 monclient: _finish_auth 0 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 1 -- 172.21.15.98:0/3661556933 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16e00044a0 con 0x7f16f00a52a0 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16f5385640 5 monclient: authenticate success, global_id 4547 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16f5385640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16f5385640 10 monclient: get_monmap_and_config success 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16f5385640 4 set_mon_vals no callback set 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 1 -- 172.21.15.98:0/3661556933 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16e0003a80 con 0x7f16f00a52a0 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.083+0000 7f16ee7fc640 10 monclient: dump: 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.084+0000 7f16f5385640 10 monclient: shutdown 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.084+0000 7f16f5385640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.084+0000 7f16f5385640 1 -- 172.21.15.98:0/3661556933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f00a52a0 msgr2=0x7f16f00a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.084+0000 7f16f5385640 1 --2- 172.21.15.98:0/3661556933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f00a52a0 0x7f16f00a1910 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009800 tx=0x7f16e002b760 comp rx=0 tx=0).stop 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.084+0000 7f16f5385640 1 -- 172.21.15.98:0/3661556933 shutdown_connections 2024-06-11T19:09:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.084+0000 7f16f5385640 1 --2- 172.21.15.98:0/3661556933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f16f009d520 0x7f16f00a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.084+0000 7f16f5385640 1 --2- 172.21.15.98:0/3661556933 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f00a52a0 0x7f16f00a1910 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.084+0000 7f16f5385640 1 -- 172.21.15.98:0/3661556933 >> 172.21.15.98:0/3661556933 conn(0x7f16f00a4780 msgr2=0x7f16f00a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.084+0000 7f16f5385640 1 -- 172.21.15.98:0/3661556933 shutdown_connections 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.084+0000 7f16f5385640 1 -- 172.21.15.98:0/3661556933 wait complete. 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.084+0000 7f16f5385640 10 monclient: build_initial_monmap 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.084+0000 7f16f5385640 10 monclient: monmap: 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:09:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 1 librados: starting msgr at 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 1 librados: starting objecter 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 1 Processor -- start 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 1 -- start start 2024-06-11T19:09:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 1 librados: setting wanted keys 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 1 librados: calling monclient init 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 10 monclient: init 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 10 monclient: _reopen_session rank -1 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 10 monclient: _add_conns ranks=[4,2,5,7,3,6,0,1,8] 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16f0073830 0x7f16f01cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 10 monclient(hunting): picked mon.noname-c con 0x7f16f0073830 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f009d520 0x7f16f01325c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 10 monclient(hunting): picked mon.noname-b con 0x7f16f009d520 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f16f00a52a0 0x7f16f006f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 10 monclient(hunting): picked mon.noname-h con 0x7f16f00a52a0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f16f00a4460 con 0x7f16f009d520 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f16f00a42e0 con 0x7f16f0073830 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f16f00a1e50 con 0x7f16f00a52a0 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 10 monclient(hunting): _renew_subs 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16f5385640 10 monclient(hunting): authenticate will time out at 806.892834s 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16e67fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f009d520 0x7f16f01325c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16eeffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16f0073830 0x7f16f01cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16e67fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f009d520 0x7f16f01325c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50734/0 (socket says 172.21.15.98:50734) 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16eeffd640 10 monclient(hunting): get_auth_request con 0x7f16f0073830 auth_method 0 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16e67fc640 1 -- 172.21.15.98:0/1778345655 learned_addr learned my addr 172.21.15.98:0/1778345655 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16e67fc640 10 monclient(hunting): get_auth_request con 0x7f16f009d520 auth_method 0 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16e67fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.085+0000 7f16ef7fe640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f16f00a52a0 0x7f16f006f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16ef7fe640 10 monclient(hunting): get_auth_request con 0x7f16f00a52a0 auth_method 0 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 10 monclient(hunting): handle_auth_done global_id 4576 payload 824 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 1 -- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f16f00a52a0 msgr2=0x7f16f006f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f16f00a52a0 0x7f16f006f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 1 -- 172.21.15.98:0/1778345655 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f009d520 msgr2=0x7f16f01325c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f009d520 0x7f16f01325c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 10 monclient: _finish_hunting 0 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 1 monclient: found mon.noname-c 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:09:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 1 -- 172.21.15.98:0/1778345655 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f00a12c0 con 0x7f16f0073830 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e67fc640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f009d520 0x7f16f01325c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16eeffd640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16f0073830 0x7f16f01cdb90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f16e0004000 tx=0x7f16e00046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 10 client.?.objecter ms_handle_connect 0x7f16f0073830 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16e0042020 con 0x7f16f0073830 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 10 monclient: dump: 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 10 monclient: _finish_auth 0 2024-06-11T19:09:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16e0003c90 con 0x7f16f0073830 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16f5385640 5 monclient: authenticate success, global_id 4576 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16e0037040 con 0x7f16f0073830 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16f5385640 10 monclient: _renew_subs 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16f5385640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16f5385640 1 -- 172.21.15.98:0/1778345655 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f006e710 con 0x7f16f0073830 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16edffb640 4 set_mon_vals no callback set 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16e7fff640 10 monclient: dump: 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16f5385640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16f5385640 10 monclient: _renew_subs 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16f5385640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16f5385640 1 -- 172.21.15.98:0/1778345655 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f00731d0 con 0x7f16f0073830 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.086+0000 7f16f5385640 1 librados: init done 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16f5385640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16f5385640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f0071cb0 0x7f16f006fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16f5385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16f5385640 1 -- 172.21.15.98:0/1778345655 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f16f0050830 con 0x7f16f0071cb0 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f16e0045020 con 0x7f16f0073830 2024-06-11T19:09:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e67fc640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f0071cb0 0x7f16f006fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e67fc640 10 monclient: get_auth_request con 0x7f16f0071cb0 auth_method 0 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e67fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e7fff640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f16d003fdc0 0x7f16d0042280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7f16e0072da0 con 0x7f16f0073830 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e7fff640 10 client.4576.objecter ms_dispatch 0x7f16f0000b90 osd_map(18..18 src has 1..18) 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e7fff640 3 client.4576.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e7fff640 3 client.4576.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16e7fff640 20 client.4576.objecter dump_active .. 0 homeless 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.087+0000 7f16ef7fe640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f16d003fdc0 0x7f16d0042280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e67fc640 10 monclient(hunting): handle_auth_done global_id 4525 payload 824 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e67fc640 1 -- 172.21.15.98:0/1778345655 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16f0071150 con 0x7f16f0071cb0 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16ef7fe640 10 monclient: get_auth_request con 0x7f16d003fdc0 auth_method 0 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 10 client.4576.objecter ms_handle_connect 0x7f16f0071cb0 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 10 client.4576.objecter resend_mon_ops 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e67fc640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f0071cb0 0x7f16f006fab0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f16f0070cd0 tx=0x7f16d800f610 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16d8019070 con 0x7f16f0071cb0 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 10 monclient: dump: 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 10 client.4576.objecter ms_handle_connect 0x7f16d003fdc0 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16ef7fe640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f16d003fdc0 0x7f16d0042280 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f16f009e410 tx=0x7f16e8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f16d8004030 con 0x7f16f0071cb0 2024-06-11T19:09:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:09:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:09:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f0071cb0 msgr2=0x7f16f006fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f0071cb0 0x7f16f006fab0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f16f0070cd0 tx=0x7f16d800f610 comp rx=0 tx=0).stop 2024-06-11T19:09:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f0071cb0 msgr2=0x7f16f006fab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.088+0000 7f16e7fff640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f0071cb0 0x7f16f006fab0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f16f0070cd0 tx=0x7f16d800f610 comp rx=0 tx=0).stop 2024-06-11T19:09:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.094+0000 7f16f5385640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.094+0000 7f16f5385640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f00a16f0 0x7f16f006fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16f5385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16f5385640 1 -- 172.21.15.98:0/1778345655 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f16f0075170 con 0x7f16f00a16f0 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e67fc640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f00a16f0 0x7f16f006fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e67fc640 10 monclient: get_auth_request con 0x7f16f00a16f0 auth_method 0 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e67fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e67fc640 10 monclient(hunting): handle_auth_done global_id 4534 payload 824 2024-06-11T19:09:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e67fc640 1 -- 172.21.15.98:0/1778345655 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f16f00a3410 con 0x7f16f00a16f0 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e7fff640 10 client.4576.objecter ms_handle_connect 0x7f16f00a16f0 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e67fc640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f00a16f0 0x7f16f006fff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16f0070cd0 tx=0x7f16d80046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e7fff640 10 client.4576.objecter resend_mon_ops 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16d8019040 con 0x7f16f00a16f0 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e7fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.095+0000 7f16e7fff640 10 monclient: dump: 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:09:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9261 (secure 0 0 0) 0x7f16d801e660 con 0x7f16f00a16f0 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16e7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16e7fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:09:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f00a16f0 msgr2=0x7f16f006fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16e7fff640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f00a16f0 0x7f16f006fff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16f0070cd0 tx=0x7f16d80046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16e7fff640 1 -- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f00a16f0 msgr2=0x7f16f006fff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:09:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16e7fff640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f16f00a16f0 0x7f16f006fff0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16f0070cd0 tx=0x7f16d80046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:57.120 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:09:57.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:57.120 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:57.120 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:09:57.120 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 10, 2024-06-11T19:09:57.120 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 47312, 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:57.121 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:09:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:09:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:09:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:09:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:09:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:09:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:09:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:09:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:09:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:09:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:09:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:09:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:09:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:09:57.142 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:09:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 10 librados: watch_flush enter 2024-06-11T19:09:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 10 librados: watch_flush exit 2024-06-11T19:09:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 20 client.4576.objecter shutdown clearing up homeless session... 2024-06-11T19:09:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 10 client.4576.objecter successfully canceled tick 2024-06-11T19:09:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 1 -- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f16d003fdc0 msgr2=0x7f16d0042280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f16d003fdc0 0x7f16d0042280 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f16f009e410 tx=0x7f16e8009040 comp rx=0 tx=0).stop 2024-06-11T19:09:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 10 monclient: shutdown 2024-06-11T19:09:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:09:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 1 -- 172.21.15.98:0/1778345655 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16f0073830 msgr2=0x7f16f01cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:09:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 1 --2- 172.21.15.98:0/1778345655 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16f0073830 0x7f16f01cdb90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f16e0004000 tx=0x7f16e00046c0 comp rx=0 tx=0).stop 2024-06-11T19:09:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16ef7fe640 1 -- 172.21.15.98:0/1778345655 reap_dead start 2024-06-11T19:09:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16ef7fe640 1 -- 172.21.15.98:0/1778345655 reap_dead start 2024-06-11T19:09:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 1 -- 172.21.15.98:0/1778345655 shutdown_connections 2024-06-11T19:09:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 1 -- 172.21.15.98:0/1778345655 >> 172.21.15.98:0/1778345655 conn(0x7f16f00a4780 msgr2=0x7f16f0134440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:09:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 1 -- 172.21.15.98:0/1778345655 shutdown_connections 2024-06-11T19:09:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 1 -- 172.21.15.98:0/1778345655 wait complete. 2024-06-11T19:09:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:09:57.096+0000 7f16f5385640 1 librados: shutdown 2024-06-11T19:09:57.146 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:09:57.146 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-06-11T19:10:33.108 INFO:tasks.workunit.client.0.smithi117.stderr:Note: switching to '37bbe1d8d788b7c9a71280e7d510c295a5853f0b'. 2024-06-11T19:10:33.109 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-06-11T19:10:33.109 INFO:tasks.workunit.client.0.smithi117.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-06-11T19:10:33.109 INFO:tasks.workunit.client.0.smithi117.stderr:changes and commit them, and you can discard any commits you make in this 2024-06-11T19:10:33.109 INFO:tasks.workunit.client.0.smithi117.stderr:state without impacting any branches by switching back to a branch. 2024-06-11T19:10:33.109 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-06-11T19:10:33.109 INFO:tasks.workunit.client.0.smithi117.stderr:If you want to create a new branch to retain commits you create, you may 2024-06-11T19:10:33.109 INFO:tasks.workunit.client.0.smithi117.stderr:do so (now or later) by using -c with the switch command. Example: 2024-06-11T19:10:33.109 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-06-11T19:10:33.109 INFO:tasks.workunit.client.0.smithi117.stderr: git switch -c 2024-06-11T19:10:33.109 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-06-11T19:10:33.110 INFO:tasks.workunit.client.0.smithi117.stderr:Or undo this operation with: 2024-06-11T19:10:33.110 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-06-11T19:10:33.110 INFO:tasks.workunit.client.0.smithi117.stderr: git switch - 2024-06-11T19:10:33.110 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-06-11T19:10:33.110 INFO:tasks.workunit.client.0.smithi117.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-06-11T19:10:33.110 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-06-11T19:10:33.110 INFO:tasks.workunit.client.0.smithi117.stderr:HEAD is now at 37bbe1d8d78 Merge branch 'wip-osd-ec-partial-reads' of https://github.com/rzarzynski/ceph into wip-lflores-testing-2-2024-06-10-1925 2024-06-11T19:10:33.114 DEBUG:teuthology.orchestra.run.smithi117:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-06-11T19:10:33.180 INFO:tasks.workunit.client.0.smithi117.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-06-11T19:10:33.182 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-06-11T19:10:33.182 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-06-11T19:10:33.759 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-06-11T19:10:33.819 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-06-11T19:10:33.862 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-06-11T19:10:33.864 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-06-11T19:10:33.864 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-06-11T19:10:33.902 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-06-11T19:10:33.905 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-06-11T19:10:33.905 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-06-11T19:10:33.961 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-06-11T19:10:33.962 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-06-11T19:10:33.962 DEBUG:teuthology.orchestra.run.smithi117:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37bbe1d8d788b7c9a71280e7d510c295a5853f0b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-06-11T19:10:34.021 INFO:tasks.workunit.client.0.smithi117.stderr:+ parallel=1 2024-06-11T19:10:34.022 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' '' = --serial ']' 2024-06-11T19:10:34.022 INFO:tasks.workunit.client.0.smithi117.stderr:+ crimson=0 2024-06-11T19:10:34.022 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' '' = --crimson ']' 2024-06-11T19:10:34.022 INFO:tasks.workunit.client.0.smithi117.stderr:+ color= 2024-06-11T19:10:34.022 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -t 1 ']' 2024-06-11T19:10:34.022 INFO:tasks.workunit.client.0.smithi117.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-06-11T19:10:34.022 INFO:tasks.workunit.client.0.smithi117.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-06-11T19:10:34.022 INFO:tasks.workunit.client.0.smithi117.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-06-11T19:10:34.023 INFO:tasks.workunit.client.0.smithi117.stderr:+ declare -A pids 2024-06-11T19:10:34.023 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.023 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.023 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_aio 2024-06-11T19:10:34.023 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_aio' 2024-06-11T19:10:34.024 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_aio 2024-06-11T19:10:34.024 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.024 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_aio 2024-06-11T19:10:34.024 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36408 2024-06-11T19:10:34.024 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_aio on pid 36408' 2024-06-11T19:10:34.025 INFO:tasks.workunit.client.0.smithi117.stdout:test api_aio on pid 36408 2024-06-11T19:10:34.028 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36408 2024-06-11T19:10:34.028 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.028 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.028 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-06-11T19:10:34.028 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_aio_pp 2024-06-11T19:10:34.028 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_aio_pp' 2024-06-11T19:10:34.028 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_aio_pp 2024-06-11T19:10:34.028 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.028 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.028 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.029 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.029 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.029 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.029 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.029 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.029 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.029 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.029 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.029 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.029 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.029 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.029 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_aio_pp 2024-06-11T19:10:34.030 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36415 2024-06-11T19:10:34.031 INFO:tasks.workunit.client.0.smithi117.stdout:test api_aio_pp on pid 36415 2024-06-11T19:10:34.034 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_aio_pp on pid 36415' 2024-06-11T19:10:34.034 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36415 2024-06-11T19:10:34.034 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_io 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_io' 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_io 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.035 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.036 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.037 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.038 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.038 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.038 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.038 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.038 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.038 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.038 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.038 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.038 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.038 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_io 2024-06-11T19:10:34.038 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36424 2024-06-11T19:10:34.038 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_io on pid 36424' 2024-06-11T19:10:34.039 INFO:tasks.workunit.client.0.smithi117.stdout:test api_io on pid 36424 2024-06-11T19:10:34.043 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36424 2024-06-11T19:10:34.043 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.043 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.043 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-06-11T19:10:34.043 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_io_pp 2024-06-11T19:10:34.043 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_io_pp' 2024-06-11T19:10:34.043 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_io_pp 2024-06-11T19:10:34.043 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.044 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.045 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.046 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.047 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.048 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.048 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.048 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_io_pp 2024-06-11T19:10:34.048 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36435 2024-06-11T19:10:34.048 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_io_pp on pid 36435' 2024-06-11T19:10:34.048 INFO:tasks.workunit.client.0.smithi117.stdout:test api_io_pp on pid 36435 2024-06-11T19:10:34.053 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36435 2024-06-11T19:10:34.053 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.053 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-06-11T19:10:34.053 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.053 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_asio 2024-06-11T19:10:34.053 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_asio' 2024-06-11T19:10:34.053 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_asio 2024-06-11T19:10:34.053 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.053 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.053 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.053 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.054 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.055 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.056 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.057 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.058 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_asio 2024-06-11T19:10:34.059 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36449 2024-06-11T19:10:34.059 INFO:tasks.workunit.client.0.smithi117.stdout:test api_asio on pid 36449 2024-06-11T19:10:34.065 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_asio on pid 36449' 2024-06-11T19:10:34.065 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36449 2024-06-11T19:10:34.065 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_list 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_list' 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_list 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.066 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.067 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.068 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.069 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.070 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.071 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.072 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36408/exe 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_list 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36466 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_list on pid 36466' 2024-06-11T19:10:34.073 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-06-11T19:10:34.074 INFO:tasks.workunit.client.0.smithi117.stdout:test api_list on pid 36466 2024-06-11T19:10:34.082 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36466 2024-06-11T19:10:34.082 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.082 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.082 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.082 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.082 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.082 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_lock 2024-06-11T19:10:34.082 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_lock' 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_lock 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.083 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.084 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.085 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.086 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.087 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-06-11T19:10:34.088 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.089 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.089 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.089 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.089 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.090 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.090 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.090 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_aio: /' 2024-06-11T19:10:34.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.090 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.090 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.091 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.091 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.091 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.091 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.091 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_lock 2024-06-11T19:10:34.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36415/exe 2024-06-11T19:10:34.091 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.092 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36487 2024-06-11T19:10:34.092 INFO:tasks.workunit.client.0.smithi117.stdout:test api_lock on pid 36487 2024-06-11T19:10:34.098 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.099 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_lock on pid 36487' 2024-06-11T19:10:34.099 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36487 2024-06-11T19:10:34.099 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_aio.log 2024-06-11T19:10:34.099 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.099 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.099 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.099 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_lock_pp 2024-06-11T19:10:34.099 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_lock_pp' 2024-06-11T19:10:34.099 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.099 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-06-11T19:10:34.099 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_lock_pp 2024-06-11T19:10:34.099 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.100 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.101 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.102 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.103 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.104 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_aio_pp: /' 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.105 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36424/exe 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_lock_pp 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.106 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36506 2024-06-11T19:10:34.107 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.107 INFO:tasks.workunit.client.0.smithi117.stdout:test api_lock_pp on pid 36506 2024-06-11T19:10:34.114 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_lock_pp on pid 36506' 2024-06-11T19:10:34.114 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36506 2024-06-11T19:10:34.114 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.114 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.114 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-06-11T19:10:34.114 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_misc 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_misc' 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.115 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_misc 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.116 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.117 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.118 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_io.log 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.119 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.120 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_misc 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.121 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.122 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36523 2024-06-11T19:10:34.122 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.122 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.122 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.122 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.122 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.122 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.122 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_misc on pid 36523' 2024-06-11T19:10:34.122 INFO:tasks.workunit.client.0.smithi117.stdout:test api_misc on pid 36523 2024-06-11T19:10:34.145 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36523 2024-06-11T19:10:34.145 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.145 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.145 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_misc_pp 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_misc_pp' 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36435/exe 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.146 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.147 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_misc_pp 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.148 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_io: /' 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.149 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.150 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.151 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.152 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.153 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.153 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.153 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.153 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.153 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-06-11T19:10:34.153 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.153 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.153 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.153 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.153 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.154 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.155 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_io_pp: /' 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36449/exe 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.156 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.157 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.158 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_asio.log 2024-06-11T19:10:34.159 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.160 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.161 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.162 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36466/exe 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.163 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.164 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36487/exe 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_list: /' 2024-06-11T19:10:34.165 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.166 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.167 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.168 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.168 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_lock: /' 2024-06-11T19:10:34.168 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_misc_pp 2024-06-11T19:10:34.168 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.168 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36573 2024-06-11T19:10:34.168 INFO:tasks.workunit.client.0.smithi117.stdout:test api_misc_pp on pid 36573 2024-06-11T19:10:34.179 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_misc_pp on pid 36573' 2024-06-11T19:10:34.179 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36573 2024-06-11T19:10:34.179 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.179 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.180 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.181 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_asio: /' 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.182 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.183 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.183 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.183 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.183 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.183 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.183 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.183 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.183 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.183 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.183 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_lock.log 2024-06-11T19:10:34.183 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.185 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.185 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.185 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.185 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.185 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_tier_pp 2024-06-11T19:10:34.185 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_tier_pp' 2024-06-11T19:10:34.185 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.185 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_list.log 2024-06-11T19:10:34.185 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.185 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.185 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.186 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36506/exe 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.187 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36523/exe 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.188 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.189 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.190 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.191 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.191 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.191 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_misc.log 2024-06-11T19:10:34.191 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-06-11T19:10:34.191 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_lock_pp: /' 2024-06-11T19:10:34.191 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_tier_pp 2024-06-11T19:10:34.191 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_tier_pp 2024-06-11T19:10:34.191 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36616 2024-06-11T19:10:34.191 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_tier_pp on pid 36616' 2024-06-11T19:10:34.191 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-06-11T19:10:34.192 INFO:tasks.workunit.client.0.smithi117.stdout:test api_tier_pp on pid 36616 2024-06-11T19:10:34.193 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36616 2024-06-11T19:10:34.194 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.194 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.194 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_pool 2024-06-11T19:10:34.194 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_pool' 2024-06-11T19:10:34.194 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_pool 2024-06-11T19:10:34.194 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.194 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.194 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.194 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.194 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.194 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.194 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.195 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_misc: /' 2024-06-11T19:10:34.195 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_pool 2024-06-11T19:10:34.195 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36629 2024-06-11T19:10:34.195 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_pool on pid 36629' 2024-06-11T19:10:34.195 INFO:tasks.workunit.client.0.smithi117.stdout:test api_pool on pid 36629 2024-06-11T19:10:34.202 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36629 2024-06-11T19:10:34.202 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.202 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.202 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-06-11T19:10:34.202 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.202 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.202 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.202 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.203 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.204 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.205 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.206 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_snapshots 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_snapshots' 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_snapshots 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.207 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.208 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.208 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.208 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.208 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_snapshots 2024-06-11T19:10:34.208 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36687 2024-06-11T19:10:34.208 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_snapshots on pid 36687' 2024-06-11T19:10:34.208 INFO:tasks.workunit.client.0.smithi117.stdout:test api_snapshots on pid 36687 2024-06-11T19:10:34.217 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36687 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_snapshots_pp 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_snapshots_pp' 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.218 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.219 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.220 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_snapshots_pp 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.221 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.222 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.223 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.224 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.225 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.226 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.226 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_snapshots_pp 2024-06-11T19:10:34.226 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.226 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.226 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.226 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.226 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36707 2024-06-11T19:10:34.226 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.226 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.226 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-06-11T19:10:34.226 INFO:tasks.workunit.client.0.smithi117.stdout:test api_snapshots_pp on pid 36707 2024-06-11T19:10:34.231 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_snapshots_pp on pid 36707' 2024-06-11T19:10:34.231 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36707 2024-06-11T19:10:34.231 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.231 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.231 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_stat 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_stat' 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.232 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_stat 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.233 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.234 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_stat 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.235 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.236 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36721 2024-06-11T19:10:34.236 INFO:tasks.workunit.client.0.smithi117.stdout:test api_stat on pid 36721 2024-06-11T19:10:34.239 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_stat on pid 36721' 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36721 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.240 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.241 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.242 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.243 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.243 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-06-11T19:10:34.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [----------] Global test environment set-up. 2024-06-11T19:10:34.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [----------] 12 tests from AsioRados 2024-06-11T19:10:34.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.128+0000 7efc0c645780 10 monclient: get_monmap_and_config 2024-06-11T19:10:34.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.128+0000 7efc0c645780 10 monclient: build_initial_monmap 2024-06-11T19:10:34.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.128+0000 7efc0c645780 10 monclient: monmap: 2024-06-11T19:10:34.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: epoch 0 2024-06-11T19:10:34.243 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: last_changed 0.000000 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: created 0.000000 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: min_mon_release 0 (unknown) 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: election_strategy: 1 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:10:34.244 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.128+0000 7efc0c645780 1 Processor -- start 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.128+0000 7efc0c645780 1 -- start start 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.128+0000 7efc0c645780 10 monclient: init 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 10 monclient: _reopen_session rank -1 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 10 monclient: _add_conns ranks=[2,0,5,3,4,8,1,6,7] 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558e13140980 0x558e131d79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 10 monclient(hunting): picked mon.noname-b con 0x558e13140980 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e13142e50 0x558e131d7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 10 monclient(hunting): picked mon.noname-a con 0x558e13142e50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x558e13146260 0x558e13128010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 10 monclient(hunting): picked mon.noname-h con 0x558e13146260 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 10 monclient(hunting): start opening mon connection 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558e13074500 con 0x558e13142e50 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 10 monclient(hunting): start opening mon connection 2024-06-11T19:10:34.245 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x558e130d1560 con 0x558e13140980 2024-06-11T19:10:34.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.247 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.247 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.247 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.247 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.248 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.248 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 10 monclient(hunting): start opening mon connection 2024-06-11T19:10:34.248 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x558e13145680 con 0x558e13146260 2024-06-11T19:10:34.248 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 10 monclient(hunting): _renew_subs 2024-06-11T19:10:34.248 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0c645780 10 monclient(hunting): authenticate will time out at 856.941314s 2024-06-11T19:10:34.248 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x558e13146260 0x558e13128010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:10:34.248 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x558e13146260 0x558e13128010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.117:60320/0 (socket says 172.21.15.117:60320) 2024-06-11T19:10:34.248 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 1 -- 172.21.15.117:0/2533539005 learned_addr learned my addr 172.21.15.117:0/2533539005 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 10 monclient(hunting): get_auth_request con 0x558e13146260 auth_method 0 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0914c640 1 --2- 172.21.15.117:0/2533539005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e13142e50 0x558e131d7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0914c640 10 monclient(hunting): get_auth_request con 0x558e13142e50 auth_method 0 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0914c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0914c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0914c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0994d640 1 --2- 172.21.15.117:0/2533539005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558e13140980 0x558e131d79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:10:34.249 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0994d640 10 monclient(hunting): get_auth_request con 0x558e13140980 auth_method 0 2024-06-11T19:10:34.250 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0994d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:10:34.250 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0994d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:10:34.250 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0994d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:10:34.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.251 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.251 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.251 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.251 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.251 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.251 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.251 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0914c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0914c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0914c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 10 monclient(hunting): handle_auth_done global_id 4469 payload 274 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 1 -- 172.21.15.117:0/2533539005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558e13140980 msgr2=0x558e131d79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 1 --2- 172.21.15.117:0/2533539005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558e13140980 0x558e131d79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 1 -- 172.21.15.117:0/2533539005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e13142e50 msgr2=0x558e131d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.129+0000 7efc0a14e640 1 --2- 172.21.15.117:0/2533539005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e13142e50 0x558e131d7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efc0a14e640 10 monclient: _finish_hunting 0 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efc0a14e640 1 monclient: found mon.noname-h 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efc0a14e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efc0a14e640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efc0a14e640 1 -- 172.21.15.117:0/2533539005 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e1312a260 con 0x558e13146260 2024-06-11T19:10:34.252 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efc0994d640 1 --2- 172.21.15.117:0/2533539005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558e13140980 0x558e131d79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:10:34.253 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efc0a14e640 1 --2- 172.21.15.117:0/2533539005 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x558e13146260 0x558e13128010 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efc0000b240 tx=0x7efc0000b700 comp rx=0 tx=0).ready entity=mon.5 client_cookie=fc9cd2a4c52327f3 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:10:34.253 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 1 -- 172.21.15.117:0/2533539005 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc00008070 con 0x558e13146260 2024-06-11T19:10:34.253 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:10:34.253 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:10:34.253 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 10 monclient: dump: 2024-06-11T19:10:34.253 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: epoch 1 2024-06-11T19:10:34.253 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:10:34.253 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:10:34.253 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:10:34.253 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: min_mon_release 19 (squid) 2024-06-11T19:10:34.254 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: election_strategy: 3 2024-06-11T19:10:34.254 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:10:34.254 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:10:34.254 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:10:34.255 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.255 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.255 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.255 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.256 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.256 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.256 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.256 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.256 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.256 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.256 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.256 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.256 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.256 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.256 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.256 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 10 monclient: _finish_auth 0 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 10 monclient: _check_auth_tickets 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 1 -- 172.21.15.117:0/2533539005 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc0000d970 con 0x558e13146260 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:10:34.257 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 1 -- 172.21.15.117:0/2533539005 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc0000f3f0 con 0x558e13146260 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efbfbfff640 10 monclient: dump: 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: epoch 1 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: min_mon_release 19 (squid) 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: election_strategy: 3 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:10:34.258 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efc0c645780 5 monclient: authenticate success, global_id 4469 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efc0c645780 10 monclient: get_monmap_and_config success 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.130+0000 7efc0c645780 4 set_mon_vals no callback set 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.131+0000 7efc0c645780 10 monclient: shutdown 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.131+0000 7efc0c645780 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.131+0000 7efc0c645780 1 -- 172.21.15.117:0/2533539005 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x558e13146260 msgr2=0x558e13128010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.131+0000 7efc0c645780 1 --2- 172.21.15.117:0/2533539005 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x558e13146260 0x558e13128010 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efc0000b240 tx=0x7efc0000b700 comp rx=0 tx=0).stop 2024-06-11T19:10:34.259 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 1 -- 172.21.15.117:0/2533539005 shutdown_connections 2024-06-11T19:10:34.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.263 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.263 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.263 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.263 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_stat_pp 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_stat_pp' 2024-06-11T19:10:34.264 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36629/exe 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.265 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.266 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.266 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.266 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_stat_pp 2024-06-11T19:10:34.266 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 1 --2- 172.21.15.117:0/2533539005 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x558e13146260 0x558e13128010 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:10:34.266 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 1 --2- 172.21.15.117:0/2533539005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558e13142e50 0x558e131d7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:10:34.266 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 1 --2- 172.21.15.117:0/2533539005 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x558e13140980 0x558e131d79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:10:34.266 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 1 -- 172.21.15.117:0/2533539005 >> 172.21.15.117:0/2533539005 conn(0x558e1314a900 msgr2=0x558e1312b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:10:34.266 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 1 -- 172.21.15.117:0/2533539005 shutdown_connections 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 1 -- 172.21.15.117:0/2533539005 wait complete. 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 10 monclient: build_initial_monmap 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 10 monclient: monmap: 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: epoch 0 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: last_changed 0.000000 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: created 0.000000 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: min_mon_release 0 (unknown) 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: election_strategy: 1 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:10:34.267 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 1 librados: starting msgr at 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 1 librados: starting objecter 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 1 Processor -- start 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.132+0000 7efc0c645780 1 -- start start 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 1 librados: setting wanted keys 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 1 librados: calling monclient init 2024-06-11T19:10:34.268 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 10 monclient: init 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 10 monclient: _reopen_session rank -1 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 10 monclient: _add_conns ranks=[1,4,7,0,2,3,5,8,6] 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558e13140980 0x558e1327d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 10 monclient(hunting): picked mon.noname-f con 0x558e13140980 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x558e13142e50 0x558e1327d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 10 monclient(hunting): picked mon.noname-c con 0x558e13142e50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x558e13146260 0x558e1327dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 10 monclient(hunting): picked mon.noname-i con 0x558e13146260 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 10 monclient(hunting): start opening mon connection 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558e13143580 con 0x558e13140980 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 10 monclient(hunting): start opening mon connection 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x558e1313dbc0 con 0x558e13142e50 2024-06-11T19:10:34.269 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 10 monclient(hunting): start opening mon connection 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x558e131d7b50 con 0x558e13146260 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 10 monclient(hunting): _renew_subs 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.133+0000 7efc0c645780 10 monclient(hunting): authenticate will time out at 856.945881s 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x558e13146260 0x558e1327dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0994d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558e13140980 0x558e1327d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x558e13146260 0x558e1327dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.117:49990/0 (socket says 172.21.15.117:49990) 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 1 -- 172.21.15.117:0/1082957771 learned_addr learned my addr 172.21.15.117:0/1082957771 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 10 monclient(hunting): get_auth_request con 0x558e13146260 auth_method 0 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0994d640 10 monclient(hunting): get_auth_request con 0x558e13140980 auth_method 0 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0994d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:10:34.270 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0994d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:10:34.271 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0994d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:10:34.272 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.272 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.272 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.272 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.272 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.272 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.272 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.272 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.272 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.272 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.273 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0914c640 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x558e13142e50 0x558e1327d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:10:34.273 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0914c640 10 monclient(hunting): get_auth_request con 0x558e13142e50 auth_method 0 2024-06-11T19:10:34.273 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0914c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:10:34.273 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0914c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:10:34.273 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0914c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:10:34.273 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:10:34.273 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:10:34.273 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:10:34.273 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0994d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:10:34.274 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0994d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:10:34.274 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0994d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:10:34.274 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0914c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:10:34.274 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0914c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:10:34.274 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0914c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:10:34.274 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 10 monclient(hunting): handle_auth_done global_id 4579 payload 824 2024-06-11T19:10:34.274 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 1 -- 172.21.15.117:0/1082957771 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x558e13142e50 msgr2=0x558e1327d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:10:34.274 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x558e13142e50 0x558e1327d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:10:34.274 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 1 -- 172.21.15.117:0/1082957771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558e13140980 msgr2=0x558e1327d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:10:34.274 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558e13140980 0x558e1327d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:10:34.274 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 10 monclient: _finish_hunting 0 2024-06-11T19:10:34.274 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 1 monclient: found mon.noname-i 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0a14e640 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e1313e4a0 con 0x558e13146260 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.134+0000 7efc0994d640 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558e13140980 0x558e1327d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efc0a14e640 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x558e13146260 0x558e1327dcb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efc00000fb0 tx=0x7efc00000ed0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 10 client.?.objecter ms_handle_connect 0x558e13146260 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 1 -- 172.21.15.117:0/1082957771 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc000012f0 con 0x558e13146260 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 10 monclient: dump: 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: epoch 1 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:10:34.275 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: min_mon_release 19 (squid) 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: election_strategy: 3 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.276 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 10 monclient: _finish_auth 0 2024-06-11T19:10:34.277 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 10 monclient: _check_auth_tickets 2024-06-11T19:10:34.277 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:10:34.277 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efc0c645780 5 monclient: authenticate success, global_id 4579 2024-06-11T19:10:34.277 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efc0c645780 10 monclient: _renew_subs 2024-06-11T19:10:34.277 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efc0c645780 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:10:34.277 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x558e1313e780 con 0x558e13146260 2024-06-11T19:10:34.277 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efc0c645780 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:10:34.277 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efc0c645780 10 monclient: _renew_subs 2024-06-11T19:10:34.277 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efc0c645780 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:10:34.277 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({osdmap=0}) -- 0x558e13141730 con 0x558e13146260 2024-06-11T19:10:34.277 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efc0c645780 1 librados: init done 2024-06-11T19:10:34.280 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.281 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.282 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_pool: /' 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_pool.log 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.283 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.284 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efc0c645780 10 librados: wait_for_osdmap waiting 2024-06-11T19:10:34.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 1 -- 172.21.15.117:0/1082957771 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc0001a480 con 0x558e13146260 2024-06-11T19:10:34.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:10:34.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 1 -- 172.21.15.117:0/1082957771 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc0001aaa0 con 0x558e13146260 2024-06-11T19:10:34.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:10:34.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:10:34.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfbfff640 4 set_mon_vals no callback set 2024-06-11T19:10:34.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.135+0000 7efbfaffd640 10 monclient: dump: 2024-06-11T19:10:34.284 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: epoch 1 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: min_mon_release 19 (squid) 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: election_strategy: 3 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:10:34.285 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:10:34.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:10:34.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.136+0000 7efbfaffd640 1 -- 172.21.15.117:0/1082957771 <== mon.7 v2:172.21.15.117:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7efc00017070 con 0x558e13146260 2024-06-11T19:10:34.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efbfaffd640 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efbe803e430 0x7efbe80408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:10:34.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efbfaffd640 1 -- 172.21.15.117:0/1082957771 <== mon.7 v2:172.21.15.117:3303/0 5 ==== osd_map(18..18 src has 1..18) ==== 4346+0+0 (secure 0 0 0) 0x7efc00015020 con 0x558e13146260 2024-06-11T19:10:34.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efbfaffd640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_map(18..18 src has 1..18) 2024-06-11T19:10:34.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efbfaffd640 3 client.4579.objecter handle_osd_map got epochs [18,18] > 0 2024-06-11T19:10:34.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efbfaffd640 3 client.4579.objecter handle_osd_map decoding full epoch 18 2024-06-11T19:10:34.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efbfaffd640 20 client.4579.objecter dump_active .. 0 homeless 2024-06-11T19:10:34.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efc0c645780 10 librados: wait_for_osdmap done waiting 2024-06-11T19:10:34.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efc0c645780 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:10:34.286 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efbfc04ad60 con 0x558e13146260 2024-06-11T19:10:34.290 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36687/exe 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.291 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.292 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36573/exe 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.293 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.294 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.294 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.294 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.294 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_snapshots: /' 2024-06-11T19:10:34.294 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.294 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_stat_pp 2024-06-11T19:10:34.294 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36799 2024-06-11T19:10:34.294 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efc0994d640 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efbe803e430 0x7efbe80408f0 test api_stat_pp on pid 36799 2024-06-11T19:10:34.305 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_stat_pp on pid 36799' 2024-06-11T19:10:34.305 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36799 2024-06-11T19:10:34.305 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.305 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.305 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.305 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_watch_notify 2024-06-11T19:10:34.305 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-06-11T19:10:34.305 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_watch_notify' 2024-06-11T19:10:34.305 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.305 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.306 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.307 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.308 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_misc_pp: /' 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.309 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_watch_notify 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.310 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.311 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.312 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.313 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.314 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.315 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.315 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.315 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.315 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.315 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.315 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.315 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.315 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_watch_notify 2024-06-11T19:10:34.315 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36838 2024-06-11T19:10:34.315 INFO:tasks.workunit.client.0.smithi117.stdout:test api_watch_notify on pid 36838 2024-06-11T19:10:34.323 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_watch_notify on pid 36838' 2024-06-11T19:10:34.323 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36838 2024-06-11T19:10:34.323 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.323 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.323 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.324 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_watch_notify_pp 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.325 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36616/exe 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_watch_notify_pp' 2024-06-11T19:10:34.326 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.327 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_watch_notify_pp 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.328 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.329 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.330 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.330 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.330 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.330 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_watch_notify_pp 2024-06-11T19:10:34.330 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.330 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.330 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.330 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36851 2024-06-11T19:10:34.330 INFO:tasks.workunit.client.0.smithi117.stdout:test api_watch_notify_pp on pid 36851 2024-06-11T19:10:34.339 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_watch_notify_pp on pid 36851' 2024-06-11T19:10:34.339 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36851 2024-06-11T19:10:34.339 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.340 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.341 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_cmd 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_cmd' 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.342 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36707/exe 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.343 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_tier_pp: /' 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.344 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.345 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_cmd 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.346 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.347 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.348 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.348 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.348 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.348 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.348 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.348 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_cmd 2024-06-11T19:10:34.348 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-06-11T19:10:34.348 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36870 2024-06-11T19:10:34.348 INFO:tasks.workunit.client.0.smithi117.stdout:test api_cmd on pid 36870 2024-06-11T19:10:34.358 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_cmd on pid 36870' 2024-06-11T19:10:34.359 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36870 2024-06-11T19:10:34.359 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.359 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.359 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.359 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.359 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.359 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-06-11T19:10:34.359 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36721/exe 2024-06-11T19:10:34.359 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-06-11T19:10:34.359 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.359 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.359 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.360 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.361 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_stat.log 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_stat: /' 2024-06-11T19:10:34.362 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.363 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36799/exe 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.364 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.365 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.366 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_cmd_pp 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_cmd_pp' 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_stat_pp: /' 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_cmd_pp 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.367 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_cmd_pp 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36919 2024-06-11T19:10:34.368 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_cmd_pp on pid 36919' 2024-06-11T19:10:34.369 INFO:tasks.workunit.client.0.smithi117.stdout:test api_cmd_pp on pid 36919 2024-06-11T19:10:34.374 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36919 2024-06-11T19:10:34.374 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.374 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.374 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-06-11T19:10:34.374 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_service 2024-06-11T19:10:34.374 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_service' 2024-06-11T19:10:34.374 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_service 2024-06-11T19:10:34.374 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.374 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.374 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.375 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.376 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.377 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.378 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.378 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.378 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_service 2024-06-11T19:10:34.378 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36940 2024-06-11T19:10:34.378 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_service on pid 36940' 2024-06-11T19:10:34.378 INFO:tasks.workunit.client.0.smithi117.stdout:test api_service on pid 36940 2024-06-11T19:10:34.389 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36940 2024-06-11T19:10:34.389 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.389 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.389 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-06-11T19:10:34.389 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.389 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.389 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.389 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.390 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.390 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.390 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.390 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.390 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.390 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.390 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.390 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.390 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.390 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.390 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.390 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.391 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_service_pp 2024-06-11T19:10:34.392 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_service_pp' 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.393 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.394 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_service_pp 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.395 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.396 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.397 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_service_pp 2024-06-11T19:10:34.398 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36964 2024-06-11T19:10:34.398 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.398 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.398 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_service_pp on pid 36964' 2024-06-11T19:10:34.398 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.398 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.398 INFO:tasks.workunit.client.0.smithi117.stdout:test api_service_pp on pid 36964 2024-06-11T19:10:34.414 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=36964 2024-06-11T19:10:34.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.414 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.414 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.414 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.414 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.414 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.414 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.414 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.415 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_c_write_operations 2024-06-11T19:10:34.416 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_c_write_operations' 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.417 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.418 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.419 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36851/exe 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.420 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.421 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.422 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_c_write_operations 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.423 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.424 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.425 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.426 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.427 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.428 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.428 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.428 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.428 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_c_write_operations 2024-06-11T19:10:34.428 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37034 2024-06-11T19:10:34.428 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_c_write_operations on pid 37034' 2024-06-11T19:10:34.428 INFO:tasks.workunit.client.0.smithi117.stdout:test api_c_write_operations on pid 37034 2024-06-11T19:10:34.434 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37034 2024-06-11T19:10:34.434 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.434 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.434 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-06-11T19:10:34.434 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36919/exe 2024-06-11T19:10:34.434 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.434 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.434 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.434 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.434 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.434 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s api_c_read_operations 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.435 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' api_c_read_operations' 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo api_c_read_operations 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.436 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.437 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.438 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.439 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.439 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36870/exe 2024-06-11T19:10:34.439 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=api_c_read_operations 2024-06-11T19:10:34.439 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37059 2024-06-11T19:10:34.439 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test api_c_read_operations on pid 37059' 2024-06-11T19:10:34.439 INFO:tasks.workunit.client.0.smithi117.stdout:test api_c_read_operations on pid 37059 2024-06-11T19:10:34.450 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37059 2024-06-11T19:10:34.450 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.450 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.450 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-06-11T19:10:34.450 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.451 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.452 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.453 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_cmd: /' 2024-06-11T19:10:34.454 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36838/exe 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.455 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.456 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.457 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.458 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.459 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s list_parallel 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' list_parallel' 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_watch_notify: /' 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo list_parallel 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_cmd.log 2024-06-11T19:10:34.460 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=list_parallel 2024-06-11T19:10:34.461 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.461 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.461 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.461 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.461 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.461 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.461 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.461 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37107 2024-06-11T19:10:34.461 INFO:tasks.workunit.client.0.smithi117.stdout:test list_parallel on pid 37107 2024-06-11T19:10:34.472 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test list_parallel on pid 37107' 2024-06-11T19:10:34.472 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37107 2024-06-11T19:10:34.472 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s open_pools_parallel 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' open_pools_parallel' 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.473 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.474 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.475 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.476 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.477 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.478 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36940/exe 2024-06-11T19:10:34.479 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-06-11T19:10:34.480 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.481 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo open_pools_parallel 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.482 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.483 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.483 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.483 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.483 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.483 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_service: /' 2024-06-11T19:10:34.483 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.483 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.483 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=open_pools_parallel 2024-06-11T19:10:34.483 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.483 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37146 2024-06-11T19:10:34.483 INFO:tasks.workunit.client.0.smithi117.stdout:test open_pools_parallel on pid 37146 2024-06-11T19:10:34.492 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test open_pools_parallel on pid 37146' 2024-06-11T19:10:34.493 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37146 2024-06-11T19:10:34.493 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-06-11T19:10:34.493 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.493 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s delete_pools_parallel 2024-06-11T19:10:34.493 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' delete_pools_parallel' 2024-06-11T19:10:34.493 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-06-11T19:10:34.493 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.493 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.493 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.493 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.493 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.494 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.495 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.496 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.497 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_service.log 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37034/exe 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.498 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.499 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo delete_pools_parallel 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.500 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.501 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.501 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.501 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.501 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.501 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-06-11T19:10:34.501 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-06-11T19:10:34.501 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.501 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-06-11T19:10:34.501 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=delete_pools_parallel 2024-06-11T19:10:34.501 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37174 2024-06-11T19:10:34.501 INFO:tasks.workunit.client.0.smithi117.stdout:test delete_pools_parallel on pid 37174 2024-06-11T19:10:34.515 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test delete_pools_parallel on pid 37174' 2024-06-11T19:10:34.515 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37174 2024-06-11T19:10:34.515 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.515 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.516 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.516 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.516 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.516 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.516 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.516 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.516 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.516 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.516 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.516 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.516 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.516 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.517 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/36964/exe 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.518 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.519 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_service_pp: /' 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.520 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.521 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.522 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.523 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s cls 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' cls' 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37059/exe 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo cls 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.524 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-06-11T19:10:34.525 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.526 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37107/exe 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.527 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.528 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.528 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.528 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.528 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.528 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.528 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.528 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=cls 2024-06-11T19:10:34.528 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.528 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37242 2024-06-11T19:10:34.528 INFO:tasks.workunit.client.0.smithi117.stdout:test cls on pid 37242 2024-06-11T19:10:34.539 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test cls on pid 37242' 2024-06-11T19:10:34.539 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37242 2024-06-11T19:10:34.539 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.539 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.540 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_list_parallel.log 2024-06-11T19:10:34.541 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.542 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.543 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.544 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37146/exe 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.545 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.546 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s cmd 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' cmd' 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.547 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.548 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.549 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.549 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.549 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.549 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ list_parallel: /' 2024-06-11T19:10:34.549 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo cmd 2024-06-11T19:10:34.549 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=cmd 2024-06-11T19:10:34.549 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37302 2024-06-11T19:10:34.549 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test cmd on pid 37302' 2024-06-11T19:10:34.549 INFO:tasks.workunit.client.0.smithi117.stdout:test cmd on pid 37302 2024-06-11T19:10:34.552 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37302 2024-06-11T19:10:34.552 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.552 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.552 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.553 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s handler_error 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' handler_error' 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.554 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo handler_error 2024-06-11T19:10:34.555 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=handler_error 2024-06-11T19:10:34.555 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37326 2024-06-11T19:10:34.555 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test handler_error on pid 37326' 2024-06-11T19:10:34.555 INFO:tasks.workunit.client.0.smithi117.stdout:test handler_error on pid 37326 2024-06-11T19:10:34.561 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37326 2024-06-11T19:10:34.561 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.561 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.561 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-06-11T19:10:34.561 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s io 2024-06-11T19:10:34.561 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' io' 2024-06-11T19:10:34.561 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo io 2024-06-11T19:10:34.561 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.561 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.562 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.563 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.564 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.565 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.566 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.566 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.566 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=io 2024-06-11T19:10:34.566 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.566 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.566 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.566 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.566 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.566 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.566 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.566 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37359 2024-06-11T19:10:34.567 INFO:tasks.workunit.client.0.smithi117.stdout:test io on pid 37359 2024-06-11T19:10:34.581 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test io on pid 37359' 2024-06-11T19:10:34.581 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37359 2024-06-11T19:10:34.581 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.581 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.581 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.581 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.581 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.582 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.582 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.582 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.582 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.582 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.582 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.582 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.582 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.582 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.582 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s ec_io 2024-06-11T19:10:34.582 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.582 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' ec_io' 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.583 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37174/exe 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.584 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.585 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.586 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.587 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.588 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.589 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.590 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.591 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo ec_io 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.592 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.593 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.594 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.595 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.595 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.595 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.595 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.595 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.595 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.595 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=ec_io 2024-06-11T19:10:34.595 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37433 2024-06-11T19:10:34.595 INFO:tasks.workunit.client.0.smithi117.stdout:test ec_io on pid 37433 2024-06-11T19:10:34.604 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test ec_io on pid 37433' 2024-06-11T19:10:34.604 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37433 2024-06-11T19:10:34.604 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.604 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s list 2024-06-11T19:10:34.604 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.604 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' list' 2024-06-11T19:10:34.604 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-06-11T19:10:34.604 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-06-11T19:10:34.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo list 2024-06-11T19:10:34.604 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.604 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.605 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.606 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.607 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.608 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=list 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.609 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.610 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.610 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.610 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.610 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.610 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.610 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.610 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.610 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.610 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.610 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.610 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37475 2024-06-11T19:10:34.610 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test list on pid 37475' 2024-06-11T19:10:34.611 INFO:tasks.workunit.client.0.smithi117.stdout:test list on pid 37475 2024-06-11T19:10:34.614 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37475 2024-06-11T19:10:34.614 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s ec_list 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' ec_list' 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo ec_list 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.615 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37326/exe 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37242/exe 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.616 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=ec_list 2024-06-11T19:10:34.617 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.618 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37495 2024-06-11T19:10:34.618 INFO:tasks.workunit.client.0.smithi117.stdout:test ec_list on pid 37495 2024-06-11T19:10:34.628 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test ec_list on pid 37495' 2024-06-11T19:10:34.628 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37495 2024-06-11T19:10:34.628 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.628 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s misc 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.629 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' misc' 2024-06-11T19:10:34.630 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_handler_error 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_handler_error.log 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ cls: /' 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_cls 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_cls.log 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.631 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.632 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.633 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ handler_error: /' 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.634 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.635 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.636 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.637 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.638 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo misc 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=misc 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37550 2024-06-11T19:10:34.639 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test misc on pid 37550' 2024-06-11T19:10:34.640 INFO:tasks.workunit.client.0.smithi117.stdout:test misc on pid 37550 2024-06-11T19:10:34.651 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37550 2024-06-11T19:10:34.651 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.651 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.651 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37359/exe 2024-06-11T19:10:34.651 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.651 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.651 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.651 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.651 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.651 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.651 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.651 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.652 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.653 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.654 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ io: /' 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.655 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_io.log 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.656 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s pool 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' pool' 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_io 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.657 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.658 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo pool 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.659 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.660 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.661 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.661 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.661 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.661 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.661 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.661 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.661 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.661 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.661 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.661 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=pool 2024-06-11T19:10:34.661 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37591 2024-06-11T19:10:34.661 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test pool on pid 37591' 2024-06-11T19:10:34.662 INFO:tasks.workunit.client.0.smithi117.stdout:test pool on pid 37591 2024-06-11T19:10:34.667 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37591 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s read_operations 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' read_operations' 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.668 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.669 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo read_operations 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.670 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.671 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37302/exe 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=read_operations 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.672 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.673 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.673 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.673 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.673 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.673 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.673 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.673 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37612 2024-06-11T19:10:34.673 INFO:tasks.workunit.client.0.smithi117.stdout:test read_operations on pid 37612 2024-06-11T19:10:34.686 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test read_operations on pid 37612' 2024-06-11T19:10:34.686 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37612 2024-06-11T19:10:34.686 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.686 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.687 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.688 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s snapshots 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' snapshots' 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.689 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_cmd 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ cmd: /' 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.690 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.691 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_cmd.log 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.692 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.693 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.694 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37433/exe 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo snapshots 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.695 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.696 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.697 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.698 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.698 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.698 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.698 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.698 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.698 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.698 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.698 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.698 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=snapshots 2024-06-11T19:10:34.698 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37659 2024-06-11T19:10:34.698 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.699 INFO:tasks.workunit.client.0.smithi117.stdout:test snapshots on pid 37659 2024-06-11T19:10:34.705 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test snapshots on pid 37659' 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37659 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.706 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.707 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s watch_notify 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' watch_notify' 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.708 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.709 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_ec_io.log 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo watch_notify 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.710 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.711 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.712 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.713 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.713 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.713 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=watch_notify 2024-06-11T19:10:34.713 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37706 2024-06-11T19:10:34.713 INFO:tasks.workunit.client.0.smithi117.stdout:test watch_notify on pid 37706 2024-06-11T19:10:34.726 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test watch_notify on pid 37706' 2024-06-11T19:10:34.726 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37706 2024-06-11T19:10:34.726 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.727 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:++ printf %25s write_operations 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37475/exe 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:+ r=' write_operations' 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.728 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.729 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.730 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.731 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.732 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_ec_io 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_list 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_list.log 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ list: /' 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ ec_io: /' 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.733 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.734 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.735 INFO:tasks.workunit.client.0.smithi117.stderr:++ echo write_operations 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:++ awk '{print $1}' 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37495/exe 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37550/exe 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.736 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.737 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.738 INFO:tasks.workunit.client.0.smithi117.stderr:+ ff=write_operations 2024-06-11T19:10:34.739 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.739 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.739 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.739 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37757 2024-06-11T19:10:34.739 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.739 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'test write_operations on pid 37757' 2024-06-11T19:10:34.739 INFO:tasks.workunit.client.0.smithi117.stdout:test write_operations on pid 37757 2024-06-11T19:10:34.766 INFO:tasks.workunit.client.0.smithi117.stderr:+ pids[$f]=37757 2024-06-11T19:10:34.766 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.766 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.766 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.766 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:+ ret=0 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' 1 -eq 1 ']' 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36616 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36616 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.767 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.768 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_ec_list 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -f /etc/bashrc ']' 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:+ . /etc/bashrc 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -z '' ']' 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_ec_list.log 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:++ BASHRCSOURCED=Y 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:++ shopt -q login_shell 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ misc: /' 2024-06-11T19:10:34.769 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.770 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.771 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.771 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.771 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.771 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.771 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.771 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.771 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.771 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.771 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.771 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.771 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.771 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ ec_list: /' 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_misc.log 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_misc 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:+++ umask 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' 0022 -eq 0 ']' 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:++ SHELL=/bin/bash 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37591/exe 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.772 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.773 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.774 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.775 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.776 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37612/exe 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_pool.log 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_pool 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'grep=grep --color=auto' 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ pool: /' 2024-06-11T19:10:34.777 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorls.sh 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' '!' -t 0 ']' 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:+++ return 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export S_COLORS=auto 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:+++ S_COLORS=auto 2024-06-11T19:10:34.778 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.779 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_read_operations 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_read_operations.log 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.780 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ read_operations: /' 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.781 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.782 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-11T19:10:34.783 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/gawk.sh 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/lang.sh 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.784 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.785 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/locale 2024-06-11T19:10:34.785 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.785 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.785 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.785 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.785 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-11T19:10:34.785 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.785 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.785 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.785 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.785 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-11T19:10:34.785 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.786 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37659/exe 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.787 INFO:tasks.workunit.client.0.smithi117.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:++++ export LANG=en_US.UTF-8 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:++++ LANG=en_US.UTF-8 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ LANG=en_US.UTF-8 2024-06-11T19:10:34.788 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:+++ unset LANG_backup config 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n '' ']' 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' dumb = linux ']' 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.789 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/less.sh 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ for i in /etc/profile.d/*.sh 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ '[' '' ']' 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:++ . /etc/profile.d/which2.sh 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.790 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37706/exe 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ snapshots: /' 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.791 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_snapshots 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:+++++ readlink /proc/37757/exe 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:++++ basename /usr/bin/bash 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_declare='declare -f' 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:+++ which_opt=-f 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export which_declare 2024-06-11T19:10:34.792 INFO:tasks.workunit.client.0.smithi117.stderr:+++ export -f which 2024-06-11T19:10:34.793 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset i 2024-06-11T19:10:34.793 INFO:tasks.workunit.client.0.smithi117.stderr:++ unset -f pathmunge 2024-06-11T19:10:34.793 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-11T19:10:34.793 INFO:tasks.workunit.client.0.smithi117.stderr:+ export PATH 2024-06-11T19:10:34.793 INFO:tasks.workunit.client.0.smithi117.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-11T19:10:34.793 INFO:tasks.workunit.client.0.smithi117.stderr:+ unset rc 2024-06-11T19:10:34.793 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ watch_notify: /' 2024-06-11T19:10:34.793 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_write_operations.log 2024-06-11T19:10:34.793 INFO:tasks.workunit.client.0.smithi117.stderr:+ tee ceph_test_neorados_snapshots.log 2024-06-11T19:10:34.793 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_watch_notify 2024-06-11T19:10:34.793 INFO:tasks.workunit.client.0.smithi117.stderr:+ sed 's/^/ write_operations: /' 2024-06-11T19:10:34.793 INFO:tasks.workunit.client.0.smithi117.stderr:+ ceph_test_neorados_write_operations 2024-06-11T19:10:34.865 INFO:tasks.workunit.client.0.smithi117.stdout:unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:10:34.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efc0994d640 10 monclient: get_auth_request con 0x7efbe803e430 auth_method 0 2024-06-11T19:10:34.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efbfaffd640 1 -- 172.21.15.117:0/1082957771 <== mon.7 v2:172.21.15.117:3303/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7efc0000ec60 con 0x558e13146260 2024-06-11T19:10:34.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efbfaffd640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-06-11T19:10:34.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efbfb7fe640 10 client.4579.objecter _get_latest_version latest 18, have it 2024-06-11T19:10:34.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efc0c645780 10 client.4579.objecter create_pool name=ceph_test_rados_api_asio 2024-06-11T19:10:34.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efc0c645780 10 client.4579.objecter pool_op_submit 1 2024-06-11T19:10:34.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efc0c645780 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:10:34.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.137+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x558e130d3eb0 con 0x558e13146260 2024-06-11T19:10:34.866 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.138+0000 7efc0994d640 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efbe803e430 0x7efbe80408f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efbfc053000 tx=0x7efbfc04e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.138+0000 7efbfaffd640 10 client.4579.objecter ms_handle_connect 0x7efbe803e430 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.863+0000 7efbfaffd640 1 -- 172.21.15.117:0/1082957771 <== mon.7 v2:172.21.15.117:3303/0 7 ==== pool_op_reply(tid 1 (0) Success v19) ==== 43+0+0 (secure 0 0 0) 0x558e1312a260 con 0x558e13146260 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.863+0000 7efbfaffd640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 pool_op_reply(tid 1 (0) Success v19) 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.863+0000 7efbfaffd640 10 client.4579.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v19) 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.863+0000 7efbfaffd640 10 client.4579.objecter have request 1 at 0x558e1313f7a0 Op: create 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.863+0000 7efbfaffd640 20 client.4579.objecter waiting for client to reach epoch 19 before calling back 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.863+0000 7efbfaffd640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.863+0000 7efbfaffd640 10 monclient: _renew_subs 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.863+0000 7efbfaffd640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.863+0000 7efbfaffd640 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({osdmap=19}) -- 0x7efbe8043b90 con 0x558e13146260 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.863+0000 7efbfaffd640 10 client.4579.objecter done 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.865+0000 7efbfaffd640 1 -- 172.21.15.117:0/1082957771 <== mon.7 v2:172.21.15.117:3303/0 8 ==== osd_map(19..19 src has 1..19) ==== 13849+0+0 (secure 0 0 0) 0x7efc00052080 con 0x558e13146260 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.865+0000 7efbfaffd640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_map(19..19 src has 1..19) 2024-06-11T19:10:34.867 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.865+0000 7efbfaffd640 3 client.4579.objecter handle_osd_map got epochs [19,19] > 18 2024-06-11T19:10:36.040 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.865+0000 7efbfaffd640 3 client.4579.objecter handle_osd_map decoding incremental epoch 19 2024-06-11T19:10:36.040 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.866+0000 7efbfaffd640 20 client.4579.objecter dump_active .. 0 homeless 2024-06-11T19:10:36.040 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 10 librados: set snap read head -> 1 2024-06-11T19:10:36.040 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 10 librados: writefull oid=exist nspace= 2024-06-11T19:10:36.040 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e13284480 2024-06-11T19:10:36.040 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-06-11T19:10:36.040 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.040 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x558e13284e40 0x558e13287230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 2 osd.2 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 2 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 15 client.4579.objecter _send_op 2 to 7.1 on osd.2 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:2 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x558e13287770 con 0x558e13284e40 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.872+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.875+0000 7efc0914c640 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x558e13284e40 0x558e13287230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.875+0000 7efc0914c640 10 monclient: get_auth_request con 0x558e13284e40 auth_method 0 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.877+0000 7efc0914c640 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x558e13284e40 0x558e13287230 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:10:36.041 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:34.878+0000 7efbfaffd640 10 client.4579.objecter ms_handle_connect 0x558e13284e40 2024-06-11T19:10:36.042 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.040+0000 7efbfaffd640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 1 ==== osd_map(20..20 src has 1..20) ==== 8029+0+0 (crc 0 0 0) 0x7efbf40099b0 con 0x558e13284e40 2024-06-11T19:10:36.042 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.040+0000 7efbfaffd640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_map(20..20 src has 1..20) 2024-06-11T19:10:36.042 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.040+0000 7efbfaffd640 3 client.4579.objecter handle_osd_map got epochs [20,20] > 19 2024-06-11T19:10:36.123 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.040+0000 7efbfaffd640 3 client.4579.objecter handle_osd_map decoding incremental epoch 20 2024-06-11T19:10:36.123 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.040+0000 7efbfaffd640 10 client.4579.objecter checking op 2 2024-06-11T19:10:36.123 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.040+0000 7efbfaffd640 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-06-11T19:10:36.124 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.040+0000 7efbfaffd640 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.124 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.040+0000 7efbfaffd640 20 client.4579.objecter dump_active .. 0 homeless 2024-06-11T19:10:36.124 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.040+0000 7efbfaffd640 20 client.4579.objecter 2 7.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2024-06-11T19:10:36.124 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-06-11T19:10:36.124 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 2 ==== osd_op_reply(2 exist [writefull 0~5] v20'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.124 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(2 exist [writefull 0~5] v20'1 uv1 ondisk = 0) 2024-06-11T19:10:36.124 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.124 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 2 ondisk uv 1 in 7.1 attempt 0 2024-06-11T19:10:36.124 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.124 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 2 2024-06-11T19:10:36.124 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0914c640 15 client.4579.objecter _finish_op 2 2024-06-11T19:10:36.124 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 2 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 10 librados: Objecter returned from writefull r=0 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e13284480 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 3 osd.2 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 3 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 15 client.4579.objecter _send_op 3 to 7.1 on osd.2 2024-06-11T19:10:36.125 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:3 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) -- 0x558e1313bd80 con 0x558e13284e40 2024-06-11T19:10:36.126 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.126 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.126 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e1313c190 2024-06-11T19:10:36.126 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 20 client.4579.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 10 client.4579.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 4 osd.2 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 4 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 15 client.4579.objecter _send_op 4 to 7.1 on osd.2 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:4 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) -- 0x558e1313c640 con 0x558e13284e40 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 5 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.124+0000 7efc0c645780 5 client.4579.objecter 2 in flight 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-06-11T19:10:36.127 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 3 ondisk uv 1 in 7.1 attempt 0 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 5 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 3 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 15 client.4579.objecter _finish_op 3 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 5 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 3 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 4 ondisk uv 0 in 7.1 attempt 0 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 10 client.4579.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-06-11T19:10:36.128 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 4 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 15 client.4579.objecter _finish_op 4 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 4 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (4 ms) 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e1313c190 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 5 osd.2 2024-06-11T19:10:36.129 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 5 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 15 client.4579.objecter _send_op 5 to 7.1 on osd.2 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:5 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) -- 0x558e13284480 con 0x558e13284e40 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e13117100 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 20 client.4579.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 10 client.4579.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 6 osd.2 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.130 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 6 2024-06-11T19:10:36.131 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 15 client.4579.objecter _send_op 6 to 7.1 on osd.2 2024-06-11T19:10:36.131 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:6 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) -- 0x558e131175b0 con 0x558e13284e40 2024-06-11T19:10:36.131 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 5 2024-06-11T19:10:36.131 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0c645780 5 client.4579.objecter 2 in flight 2024-06-11T19:10:36.131 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.131 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 5 ondisk uv 1 in 7.1 attempt 0 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 5 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 5 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 15 client.4579.objecter _finish_op 5 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 5 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 5 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.127+0000 7efc0914c640 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 6 ondisk uv 0 in 7.1 attempt 0 2024-06-11T19:10:36.132 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 10 client.4579.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 6 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 15 client.4579.objecter _finish_op 6 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 6 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e13117100 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.133 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 7 osd.2 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 7 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 15 client.4579.objecter _send_op 7 to 7.1 on osd.2 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:7 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) -- 0x558e1313c170 con 0x558e13284e40 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e131179a0 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 10 client.4579.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 8 osd.2 2024-06-11T19:10:36.134 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 8 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 15 client.4579.objecter _send_op 8 to 7.1 on osd.2 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:8 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) -- 0x558e13117e50 con 0x558e13284e40 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 5 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 5 client.4579.objecter 2 in flight 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 7 ondisk uv 1 in 7.1 attempt 0 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 5 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 7 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 15 client.4579.objecter _finish_op 7 2024-06-11T19:10:36.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 5 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 7 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (0 ms) 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 8 ondisk uv 0 in 7.1 attempt 0 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 10 client.4579.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 8 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 15 client.4579.objecter _finish_op 8 2024-06-11T19:10:36.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 8 2024-06-11T19:10:36.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-06-11T19:10:36.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 librados: queue_aio_write 0x558e1308b300 completion 0x558e1313e1c0 write_seq 1 2024-06-11T19:10:36.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e131179a0 2024-06-11T19:10:36.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-06-11T19:10:36.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 9 osd.2 2024-06-11T19:10:36.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 9 2024-06-11T19:10:36.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 15 client.4579.objecter _send_op 9 to 7.1 on osd.2 2024-06-11T19:10:36.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:9 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) -- 0x558e13117100 con 0x558e13284e40 2024-06-11T19:10:36.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.128+0000 7efc0c645780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-06-11T19:10:36.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (7 ms) 2024-06-11T19:10:36.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-06-11T19:10:36.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.135+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 9 ==== osd_op_reply(9 exist [write 0~5] v20'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.135+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(9 exist [write 0~5] v20'2 uv2 ondisk = 0) 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.135+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.135+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 9 ondisk uv 2 in 7.1 attempt 0 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.135+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.135+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 9 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.135+0000 7efc0914c640 15 client.4579.objecter _finish_op 9 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.135+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.135+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 9 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.135+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.135+0000 7efc0914c640 20 librados: complete_aio_write 0x558e1313e1c0 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 20 librados: queue_aio_write 0x558e1308b300 completion 0x558e1313e2d0 write_seq 2 2024-06-11T19:10:36.139 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e131179a0 2024-06-11T19:10:36.140 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-06-11T19:10:36.140 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.140 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.140 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.140 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 10 osd.2 2024-06-11T19:10:36.140 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.140 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 10 2024-06-11T19:10:36.140 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 15 client.4579.objecter _send_op 10 to 7.1 on osd.2 2024-06-11T19:10:36.140 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:10 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) -- 0x558e13117100 con 0x558e13284e40 2024-06-11T19:10:36.140 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.144 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.144 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.136+0000 7efc0c645780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-06-11T19:10:36.144 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 10 ==== osd_op_reply(10 exist [write 0~5] v20'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.144 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(10 exist [write 0~5] v20'3 uv3 ondisk = 0) 2024-06-11T19:10:36.144 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.144 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 10 ondisk uv 3 in 7.1 attempt 0 2024-06-11T19:10:36.144 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 10 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0914c640 15 client.4579.objecter _finish_op 10 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 10 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0914c640 20 librados: complete_aio_write 0x558e1313e2d0 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (7 ms) 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 20 librados: queue_aio_write 0x558e1308b300 completion 0x558e131d7420 write_seq 3 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e131179a0 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-06-11T19:10:36.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 11 osd.2 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 11 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 15 client.4579.objecter _send_op 11 to 7.1 on osd.2 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:11 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) -- 0x558e1311b060 con 0x558e13284e40 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.142+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.143+0000 7efc0c645780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 11 ==== osd_op_reply(11 exist [write 0~5] v20'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(11 exist [write 0~5] v20'4 uv4 ondisk = 0) 2024-06-11T19:10:36.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 11 ondisk uv 4 in 7.1 attempt 0 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 11 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0914c640 15 client.4579.objecter _finish_op 11 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 11 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0914c640 20 librados: complete_aio_write 0x558e131d7420 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (2 ms) 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e131179a0 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-06-11T19:10:36.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.144+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 12 osd.2 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 12 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 15 client.4579.objecter _send_op 12 to 7.1 on osd.2 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:12 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) -- 0x558e1311b060 con 0x558e13284e40 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e13118240 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-06-11T19:10:36.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 10 client.4579.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 13 osd.2 2024-06-11T19:10:36.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 13 2024-06-11T19:10:36.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 15 client.4579.objecter _send_op 13 to 7.1 on osd.2 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:13 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) -- 0x558e131186f0 con 0x558e13284e40 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 5 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 5 client.4579.objecter 2 in flight 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 12 ondisk uv 4 in 7.1 attempt 0 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 5 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 12 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 15 client.4579.objecter _finish_op 12 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 5 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 12 2024-06-11T19:10:36.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 13 ondisk uv 0 in 7.1 attempt 0 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 10 client.4579.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 13 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 15 client.4579.objecter _finish_op 13 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 13 2024-06-11T19:10:36.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e131179a0 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 14 osd.2 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 14 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 15 client.4579.objecter _send_op 14 to 7.1 on osd.2 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:14 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) -- 0x558e1311b060 con 0x558e13284e40 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e13118bf0 2024-06-11T19:10:36.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 10 client.4579.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 15 osd.2 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 15 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 15 client.4579.objecter _send_op 15 to 7.1 on osd.2 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:15 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) -- 0x558e13118240 con 0x558e13284e40 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 5 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.145+0000 7efc0c645780 5 client.4579.objecter 2 in flight 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-06-11T19:10:36.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 14 ondisk uv 4 in 7.1 attempt 0 2024-06-11T19:10:36.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 5 2024-06-11T19:10:36.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 14 2024-06-11T19:10:36.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 15 client.4579.objecter _finish_op 14 2024-06-11T19:10:36.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 5 2024-06-11T19:10:36.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 14 2024-06-11T19:10:36.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-06-11T19:10:36.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 15 ondisk uv 0 in 7.1 attempt 0 2024-06-11T19:10:36.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 10 client.4579.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 15 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 15 client.4579.objecter _finish_op 15 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 15 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.146+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (2 ms) 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e131179a0 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 16 osd.2 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 16 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 15 client.4579.objecter _send_op 16 to 7.1 on osd.2 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:16 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) -- 0x558e13118240 con 0x558e13284e40 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e13118bf0 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 20 client.4579.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 10 client.4579.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 17 osd.2 2024-06-11T19:10:36.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 17 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 15 client.4579.objecter _send_op 17 to 7.1 on osd.2 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:17 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) -- 0x558e13118630 con 0x558e13284e40 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 5 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.147+0000 7efc0c645780 5 client.4579.objecter 2 in flight 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 16 ondisk uv 4 in 7.1 attempt 0 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 5 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 16 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 15 client.4579.objecter _finish_op 16 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 5 2024-06-11T19:10:36.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 16 2024-06-11T19:10:36.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-06-11T19:10:36.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 17 ondisk uv 0 in 7.1 attempt 0 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 10 client.4579.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 17 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 15 client.4579.objecter _finish_op 17 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 17 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0c645780 20 librados: queue_aio_write 0x558e1308b300 completion 0x558e1311b170 write_seq 4 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e131179a0 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.148+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 18 osd.2 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.149+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.149+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 18 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.149+0000 7efc0c645780 15 client.4579.objecter _send_op 18 to 7.1 on osd.2 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.149+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:18 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) -- 0x558e13118240 con 0x558e13284e40 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.149+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.149+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.151+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 18 ==== osd_op_reply(18 exist [writefull 0~5] v20'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.151+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(18 exist [writefull 0~5] v20'5 uv5 ondisk = 0) 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.151+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.151+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 18 ondisk uv 5 in 7.1 attempt 0 2024-06-11T19:10:36.160 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.151+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.151+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 18 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.151+0000 7efc0914c640 15 client.4579.objecter _finish_op 18 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.151+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.151+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 18 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.151+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.151+0000 7efc0914c640 20 librados: complete_aio_write 0x558e1311b170 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (4 ms) 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 20 librados: queue_aio_write 0x558e1308b300 completion 0x558e1311b280 write_seq 5 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e131179a0 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.161 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 19 osd.2 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 19 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 15 client.4579.objecter _send_op 19 to 7.1 on osd.2 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:19 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) -- 0x558e13118240 con 0x558e13284e40 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.152+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 19 ==== osd_op_reply(19 exist [writefull 0~5] v20'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(19 exist [writefull 0~5] v20'6 uv6 ondisk = 0) 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 19 ondisk uv 6 in 7.1 attempt 0 2024-06-11T19:10:36.162 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 19 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0914c640 15 client.4579.objecter _finish_op 19 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 19 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0914c640 20 librados: complete_aio_write 0x558e1311b280 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (2 ms) 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 20 librados: queue_aio_write 0x558e1308b300 completion 0x558e1311b170 write_seq 6 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 10 client.4579.objecter _op_submit op 0x558e131179a0 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 20 client.4579.objecter _calc_target epoch 20 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 20 client.4579.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 10 client.4579.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [2,1] primary 2 2024-06-11T19:10:36.163 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 20 client.4579.objecter _get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 10 client.4579.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 20 osd.2 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 20 client.4579.objecter get_session s=0x558e13115640 osd=2 3 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 15 client.4579.objecter _session_op_assign 2 20 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 15 client.4579.objecter _send_op 20 to 7.1 on osd.2 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- osd_op(unknown.0.0:20 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) -- 0x558e13118240 con 0x558e13284e40 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.154+0000 7efc0c645780 5 client.4579.objecter 1 in flight 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.156+0000 7efc0914c640 1 -- 172.21.15.117:0/1082957771 <== osd.2 v2:172.21.15.98:6808/3751501255 20 ==== osd_op_reply(20 exist [writefull 0~5] v20'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7efbf4009cb0 con 0x558e13284e40 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.156+0000 7efc0914c640 10 client.4579.objecter ms_dispatch 0x558e130a3a70 osd_op_reply(20 exist [writefull 0~5] v20'7 uv7 ondisk = 0) 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.156+0000 7efc0914c640 10 client.4579.objecter in handle_osd_op_reply 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.156+0000 7efc0914c640 7 client.4579.objecter handle_osd_op_reply 20 ondisk uv 7 in 7.1 attempt 0 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.156+0000 7efc0914c640 10 client.4579.objecter op 0 rval 0 len 0 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.156+0000 7efc0914c640 15 client.4579.objecter handle_osd_op_reply completed tid 20 2024-06-11T19:10:36.164 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.156+0000 7efc0914c640 15 client.4579.objecter _finish_op 20 2024-06-11T19:10:36.165 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.156+0000 7efc0914c640 20 client.4579.objecter put_session s=0x558e13115640 osd=2 4 2024-06-11T19:10:36.165 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.156+0000 7efc0914c640 15 client.4579.objecter _session_op_remove 2 20 2024-06-11T19:10:36.165 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.156+0000 7efc0914c640 5 client.4579.objecter 0 in flight 2024-06-11T19:10:36.165 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.156+0000 7efc0914c640 20 librados: complete_aio_write 0x558e1311b170 2024-06-11T19:10:36.165 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2 ms) 2024-06-11T19:10:36.165 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.156+0000 7efc0c645780 10 librados: watch_flush enter 2024-06-11T19:10:36.165 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 10 librados: watch_flush exit 2024-06-11T19:10:36.165 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 10 client.4579.objecter close_session for osd.2 2024-06-11T19:10:36.165 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x558e13284e40 msgr2=0x558e13287230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x558e13284e40 0x558e13287230 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 20 client.4579.objecter put_session s=0x558e13115640 osd=2 1 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 20 client.4579.objecter shutdown clearing up homeless session... 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 10 client.4579.objecter successfully canceled tick 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efbe803e430 msgr2=0x7efbe80408f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efbe803e430 0x7efbe80408f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efbfc053000 tx=0x7efbfc04e000 comp rx=0 tx=0).stop 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 10 monclient: shutdown 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x558e13146260 msgr2=0x558e1327dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 1 --2- 172.21.15.117:0/1082957771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x558e13146260 0x558e1327dcb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efc00000fb0 tx=0x7efc00000ed0 comp rx=0 tx=0).stop 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0a14e640 1 -- 172.21.15.117:0/1082957771 reap_dead start 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 shutdown_connections 2024-06-11T19:10:36.166 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 >> 172.21.15.117:0/1082957771 conn(0x558e1314a900 msgr2=0x558e1312a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:10:36.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 shutdown_connections 2024-06-11T19:10:36.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.157+0000 7efc0c645780 1 -- 172.21.15.117:0/1082957771 wait complete. 2024-06-11T19:10:36.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.158+0000 7efc0c645780 1 librados: shutdown 2024-06-11T19:10:36.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [----------] 12 tests from AsioRados (33 ms total) 2024-06-11T19:10:36.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: 2024-06-11T19:10:36.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [----------] Global test environment tear-down 2024-06-11T19:10:36.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (2031 ms total) 2024-06-11T19:10:36.167 INFO:tasks.workunit.client.0.smithi117.stdout: api_asio: [ PASSED ] 12 tests. 2024-06-11T19:10:36.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-06-11T19:10:36.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] Global test environment set-up. 2024-06-11T19:10:36.548 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] 7 tests from LibRadosList 2024-06-11T19:10:36.549 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-06-11T19:10:36.549 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.ListObjects (298 ms) 2024-06-11T19:10:36.549 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-06-11T19:10:36.549 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (35 ms) 2024-06-11T19:10:36.549 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-06-11T19:10:36.549 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset foo1,foo2,foo3 2024-06-11T19:10:36.550 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo1 2024-06-11T19:10:36.550 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo2 2024-06-11T19:10:36.550 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo3 2024-06-11T19:10:36.550 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset foo1,foo4,foo5 2024-06-11T19:10:36.550 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo4 2024-06-11T19:10:36.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo5 2024-06-11T19:10:36.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo1 2024-06-11T19:10:36.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset foo6,foo7 2024-06-11T19:10:36.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo7 2024-06-11T19:10:36.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo6 2024-06-11T19:10:36.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-06-11T19:10:36.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns1:foo4 2024-06-11T19:10:36.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns1:foo5 2024-06-11T19:10:36.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns2:foo7 2024-06-11T19:10:36.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns2:foo6 2024-06-11T19:10:36.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns1:foo1 2024-06-11T19:10:36.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: :foo1 2024-06-11T19:10:36.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: :foo2 2024-06-11T19:10:36.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: :foo3 2024-06-11T19:10:36.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (121 ms) 2024-06-11T19:10:36.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-06-11T19:10:36.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 1 0 2024-06-11T19:10:36.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 10 0 2024-06-11T19:10:36.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 13 0 2024-06-11T19:10:36.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 7 0 2024-06-11T19:10:36.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 14 0 2024-06-11T19:10:36.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 0 0 2024-06-11T19:10:36.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 15 0 2024-06-11T19:10:36.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 11 0 2024-06-11T19:10:36.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 5 0 2024-06-11T19:10:36.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 8 0 2024-06-11T19:10:36.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 6 0 2024-06-11T19:10:36.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 3 0 2024-06-11T19:10:36.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 4 0 2024-06-11T19:10:36.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 12 0 2024-06-11T19:10:36.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 9 0 2024-06-11T19:10:36.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 2 0 2024-06-11T19:10:36.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-06-11T19:10:36.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (115 ms) 2024-06-11T19:10:36.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-06-11T19:10:36.557 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: x cursor=MIN 2024-06-11T19:10:36.558 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=1 cursor=10:02547ec2:::1:head 2024-06-11T19:10:36.558 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=10 cursor=10:52ea6a34:::10:head 2024-06-11T19:10:36.558 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=13 cursor=10:566253c9:::13:head 2024-06-11T19:10:36.558 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=7 cursor=10:5c6b0b28:::7:head 2024-06-11T19:10:36.559 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=14 cursor=10:62a1935d:::14:head 2024-06-11T19:10:36.559 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=0 cursor=10:6cac518f:::0:head 2024-06-11T19:10:36.559 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=15 cursor=10:863748b0:::15:head 2024-06-11T19:10:36.559 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=11 cursor=10:89d3ae78:::11:head 2024-06-11T19:10:36.559 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=5 cursor=10:b29083e3:::5:head 2024-06-11T19:10:36.559 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=8 cursor=10:bd63b0f1:::8:head 2024-06-11T19:10:36.559 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=6 cursor=10:c4fdafeb:::6:head 2024-06-11T19:10:36.560 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=3 cursor=10:cfc208b3:::3:head 2024-06-11T19:10:36.560 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=4 cursor=10:d83876eb:::4:head 2024-06-11T19:10:36.560 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=12 cursor=10:de5d7c5f:::12:head 2024-06-11T19:10:36.560 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=9 cursor=10:e960b815:::9:head 2024-06-11T19:10:36.560 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > oid=2 cursor=10:f905c69b:::2:head 2024-06-11T19:10:36.560 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: FIRST> seek to MIN oid=1 2024-06-11T19:10:36.561 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=1 cursor=10:02547ec2:::1:head 2024-06-11T19:10:36.561 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:02547ec2:::1:head 2024-06-11T19:10:36.561 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:02547ec2:::1:head -> 1 2024-06-11T19:10:36.562 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=10 cursor=10:52ea6a34:::10:head 2024-06-11T19:10:36.562 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:52ea6a34:::10:head 2024-06-11T19:10:36.562 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:52ea6a34:::10:head -> 10 2024-06-11T19:10:36.562 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=13 cursor=10:566253c9:::13:head 2024-06-11T19:10:36.562 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:566253c9:::13:head 2024-06-11T19:10:36.562 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:566253c9:::13:head -> 13 2024-06-11T19:10:36.563 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=7 cursor=10:5c6b0b28:::7:head 2024-06-11T19:10:36.563 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:5c6b0b28:::7:head 2024-06-11T19:10:36.563 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:5c6b0b28:::7:head -> 7 2024-06-11T19:10:36.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=14 cursor=10:62a1935d:::14:head 2024-06-11T19:10:36.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:62a1935d:::14:head 2024-06-11T19:10:36.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:62a1935d:::14:head -> 14 2024-06-11T19:10:36.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=0 cursor=10:6cac518f:::0:head 2024-06-11T19:10:36.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:6cac518f:::0:head 2024-06-11T19:10:36.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:6cac518f:::0:head -> 0 2024-06-11T19:10:36.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=15 cursor=10:863748b0:::15:head 2024-06-11T19:10:36.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:863748b0:::15:head 2024-06-11T19:10:36.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:863748b0:::15:head -> 15 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=11 cursor=10:89d3ae78:::11:head 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:89d3ae78:::11:head 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:89d3ae78:::11:head -> 11 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=5 cursor=10:b29083e3:::5:head 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:b29083e3:::5:head 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:b29083e3:::5:head -> 5 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=8 cursor=10:bd63b0f1:::8:head 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:bd63b0f1:::8:head 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:bd63b0f1:::8:head -> 8 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=6 cursor=10:c4fdafeb:::6:head 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:c4fdafeb:::6:head 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:c4fdafeb:::6:head -> 6 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=3 cursor=10:cfc208b3:::3:head 2024-06-11T19:10:36.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:cfc208b3:::3:head 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:cfc208b3:::3:head -> 3 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=4 cursor=10:d83876eb:::4:head 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:d83876eb:::4:head 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:d83876eb:::4:head -> 4 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=12 cursor=10:de5d7c5f:::12:head 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:de5d7c5f:::12:head 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:de5d7c5f:::12:head -> 12 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=9 cursor=10:e960b815:::9:head 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:e960b815:::9:head 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:e960b815:::9:head -> 9 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : oid=2 cursor=10:f905c69b:::2:head 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:f905c69b:::2:head 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:f905c69b:::2:head -> 2 2024-06-11T19:10:36.605 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:f905c69b:::2:head 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:f905c69b:::2:head expected=10:f905c69b:::2:head 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:f905c69b:::2:head -> 2 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=2 expected=2 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:e960b815:::9:head 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:e960b815:::9:head expected=10:e960b815:::9:head 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:e960b815:::9:head -> 9 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=9 expected=9 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:de5d7c5f:::12:head 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:de5d7c5f:::12:head expected=10:de5d7c5f:::12:head 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:de5d7c5f:::12:head -> 12 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=12 expected=12 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:d83876eb:::4:head 2024-06-11T19:10:36.606 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:d83876eb:::4:head expected=10:d83876eb:::4:head 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:d83876eb:::4:head -> 4 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=4 expected=4 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:cfc208b3:::3:head 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:cfc208b3:::3:head expected=10:cfc208b3:::3:head 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:cfc208b3:::3:head -> 3 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=3 expected=3 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:c4fdafeb:::6:head 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:c4fdafeb:::6:head expected=10:c4fdafeb:::6:head 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:c4fdafeb:::6:head -> 6 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=6 expected=6 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:bd63b0f1:::8:head 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:bd63b0f1:::8:head expected=10:bd63b0f1:::8:head 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:bd63b0f1:::8:head -> 8 2024-06-11T19:10:36.607 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=8 expected=8 2024-06-11T19:10:36.608 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:b29083e3:::5:head 2024-06-11T19:10:36.608 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:b29083e3:::5:head expected=10:b29083e3:::5:head 2024-06-11T19:10:36.608 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:b29083e3:::5:head -> 5 2024-06-11T19:10:36.902 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=5 expec api_cmd_pp: Running main() from gmock_main.cc 2024-06-11T19:10:36.902 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-06-11T19:10:36.902 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-06-11T19:10:36.903 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-06-11T19:10:36.903 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-06-11T19:10:36.903 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (52 ms) 2024-06-11T19:10:36.903 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-06-11T19:10:36.903 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (63 ms) 2024-06-11T19:10:36.903 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-06-11T19:10:36.903 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2537 ms) 2024-06-11T19:10:36.903 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (2652 ms total) 2024-06-11T19:10:36.903 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: 2024-06-11T19:10:36.903 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-06-11T19:10:36.903 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (2653 ms total) 2024-06-11T19:10:36.903 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-06-11T19:10:36.906 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: Running main() from gmock_main.cc 2024-06-11T19:10:36.906 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-06-11T19:10:36.906 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [----------] Global test environment set-up. 2024-06-11T19:10:36.906 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-06-11T19:10:36.906 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-06-11T19:10:36.906 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2504 ms) 2024-06-11T19:10:36.906 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [----------] 1 test from neocls_handler_error (2504 ms total) 2024-06-11T19:10:36.907 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: 2024-06-11T19:10:36.907 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [----------] Global test environment tear-down 2024-06-11T19:10:36.907 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2504 ms total) 2024-06-11T19:10:36.907 INFO:tasks.workunit.client.0.smithi117.stdout: handler_error: [ PASSED ] 1 test. 2024-06-11T19:10:36.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-06-11T19:10:36.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-06-11T19:10:36.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-06-11T19:10:36.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-06-11T19:10:36.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-06-11T19:10:36.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-06-11T19:10:36.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-06-11T19:10:36.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (251 ms) 2024-06-11T19:10:36.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-06-11T19:10:36.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (6 ms) 2024-06-11T19:10:36.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-06-11T19:10:36.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (16 ms) 2024-06-11T19:10:36.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-06-11T19:10:36.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (28 ms) 2024-06-11T19:10:36.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (12 ms) 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (11 ms) 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (11 ms) 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (12 ms) 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5 ms) 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (10 ms) 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-06-11T19:10:36.911 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (14 ms) 2024-06-11T19:10:36.912 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-06-11T19:10:36.912 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (12 ms) 2024-06-11T19:10:36.912 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-06-11T19:10:36.912 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (32 ms) 2024-06-11T19:10:36.912 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-06-11T19:10:36.912 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (34 ms) 2024-06-11T19:10:36.912 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-06-11T19:10:36.912 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (13 ms) 2024-06-11T19:10:36.912 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-06-11T19:10:36.912 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (8 ms) 2024-06-11T19:10:36.912 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (475 ms total) 2024-06-11T19:10:36.912 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: 2024-06-11T19:10:36.913 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-06-11T19:10:36.913 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (2607 ms total) 2024-06-11T19:10:36.913 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-06-11T19:10:37.073 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: Running main() from gmock_main.cc 2024-06-11T19:10:37.074 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-06-11T19:10:37.074 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [----------] Global test environment set-up. 2024-06-11T19:10:37.074 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-06-11T19:10:37.074 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-06-11T19:10:37.074 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (77 ms) 2024-06-11T19:10:37.074 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-06-11T19:10:37.074 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (47 ms) 2024-06-11T19:10:37.074 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-06-11T19:10:37.074 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2516 ms) 2024-06-11T19:10:37.074 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-06-11T19:10:37.075 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.844594+0000 mon.a [WRN] Health check failed: 34 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-11T19:10:37.075 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.858019+0000 mon.a [INF] from='client.? 172.21.15.117:0/936665631' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi117-37207-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.075 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.858106+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi117-36785-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.075 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.858313+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-37069-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.075 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.858484+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi117-36868-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.075 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.858664+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi117-36569-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.075 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.858756+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi117-36480-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.075 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.858844+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-37168-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.075 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.858932+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi117-36498-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.075 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.859082+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi117-36879-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.076 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.859154+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi117-36584-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.076 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.859219+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi117-36854-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.076 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.859269+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi117-36891-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.416 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35 list_parallel: process_1_[37278]: starting. 2024-06-11T19:10:37.421 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[37278]: creating pool ceph_test_rados_list_parallel.smithi117-37244 2024-06-11T19:10:37.421 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[37278]: created object 0... 2024-06-11T19:10:37.421 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[37278]: created object 25... 2024-06-11T19:10:37.422 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[37278]: created object 49... 2024-06-11T19:10:37.422 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[37278]: finishing. 2024-06-11T19:10:37.422 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_1_[37278]: shutting down. 2024-06-11T19:10:37.422 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_2_[37281]: starting. 2024-06-11T19:10:37.422 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_2_[37281]: listing objects. 2024-06-11T19:10:37.422 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_2_[37281]: listed object 0... 2024-06-11T19:10:37.423 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_2_[37281]: listed object 25... 2024-06-11T19:10:37.423 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_2_[37281]: saw 50 objects 2024-06-11T19:10:37.423 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_2_[37281]: shutting down. 2024-06-11T19:10:37.423 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.423 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[38075]: starting. 2024-06-11T19:10:37.423 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[38075]: creating pool ceph_test_rados_list_parallel.smithi117-37244 2024-06-11T19:10:37.424 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[38075]: created object 0... 2024-06-11T19:10:37.424 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[38075]: created object 25... 2024-06-11T19:10:37.424 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[38075]: created object 49... 2024-06-11T19:10:37.424 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[38075]: finishing. 2024-06-11T19:10:37.424 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_3_[38075]: shutting down. 2024-06-11T19:10:37.424 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.424 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_4_[38076]: starting. 2024-06-11T19:10:37.425 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_4_[38076]: listing objects. 2024-06-11T19:10:37.425 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_4_[38076]: listed object 0... 2024-06-11T19:10:37.425 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_4_[38076]: listed object 25... 2024-06-11T19:10:37.425 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_4_[38076]: saw 43 objects 2024-06-11T19:10:37.425 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_4_[38076]: shutting down. 2024-06-11T19:10:37.425 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.425 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_5_[38077]: starting. 2024-06-11T19:10:37.426 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_5_[38077]: removed 25 objects... 2024-06-11T19:10:37.426 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_5_[38077]: removed half of the objects 2024-06-11T19:10:37.426 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_5_[38077]: removed 50 objects... 2024-06-11T19:10:37.426 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_5_[38077]: removed 50 objects 2024-06-11T19:10:37.426 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_5_[38077]: shutting down. 2024-06-11T19:10:37.426 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.426 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.426 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[38126]: starting. 2024-06-11T19:10:37.427 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[38126]: creating pool ceph_test_rados_list_parallel.smithi117-37244 2024-06-11T19:10:37.427 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[38126]: created object 0... 2024-06-11T19:10:37.427 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[38126]: created object 25... 2024-06-11T19:10:37.427 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[38126]: created object 49... 2024-06-11T19:10:37.427 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[38126]: finishing. 2024-06-11T19:10:37.427 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_6_[38126]: shutting down. 2024-06-11T19:10:37.428 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.428 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.428 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[38127]: starting. 2024-06-11T19:10:37.428 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[38127]: listing objects. 2024-06-11T19:10:37.428 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[38127]: listed object 0... 2024-06-11T19:10:37.428 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[38127]: listed object 25... 2024-06-11T19:10:37.429 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[38127]: listed object 50... 2024-06-11T19:10:37.429 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[38127]: saw 57 objects 2024-06-11T19:10:37.429 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_7_[38127]: shutting down. 2024-06-11T19:10:37.429 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.429 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.429 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_8_[38128]: starting. 2024-06-11T19:10:37.429 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_8_[38128]: added 25 objects... 2024-06-11T19:10:37.430 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_8_[38128]: added half of the objects 2024-06-11T19:10:37.430 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_8_[38128]: added 50 objects... 2024-06-11T19:10:37.430 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_8_[38128]: added 50 objects 2024-06-11T19:10:37.430 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_8_[38128]: shutting down. 2024-06-11T19:10:37.430 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.430 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.817 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.817 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[38468]: starting. 2024-06-11T19:10:37.818 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[38468]: creating pool ceph_test_rados_list_parallel.smithi117-37244 2024-06-11T19:10:37.818 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[38468]: created object 0... 2024-06-11T19:10:37.818 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[38468]: created object 25... 2024-06-11T19:10:37.818 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[38468]: created object 49... 2024-06-11T19:10:37.818 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[38468]: finishing. 2024-06-11T19:10:37.818 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_9_[38468]: shutting down. 2024-06-11T19:10:37.818 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.818 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.818 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.818 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_13_[38472]: starting. 2024-06-11T19:10:37.818 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_13_[38472]: removed 25 objects... 2024-06-11T19:10:37.818 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_13_[38472]: removed half of the objects 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_13_[38472]: removed 50 objects... 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_13_[38472]: removed 50 objects 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_13_[38472]: shutting down. 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_12_[38471]: starting. 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_12_[38471]: added 25 objects... 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_12_[38471]: added half of the objects 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_12_[38471]: added 50 objects... 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_12_[38471]: added 50 objects 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_12_[38471]: shutting down. 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.819 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_11_[38470]: starting. 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_11_[38470]: added 25 objects... 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_11_[38470]: added half of the objects 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_11_[38470]: added 50 objects... 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_11_[38470]: added 50 objects 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_11_[38470]: shutting down. 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[38469]: starting. 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[38469]: listing objects. 2024-06-11T19:10:37.820 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[38469]: listed object 0... 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[38469]: listed object 25... 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[38469]: listed object 50... 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[38469]: listed object 75... 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[38469]: saw 95 objects 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_10_[38469]: shutting down. 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[38553]: starting. 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[38553]: creating pool ceph_test_rados_list_parallel.smithi117-37244 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[38553]: created object 0... 2024-06-11T19:10:37.821 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[38553]: created object 25... 2024-06-11T19:10:37.822 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[38553]: created object 49... 2024-06-11T19:10:37.822 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[38553]: finishing. 2024-06-11T19:10:37.822 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_14_[38553]: shutting down. 2024-06-11T19:10:37.822 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.822 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.822 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.822 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:37.822 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[38554]: starting. 2024-06-11T19:10:37.822 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[38554]: listing objects. 2024-06-11T19:10:37.822 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[38554]: listed object 0... 2024-06-11T19:10:37.889 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[38554]: listed object 2.859323+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-37044-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.859383+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi117-36537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.859441+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi117-36810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.859509+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi117-36516-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.859547+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-36768-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.859585+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi117-36564-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:35.859623+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi117-37084-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:37.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:36.871374+0000 mon.a [INF] from='client.? 172.21.15.117:0/1675178955' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi117-37011-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi117-37011-1"}]': finished 2024-06-11T19:10:37.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:36.871471+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi117-37764-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi117-37764-1"}]': finished 2024-06-11T19:10:37.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:36.871573+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi117-37672-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi117-37672-1"}]': finished 2024-06-11T19:10:37.890 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:36.899781+0000 mon.b [INF] from='client.? 172.21.15.117:0/2492738716' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-36854-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-06-11T19:10:37.890 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:36.900386+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-36854-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-06-11T19:10:37.890 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.033757+0000 mon.h [INF] from='client.? 172.21.15.117:0/2375645083' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-06-11T19:10:37.890 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.033989+0000 client.admin [INF] onexx 2024-06-11T19:10:37.890 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.034195+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-06-11T19:10:37.890 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.048138+0000 mon.h [INF] from='client.? 172.21.15.117:0/1815703428' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi117-36516-16"}]: dispatch 2024-06-11T19:10:37.890 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.051152+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi117-36516-16"}]: dispatch 2024-06-11T19:10:37.890 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.064207+0000 mon.h [INF] from='client.? 172.21.15.117:0/1815703428' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi117-36516-16"}]: dispatch 2024-06-11T19:10:37.890 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.066588+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi117-36516-16"}]: dispatch 2024-06-11T19:10:37.890 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.070197+0000 mon.h [INF] from='client.? 172.21.15.117:0/1815703428' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi117-36516-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-11T19:10:37.890 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.070520+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi117-36516-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-11T19:10:37.890 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.100432+0000 mon.g [INF] from='client.? 172.21.15.117:0/1811073270' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi117-36879-7"}]: dispatch 2024-06-11T19:10:37.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.100882+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi117-36879-7"}]: dispatch 2024-06-11T19:10:37.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.106103+0000 mon.g [INF] from='client.? 172.21.15.117:0/1811073270' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi117-36879-7"}]: dispatch 2024-06-11T19:10:37.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.106584+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi117-36879-7"}]: dispatch 2024-06-11T19:10:37.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.107190+0000 mon.g [INF] from='client.? 172.21.15.117:0/1811073270' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi117-36879-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-11T19:10:37.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.107555+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi117-36879-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-11T19:10:37.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.117464+0000 mon.g [INF] from='client.? 172.21.15.117:0/3541246497' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi117-36891-7"}]: dispatch 2024-06-11T19:10:37.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.117812+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi117-36891-7"}]: dispatch 2024-06-11T19:10:37.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.119917+0000 mon.g [INF] from='client.? 172.21.15.117:0/3541246497' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi117-36891-7"}]: dispatch 2024-06-11T19:10:37.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.120205+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi117-36891-7"}]: dispatch 2024-06-11T19:10:37.891 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.120791+0000 mon.g [INF] from='client.? 172.21.15.117:0/3541246497' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi117-36891-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-11T19:10:38.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:37.121126+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix"5... 2024-06-11T19:10:38.917 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[38554]: listed object 50... 2024-06-11T19:10:38.917 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[38554]: listed object 75... 2024-06-11T19:10:38.917 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[38554]: listed object 100... 2024-06-11T19:10:38.917 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[38554]: listed object 125... 2024-06-11T19:10:38.917 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[38554]: saw 150 objects 2024-06-11T19:10:38.918 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_15_[38554]: shutting down. 2024-06-11T19:10:38.918 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:38.918 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:38.918 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:38.918 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:38.918 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_16_[38555]: starting. 2024-06-11T19:10:38.918 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_16_[38555]: added 25 objects... 2024-06-11T19:10:38.918 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_16_[38555]: added half of the objects 2024-06-11T19:10:38.918 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_16_[38555]: added 50 objects... 2024-06-11T19:10:38.918 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_16_[38555]: added 50 objects 2024-06-11T19:10:38.918 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: process_16_[38555]: shutting down. 2024-06-11T19:10:38.918 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:38.919 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:38.919 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:38.919 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:38.919 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******************************* 2024-06-11T19:10:38.919 INFO:tasks.workunit.client.0.smithi117.stdout: list_parallel: ******* SUCCESS ********** 2024-06-11T19:10:39.937 INFO:tasks.workunit.client.0.smithi117.stdout: cls: Running main() from gmock_main.cc 2024-06-11T19:10:39.938 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-06-11T19:10:39.938 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [----------] Global test environment set-up. 2024-06-11T19:10:39.938 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-06-11T19:10:39.938 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-06-11T19:10:39.938 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [ OK ] NeoRadosCls.DNE (2506 ms) 2024-06-11T19:10:39.938 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-06-11T19:10:39.938 INFO:tasks.workunit.client.0.smithi117.stdout: cls: unknown file: Failure 2024-06-11T19:10:39.938 INFO:tasks.workunit.client.0.smithi117.stdout: cls: C++ exception with description "(95) Operation not supported [osd:95]" thrown in the test body. 2024-06-11T19:10:39.938 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [ FAILED ] NeoRadosCls.RemoteReads (3029 ms) 2024-06-11T19:10:39.938 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [----------] 2 tests from NeoRadosCls (5535 ms total) 2024-06-11T19:10:39.938 INFO:tasks.workunit.client.0.smithi117.stdout: cls: 2024-06-11T19:10:39.938 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [----------] Global test environment tear-down 2024-06-11T19:10:39.939 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [==========] 2 tests from 1 test suite ran. (5535 ms total) 2024-06-11T19:10:39.939 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [ PASSED ] 1 test. 2024-06-11T19:10:39.939 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [ FAILED ] 1 test, listed below: 2024-06-11T19:10:39.939 INFO:tasks.workunit.client.0.smithi117.stdout: cls: [ FAILED ] NeoRadosCls.RemoteReads 2024-06-11T19:10:39.939 INFO:tasks.workunit.client.0.smithi117.stdout: cls: 2024-06-11T19:10:39.939 INFO:tasks.workunit.client.0.smithi117.stdout: cls: 1 FAILED TEST 2024-06-11T19:10:39.955 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[37295]: starting. 2024-06-11T19:10:39.955 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[37295]: creating pool ceph_test_rados_open_pools_parallel.smithi117-37269 2024-06-11T19:10:39.956 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[37295]: created object 0... 2024-06-11T19:10:39.956 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[37295]: created object 25... 2024-06-11T19:10:39.956 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[37295]: created object 49... 2024-06-11T19:10:39.956 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[37295]: finishing. 2024-06-11T19:10:39.956 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_1_[37295]: shutting down. 2024-06-11T19:10:39.956 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_2_[37296]: starting. 2024-06-11T19:10:39.956 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_2_[37296]: rados_pool_create. 2024-06-11T19:10:39.956 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_2_[37296]: rados_ioctx_create. 2024-06-11T19:10:39.956 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_2_[37296]: shutting down. 2024-06-11T19:10:39.956 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: ******************************* 2024-06-11T19:10:39.956 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[38155]: starting. 2024-06-11T19:10:39.957 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[38155]: creating pool ceph_test_rados_open_pools_parallel.smithi117-37269 2024-06-11T19:10:39.957 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[38155]: created object 0... 2024-06-11T19:10:39.957 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[38155]: created object 25... 2024-06-11T19:10:39.957 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[38155]: created object 49... 2024-06-11T19:10:39.957 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[38155]: finishing. 2024-06-11T19:10:39.957 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_3_[38155]: shutting down. 2024-06-11T19:10:39.957 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: ******************************* 2024-06-11T19:10:39.957 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_4_[38156]: starting. 2024-06-11T19:10:39.957 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_4_[38156]: rados_pool_create. 2024-06-11T19:10:39.957 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_4_[38156]: rados_ioctx_create. 2024-06-11T19:10:39.957 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: process_4_[38156]: shutting down. 2024-06-11T19:10:39.957 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: ******************************* 2024-06-11T19:10:39.958 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: ******************************* 2024-06-11T19:10:39.958 INFO:tasks.workunit.client.0.smithi117.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-06-11T19:10:39.960 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[37560]: starting. 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[37560]: creating pool ceph_test_rados_delete_pools_parallel.smithi117-37397 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[37560]: created object 0... 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[37560]: created object 25... 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[37560]: created object 49... 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[37560]: finishing. 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_1_[37560]: shutting down. 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_2_[37562]: starting. 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_2_[37562]: deleting pool ceph_test_rados_delete_pools_parallel.smithi117-37397 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_2_[37562]: shutting down. 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: ******************************* 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[38167]: starting. 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[38167]: creating pool ceph_test_rados_delete_pools_parallel.smithi117-37397 2024-06-11T19:10:39.961 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[38167]: created object 0... 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[38167]: created object 25... 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[38167]: created object 49... 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[38167]: finishing. 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_3_[38167]: shutting down. 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: ******************************* 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_5_[38170]: starting. 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_5_[38170]: listing objects. 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_5_[38170]: listed object 0... 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_5_[38170]: listed object 25... 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_5_[38170]: saw 50 objects 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_5_[38170]: shutting down. 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: ******************************* 2024-06-11T19:10:39.962 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_4_[38169]: starting. 2024-06-11T19:10:39.963 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_4_[38169]: deleting pool ceph_test_rados_delete_pools_parallel.smithi117-37397 2024-06-11T19:10:39.963 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: process_4_[38169]: shutting down. 2024-06-11T19:10:39.963 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: ******************************* 2024-06-11T19:10:39.963 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: ******************************* 2024-06-11T19:10:39.963 INFO:tasks.workunit.client.0.smithi117.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-06-11T19:10:40.886 INFO:tasks.workunit.client.0.smithi117.stdout:: "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi117-36891-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-11T19:10:40.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.884419+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-06-11T19:10:40.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.892118+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi117-37672-1"}]': finished 2024-06-11T19:10:40.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.892205+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-36768-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:40.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.892316+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-37168-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:40.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.892381+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi117-36498-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:40.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.892446+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi117-36480-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:40.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.892519+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi117-36537-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-06-11T19:10:40.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.897942+0000 mon.g [INF] from='client.? 172.21.15.117:0/2875330109' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi117-36537-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi117-36537-24"}]: dispatch 2024-06-11T19:10:40.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.898249+0000 mon.h [INF] from='client.? 172.21.15.117:0/676065082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi117-37672-1"}]: dispatch 2024-06-11T19:10:40.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.914280+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi117-36537-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi117-36537-24"}]: dispatch 2024-06-11T19:10:40.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.914596+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi117-37672-1"}]: dispatch 2024-06-11T19:10:40.887 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.915367+0000 mon.b [INF] from='client.? 172.21.15.117:0/2492738716' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-36854-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-06-11T19:10:40.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.916573+0000 mon.c [INF] from='client.? 172.21.15.117:0/2338303920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi117-37764-1"}]: dispatch 2024-06-11T19:10:40.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.922752+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi117-37764-1"}]: dispatch 2024-06-11T19:10:40.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.923014+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-36854-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-06-11T19:10:40.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.924141+0000 mon.a [INF] from='client.? 172.21.15.117:0/1675178955' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi117-37011-1"}]: dispatch 2024-06-11T19:10:40.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.966587+0000 mon.g [INF] from='client.? 172.21.15.117:0/3716779790' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi117-36569-10"}]: dispatch 2024-06-11T19:10:40.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.967245+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi117-36569-10"}]: dispatch 2024-06-11T19:10:40.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.968653+0000 mon.g [INF] from='client.? 172.21.15.117:0/3716779790' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi117-36569-10"}]: dispatch 2024-06-11T19:10:40.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.969302+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi117-36569-10"}]: dispatch 2024-06-11T19:10:40.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.969952+0000 mon.g [INF] from='client.? 172.21.15.117:0/3716779790' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi117-36569-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-11T19:10:40.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.970675+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi117-36569-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-11T19:10:40.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.972402+0000 mon.g [INF] from='client.? 172.21.15.117:0/3128758985' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi117-36584-10"}]: dispatch 2024-06-11T19:10:40.888 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.972764+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi117-36584-10"}]: dispatch 2024-06-11T19:10:40.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.978992+0000 mon.g [INF] from='client.? 172.21.15.117:0/3128758985' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi117-36584-10"}]: dispatch 2024-06-11T19:10:40.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.979605+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi117-36584-10"}]: dispatch 2024-06-11T19:10:40.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.980410+0000 mon.g [INF] from='client.? 172.21.15.117:0/3128758985' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi117-36584-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-11T19:10:40.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:38.980815+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi117-36584-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-11T19:10:40.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.487196+0000 mon.i [INF] from='client.? 172.21.15.117:0/1759868744' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.0"}]: dispatch 2024-06-11T19:10:40.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.487973+0000 mon.i [INF] from='client.? 172.21.15.117:0/1759868744' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.1"}]: dispatch 2024-06-11T19:10:40.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.488614+0000 mon.i [INF] from='client.? 172.21.15.117:0/1759868744' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.2"}]: dispatch 2024-06-11T19:10:40.889 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.489217+0000 mon.i [INF] from='client.? 172.21.15.117:0/1759868744' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.3"}]: dispatch 2024-06-11T19:10:40.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.489948+0000 mon.i [INF] from='client.? 172.21.15.117:0/1759868744' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.4"}]: dispatch 2024-06-11T19:10:40.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.490568+0000 mon.i [INF] from='client.? 172.21.15.117:0/1759868744' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.5"}]: dispatch 2024-06-11T19:10:40.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.491175+0000 mon.i [INF] from='client.? 172.21.15.117:0/1759868744' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.6"}]: dispatch 2024-06-11T19:10:40.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.492627+0000 mon.i [INF] from='client.? 172.21.15.117:0/1759868744' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.7"}]: dispatch 2024-06-11T19:10:40.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.493326+0000 mon.i [INF] from='client.? 172.21.15.117:0/1759868744' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.8"}]: dispatch 2024-06-11T19:10:40.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.493996+0000 mon.i [INF] from='client.? 172.21.15.117:0/1759868744' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "13.9"}]: dispatch 2024-06-11T19:10:40.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.905177+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi117-36516-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi117-36516-16"}]': finished 2024-06-11T19:10:40.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.905333+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi117-36879-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi117-36879-7"}]': finished 2024-06-11T19:10:40.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.905427+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi117-36891-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi117-36891-7"}]': finished 2024-06-11T19:10:40.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.905488+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi117-37672-1"}]': finished 2024-06-11T19:10:40.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.905550+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi117-37764-1"}]': finished 2024-06-11T19:10:40.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.905589+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-36854-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-11T19:10:40.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.905628+0000 mon.a [INF] from='client.? 172.21.15.117:0/1675178955' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi117-37011-1"}]': finished 2024-06-11T19:10:40.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.905658+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi117-36569-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-06-11T19:10:40.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.905702+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi117-36584-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-06-11T19:10:40.910 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.910205+0000 mon.g [INF] from='client.? 172.21.15.117:0/3716779790' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi117-36569-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi117-36569-10"}]: dispatch 2024-06-11T19:10:40.953 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T1 cmd: Running main() from gmock_main.cc 2024-06-11T19:10:40.953 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-06-11T19:10:40.954 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [----------] Global test environment set-up. 2024-06-11T19:10:40.954 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-06-11T19:10:40.954 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-06-11T19:10:40.954 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1448 ms) 2024-06-11T19:10:40.954 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-06-11T19:10:40.955 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (2010 ms) 2024-06-11T19:10:40.955 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-06-11T19:10:40.955 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3046 ms) 2024-06-11T19:10:40.955 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [----------] 3 tests from NeoRadosCmd (6504 ms total) 2024-06-11T19:10:40.955 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: 2024-06-11T19:10:40.955 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [----------] Global test environment tear-down 2024-06-11T19:10:40.955 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [==========] 3 tests from 1 test suite ran. (6504 ms total) 2024-06-11T19:10:40.956 INFO:tasks.workunit.client.0.smithi117.stdout: cmd: [ PASSED ] 3 tests. 2024-06-11T19:10:42.060 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Running main() from gmock_main.cc 2024-06-11T19:10:42.060 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [==========] Running 40 tests from 2 test suites. 2024-06-11T19:10:42.060 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [----------] Global test environment set-up. 2024-06-11T19:10:42.060 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP 2024-06-11T19:10:42.060 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: seed 36537 2024-06-11T19:10:42.060 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-06-11T19:10:42.060 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-06-11T19:10:42.060 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-06-11T19:10:42.060 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (247 ms) 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (10 ms) 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (14 ms) 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (598 ms) 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (3 ms) 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2 ms) 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (5 ms) 2024-06-11T19:10:42.061 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-06-11T19:10:42.062 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (3 ms) 2024-06-11T19:10:42.062 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-06-11T19:10:42.062 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (6 ms) 2024-06-11T19:10:42.062 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-06-11T19:10:42.062 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (4 ms) 2024-06-11T19:10:42.062 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-06-11T19:10:42.062 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (4 ms) 2024-06-11T19:10:42.062 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-06-11T19:10:42.062 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (9 ms) 2024-06-11T19:10:42.062 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-06-11T19:10:42.062 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (10 ms) 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (10 ms) 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (10 ms) 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (23 ms) 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (12 ms) 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CrcZeroWrite 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.CrcZeroWrite (3 ms) 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (8 ms) 2024-06-11T19:10:42.063 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-06-11T19:10:42.064 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (2 ms) 2024-06-11T19:10:42.064 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-06-11T19:10:42.064 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (6 ms) 2024-06-11T19:10:42.064 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP (989 ms total) 2024-06-11T19:10:42.064 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: 2024-06-11T19:10:42.064 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP 2024-06-11T19:10:42.064 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-06-11T19:10:42.064 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1122 ms) 2024-06-11T19:10:42.064 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-06-11T19:10:42.064 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (11 ms) 2024-06-11T19:10:42.064 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-06-11T19:10:42.064 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (5 ms) 2024-06-11T19:10:42.952 INFO:tasks.workunit.client.0.smithi117.stdout: list: Running main() from gmock_main.cc 2024-06-11T19:10:42.952 INFO:tasks.workunit.client.0.smithi117.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-06-11T19:10:42.953 INFO:tasks.workunit.client.0.smithi117.stdout: list: [----------] Global test environment set-up. 2024-06-11T19:10:42.953 INFO:tasks.workunit.client.0.smithi117.stdout: list: [----------] 3 tests from NeoradosList 2024-06-11T19:10:42.953 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-06-11T19:10:42.953 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ OK ] NeoradosList.ListObjects (2430 ms) 2024-06-11T19:10:42.953 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-06-11T19:10:42.954 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3046 ms) 2024-06-11T19:10:42.954 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-06-11T19:10:42.954 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3007 ms) 2024-06-11T19:10:42.954 INFO:tasks.workunit.client.0.smithi117.stdout: list: [----------] 3 tests from NeoradosList (8483 ms total) 2024-06-11T19:10:42.954 INFO:tasks.workunit.client.0.smithi117.stdout: list: 2024-06-11T19:10:42.954 INFO:tasks.workunit.client.0.smithi117.stdout: list: [----------] Global test environment tear-down 2024-06-11T19:10:42.955 INFO:tasks.workunit.client.0.smithi117.stdout: list: [==========] 3 tests from 1 test suite ran. (8483 ms total) 2024-06-11T19:10:42.955 INFO:tasks.workunit.client.0.smithi117.stdout: list: [ PASSED ] 3 tests. 2024-06-11T19:10:43.550 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-06-11T19:10:43.550 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [----------] Global test environment set-up. 2024-06-11T19:10:43.550 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [----------] 4 tests from LibRadosService 2024-06-11T19:10:43.550 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-06-11T19:10:43.550 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5035 ms) 2024-06-11T19:10:43.550 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-06-11T19:10:43.550 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ OK ] LibRadosService.RegisterLate (20 ms) 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: cluster: 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: id: 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: health: HEALTH_WARN 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 11 pool(s) do not have an application enabled 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: services: 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: mon: 9 daemons, quorum a,f,b,g,c,h,i,e (age 44s), out of quorum: d 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: mgr: x(active, since 80s) 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: osd: 6 osds: 6 up (since 74s), 6 in (since 82s) 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:10:43.551 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: data: 2024-06-11T19:10:43.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: pools: 22 pools, 577 pgs 2024-06-11T19:10:43.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: objects: 2 objects, 577 KiB 2024-06-11T19:10:43.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: usage: 160 MiB used, 540 GiB / 540 GiB avail 2024-06-11T19:10:43.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: pgs: 75.737% pgs unknown 2024-06-11T19:10:43.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 22.704% pgs not active 2024-06-11T19:10:43.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 437 unknown 2024-06-11T19:10:43.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 131 creating+peering 2024-06-11T19:10:43.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 9 active+clean 2024-06-11T19:10:43.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:10:43.552 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:10:43.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: cluster: 2024-06-11T19:10:43.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: id: 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:10:43.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: health: HEALTH_WARN 2024-06-11T19:10:43.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 15 pool(s) do not have an application enabled 2024-06-11T19:10:43.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:10:43.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: services: 2024-06-11T19:10:43.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: mon: 9 daemons, quorum a,f,b,g,c,h,i,e (age 46s), out of quorum: d 2024-06-11T19:10:43.553 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: mgr: x(active, since 82s) 2024-06-11T19:10:43.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: osd: 6 osds: 6 up (since 76s), 6 in (since 84s) 2024-06-11T19:10:43.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:10:43.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: data: 2024-06-11T19:10:43.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: pools: 28 pools, 841 pgs 2024-06-11T19:10:43.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: objects: 2 objects, 577 KiB 2024-06-11T19:10:43.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: usage: 160 MiB used, 540 GiB / 540 GiB avail 2024-06-11T19:10:43.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: pgs: 90.131% pgs unknown 2024-06-11T19:10:43.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 8.799% pgs not active 2024-06-11T19:10:43.554 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 758 unknown 2024-06-11T19:10:43.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 74 creating+peering 2024-06-11T19:10:43.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 9 active+clean 2024-06-11T19:10:43.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:10:43.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:10:43.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: cluster: 2024-06-11T19:10:43.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: id: 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:10:43.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: health: HEALTH_WARN 2024-06-11T19:10:43.555 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 1 cache pools are missing hit_sets 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 16 pool(s) do not have an application enabled 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: services: 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: mon: 9 daemons, quorum a,f,b,g,c,h,i,e (age 48s), out of quorum: d 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: mgr: x(active, since 84s) 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: osd: 6 osds: 6 up (since 78s), 6 in (since 86s) 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: data: 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: pools: 28 pools, 745 pgs 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: objects: 60 objects, 121 MiB 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: usage: 180 MiB used, 540 GiB / 540 GiB avail 2024-06-11T19:10:43.556 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: pgs: 42.953% pgs unknown 2024-06-11T19:10:43.916 INFO:tasks.workunit.client.0.smithi117.stdout: 9:10:39.910355+0000 mon.g [INF] from='client.? 172.21.15.117:0/3128758985' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi117-36584-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi117-36584-10"}]: dispatch 2024-06-11T19:10:43.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.916945+0000 mon.c [INF] from='client.? 172.21.15.117:0/2338303920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi117-37764-1"}]: dispatch 2024-06-11T19:10:43.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.921095+0000 mon.b [INF] from='client.? 172.21.15.117:0/2492738716' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi117-36854-4", "tierpool": "test-rados-api-smithi117-36854-4-cache"}]: dispatch 2024-06-11T19:10:43.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.922864+0000 mon.a [INF] from='client.? 172.21.15.117:0/1675178955' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi117-37011-1"}]: dispatch 2024-06-11T19:10:43.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.924844+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi117-36569-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi117-36569-10"}]: dispatch 2024-06-11T19:10:43.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.925189+0000 mon.b [INF] from='client.? 172.21.15.117:0/1009798656' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi117-36498-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-06-11T19:10:43.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.925382+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi117-37764-1"}]: dispatch 2024-06-11T19:10:43.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.926215+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi117-36584-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi117-36584-10"}]: dispatch 2024-06-11T19:10:43.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.926877+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi117-36854-4", "tierpool": "test-rados-api-smithi117-36854-4-cache"}]: dispatch 2024-06-11T19:10:43.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:39.927315+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi117-36498-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-06-11T19:10:43.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:40.039640+0000 mon.a [INF] from='client.? 172.21.15.117:0/1429340674' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi117-37672-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-11T19:10:43.918 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:40.887892+0000 mon.h [INF] from='client.? 172.21.15.117:0/2375645083' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-06-11T19:10:43.918 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:40.888154+0000 client.admin [INF] threexx 2024-06-11T19:10:43.918 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: got: 2024-06-11T19:10:40.888426+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-06-11T19:10:43.918 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7015 ms) 2024-06-11T19:10:43.918 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (9655 ms total) 2024-06-11T19:10:43.918 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: 2024-06-11T19:10:43.918 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [----------] Global test environment tear-down 2024-06-11T19:10:43.918 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (9655 ms total) 2024-06-11T19:10:43.918 INFO:tasks.workunit.client.0.smithi117.stdout: api_cmd: [ PASSED ] 4 tests. 2024-06-11T19:10:43.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: Running main() from gmock_main.cc 2024-06-11T19:10:43.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-06-11T19:10:43.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [----------] Global test environment set-up. 2024-06-11T19:10:43.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-06-11T19:10:43.962 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (267 ms) 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: no timeout :/ 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: no timeout :/ 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: no timeout :/ 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: no timeout :/ 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: no timeout :/ 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (45 ms) 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (11 ms) 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-06-11T19:10:43.963 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.Checksum (5 ms) 2024-06-11T19:10:43.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-06-11T19:10:43.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (5 ms) 2024-06-11T19:10:43.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-06-11T19:10:43.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (10 ms) 2024-06-11T19:10:43.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-06-11T19:10:43.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (8 ms) 2024-06-11T19:10:43.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-06-11T19:10:43.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (5 ms) 2024-06-11T19:10:43.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-06-11T19:10:43.964 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.TruncTest (8 ms) 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (9 ms) 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (11 ms) 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.RmXattr (32 ms) 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIo.XattrIter (15 ms) 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [----------] 14 tests from LibRadosIo (433 ms total) 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1100 ms) 2024-06-11T19:10:43.965 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-06-11T19:10:43.966 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (24 ms) 2024-06-11T19:10:43.966 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-06-11T19:10:43.966 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (19 ms) 2024-06-11T19:10:43.966 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-06-11T19:10:43.966 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (15 ms) 2024-06-11T19:10:43.966 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-06-11T19:10:43.966 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (11 ms) 2024-06-11T19:10:43.966 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-06-11T19:10:43.966 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (7 ms) 2024-06-11T19:10:43.966 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-06-11T19:10:43.967 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (7 ms) 2024-06-11T19:10:43.967 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-06-11T19:10:43.967 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (6 ms) 2024-06-11T19:10:43.967 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-06-11T19:10:43.967 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (15 ms) 2024-06-11T19:10:43.967 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-06-11T19:10:43.967 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (5 ms) 2024-06-11T19:10:43.967 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1209 ms total) 2024-06-11T19:10:43.967 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: 2024-06-11T19:10:43.968 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: watch_notify: Running main() from gmock_main.cc 2024-06-11T19:10:43.968 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-06-11T19:10:43.968 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [----------] Global test environment set-up. 2024-06-11T19:10:43.968 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-06-11T19:10:43.968 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-06-11T19:10:43.968 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: handle_notify cookie 94160274022880 notify_id 85899345920 notifier_gid 4649 2024-06-11T19:10:43.968 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (2376 ms) 2024-06-11T19:10:43.968 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-06-11T19:10:43.968 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: Trying... 2024-06-11T19:10:43.968 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: handle_notify cookie 94160285995536 notify_id 98784247810 notifier_gid 4978 2024-06-11T19:10:43.968 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: Waiting for 3.000000000s 2024-06-11T19:10:43.969 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: Timed out. 2024-06-11T19:10:43.969 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: Flushing... 2024-06-11T19:10:43.969 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: Flushed... 2024-06-11T19:10:43.969 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7076 ms) 2024-06-11T19:10:43.969 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (9452 ms total) 2024-06-11T19:10:43.969 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: 2024-06-11T19:10:43.969 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [----------] Global test environment tear-down 2024-06-11T19:10:43.969 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (9452 ms total) 2024-06-11T19:10:43.969 INFO:tasks.workunit.client.0.smithi117.stdout: watch_notify: [ PASSED ] 2 tests. 2024-06-11T19:10:43.969 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: seed 36891 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (251 ms) 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (10 ms) 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (9 ms) 2024-06-11T19:10:43.970 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (17 ms) 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (287 ms total) 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1098 ms) 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (6 ms) 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (20 ms) 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1124 ms total) 2024-06-11T19:10:43.971 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: 2024-06-11T19:10:43.972 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-06-11T19:10:43.972 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (9767 ms total) 2024-06-11T19:10:43.972 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-06-11T19:10:43.978 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: Running main() from gmock_main.cc 2024-06-11T19:10:43.978 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-06-11T19:10:43.978 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [----------] Global test environment set-up. 2024-06-11T19:10:43.978 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStat.Stat (252 ms) 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (12 ms) 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStat.StatNS (20 ms) 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (6 ms) 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [----------] 5 tests from LibRadosStat (291 ms total) 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: 2024-06-11T19:10:43.979 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1084 ms) 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (18 ms) 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (4 ms) 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1106 ms total) 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [----------] Global test environment tear-down 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (9773 ms total) 2024-06-11T19:10:43.980 INFO:tasks.workunit.client.0.smithi117.stdout: api_stat: [ PASSED ] 9 tests. 2024-06-11T19:10:43.988 INFO:tasks.workunit.client.0.smithi117.stdout: [----------] Global test environment tear-down 2024-06-11T19:10:43.988 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9897 ms total) 2024-06-11T19:10:43.988 INFO:tasks.workunit.client.0.smithi117.stdout: api_io: [ PASSED ] 24 tests. 2024-06-11T19:10:45.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (3 ms) 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (3 ms) 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (4 ms) 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (2 ms) 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (6 ms) 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-06-11T19:10:45.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (7 ms) 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (10 ms) 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (10 ms) 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (30 ms) 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CrcZeroWrite 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-4216-g37bbe1d8/rpm/el9/BUILD/ceph-19.0.0-4216-g37bbe1d8/src/test/librados/io_cxx.cc:888: Failure 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Expected equality of these values: 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: 0 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: ioctx.write("foo", bl, 0, sizeof(buf)) 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Which is: -95 2024-06-11T19:10:45.825 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ FAILED ] LibRadosIoECPP.CrcZeroWrite (856 ms) 2024-06-11T19:10:45.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-06-11T19:10:45.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (52 ms) 2024-06-11T19:10:45.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-06-11T19:10:45.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-4216-g37bbe1d8/rpm/el9/BUILD/ceph-19.0.0-4216-g37bbe1d8/src/test/librados/testcase_cxx.cc:392: Failure 2024-06-11T19:10:45.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Value of: req 2024-06-11T19:10:45.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Actual: false 2024-06-11T19:10:45.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Expected: true 2024-06-11T19:10:45.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ FAILED ] LibRadosIoECPP.CmpExtPP (0 ms) 2024-06-11T19:10:45.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-06-11T19:10:45.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-4216-g37bbe1d8/rpm/el9/BUILD/ceph-19.0.0-4216-g37bbe1d8/src/test/librados/testcase_cxx.cc:392: Failure 2024-06-11T19:10:45.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Value of: req 2024-06-11T19:10:45.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Actual: false 2024-06-11T19:10:45.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Expected: true 2024-06-11T19:10:45.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ FAILED ] LibRadosIoECPP.CmpExtDNEPP (0 ms) 2024-06-11T19:10:45.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-06-11T19:10:45.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-4216-g37bbe1d8/rpm/el9/BUILD/ceph-19.0.0-4216-g37bbe1d8/src/test/librados/testcase_cxx.cc:392: Failure 2024-06-11T19:10:45.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Value of: req 2024-06-11T19:10:45.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Actual: false 2024-06-11T19:10:45.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: Expected: true 2024-06-11T19:10:45.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ FAILED ] LibRadosIoECPP.CmpExtMismatchPP (0 ms) 2024-06-11T19:10:45.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP (2126 ms total) 2024-06-11T19:10:45.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: 2024-06-11T19:10:45.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [----------] Global test environment tear-down 2024-06-11T19:10:45.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [==========] 40 tests from 2 test suites ran. (11744 ms total) 2024-06-11T19:10:45.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ PASSED ] 36 tests. 2024-06-11T19:10:45.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ FAILED ] 4 tests, listed below: 2024-06-11T19:10:45.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ FAILED ] LibRadosIoECPP.CrcZeroWrite 2024-06-11T19:10:45.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ FAILED ] LibRadosIoECPP.CmpExtPP 2024-06-11T19:10:45.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ FAILED ] LibRadosIoECPP.CmpExtDNEPP 2024-06-11T19:10:45.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: [ FAILED ] LibRadosIoECPP.CmpExtMismatchPP 2024-06-11T19:10:45.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: 2024-06-11T19:10:45.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_io_pp: 4 FAILED TESTS 2024-06-11T19:10:46.796 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: Running main() from gmock_main.cc 2024-06-11T19:10:46.796 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-06-11T19:10:46.796 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [----------] Global test environment set-up. 2024-06-11T19:10:46.797 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-06-11T19:10:46.797 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-06-11T19:10:46.797 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5452 ms) 2024-06-11T19:10:46.797 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-06-11T19:10:46.797 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6882 ms) 2024-06-11T19:10:46.797 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12334 ms total) 2024-06-11T19:10:46.797 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: 2024-06-11T19:10:46.797 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [----------] Global test environment tear-down 2024-06-11T19:10:46.797 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12335 ms total) 2024-06-11T19:10:46.797 INFO:tasks.workunit.client.0.smithi117.stdout: ec_io: [ PASSED ] 2 tests. 2024-06-11T19:10:47.819 INFO:tasks.workunit.client.0.smithi117.stdout: pool: Running main() from gmock_main.cc 2024-06-11T19:10:47.819 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-06-11T19:10:47.819 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [----------] Global test environment set-up. 2024-06-11T19:10:47.819 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-06-11T19:10:47.819 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ OK ] NeoRadosPools.PoolList (1398 ms) 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2032 ms) 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2023 ms) 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4013 ms) 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1854 ms) 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2007 ms) 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [----------] 6 tests from NeoRadosPools (13328 ms total) 2024-06-11T19:10:47.820 INFO:tasks.workunit.client.0.smithi117.stdout: pool: 2024-06-11T19:10:47.821 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [----------] Global test environment tear-down 2024-06-11T19:10:47.821 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [==========] 6 tests from 1 test suite ran. (13328 ms total) 2024-06-11T19:10:47.821 INFO:tasks.workunit.client.0.smithi117.stdout: pool: [ PASSED ] 6 tests. 2024-06-11T19:10:47.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-06-11T19:10:47.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-06-11T19:10:47.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-06-11T19:10:47.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-06-11T19:10:47.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: seed 36584 2024-06-11T19:10:47.821 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (220 ms) 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (23 ms) 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1039 ms) 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1006 ms) 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (9 ms) 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (12 ms) 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-06-11T19:10:47.822 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (6 ms) 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2318 ms total) 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1106 ms) 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (14 ms) 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1072 ms) 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-06-11T19:10:47.823 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1013 ms) 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (14 ms) 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (8 ms) 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (10 ms) 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (8 ms) 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3245 ms total) 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13713 ms total) 2024-06-11T19:10:47.824 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-06-11T19:10:47.843 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: Running main() from gmock_main.cc 2024-06-11T19:10:47.843 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-06-11T19:10:47.843 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [----------] Global test environment set-up. 2024-06-11T19:10:47.843 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-06-11T19:10:47.843 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-06-11T19:10:47.843 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (253 ms) 2024-06-11T19:10:47.843 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-06-11T19:10:47.843 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.LockShared (14 ms) 2024-06-11T19:10:47.843 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-06-11T19:10:47.843 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1021 ms) 2024-06-11T19:10:47.843 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1003 ms) 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (14 ms) 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.Unlock (10 ms) 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (5 ms) 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (4 ms) 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [----------] 8 tests from LibRadosLock (2324 ms total) 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-06-11T19:10:47.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1097 ms) 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (18 ms) 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1065 ms) 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1010 ms) 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (11 ms) 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (17 ms) 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (11 ms) 2024-06-11T19:10:47.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-06-11T19:10:47.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (7 ms) 2024-06-11T19:10:47.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3236 ms total) 2024-06-11T19:10:47.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: 2024-06-11T19:10:47.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [----------] Global test environment tear-down 2024-06-11T19:10:47.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13742 ms total) 2024-06-11T19:10:47.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_lock: [ PASSED ] 16 tests. 2024-06-11T19:10:50.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-06-11T19:10:50.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-06-11T19:10:50.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-06-11T19:10:50.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-06-11T19:10:50.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-06-11T19:10:50.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify 2024-06-11T19:10:50.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1091 ms) 2024-06-11T19:10:50.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-06-11T19:10:50.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (32 ms) 2024-06-11T19:10:50.135 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1123 ms total) 2024-06-11T19:10:50.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: 2024-06-11T19:10:50.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-06-11T19:10:50.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-06-11T19:10:50.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify 2024-06-11T19:10:50.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (123 ms) 2024-06-11T19:10:50.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-06-11T19:10:50.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify 2024-06-11T19:10:50.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3810 ms) 2024-06-11T19:10:50.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-06-11T19:10:50.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (20 ms) 2024-06-11T19:10:50.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-06-11T19:10:50.136 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (9 ms) 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94707273009104 notify_id 133143986180 notifier_gid 5070 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (7 ms) 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94707273009104 notify_id 133143986181 notifier_gid 5070 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (6 ms) 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94707273009104 notify_id 133143986182 notifier_gid 5070 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (5 ms) 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94707272996928 notify_id 133143986176 notifier_gid 5070 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (6 ms) 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-06-11T19:10:50.137 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94707273021584 notify_id 133143986177 notifier_gid 5070 2024-06-11T19:10:50.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (208 ms) 2024-06-11T19:10:50.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-06-11T19:10:50.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94707273021584 notify_id 133143986183 notifier_gid 5070 2024-06-11T19:10:50.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (5 ms) 2024-06-11T19:10:50.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-06-11T19:10:50.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: trying... 2024-06-11T19:10:50.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94707273036160 notify_id 133143986184 notifier_gid 5070 2024-06-11T19:10:50.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: timed out 2024-06-11T19:10:50.138 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: flushing 2024-06-11T19:10:54.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_n api_watch_notify: Running main() from gmock_main.cc 2024-06-11T19:10:54.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-06-11T19:10:54.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-06-11T19:10:54.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify_test_cb 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (277 ms) 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94312138374720 err -107 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1016 ms) 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94312138386256 err -107 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1008 ms) 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4642 notify_id 94489280512 cookie 94312138392496 2024-06-11T19:10:54.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (9 ms) 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4642 notify_id 94489280513 cookie 94312138407952 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (9 ms) 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4642 notify_id 94489280513 cookie 94312138420784 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (5 ms) 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4642 notify_id 94489280513 cookie 94312138420784 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4642 notify_id 94489280513 cookie 94312138425824 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (7 ms) 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4642 notify_id 94489280514 cookie 94312138425824 2024-06-11T19:10:54.850 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4642 notify_id 98784247811 cookie 94312138425824 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3004 ms) 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94312138420176 err -107 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_cb from 4642 notify_id 128849018881 cookie 94312138420176 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5009 ms) 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94312138411232 err -107 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1004 ms) 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11348 ms total) 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-06-11T19:10:54.851 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-06-11T19:10:54.852 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: watch_notify_test_cb 2024-06-11T19:10:54.852 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1073 ms) 2024-06-11T19:10:54.852 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1073 ms total) 2024-06-11T19:10:54.852 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: 2024-06-11T19:10:54.852 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-06-11T19:10:54.852 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (20585 ms total) 2024-06-11T19:10:54.852 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-06-11T19:10:55.847 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: Running main() from gmock_main.cc 2024-06-11T19:10:55.847 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-06-11T19:10:55.847 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [----------] Global test environment set-up. 2024-06-11T19:10:55.847 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-06-11T19:10:55.847 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-06-11T19:10:55.847 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (6456 ms) 2024-06-11T19:10:55.847 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-06-11T19:10:55.848 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6885 ms) 2024-06-11T19:10:55.848 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-06-11T19:10:55.848 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (8022 ms) 2024-06-11T19:10:55.848 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [----------] 3 tests from NeoradosECList (21363 ms total) 2024-06-11T19:10:55.848 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: 2024-06-11T19:10:55.848 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [----------] Global test environment tear-down 2024-06-11T19:10:55.848 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (21363 ms total) 2024-06-11T19:10:55.848 INFO:tasks.workunit.client.0.smithi117.stdout: ec_list: [ PASSED ] 3 tests. 2024-06-11T19:10:56.896 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: Running main() from gmock_main.cc 2024-06-11T19:10:56.896 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-06-11T19:10:56.896 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [----------] Global test environment set-up. 2024-06-11T19:10:56.896 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-06-11T19:10:56.896 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-06-11T19:10:56.896 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (2376 ms) 2024-06-11T19:10:56.896 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-06-11T19:10:56.896 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3053 ms) 2024-06-11T19:10:56.896 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-06-11T19:10:56.896 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3005 ms) 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (2862 ms) 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3010 ms) 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3014 ms) 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (5038 ms) 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (22358 ms total) 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [----------] Global test environment tear-down 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (22358 ms total) 2024-06-11T19:10:56.897 INFO:tasks.workunit.client.0.smithi117.stdout: write_operations: [ PASSED ] 7 tests. 2024-06-11T19:10:56.905 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-06-11T19:10:56.905 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-06-11T19:10:56.905 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-06-11T19:10:56.905 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-06-11T19:10:56.905 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-06-11T19:10:56.905 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-06-11T19:10:56.905 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-06-11T19:10:56.905 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: 2024-06-11T19:10:56.905 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-06-11T19:10:56.905 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-06-11T19:10:56.905 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (2639 ms) 2024-06-11T19:10:56.905 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-06-11T19:10:56.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3033 ms) 2024-06-11T19:10:56.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-06-11T19:10:56.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2986 ms) 2024-06-11T19:10:56.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-06-11T19:10:56.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2867 ms) 2024-06-11T19:10:56.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-06-11T19:10:56.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3006 ms) 2024-06-11T19:10:56.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-06-11T19:10:56.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3007 ms) 2024-06-11T19:10:56.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-06-11T19:10:56.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (5055 ms) 2024-06-11T19:10:56.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (22593 ms total) 2024-06-11T19:10:56.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: 2024-06-11T19:10:56.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-06-11T19:10:56.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (22595 ms total) 2024-06-11T19:10:56.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-06-11T19:11:02.860 INFO:tasks.workunit.client.0.smithi117.stdout:otify_pp: flushed 2024-06-11T19:11:02.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3004 ms) 2024-06-11T19:11:02.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-06-11T19:11:02.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: trying... 2024-06-11T19:11:02.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94707273036160 notify_id 146028888073 notifier_gid 5070 2024-06-11T19:11:02.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: timed out 2024-06-11T19:11:02.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: flushing 2024-06-11T19:11:02.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: flushed 2024-06-11T19:11:02.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3006 ms) 2024-06-11T19:11:02.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: List watches 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify2 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94707273021584 notify_id 158913789963 notifier_gid 5070 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify2 done 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: watch_check 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: unwatch2 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: flushing 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: done 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3010 ms) 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: List watches 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify2 2024-06-11T19:11:02.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: handle_notify cookie 94707273021584 notify_id 171798691842 notifier_gid 5070 2024-06-11T19:11:02.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: notify2 done 2024-06-11T19:11:02.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: watch_check 2024-06-11T19:11:02.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: unwatch2 2024-06-11T19:11:02.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: flushing 2024-06-11T19:11:02.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: done 2024-06-11T19:11:02.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3012 ms) 2024-06-11T19:11:02.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16231 ms total) 2024-06-11T19:11:02.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: 2024-06-11T19:11:02.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-06-11T19:11:02.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28597 ms total) 2024-06-11T19:11:02.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-06-11T19:11:03.602 INFO:tasks.workunit.client.0.smithi117.stdout:api_service: 6.980% pgs not active 2024-06-11T19:11:03.602 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 373 active+clean 2024-06-11T19:11:03.602 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 320 unknown 2024-06-11T19:11:03.602 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 52 creating+peering 2024-06-11T19:11:03.602 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:11:03.602 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: io: 2024-06-11T19:11:03.602 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: client: 12 MiB/s rd, 31 MiB/s wr, 255 op/s rd, 290 op/s wr 2024-06-11T19:11:03.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:11:03.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:11:03.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4208 ms) 2024-06-11T19:11:03.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ RUN ] LibRadosService.Status 2024-06-11T19:11:03.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ OK ] LibRadosService.Status (20024 ms) 2024-06-11T19:11:03.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [----------] 4 tests from LibRadosService (29287 ms total) 2024-06-11T19:11:03.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: 2024-06-11T19:11:03.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [----------] Global test environment tear-down 2024-06-11T19:11:03.603 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29291 ms total) 2024-06-11T19:11:03.604 INFO:tasks.workunit.client.0.smithi117.stdout: api_service: [ PASSED ] 4 tests. 2024-06-11T19:11:04.672 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-06-11T19:11:04.673 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [----------] Global test environment set-up. 2024-06-11T19:11:04.673 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-06-11T19:11:04.673 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-06-11T19:11:04.673 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5043 ms) 2024-06-11T19:11:04.673 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-06-11T19:11:04.673 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (57 ms) 2024-06-11T19:11:04.673 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-06-11T19:11:04.673 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20043 ms) 2024-06-11T19:11:04.673 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-06-11T19:11:04.673 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: attempt 0 of 20 2024-06-11T19:11:04.673 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5201 ms) 2024-06-11T19:11:04.673 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30344 ms total) 2024-06-11T19:11:04.674 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: 2024-06-11T19:11:04.674 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [----------] Global test environment tear-down 2024-06-11T19:11:04.674 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30344 ms total) 2024-06-11T19:11:04.674 INFO:tasks.workunit.client.0.smithi117.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-06-11T19:11:06.911 INFO:tasks.workunit.client.0.smithi117.stdout: misc: Running main() from gmock_main.cc 2024-06-11T19:11:06.911 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-06-11T19:11:06.911 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [----------] Global test environment set-up. 2024-06-11T19:11:06.911 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-06-11T19:11:06.911 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-06-11T19:11:06.911 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.Version (1411 ms) 2024-06-11T19:11:06.911 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-06-11T19:11:06.911 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2034 ms) 2024-06-11T19:11:06.911 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-06-11T19:11:06.911 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.LongName (3024 ms) 2024-06-11T19:11:06.911 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3022 ms) 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (2834 ms) 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3022 ms) 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.Exec (2997 ms) 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3003 ms) 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3011 ms) 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3005 ms) 2024-06-11T19:11:06.912 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-06-11T19:11:06.913 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2012 ms) 2024-06-11T19:11:06.913 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-06-11T19:11:06.913 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3038 ms) 2024-06-11T19:11:06.913 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [----------] 12 tests from NeoRadosMisc (32413 ms total) 2024-06-11T19:11:06.913 INFO:tasks.workunit.client.0.smithi117.stdout: misc: 2024-06-11T19:11:06.913 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [----------] Global test environment tear-down 2024-06-11T19:11:06.913 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [==========] 12 tests from 1 test suite ran. (32413 ms total) 2024-06-11T19:11:06.913 INFO:tasks.workunit.client.0.smithi117.stdout: misc: [ PASSED ] 12 tests. 2024-06-11T19:11:10.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-06-11T19:11:10.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] Global test environment set-up. 2024-06-11T19:11:10.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-06-11T19:11:10.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-06-11T19:11:10.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-06-11T19:11:10.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-06-11T19:11:10.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: 2024-06-11T19:11:10.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-06-11T19:11:10.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-06-11T19:11:10.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-06-11T19:11:10.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: 2024-06-11T19:10:34.215+0000 7f1d82971780 -1 failed for service _ceph-mon._tcp 2024-06-11T19:11:10.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: 2024-06-11T19:10:34.215+0000 7f1d82971780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-06-11T19:11:10.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (75 ms) 2024-06-11T19:11:10.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-06-11T19:11:10.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2012 ms) 2024-06-11T19:11:10.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2087 ms total) 2024-06-11T19:11:10.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: 2024-06-11T19:11:10.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-06-11T19:11:10.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-06-11T19:11:10.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x563a52742a40 2024-06-11T19:11:10.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 0x563a5296da20 2024-06-11T19:11:10.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: started 2 aios 2024-06-11T19:11:10.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x563a52742a40 2024-06-11T19:11:10.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: waiting 0x563a5296da20 2024-06-11T19:11:10.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: done. 2024-06-11T19:11:10.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5726 ms) 2024-06-11T19:11:10.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5726 ms total) 2024-06-11T19:11:10.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: 2024-06-11T19:11:10.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-06-11T19:11:10.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-06-11T19:11:10.859 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-06-11T19:11:10.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-06-11T19:11:10.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.Exec (61 ms) 2024-06-11T19:11:10.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-06-11T19:11:10.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (22 ms) 2024-06-11T19:11:10.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-06-11T19:11:10.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (4 ms) 2024-06-11T19:11:10.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-06-11T19:11:10.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4771 ms) 2024-06-11T19:11:10.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-06-11T19:11:10.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-06-11T19:11:10.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-06-11T19:11:10.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-06-11T19:11:10.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-06-11T19:11:10.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (21820 ms) 2024-06-11T19:11:10.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] 8 tests from LibRadosMisc (26678 ms total) 2024-06-11T19:11:10.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: 2024-06-11T19:11:10.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [----------] Global test environment tear-down 2024-06-11T19:11:10.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (36699 ms total) 2024-06-11T19:11:10.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc: [ PASSED ] 12 tests. 2024-06-11T19:11:11.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: Running main() from gmock_main.cc 2024-06-11T19:11:11.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-06-11T19:11:11.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [----------] Global test environment set-up. 2024-06-11T19:11:11.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-06-11T19:11:11.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-06-11T19:11:11.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolList (2769 ms) 2024-06-11T19:11:11.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-06-11T19:11:11.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3019 ms) 2024-06-11T19:11:11.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-06-11T19:11:11.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3005 ms) 2024-06-11T19:11:11.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-06-11T19:11:11.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2878 ms) 2024-06-11T19:11:11.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-06-11T19:11:11.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3008 ms) 2024-06-11T19:11:11.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-06-11T19:11:11.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (4999 ms) 2024-06-11T19:11:11.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-06-11T19:11:11.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5011 ms) 2024-06-11T19:11:11.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-06-11T19:11:11.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5013 ms) 2024-06-11T19:11:11.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-06-11T19:11:11.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7963 ms) 2024-06-11T19:11:11.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [----------] 9 tests from LibRadosPools (37665 ms total) 2024-06-11T19:11:11.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: 2024-06-11T19:11:11.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [----------] Global test environment tear-down 2024-06-11T19:11:11.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (37665 ms total) 2024-06-11T19:11:11.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_pool: [ PASSED ] 9 tests. 2024-06-11T19:11:15.813 INFO:tasks.workunit.client.0.smithi117.stdout: io: Running main() from gmock_main.cc 2024-06-11T19:11:15.813 INFO:tasks.workunit.client.0.smithi117.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-06-11T19:11:15.813 INFO:tasks.workunit.client.0.smithi117.stdout: io: [----------] Global test environment set-up. 2024-06-11T19:11:15.813 INFO:tasks.workunit.client.0.smithi117.stdout: io: [----------] 14 tests from NeoRadosIo 2024-06-11T19:11:15.813 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-06-11T19:11:15.813 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.Limits (2483 ms) 2024-06-11T19:11:15.813 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-06-11T19:11:15.813 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3026 ms) 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.ReadOp (3011 ms) 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.SparseRead (2870 ms) 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3006 ms) 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3006 ms) 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (2998 ms) 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3022 ms) 2024-06-11T19:11:15.814 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-06-11T19:11:15.815 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2984 ms) 2024-06-11T19:11:15.815 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-06-11T19:11:15.815 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.Trunc (3026 ms) 2024-06-11T19:11:15.815 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-06-11T19:11:15.815 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.Remove (2974 ms) 2024-06-11T19:11:15.815 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-06-11T19:11:15.815 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3018 ms) 2024-06-11T19:11:15.815 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-06-11T19:11:15.815 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.RmXattr (2980 ms) 2024-06-11T19:11:15.815 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-06-11T19:11:15.815 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2982 ms) 2024-06-11T19:11:15.815 INFO:tasks.workunit.client.0.smithi117.stdout: io: [----------] 14 tests from NeoRadosIo (41386 ms total) 2024-06-11T19:11:15.816 INFO:tasks.workunit.client.0.smithi117.stdout: io: 2024-06-11T19:11:15.816 INFO:tasks.workunit.client.0.smithi117.stdout: io: [----------] Global test environment tear-down 2024-06-11T19:11:15.816 INFO:tasks.workunit.client.0.smithi117.stdout: io: [==========] 14 tests from 1 test suite ran. (41386 ms total) 2024-06-11T19:11:15.816 INFO:tasks.workunit.client.0.smithi117.stdout: io: [ PASSED ] 14 tests. 2024-06-11T19:11:15.817 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: Running main() from gmock_main.cc 2024-06-11T19:11:15.817 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-06-11T19:11:15.818 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [----------] Global test environment set-up. 2024-06-11T19:11:15.818 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-06-11T19:11:15.818 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-06-11T19:11:15.818 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2437 ms) 2024-06-11T19:11:15.818 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-06-11T19:11:15.818 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3008 ms) 2024-06-11T19:11:15.818 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-06-11T19:11:15.818 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3019 ms) 2024-06-11T19:11:15.818 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-06-11T19:11:15.818 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (2850 ms) 2024-06-11T19:11:15.818 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-06-11T19:11:15.818 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3022 ms) 2024-06-11T19:11:15.819 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-06-11T19:11:15.819 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (2995 ms) 2024-06-11T19:11:15.819 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-06-11T19:11:15.819 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3011 ms) 2024-06-11T19:11:15.819 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-06-11T19:11:15.819 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3019 ms) 2024-06-11T19:11:15.819 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-06-11T19:11:15.819 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (2984 ms) 2024-06-11T19:11:15.819 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-06-11T19:11:15.819 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3013 ms) 2024-06-11T19:11:15.819 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-06-11T19:11:15.819 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3038 ms) 2024-06-11T19:11:15.820 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-06-11T19:11:15.820 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (2926 ms) 2024-06-11T19:11:15.820 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-06-11T19:11:15.820 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3026 ms) 2024-06-11T19:11:15.820 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-06-11T19:11:15.820 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (2978 ms) 2024-06-11T19:11:15.820 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (41326 ms total) 2024-06-11T19:11:15.820 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: 2024-06-11T19:11:15.820 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [----------] Global test environment tear-down 2024-06-11T19:11:15.820 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (41326 ms total) 2024-06-11T19:11:15.820 INFO:tasks.workunit.client.0.smithi117.stdout: read_operations: [ PASSED ] 14 tests. 2024-06-11T19:11:27.147 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-06-11T19:11:27.147 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-06-11T19:11:27.147 INFO:tasks.ceph.mon.d:Restarting daemon 2024-06-11T19:11:27.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-06-11T19:11:27.150 INFO:tasks.ceph.mon.d:Started 2024-06-11T19:11:27.151 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:11:27.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:11:27.245 INFO:tasks.ceph.mon.d.smithi098.stderr:2024-06-11T19:11:27.245+0000 7fd21a1dcb00 -1 mon.d@-1(???).log v0 log_external_backlog rewinding external_log_to from 24 -> 0 (sync_force? mon rebuild?) 2024-06-11T19:11:27.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.257+0000 7fdead0d8640 10 monclient: get_monmap_and_config 2024-06-11T19:11:27.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.257+0000 7fdead0d8640 10 monclient: build_initial_monmap 2024-06-11T19:11:27.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.257+0000 7fdead0d8640 10 monclient: monmap: 2024-06-11T19:11:27.257 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:27.257 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:27.258961+0000 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:27.258961+0000 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:27.258 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:27.259 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:27.259 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:27.259 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.258+0000 7fdead0d8640 1 Processor -- start 2024-06-11T19:11:27.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.258+0000 7fdead0d8640 1 -- start start 2024-06-11T19:11:27.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.258+0000 7fdead0d8640 10 monclient: init 2024-06-11T19:11:27.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:27.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 10 monclient: _add_conns ranks=[8,3,7,9,0,4,1,5,6,2,10] 2024-06-11T19:11:27.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdea80a4920 0x7fdea80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 10 monclient(hunting): picked mon.noname-e con 0x7fdea80a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:27.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdea80a5240 0x7fdea80a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 10 monclient(hunting): picked mon.noname-g con 0x7fdea80a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdea80a2530 0x7fdea80a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 10 monclient(hunting): picked mon.noname-i con 0x7fdea80a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fdea80a7140 con 0x7fdea80a5240 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fdea80a2b50 con 0x7fdea80a2530 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fdea80a2e30 con 0x7fdea80a4920 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdead0d8640 10 monclient(hunting): authenticate will time out at 897.066321s 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdea80a4920 0x7fdea80a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdea80a4920 0x7fdea80a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34450/0 (socket says 172.21.15.98:34450) 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 1 -- 172.21.15.98:0/3948598794 learned_addr learned my addr 172.21.15.98:0/3948598794 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:27.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 10 monclient(hunting): get_auth_request con 0x7fdea80a4920 auth_method 0 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fde9ffff640 1 --2- 172.21.15.98:0/3948598794 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdea80a5240 0x7fdea80a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea759e640 1 --2- 172.21.15.98:0/3948598794 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdea80a2530 0x7fdea80a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fde9ffff640 10 monclient(hunting): get_auth_request con 0x7fdea80a5240 auth_method 0 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fde9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fde9ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fde9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea759e640 10 monclient(hunting): get_auth_request con 0x7fdea80a2530 auth_method 0 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea759e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea759e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 10 monclient(hunting): handle_auth_done global_id 22679 payload 274 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 1 -- 172.21.15.98:0/3948598794 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdea80a2530 msgr2=0x7fdea80a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 1 --2- 172.21.15.98:0/3948598794 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdea80a2530 0x7fdea80a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 1 -- 172.21.15.98:0/3948598794 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdea80a5240 msgr2=0x7fdea80a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 1 --2- 172.21.15.98:0/3948598794 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdea80a5240 0x7fdea80a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 1 monclient: found mon.noname-e 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.259+0000 7fdea6d9d640 1 -- 172.21.15.98:0/3948598794 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea80a3110 con 0x7fdea80a4920 2024-06-11T19:11:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea6d9d640 1 --2- 172.21.15.98:0/3948598794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdea80a4920 0x7fdea80a4d00 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fde980098a0 tx=0x7fde9802b7a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=d4bd196ec2b091cb server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 1 -- 172.21.15.98:0/3948598794 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde98038070 con 0x7fdea80a4920 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 10 monclient: dump: 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.273 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 10 monclient: _finish_auth 0 2024-06-11T19:11:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 10 monclient: _check_auth_tickets 2024-06-11T19:11:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdead0d8640 5 monclient: authenticate success, global_id 22679 2024-06-11T19:11:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 1 -- 172.21.15.98:0/3948598794 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde98031970 con 0x7fdea80a4920 2024-06-11T19:11:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdead0d8640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdead0d8640 4 set_mon_vals no callback set 2024-06-11T19:11:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 1 -- 172.21.15.98:0/3948598794 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde98031ba0 con 0x7fdea80a4920 2024-06-11T19:11:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.260+0000 7fdea659c640 10 monclient: dump: 2024-06-11T19:11:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.261+0000 7fdead0d8640 10 monclient: shutdown 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.261+0000 7fdead0d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.261+0000 7fdead0d8640 1 -- 172.21.15.98:0/3948598794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdea80a4920 msgr2=0x7fdea80a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.261+0000 7fdead0d8640 1 --2- 172.21.15.98:0/3948598794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdea80a4920 0x7fdea80a4d00 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fde980098a0 tx=0x7fde9802b7a0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.261+0000 7fdead0d8640 1 -- 172.21.15.98:0/3948598794 shutdown_connections 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.261+0000 7fdead0d8640 1 --2- 172.21.15.98:0/3948598794 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdea80a2530 0x7fdea80a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.261+0000 7fdead0d8640 1 --2- 172.21.15.98:0/3948598794 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdea80a5240 0x7fdea80a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.261+0000 7fdead0d8640 1 --2- 172.21.15.98:0/3948598794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdea80a4920 0x7fdea80a4d00 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.261+0000 7fdead0d8640 1 -- 172.21.15.98:0/3948598794 >> 172.21.15.98:0/3948598794 conn(0x7fdea80ab850 msgr2=0x7fdea80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.261+0000 7fdead0d8640 1 -- 172.21.15.98:0/3948598794 shutdown_connections 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.261+0000 7fdead0d8640 1 -- 172.21.15.98:0/3948598794 wait complete. 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.262+0000 7fdead0d8640 10 monclient: build_initial_monmap 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.262+0000 7fdead0d8640 10 monclient: monmap: 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.262+0000 7fdead0d8640 1 librados: starting msgr at 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.262+0000 7fdead0d8640 1 librados: starting objecter 2024-06-11T19:11:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.262+0000 7fdead0d8640 1 Processor -- start 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.262+0000 7fdead0d8640 1 -- start start 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.262+0000 7fdead0d8640 1 librados: setting wanted keys 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.262+0000 7fdead0d8640 1 librados: calling monclient init 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.262+0000 7fdead0d8640 10 monclient: init 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 10 monclient: _add_conns ranks=[5,3,6,2,8,4,7,1,0] 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdea80a2530 0x7fdea8137740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 10 monclient(hunting): picked mon.noname-h con 0x7fdea80a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdea80a4920 0x7fdea8137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 10 monclient(hunting): picked mon.noname-g con 0x7fdea80a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdea80a5240 0x7fdea81232a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 10 monclient(hunting): picked mon.noname-d con 0x7fdea80a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fdea80a6da0 con 0x7fdea80a4920 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fdea8138e50 con 0x7fdea80a2530 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fdea80a6420 con 0x7fdea80a5240 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdead0d8640 10 monclient(hunting): authenticate will time out at 897.070436s 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea759e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdea80a5240 0x7fdea81232a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea759e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdea80a5240 0x7fdea81232a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:41502/0 (socket says 172.21.15.98:41502) 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea759e640 1 -- 172.21.15.98:0/2011909464 learned_addr learned my addr 172.21.15.98:0/2011909464 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea759e640 10 monclient(hunting): get_auth_request con 0x7fdea80a5240 auth_method 0 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea759e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea759e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fde9ffff640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdea80a4920 0x7fdea8137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea6d9d640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdea80a2530 0x7fdea8137740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fde9ffff640 10 monclient(hunting): get_auth_request con 0x7fdea80a4920 auth_method 0 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fde9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fde9ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fde9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea759e640 1 -- 172.21.15.98:0/2011909464 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdea80a5240 msgr2=0x7fdea81232a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea6d9d640 10 monclient(hunting): get_auth_request con 0x7fdea80a2530 auth_method 0 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea759e640 1 -- 172.21.15.98:0/2011909464 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdea80a5240 msgr2=0x7fdea81232a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea6d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea759e640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdea80a5240 0x7fdea81232a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea759e640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdea80a5240 0x7fdea81232a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fde9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fde9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fde9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.263+0000 7fdea6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9ffff640 10 monclient(hunting): handle_auth_done global_id 21477 payload 824 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9ffff640 1 -- 172.21.15.98:0/2011909464 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdea80a5240 msgr2=0x7fdea81232a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9ffff640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdea80a5240 0x7fdea81232a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9ffff640 1 -- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdea80a2530 msgr2=0x7fdea8137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9ffff640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdea80a2530 0x7fdea8137740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9ffff640 1 monclient: found mon.noname-g 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9ffff640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:11:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9ffff640 1 -- 172.21.15.98:0/2011909464 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea81c0d10 con 0x7fdea80a4920 2024-06-11T19:11:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fdea6d9d640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdea80a2530 0x7fdea8137740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9ffff640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdea80a4920 0x7fdea8137c80 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fdea8073fc0 tx=0x7fde9000d980 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 10 client.?.objecter ms_handle_connect 0x7fdea80a4920 2024-06-11T19:11:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 1 -- 172.21.15.98:0/2011909464 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde90012a60 con 0x7fdea80a4920 2024-06-11T19:11:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:11:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 10 monclient: dump: 2024-06-11T19:11:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 1 -- 172.21.15.98:0/2011909464 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde90019070 con 0x7fdea80a4920 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fdead0d8640 5 monclient: authenticate success, global_id 21477 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 1 -- 172.21.15.98:0/2011909464 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde90012c40 con 0x7fdea80a4920 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fde9f7fe640 10 monclient: dump: 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.285 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fdea5d9b640 4 set_mon_vals no callback set 2024-06-11T19:11:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fdead0d8640 10 monclient: _renew_subs 2024-06-11T19:11:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fdead0d8640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:11:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fdead0d8640 1 -- 172.21.15.98:0/2011909464 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdea8126b80 con 0x7fdea80a4920 2024-06-11T19:11:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fdead0d8640 10 client.21477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fdead0d8640 10 monclient: _renew_subs 2024-06-11T19:11:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fdead0d8640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:11:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fdead0d8640 1 -- 172.21.15.98:0/2011909464 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdea80a92f0 con 0x7fdea80a4920 2024-06-11T19:11:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.264+0000 7fdead0d8640 1 librados: init done 2024-06-11T19:11:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.265+0000 7fde9f7fe640 1 -- 172.21.15.98:0/2011909464 <== mon.3 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fde900125f0 con 0x7fdea80a4920 2024-06-11T19:11:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.265+0000 7fdead0d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.265+0000 7fdead0d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.265+0000 7fdead0d8640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:11:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.265+0000 7fdead0d8640 1 -- 172.21.15.98:0/2011909464 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdea8073550 con 0x7fdea80a4920 2024-06-11T19:11:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.266+0000 7fde9f7fe640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fde8803fdb0 0x7fde88042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.266+0000 7fde9f7fe640 1 -- 172.21.15.98:0/2011909464 <== mon.3 v2:172.21.15.117:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 8794+0+0 (secure 0 0 0) 0x7fde9000f040 con 0x7fdea80a4920 2024-06-11T19:11:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.266+0000 7fde9f7fe640 10 client.21477.objecter ms_dispatch 0x7fdea8000b90 osd_map(71..71 src has 1..71) 2024-06-11T19:11:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.266+0000 7fde9f7fe640 3 client.21477.objecter handle_osd_map got epochs [71,71] > 0 2024-06-11T19:11:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.266+0000 7fde9f7fe640 3 client.21477.objecter handle_osd_map decoding full epoch 71 2024-06-11T19:11:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.266+0000 7fdea6d9d640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fde8803fdb0 0x7fde88042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.266+0000 7fde9f7fe640 20 client.21477.objecter dump_active .. 0 homeless 2024-06-11T19:11:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.266+0000 7fdea6d9d640 10 monclient: get_auth_request con 0x7fde8803fdb0 auth_method 0 2024-06-11T19:11:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.266+0000 7fdea6d9d640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fde8803fdb0 0x7fde88042270 secure :-1 s=READY pgs=3965 cs=0 l=1 rev1=1 crypto rx=0x7fde98035000 tx=0x7fde98008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.266+0000 7fde9f7fe640 10 client.21477.objecter ms_handle_connect 0x7fde8803fdb0 2024-06-11T19:11:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.270+0000 7fde9f7fe640 1 -- 172.21.15.98:0/2011909464 <== mon.3 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdea8073550 con 0x7fdea80a4920 2024-06-11T19:11:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.271+0000 7fde9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.271+0000 7fde9f7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:27.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.539+0000 7fdead0d8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:11:27.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.539+0000 7fdead0d8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:11:27.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.539+0000 7fdead0d8640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:11:27.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.539+0000 7fdead0d8640 1 -- 172.21.15.98:0/2011909464 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdea80a66f0 con 0x7fdea80a4920 2024-06-11T19:11:27.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.540+0000 7fde9f7fe640 1 -- 172.21.15.98:0/2011909464 <== mon.3 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fdea81c0d10 con 0x7fdea80a4920 2024-06-11T19:11:27.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.540+0000 7fde9f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:11:27.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.540+0000 7fde9f7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:27.540 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":12,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:11:27.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 10 librados: watch_flush enter 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 10 librados: watch_flush exit 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 20 client.21477.objecter shutdown clearing up homeless session... 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 10 client.21477.objecter successfully canceled tick 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 -- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fde8803fdb0 msgr2=0x7fde88042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fde8803fdb0 0x7fde88042270 secure :-1 s=READY pgs=3965 cs=0 l=1 rev1=1 crypto rx=0x7fde98035000 tx=0x7fde98008030 comp rx=0 tx=0).stop 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 10 monclient: shutdown 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 -- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdea80a4920 msgr2=0x7fdea8137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdea80a4920 0x7fdea8137c80 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fdea8073fc0 tx=0x7fde9000d980 comp rx=0 tx=0).stop 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 -- 172.21.15.98:0/2011909464 shutdown_connections 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fde8803fdb0 0x7fde88042270 unknown :-1 s=CLOSED pgs=3965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdea80a5240 0x7fdea81232a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdea80a4920 0x7fdea8137c80 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 --2- 172.21.15.98:0/2011909464 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdea80a2530 0x7fdea8137740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 -- 172.21.15.98:0/2011909464 >> 172.21.15.98:0/2011909464 conn(0x7fdea80ab850 msgr2=0x7fdea80a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:27.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 -- 172.21.15.98:0/2011909464 shutdown_connections 2024-06-11T19:11:27.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 -- 172.21.15.98:0/2011909464 wait complete. 2024-06-11T19:11:27.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.542+0000 7fdead0d8640 1 librados: shutdown 2024-06-11T19:11:27.552 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:11:27.553 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:11:27.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:11:27.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.653+0000 7f0585c2e640 10 monclient: get_monmap_and_config 2024-06-11T19:11:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.653+0000 7f0585c2e640 10 monclient: build_initial_monmap 2024-06-11T19:11:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.653+0000 7f0585c2e640 10 monclient: monmap: 2024-06-11T19:11:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:27.654205+0000 2024-06-11T19:11:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:27.654205+0000 2024-06-11T19:11:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:27.653 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:27.654 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:27.654 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 1 Processor -- start 2024-06-11T19:11:27.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 1 -- start start 2024-06-11T19:11:27.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 10 monclient: init 2024-06-11T19:11:27.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:27.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 10 monclient: _add_conns ranks=[9,10,1,8,2,4,5,0,7,3,6] 2024-06-11T19:11:27.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0580131c60 0x7f05800a1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 10 monclient(hunting): picked mon.noname-a con 0x7f0580131c60 addr v2:172.21.15.98:3300/0 2024-06-11T19:11:27.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0580132040 0x7f0580132460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 10 monclient(hunting): picked mon.noname-f con 0x7f0580132040 addr v2:172.21.15.117:3300/0 2024-06-11T19:11:27.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f0580134e30 addr v1:172.21.15.117:6789/0 2024-06-11T19:11:27.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05800a02a0 con 0x7f0580134e30 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f05800a0580 con 0x7f0580131c60 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f05800a0860 con 0x7f0580132040 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f0585c2e640 10 monclient(hunting): authenticate will time out at 897.461665s 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f057f7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0580131c60 0x7f05800a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f057f7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0580131c60 0x7f05800a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53662/0 (socket says 172.21.15.98:53662) 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f057f7fe640 1 -- 172.21.15.98:0/3508262580 learned_addr learned my addr 172.21.15.98:0/3508262580 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f057f7fe640 10 monclient(hunting): get_auth_request con 0x7f0580131c60 auth_method 0 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f057f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f057f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f057f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f057f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f057f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f057f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.654+0000 7f057f7fe640 10 monclient(hunting): handle_auth_done global_id 30744 payload 274 2024-06-11T19:11:27.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057f7fe640 1 -- 172.21.15.98:0/3508262580 >> v2:172.21.15.117:3300/0 conn(0x7f0580132040 msgr2=0x7f0580132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:27.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057f7fe640 1 --2- 172.21.15.98:0/3508262580 >> v2:172.21.15.117:3300/0 conn(0x7f0580132040 0x7f0580132460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057f7fe640 1 -- 172.21.15.98:0/3508262580 >> v1:172.21.15.117:6789/0 conn(0x7f0580134e30 legacy=0x7f0580135280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:27.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057f7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:11:27.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057f7fe640 1 monclient: found mon.noname-a 2024-06-11T19:11:27.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:27.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:11:27.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057f7fe640 1 -- 172.21.15.98:0/3508262580 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058009f570 con 0x7f0580131c60 2024-06-11T19:11:27.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057f7fe640 1 --2- 172.21.15.98:0/3508262580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580131c60 0x7f05800a1520 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f05800a84b0 tx=0x7f057002c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a70e2c768ae00f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057effd640 1 -- 172.21.15.98:0/3508262580 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0570036020 con 0x7f0580131c60 2024-06-11T19:11:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:11:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057effd640 10 monclient: dump: 2024-06-11T19:11:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.667 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057effd640 10 monclient: _finish_auth 0 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057effd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057effd640 1 -- 172.21.15.98:0/3508262580 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f057002e040 con 0x7f0580131c60 2024-06-11T19:11:27.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f0585c2e640 5 monclient: authenticate success, global_id 30744 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f0585c2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f0585c2e640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057effd640 1 -- 172.21.15.98:0/3508262580 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05700313b0 con 0x7f0580131c60 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f0585c2e640 4 set_mon_vals no callback set 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.655+0000 7f057effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.656+0000 7f057effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.656+0000 7f057effd640 10 monclient: dump: 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.669 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.670 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.670 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.670 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.670 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.670 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.670 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.670 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.670 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.670 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.656+0000 7f0585c2e640 10 monclient: shutdown 2024-06-11T19:11:27.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.656+0000 7f0585c2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:27.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.656+0000 7f0585c2e640 1 -- 172.21.15.98:0/3508262580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580131c60 msgr2=0x7f05800a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.656+0000 7f0585c2e640 1 --2- 172.21.15.98:0/3508262580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580131c60 0x7f05800a1520 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f05800a84b0 tx=0x7f057002c900 comp rx=0 tx=0).stop 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.656+0000 7f0585c2e640 1 -- 172.21.15.98:0/3508262580 shutdown_connections 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.656+0000 7f0585c2e640 1 --2- 172.21.15.98:0/3508262580 >> v2:172.21.15.117:3300/0 conn(0x7f0580132040 0x7f0580132460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.656+0000 7f0585c2e640 1 --2- 172.21.15.98:0/3508262580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580131c60 0x7f05800a1520 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.656+0000 7f0585c2e640 1 -- 172.21.15.98:0/3508262580 >> 172.21.15.98:0/3508262580 conn(0x7f05800a7850 msgr2=0x7f05800a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.656+0000 7f0585c2e640 1 -- 172.21.15.98:0/3508262580 shutdown_connections 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.656+0000 7f0585c2e640 1 -- 172.21.15.98:0/3508262580 wait complete. 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 10 monclient: build_initial_monmap 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 10 monclient: monmap: 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:27.671 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 1 librados: starting msgr at 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 1 librados: starting objecter 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 1 Processor -- start 2024-06-11T19:11:27.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 1 -- start start 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 1 librados: setting wanted keys 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 1 librados: calling monclient init 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 10 monclient: init 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 10 monclient: _add_conns ranks=[0,2,8,4,3,1,5,7,6] 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580131c60 0x7f05801c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 10 monclient(hunting): picked mon.noname-a con 0x7f0580131c60 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0580132040 0x7f058009a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 10 monclient(hunting): picked mon.noname-b con 0x7f0580132040 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.657+0000 7f0585c2e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0580134e30 0x7f05801ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 10 monclient(hunting): picked mon.noname-e con 0x7f0580134e30 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0580135b10 con 0x7f0580131c60 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0580135990 con 0x7f0580132040 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0580135c90 con 0x7f0580134e30 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 10 monclient(hunting): authenticate will time out at 897.465018s 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580131c60 0x7f05801c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580131c60 0x7f05801c94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53676/0 (socket says 172.21.15.98:53676) 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 1 -- 172.21.15.98:0/34345523 learned_addr learned my addr 172.21.15.98:0/34345523 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 10 monclient(hunting): get_auth_request con 0x7f0580131c60 auth_method 0 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0577fff640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0580132040 0x7f058009a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057ffff640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0580134e30 0x7f05801ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0577fff640 10 monclient(hunting): get_auth_request con 0x7f0580132040 auth_method 0 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0577fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0577fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0577fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057ffff640 10 monclient(hunting): get_auth_request con 0x7f0580134e30 auth_method 0 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0577fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0577fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0577fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 10 monclient(hunting): handle_auth_done global_id 30753 payload 824 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 1 -- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0580134e30 msgr2=0x7f05801ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0580134e30 0x7f05801ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 1 -- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0580132040 msgr2=0x7f058009a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0580132040 0x7f058009a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 1 monclient: found mon.noname-a 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:27.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 1 -- 172.21.15.98:0/34345523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058009deb0 con 0x7f0580131c60 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057f7fe640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580131c60 0x7f05801c94d0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f0570000c00 tx=0x7f05700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 10 client.?.objecter ms_handle_connect 0x7f0580131c60 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0570036040 con 0x7f0580131c60 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 10 monclient: dump: 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.677 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 10 monclient: _finish_auth 0 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 10 monclient: _check_auth_tickets 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0570003a80 con 0x7f0580131c60 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 5 monclient: authenticate success, global_id 30753 2024-06-11T19:11:27.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0570003c30 con 0x7f0580131c60 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 10 monclient: _renew_subs 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 1 -- 172.21.15.98:0/34345523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05800987b0 con 0x7f0580131c60 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057e7fc640 4 set_mon_vals no callback set 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f057cff9640 10 monclient: dump: 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.679 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 10 client.30753.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 10 monclient: _renew_subs 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 1 -- 172.21.15.98:0/34345523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0580135310 con 0x7f0580131c60 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.658+0000 7f0585c2e640 1 librados: init done 2024-06-11T19:11:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0585c2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0585c2e640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580097850 0x7f05801caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0585c2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0585c2e640 1 -- 172.21.15.98:0/34345523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0580050830 con 0x7f0580097850 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0577fff640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580097850 0x7f05801caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0577fff640 10 monclient: get_auth_request con 0x7f0580097850 auth_method 0 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0577fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0577fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0577fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0577fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f0570039020 con 0x7f0580131c60 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0577fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0577fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0577fff640 10 monclient(hunting): handle_auth_done global_id 30762 payload 824 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.659+0000 7f0577fff640 1 -- 172.21.15.98:0/34345523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0580098fb0 con 0x7f0580097850 2024-06-11T19:11:27.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f056003fdb0 0x7f0560042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8794+0+0 (secure 0 0 0) 0x7f05700745b0 con 0x7f0580131c60 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 10 client.30753.objecter ms_dispatch 0x7f0580000b90 osd_map(71..71 src has 1..71) 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 3 client.30753.objecter handle_osd_map got epochs [71,71] > 0 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 3 client.30753.objecter handle_osd_map decoding full epoch 71 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f0577fff640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580097850 0x7f05801caa90 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f058012f800 tx=0x7f056800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 20 client.30753.objecter dump_active .. 0 homeless 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 10 client.30753.objecter ms_handle_connect 0x7f0580097850 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 10 client.30753.objecter resend_mon_ops 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f056800c7a0 con 0x7f0580097850 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 10 monclient: dump: 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.682 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057ffff640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f056003fdb0 0x7f0560042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057ffff640 10 monclient: get_auth_request con 0x7f056003fdb0 auth_method 0 2024-06-11T19:11:27.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0568005a10 con 0x7f0580097850 2024-06-11T19:11:27.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:27.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:27.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580097850 msgr2=0x7f05801caa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580097850 0x7f05801caa90 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f058012f800 tx=0x7f056800ec00 comp rx=0 tx=0).stop 2024-06-11T19:11:27.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580097850 msgr2=0x7f05801caa90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:27.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580097850 0x7f05801caa90 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057ffff640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f056003fdb0 0x7f0560042270 secure :-1 s=READY pgs=3967 cs=0 l=1 rev1=1 crypto rx=0x7f0578009fa0 tx=0x7f057800d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.660+0000 7f057cff9640 10 client.30753.objecter ms_handle_connect 0x7f056003fdb0 2024-06-11T19:11:27.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.666+0000 7f0585c2e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:27.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.666+0000 7f0585c2e640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058009aac0 0x7f05801cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.666+0000 7f0585c2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.666+0000 7f0585c2e640 1 -- 172.21.15.98:0/34345523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05800a1520 con 0x7f058009aac0 2024-06-11T19:11:27.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f0577fff640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058009aac0 0x7f05801cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f0577fff640 10 monclient: get_auth_request con 0x7f058009aac0 auth_method 0 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f0577fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f0577fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f0577fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f0577fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f0577fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f0577fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f0577fff640 10 monclient(hunting): handle_auth_done global_id 30771 payload 824 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f0577fff640 1 -- 172.21.15.98:0/34345523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0580068c60 con 0x7f058009aac0 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 10 client.30753.objecter ms_handle_connect 0x7f058009aac0 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f0577fff640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058009aac0 0x7f05801cafd0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f058012f800 tx=0x7f0568004820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 10 client.30753.objecter resend_mon_ops 2024-06-11T19:11:27.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0568006670 con 0x7f058009aac0 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 10 monclient: dump: 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.688 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.689 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7f0568006670 con 0x7f058009aac0 2024-06-11T19:11:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058009aac0 msgr2=0x7f05801cafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058009aac0 0x7f05801cafd0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f058012f800 tx=0x7f0568004820 comp rx=0 tx=0).stop 2024-06-11T19:11:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 1 -- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058009aac0 msgr2=0x7f05801cafd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.667+0000 7f057cff9640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f058009aac0 0x7f05801cafd0 secure :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f058012f800 tx=0x7f0568004820 comp rx=0 tx=0).stop 2024-06-11T19:11:27.690 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:11:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 137904, 2024-06-11T19:11:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:27.692 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:27.693 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:27.694 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:27.698 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.700 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:27.701 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:27.701 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:27.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:27.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:27.708 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:27.712 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:27.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 10 librados: watch_flush enter 2024-06-11T19:11:27.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 10 librados: watch_flush exit 2024-06-11T19:11:27.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 20 client.30753.objecter shutdown clearing up homeless session... 2024-06-11T19:11:27.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 10 client.30753.objecter successfully canceled tick 2024-06-11T19:11:27.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 1 -- 172.21.15.98:0/34345523 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f056003fdb0 msgr2=0x7f0560042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f056003fdb0 0x7f0560042270 secure :-1 s=READY pgs=3967 cs=0 l=1 rev1=1 crypto rx=0x7f0578009fa0 tx=0x7f057800d040 comp rx=0 tx=0).stop 2024-06-11T19:11:27.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 10 monclient: shutdown 2024-06-11T19:11:27.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:27.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 1 -- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580131c60 msgr2=0x7f05801c94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 1 --2- 172.21.15.98:0/34345523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0580131c60 0x7f05801c94d0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f0570000c00 tx=0x7f05700046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f057ffff640 1 -- 172.21.15.98:0/34345523 reap_dead start 2024-06-11T19:11:27.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f057ffff640 1 -- 172.21.15.98:0/34345523 reap_dead start 2024-06-11T19:11:27.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 1 -- 172.21.15.98:0/34345523 shutdown_connections 2024-06-11T19:11:27.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 1 -- 172.21.15.98:0/34345523 >> 172.21.15.98:0/34345523 conn(0x7f05800a7850 msgr2=0x7f05800a21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:27.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 1 -- 172.21.15.98:0/34345523 shutdown_connections 2024-06-11T19:11:27.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 1 -- 172.21.15.98:0/34345523 wait complete. 2024-06-11T19:11:27.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.668+0000 7f0585c2e640 1 librados: shutdown 2024-06-11T19:11:27.719 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:27.719 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:11:27.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:11:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.819+0000 7ff5417fb640 10 monclient: get_monmap_and_config 2024-06-11T19:11:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.819+0000 7ff5417fb640 10 monclient: build_initial_monmap 2024-06-11T19:11:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.819+0000 7ff5417fb640 10 monclient: monmap: 2024-06-11T19:11:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:27.820566+0000 2024-06-11T19:11:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:27.820566+0000 2024-06-11T19:11:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:27.820 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 1 Processor -- start 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 1 -- start start 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 10 monclient: init 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 10 monclient: _add_conns ranks=[9,8,1,7,2,4,10,0,5,6,3] 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff53c0a4920 0x7ff53c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 10 monclient(hunting): picked mon.noname-a con 0x7ff53c0a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff53c0a5240 0x7ff53c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 10 monclient(hunting): picked mon.noname-e con 0x7ff53c0a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff53c0a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff53c0a7140 con 0x7ff53c0a2530 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff53c0a2b50 con 0x7ff53c0a4920 2024-06-11T19:11:27.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff53c0a2e30 con 0x7ff53c0a5240 2024-06-11T19:11:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff5417fb640 10 monclient(hunting): authenticate will time out at 897.627633s 2024-06-11T19:11:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53a7fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff53c0a5240 0x7ff53c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53affd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff53c0a4920 0x7ff53c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53affd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff53c0a4920 0x7ff53c0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53706/0 (socket says 172.21.15.98:53706) 2024-06-11T19:11:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53affd640 1 -- 172.21.15.98:0/3075327474 learned_addr learned my addr 172.21.15.98:0/3075327474 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53affd640 10 monclient(hunting): get_auth_request con 0x7ff53c0a4920 auth_method 0 2024-06-11T19:11:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53a7fc640 10 monclient(hunting): get_auth_request con 0x7ff53c0a5240 auth_method 0 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.820+0000 7ff53affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_done global_id 22697 payload 274 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff53a7fc640 1 -- 172.21.15.98:0/3075327474 >> v2:172.21.15.98:3300/0 conn(0x7ff53c0a4920 msgr2=0x7ff53c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff53a7fc640 1 --2- 172.21.15.98:0/3075327474 >> v2:172.21.15.98:3300/0 conn(0x7ff53c0a4920 0x7ff53c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff53a7fc640 1 -- 172.21.15.98:0/3075327474 >> v1:172.21.15.117:6789/0 conn(0x7ff53c0a2530 legacy=0x7ff53c0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff53a7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff53a7fc640 1 monclient: found mon.noname-e 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff53a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff53a7fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff53a7fc640 1 -- 172.21.15.98:0/3075327474 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53c0a3110 con 0x7ff53c0a5240 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff53affd640 1 --2- 172.21.15.98:0/3075327474 >> v2:172.21.15.98:3300/0 conn(0x7ff53c0a4920 0x7ff53c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff53a7fc640 1 --2- 172.21.15.98:0/3075327474 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff53c0a5240 0x7ff53c0a59a0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7ff5300097a0 tx=0x7ff53002b6f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=353c2e8f2f399de3 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff539ffb640 1 -- 172.21.15.98:0/3075327474 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff530037070 con 0x7ff53c0a5240 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff539ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff539ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff539ffb640 10 monclient: dump: 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff539ffb640 10 monclient: _finish_auth 0 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff539ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff539ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff5417fb640 5 monclient: authenticate success, global_id 22697 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff539ffb640 1 -- 172.21.15.98:0/3075327474 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff530030980 con 0x7ff53c0a5240 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff5417fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.821+0000 7ff539ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff5417fb640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff5417fb640 4 set_mon_vals no callback set 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff539ffb640 1 -- 172.21.15.98:0/3075327474 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff530030bb0 con 0x7ff53c0a5240 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff539ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff539ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff539ffb640 10 monclient: dump: 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.837 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff5417fb640 10 monclient: shutdown 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff5417fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff5417fb640 1 -- 172.21.15.98:0/3075327474 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff53c0a5240 msgr2=0x7ff53c0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff5417fb640 1 --2- 172.21.15.98:0/3075327474 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff53c0a5240 0x7ff53c0a59a0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7ff5300097a0 tx=0x7ff53002b6f0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff5417fb640 1 -- 172.21.15.98:0/3075327474 shutdown_connections 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff5417fb640 1 --2- 172.21.15.98:0/3075327474 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff53c0a5240 0x7ff53c0a59a0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff5417fb640 1 --2- 172.21.15.98:0/3075327474 >> v2:172.21.15.98:3300/0 conn(0x7ff53c0a4920 0x7ff53c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff5417fb640 1 -- 172.21.15.98:0/3075327474 >> 172.21.15.98:0/3075327474 conn(0x7ff53c0ab850 msgr2=0x7ff53c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff5417fb640 1 -- 172.21.15.98:0/3075327474 shutdown_connections 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.822+0000 7ff5417fb640 1 -- 172.21.15.98:0/3075327474 wait complete. 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 10 monclient: build_initial_monmap 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 10 monclient: monmap: 2024-06-11T19:11:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 1 librados: starting msgr at 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 1 librados: starting objecter 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 1 Processor -- start 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 1 -- start start 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 1 librados: setting wanted keys 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 1 librados: calling monclient init 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 10 monclient: init 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 10 monclient: _add_conns ranks=[8,7,4,3,1,2,0,6,5] 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff53c0a2530 0x7ff53c1376f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 10 monclient(hunting): picked mon.noname-e con 0x7ff53c0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff53c0a4920 0x7ff53c137c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.823+0000 7ff5417fb640 10 monclient(hunting): picked mon.noname-i con 0x7ff53c0a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff53c0a5240 0x7ff53c123160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 10 monclient(hunting): picked mon.noname-c con 0x7ff53c0a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff53c0a6da0 con 0x7ff53c0a5240 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff53c138e50 con 0x7ff53c0a4920 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff53c0a65f0 con 0x7ff53c0a2530 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 10 monclient(hunting): authenticate will time out at 897.631035s 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff53c0a2530 0x7ff53c1376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff53c0a2530 0x7ff53c1376f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34476/0 (socket says 172.21.15.98:34476) 2024-06-11T19:11:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 1 -- 172.21.15.98:0/3393435857 learned_addr learned my addr 172.21.15.98:0/3393435857 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 10 monclient(hunting): get_auth_request con 0x7ff53c0a2530 auth_method 0 2024-06-11T19:11:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53b7fe640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff53c0a5240 0x7ff53c123160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53b7fe640 10 monclient(hunting): get_auth_request con 0x7ff53c0a5240 auth_method 0 2024-06-11T19:11:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53a7fc640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff53c0a4920 0x7ff53c137c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53a7fc640 10 monclient(hunting): get_auth_request con 0x7ff53c0a4920 auth_method 0 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 10 monclient(hunting): handle_auth_done global_id 22706 payload 824 2024-06-11T19:11:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 1 -- 172.21.15.98:0/3393435857 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff53c0a4920 msgr2=0x7ff53c137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff53c0a4920 0x7ff53c137c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 1 -- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff53c0a5240 msgr2=0x7ff53c123160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff53c0a5240 0x7ff53c123160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 10 monclient: _finish_hunting 0 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 1 monclient: found mon.noname-e 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 1 -- 172.21.15.98:0/3393435857 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53c1c0bf0 con 0x7ff53c0a2530 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff53affd640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff53c0a2530 0x7ff53c1376f0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00ea30 tx=0x7ff52c00eef0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 10 client.?.objecter ms_handle_connect 0x7ff53c0a2530 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff52c017070 con 0x7ff53c0a2530 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 10 monclient: dump: 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.846 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 10 monclient: _finish_auth 0 2024-06-11T19:11:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 10 monclient: _check_auth_tickets 2024-06-11T19:11:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 5 monclient: authenticate success, global_id 22706 2024-06-11T19:11:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff52c013470 con 0x7ff53c0a2530 2024-06-11T19:11:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff52c013a90 con 0x7ff53c0a2530 2024-06-11T19:11:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5237fe640 10 monclient: dump: 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5397fa640 4 set_mon_vals no callback set 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 10 monclient: _renew_subs 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 1 -- 172.21.15.98:0/3393435857 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff53c125940 con 0x7ff53c0a2530 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 10 client.22706.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 10 monclient: _renew_subs 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 1 -- 172.21.15.98:0/3393435857 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7ff53c0a9380 con 0x7ff53c0a2530 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.824+0000 7ff5417fb640 1 librados: init done 2024-06-11T19:11:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff5417fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff5417fb640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c126a70 0x7ff53c123e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff5417fb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff5417fb640 1 -- 172.21.15.98:0/3393435857 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff53c050830 con 0x7ff53c126a70 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff52c01a070 con 0x7ff53c0a2530 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff53a7fc640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c126a70 0x7ff53c123e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff53a7fc640 10 monclient: get_auth_request con 0x7ff53c126a70 auth_method 0 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff53a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff53a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff53a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff5237fe640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff52803fdb0 0x7ff528042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_done global_id 21458 payload 824 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.825+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(71..71 src has 1..71) ==== 8794+0+0 (secure 0 0 0) 0x7ff52c00f040 con 0x7ff53c0a2530 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff53a7fc640 1 -- 172.21.15.98:0/3393435857 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff53c0a4d00 con 0x7ff53c126a70 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 10 client.22706.objecter ms_dispatch 0x7ff53c000b90 osd_map(71..71 src has 1..71) 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 3 client.22706.objecter handle_osd_map got epochs [71,71] > 0 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 3 client.22706.objecter handle_osd_map decoding full epoch 71 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff53b7fe640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff52803fdb0 0x7ff528042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff53b7fe640 10 monclient: get_auth_request con 0x7ff52803fdb0 auth_method 0 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 20 client.22706.objecter dump_active .. 0 homeless 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff53a7fc640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c126a70 0x7ff53c123e60 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0abfb0 tx=0x7ff5300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 10 client.22706.objecter ms_handle_connect 0x7ff53c126a70 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 10 client.22706.objecter resend_mon_ops 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff530042030 con 0x7ff53c126a70 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 10 monclient: dump: 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff53b7fe640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff52803fdb0 0x7ff528042270 secure :-1 s=READY pgs=3968 cs=0 l=1 rev1=1 crypto rx=0x7ff53c138740 tx=0x7ff534003d10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 10 client.22706.objecter ms_handle_connect 0x7ff52803fdb0 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff530003c90 con 0x7ff53c126a70 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c126a70 msgr2=0x7ff53c123e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c126a70 0x7ff53c123e60 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0abfb0 tx=0x7ff5300046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c126a70 msgr2=0x7ff53c123e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.826+0000 7ff5237fe640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c126a70 0x7ff53c123e60 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5417fb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5417fb640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c02cb40 0x7ff53c02cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5417fb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5417fb640 1 -- 172.21.15.98:0/3393435857 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff53c0a8e80 con 0x7ff53c02cb40 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff53a7fc640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c02cb40 0x7ff53c02cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff53a7fc640 10 monclient: get_auth_request con 0x7ff53c02cb40 auth_method 0 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff53a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff53a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff53a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff53a7fc640 10 monclient(hunting): handle_auth_done global_id 21467 payload 824 2024-06-11T19:11:27.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff53a7fc640 1 -- 172.21.15.98:0/3393435857 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff53c0a0100 con 0x7ff53c02cb40 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 10 client.22706.objecter ms_handle_connect 0x7ff53c02cb40 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff53a7fc640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c02cb40 0x7ff53c02cf20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ff530000c00 tx=0x7ff530030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 10 client.22706.objecter resend_mon_ops 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff530042040 con 0x7ff53c02cb40 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 10 monclient: dump: 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.856 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7ff530003e40 con 0x7ff53c02cb40 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c02cb40 msgr2=0x7ff53c02cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c02cb40 0x7ff53c02cf20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ff530000c00 tx=0x7ff530030000 comp rx=0 tx=0).stop 2024-06-11T19:11:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 1 -- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c02cb40 msgr2=0x7ff53c02cf20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:27.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.833+0000 7ff5237fe640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff53c02cb40 0x7ff53c02cf20 secure :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7ff530000c00 tx=0x7ff530030000 comp rx=0 tx=0).stop 2024-06-11T19:11:27.858 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:27.858 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:27.858 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:27.858 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:27.858 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:27.858 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 138066, 2024-06-11T19:11:27.858 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:27.858 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:27.859 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:27.860 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:27.861 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:27.862 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:27.868 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.871 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:27.871 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:27.871 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.871 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.871 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:27.871 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:27.873 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.874 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.874 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.874 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:27.874 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:27.874 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.874 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.874 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.874 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:27.874 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.876 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.877 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.879 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:27.880 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:27.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:27.881 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:27.881 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:27.881 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:27.881 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:27.881 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:27.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:27.881 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:27.881 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:27.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 10 librados: watch_flush enter 2024-06-11T19:11:27.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 10 librados: watch_flush exit 2024-06-11T19:11:27.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 20 client.22706.objecter shutdown clearing up homeless session... 2024-06-11T19:11:27.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 10 client.22706.objecter successfully canceled tick 2024-06-11T19:11:27.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 1 -- 172.21.15.98:0/3393435857 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff52803fdb0 msgr2=0x7ff528042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff52803fdb0 0x7ff528042270 secure :-1 s=READY pgs=3968 cs=0 l=1 rev1=1 crypto rx=0x7ff53c138740 tx=0x7ff534003d10 comp rx=0 tx=0).stop 2024-06-11T19:11:27.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 10 monclient: shutdown 2024-06-11T19:11:27.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:27.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 1 -- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff53c0a2530 msgr2=0x7ff53c1376f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 1 --2- 172.21.15.98:0/3393435857 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff53c0a2530 0x7ff53c1376f0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00ea30 tx=0x7ff52c00eef0 comp rx=0 tx=0).stop 2024-06-11T19:11:27.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff53b7fe640 1 -- 172.21.15.98:0/3393435857 reap_dead start 2024-06-11T19:11:27.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff53b7fe640 1 -- 172.21.15.98:0/3393435857 reap_dead start 2024-06-11T19:11:27.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 1 -- 172.21.15.98:0/3393435857 shutdown_connections 2024-06-11T19:11:27.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 1 -- 172.21.15.98:0/3393435857 >> 172.21.15.98:0/3393435857 conn(0x7ff53c0ab850 msgr2=0x7ff53c0726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:27.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 1 -- 172.21.15.98:0/3393435857 shutdown_connections 2024-06-11T19:11:27.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 1 -- 172.21.15.98:0/3393435857 wait complete. 2024-06-11T19:11:27.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.834+0000 7ff5417fb640 1 librados: shutdown 2024-06-11T19:11:27.884 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:27.884 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:11:27.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:11:27.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.985+0000 7f072455f640 10 monclient: get_monmap_and_config 2024-06-11T19:11:27.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.985+0000 7f072455f640 10 monclient: build_initial_monmap 2024-06-11T19:11:27.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.985+0000 7f072455f640 10 monclient: monmap: 2024-06-11T19:11:27.985 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:27.985 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.985 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:27.986355+0000 2024-06-11T19:11:27.985 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:27.986355+0000 2024-06-11T19:11:27.985 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:27.985 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.985 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:27.985 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:27.986 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:27.986 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:27.986 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:27.986 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:27.986 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:27.986 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:27.986 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:27.986 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:27.986 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:27.986 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:27.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.985+0000 7f072455f640 1 Processor -- start 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.985+0000 7f072455f640 1 -- start start 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.985+0000 7f072455f640 10 monclient: init 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f072455f640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f072455f640 10 monclient: _add_conns ranks=[1,0,8,3,2,7,9,5,4,6,10] 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f072455f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f071c0a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f072455f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f071c0a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f072455f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f071c13f380 0x7f071c141770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f072455f640 10 monclient(hunting): picked mon.noname-e con 0x7f071c13f380 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f072455f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f071c0a7140 con 0x7f071c0a2530 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f072455f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f071c0a2b50 con 0x7f071c0a4920 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f072455f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f072455f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f071c0a2e30 con 0x7f071c13f380 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f072455f640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:27.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f072455f640 10 monclient(hunting): authenticate will time out at 897.793684s 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f0721afa640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f071c0a2530 0x7f071c13dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52844/0 (socket says 172.21.15.98:52844) 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f0722afc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f071c13f380 0x7f071c141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f0721afa640 1 -- 172.21.15.98:0/4072791598 learned_addr learned my addr 172.21.15.98:0/4072791598 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f0722afc640 10 monclient(hunting): get_auth_request con 0x7f071c13f380 auth_method 0 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f0722afc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f0722afc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f0722afc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f0722afc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f0722afc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.986+0000 7f0722afc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f0722afc640 10 monclient(hunting): handle_auth_done global_id 22715 payload 274 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 1 -- 172.21.15.98:0/4072791598 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1487558118 0 0) 0x7f071c0a7140 con 0x7f071c0a2530 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f0722afc640 1 -- 172.21.15.98:0/4072791598 >> v1:172.21.15.117:6789/0 conn(0x7f071c0a4920 legacy=0x7f071c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:27.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f0722afc640 1 -- 172.21.15.98:0/4072791598 >> v1:172.21.15.98:6789/0 conn(0x7f071c0a2530 legacy=0x7f071c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f0722afc640 10 monclient: _finish_hunting 0 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f0722afc640 1 monclient: found mon.noname-e 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f0722afc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f0722afc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f0722afc640 1 -- 172.21.15.98:0/4072791598 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071c0a3110 con 0x7f071c13f380 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f0722afc640 1 --2- 172.21.15.98:0/4072791598 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f071c13f380 0x7f071c141770 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f0714009dc0 tx=0x7f071402bd50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=d213a4be46fe828b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 1 -- 172.21.15.98:0/4072791598 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0714039070 con 0x7f071c13f380 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 10 monclient: dump: 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:27.998 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:27.999 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 10 monclient: _finish_auth 0 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 10 monclient: _check_auth_tickets 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 1 -- 172.21.15.98:0/4072791598 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f071402cd70 con 0x7f071c13f380 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f072455f640 5 monclient: authenticate success, global_id 22715 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f072455f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f072455f640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 1 -- 172.21.15.98:0/4072791598 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07140327b0 con 0x7f071c13f380 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f072455f640 4 set_mon_vals no callback set 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.987+0000 7f07212f9640 10 monclient: dump: 2024-06-11T19:11:28.000 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:28.001 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.988+0000 7f072455f640 10 monclient: shutdown 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.988+0000 7f072455f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.988+0000 7f072455f640 1 -- 172.21.15.98:0/4072791598 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f071c13f380 msgr2=0x7f071c141770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.988+0000 7f072455f640 1 --2- 172.21.15.98:0/4072791598 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f071c13f380 0x7f071c141770 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f0714009dc0 tx=0x7f071402bd50 comp rx=0 tx=0).stop 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.988+0000 7f072455f640 1 -- 172.21.15.98:0/4072791598 shutdown_connections 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.988+0000 7f072455f640 1 --2- 172.21.15.98:0/4072791598 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f071c13f380 0x7f071c141770 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.988+0000 7f072455f640 1 -- 172.21.15.98:0/4072791598 >> 172.21.15.98:0/4072791598 conn(0x7f071c0ab850 msgr2=0x7f071c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.988+0000 7f072455f640 1 -- 172.21.15.98:0/4072791598 shutdown_connections 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.988+0000 7f072455f640 1 -- 172.21.15.98:0/4072791598 wait complete. 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient: build_initial_monmap 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient: monmap: 2024-06-11T19:11:28.002 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:28.003 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 1 librados: starting msgr at 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 1 librados: starting objecter 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 1 Processor -- start 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 1 -- start start 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 1 librados: setting wanted keys 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 1 librados: calling monclient init 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient: init 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient: _add_conns ranks=[0,8,7,2,4,6,3,1,5] 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f071c0a2530 0x7f071c1cd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient(hunting): picked mon.noname-a con 0x7f071c0a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f071c0a4920 0x7f071c135870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient(hunting): picked mon.noname-e con 0x7f071c0a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f071c13f380 0x7f071c09d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient(hunting): picked mon.noname-i con 0x7f071c13f380 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f071c138e50 con 0x7f071c0a2530 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f071c0a84b0 con 0x7f071c13f380 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f071c0a6da0 con 0x7f071c0a4920 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f072455f640 10 monclient(hunting): authenticate will time out at 897.796846s 2024-06-11T19:11:28.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f07222fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f071c0a2530 0x7f071c1cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f0721afa640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f071c0a4920 0x7f071c135870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f07222fb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f071c0a2530 0x7f071c1cd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53710/0 (socket says 172.21.15.98:53710) 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f07222fb640 1 -- 172.21.15.98:0/2424087275 learned_addr learned my addr 172.21.15.98:0/2424087275 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f07222fb640 10 monclient(hunting): get_auth_request con 0x7f071c0a2530 auth_method 0 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f07222fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f07222fb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f07222fb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f0721afa640 10 monclient(hunting): get_auth_request con 0x7f071c0a4920 auth_method 0 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f0721afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f0721afa640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.989+0000 7f0721afa640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f07222fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:28.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f07222fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f07222fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0722afc640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f071c13f380 0x7f071c09d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0722afc640 10 monclient(hunting): get_auth_request con 0x7f071c13f380 auth_method 0 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0722afc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0722afc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0722afc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 10 monclient(hunting): handle_auth_done global_id 22724 payload 824 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 1 -- 172.21.15.98:0/2424087275 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f071c13f380 msgr2=0x7f071c09d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f071c13f380 0x7f071c09d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 1 -- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f071c0a2530 msgr2=0x7f071c1cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f071c0a2530 0x7f071c1cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 10 monclient: _finish_hunting 0 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 1 monclient: found mon.noname-e 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 1 -- 172.21.15.98:0/2424087275 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071c13f130 con 0x7f071c0a4920 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f07222fb640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f071c0a2530 0x7f071c1cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0722afc640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f071c13f380 0x7f071c09d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0721afa640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f071c0a4920 0x7f071c135870 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f071800c410 tx=0x7f071800c8d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 10 client.?.objecter ms_handle_connect 0x7f071c0a4920 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07180047e0 con 0x7f071c0a4920 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:28.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 10 monclient: dump: 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:28.009 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 10 monclient: _finish_auth 0 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07180049e0 con 0x7f071c0a4920 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f072455f640 5 monclient: authenticate success, global_id 22724 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f072455f640 10 monclient: _renew_subs 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f072455f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f072455f640 1 -- 172.21.15.98:0/2424087275 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071c0a1550 con 0x7f071c0a4920 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:28.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07180133c0 con 0x7f071c0a4920 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f070effd640 10 monclient: dump: 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:28.011 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f0720af8640 4 set_mon_vals no callback set 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f072455f640 10 client.22724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f072455f640 10 monclient: _renew_subs 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f072455f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f072455f640 1 -- 172.21.15.98:0/2424087275 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f071c0a00a0 con 0x7f071c0a4920 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.990+0000 7f072455f640 1 librados: init done 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f0718011070 con 0x7f071c0a4920 2024-06-11T19:11:28.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f072455f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f072455f640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c135db0 0x7f071c09de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f072455f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f072455f640 1 -- 172.21.15.98:0/2424087275 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f071c13eb70 con 0x7f071c135db0 2024-06-11T19:11:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f07222fb640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c135db0 0x7f071c09de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f07222fb640 10 monclient: get_auth_request con 0x7f071c135db0 auth_method 0 2024-06-11T19:11:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f07222fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f07222fb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f07222fb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f070effd640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f070003fdb0 0x7f0700042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f07222fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f07222fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:28.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f07222fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(71..71 src has 1..71) ==== 8794+0+0 (secure 0 0 0) 0x7f0718051fa0 con 0x7f071c0a4920 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f070effd640 10 client.22724.objecter ms_dispatch 0x7f071c000b90 osd_map(71..71 src has 1..71) 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f070effd640 3 client.22724.objecter handle_osd_map got epochs [71,71] > 0 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f070effd640 3 client.22724.objecter handle_osd_map decoding full epoch 71 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f070effd640 20 client.22724.objecter dump_active .. 0 homeless 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f07222fb640 10 monclient(hunting): handle_auth_done global_id 22711 payload 824 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f07222fb640 1 -- 172.21.15.98:0/2424087275 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f071c0733b0 con 0x7f071c135db0 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.991+0000 7f0722afc640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f070003fdb0 0x7f0700042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f0722afc640 10 monclient: get_auth_request con 0x7f070003fdb0 auth_method 0 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f07222fb640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c135db0 0x7f071c09de50 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f071000fb10 tx=0x7f0710004580 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 10 client.22724.objecter ms_handle_connect 0x7f071c135db0 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 10 client.22724.objecter resend_mon_ops 2024-06-11T19:11:28.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0710018020 con 0x7f071c135db0 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 10 monclient: dump: 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:28.015 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 10 client.22724.objecter ms_handle_connect 0x7f070003fdb0 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f0722afc640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f070003fdb0 0x7f0700042270 secure :-1 s=READY pgs=3969 cs=0 l=1 rev1=1 crypto rx=0x7f0714036000 tx=0x7f0714030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f071000db60 con 0x7f071c135db0 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c135db0 msgr2=0x7f071c09de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c135db0 0x7f071c09de50 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f071000fb10 tx=0x7f0710004580 comp rx=0 tx=0).stop 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c135db0 msgr2=0x7f071c09de50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:28.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.992+0000 7f070effd640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c135db0 0x7f071c09de50 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.998+0000 7f072455f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:28.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.998+0000 7f072455f640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c09e390 0x7f071c139bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f072455f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f072455f640 1 -- 172.21.15.98:0/2424087275 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f071c050830 con 0x7f071c09e390 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f07222fb640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c09e390 0x7f071c139bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f07222fb640 10 monclient: get_auth_request con 0x7f071c09e390 auth_method 0 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f07222fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f07222fb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f07222fb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f07222fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f07222fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f07222fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f07222fb640 10 monclient(hunting): handle_auth_done global_id 22720 payload 824 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f07222fb640 1 -- 172.21.15.98:0/2424087275 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f071c09e7a0 con 0x7f071c09e390 2024-06-11T19:11:28.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 10 client.22724.objecter ms_handle_connect 0x7f071c09e390 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 10 client.22724.objecter resend_mon_ops 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f07222fb640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c09e390 0x7f071c139bb0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f0710019000 tx=0x7f0710003760 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0710018040 con 0x7f071c09e390 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 10 monclient: dump: 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:28.020 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f0710014da0 con 0x7f071c09e390 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c09e390 msgr2=0x7f071c139bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:28.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c09e390 0x7f071c139bb0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f0710019000 tx=0x7f0710003760 comp rx=0 tx=0).stop 2024-06-11T19:11:28.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 1 -- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c09e390 msgr2=0x7f071c139bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:28.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:27.999+0000 7f070effd640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f071c09e390 0x7f071c139bb0 secure :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f0710019000 tx=0x7f0710003760 comp rx=0 tx=0).stop 2024-06-11T19:11:28.022 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 138217, 2024-06-11T19:11:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:28.044 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:28.048 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:28.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:28.048 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:28.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:28.048 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:28.048 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 10 librados: watch_flush enter 2024-06-11T19:11:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 10 librados: watch_flush exit 2024-06-11T19:11:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 20 client.22724.objecter shutdown clearing up homeless session... 2024-06-11T19:11:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 10 client.22724.objecter successfully canceled tick 2024-06-11T19:11:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 1 -- 172.21.15.98:0/2424087275 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f070003fdb0 msgr2=0x7f0700042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f070003fdb0 0x7f0700042270 secure :-1 s=READY pgs=3969 cs=0 l=1 rev1=1 crypto rx=0x7f0714036000 tx=0x7f0714030000 comp rx=0 tx=0).stop 2024-06-11T19:11:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 10 monclient: shutdown 2024-06-11T19:11:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 1 -- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f071c0a4920 msgr2=0x7f071c135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f071c0a4920 0x7f071c135870 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f071800c410 tx=0x7f071800c8d0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f0722afc640 1 -- 172.21.15.98:0/2424087275 reap_dead start 2024-06-11T19:11:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 1 -- 172.21.15.98:0/2424087275 shutdown_connections 2024-06-11T19:11:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 1 --2- 172.21.15.98:0/2424087275 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f071c0a4920 0x7f071c135870 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 1 -- 172.21.15.98:0/2424087275 >> 172.21.15.98:0/2424087275 conn(0x7f071c0ab850 msgr2=0x7f071c0a8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 1 -- 172.21.15.98:0/2424087275 shutdown_connections 2024-06-11T19:11:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 1 -- 172.21.15.98:0/2424087275 wait complete. 2024-06-11T19:11:28.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.000+0000 7f072455f640 1 librados: shutdown 2024-06-11T19:11:28.051 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:28.051 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:11:28.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:11:28.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.151+0000 7f857a351640 10 monclient: get_monmap_and_config 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.151+0000 7f857a351640 10 monclient: build_initial_monmap 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.151+0000 7f857a351640 10 monclient: monmap: 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:28.152218+0000 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:28.152218+0000 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:28.151 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:28.152 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:28.152 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:28.152 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:28.152 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:28.152 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:28.152 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:28.152 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 1 Processor -- start 2024-06-11T19:11:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 1 -- start start 2024-06-11T19:11:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 10 monclient: init 2024-06-11T19:11:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 10 monclient: _add_conns ranks=[7,10,5,2,9,1,0,3,6,8,4] 2024-06-11T19:11:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85740a4920 0x7f85740a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 10 monclient(hunting): picked mon.noname-i con 0x7f85740a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f85740a5240 0x7f85740a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 10 monclient(hunting): picked mon.noname-f con 0x7f85740a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f85740a2530 0x7f85740a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 10 monclient(hunting): picked mon.noname-h con 0x7f85740a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f85740a7140 con 0x7f85740a5240 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f85740a2b50 con 0x7f85740a2530 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f85740a2e30 con 0x7f85740a4920 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f857a351640 10 monclient(hunting): authenticate will time out at 897.959636s 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f8573fff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85740a4920 0x7f85740a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f85788ee640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f85740a2530 0x7f85740a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f85737fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f85740a5240 0x7f85740a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f85788ee640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f85740a2530 0x7f85740a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:56792/0 (socket says 172.21.15.98:56792) 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f85788ee640 1 -- 172.21.15.98:0/1672643877 learned_addr learned my addr 172.21.15.98:0/1672643877 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f85788ee640 10 monclient(hunting): get_auth_request con 0x7f85740a2530 auth_method 0 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f85788ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f85788ee640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f85788ee640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f8573fff640 10 monclient(hunting): get_auth_request con 0x7f85740a4920 auth_method 0 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f8573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f8573fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f8573fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f85737fe640 10 monclient(hunting): get_auth_request con 0x7f85740a5240 auth_method 0 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f85737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f85737fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.152+0000 7f85737fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85788ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85788ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85788ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 10 monclient(hunting): handle_auth_done global_id 30655 payload 274 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 1 -- 172.21.15.98:0/1672643877 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85740a4920 msgr2=0x7f85740a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 1 --2- 172.21.15.98:0/1672643877 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85740a4920 0x7f85740a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 1 -- 172.21.15.98:0/1672643877 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f85740a2530 msgr2=0x7f85740a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 1 --2- 172.21.15.98:0/1672643877 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f85740a2530 0x7f85740a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 10 monclient: _finish_hunting 0 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 1 monclient: found mon.noname-f 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 1 -- 172.21.15.98:0/1672643877 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85740a3110 con 0x7f85740a5240 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85788ee640 1 --2- 172.21.15.98:0/1672643877 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f85740a2530 0x7f85740a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f8573fff640 1 --2- 172.21.15.98:0/1672643877 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85740a4920 0x7f85740a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.153+0000 7f85737fe640 1 --2- 172.21.15.98:0/1672643877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85740a5240 0x7f85740a59a0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f85680097a0 tx=0x7f856802b6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e3f1dcc4a804e1 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 1 -- 172.21.15.98:0/1672643877 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8568037070 con 0x7f85740a5240 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 10 monclient: dump: 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:28.166 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:28.167 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 10 monclient: _finish_auth 0 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 1 -- 172.21.15.98:0/1672643877 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8568030980 con 0x7f85740a5240 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 5 monclient: authenticate success, global_id 30655 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 1 -- 172.21.15.98:0/1672643877 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8568030bb0 con 0x7f85740a5240 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 4 set_mon_vals no callback set 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f8572ffd640 10 monclient: dump: 2024-06-11T19:11:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 10 monclient: shutdown 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 1 -- 172.21.15.98:0/1672643877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85740a5240 msgr2=0x7f85740a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 1 --2- 172.21.15.98:0/1672643877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85740a5240 0x7f85740a59a0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f85680097a0 tx=0x7f856802b6f0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 1 -- 172.21.15.98:0/1672643877 shutdown_connections 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 1 --2- 172.21.15.98:0/1672643877 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f85740a2530 0x7f85740a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 1 --2- 172.21.15.98:0/1672643877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85740a5240 0x7f85740a59a0 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 1 --2- 172.21.15.98:0/1672643877 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85740a4920 0x7f85740a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 1 -- 172.21.15.98:0/1672643877 >> 172.21.15.98:0/1672643877 conn(0x7f85740ab850 msgr2=0x7f85740a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 1 -- 172.21.15.98:0/1672643877 shutdown_connections 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.154+0000 7f857a351640 1 -- 172.21.15.98:0/1672643877 wait complete. 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.155+0000 7f857a351640 10 monclient: build_initial_monmap 2024-06-11T19:11:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.155+0000 7f857a351640 10 monclient: monmap: 2024-06-11T19:11:28.171 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:28.171 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:28.171 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:28.171 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:28.171 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:28.171 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:28.171 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:28.171 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:28.171 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:28.171 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:28.171 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:28.171 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.155+0000 7f857a351640 1 librados: starting msgr at 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.155+0000 7f857a351640 1 librados: starting objecter 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.155+0000 7f857a351640 1 Processor -- start 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.155+0000 7f857a351640 1 -- start start 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.155+0000 7f857a351640 1 librados: setting wanted keys 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.155+0000 7f857a351640 1 librados: calling monclient init 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.155+0000 7f857a351640 10 monclient: init 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 10 monclient: _add_conns ranks=[4,0,7,3,6,8,1,2,5] 2024-06-11T19:11:28.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f85740a2530 0x7f85741cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 10 monclient(hunting): picked mon.noname-c con 0x7f85740a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85740a4920 0x7f85741ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 10 monclient(hunting): picked mon.noname-a con 0x7f85740a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85740a5240 0x7f8574122e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 10 monclient(hunting): picked mon.noname-i con 0x7f85740a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85740a6da0 con 0x7f85740a4920 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f8574138e50 con 0x7f85740a2530 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f85740a6420 con 0x7f85740a5240 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f857a351640 10 monclient(hunting): authenticate will time out at 897.963361s 2024-06-11T19:11:28.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f8573fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f85740a2530 0x7f85741cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85740a4920 0x7f85741ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f8573fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f85740a2530 0x7f85741cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46396/0 (socket says 172.21.15.98:46396) 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 learned_addr learned my addr 172.21.15.98:0/1992140755 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f8573fff640 10 monclient(hunting): get_auth_request con 0x7f85740a2530 auth_method 0 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f8573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f8573fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f8573fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 10 monclient(hunting): get_auth_request con 0x7f85740a4920 auth_method 0 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:28.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85788ee640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85740a5240 0x7f8574122e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85788ee640 10 monclient(hunting): get_auth_request con 0x7f85740a5240 auth_method 0 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85788ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85788ee640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85788ee640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85788ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85788ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85788ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 10 monclient(hunting): handle_auth_done global_id 30807 payload 824 2024-06-11T19:11:28.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85740a5240 msgr2=0x7f8574122e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85740a5240 0x7f8574122e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f85740a2530 msgr2=0x7f85741cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f85740a2530 0x7f85741cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 10 monclient: _finish_hunting 0 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 1 monclient: found mon.noname-a 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 1 -- 172.21.15.98:0/1992140755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8574126af0 con 0x7f85740a4920 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f85740a2530 0x7f85741cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f85737fe640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85740a4920 0x7f85741ce080 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f8568032000 tx=0x7f85680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.156+0000 7f8570ff9640 10 client.?.objecter ms_handle_connect 0x7f85740a4920 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:28.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8568037050 con 0x7f85740a4920 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 10 monclient: dump: 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:28.177 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 10 monclient: _finish_auth 0 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8568038040 con 0x7f85740a4920 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f857a351640 5 monclient: authenticate success, global_id 30807 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8568004310 con 0x7f85740a4920 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:28.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8571ffb640 4 set_mon_vals no callback set 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f8570ff9640 10 monclient: dump: 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f857a351640 10 monclient: _renew_subs 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f857a351640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f857a351640 1 -- 172.21.15.98:0/1992140755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8574125640 con 0x7f85740a4920 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f857a351640 10 client.30807.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f857a351640 10 monclient: _renew_subs 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f857a351640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f857a351640 1 -- 172.21.15.98:0/1992140755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85740a9240 con 0x7f85740a4920 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.157+0000 7f857a351640 1 librados: init done 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f857a351640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:28.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f857a351640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f857a351640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f857a351640 1 -- 172.21.15.98:0/1992140755 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8574050830 con 0x7f8574123390 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8568045030 con 0x7f85740a4920 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8573fff640 10 monclient: get_auth_request con 0x7f8574123390 auth_method 0 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8573fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8573fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 msgr2=0x7f8574123770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 msgr2=0x7f8574123770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8570ff9640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f856003fdb0 0x7f8560042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 8794+0+0 (secure 0 0 0) 0x7f8568043020 con 0x7f85740a4920 2024-06-11T19:11:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8570ff9640 10 client.30807.objecter ms_dispatch 0x7f8574000b90 osd_map(71..71 src has 1..71) 2024-06-11T19:11:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8570ff9640 3 client.30807.objecter handle_osd_map got epochs [71,71] > 0 2024-06-11T19:11:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8570ff9640 3 client.30807.objecter handle_osd_map decoding full epoch 71 2024-06-11T19:11:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f8570ff9640 20 client.30807.objecter dump_active .. 0 homeless 2024-06-11T19:11:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f85788ee640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f856003fdb0 0x7f8560042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.158+0000 7f85788ee640 10 monclient: get_auth_request con 0x7f856003fdb0 auth_method 0 2024-06-11T19:11:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.159+0000 7f8570ff9640 10 client.30807.objecter ms_handle_connect 0x7f856003fdb0 2024-06-11T19:11:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.159+0000 7f85788ee640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f856003fdb0 0x7f8560042270 secure :-1 s=READY pgs=3970 cs=0 l=1 rev1=1 crypto rx=0x7f856c009c90 tx=0x7f856c00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:28.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.359+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.359+0000 7f8573fff640 10 monclient: get_auth_request con 0x7f8574123390 auth_method 0 2024-06-11T19:11:28.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.359+0000 7f8573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.359+0000 7f8573fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.359+0000 7f8573fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:28.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.359+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 msgr2=0x7f8574123770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:11:28.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.359+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 msgr2=0x7f8574123770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:11:28.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.359+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:28.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.359+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:11:28.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.760+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:28.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.760+0000 7f8573fff640 10 monclient: get_auth_request con 0x7f8574123390 auth_method 0 2024-06-11T19:11:28.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.760+0000 7f8573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:28.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.760+0000 7f8573fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:28.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.760+0000 7f8573fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:28.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.760+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 msgr2=0x7f8574123770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:11:28.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.760+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 msgr2=0x7f8574123770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:11:28.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.760+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:28.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:28.760+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:11:29.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:29.561+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:29.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:29.561+0000 7f8573fff640 10 monclient: get_auth_request con 0x7f8574123390 auth_method 0 2024-06-11T19:11:29.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:29.561+0000 7f8573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:29.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:29.561+0000 7f8573fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:29.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:29.561+0000 7f8573fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:29.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:29.561+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 msgr2=0x7f8574123770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:11:29.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:29.561+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 msgr2=0x7f8574123770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:11:29.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:29.561+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:29.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:29.561+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:11:30.341 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: Running main() from gmock_main.cc 2024-06-11T19:11:30.342 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-06-11T19:11:30.342 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] Global test environment set-up. 2024-06-11T19:11:30.342 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-06-11T19:11:30.342 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-06-11T19:11:30.342 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2026 ms) 2024-06-11T19:11:30.342 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-06-11T19:11:30.342 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2030 ms) 2024-06-11T19:11:30.342 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-06-11T19:11:30.342 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2013 ms) 2024-06-11T19:11:30.342 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-06-11T19:11:30.342 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2004 ms) 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8073 ms total) 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4054 ms) 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (3998 ms) 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (5000 ms) 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (13052 ms total) 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-06-11T19:11:30.343 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2956 ms) 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2008 ms) 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2011 ms) 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2005 ms) 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8980 ms total) 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4019 ms) 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4001 ms) 2024-06-11T19:11:30.344 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8020 ms total) 2024-06-11T19:11:30.345 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: 2024-06-11T19:11:30.345 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [----------] Global test environment tear-down 2024-06-11T19:11:30.345 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (56180 ms total) 2024-06-11T19:11:30.345 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-06-11T19:11:31.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.156+0000 7f855b7fe640 10 monclient: tick 2024-06-11T19:11:31.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.156+0000 7f855b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:11:31.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.156+0000 7f855b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:31.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.156+0000 7f855b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:31.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.163+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:31.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.163+0000 7f8573fff640 10 monclient: get_auth_request con 0x7f8574123390 auth_method 0 2024-06-11T19:11:31.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.163+0000 7f8573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:31.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.163+0000 7f8573fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:31.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.163+0000 7f8573fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:31.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.163+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 msgr2=0x7f8574123770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:11:31.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.163+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 msgr2=0x7f8574123770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:11:31.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.163+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:31.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:31.163+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-06-11T19:11:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.156+0000 7f855b7fe640 10 monclient: tick 2024-06-11T19:11:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.156+0000 7f855b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:11:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.156+0000 7f855b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.156+0000 7f855b7fe640 5 monclient: _check_tell_commands timeout tell command 1 2024-06-11T19:11:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.156+0000 7f855b7fe640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 msgr2=0x7f8574123770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:11:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.156+0000 7f855b7fe640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.156+0000 7f855b7fe640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f854c003540 0x7f854c0059d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.156+0000 7f855b7fe640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 msgr2=0x7f8574123770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.156+0000 7f855b7fe640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123390 0x7f8574123770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.156+0000 7f855b7fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.156+0000 7f855b7fe640 1 -- 172.21.15.98:0/1992140755 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f854c006080 con 0x7f854c003540 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.157+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f854c003540 0x7f854c0059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.157+0000 7f8573fff640 10 monclient: get_auth_request con 0x7f854c003540 auth_method 0 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.157+0000 7f8573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.157+0000 7f8573fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.157+0000 7f8573fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.157+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f854c003540 msgr2=0x7f854c0059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.157+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f854c003540 msgr2=0x7f854c0059d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.157+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f854c003540 0x7f854c0059d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.157+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f854c003540 0x7f854c0059d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:32.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.358+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f854c003540 0x7f854c0059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.358+0000 7f8573fff640 10 monclient: get_auth_request con 0x7f854c003540 auth_method 0 2024-06-11T19:11:32.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.358+0000 7f8573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.358+0000 7f8573fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.358+0000 7f8573fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:32.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.358+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.358+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.358+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.358+0000 7f8573fff640 10 monclient(hunting): handle_auth_done global_id 54105 payload 824 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.358+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f854c006230 con 0x7f854c003540 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f854c003540 0x7f854c0059d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8564009110 tx=0x7f85640168c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 10 client.30807.objecter ms_handle_connect 0x7f854c003540 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 10 client.30807.objecter resend_mon_ops 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8564027070 con 0x7f854c003540 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 10 monclient: dump: 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.360 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.361 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.361 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.361 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.361 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.361 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.361 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.361 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.361 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.361 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.361 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.361 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8564015690 con 0x7f854c003540 2024-06-11T19:11:32.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:32.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:32.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f854c003540 msgr2=0x7f854c0059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f854c003540 0x7f854c0059d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8564009110 tx=0x7f85640168c0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f854c003540 msgr2=0x7f854c0059d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:32.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.359+0000 7f8570ff9640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f854c003540 0x7f854c0059d0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8564009110 tx=0x7f85640168c0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.368+0000 7f857a351640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.368+0000 7f857a351640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123cb0 0x7f857409e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.368+0000 7f857a351640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.368+0000 7f857a351640 1 -- 172.21.15.98:0/1992140755 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f85740a6750 con 0x7f8574123cb0 2024-06-11T19:11:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123cb0 0x7f857409e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8573fff640 10 monclient: get_auth_request con 0x7f8574123cb0 auth_method 0 2024-06-11T19:11:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8573fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8573fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8573fff640 10 monclient(hunting): handle_auth_done global_id 54114 payload 824 2024-06-11T19:11:32.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8573fff640 1 -- 172.21.15.98:0/1992140755 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8574069690 con 0x7f8574123cb0 2024-06-11T19:11:32.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8570ff9640 10 client.30807.objecter ms_handle_connect 0x7f8574123cb0 2024-06-11T19:11:32.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8573fff640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123cb0 0x7f857409e9f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8564009110 tx=0x7f85640046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.369+0000 7f8570ff9640 10 client.30807.objecter resend_mon_ops 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8564027050 con 0x7f8574123cb0 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f8570ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f8570ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f8570ff9640 10 monclient: dump: 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9271 (secure 0 0 0) 0x7f8564028040 con 0x7f8574123cb0 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f8570ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f8570ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123cb0 msgr2=0x7f857409e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f8570ff9640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123cb0 0x7f857409e9f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8564009110 tx=0x7f85640046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f8570ff9640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123cb0 msgr2=0x7f857409e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:32.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f85788ee640 1 -- 172.21.15.98:0/1992140755 reap_dead start 2024-06-11T19:11:32.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f8570ff9640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8574123cb0 0x7f857409e9f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.377 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 5131, 2024-06-11T19:11:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:32.377 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:11:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:32.378 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:32.379 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:32.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:32.382 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:32.384 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:32.387 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:32.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.391 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:32.396 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:32.398 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.399 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:32.399 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:32.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f857a351640 10 librados: watch_flush enter 2024-06-11T19:11:32.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f857a351640 10 librados: watch_flush exit 2024-06-11T19:11:32.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f857a351640 20 client.30807.objecter shutdown clearing up homeless session... 2024-06-11T19:11:32.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f857a351640 10 client.30807.objecter successfully canceled tick 2024-06-11T19:11:32.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f857a351640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f856003fdb0 msgr2=0x7f8560042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f857a351640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f856003fdb0 0x7f8560042270 secure :-1 s=READY pgs=3970 cs=0 l=1 rev1=1 crypto rx=0x7f856c009c90 tx=0x7f856c00b040 comp rx=0 tx=0).stop 2024-06-11T19:11:32.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f857a351640 10 monclient: shutdown 2024-06-11T19:11:32.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f857a351640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:32.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f857a351640 1 -- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85740a4920 msgr2=0x7f85741ce080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.375+0000 7f857a351640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85740a4920 0x7f85741ce080 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f8568032000 tx=0x7f85680046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.376+0000 7f857a351640 1 -- 172.21.15.98:0/1992140755 shutdown_connections 2024-06-11T19:11:32.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.376+0000 7f857a351640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f856003fdb0 0x7f8560042270 unknown :-1 s=CLOSED pgs=3970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.376+0000 7f857a351640 1 --2- 172.21.15.98:0/1992140755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85740a4920 0x7f85741ce080 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.376+0000 7f857a351640 1 -- 172.21.15.98:0/1992140755 >> 172.21.15.98:0/1992140755 conn(0x7f85740ab850 msgr2=0x7f85741be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:32.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.376+0000 7f857a351640 1 -- 172.21.15.98:0/1992140755 shutdown_connections 2024-06-11T19:11:32.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.376+0000 7f857a351640 1 -- 172.21.15.98:0/1992140755 wait complete. 2024-06-11T19:11:32.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.376+0000 7f857a351640 1 librados: shutdown 2024-06-11T19:11:32.403 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:32.403 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:11:32.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:11:32.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.503+0000 7fbcc1255640 10 monclient: get_monmap_and_config 2024-06-11T19:11:32.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.503+0000 7fbcc1255640 10 monclient: build_initial_monmap 2024-06-11T19:11:32.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.503+0000 7fbcc1255640 10 monclient: monmap: 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:32.504626+0000 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:32.504626+0000 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:32.505 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 1 Processor -- start 2024-06-11T19:11:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 1 -- start start 2024-06-11T19:11:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 10 monclient: init 2024-06-11T19:11:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 10 monclient: _add_conns ranks=[5,9,6,3,10,1,8,0,7,4,2] 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbcbc0a4920 0x7fbcbc0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 10 monclient(hunting): picked mon.noname-h con 0x7fbcbc0a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbcbc0a5240 0x7fbcbc0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 10 monclient(hunting): picked mon.noname-a con 0x7fbcbc0a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbcbc0a2530 0x7fbcbc0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 10 monclient(hunting): picked mon.noname-d con 0x7fbcbc0a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbcbc0a7140 con 0x7fbcbc0a5240 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fbcbc0a2b50 con 0x7fbcbc0a4920 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fbcbc0a2e30 con 0x7fbcbc0a2530 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcc1255640 10 monclient(hunting): authenticate will time out at 902.311792s 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcbb59e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbcbc0a2530 0x7fbcbc0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcb259c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbcbc0a5240 0x7fbcbc0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcb259c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbcbc0a5240 0x7fbcbc0a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53744/0 (socket says 172.21.15.98:53744) 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcbb59e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbcbc0a2530 0x7fbcbc0a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:41602/0 (socket says 172.21.15.98:41602) 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcb259c640 1 -- 172.21.15.98:0/195120526 learned_addr learned my addr 172.21.15.98:0/195120526 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcb259c640 10 monclient(hunting): get_auth_request con 0x7fbcbc0a5240 auth_method 0 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcb259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcb259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcbb59e640 10 monclient(hunting): get_auth_request con 0x7fbcbc0a2530 auth_method 0 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcbb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcbb59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.504+0000 7fbcbb59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcbb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcbb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcbb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcbad9d640 1 --2- 172.21.15.98:0/195120526 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbcbc0a4920 0x7fbcbc0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcbad9d640 10 monclient(hunting): get_auth_request con 0x7fbcbc0a4920 auth_method 0 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcbad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 10 monclient(hunting): handle_auth_done global_id 30879 payload 274 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 1 -- 172.21.15.98:0/195120526 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbcbc0a2530 msgr2=0x7fbcbc0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 1 --2- 172.21.15.98:0/195120526 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbcbc0a2530 0x7fbcbc0a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 1 -- 172.21.15.98:0/195120526 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbcbc0a4920 msgr2=0x7fbcbc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 1 --2- 172.21.15.98:0/195120526 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbcbc0a4920 0x7fbcbc0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 10 monclient: _finish_hunting 0 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 1 monclient: found mon.noname-a 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 1 -- 172.21.15.98:0/195120526 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcbc0a3110 con 0x7fbcbc0a5240 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcbb59e640 1 --2- 172.21.15.98:0/195120526 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbcbc0a2530 0x7fbcbc0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.505+0000 7fbcb259c640 1 --2- 172.21.15.98:0/195120526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcbc0a5240 0x7fbcbc0a59a0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009870 tx=0x7fbca402b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcdceaa323f15346 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 1 -- 172.21.15.98:0/195120526 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbca4038070 con 0x7fbcbc0a5240 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 10 monclient: dump: 2024-06-11T19:11:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 10 monclient: _finish_auth 0 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 10 monclient: _check_auth_tickets 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 5 monclient: authenticate success, global_id 30879 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 1 -- 172.21.15.98:0/195120526 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca4031990 con 0x7fbcbc0a5240 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 4 set_mon_vals no callback set 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 1 -- 172.21.15.98:0/195120526 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbca40323b0 con 0x7fbcbc0a5240 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcba59c640 10 monclient: dump: 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 10 monclient: shutdown 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 1 -- 172.21.15.98:0/195120526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcbc0a5240 msgr2=0x7fbcbc0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 1 --2- 172.21.15.98:0/195120526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcbc0a5240 0x7fbcbc0a59a0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009870 tx=0x7fbca402b7c0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 1 -- 172.21.15.98:0/195120526 shutdown_connections 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 1 --2- 172.21.15.98:0/195120526 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbcbc0a2530 0x7fbcbc0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 1 --2- 172.21.15.98:0/195120526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcbc0a5240 0x7fbcbc0a59a0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 1 --2- 172.21.15.98:0/195120526 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbcbc0a4920 0x7fbcbc0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 1 -- 172.21.15.98:0/195120526 >> 172.21.15.98:0/195120526 conn(0x7fbcbc0ab850 msgr2=0x7fbcbc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 1 -- 172.21.15.98:0/195120526 shutdown_connections 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.506+0000 7fbcc1255640 1 -- 172.21.15.98:0/195120526 wait complete. 2024-06-11T19:11:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.507+0000 7fbcc1255640 10 monclient: build_initial_monmap 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.507+0000 7fbcc1255640 10 monclient: monmap: 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.507+0000 7fbcc1255640 1 librados: starting msgr at 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.507+0000 7fbcc1255640 1 librados: starting objecter 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.507+0000 7fbcc1255640 1 Processor -- start 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.507+0000 7fbcc1255640 1 -- start start 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.507+0000 7fbcc1255640 1 librados: setting wanted keys 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.507+0000 7fbcc1255640 1 librados: calling monclient init 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.507+0000 7fbcc1255640 10 monclient: init 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 10 monclient: _add_conns ranks=[7,1,3,4,6,5,2,8,0] 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbcbc0a2530 0x7fbcbc1cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 10 monclient(hunting): picked mon.noname-i con 0x7fbcbc0a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbcbc0a4920 0x7fbcbc1cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 10 monclient(hunting): picked mon.noname-f con 0x7fbcbc0a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbcbc0a5240 0x7fbcbc1251d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 10 monclient(hunting): picked mon.noname-g con 0x7fbcbc0a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbcbc0a6da0 con 0x7fbcbc0a4920 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fbcbc138e50 con 0x7fbcbc0a5240 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fbcbc0a6420 con 0x7fbcbc0a2530 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcc1255640 10 monclient(hunting): authenticate will time out at 902.315379s 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbb59e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbcbc0a5240 0x7fbcbc1251d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbad9d640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbcbc0a2530 0x7fbcbc1cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcb259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbcbc0a4920 0x7fbcbc1cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbad9d640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbcbc0a2530 0x7fbcbc1cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:53440/0 (socket says 172.21.15.98:53440) 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbad9d640 1 -- 172.21.15.98:0/302125210 learned_addr learned my addr 172.21.15.98:0/302125210 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbb59e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbcbc0a5240 0x7fbcbc1251d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:44694/0 (socket says 172.21.15.98:44694) 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbad9d640 10 monclient(hunting): get_auth_request con 0x7fbcbc0a2530 auth_method 0 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbb59e640 10 monclient(hunting): get_auth_request con 0x7fbcbc0a5240 auth_method 0 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbb59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbb59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcb259c640 10 monclient(hunting): get_auth_request con 0x7fbcbc0a4920 auth_method 0 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcb259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcb259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcb259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcb259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcb259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcbb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcb259c640 10 monclient(hunting): handle_auth_done global_id 30691 payload 824 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcb259c640 1 -- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbcbc0a2530 msgr2=0x7fbcbc1cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.508+0000 7fbcb259c640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbcbc0a2530 0x7fbcbc1cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb259c640 1 -- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbcbc0a5240 msgr2=0x7fbcbc1251d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb259c640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbcbc0a5240 0x7fbcbc1251d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb259c640 10 monclient: _finish_hunting 0 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb259c640 1 monclient: found mon.noname-f 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb259c640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb259c640 1 -- 172.21.15.98:0/302125210 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcbc127130 con 0x7fbcbc0a4920 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcbad9d640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbcbc0a2530 0x7fbcbc1cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcbb59e640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbcbc0a5240 0x7fbcbc1251d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb259c640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbcbc0a4920 0x7fbcbc1cdfe0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7fbca4033000 tx=0x7fbca40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 10 client.?.objecter ms_handle_connect 0x7fbcbc0a4920 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbca4042020 con 0x7fbcbc0a4920 2024-06-11T19:11:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 10 monclient: dump: 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 10 monclient: _finish_auth 0 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 10 monclient: _check_auth_tickets 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca4003610 con 0x7fbcbc0a4920 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbca4038040 con 0x7fbcbc0a4920 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcc1255640 5 monclient: authenticate success, global_id 30691 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcc1255640 10 monclient: _renew_subs 2024-06-11T19:11:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcc1255640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcc1255640 1 -- 172.21.15.98:0/302125210 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcbc1387f0 con 0x7fbcbc0a4920 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb9d9b640 4 set_mon_vals no callback set 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcb3fff640 10 monclient: dump: 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcc1255640 10 client.30691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcc1255640 10 monclient: _renew_subs 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcc1255640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcc1255640 1 -- 172.21.15.98:0/302125210 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcbc128750 con 0x7fbcbc0a4920 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.509+0000 7fbcc1255640 1 librados: init done 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcc1255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcc1255640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128990 0x7fbcbc125710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcc1255640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcc1255640 1 -- 172.21.15.98:0/302125210 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbcbc050830 con 0x7fbcbc128990 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbad9d640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128990 0x7fbcbc125710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbad9d640 10 monclient: get_auth_request con 0x7fbcbc128990 auth_method 0 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fbca4045020 con 0x7fbcbc0a4920 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbad9d640 10 monclient(hunting): handle_auth_done global_id 22760 payload 824 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcb3fff640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbc9c03fdb0 0x7fbc9c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbad9d640 1 -- 172.21.15.98:0/302125210 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbcbc1277f0 con 0x7fbcbc128990 2024-06-11T19:11:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 8202+0+0 (secure 0 0 0) 0x7fbca4030070 con 0x7fbcbc0a4920 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcb3fff640 10 client.30691.objecter ms_dispatch 0x7fbcbc000b90 osd_map(76..76 src has 1..76) 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcb3fff640 3 client.30691.objecter handle_osd_map got epochs [76,76] > 0 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcb3fff640 3 client.30691.objecter handle_osd_map decoding full epoch 76 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcb3fff640 20 client.30691.objecter dump_active .. 0 homeless 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbad9d640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128990 0x7fbcbc125710 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fbcac010c60 tx=0x7fbcac00e5a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcb3fff640 10 client.30691.objecter ms_handle_connect 0x7fbcbc128990 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcb3fff640 10 client.30691.objecter resend_mon_ops 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbb59e640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbc9c03fdb0 0x7fbc9c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.510+0000 7fbcbb59e640 10 monclient: get_auth_request con 0x7fbc9c03fdb0 auth_method 0 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbcac018070 con 0x7fbcbc128990 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcb3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcb3fff640 10 monclient: dump: 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.535 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.536 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbcac014430 con 0x7fbcbc128990 2024-06-11T19:11:32.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcb3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:32.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcb3fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:32.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128990 msgr2=0x7fbcbc125710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcb3fff640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128990 0x7fbcbc125710 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fbcac010c60 tx=0x7fbcac00e5a0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128990 msgr2=0x7fbcbc125710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:32.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcb3fff640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128990 0x7fbcbc125710 secure :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fbcac010c60 tx=0x7fbcac00e5a0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcb3fff640 10 client.30691.objecter ms_handle_connect 0x7fbc9c03fdb0 2024-06-11T19:11:32.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.511+0000 7fbcbb59e640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbc9c03fdb0 0x7fbc9c042270 secure :-1 s=READY pgs=3976 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc0755f0 tx=0x7fbcb400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.517+0000 7fbcc1255640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.517+0000 7fbcc1255640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128d70 0x7fbcbc125c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.517+0000 7fbcc1255640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.517+0000 7fbcc1255640 1 -- 172.21.15.98:0/302125210 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbcbc0a6720 con 0x7fbcbc128d70 2024-06-11T19:11:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.517+0000 7fbcbad9d640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128d70 0x7fbcbc125c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.517+0000 7fbcbad9d640 10 monclient: get_auth_request con 0x7fbcbc128d70 auth_method 0 2024-06-11T19:11:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.517+0000 7fbcbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.517+0000 7fbcbad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.517+0000 7fbcbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.517+0000 7fbcbad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.517+0000 7fbcbad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.517+0000 7fbcbad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcbad9d640 10 monclient(hunting): handle_auth_done global_id 22769 payload 824 2024-06-11T19:11:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcbad9d640 1 -- 172.21.15.98:0/302125210 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbcbc126c90 con 0x7fbcbc128d70 2024-06-11T19:11:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcbad9d640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128d70 0x7fbcbc125c50 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fbcac000c00 tx=0x7fbcac0046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 10 client.30691.objecter ms_handle_connect 0x7fbcbc128d70 2024-06-11T19:11:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 10 client.30691.objecter resend_mon_ops 2024-06-11T19:11:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbcac010ea0 con 0x7fbcbc128d70 2024-06-11T19:11:32.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:32.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 10 monclient: dump: 2024-06-11T19:11:32.541 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.541 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.541 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.542 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fbcac003e70 con 0x7fbcbc128d70 2024-06-11T19:11:32.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:32.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:32.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128d70 msgr2=0x7fbcbc125c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128d70 0x7fbcbc125c50 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fbcac000c00 tx=0x7fbcac0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 1 -- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128d70 msgr2=0x7fbcbc125c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:32.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcb3fff640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcbc128d70 0x7fbcbc125c50 secure :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fbcac000c00 tx=0x7fbcac0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.543 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 142755, 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:32.566 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcc1255640 10 librados: watch_flush enter 2024-06-11T19:11:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcc1255640 10 librados: watch_flush exit 2024-06-11T19:11:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcc1255640 20 client.30691.objecter shutdown clearing up homeless session... 2024-06-11T19:11:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcc1255640 10 client.30691.objecter successfully canceled tick 2024-06-11T19:11:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcc1255640 1 -- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbc9c03fdb0 msgr2=0x7fbc9c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcc1255640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbc9c03fdb0 0x7fbc9c042270 secure :-1 s=READY pgs=3976 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc0755f0 tx=0x7fbcb400b040 comp rx=0 tx=0).stop 2024-06-11T19:11:32.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcc1255640 10 monclient: shutdown 2024-06-11T19:11:32.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcc1255640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:32.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcc1255640 1 -- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbcbc0a4920 msgr2=0x7fbcbc1cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcc1255640 1 --2- 172.21.15.98:0/302125210 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbcbc0a4920 0x7fbcbc1cdfe0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7fbca4033000 tx=0x7fbca40046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcbb59e640 1 -- 172.21.15.98:0/302125210 reap_dead start 2024-06-11T19:11:32.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcbb59e640 1 -- 172.21.15.98:0/302125210 reap_dead start 2024-06-11T19:11:32.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcc1255640 1 -- 172.21.15.98:0/302125210 shutdown_connections 2024-06-11T19:11:32.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.518+0000 7fbcc1255640 1 -- 172.21.15.98:0/302125210 >> 172.21.15.98:0/302125210 conn(0x7fbcbc0ab850 msgr2=0x7fbcbc09c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:32.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.519+0000 7fbcc1255640 1 -- 172.21.15.98:0/302125210 shutdown_connections 2024-06-11T19:11:32.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.519+0000 7fbcc1255640 1 -- 172.21.15.98:0/302125210 wait complete. 2024-06-11T19:11:32.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.519+0000 7fbcc1255640 1 librados: shutdown 2024-06-11T19:11:32.569 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:32.570 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:11:32.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:11:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.669+0000 7f57ccea4640 10 monclient: get_monmap_and_config 2024-06-11T19:11:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.669+0000 7f57ccea4640 10 monclient: build_initial_monmap 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.669+0000 7f57ccea4640 10 monclient: monmap: 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:32.670224+0000 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:32.670224+0000 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:32.670 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.669+0000 7f57ccea4640 1 Processor -- start 2024-06-11T19:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.669+0000 7f57ccea4640 1 -- start start 2024-06-11T19:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.669+0000 7f57ccea4640 10 monclient: init 2024-06-11T19:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 10 monclient: _add_conns ranks=[4,7,9,1,10,6,3,2,0,5,8] 2024-06-11T19:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c8136c90 0x7f57c8137070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 10 monclient(hunting): picked mon.noname-c con 0x7f57c8136c90 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f57c81367d0 0x7f57c80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 10 monclient(hunting): picked mon.noname-i con 0x7f57c81367d0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f57c80a3bd0 0x7f57c80a4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 10 monclient(hunting): picked mon.noname-a con 0x7f57c80a3bd0 addr v2:172.21.15.98:3300/0 2024-06-11T19:11:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f57c8068180 con 0x7f57c80a3bd0 2024-06-11T19:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f57c80a0270 con 0x7f57c8136c90 2024-06-11T19:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f57c809fc20 con 0x7f57c81367d0 2024-06-11T19:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57ccea4640 10 monclient(hunting): authenticate will time out at 902.477507s 2024-06-11T19:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f57c80a3bd0 0x7f57c80a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c659c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c8136c90 0x7f57c8137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c659c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c8136c90 0x7f57c8137070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46418/0 (socket says 172.21.15.98:46418) 2024-06-11T19:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f57c80a3bd0 0x7f57c80a4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53756/0 (socket says 172.21.15.98:53756) 2024-06-11T19:11:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c659c640 1 -- 172.21.15.98:0/3134586479 learned_addr learned my addr 172.21.15.98:0/3134586479 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 10 monclient(hunting): get_auth_request con 0x7f57c80a3bd0 auth_method 0 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c659c640 10 monclient(hunting): get_auth_request con 0x7f57c8136c90 auth_method 0 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c5d9b640 1 --2- 172.21.15.98:0/3134586479 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f57c81367d0 0x7f57c80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 10 monclient(hunting): handle_auth_done global_id 30888 payload 274 2024-06-11T19:11:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 1 -- 172.21.15.98:0/3134586479 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f57c81367d0 msgr2=0x7f57c80a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 1 --2- 172.21.15.98:0/3134586479 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f57c81367d0 0x7f57c80a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 1 -- 172.21.15.98:0/3134586479 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c8136c90 msgr2=0x7f57c8137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 1 --2- 172.21.15.98:0/3134586479 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c8136c90 0x7f57c8137070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:11:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 1 monclient: found mon.noname-a 2024-06-11T19:11:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:11:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c6d9d640 1 -- 172.21.15.98:0/3134586479 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c809ff00 con 0x7f57c80a3bd0 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c659c640 1 --2- 172.21.15.98:0/3134586479 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c8136c90 0x7f57c8137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c5d9b640 10 monclient: get_auth_request con 0x7f57c81367d0 auth_method 0 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.670+0000 7f57c5d9b640 1 --2- 172.21.15.98:0/3134586479 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f57c81367d0 0x7f57c80a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c6d9d640 1 --2- 172.21.15.98:0/3134586479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57c80a3bd0 0x7f57c80a4050 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009d90 tx=0x7f57b802bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d3402ef4ec566a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 1 -- 172.21.15.98:0/3134586479 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f57b8039070 con 0x7f57c80a3bd0 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 10 monclient: dump: 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 10 monclient: _finish_auth 0 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 10 monclient: _check_auth_tickets 2024-06-11T19:11:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 1 -- 172.21.15.98:0/3134586479 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57b802cd90 con 0x7f57c80a3bd0 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57ccea4640 5 monclient: authenticate success, global_id 30888 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57ccea4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57ccea4640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 1 -- 172.21.15.98:0/3134586479 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f57b80327b0 con 0x7f57c80a3bd0 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57ccea4640 4 set_mon_vals no callback set 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.671+0000 7f57c559a640 10 monclient: dump: 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.672+0000 7f57ccea4640 10 monclient: shutdown 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.672+0000 7f57ccea4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.672+0000 7f57ccea4640 1 -- 172.21.15.98:0/3134586479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57c80a3bd0 msgr2=0x7f57c80a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.672+0000 7f57ccea4640 1 --2- 172.21.15.98:0/3134586479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57c80a3bd0 0x7f57c80a4050 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009d90 tx=0x7f57b802bd70 comp rx=0 tx=0).stop 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.672+0000 7f57ccea4640 1 -- 172.21.15.98:0/3134586479 shutdown_connections 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.672+0000 7f57ccea4640 1 --2- 172.21.15.98:0/3134586479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57c80a3bd0 0x7f57c80a4050 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.672+0000 7f57ccea4640 1 --2- 172.21.15.98:0/3134586479 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f57c81367d0 0x7f57c80a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.672+0000 7f57ccea4640 1 --2- 172.21.15.98:0/3134586479 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c8136c90 0x7f57c8137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.672+0000 7f57ccea4640 1 -- 172.21.15.98:0/3134586479 >> 172.21.15.98:0/3134586479 conn(0x7f57c80a4780 msgr2=0x7f57c80a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.672+0000 7f57ccea4640 1 -- 172.21.15.98:0/3134586479 shutdown_connections 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.672+0000 7f57ccea4640 1 -- 172.21.15.98:0/3134586479 wait complete. 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.673+0000 7f57ccea4640 10 monclient: build_initial_monmap 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.673+0000 7f57ccea4640 10 monclient: monmap: 2024-06-11T19:11:32.688 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.673+0000 7f57ccea4640 1 librados: starting msgr at 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.673+0000 7f57ccea4640 1 librados: starting objecter 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.673+0000 7f57ccea4640 1 Processor -- start 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.673+0000 7f57ccea4640 1 -- start start 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.673+0000 7f57ccea4640 1 librados: setting wanted keys 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.673+0000 7f57ccea4640 1 librados: calling monclient init 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.673+0000 7f57ccea4640 10 monclient: init 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient: _add_conns ranks=[4,8,1,2,3,6,5,0,7] 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c80a3bd0 0x7f57c81cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient(hunting): picked mon.noname-c con 0x7f57c80a3bd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57c81367d0 0x7f57c81cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient(hunting): picked mon.noname-e con 0x7f57c81367d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8136c90 0x7f57c8120c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient(hunting): picked mon.noname-f con 0x7f57c8136c90 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81375b0 con 0x7f57c8136c90 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f57c8068560 con 0x7f57c80a3bd0 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f57c8124bd0 con 0x7f57c81367d0 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient(hunting): authenticate will time out at 902.481172s 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c80a3bd0 0x7f57c81cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c80a3bd0 0x7f57c81cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46420/0 (socket says 172.21.15.98:46420) 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 1 -- 172.21.15.98:0/169457769 learned_addr learned my addr 172.21.15.98:0/169457769 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 10 monclient(hunting): get_auth_request con 0x7f57c80a3bd0 auth_method 0 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c5d9b640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57c81367d0 0x7f57c81cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c5d9b640 10 monclient(hunting): get_auth_request con 0x7f57c81367d0 auth_method 0 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c5d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c6d9d640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8136c90 0x7f57c8120c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c6d9d640 10 monclient(hunting): get_auth_request con 0x7f57c8136c90 auth_method 0 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c6d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 10 monclient(hunting): handle_auth_done global_id 22783 payload 824 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 1 -- 172.21.15.98:0/169457769 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57c81367d0 msgr2=0x7f57c81cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57c81367d0 0x7f57c81cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 1 -- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8136c90 msgr2=0x7f57c8120c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8136c90 0x7f57c8120c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 10 monclient: _finish_hunting 0 2024-06-11T19:11:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 1 monclient: found mon.noname-c 2024-06-11T19:11:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:11:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 1 -- 172.21.15.98:0/169457769 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c80a7710 con 0x7f57c80a3bd0 2024-06-11T19:11:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c5d9b640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57c81367d0 0x7f57c81cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c659c640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c80a3bd0 0x7f57c81cba40 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f57c000e9e0 tx=0x7f57c000eea0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 10 client.?.objecter ms_handle_connect 0x7f57c80a3bd0 2024-06-11T19:11:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f57c0017070 con 0x7f57c80a3bd0 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 10 monclient: dump: 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 10 monclient: _finish_auth 0 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57c000ce40 con 0x7f57c80a3bd0 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 5 monclient: authenticate success, global_id 22783 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f57c00138e0 con 0x7f57c80a3bd0 2024-06-11T19:11:32.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient: _renew_subs 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 1 -- 172.21.15.98:0/169457769 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c81248f0 con 0x7f57c80a3bd0 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57b6ffd640 10 monclient: dump: 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.697 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57c4d99640 4 set_mon_vals no callback set 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 client.22783.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient: _renew_subs 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:11:32.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 1 -- 172.21.15.98:0/169457769 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c8124610 con 0x7f57c80a3bd0 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.674+0000 7f57ccea4640 1 librados: init done 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.675+0000 7f57ccea4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.675+0000 7f57ccea4640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121190 0x7f57c8121570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.675+0000 7f57ccea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.675+0000 7f57ccea4640 1 -- 172.21.15.98:0/169457769 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f57c8137730 con 0x7f57c8121190 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.675+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f57c001a070 con 0x7f57c80a3bd0 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.675+0000 7f57c5d9b640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121190 0x7f57c8121570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c5d9b640 10 monclient: get_auth_request con 0x7f57c8121190 auth_method 0 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c5d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f57a803fdb0 0x7f57a8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(76..76 src has 1..76) ==== 8202+0+0 (secure 0 0 0) 0x7f57c000f040 con 0x7f57c80a3bd0 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 10 client.22783.objecter ms_dispatch 0x7f57c8000b90 osd_map(76..76 src has 1..76) 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 3 client.22783.objecter handle_osd_map got epochs [76,76] > 0 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 3 client.22783.objecter handle_osd_map decoding full epoch 76 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c5d9b640 10 monclient(hunting): handle_auth_done global_id 30709 payload 824 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c5d9b640 1 -- 172.21.15.98:0/169457769 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f57c80a3410 con 0x7f57c8121190 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 20 client.22783.objecter dump_active .. 0 homeless 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c6d9d640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f57a803fdb0 0x7f57a8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c6d9d640 10 monclient: get_auth_request con 0x7f57a803fdb0 auth_method 0 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 10 client.22783.objecter ms_handle_connect 0x7f57c8121190 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 10 client.22783.objecter resend_mon_ops 2024-06-11T19:11:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c5d9b640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121190 0x7f57c8121570 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f57c80a8d40 tx=0x7f57bc00d590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f57bc00fe50 con 0x7f57c8121190 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 10 monclient: dump: 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.702 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 10 client.22783.objecter ms_handle_connect 0x7f57a803fdb0 2024-06-11T19:11:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57c6d9d640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f57a803fdb0 0x7f57a8042270 secure :-1 s=READY pgs=3977 cs=0 l=1 rev1=1 crypto rx=0x7f57b8036000 tx=0x7f57b8030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f57bc00fe50 con 0x7f57c8121190 2024-06-11T19:11:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121190 msgr2=0x7f57c8121570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121190 0x7f57c8121570 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f57c80a8d40 tx=0x7f57bc00d590 comp rx=0 tx=0).stop 2024-06-11T19:11:32.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121190 msgr2=0x7f57c8121570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:32.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.676+0000 7f57b6ffd640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121190 0x7f57c8121570 secure :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f57c80a8d40 tx=0x7f57bc00d590 comp rx=0 tx=0).stop 2024-06-11T19:11:32.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57ccea4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57ccea4640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121ab0 0x7f57c8071540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57ccea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57ccea4640 1 -- 172.21.15.98:0/169457769 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f57c8050830 con 0x7f57c8121ab0 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57c5d9b640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121ab0 0x7f57c8071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57c5d9b640 10 monclient: get_auth_request con 0x7f57c8121ab0 auth_method 0 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57c5d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57c5d9b640 10 monclient(hunting): handle_auth_done global_id 30718 payload 824 2024-06-11T19:11:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.683+0000 7f57c5d9b640 1 -- 172.21.15.98:0/169457769 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f57c809f2a0 con 0x7f57c8121ab0 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 10 client.22783.objecter ms_handle_connect 0x7f57c8121ab0 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57c5d9b640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121ab0 0x7f57c8071540 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f57bc008010 tx=0x7f57bc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 10 client.22783.objecter resend_mon_ops 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f57bc006670 con 0x7f57c8121ab0 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 10 monclient: dump: 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f57bc024070 con 0x7f57c8121ab0 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:32.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121ab0 msgr2=0x7f57c8071540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121ab0 0x7f57c8071540 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f57bc008010 tx=0x7f57bc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 1 -- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121ab0 msgr2=0x7f57c8071540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57b6ffd640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c8121ab0 0x7f57c8071540 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.708 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 142902, 2024-06-11T19:11:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.715 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:32.717 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.720 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:32.722 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.725 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:32.727 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.729 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.730 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:32.730 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:32.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 10 librados: watch_flush enter 2024-06-11T19:11:32.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 10 librados: watch_flush exit 2024-06-11T19:11:32.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 20 client.22783.objecter shutdown clearing up homeless session... 2024-06-11T19:11:32.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 10 client.22783.objecter successfully canceled tick 2024-06-11T19:11:32.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 1 -- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f57a803fdb0 msgr2=0x7f57a8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f57a803fdb0 0x7f57a8042270 secure :-1 s=READY pgs=3977 cs=0 l=1 rev1=1 crypto rx=0x7f57b8036000 tx=0x7f57b8030000 comp rx=0 tx=0).stop 2024-06-11T19:11:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 10 monclient: shutdown 2024-06-11T19:11:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 1 -- 172.21.15.98:0/169457769 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c80a3bd0 msgr2=0x7f57c81cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 1 --2- 172.21.15.98:0/169457769 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57c80a3bd0 0x7f57c81cba40 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f57c000e9e0 tx=0x7f57c000eea0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57c6d9d640 1 -- 172.21.15.98:0/169457769 reap_dead start 2024-06-11T19:11:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57c6d9d640 1 -- 172.21.15.98:0/169457769 reap_dead start 2024-06-11T19:11:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 1 -- 172.21.15.98:0/169457769 shutdown_connections 2024-06-11T19:11:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 1 -- 172.21.15.98:0/169457769 >> 172.21.15.98:0/169457769 conn(0x7f57c80a4780 msgr2=0x7f57c81bc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 1 -- 172.21.15.98:0/169457769 shutdown_connections 2024-06-11T19:11:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 1 -- 172.21.15.98:0/169457769 wait complete. 2024-06-11T19:11:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.684+0000 7f57ccea4640 1 librados: shutdown 2024-06-11T19:11:32.734 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:32.734 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:11:32.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:11:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.834+0000 7f6f00c87640 10 monclient: get_monmap_and_config 2024-06-11T19:11:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.834+0000 7f6f00c87640 10 monclient: build_initial_monmap 2024-06-11T19:11:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.834+0000 7f6f00c87640 10 monclient: monmap: 2024-06-11T19:11:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:32.835801+0000 2024-06-11T19:11:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:32.835801+0000 2024-06-11T19:11:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:32.836 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.835+0000 7f6f00c87640 1 Processor -- start 2024-06-11T19:11:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.835+0000 7f6f00c87640 1 -- start start 2024-06-11T19:11:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.835+0000 7f6f00c87640 10 monclient: init 2024-06-11T19:11:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 10 monclient: _add_conns ranks=[3,8,9,7,4,0,6,10,2,1,5] 2024-06-11T19:11:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc09d330 0x7f6efc09d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 10 monclient(hunting): picked mon.noname-g con 0x7f6efc09d330 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6efc09dc50 0x7f6efc136770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 10 monclient(hunting): picked mon.noname-e con 0x7f6efc09dc50 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6efc0a2850 0x7f6efc136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 10 monclient(hunting): picked mon.noname-a con 0x7f6efc0a2850 addr v2:172.21.15.98:3300/0 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6efc09bb20 con 0x7f6efc0a2850 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f6efc09b560 con 0x7f6efc09d330 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6efc09b840 con 0x7f6efc09dc50 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6f00c87640 10 monclient(hunting): authenticate will time out at 902.643130s 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6efc09dc50 0x7f6efc136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efad9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6efc0a2850 0x7f6efc136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6efc09dc50 0x7f6efc136770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34528/0 (socket says 172.21.15.98:34528) 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 1 -- 172.21.15.98:0/3466892121 learned_addr learned my addr 172.21.15.98:0/3466892121 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efad9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6efc0a2850 0x7f6efc136cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53762/0 (socket says 172.21.15.98:53762) 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 10 monclient(hunting): get_auth_request con 0x7f6efc09dc50 auth_method 0 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efad9d640 10 monclient(hunting): get_auth_request con 0x7f6efc0a2850 auth_method 0 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efa59c640 1 --2- 172.21.15.98:0/3466892121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc09d330 0x7f6efc09d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efa59c640 10 monclient(hunting): get_auth_request con 0x7f6efc09d330 auth_method 0 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efa59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efa59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 10 monclient(hunting): handle_auth_done global_id 22787 payload 274 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 1 -- 172.21.15.98:0/3466892121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc09d330 msgr2=0x7f6efc09d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 1 --2- 172.21.15.98:0/3466892121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc09d330 0x7f6efc09d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 1 -- 172.21.15.98:0/3466892121 >> v2:172.21.15.98:3300/0 conn(0x7f6efc0a2850 msgr2=0x7f6efc136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 1 --2- 172.21.15.98:0/3466892121 >> v2:172.21.15.98:3300/0 conn(0x7f6efc0a2850 0x7f6efc136cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 1 monclient: found mon.noname-e 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6ef9d9b640 1 -- 172.21.15.98:0/3466892121 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6efc0769a0 con 0x7f6efc09dc50 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.836+0000 7f6efad9d640 1 --2- 172.21.15.98:0/3466892121 >> v2:172.21.15.98:3300/0 conn(0x7f6efc0a2850 0x7f6efc136cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef9d9b640 1 --2- 172.21.15.98:0/3466892121 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6efc09dc50 0x7f6efc136770 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009850 tx=0x7f6ef002b7a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=9a61a50997b03ccb server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 1 -- 172.21.15.98:0/3466892121 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ef0038070 con 0x7f6efc09dc50 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 10 monclient: dump: 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.852 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 10 monclient: _finish_auth 0 2024-06-11T19:11:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 10 monclient: _check_auth_tickets 2024-06-11T19:11:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6f00c87640 5 monclient: authenticate success, global_id 22787 2024-06-11T19:11:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6f00c87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 1 -- 172.21.15.98:0/3466892121 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef00044a0 con 0x7f6efc09dc50 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6f00c87640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6f00c87640 4 set_mon_vals no callback set 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 1 -- 172.21.15.98:0/3466892121 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ef0003a80 con 0x7f6efc09dc50 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.837+0000 7f6ef959a640 10 monclient: dump: 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 10 monclient: shutdown 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 1 -- 172.21.15.98:0/3466892121 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6efc09dc50 msgr2=0x7f6efc136770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 1 --2- 172.21.15.98:0/3466892121 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6efc09dc50 0x7f6efc136770 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009850 tx=0x7f6ef002b7a0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 1 -- 172.21.15.98:0/3466892121 shutdown_connections 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 1 --2- 172.21.15.98:0/3466892121 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6efc09dc50 0x7f6efc136770 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 1 --2- 172.21.15.98:0/3466892121 >> v2:172.21.15.98:3300/0 conn(0x7f6efc0a2850 0x7f6efc136cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 1 --2- 172.21.15.98:0/3466892121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc09d330 0x7f6efc09d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 1 -- 172.21.15.98:0/3466892121 >> 172.21.15.98:0/3466892121 conn(0x7f6efc0a7850 msgr2=0x7f6efc0a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 1 -- 172.21.15.98:0/3466892121 shutdown_connections 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 1 -- 172.21.15.98:0/3466892121 wait complete. 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 10 monclient: build_initial_monmap 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.838+0000 7f6f00c87640 10 monclient: monmap: 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 1 librados: starting msgr at 2024-06-11T19:11:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 1 librados: starting objecter 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 1 Processor -- start 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 1 -- start start 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 1 librados: setting wanted keys 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 1 librados: calling monclient init 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 10 monclient: init 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 10 monclient: _add_conns ranks=[1,6,0,2,4,5,3,7,8] 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6efc09d330 0x7f6efc130570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 10 monclient(hunting): picked mon.noname-f con 0x7f6efc09d330 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6efc0a2850 0x7f6efc130ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 10 monclient(hunting): picked mon.noname-d con 0x7f6efc0a2850 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc130ff0 0x7f6efc1cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 10 monclient(hunting): picked mon.noname-a con 0x7f6efc130ff0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc1371f0 con 0x7f6efc130ff0 2024-06-11T19:11:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc09e270 con 0x7f6efc09d330 2024-06-11T19:11:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f6efc137370 con 0x7f6efc0a2850 2024-06-11T19:11:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6f00c87640 10 monclient(hunting): authenticate will time out at 902.646732s 2024-06-11T19:11:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6efad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc130ff0 0x7f6efc1cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6ef9d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6efc0a2850 0x7f6efc130ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6efad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc130ff0 0x7f6efc1cb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53772/0 (socket says 172.21.15.98:53772) 2024-06-11T19:11:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6efad9d640 1 -- 172.21.15.98:0/3323389650 learned_addr learned my addr 172.21.15.98:0/3323389650 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6efad9d640 10 monclient(hunting): get_auth_request con 0x7f6efc130ff0 auth_method 0 2024-06-11T19:11:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6efad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6efad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6efad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6ef9d9b640 10 monclient(hunting): get_auth_request con 0x7f6efc0a2850 auth_method 0 2024-06-11T19:11:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6ef9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6ef9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6ef9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6efad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6efad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6efad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6ef9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6ef9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.839+0000 7f6ef9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efa59c640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6efc09d330 0x7f6efc130570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efa59c640 10 monclient(hunting): get_auth_request con 0x7f6efc09d330 auth_method 0 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efa59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efa59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efad9d640 10 monclient(hunting): handle_auth_done global_id 30906 payload 824 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efad9d640 1 -- 172.21.15.98:0/3323389650 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6efc0a2850 msgr2=0x7f6efc130ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efad9d640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6efc0a2850 0x7f6efc130ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efad9d640 1 -- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6efc09d330 msgr2=0x7f6efc130570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efad9d640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6efc09d330 0x7f6efc130570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efad9d640 1 monclient: found mon.noname-a 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efad9d640 1 -- 172.21.15.98:0/3323389650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6efc134190 con 0x7f6efc130ff0 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6ef9d9b640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6efc0a2850 0x7f6efc130ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efa59c640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6efc09d330 0x7f6efc130570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6efad9d640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc130ff0 0x7f6efc1cb820 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400efb0 tx=0x7f6ef400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 10 client.?.objecter ms_handle_connect 0x7f6efc130ff0 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ef4018020 con 0x7f6efc130ff0 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 10 monclient: dump: 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 10 monclient: _finish_auth 0 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6f00c87640 5 monclient: authenticate success, global_id 30906 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6f00c87640 10 monclient: _renew_subs 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef400cd30 con 0x7f6efc130ff0 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6f00c87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6f00c87640 1 -- 172.21.15.98:0/3323389650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6efc134470 con 0x7f6efc130ff0 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ef4014d60 con 0x7f6efc130ff0 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6eeaffd640 10 monclient: dump: 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6ef8d99640 4 set_mon_vals no callback set 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6f00c87640 10 client.30906.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6f00c87640 10 monclient: _renew_subs 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6f00c87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6f00c87640 1 -- 172.21.15.98:0/3323389650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6efc0a4980 con 0x7f6efc130ff0 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.840+0000 7f6f00c87640 1 librados: init done 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6f00c87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6f00c87640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc09dc50 0x7f6efc02cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6f00c87640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6f00c87640 1 -- 172.21.15.98:0/3323389650 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f6efc132c70 con 0x7f6efc09dc50 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f6ef4005d10 con 0x7f6efc130ff0 2024-06-11T19:11:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6efa59c640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc09dc50 0x7f6efc02cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6efa59c640 10 monclient: get_auth_request con 0x7f6efc09dc50 auth_method 0 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6efa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6efa59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6efa59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6eeaffd640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6edc03fde0 0x7f6edc0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6efa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6efa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 8202+0+0 (secure 0 0 0) 0x7f6ef4052ef0 con 0x7f6efc130ff0 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6eeaffd640 10 client.30906.objecter ms_dispatch 0x7f6efc000b90 osd_map(76..76 src has 1..76) 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6eeaffd640 3 client.30906.objecter handle_osd_map got epochs [76,76] > 0 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6efa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6eeaffd640 3 client.30906.objecter handle_osd_map decoding full epoch 76 2024-06-11T19:11:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6eeaffd640 20 client.30906.objecter dump_active .. 0 homeless 2024-06-11T19:11:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6ef9d9b640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6edc03fde0 0x7f6edc0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.841+0000 7f6ef9d9b640 10 monclient: get_auth_request con 0x7f6edc03fde0 auth_method 0 2024-06-11T19:11:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6efa59c640 10 monclient(hunting): handle_auth_done global_id 21549 payload 824 2024-06-11T19:11:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6efa59c640 1 -- 172.21.15.98:0/3323389650 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6efc073b50 con 0x7f6efc09dc50 2024-06-11T19:11:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 10 client.30906.objecter ms_handle_connect 0x7f6efc09dc50 2024-06-11T19:11:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 10 client.30906.objecter resend_mon_ops 2024-06-11T19:11:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6efa59c640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc09dc50 0x7f6efc02cb40 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f6eec011830 tx=0x7f6eec011cf0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 10 client.30906.objecter ms_handle_connect 0x7f6edc03fde0 2024-06-11T19:11:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6ef9d9b640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6edc03fde0 0x7f6edc0422a0 secure :-1 s=READY pgs=3978 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0035000 tx=0x7f6ef0008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6eec00fca0 con 0x7f6efc09dc50 2024-06-11T19:11:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 10 monclient: dump: 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.869 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6eec00f830 con 0x7f6efc09dc50 2024-06-11T19:11:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc09dc50 msgr2=0x7f6efc02cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc09dc50 0x7f6efc02cb40 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f6eec011830 tx=0x7f6eec011cf0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc09dc50 msgr2=0x7f6efc02cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.842+0000 7f6eeaffd640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc09dc50 0x7f6efc02cb40 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6f00c87640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6f00c87640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc1cc520 0x7f6efc02d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6f00c87640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6f00c87640 1 -- 172.21.15.98:0/3323389650 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f6efc050830 con 0x7f6efc1cc520 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6efa59c640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc1cc520 0x7f6efc02d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6efa59c640 10 monclient: get_auth_request con 0x7f6efc1cc520 auth_method 0 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6efa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6efa59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6efa59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6efa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6efa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6efa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6efa59c640 10 monclient(hunting): handle_auth_done global_id 21558 payload 824 2024-06-11T19:11:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6efa59c640 1 -- 172.21.15.98:0/3323389650 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6efc09d710 con 0x7f6efc1cc520 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6eeaffd640 10 client.30906.objecter ms_handle_connect 0x7f6efc1cc520 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6efa59c640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc1cc520 0x7f6efc02d080 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6eec011830 tx=0x7f6eec0046c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6eeaffd640 10 client.30906.objecter resend_mon_ops 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6eec0067b0 con 0x7f6efc1cc520 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6eeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6eeaffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.849+0000 7f6eeaffd640 10 monclient: dump: 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f6eec026070 con 0x7f6efc1cc520 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6eeaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6eeaffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc1cc520 msgr2=0x7f6efc02d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6eeaffd640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc1cc520 0x7f6efc02d080 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6eec011830 tx=0x7f6eec0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6eeaffd640 1 -- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc1cc520 msgr2=0x7f6efc02d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6eeaffd640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6efc1cc520 0x7f6efc02d080 secure :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6eec011830 tx=0x7f6eec0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.875 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:32.875 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 143067, 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:11:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:32.898 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:32.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 10 librados: watch_flush enter 2024-06-11T19:11:32.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 10 librados: watch_flush exit 2024-06-11T19:11:32.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 20 client.30906.objecter shutdown clearing up homeless session... 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 10 client.30906.objecter successfully canceled tick 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 1 -- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6edc03fde0 msgr2=0x7f6edc0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6edc03fde0 0x7f6edc0422a0 secure :-1 s=READY pgs=3978 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0035000 tx=0x7f6ef0008030 comp rx=0 tx=0).stop 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 10 monclient: shutdown 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 1 -- 172.21.15.98:0/3323389650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc130ff0 msgr2=0x7f6efc1cb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6efad9d640 1 -- 172.21.15.98:0/3323389650 reap_dead start 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc130ff0 0x7f6efc1cb820 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400efb0 tx=0x7f6ef400c490 comp rx=0 tx=0).stop 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 1 -- 172.21.15.98:0/3323389650 shutdown_connections 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 1 --2- 172.21.15.98:0/3323389650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6efc130ff0 0x7f6efc1cb820 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 1 -- 172.21.15.98:0/3323389650 >> 172.21.15.98:0/3323389650 conn(0x7f6efc0a7850 msgr2=0x7f6efc072d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 1 -- 172.21.15.98:0/3323389650 shutdown_connections 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 1 -- 172.21.15.98:0/3323389650 wait complete. 2024-06-11T19:11:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:32.850+0000 7f6f00c87640 1 librados: shutdown 2024-06-11T19:11:32.902 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:32.902 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:11:32.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:11:33.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.002+0000 7f3990be3640 10 monclient: get_monmap_and_config 2024-06-11T19:11:33.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.002+0000 7f3990be3640 10 monclient: build_initial_monmap 2024-06-11T19:11:33.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.002+0000 7f3990be3640 10 monclient: monmap: 2024-06-11T19:11:33.002 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:33.002 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.002 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:33.003237+0000 2024-06-11T19:11:33.002 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:33.003237+0000 2024-06-11T19:11:33.002 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:33.002 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.002 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:33.002 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:33.003 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:33.003 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:33.003 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:33.003 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:33.003 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:33.003 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:33.003 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:33.003 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:33.003 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:33.003 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.002+0000 7f3990be3640 1 Processor -- start 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 1 -- start start 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 10 monclient: init 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 10 monclient: _add_conns ranks=[4,0,8,6,1,10,9,2,3,7,5] 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f398c0a52a0 0x7f398c0a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 10 monclient(hunting): picked mon.noname-c con 0x7f398c0a52a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f398c09d520 addr v1:172.21.15.98:6789/0 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 0x7f398c0a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 10 monclient(hunting): picked mon.noname-e con 0x7f398c073830 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f398c09d2e0 con 0x7f398c09d520 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f398c09daa0 con 0x7f398c0a52a0 2024-06-11T19:11:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f398c0a5680 con 0x7f398c073830 2024-06-11T19:11:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3990be3640 10 monclient(hunting): authenticate will time out at 902.810723s 2024-06-11T19:11:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398ad9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 0x7f398c0a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3989d9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f398c09d520 0x7f398c0a3690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51570/0 (socket says 172.21.15.98:51570) 2024-06-11T19:11:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f3989d9b640 1 -- 172.21.15.98:0/2074820574 learned_addr learned my addr 172.21.15.98:0/2074820574 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:11:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398ad9d640 10 monclient(hunting): get_auth_request con 0x7f398c073830 auth_method 0 2024-06-11T19:11:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398a59c640 1 --2- 172.21.15.98:0/2074820574 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f398c0a52a0 0x7f398c0a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398a59c640 10 monclient(hunting): get_auth_request con 0x7f398c0a52a0 auth_method 0 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.003+0000 7f398a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398ad9d640 10 monclient(hunting): handle_auth_done global_id 22796 payload 274 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398959a640 1 -- 172.21.15.98:0/2074820574 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1070832635 0 0) 0x7f398c09d2e0 con 0x7f398c09d520 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398ad9d640 1 -- 172.21.15.98:0/2074820574 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f398c0a52a0 msgr2=0x7f398c0a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398ad9d640 1 --2- 172.21.15.98:0/2074820574 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f398c0a52a0 0x7f398c0a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398ad9d640 1 -- 172.21.15.98:0/2074820574 >> v1:172.21.15.98:6789/0 conn(0x7f398c09d520 legacy=0x7f398c0a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398ad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398ad9d640 1 monclient: found mon.noname-e 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398ad9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398ad9d640 1 -- 172.21.15.98:0/2074820574 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398c09e9d0 con 0x7f398c073830 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398a59c640 1 --2- 172.21.15.98:0/2074820574 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f398c0a52a0 0x7f398c0a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.004+0000 7f398ad9d640 1 --2- 172.21.15.98:0/2074820574 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 0x7f398c0a3da0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f3984009df0 tx=0x7f398402bd50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=cd6bb4cbfeaa0a2b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 1 -- 172.21.15.98:0/2074820574 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3984039070 con 0x7f398c073830 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 10 monclient: dump: 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.018 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 10 monclient: _finish_auth 0 2024-06-11T19:11:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 10 monclient: _check_auth_tickets 2024-06-11T19:11:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 5 monclient: authenticate success, global_id 22796 2024-06-11T19:11:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 1 -- 172.21.15.98:0/2074820574 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f398402cd70 con 0x7f398c073830 2024-06-11T19:11:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 4 set_mon_vals no callback set 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 1 -- 172.21.15.98:0/2074820574 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f39840327b0 con 0x7f398c073830 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f398959a640 10 monclient: dump: 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.019 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 10 monclient: shutdown 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:33.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 1 -- 172.21.15.98:0/2074820574 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 msgr2=0x7f398c0a3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 1 --2- 172.21.15.98:0/2074820574 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 0x7f398c0a3da0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f3984009df0 tx=0x7f398402bd50 comp rx=0 tx=0).stop 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 1 -- 172.21.15.98:0/2074820574 shutdown_connections 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 1 --2- 172.21.15.98:0/2074820574 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 0x7f398c0a3da0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 1 --2- 172.21.15.98:0/2074820574 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f398c0a52a0 0x7f398c0a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 1 -- 172.21.15.98:0/2074820574 >> 172.21.15.98:0/2074820574 conn(0x7f398c0a4780 msgr2=0x7f398c0a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 1 -- 172.21.15.98:0/2074820574 shutdown_connections 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.005+0000 7f3990be3640 1 -- 172.21.15.98:0/2074820574 wait complete. 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.006+0000 7f3990be3640 10 monclient: build_initial_monmap 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.006+0000 7f3990be3640 10 monclient: monmap: 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:33.021 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.006+0000 7f3990be3640 1 librados: starting msgr at 2024-06-11T19:11:33.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.006+0000 7f3990be3640 1 librados: starting objecter 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.006+0000 7f3990be3640 1 Processor -- start 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.006+0000 7f3990be3640 1 -- start start 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.006+0000 7f3990be3640 1 librados: setting wanted keys 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.006+0000 7f3990be3640 1 librados: calling monclient init 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.006+0000 7f3990be3640 10 monclient: init 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 10 monclient: _add_conns ranks=[8,6,7,4,3,0,5,1,2] 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 0x7f398c1ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 10 monclient(hunting): picked mon.noname-e con 0x7f398c073830 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f398c09d520 0x7f398c1cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 10 monclient(hunting): picked mon.noname-d con 0x7f398c09d520 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f398c0a52a0 0x7f398c11e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 10 monclient(hunting): picked mon.noname-i con 0x7f398c0a52a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f398c0a4460 con 0x7f398c09d520 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f398c0a42e0 con 0x7f398c0a52a0 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f398c0a1e50 con 0x7f398c073830 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3990be3640 10 monclient(hunting): authenticate will time out at 902.814246s 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 0x7f398c1ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 0x7f398c1ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:45482/0 (socket says 172.21.15.98:45482) 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 1 -- 172.21.15.98:0/17692794 learned_addr learned my addr 172.21.15.98:0/17692794 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 10 monclient(hunting): get_auth_request con 0x7f398c073830 auth_method 0 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f398c09d520 0x7f398c1cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 10 monclient(hunting): get_auth_request con 0x7f398c09d520 auth_method 0 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 1 -- 172.21.15.98:0/17692794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 msgr2=0x7f398c1ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:11:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 1 -- 172.21.15.98:0/17692794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 msgr2=0x7f398c1ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:11:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 0x7f398c1ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398a59c640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 0x7f398c1ca7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 10 monclient(hunting): handle_auth_done global_id 54141 payload 824 2024-06-11T19:11:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f398ad9d640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f398c0a52a0 0x7f398c11e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 1 -- 172.21.15.98:0/17692794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 msgr2=0x7f398c1ca7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:11:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f398c073830 0x7f398c1ca7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 1 -- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f398c0a52a0 msgr2=0x7f398c11e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f398c0a52a0 0x7f398c11e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:11:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 1 monclient: found mon.noname-d 2024-06-11T19:11:33.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 1 -- 172.21.15.98:0/17692794 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398c11fca0 con 0x7f398c09d520 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f3989d9b640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f398c09d520 0x7f398c1cad10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f398c0a8d40 tx=0x7f398000c870 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f397affd640 10 client.?.objecter ms_handle_connect 0x7f398c09d520 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f397affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f39800047e0 con 0x7f398c09d520 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.007+0000 7f397affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f397affd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f397affd640 10 monclient: dump: 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.027 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f397affd640 10 monclient: _finish_auth 0 2024-06-11T19:11:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f397affd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f397affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f3990be3640 5 monclient: authenticate success, global_id 54141 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f3990be3640 10 monclient: _renew_subs 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39800049c0 con 0x7f398c09d520 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f3990be3640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f3990be3640 1 -- 172.21.15.98:0/17692794 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f398c0a4bd0 con 0x7f398c09d520 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f397affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f39800133c0 con 0x7f398c09d520 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f397affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f397affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f3990be3640 10 client.54141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f3988d99640 4 set_mon_vals no callback set 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f397affd640 10 monclient: dump: 2024-06-11T19:11:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f3990be3640 10 monclient: _renew_subs 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f3990be3640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f3990be3640 1 -- 172.21.15.98:0/17692794 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f398c1212c0 con 0x7f398c09d520 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f3990be3640 1 librados: init done 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f3990be3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.008+0000 7f3990be3640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11e810 0x7f398c11ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f3990be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f3990be3640 1 -- 172.21.15.98:0/17692794 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f398c050830 con 0x7f398c11e810 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3980011070 con 0x7f398c09d520 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398a59c640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11e810 0x7f398c11ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398a59c640 10 monclient: get_auth_request con 0x7f398c11e810 auth_method 0 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f396c03fdb0 0x7f396c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(76..76 src has 1..76) ==== 8202+0+0 (secure 0 0 0) 0x7f3980051d30 con 0x7f398c09d520 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 10 client.54141.objecter ms_dispatch 0x7f398c000b90 osd_map(76..76 src has 1..76) 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 3 client.54141.objecter handle_osd_map got epochs [76,76] > 0 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 3 client.54141.objecter handle_osd_map decoding full epoch 76 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 20 client.54141.objecter dump_active .. 0 homeless 2024-06-11T19:11:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398a59c640 10 monclient(hunting): handle_auth_done global_id 22028 payload 824 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398ad9d640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f396c03fdb0 0x7f396c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398a59c640 1 -- 172.21.15.98:0/17692794 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f398c0a1910 con 0x7f398c11e810 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398ad9d640 10 monclient: get_auth_request con 0x7f396c03fdb0 auth_method 0 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 10 client.54141.objecter ms_handle_connect 0x7f398c11e810 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 10 client.54141.objecter resend_mon_ops 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398a59c640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11e810 0x7f398c11ebf0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f397c011980 tx=0x7f397c011e40 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f397c014440 con 0x7f398c11e810 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 10 monclient: dump: 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f397affd640 10 client.54141.objecter ms_handle_connect 0x7f396c03fdb0 2024-06-11T19:11:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.009+0000 7f398ad9d640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f396c03fdb0 0x7f396c042270 secure :-1 s=READY pgs=3979 cs=0 l=1 rev1=1 crypto rx=0x7f3984036000 tx=0x7f3984030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.010+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f397c00fb60 con 0x7f398c11e810 2024-06-11T19:11:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.010+0000 7f397affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.010+0000 7f397affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.010+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11e810 msgr2=0x7f398c11ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.010+0000 7f397affd640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11e810 0x7f398c11ebf0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f397c011980 tx=0x7f397c011e40 comp rx=0 tx=0).stop 2024-06-11T19:11:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.010+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11e810 msgr2=0x7f398c11ebf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.010+0000 7f397affd640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11e810 0x7f398c11ebf0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.016+0000 7f3990be3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:33.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.016+0000 7f3990be3640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11f130 0x7f398c067ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.016+0000 7f3990be3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.016+0000 7f3990be3640 1 -- 172.21.15.98:0/17692794 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f398c121c20 con 0x7f398c11f130 2024-06-11T19:11:33.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.016+0000 7f398a59c640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11f130 0x7f398c067ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f398a59c640 10 monclient: get_auth_request con 0x7f398c11f130 auth_method 0 2024-06-11T19:11:33.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f398a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f398a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f398a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f398a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f398a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f398a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f398a59c640 10 monclient(hunting): handle_auth_done global_id 22037 payload 824 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f398a59c640 1 -- 172.21.15.98:0/17692794 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f398c122590 con 0x7f398c11f130 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f398a59c640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11f130 0x7f398c067ce0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f397c011980 tx=0x7f397c0046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 10 client.54141.objecter ms_handle_connect 0x7f398c11f130 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 10 client.54141.objecter resend_mon_ops 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f397c00fea0 con 0x7f398c11f130 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:11:33.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 10 monclient: dump: 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.039 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.040 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7f397c01d400 con 0x7f398c11f130 2024-06-11T19:11:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11f130 msgr2=0x7f398c067ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11f130 0x7f398c067ce0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f397c011980 tx=0x7f397c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 1 -- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11f130 msgr2=0x7f398c067ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.017+0000 7f397affd640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f398c11f130 0x7f398c067ce0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.040 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 143211, 2024-06-11T19:11:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:33.041 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:33.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:33.043 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:33.044 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:33.045 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:33.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:33.048 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.050 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.053 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.054 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.056 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:33.059 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:33.060 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.062 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.063 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:33.064 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:33.066 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:33.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 10 librados: watch_flush enter 2024-06-11T19:11:33.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 10 librados: watch_flush exit 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 20 client.54141.objecter shutdown clearing up homeless session... 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 10 client.54141.objecter successfully canceled tick 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 1 -- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f396c03fdb0 msgr2=0x7f396c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f396c03fdb0 0x7f396c042270 secure :-1 s=READY pgs=3979 cs=0 l=1 rev1=1 crypto rx=0x7f3984036000 tx=0x7f3984030000 comp rx=0 tx=0).stop 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 10 monclient: shutdown 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 1 -- 172.21.15.98:0/17692794 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f398c09d520 msgr2=0x7f398c1cad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 1 --2- 172.21.15.98:0/17692794 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f398c09d520 0x7f398c1cad10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f398c0a8d40 tx=0x7f398000c870 comp rx=0 tx=0).stop 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f398ad9d640 1 -- 172.21.15.98:0/17692794 reap_dead start 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f398ad9d640 1 -- 172.21.15.98:0/17692794 reap_dead start 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 1 -- 172.21.15.98:0/17692794 shutdown_connections 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 1 -- 172.21.15.98:0/17692794 >> 172.21.15.98:0/17692794 conn(0x7f398c0a4780 msgr2=0x7f398c1bb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 1 -- 172.21.15.98:0/17692794 shutdown_connections 2024-06-11T19:11:33.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 1 -- 172.21.15.98:0/17692794 wait complete. 2024-06-11T19:11:33.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.018+0000 7f3990be3640 1 librados: shutdown 2024-06-11T19:11:33.070 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:33.070 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:11:33.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:11:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.170+0000 7f8d21dd5640 10 monclient: get_monmap_and_config 2024-06-11T19:11:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.170+0000 7f8d21dd5640 10 monclient: build_initial_monmap 2024-06-11T19:11:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.170+0000 7f8d21dd5640 10 monclient: monmap: 2024-06-11T19:11:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:33.171317+0000 2024-06-11T19:11:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:33.171317+0000 2024-06-11T19:11:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:33.171 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 1 Processor -- start 2024-06-11T19:11:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 1 -- start start 2024-06-11T19:11:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 10 monclient: init 2024-06-11T19:11:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 10 monclient: _add_conns ranks=[10,7,6,1,8,9,3,5,2,4,0] 2024-06-11T19:11:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8d1c0a0480 0x7f8d1c0a0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 10 monclient(hunting): picked mon.noname-f con 0x7f8d1c0a0480 addr v2:172.21.15.117:3300/0 2024-06-11T19:11:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c0a10c0 0x7f8d1c0a1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 10 monclient(hunting): picked mon.noname-i con 0x7f8d1c0a10c0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8d1c0a1ea0 0x7f8d1c0a2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 10 monclient(hunting): picked mon.noname-d con 0x7f8d1c0a1ea0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f8d1c09da80 con 0x7f8d1c0a0480 2024-06-11T19:11:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8d1c0a2aa0 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8d1c0a2d80 con 0x7f8d1c0a10c0 2024-06-11T19:11:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d21dd5640 10 monclient(hunting): authenticate will time out at 902.978612s 2024-06-11T19:11:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1bfff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8d1c0a1ea0 0x7f8d1c0a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1bfff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8d1c0a1ea0 0x7f8d1c0a2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39764/0 (socket says 172.21.15.98:39764) 2024-06-11T19:11:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1bfff640 1 -- 172.21.15.98:0/2164411257 learned_addr learned my addr 172.21.15.98:0/2164411257 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1bfff640 10 monclient(hunting): get_auth_request con 0x7f8d1c0a1ea0 auth_method 0 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d12ffd640 1 --2- 172.21.15.98:0/2164411257 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c0a10c0 0x7f8d1c0a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1b7fe640 1 --2- 172.21.15.98:0/2164411257 >> v2:172.21.15.117:3300/0 conn(0x7f8d1c0a0480 0x7f8d1c0a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1b7fe640 10 monclient(hunting): get_auth_request con 0x7f8d1c0a0480 auth_method 0 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d12ffd640 10 monclient(hunting): get_auth_request con 0x7f8d1c0a10c0 auth_method 0 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d12ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.171+0000 7f8d12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1bfff640 10 monclient(hunting): handle_auth_done global_id 54150 payload 274 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1bfff640 1 -- 172.21.15.98:0/2164411257 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c0a10c0 msgr2=0x7f8d1c0a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1bfff640 1 --2- 172.21.15.98:0/2164411257 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c0a10c0 0x7f8d1c0a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1bfff640 1 -- 172.21.15.98:0/2164411257 >> v2:172.21.15.117:3300/0 conn(0x7f8d1c0a0480 msgr2=0x7f8d1c0a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1bfff640 1 --2- 172.21.15.98:0/2164411257 >> v2:172.21.15.117:3300/0 conn(0x7f8d1c0a0480 0x7f8d1c0a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1bfff640 1 monclient: found mon.noname-d 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1bfff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1bfff640 1 -- 172.21.15.98:0/2164411257 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1c0a3060 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1b7fe640 1 --2- 172.21.15.98:0/2164411257 >> v2:172.21.15.117:3300/0 conn(0x7f8d1c0a0480 0x7f8d1c0a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:11:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1bfff640 1 --2- 172.21.15.98:0/2164411257 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8d1c0a1ea0 0x7f8d1c0a2320 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009e50 tx=0x7f8d1402bdb0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=b96b5d3189bc1031 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1affd640 1 -- 172.21.15.98:0/2164411257 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d1402cbc0 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1affd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1affd640 10 monclient: dump: 2024-06-11T19:11:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.172+0000 7f8d1affd640 10 monclient: _finish_auth 0 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d1affd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d1affd640 1 -- 172.21.15.98:0/2164411257 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d1402cd80 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d1affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 5 monclient: authenticate success, global_id 54150 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 4 set_mon_vals no callback set 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d1affd640 1 -- 172.21.15.98:0/2164411257 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d14034860 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d1affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d1affd640 10 monclient: dump: 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 10 monclient: shutdown 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 1 -- 172.21.15.98:0/2164411257 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8d1c0a1ea0 msgr2=0x7f8d1c0a2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 1 --2- 172.21.15.98:0/2164411257 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8d1c0a1ea0 0x7f8d1c0a2320 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009e50 tx=0x7f8d1402bdb0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 1 -- 172.21.15.98:0/2164411257 shutdown_connections 2024-06-11T19:11:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 1 --2- 172.21.15.98:0/2164411257 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8d1c0a1ea0 0x7f8d1c0a2320 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 1 --2- 172.21.15.98:0/2164411257 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c0a10c0 0x7f8d1c0a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 1 --2- 172.21.15.98:0/2164411257 >> v2:172.21.15.117:3300/0 conn(0x7f8d1c0a0480 0x7f8d1c0a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 1 -- 172.21.15.98:0/2164411257 >> 172.21.15.98:0/2164411257 conn(0x7f8d1c0a3a60 msgr2=0x7f8d1c0a3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 1 -- 172.21.15.98:0/2164411257 shutdown_connections 2024-06-11T19:11:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.173+0000 7f8d21dd5640 1 -- 172.21.15.98:0/2164411257 wait complete. 2024-06-11T19:11:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.174+0000 7f8d21dd5640 10 monclient: build_initial_monmap 2024-06-11T19:11:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.174+0000 7f8d21dd5640 10 monclient: monmap: 2024-06-11T19:11:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:33.191 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.174+0000 7f8d21dd5640 1 librados: starting msgr at 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.174+0000 7f8d21dd5640 1 librados: starting objecter 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.174+0000 7f8d21dd5640 1 Processor -- start 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.174+0000 7f8d21dd5640 1 -- start start 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.174+0000 7f8d21dd5640 1 librados: setting wanted keys 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.174+0000 7f8d21dd5640 1 librados: calling monclient init 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.174+0000 7f8d21dd5640 10 monclient: init 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient: _add_conns ranks=[7,4,0,1,2,3,6,5,8] 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c0a0480 0x7f8d1c1cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient(hunting): picked mon.noname-i con 0x7f8d1c0a0480 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8d1c0a10c0 0x7f8d1c1cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient(hunting): picked mon.noname-c con 0x7f8d1c0a10c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d1c0a1ea0 0x7f8d1c1ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient(hunting): picked mon.noname-a con 0x7f8d1c0a1ea0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c0a0da0 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f8d1c0a4850 con 0x7f8d1c0a10c0 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8d1c0a0f20 con 0x7f8d1c0a0480 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient(hunting): authenticate will time out at 902.982112s 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d12ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8d1c0a10c0 0x7f8d1c1cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d1c0a1ea0 0x7f8d1c1ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d12ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8d1c0a10c0 0x7f8d1c1cde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:49912/0 (socket says 172.21.15.98:49912) 2024-06-11T19:11:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d1c0a1ea0 0x7f8d1c1ce390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43448/0 (socket says 172.21.15.98:43448) 2024-06-11T19:11:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d12ffd640 1 -- 172.21.15.98:0/333398993 learned_addr learned my addr 172.21.15.98:0/333398993 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d12ffd640 10 monclient(hunting): get_auth_request con 0x7f8d1c0a10c0 auth_method 0 2024-06-11T19:11:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d12ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 10 monclient(hunting): get_auth_request con 0x7f8d1c0a1ea0 auth_method 0 2024-06-11T19:11:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1b7fe640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c0a0480 0x7f8d1c1cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1b7fe640 10 monclient(hunting): get_auth_request con 0x7f8d1c0a0480 auth_method 0 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 10 monclient(hunting): handle_auth_done global_id 30924 payload 824 2024-06-11T19:11:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 1 -- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c0a0480 msgr2=0x7f8d1c1cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c0a0480 0x7f8d1c1cd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 1 -- 172.21.15.98:0/333398993 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8d1c0a10c0 msgr2=0x7f8d1c1cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8d1c0a10c0 0x7f8d1c1cde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:11:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 1 monclient: found mon.noname-a 2024-06-11T19:11:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:11:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 1 -- 172.21.15.98:0/333398993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1c1257a0 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d12ffd640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8d1c0a10c0 0x7f8d1c1cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1bfff640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d1c0a1ea0 0x7f8d1c1ce390 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f8d14032000 tx=0x7f8d14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 10 client.?.objecter ms_handle_connect 0x7f8d1c0a1ea0 2024-06-11T19:11:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d14042070 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:11:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 10 monclient: dump: 2024-06-11T19:11:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.198 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 10 monclient: _finish_auth 0 2024-06-11T19:11:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:11:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d140046c0 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 5 monclient: authenticate success, global_id 30924 2024-06-11T19:11:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient: _renew_subs 2024-06-11T19:11:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 1 -- 172.21.15.98:0/333398993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1c126350 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d14009510 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d18ff9640 10 monclient: dump: 2024-06-11T19:11:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 client.30924.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d1a7fc640 4 set_mon_vals no callback set 2024-06-11T19:11:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient: _renew_subs 2024-06-11T19:11:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 1 -- 172.21.15.98:0/333398993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d1c124c20 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.175+0000 7f8d21dd5640 1 librados: init done 2024-06-11T19:11:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.176+0000 7f8d21dd5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.176+0000 7f8d21dd5640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c122db0 0x7f8d1c123190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.176+0000 7f8d21dd5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.176+0000 7f8d21dd5640 1 -- 172.21.15.98:0/333398993 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8d1c126b50 con 0x7f8d1c122db0 2024-06-11T19:11:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.176+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8d14045070 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d1b7fe640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c122db0 0x7f8d1c123190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8cfc03fdb0 0x7f8cfc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d1b7fe640 10 monclient: get_auth_request con 0x7f8d1c122db0 auth_method 0 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 8202+0+0 (secure 0 0 0) 0x7f8d140747b0 con 0x7f8d1c0a1ea0 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 10 client.30924.objecter ms_dispatch 0x7f8d1c000b90 osd_map(76..76 src has 1..76) 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 3 client.30924.objecter handle_osd_map got epochs [76,76] > 0 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 3 client.30924.objecter handle_osd_map decoding full epoch 76 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 20 client.30924.objecter dump_active .. 0 homeless 2024-06-11T19:11:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d12ffd640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8cfc03fdb0 0x7f8cfc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d12ffd640 10 monclient: get_auth_request con 0x7f8cfc03fdb0 auth_method 0 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d1b7fe640 10 monclient(hunting): handle_auth_done global_id 21922 payload 824 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d1b7fe640 1 -- 172.21.15.98:0/333398993 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d1c126520 con 0x7f8d1c122db0 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 10 client.30924.objecter ms_handle_connect 0x7f8d1c122db0 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 10 client.30924.objecter resend_mon_ops 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d1b7fe640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c122db0 0x7f8d1c123190 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c011510 tx=0x7f8d0c0119d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 10 client.30924.objecter ms_handle_connect 0x7f8cfc03fdb0 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d12ffd640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8cfc03fdb0 0x7f8cfc042270 secure :-1 s=READY pgs=3980 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c124490 tx=0x7f8d04008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d0c019020 con 0x7f8d1c122db0 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.177+0000 7f8d18ff9640 10 monclient: dump: 2024-06-11T19:11:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.205 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.178+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8d0c004300 con 0x7f8d1c122db0 2024-06-11T19:11:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.178+0000 7f8d18ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.178+0000 7f8d18ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.178+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c122db0 msgr2=0x7f8d1c123190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.178+0000 7f8d18ff9640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c122db0 0x7f8d1c123190 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c011510 tx=0x7f8d0c0119d0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.178+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c122db0 msgr2=0x7f8d1c123190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.178+0000 7f8d18ff9640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c122db0 0x7f8d1c123190 secure :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c011510 tx=0x7f8d0c0119d0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.184+0000 7f8d21dd5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.184+0000 7f8d21dd5640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c1236d0 0x7f8d1c1be820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.184+0000 7f8d21dd5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.184+0000 7f8d21dd5640 1 -- 172.21.15.98:0/333398993 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8d1c050830 con 0x7f8d1c1236d0 2024-06-11T19:11:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.184+0000 7f8d1b7fe640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c1236d0 0x7f8d1c1be820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.184+0000 7f8d1b7fe640 10 monclient: get_auth_request con 0x7f8d1c1236d0 auth_method 0 2024-06-11T19:11:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.184+0000 7f8d1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.184+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.184+0000 7f8d1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d1b7fe640 10 monclient(hunting): handle_auth_done global_id 21931 payload 824 2024-06-11T19:11:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d1b7fe640 1 -- 172.21.15.98:0/333398993 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8d1c138ea0 con 0x7f8d1c1236d0 2024-06-11T19:11:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 10 client.30924.objecter ms_handle_connect 0x7f8d1c1236d0 2024-06-11T19:11:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 10 client.30924.objecter resend_mon_ops 2024-06-11T19:11:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d1b7fe640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c1236d0 0x7f8d1c1be820 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c011510 tx=0x7f8d0c0046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d0c019040 con 0x7f8d1c1236d0 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 10 monclient: dump: 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.210 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f8d0c003df0 con 0x7f8d1c1236d0 2024-06-11T19:11:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c1236d0 msgr2=0x7f8d1c1be820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c1236d0 0x7f8d1c1be820 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c011510 tx=0x7f8d0c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 1 -- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c1236d0 msgr2=0x7f8d1c1be820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:33.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.185+0000 7f8d18ff9640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d1c1236d0 0x7f8d1c1be820 secure :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c011510 tx=0x7f8d0c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.211 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 143401, 2024-06-11T19:11:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:33.211 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:33.212 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:33.213 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:33.234 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 10 librados: watch_flush enter 2024-06-11T19:11:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 10 librados: watch_flush exit 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 20 client.30924.objecter shutdown clearing up homeless session... 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 10 client.30924.objecter successfully canceled tick 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 1 -- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8cfc03fdb0 msgr2=0x7f8cfc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8cfc03fdb0 0x7f8cfc042270 secure :-1 s=READY pgs=3980 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c124490 tx=0x7f8d04008040 comp rx=0 tx=0).stop 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 10 monclient: shutdown 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 1 -- 172.21.15.98:0/333398993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d1c0a1ea0 msgr2=0x7f8d1c1ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 1 --2- 172.21.15.98:0/333398993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d1c0a1ea0 0x7f8d1c1ce390 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f8d14032000 tx=0x7f8d14004a40 comp rx=0 tx=0).stop 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d1bfff640 1 -- 172.21.15.98:0/333398993 reap_dead start 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d1bfff640 1 -- 172.21.15.98:0/333398993 reap_dead start 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 1 -- 172.21.15.98:0/333398993 shutdown_connections 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 1 -- 172.21.15.98:0/333398993 >> 172.21.15.98:0/333398993 conn(0x7f8d1c0a3a60 msgr2=0x7f8d1c0a6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 1 -- 172.21.15.98:0/333398993 shutdown_connections 2024-06-11T19:11:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 1 -- 172.21.15.98:0/333398993 wait complete. 2024-06-11T19:11:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.186+0000 7f8d21dd5640 1 librados: shutdown 2024-06-11T19:11:33.237 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:33.237 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-06-11T19:11:33.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-06-11T19:11:33.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.336+0000 7fd8af53f640 10 monclient: get_monmap_and_config 2024-06-11T19:11:33.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.336+0000 7fd8af53f640 10 monclient: build_initial_monmap 2024-06-11T19:11:33.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.336+0000 7fd8af53f640 10 monclient: monmap: 2024-06-11T19:11:33.336 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:33.336 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.336 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:33.337341+0000 2024-06-11T19:11:33.336 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:33.337341+0000 2024-06-11T19:11:33.336 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.336+0000 7fd8af53f640 1 Processor -- start 2024-06-11T19:11:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 1 -- start start 2024-06-11T19:11:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 10 monclient: init 2024-06-11T19:11:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 10 monclient: _add_conns ranks=[10,1,7,0,4,6,2,9,5,8,3] 2024-06-11T19:11:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd8a80a2f40 0x7fd8a80a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 10 monclient(hunting): picked mon.noname-f con 0x7fd8a80a2f40 addr v2:172.21.15.117:3300/0 2024-06-11T19:11:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd8a80a4df0 addr v1:172.21.15.117:6789/0 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8a80a5200 0x7fd8a8136de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 10 monclient(hunting): picked mon.noname-i con 0x7fd8a80a5200 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a80a0ed0 con 0x7fd8a80a4df0 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fd8a80a3320 con 0x7fd8a80a2f40 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd8a80a2460 con 0x7fd8a80a5200 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.337+0000 7fd8af53f640 10 monclient(hunting): authenticate will time out at 903.144955s 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd8a80a2f40 0x7fd8a80a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8adadc640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8a80a5200 0x7fd8a8136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd8a80a2f40 0x7fd8a80a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:59374/0 (socket says 172.21.15.98:59374) 2024-06-11T19:11:33.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8adadc640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8a80a5200 0x7fd8a8136de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:36612/0 (socket says 172.21.15.98:36612) 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 1 -- 172.21.15.98:0/387252459 learned_addr learned my addr 172.21.15.98:0/387252459 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 10 monclient(hunting): get_auth_request con 0x7fd8a80a2f40 auth_method 0 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8adadc640 10 monclient(hunting): get_auth_request con 0x7fd8a80a5200 auth_method 0 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8adadc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8adadc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8adadc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8adadc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8adadc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8adadc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 10 monclient(hunting): handle_auth_done global_id 30745 payload 274 2024-06-11T19:11:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 1 -- 172.21.15.98:0/387252459 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8a80a5200 msgr2=0x7fd8a8136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 1 --2- 172.21.15.98:0/387252459 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8a80a5200 0x7fd8a8136de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 1 -- 172.21.15.98:0/387252459 >> v1:172.21.15.117:6789/0 conn(0x7fd8a80a4df0 legacy=0x7fd8a81366d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 10 monclient: _finish_hunting 0 2024-06-11T19:11:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 1 monclient: found mon.noname-f 2024-06-11T19:11:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:11:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.338+0000 7fd8ad2db640 1 -- 172.21.15.98:0/387252459 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a80a2180 con 0x7fd8a80a2f40 2024-06-11T19:11:33.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd8ad2db640 1 --2- 172.21.15.98:0/387252459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8a80a2f40 0x7fd8a80a48b0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fd8980098a0 tx=0x7fd89802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc525572118918f6 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 1 -- 172.21.15.98:0/387252459 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd89802e070 con 0x7fd8a80a2f40 2024-06-11T19:11:33.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:11:33.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 10 monclient: dump: 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.346 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 10 monclient: _finish_auth 0 2024-06-11T19:11:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:11:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 1 -- 172.21.15.98:0/387252459 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd898037070 con 0x7fd8a80a2f40 2024-06-11T19:11:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 1 -- 172.21.15.98:0/387252459 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd898032470 con 0x7fd8a80a2f40 2024-06-11T19:11:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:11:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd89ffff640 10 monclient: dump: 2024-06-11T19:11:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd8af53f640 5 monclient: authenticate success, global_id 30745 2024-06-11T19:11:33.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd8af53f640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:33.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.341+0000 7fd8af53f640 4 set_mon_vals no callback set 2024-06-11T19:11:33.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.342+0000 7fd8af53f640 10 monclient: shutdown 2024-06-11T19:11:33.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.342+0000 7fd8af53f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:33.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.342+0000 7fd8af53f640 1 -- 172.21.15.98:0/387252459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8a80a2f40 msgr2=0x7fd8a80a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.342+0000 7fd8af53f640 1 --2- 172.21.15.98:0/387252459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8a80a2f40 0x7fd8a80a48b0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fd8980098a0 tx=0x7fd89802b880 comp rx=0 tx=0).stop 2024-06-11T19:11:33.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.346+0000 7fd8af53f640 1 -- 172.21.15.98:0/387252459 shutdown_connections 2024-06-11T19:11:33.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.346+0000 7fd8af53f640 1 --2- 172.21.15.98:0/387252459 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8a80a5200 0x7fd8a8136de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.346+0000 7fd8af53f640 1 --2- 172.21.15.98:0/387252459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8a80a2f40 0x7fd8a80a48b0 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.346+0000 7fd8af53f640 1 -- 172.21.15.98:0/387252459 >> 172.21.15.98:0/387252459 conn(0x7fd8a80a95b0 msgr2=0x7fd8a80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:33.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.350+0000 7fd8af53f640 1 -- 172.21.15.98:0/387252459 shutdown_connections 2024-06-11T19:11:33.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.350+0000 7fd8af53f640 1 -- 172.21.15.98:0/387252459 wait complete. 2024-06-11T19:11:33.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.351+0000 7fd8af53f640 10 monclient: build_initial_monmap 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.351+0000 7fd8af53f640 10 monclient: monmap: 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:33.351 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:33.352 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:33.352 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:33.352 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:33.352 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.351+0000 7fd8af53f640 1 librados: starting msgr at 2024-06-11T19:11:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.351+0000 7fd8af53f640 1 librados: starting objecter 2024-06-11T19:11:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.352+0000 7fd8af53f640 1 Processor -- start 2024-06-11T19:11:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.352+0000 7fd8af53f640 1 -- start start 2024-06-11T19:11:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.352+0000 7fd8af53f640 1 librados: setting wanted keys 2024-06-11T19:11:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.352+0000 7fd8af53f640 1 librados: calling monclient init 2024-06-11T19:11:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.352+0000 7fd8af53f640 10 monclient: init 2024-06-11T19:11:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.352+0000 7fd8af53f640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.352+0000 7fd8af53f640 10 monclient: _add_conns ranks=[5,6,8,3,7,4,2,0,1] 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.352+0000 7fd8af53f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd8a80a2f40 0x7fd8a81cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.352+0000 7fd8af53f640 10 monclient(hunting): picked mon.noname-h con 0x7fd8a80a2f40 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.352+0000 7fd8af53f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8a80a4df0 0x7fd8a8135f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.352+0000 7fd8af53f640 10 monclient(hunting): picked mon.noname-d con 0x7fd8a80a4df0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.352+0000 7fd8af53f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8a80a5200 0x7fd8a81364c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8adadc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8a80a5200 0x7fd8a81364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8adadc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8a80a5200 0x7fd8a81364c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:45496/0 (socket says 172.21.15.98:45496) 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8adadc640 1 -- 172.21.15.98:0/1116438020 learned_addr learned my addr 172.21.15.98:0/1116438020 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8af53f640 10 monclient(hunting): picked mon.noname-e con 0x7fd8a80a5200 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8af53f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd8a80a1250 con 0x7fd8a80a2f40 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8af53f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd8a8137320 con 0x7fd8a80a4df0 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8a80a4df0 0x7fd8a8135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8af53f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd8a81296c0 con 0x7fd8a80a5200 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8af53f640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8af53f640 10 monclient(hunting): authenticate will time out at 903.160163s 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8adadc640 10 monclient(hunting): get_auth_request con 0x7fd8a80a5200 auth_method 0 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8adadc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8adadc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8adadc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 10 monclient(hunting): get_auth_request con 0x7fd8a80a4df0 auth_method 0 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8ad2db640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd8a80a2f40 0x7fd8a81cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8ad2db640 10 monclient(hunting): get_auth_request con 0x7fd8a80a2f40 auth_method 0 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8ad2db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8ad2db640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8ad2db640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 10 monclient(hunting): handle_auth_done global_id 54159 payload 824 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 1 -- 172.21.15.98:0/1116438020 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8a80a5200 msgr2=0x7fd8a81364c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8a80a5200 0x7fd8a81364c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 1 -- 172.21.15.98:0/1116438020 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd8a80a2f40 msgr2=0x7fd8a81cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd8a80a2f40 0x7fd8a81cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 10 monclient: _finish_hunting 0 2024-06-11T19:11:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 1 monclient: found mon.noname-d 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.353+0000 7fd8acada640 1 -- 172.21.15.98:0/1116438020 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a81cde00 con 0x7fd8a80a4df0 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.354+0000 7fd8adadc640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8a80a5200 0x7fd8a81364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.354+0000 7fd8acada640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8a80a4df0 0x7fd8a8135f80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400c3f0 tx=0x7fd8a400c8b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.355+0000 7fd89dffb640 10 client.?.objecter ms_handle_connect 0x7fd8a80a4df0 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.355+0000 7fd89dffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.355+0000 7fd89dffb640 1 -- 172.21.15.98:0/1116438020 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8a4017070 con 0x7fd8a80a4df0 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.355+0000 7fd89dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.355+0000 7fd89dffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.355+0000 7fd89dffb640 10 monclient: dump: 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.356+0000 7fd89dffb640 10 monclient: _finish_auth 0 2024-06-11T19:11:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.356+0000 7fd89dffb640 10 monclient: _check_auth_tickets 2024-06-11T19:11:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.356+0000 7fd89dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.356+0000 7fd89dffb640 1 -- 172.21.15.98:0/1116438020 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8a40035f0 con 0x7fd8a80a4df0 2024-06-11T19:11:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.356+0000 7fd89dffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.356+0000 7fd89f7fe640 4 set_mon_vals no callback set 2024-06-11T19:11:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.357+0000 7fd89dffb640 1 -- 172.21.15.98:0/1116438020 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8a4003c10 con 0x7fd8a80a4df0 2024-06-11T19:11:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.357+0000 7fd8af53f640 5 monclient: authenticate success, global_id 54159 2024-06-11T19:11:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.358+0000 7fd8af53f640 10 monclient: _renew_subs 2024-06-11T19:11:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.358+0000 7fd8af53f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.358+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a8127b50 con 0x7fd8a80a4df0 2024-06-11T19:11:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.359+0000 7fd8af53f640 10 client.54159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.359+0000 7fd89dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.359+0000 7fd89dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.359+0000 7fd89dffb640 10 monclient: dump: 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.359+0000 7fd8af53f640 10 monclient: _renew_subs 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.359+0000 7fd8af53f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.359+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a80a47f0 con 0x7fd8a80a4df0 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.359+0000 7fd8af53f640 1 librados: init done 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.361+0000 7fd8af53f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.361+0000 7fd8af53f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.361+0000 7fd8af53f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.361+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8a80a06d0 con 0x7fd8a80a4df0 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.364+0000 7fd89dffb640 1 -- 172.21.15.98:0/1116438020 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd8a40037a0 con 0x7fd8a80a4df0 2024-06-11T19:11:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.364+0000 7fd89dffb640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd88c03fd40 0x7fd88c042200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.364+0000 7fd89dffb640 1 -- 172.21.15.98:0/1116438020 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(77..77 src has 1..77) ==== 7944+0+0 (secure 0 0 0) 0x7fd8a4051430 con 0x7fd8a80a4df0 2024-06-11T19:11:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.364+0000 7fd89dffb640 10 client.54159.objecter ms_dispatch 0x7fd8a8000b90 osd_map(77..77 src has 1..77) 2024-06-11T19:11:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.364+0000 7fd89dffb640 3 client.54159.objecter handle_osd_map got epochs [77,77] > 0 2024-06-11T19:11:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.364+0000 7fd89dffb640 3 client.54159.objecter handle_osd_map decoding full epoch 77 2024-06-11T19:11:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.364+0000 7fd89dffb640 20 client.54159.objecter dump_active .. 0 homeless 2024-06-11T19:11:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.366+0000 7fd8ad2db640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd88c03fd40 0x7fd88c042200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.366+0000 7fd8ad2db640 10 monclient: get_auth_request con 0x7fd88c03fd40 auth_method 0 2024-06-11T19:11:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.367+0000 7fd8ad2db640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd88c03fd40 0x7fd88c042200 secure :-1 s=READY pgs=3981 cs=0 l=1 rev1=1 crypto rx=0x7fd8980098a0 tx=0x7fd898008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.367+0000 7fd89dffb640 10 client.54159.objecter ms_handle_connect 0x7fd88c03fd40 2024-06-11T19:11:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.371+0000 7fd89dffb640 1 -- 172.21.15.98:0/1116438020 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd8a80a06d0 con 0x7fd8a80a4df0 2024-06-11T19:11:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.371+0000 7fd89dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.371+0000 7fd89dffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.629+0000 7fd8af53f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-06-11T19:11:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.629+0000 7fd8af53f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-06-11T19:11:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.629+0000 7fd8af53f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.629+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd8a80a6a90 con 0x7fd8a80a4df0 2024-06-11T19:11:33.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.631+0000 7fd89dffb640 1 -- 172.21.15.98:0/1116438020 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd8a4050070 con 0x7fd8a80a4df0 2024-06-11T19:11:33.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.631+0000 7fd89dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-06-11T19:11:33.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.631+0000 7fd89dffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:33.635 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.635 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=9,logm=66} crc {auth=2776523883,config=692619927,health=1705939578,kv=1471832156,logm=3694693697}) 2024-06-11T19:11:33.635 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=10,kv=10,logm=66} crc {auth=2776523883,config=692619927,health=55874111,kv=2000936593,logm=3001200957}) 2024-06-11T19:11:33.635 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.635 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=9,logm=66} crc {auth=2776523883,config=692619927,health=1705939578,kv=1471832156,logm=3694693697}) 2024-06-11T19:11:33.635 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=12,kv=9,logm=66} crc {auth=3854336438,config=692619927,health=803908494,kv=1471832156,logm=888556580}) 2024-06-11T19:11:33.635 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.635 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=9,logm=66} crc {auth=2776523883,config=692619927,health=1705939578,kv=1471832156,logm=3694693697}) 2024-06-11T19:11:33.635 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=11,config=2,health=12,kv=10,logm=65} crc {auth=2054920936,config=692619927,health=4071555382,kv=2000936593,logm=4108365484}) 2024-06-11T19:11:33.636 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.636 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=9,logm=66} crc {auth=2776523883,config=692619927,health=1705939578,kv=1471832156,logm=3694693697}) 2024-06-11T19:11:33.636 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=2776523883,config=692619927,health=1878276634,kv=2000936593,logm=334768205}) 2024-06-11T19:11:33.636 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.636 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=9,logm=66} crc {auth=2776523883,config=692619927,health=1705939578,kv=1471832156,logm=3694693697}) 2024-06-11T19:11:33.636 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=12,config=2,health=12,kv=10,logm=64} crc {auth=3941922418,config=692619927,health=803908494,kv=2000936593,logm=750192681}) 2024-06-11T19:11:33.636 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.636 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=9,logm=66} crc {auth=2776523883,config=692619927,health=1705939578,kv=1471832156,logm=3694693697}) 2024-06-11T19:11:33.636 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=2776523883,config=692619927,health=4127293036,kv=2000936593,logm=3588599111}) 2024-06-11T19:11:33.636 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.631+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.636 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.632+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=9,logm=66} crc {auth=2776523883,config=692619927,health=1705939578,kv=1471832156,logm=3694693697}) 2024-06-11T19:11:33.636 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.632+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=11,config=2,health=11,kv=10,logm=66} crc {auth=3297963994,config=692619927,health=2317794993,kv=2000936593,logm=2366576576}) 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.632+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.632+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=9,logm=66} crc {auth=2776523883,config=692619927,health=1705939578,kv=1471832156,logm=3694693697}) 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.632+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=12,config=2,health=12,kv=10,logm=64} crc {auth=2776523883,config=692619927,health=803908495,kv=2000936593,logm=1582405121}) 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901243093}) 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3831270191}) 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901243093}) 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565850695}) 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901243093}) 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=666078549}) 2024-06-11T19:11:33.637 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901243093}) 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=394248747}) 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901243093}) 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3799506992}) 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901243093}) 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3429913263}) 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901243093}) 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4132501931}) 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.638 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901243093}) 2024-06-11T19:11:33.639 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.633+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=280848908}) 2024-06-11T19:11:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.633+0000 7fd8af53f640 10 librados: watch_flush enter 2024-06-11T19:11:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 10 librados: watch_flush exit 2024-06-11T19:11:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 20 client.54159.objecter shutdown clearing up homeless session... 2024-06-11T19:11:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 10 client.54159.objecter successfully canceled tick 2024-06-11T19:11:33.640 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.634+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.640 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.634+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027317980}) 2024-06-11T19:11:33.640 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.634+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3558149024}) 2024-06-11T19:11:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd88c03fd40 msgr2=0x7fd88c042200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd88c03fd40 0x7fd88c042200 secure :-1 s=READY pgs=3981 cs=0 l=1 rev1=1 crypto rx=0x7fd8980098a0 tx=0x7fd898008030 comp rx=0 tx=0).stop 2024-06-11T19:11:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 10 monclient: shutdown 2024-06-11T19:11:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8a80a4df0 msgr2=0x7fd8a8135f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:33.641 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.634+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.641 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.634+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027317980}) 2024-06-11T19:11:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8a80a4df0 0x7fd8a8135f80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400c3f0 tx=0x7fd8a400c8b0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.642 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.634+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=365412530}) 2024-06-11T19:11:33.642 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.634+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.642 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.634+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027317980}) 2024-06-11T19:11:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 shutdown_connections 2024-06-11T19:11:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd88c03fd40 0x7fd88c042200 unknown :-1 s=CLOSED pgs=3981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.643 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.634+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2483524199}) 2024-06-11T19:11:33.643 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8a80a4df0 0x7fd8a8135f80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8a80a5200 0x7fd8a81364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.644 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027317980}) 2024-06-11T19:11:33.644 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1993652465}) 2024-06-11T19:11:33.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.634+0000 7fd8af53f640 1 --2- 172.21.15.98:0/1116438020 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd8a80a2f40 0x7fd8a81cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:33.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.635+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 >> 172.21.15.98:0/1116438020 conn(0x7fd8a80a95b0 msgr2=0x7fd8a809dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:33.644 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.644 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027317980}) 2024-06-11T19:11:33.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.635+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 shutdown_connections 2024-06-11T19:11:33.645 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4001197094}) 2024-06-11T19:11:33.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.635+0000 7fd8af53f640 1 -- 172.21.15.98:0/1116438020 wait complete. 2024-06-11T19:11:33.646 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.646 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027317980}) 2024-06-11T19:11:33.646 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2922700083}) 2024-06-11T19:11:33.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:33.635+0000 7fd8af53f640 1 librados: shutdown 2024-06-11T19:11:33.662 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.662 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027317980}) 2024-06-11T19:11:33.662 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=17180869}) 2024-06-11T19:11:33.662 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.662 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027317980}) 2024-06-11T19:11:33.662 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.635+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1339273366}) 2024-06-11T19:11:33.662 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.662 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156997803}) 2024-06-11T19:11:33.662 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=814125210}) 2024-06-11T19:11:33.662 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.662 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156997803}) 2024-06-11T19:11:33.663 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2647333525}) 2024-06-11T19:11:33.663 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.663 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156997803}) 2024-06-11T19:11:33.663 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859281632}) 2024-06-11T19:11:33.663 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.663 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156997803}) 2024-06-11T19:11:33.663 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=692504394}) 2024-06-11T19:11:33.663 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.663 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156997803}) 2024-06-11T19:11:33.663 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3938562069}) 2024-06-11T19:11:33.663 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.663 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156997803}) 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=19354595}) 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156997803}) 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1833789999}) 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156997803}) 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.636+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3862518048}) 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217888437}) 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1036979256}) 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217888437}) 2024-06-11T19:11:33.664 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969592630}) 2024-06-11T19:11:33.665 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.665 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217888437}) 2024-06-11T19:11:33.665 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1005544865}) 2024-06-11T19:11:33.665 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.665 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217888437}) 2024-06-11T19:11:33.665 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3663394986}) 2024-06-11T19:11:33.665 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.665 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217888437}) 2024-06-11T19:11:33.665 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3336206978}) 2024-06-11T19:11:33.665 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.665 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217888437}) 2024-06-11T19:11:33.665 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=576831422}) 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217888437}) 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=840679296}) 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217888437}) 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.637+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3704505780}) 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.638+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.638+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321338564}) 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.638+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1165397245}) 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.638+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.638+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321338564}) 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.638+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1953915171}) 2024-06-11T19:11:33.666 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.638+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.667 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.638+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321338564}) 2024-06-11T19:11:33.667 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.638+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3849233559}) 2024-06-11T19:11:33.667 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.638+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.667 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321338564}) 2024-06-11T19:11:33.667 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2161018797}) 2024-06-11T19:11:33.667 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.667 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321338564}) 2024-06-11T19:11:33.667 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=312262644}) 2024-06-11T19:11:33.667 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.667 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321338564}) 2024-06-11T19:11:33.667 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=857620789}) 2024-06-11T19:11:33.667 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.668 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321338564}) 2024-06-11T19:11:33.668 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2005426687}) 2024-06-11T19:11:33.668 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.668 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321338564}) 2024-06-11T19:11:33.668 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.639+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=352699760}) 2024-06-11T19:11:33.668 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.668 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786820203}) 2024-06-11T19:11:33.668 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2552641395}) 2024-06-11T19:11:33.668 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.668 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786820203}) 2024-06-11T19:11:33.668 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867716195}) 2024-06-11T19:11:33.668 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786820203}) 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2001051030}) 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786820203}) 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2945219296}) 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786820203}) 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=333773903}) 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786820203}) 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3986122660}) 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.669 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786820203}) 2024-06-11T19:11:33.670 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=743231611}) 2024-06-11T19:11:33.670 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.670 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786820203}) 2024-06-11T19:11:33.670 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.640+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4058334137}) 2024-06-11T19:11:33.670 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.670 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071538131}) 2024-06-11T19:11:33.670 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1728832109}) 2024-06-11T19:11:33.670 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.670 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071538131}) 2024-06-11T19:11:33.670 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1954153066}) 2024-06-11T19:11:33.670 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071538131}) 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2911313486}) 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071538131}) 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2952334780}) 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071538131}) 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=190637166}) 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071538131}) 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=617383422}) 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071538131}) 2024-06-11T19:11:33.671 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1679301409}) 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071538131}) 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.641+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3470138625}) 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309101542}) 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2881723914}) 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309101542}) 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596101849}) 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309101542}) 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2121414172}) 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.672 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309101542}) 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1758901903}) 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309101542}) 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3807633046}) 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309101542}) 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4058763408}) 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309101542}) 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3961174458}) 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309101542}) 2024-06-11T19:11:33.673 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.643+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=770369224}) 2024-06-11T19:11:33.674 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.674 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387503350}) 2024-06-11T19:11:33.674 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2921699898}) 2024-06-11T19:11:33.674 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.674 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387503350}) 2024-06-11T19:11:33.674 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3699935829}) 2024-06-11T19:11:33.674 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.674 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387503350}) 2024-06-11T19:11:33.674 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4088110874}) 2024-06-11T19:11:33.674 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.674 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387503350}) 2024-06-11T19:11:33.674 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2989674185}) 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387503350}) 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1286484456}) 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387503350}) 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1024212395}) 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387503350}) 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3945093135}) 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387503350}) 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.644+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3001258350}) 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.675 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642273584}) 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3075164651}) 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642273584}) 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1425902834}) 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642273584}) 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2088339117}) 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642273584}) 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1554077837}) 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642273584}) 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1118093756}) 2024-06-11T19:11:33.676 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642273584}) 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2342590293}) 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642273584}) 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1015606518}) 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.645+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.646+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642273584}) 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.646+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=713323608}) 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308762377}) 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2561162460}) 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.677 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308762377}) 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=301489521}) 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308762377}) 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=242698927}) 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308762377}) 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=407631601}) 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308762377}) 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2229409369}) 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308762377}) 2024-06-11T19:11:33.678 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2515668009}) 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308762377}) 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2912643797}) 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308762377}) 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.647+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4061607486}) 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980831815}) 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=830889267}) 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980831815}) 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258740544}) 2024-06-11T19:11:33.679 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980831815}) 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3705653795}) 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980831815}) 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2314238925}) 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980831815}) 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2466724706}) 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980831815}) 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2876520643}) 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.680 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980831815}) 2024-06-11T19:11:33.681 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=814258211}) 2024-06-11T19:11:33.681 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.681 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980831815}) 2024-06-11T19:11:33.681 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.648+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3984669102}) 2024-06-11T19:11:33.682 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-06-11T19:11:33.686 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.687 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039509869}) 2024-06-11T19:11:33.687 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1833536527}) 2024-06-11T19:11:33.687 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.687 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039509869}) 2024-06-11T19:11:33.687 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2390830736}) 2024-06-11T19:11:33.687 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.687 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039509869}) 2024-06-11T19:11:33.687 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=544163885}) 2024-06-11T19:11:33.687 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.687 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039509869}) 2024-06-11T19:11:33.687 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1351575869}) 2024-06-11T19:11:33.687 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039509869}) 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1289832278}) 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039509869}) 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2655600414}) 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039509869}) 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1196294964}) 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039509869}) 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.649+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2934241268}) 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.688 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=26} crc {logm=2206495443,mdsmap=3140514009,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=67521360,osdmap=3258389416}) 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=12,osdmap=25} crc {logm=1507580180,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=1159444590,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=2241437300,osdmap=2691277469}) 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=26} crc {logm=2206495443,mdsmap=3140514009,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=67521360,osdmap=3258389416}) 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=24} crc {logm=1507580180,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=3411533383,osdmap=1784844554}) 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=26} crc {logm=2206495443,mdsmap=3140514009,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=67521360,osdmap=3258389416}) 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=25} crc {logm=1507580180,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=955939217,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=3411533383,osdmap=4042371401}) 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=26} crc {logm=2206495443,mdsmap=3140514009,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=67521360,osdmap=3258389416}) 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=13,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=12,osdmap=28} crc {logm=1426132297,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769103,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=2314039359,monmap=506305176,osd_metadata=1424309990,osd_pg_creating=1845283144,osd_snap=2241437300,osdmap=2686893937}) 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=26} crc {logm=2206495443,mdsmap=3140514009,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=67521360,osdmap=3258389416}) 2024-06-11T19:11:33.689 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=12,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=29} crc {logm=652271540,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2315787112,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=3411533383,osdmap=427513268}) 2024-06-11T19:11:33.690 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.690 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=26} crc {logm=2206495443,mdsmap=3140514009,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=67521360,osdmap=3258389416}) 2024-06-11T19:11:33.690 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=14,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=27} crc {logm=2351235528,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=1624270173,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=41437130,osdmap=1678357717}) 2024-06-11T19:11:33.690 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.690 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=26} crc {logm=2206495443,mdsmap=3140514009,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=67521360,osdmap=3258389416}) 2024-06-11T19:11:33.690 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=3,osd_pg_creating=1,osd_snap=13,osdmap=28} crc {logm=1507580180,mdsmap=2840952586,mgr=2102212137,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=2314039359,monmap=506305176,osd_metadata=1948959616,osd_pg_creating=1845283144,osd_snap=3411533383,osdmap=2903773930}) 2024-06-11T19:11:33.690 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.690 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=14,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=26} crc {logm=2206495443,mdsmap=3140514009,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=67521360,osdmap=3258389416}) 2024-06-11T19:11:33.690 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.651+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=24} crc {logm=1507580180,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1022276456,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1845283144,osd_snap=3411533383,osdmap=717762916}) 2024-06-11T19:11:33.690 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.690 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1567520678}) 2024-06-11T19:11:33.690 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2771529740}) 2024-06-11T19:11:33.691 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.691 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1567520678}) 2024-06-11T19:11:33.691 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3636845872}) 2024-06-11T19:11:33.691 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.691 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1567520678}) 2024-06-11T19:11:33.691 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=99} crc {osdmap=2969851490}) 2024-06-11T19:11:33.691 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.691 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1567520678}) 2024-06-11T19:11:33.691 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1606472013}) 2024-06-11T19:11:33.691 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.691 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1567520678}) 2024-06-11T19:11:33.691 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=98} crc {osdmap=2382917645}) 2024-06-11T19:11:33.692 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.692 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1567520678}) 2024-06-11T19:11:33.692 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=762403060}) 2024-06-11T19:11:33.692 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.692 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1567520678}) 2024-06-11T19:11:33.692 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=296488200}) 2024-06-11T19:11:33.692 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:11:33.692 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1567520678}) 2024-06-11T19:11:33.692 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-06-11T19:11:33.656+0000 7f6fda98c640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1806026070}) 2024-06-11T19:11:34.683 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:11:34.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:11:34.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.784+0000 7f2a2a3da640 10 monclient: get_monmap_and_config 2024-06-11T19:11:34.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.784+0000 7f2a2a3da640 10 monclient: build_initial_monmap 2024-06-11T19:11:34.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.784+0000 7f2a2a3da640 10 monclient: monmap: 2024-06-11T19:11:34.784 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:34.784 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:34.784 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:34.785661+0000 2024-06-11T19:11:34.784 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:34.785661+0000 2024-06-11T19:11:34.784 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:34.785 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:34.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 1 Processor -- start 2024-06-11T19:11:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 1 -- start start 2024-06-11T19:11:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 10 monclient: init 2024-06-11T19:11:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 10 monclient: _add_conns ranks=[9,10,0,3,4,1,8,2,5,7,6] 2024-06-11T19:11:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2a240a4920 0x7f2a240a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 10 monclient(hunting): picked mon.noname-a con 0x7f2a240a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:11:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2a240a5240 0x7f2a240a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 10 monclient(hunting): picked mon.noname-f con 0x7f2a240a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:11:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a240a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:11:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a240a7140 con 0x7f2a240a2530 2024-06-11T19:11:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2a240a2b50 con 0x7f2a240a4920 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f2a240a2e30 con 0x7f2a240a5240 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.785+0000 7f2a2a3da640 10 monclient(hunting): authenticate will time out at 904.592987s 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2a240a4920 0x7f2a240a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a28977640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2a240a2530 0x7f2a240a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51582/0 (socket says 172.21.15.98:51582) 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a28977640 1 -- 172.21.15.98:0/3782462547 learned_addr learned my addr 172.21.15.98:0/3782462547 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 10 monclient(hunting): get_auth_request con 0x7f2a240a4920 auth_method 0 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a237fe640 1 --2- 172.21.15.98:0/3782462547 >> v2:172.21.15.117:3300/0 conn(0x7f2a240a5240 0x7f2a240a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a237fe640 10 monclient(hunting): get_auth_request con 0x7f2a240a5240 auth_method 0 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a237fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a237fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 10 monclient(hunting): handle_auth_done global_id 30978 payload 274 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 1 -- 172.21.15.98:0/3782462547 >> v2:172.21.15.117:3300/0 conn(0x7f2a240a5240 msgr2=0x7f2a240a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 1 --2- 172.21.15.98:0/3782462547 >> v2:172.21.15.117:3300/0 conn(0x7f2a240a5240 0x7f2a240a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 1 -- 172.21.15.98:0/3782462547 >> v1:172.21.15.98:6789/0 conn(0x7f2a240a2530 legacy=0x7f2a240a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:34.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 10 monclient: _finish_hunting 0 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 1 monclient: found mon.noname-a 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a23fff640 1 -- 172.21.15.98:0/3782462547 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a240a3110 con 0x7f2a240a4920 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a22ffd640 1 -- 172.21.15.98:0/3782462547 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2739135526 0 0) 0x7f2a240a7140 con 0x7f2a240a2530 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.786+0000 7f2a22ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a23fff640 1 --2- 172.21.15.98:0/3782462547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a240a4920 0x7f2a240a4d00 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f2a24073fc0 tx=0x7f2a1402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1377e81f7c718d3a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 1 -- 172.21.15.98:0/3782462547 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2a14038070 con 0x7f2a240a4920 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 10 monclient: dump: 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:34.798 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 10 monclient: _finish_auth 0 2024-06-11T19:11:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 1 -- 172.21.15.98:0/3782462547 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a14031980 con 0x7f2a240a4920 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a2a3da640 5 monclient: authenticate success, global_id 30978 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a2a3da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a2a3da640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a2a3da640 4 set_mon_vals no callback set 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 1 -- 172.21.15.98:0/3782462547 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2a140323a0 con 0x7f2a240a4920 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a22ffd640 10 monclient: dump: 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a2a3da640 10 monclient: shutdown 2024-06-11T19:11:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a2a3da640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3782462547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a240a4920 msgr2=0x7f2a240a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a2a3da640 1 --2- 172.21.15.98:0/3782462547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a240a4920 0x7f2a240a4d00 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f2a24073fc0 tx=0x7f2a1402b7b0 comp rx=0 tx=0).stop 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3782462547 shutdown_connections 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a2a3da640 1 --2- 172.21.15.98:0/3782462547 >> v2:172.21.15.117:3300/0 conn(0x7f2a240a5240 0x7f2a240a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a2a3da640 1 --2- 172.21.15.98:0/3782462547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a240a4920 0x7f2a240a4d00 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.787+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3782462547 >> 172.21.15.98:0/3782462547 conn(0x7f2a240ab850 msgr2=0x7f2a240a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.788+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3782462547 shutdown_connections 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.788+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3782462547 wait complete. 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.788+0000 7f2a2a3da640 10 monclient: build_initial_monmap 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.788+0000 7f2a2a3da640 10 monclient: monmap: 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.788+0000 7f2a2a3da640 1 librados: starting msgr at 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.788+0000 7f2a2a3da640 1 librados: starting objecter 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.788+0000 7f2a2a3da640 1 Processor -- start 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 1 -- start start 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 1 librados: setting wanted keys 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 1 librados: calling monclient init 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 10 monclient: init 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 10 monclient: _add_conns ranks=[3,4,0,8,5,1,6,2,7] 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a240a2530 0x7f2a241cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 10 monclient(hunting): picked mon.noname-g con 0x7f2a240a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a240a4920 0x7f2a241cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 10 monclient(hunting): picked mon.noname-c con 0x7f2a240a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:34.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a240a5240 0x7f2a241250d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 10 monclient(hunting): picked mon.noname-a con 0x7f2a240a5240 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a240a6da0 con 0x7f2a240a5240 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f2a24138e50 con 0x7f2a240a2530 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2a240a65f0 con 0x7f2a240a4920 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a2a3da640 10 monclient(hunting): authenticate will time out at 904.596549s 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a240a5240 0x7f2a241250d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a240a5240 0x7f2a241250d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43486/0 (socket says 172.21.15.98:43486) 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a237fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a240a4920 0x7f2a241cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 1 -- 172.21.15.98:0/3112384879 learned_addr learned my addr 172.21.15.98:0/3112384879 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 10 monclient(hunting): get_auth_request con 0x7f2a240a5240 auth_method 0 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:34.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a237fe640 10 monclient(hunting): get_auth_request con 0x7f2a240a4920 auth_method 0 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a237fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a237fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a23fff640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a240a2530 0x7f2a241cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a23fff640 10 monclient(hunting): get_auth_request con 0x7f2a240a2530 auth_method 0 2024-06-11T19:11:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a23fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a23fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 10 monclient(hunting): handle_auth_done global_id 30996 payload 824 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 1 -- 172.21.15.98:0/3112384879 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a240a4920 msgr2=0x7f2a241cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a240a4920 0x7f2a241cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 1 -- 172.21.15.98:0/3112384879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a240a2530 msgr2=0x7f2a241cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a240a2530 0x7f2a241cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 10 monclient: _finish_hunting 0 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 1 monclient: found mon.noname-a 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:11:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a28977640 1 -- 172.21.15.98:0/3112384879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a24127780 con 0x7f2a240a5240 2024-06-11T19:11:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.789+0000 7f2a237fe640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a240a4920 0x7f2a241cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a28977640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a240a5240 0x7f2a241250d0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00c770 tx=0x7f2a1c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 10 client.?.objecter ms_handle_connect 0x7f2a240a5240 2024-06-11T19:11:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 1 -- 172.21.15.98:0/3112384879 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2a1c018070 con 0x7f2a240a5240 2024-06-11T19:11:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:11:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 10 monclient: dump: 2024-06-11T19:11:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:34.810 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 10 monclient: _finish_auth 0 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 1 -- 172.21.15.98:0/3112384879 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a1c004d70 con 0x7f2a240a5240 2024-06-11T19:11:34.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a2a3da640 5 monclient: authenticate success, global_id 30996 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a2a3da640 10 monclient: _renew_subs 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a2a3da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3112384879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a24128ca0 con 0x7f2a240a5240 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 1 -- 172.21.15.98:0/3112384879 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2a1c0070d0 con 0x7f2a240a5240 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a20ff9640 10 monclient: dump: 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:34.812 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a2a3da640 10 client.30996.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a2a3da640 10 monclient: _renew_subs 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a2a3da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a227fc640 4 set_mon_vals no callback set 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3112384879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a24138720 con 0x7f2a240a5240 2024-06-11T19:11:34.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.790+0000 7f2a2a3da640 1 librados: init done 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a2a3da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a2a3da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a2a3da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3112384879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a240696d0 con 0x7f2a240a5240 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a20ff9640 1 -- 172.21.15.98:0/3112384879 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f2a1c007270 con 0x7f2a240a5240 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a20ff9640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2a1003fde0 0x7f2a100422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a20ff9640 1 -- 172.21.15.98:0/3112384879 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7498+0+0 (secure 0 0 0) 0x7f2a1c051390 con 0x7f2a240a5240 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a20ff9640 10 client.30996.objecter ms_dispatch 0x7f2a24000b90 osd_map(78..78 src has 1..78) 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a20ff9640 3 client.30996.objecter handle_osd_map got epochs [78,78] > 0 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a20ff9640 3 client.30996.objecter handle_osd_map decoding full epoch 78 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a20ff9640 20 client.30996.objecter dump_active .. 0 homeless 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a23fff640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2a1003fde0 0x7f2a100422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:34.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.791+0000 7f2a23fff640 10 monclient: get_auth_request con 0x7f2a1003fde0 auth_method 0 2024-06-11T19:11:34.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.792+0000 7f2a20ff9640 10 client.30996.objecter ms_handle_connect 0x7f2a1003fde0 2024-06-11T19:11:34.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.792+0000 7f2a23fff640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2a1003fde0 0x7f2a100422a0 secure :-1 s=READY pgs=3985 cs=0 l=1 rev1=1 crypto rx=0x7f2a14035000 tx=0x7f2a14008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:34.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.796+0000 7f2a20ff9640 1 -- 172.21.15.98:0/3112384879 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2a1c01a310 con 0x7f2a240a5240 2024-06-11T19:11:34.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.796+0000 7f2a20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:34.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:34.796+0000 7f2a20ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:35.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.075+0000 7f2a2a3da640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:11:35.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.075+0000 7f2a2a3da640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:11:35.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.075+0000 7f2a2a3da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:35.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.075+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3112384879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2a240a4d00 con 0x7f2a240a5240 2024-06-11T19:11:35.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.075+0000 7f2a20ff9640 1 -- 172.21.15.98:0/3112384879 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f2a1c01bda0 con 0x7f2a240a5240 2024-06-11T19:11:35.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.075+0000 7f2a20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:11:35.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.075+0000 7f2a20ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:35.076 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":12,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:11:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.077+0000 7f2a2a3da640 10 librados: watch_flush enter 2024-06-11T19:11:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.077+0000 7f2a2a3da640 10 librados: watch_flush exit 2024-06-11T19:11:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.077+0000 7f2a2a3da640 20 client.30996.objecter shutdown clearing up homeless session... 2024-06-11T19:11:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.077+0000 7f2a2a3da640 10 client.30996.objecter successfully canceled tick 2024-06-11T19:11:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.077+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3112384879 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2a1003fde0 msgr2=0x7f2a100422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.077+0000 7f2a2a3da640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2a1003fde0 0x7f2a100422a0 secure :-1 s=READY pgs=3985 cs=0 l=1 rev1=1 crypto rx=0x7f2a14035000 tx=0x7f2a14008030 comp rx=0 tx=0).stop 2024-06-11T19:11:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.077+0000 7f2a2a3da640 10 monclient: shutdown 2024-06-11T19:11:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.077+0000 7f2a2a3da640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.077+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3112384879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a240a5240 msgr2=0x7f2a241250d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.077+0000 7f2a2a3da640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a240a5240 0x7f2a241250d0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00c770 tx=0x7f2a1c00cc30 comp rx=0 tx=0).stop 2024-06-11T19:11:35.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.078+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3112384879 shutdown_connections 2024-06-11T19:11:35.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.078+0000 7f2a2a3da640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2a1003fde0 0x7f2a100422a0 unknown :-1 s=CLOSED pgs=3985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.078+0000 7f2a2a3da640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a240a5240 0x7f2a241250d0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.078+0000 7f2a2a3da640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a240a4920 0x7f2a241cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.078+0000 7f2a2a3da640 1 --2- 172.21.15.98:0/3112384879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a240a2530 0x7f2a241cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.078+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3112384879 >> 172.21.15.98:0/3112384879 conn(0x7f2a240ab850 msgr2=0x7f2a2409c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:35.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.078+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3112384879 shutdown_connections 2024-06-11T19:11:35.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.078+0000 7f2a2a3da640 1 -- 172.21.15.98:0/3112384879 wait complete. 2024-06-11T19:11:35.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.078+0000 7f2a2a3da640 1 librados: shutdown 2024-06-11T19:11:35.088 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:11:35.088 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-06-11T19:11:35.088 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:11:35.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:11:35.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.187+0000 7f45cf8c0640 10 monclient: get_monmap_and_config 2024-06-11T19:11:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.187+0000 7f45cf8c0640 10 monclient: build_initial_monmap 2024-06-11T19:11:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.187+0000 7f45cf8c0640 10 monclient: monmap: 2024-06-11T19:11:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:35.188330+0000 2024-06-11T19:11:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:35.188330+0000 2024-06-11T19:11:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:35.188 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:35.188 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:35.188 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:35.188 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:35.188 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:35.188 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:35.188 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:35.188 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:35.188 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.187+0000 7f45cf8c0640 1 Processor -- start 2024-06-11T19:11:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 1 -- start start 2024-06-11T19:11:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 10 monclient: init 2024-06-11T19:11:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 10 monclient: _add_conns ranks=[7,4,8,5,10,9,0,1,2,3,6] 2024-06-11T19:11:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45c80a4920 0x7f45c80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 10 monclient(hunting): picked mon.noname-i con 0x7f45c80a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45c80a5240 0x7f45c80a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 10 monclient(hunting): picked mon.noname-c con 0x7f45c80a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c80a2530 0x7f45c80a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 10 monclient(hunting): picked mon.noname-e con 0x7f45c80a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:35.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f45c80a7140 con 0x7f45c80a5240 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f45c80a2b50 con 0x7f45c80a4920 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f45c80a2e30 con 0x7f45c80a2530 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cf8c0640 10 monclient(hunting): authenticate will time out at 904.995543s 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45c80a5240 0x7f45c80a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45c80a5240 0x7f45c80a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:49944/0 (socket says 172.21.15.98:49944) 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 1 -- 172.21.15.98:0/551226479 learned_addr learned my addr 172.21.15.98:0/551226479 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 10 monclient(hunting): get_auth_request con 0x7f45c80a5240 auth_method 0 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cde5d640 1 --2- 172.21.15.98:0/551226479 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c80a2530 0x7f45c80a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cde5d640 10 monclient(hunting): get_auth_request con 0x7f45c80a2530 auth_method 0 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cde5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cde5d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cde5d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cd65c640 1 --2- 172.21.15.98:0/551226479 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45c80a4920 0x7f45c80a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cde5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cde5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cde5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cd65c640 10 monclient(hunting): get_auth_request con 0x7f45c80a4920 auth_method 0 2024-06-11T19:11:35.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cd65c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cd65c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cd65c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 10 monclient(hunting): handle_auth_done global_id 22837 payload 274 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 1 -- 172.21.15.98:0/551226479 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c80a2530 msgr2=0x7f45c80a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 1 --2- 172.21.15.98:0/551226479 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c80a2530 0x7f45c80a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 1 -- 172.21.15.98:0/551226479 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45c80a4920 msgr2=0x7f45c80a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 1 --2- 172.21.15.98:0/551226479 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45c80a4920 0x7f45c80a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 10 monclient: _finish_hunting 0 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 1 monclient: found mon.noname-c 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cce5b640 1 -- 172.21.15.98:0/551226479 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c80a3110 con 0x7f45c80a5240 2024-06-11T19:11:35.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.188+0000 7f45cd65c640 1 --2- 172.21.15.98:0/551226479 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45c80a4920 0x7f45c80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45cde5d640 1 --2- 172.21.15.98:0/551226479 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c80a2530 0x7f45c80a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45cce5b640 1 --2- 172.21.15.98:0/551226479 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45c80a5240 0x7f45c80a59a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f45b80098a0 tx=0x7f45b802b7a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=45cf2686ae994495 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 1 -- 172.21.15.98:0/551226479 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45b8038070 con 0x7f45c80a5240 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 10 monclient: dump: 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.203 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 10 monclient: _finish_auth 0 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 10 monclient: _check_auth_tickets 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 1 -- 172.21.15.98:0/551226479 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b8031970 con 0x7f45c80a5240 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45cf8c0640 5 monclient: authenticate success, global_id 22837 2024-06-11T19:11:35.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45cf8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45cf8c0640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 1 -- 172.21.15.98:0/551226479 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45b8031ba0 con 0x7f45c80a5240 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45cf8c0640 4 set_mon_vals no callback set 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.189+0000 7f45bffff640 10 monclient: dump: 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.205 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.190+0000 7f45cf8c0640 10 monclient: shutdown 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.190+0000 7f45cf8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.190+0000 7f45cf8c0640 1 -- 172.21.15.98:0/551226479 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45c80a5240 msgr2=0x7f45c80a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.190+0000 7f45cf8c0640 1 --2- 172.21.15.98:0/551226479 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45c80a5240 0x7f45c80a59a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f45b80098a0 tx=0x7f45b802b7a0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.190+0000 7f45cf8c0640 1 -- 172.21.15.98:0/551226479 shutdown_connections 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.190+0000 7f45cf8c0640 1 --2- 172.21.15.98:0/551226479 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c80a2530 0x7f45c80a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.190+0000 7f45cf8c0640 1 --2- 172.21.15.98:0/551226479 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45c80a5240 0x7f45c80a59a0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.190+0000 7f45cf8c0640 1 --2- 172.21.15.98:0/551226479 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45c80a4920 0x7f45c80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.190+0000 7f45cf8c0640 1 -- 172.21.15.98:0/551226479 >> 172.21.15.98:0/551226479 conn(0x7f45c80ab850 msgr2=0x7f45c80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.190+0000 7f45cf8c0640 1 -- 172.21.15.98:0/551226479 shutdown_connections 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.190+0000 7f45cf8c0640 1 -- 172.21.15.98:0/551226479 wait complete. 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient: build_initial_monmap 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient: monmap: 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:35.207 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 1 librados: starting msgr at 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 1 librados: starting objecter 2024-06-11T19:11:35.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 1 Processor -- start 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 1 -- start start 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 1 librados: setting wanted keys 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 1 librados: calling monclient init 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient: init 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient: _add_conns ranks=[8,1,2,7,5,3,0,6,4] 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c80a2530 0x7f45c81371f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient(hunting): picked mon.noname-e con 0x7f45c80a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45c80a4920 0x7f45c8137730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient(hunting): picked mon.noname-f con 0x7f45c80a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c80a5240 0x7f45c81231a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient(hunting): picked mon.noname-b con 0x7f45c80a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f45c80a6da0 con 0x7f45c80a4920 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f45c8138e50 con 0x7f45c80a5240 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f45c80a6420 con 0x7f45c80a2530 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.191+0000 7f45cf8c0640 10 monclient(hunting): authenticate will time out at 904.998981s 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c80a2530 0x7f45c81371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c80a2530 0x7f45c81371f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:45514/0 (socket says 172.21.15.98:45514) 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 1 -- 172.21.15.98:0/2100220682 learned_addr learned my addr 172.21.15.98:0/2100220682 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cde5d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c80a5240 0x7f45c81231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 10 monclient(hunting): get_auth_request con 0x7f45c80a2530 auth_method 0 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cde5d640 10 monclient(hunting): get_auth_request con 0x7f45c80a5240 auth_method 0 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cde5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cde5d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cde5d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cde5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cde5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cde5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cce5b640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45c80a4920 0x7f45c8137730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cce5b640 10 monclient(hunting): get_auth_request con 0x7f45c80a4920 auth_method 0 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cce5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cce5b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cce5b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 10 monclient(hunting): handle_auth_done global_id 22850 payload 824 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 1 -- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c80a5240 msgr2=0x7f45c81231a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c80a5240 0x7f45c81231a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 1 -- 172.21.15.98:0/2100220682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45c80a4920 msgr2=0x7f45c8137730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45c80a4920 0x7f45c8137730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 10 monclient: _finish_hunting 0 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 1 monclient: found mon.noname-e 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 1 -- 172.21.15.98:0/2100220682 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c81268c0 con 0x7f45c80a2530 2024-06-11T19:11:35.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cde5d640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f45c80a5240 0x7f45c81231a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cce5b640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45c80a4920 0x7f45c8137730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cd65c640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c80a2530 0x7f45c81371f0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f45c000e760 tx=0x7f45c000eb60 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 10 client.?.objecter ms_handle_connect 0x7f45c80a2530 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45c000f040 con 0x7f45c80a2530 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 10 monclient: dump: 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.213 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 10 monclient: _finish_auth 0 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 10 monclient: _check_auth_tickets 2024-06-11T19:11:35.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cf8c0640 5 monclient: authenticate success, global_id 22850 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c00044a0 con 0x7f45c80a2530 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cf8c0640 10 monclient: _renew_subs 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cf8c0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cf8c0640 1 -- 172.21.15.98:0/2100220682 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c8126ba0 con 0x7f45c80a2530 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45c000edb0 con 0x7f45c80a2530 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bdffb640 10 monclient: dump: 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.215 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cf8c0640 10 client.22850.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45bf7fe640 4 set_mon_vals no callback set 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cf8c0640 10 monclient: _renew_subs 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cf8c0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cf8c0640 1 -- 172.21.15.98:0/2100220682 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c8126e80 con 0x7f45c80a2530 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.192+0000 7f45cf8c0640 1 librados: init done 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45cf8c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45cf8c0640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c81236e0 0x7f45c8123ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45cf8c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45cf8c0640 1 -- 172.21.15.98:0/2100220682 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45c8069750 con 0x7f45c81236e0 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f45c0004030 con 0x7f45c80a2530 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45cce5b640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c81236e0 0x7f45c8123ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45cce5b640 10 monclient: get_auth_request con 0x7f45c81236e0 auth_method 0 2024-06-11T19:11:35.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45cce5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45cce5b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45cce5b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45cce5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45cce5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.193+0000 7f45cce5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f45ac03fdb0 0x7f45ac042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(79..79 src has 1..79) ==== 8218+0+0 (secure 0 0 0) 0x7f45c0051ef0 con 0x7f45c80a2530 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 10 client.22850.objecter ms_dispatch 0x7f45c8000b90 osd_map(79..79 src has 1..79) 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 3 client.22850.objecter handle_osd_map got epochs [79,79] > 0 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 3 client.22850.objecter handle_osd_map decoding full epoch 79 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45cce5b640 10 monclient(hunting): handle_auth_done global_id 31005 payload 824 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45cce5b640 1 -- 172.21.15.98:0/2100220682 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45c80a92f0 con 0x7f45c81236e0 2024-06-11T19:11:35.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 20 client.22850.objecter dump_active .. 0 homeless 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45cde5d640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f45ac03fdb0 0x7f45ac042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45cde5d640 10 monclient: get_auth_request con 0x7f45ac03fdb0 auth_method 0 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 10 client.22850.objecter ms_handle_connect 0x7f45c81236e0 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 10 client.22850.objecter resend_mon_ops 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45cce5b640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c81236e0 0x7f45c8123ac0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009870 tx=0x7f45b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45b8043070 con 0x7f45c81236e0 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 10 monclient: dump: 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.219 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.220 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.220 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.220 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.220 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.221 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.221 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.221 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.221 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.221 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.221 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.222 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 10 client.22850.objecter ms_handle_connect 0x7f45ac03fdb0 2024-06-11T19:11:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45cde5d640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f45ac03fdb0 0x7f45ac042270 secure :-1 s=READY pgs=3986 cs=0 l=1 rev1=1 crypto rx=0x7f45c81cea20 tx=0x7f45c400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f45b8003b50 con 0x7f45c81236e0 2024-06-11T19:11:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c81236e0 msgr2=0x7f45c8123ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c81236e0 0x7f45c8123ac0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009870 tx=0x7f45b80046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c81236e0 msgr2=0x7f45c8123ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:35.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.194+0000 7f45bdffb640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c81236e0 0x7f45c8123ac0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cf8c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:35.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cf8c0640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8124000 0x7f45c809e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cf8c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cf8c0640 1 -- 172.21.15.98:0/2100220682 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45c8050830 con 0x7f45c8124000 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cce5b640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8124000 0x7f45c809e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cce5b640 10 monclient: get_auth_request con 0x7f45c8124000 auth_method 0 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cce5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cce5b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cce5b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cce5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cce5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cce5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cce5b640 10 monclient(hunting): handle_auth_done global_id 31014 payload 824 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cce5b640 1 -- 172.21.15.98:0/2100220682 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f45c80a66f0 con 0x7f45c8124000 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45bdffb640 10 client.22850.objecter ms_handle_connect 0x7f45c8124000 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45bdffb640 10 client.22850.objecter resend_mon_ops 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45cce5b640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8124000 0x7f45c809e9f0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f45b80098a0 tx=0x7f45b8031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45b8038040 con 0x7f45c8124000 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45bdffb640 10 monclient: dump: 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.226 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7f45b803e650 con 0x7f45c8124000 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45bdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45bdffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8124000 msgr2=0x7f45c809e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.201+0000 7f45bdffb640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8124000 0x7f45c809e9f0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f45b80098a0 tx=0x7f45b8031000 comp rx=0 tx=0).stop 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45bdffb640 1 -- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8124000 msgr2=0x7f45c809e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:35.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45bdffb640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45c8124000 0x7f45c809e9f0 secure :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f45b80098a0 tx=0x7f45b8031000 comp rx=0 tx=0).stop 2024-06-11T19:11:35.228 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:11:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 145438, 2024-06-11T19:11:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:35.228 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.229 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.230 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:35.231 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:35.232 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.233 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.238 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.240 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.242 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:35.245 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:35.246 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.249 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:11:35.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:35.251 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:35.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 10 librados: watch_flush enter 2024-06-11T19:11:35.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 10 librados: watch_flush exit 2024-06-11T19:11:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 20 client.22850.objecter shutdown clearing up homeless session... 2024-06-11T19:11:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 10 client.22850.objecter successfully canceled tick 2024-06-11T19:11:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 1 -- 172.21.15.98:0/2100220682 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f45ac03fdb0 msgr2=0x7f45ac042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f45ac03fdb0 0x7f45ac042270 secure :-1 s=READY pgs=3986 cs=0 l=1 rev1=1 crypto rx=0x7f45c81cea20 tx=0x7f45c400b040 comp rx=0 tx=0).stop 2024-06-11T19:11:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 10 monclient: shutdown 2024-06-11T19:11:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 1 -- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c80a2530 msgr2=0x7f45c81371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 1 --2- 172.21.15.98:0/2100220682 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f45c80a2530 0x7f45c81371f0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f45c000e760 tx=0x7f45c000eb60 comp rx=0 tx=0).stop 2024-06-11T19:11:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cde5d640 1 -- 172.21.15.98:0/2100220682 reap_dead start 2024-06-11T19:11:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cde5d640 1 -- 172.21.15.98:0/2100220682 reap_dead start 2024-06-11T19:11:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 1 -- 172.21.15.98:0/2100220682 shutdown_connections 2024-06-11T19:11:35.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 1 -- 172.21.15.98:0/2100220682 >> 172.21.15.98:0/2100220682 conn(0x7f45c80ab850 msgr2=0x7f45c81bffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:35.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 1 -- 172.21.15.98:0/2100220682 shutdown_connections 2024-06-11T19:11:35.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 1 -- 172.21.15.98:0/2100220682 wait complete. 2024-06-11T19:11:35.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.202+0000 7f45cf8c0640 1 librados: shutdown 2024-06-11T19:11:35.255 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:35.255 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:11:35.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:11:35.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.355+0000 7fde63ac8640 10 monclient: get_monmap_and_config 2024-06-11T19:11:35.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.355+0000 7fde63ac8640 10 monclient: build_initial_monmap 2024-06-11T19:11:35.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.355+0000 7fde63ac8640 10 monclient: monmap: 2024-06-11T19:11:35.355 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:35.356759+0000 2024-06-11T19:11:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:35.356759+0000 2024-06-11T19:11:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:35.357 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.356+0000 7fde63ac8640 1 Processor -- start 2024-06-11T19:11:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.356+0000 7fde63ac8640 1 -- start start 2024-06-11T19:11:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.356+0000 7fde63ac8640 10 monclient: init 2024-06-11T19:11:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.356+0000 7fde63ac8640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.356+0000 7fde63ac8640 10 monclient: _add_conns ranks=[4,8,3,10,6,9,1,7,5,2,0] 2024-06-11T19:11:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.356+0000 7fde63ac8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fde5c0a4920 0x7fde5c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.356+0000 7fde63ac8640 10 monclient(hunting): picked mon.noname-c con 0x7fde5c0a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.356+0000 7fde63ac8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a5240 0x7fde5c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde63ac8640 10 monclient(hunting): picked mon.noname-e con 0x7fde5c0a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde63ac8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde5c0a2530 0x7fde5c0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde63ac8640 10 monclient(hunting): picked mon.noname-g con 0x7fde5c0a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde63ac8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde63ac8640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fde5c0a7140 con 0x7fde5c0a2530 2024-06-11T19:11:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde63ac8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde63ac8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fde5c0a2b50 con 0x7fde5c0a4920 2024-06-11T19:11:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde63ac8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde63ac8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fde5c0a2e30 con 0x7fde5c0a5240 2024-06-11T19:11:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde63ac8640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde63ac8640 10 monclient(hunting): authenticate will time out at 905.164038s 2024-06-11T19:11:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61864640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fde5c0a4920 0x7fde5c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a5240 0x7fde5c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61864640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fde5c0a4920 0x7fde5c0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:49952/0 (socket says 172.21.15.98:49952) 2024-06-11T19:11:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61864640 1 -- 172.21.15.98:0/1744459445 learned_addr learned my addr 172.21.15.98:0/1744459445 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61864640 10 monclient(hunting): get_auth_request con 0x7fde5c0a4920 auth_method 0 2024-06-11T19:11:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61864640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61864640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 10 monclient(hunting): get_auth_request con 0x7fde5c0a5240 auth_method 0 2024-06-11T19:11:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde62065640 1 --2- 172.21.15.98:0/1744459445 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde5c0a2530 0x7fde5c0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde62065640 10 monclient(hunting): get_auth_request con 0x7fde5c0a2530 auth_method 0 2024-06-11T19:11:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde62065640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde62065640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde62065640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 10 monclient(hunting): handle_auth_done global_id 22859 payload 274 2024-06-11T19:11:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 1 -- 172.21.15.98:0/1744459445 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fde5c0a4920 msgr2=0x7fde5c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 1 --2- 172.21.15.98:0/1744459445 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fde5c0a4920 0x7fde5c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 1 -- 172.21.15.98:0/1744459445 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde5c0a2530 msgr2=0x7fde5c0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 1 --2- 172.21.15.98:0/1744459445 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde5c0a2530 0x7fde5c0a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 10 monclient: _finish_hunting 0 2024-06-11T19:11:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 1 monclient: found mon.noname-e 2024-06-11T19:11:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61063640 1 -- 172.21.15.98:0/1744459445 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde5c0a3110 con 0x7fde5c0a5240 2024-06-11T19:11:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde61864640 1 --2- 172.21.15.98:0/1744459445 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fde5c0a4920 0x7fde5c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.357+0000 7fde62065640 1 --2- 172.21.15.98:0/1744459445 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde5c0a2530 0x7fde5c0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:11:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde61063640 1 --2- 172.21.15.98:0/1744459445 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a5240 0x7fde5c0a59a0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fde4c0097d0 tx=0x7fde4c02b6d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=699993dc9d1047cd server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 1 -- 172.21.15.98:0/1744459445 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde4c037070 con 0x7fde5c0a5240 2024-06-11T19:11:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 10 monclient: dump: 2024-06-11T19:11:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.374 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 10 monclient: _finish_auth 0 2024-06-11T19:11:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 10 monclient: _check_auth_tickets 2024-06-11T19:11:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 1 -- 172.21.15.98:0/1744459445 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde4c030980 con 0x7fde5c0a5240 2024-06-11T19:11:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 5 monclient: authenticate success, global_id 22859 2024-06-11T19:11:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 1 -- 172.21.15.98:0/1744459445 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde4c030bb0 con 0x7fde5c0a5240 2024-06-11T19:11:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 4 set_mon_vals no callback set 2024-06-11T19:11:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde60862640 10 monclient: dump: 2024-06-11T19:11:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.377 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 10 monclient: shutdown 2024-06-11T19:11:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 1 -- 172.21.15.98:0/1744459445 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a5240 msgr2=0x7fde5c0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 1 --2- 172.21.15.98:0/1744459445 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a5240 0x7fde5c0a59a0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fde4c0097d0 tx=0x7fde4c02b6d0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 1 -- 172.21.15.98:0/1744459445 shutdown_connections 2024-06-11T19:11:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 1 --2- 172.21.15.98:0/1744459445 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde5c0a2530 0x7fde5c0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 1 --2- 172.21.15.98:0/1744459445 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a5240 0x7fde5c0a59a0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 1 --2- 172.21.15.98:0/1744459445 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fde5c0a4920 0x7fde5c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.358+0000 7fde63ac8640 1 -- 172.21.15.98:0/1744459445 >> 172.21.15.98:0/1744459445 conn(0x7fde5c0ab850 msgr2=0x7fde5c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.359+0000 7fde63ac8640 1 -- 172.21.15.98:0/1744459445 shutdown_connections 2024-06-11T19:11:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.359+0000 7fde63ac8640 1 -- 172.21.15.98:0/1744459445 wait complete. 2024-06-11T19:11:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.359+0000 7fde63ac8640 10 monclient: build_initial_monmap 2024-06-11T19:11:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.359+0000 7fde63ac8640 10 monclient: monmap: 2024-06-11T19:11:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:35.380 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.359+0000 7fde63ac8640 1 librados: starting msgr at 2024-06-11T19:11:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.359+0000 7fde63ac8640 1 librados: starting objecter 2024-06-11T19:11:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.359+0000 7fde63ac8640 1 Processor -- start 2024-06-11T19:11:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 1 -- start start 2024-06-11T19:11:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 1 librados: setting wanted keys 2024-06-11T19:11:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 1 librados: calling monclient init 2024-06-11T19:11:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 10 monclient: init 2024-06-11T19:11:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 10 monclient: _add_conns ranks=[3,8,0,1,4,7,6,5,2] 2024-06-11T19:11:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde5c0a2530 0x7fde5c1371f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 10 monclient(hunting): picked mon.noname-g con 0x7fde5c0a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a4920 0x7fde5c137730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 10 monclient(hunting): picked mon.noname-e con 0x7fde5c0a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde5c0a5240 0x7fde5c1231a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 10 monclient(hunting): picked mon.noname-a con 0x7fde5c0a5240 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde5c0a6da0 con 0x7fde5c0a5240 2024-06-11T19:11:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fde5c138e50 con 0x7fde5c0a2530 2024-06-11T19:11:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fde5c0a6420 con 0x7fde5c0a4920 2024-06-11T19:11:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde63ac8640 10 monclient(hunting): authenticate will time out at 905.167446s 2024-06-11T19:11:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a4920 0x7fde5c137730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde62065640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde5c0a5240 0x7fde5c1231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a4920 0x7fde5c137730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:45540/0 (socket says 172.21.15.98:45540) 2024-06-11T19:11:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde62065640 10 monclient(hunting): get_auth_request con 0x7fde5c0a5240 auth_method 0 2024-06-11T19:11:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 1 -- 172.21.15.98:0/943476806 learned_addr learned my addr 172.21.15.98:0/943476806 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde62065640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde62065640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde62065640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 10 monclient(hunting): get_auth_request con 0x7fde5c0a4920 auth_method 0 2024-06-11T19:11:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde62065640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde62065640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61864640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde5c0a2530 0x7fde5c1371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde62065640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61864640 10 monclient(hunting): get_auth_request con 0x7fde5c0a2530 auth_method 0 2024-06-11T19:11:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61864640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61864640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 10 monclient(hunting): handle_auth_done global_id 22868 payload 824 2024-06-11T19:11:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 1 -- 172.21.15.98:0/943476806 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde5c0a2530 msgr2=0x7fde5c1371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde5c0a2530 0x7fde5c1371f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 1 -- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde5c0a5240 msgr2=0x7fde5c1231a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde5c0a5240 0x7fde5c1231a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 10 monclient: _finish_hunting 0 2024-06-11T19:11:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 1 monclient: found mon.noname-e 2024-06-11T19:11:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde61063640 1 -- 172.21.15.98:0/943476806 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde5c1268c0 con 0x7fde5c0a4920 2024-06-11T19:11:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.360+0000 7fde62065640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde5c0a5240 0x7fde5c1231a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde61063640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a4920 0x7fde5c137730 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fde4c000c00 tx=0x7fde4c0046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 10 client.?.objecter ms_handle_connect 0x7fde5c0a4920 2024-06-11T19:11:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde4c041020 con 0x7fde5c0a4920 2024-06-11T19:11:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 10 monclient: dump: 2024-06-11T19:11:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.387 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 10 monclient: _finish_auth 0 2024-06-11T19:11:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 10 monclient: _check_auth_tickets 2024-06-11T19:11:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde63ac8640 5 monclient: authenticate success, global_id 22868 2024-06-11T19:11:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde4c003610 con 0x7fde5c0a4920 2024-06-11T19:11:35.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde4c037040 con 0x7fde5c0a4920 2024-06-11T19:11:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde53fff640 4 set_mon_vals no callback set 2024-06-11T19:11:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde527fc640 10 monclient: dump: 2024-06-11T19:11:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.388 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.389 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde63ac8640 10 monclient: _renew_subs 2024-06-11T19:11:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde63ac8640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:11:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde63ac8640 1 -- 172.21.15.98:0/943476806 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde5c126ba0 con 0x7fde5c0a4920 2024-06-11T19:11:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde63ac8640 10 client.22868.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde63ac8640 10 monclient: _renew_subs 2024-06-11T19:11:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde63ac8640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:11:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde63ac8640 1 -- 172.21.15.98:0/943476806 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fde5c126e80 con 0x7fde5c0a4920 2024-06-11T19:11:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.361+0000 7fde63ac8640 1 librados: init done 2024-06-11T19:11:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde63ac8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde63ac8640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c1236e0 0x7fde5c123ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde63ac8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde63ac8640 1 -- 172.21.15.98:0/943476806 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fde5c069750 con 0x7fde5c1236e0 2024-06-11T19:11:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fde4c044020 con 0x7fde5c0a4920 2024-06-11T19:11:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde61864640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c1236e0 0x7fde5c123ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde61864640 10 monclient: get_auth_request con 0x7fde5c1236e0 auth_method 0 2024-06-11T19:11:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde61864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde61864640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde61864640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde61864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde61864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde61864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde61864640 10 monclient(hunting): handle_auth_done global_id 21548 payload 824 2024-06-11T19:11:35.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde61864640 1 -- 172.21.15.98:0/943476806 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde5c0a61d0 con 0x7fde5c1236e0 2024-06-11T19:11:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fde4003fdb0 0x7fde40042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(79..79 src has 1..79) ==== 8218+0+0 (secure 0 0 0) 0x7fde4c073ac0 con 0x7fde5c0a4920 2024-06-11T19:11:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 10 client.22868.objecter ms_dispatch 0x7fde5c000b90 osd_map(79..79 src has 1..79) 2024-06-11T19:11:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 3 client.22868.objecter handle_osd_map got epochs [79,79] > 0 2024-06-11T19:11:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 3 client.22868.objecter handle_osd_map decoding full epoch 79 2024-06-11T19:11:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde61864640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c1236e0 0x7fde5c123ac0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fde580119a0 tx=0x7fde58011e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde62065640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fde4003fdb0 0x7fde40042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 20 client.22868.objecter dump_active .. 0 homeless 2024-06-11T19:11:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 10 client.22868.objecter ms_handle_connect 0x7fde5c1236e0 2024-06-11T19:11:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 10 client.22868.objecter resend_mon_ops 2024-06-11T19:11:35.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde62065640 10 monclient: get_auth_request con 0x7fde4003fdb0 auth_method 0 2024-06-11T19:11:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde58014440 con 0x7fde5c1236e0 2024-06-11T19:11:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:11:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.362+0000 7fde527fc640 10 monclient: dump: 2024-06-11T19:11:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.394 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.363+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fde5800fb60 con 0x7fde5c1236e0 2024-06-11T19:11:35.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.363+0000 7fde527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.363+0000 7fde527fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.363+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c1236e0 msgr2=0x7fde5c123ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.363+0000 7fde527fc640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c1236e0 0x7fde5c123ac0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fde580119a0 tx=0x7fde58011e60 comp rx=0 tx=0).stop 2024-06-11T19:11:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.363+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c1236e0 msgr2=0x7fde5c123ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.363+0000 7fde527fc640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c1236e0 0x7fde5c123ac0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.363+0000 7fde527fc640 10 client.22868.objecter ms_handle_connect 0x7fde4003fdb0 2024-06-11T19:11:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.363+0000 7fde62065640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fde4003fdb0 0x7fde40042270 secure :-1 s=READY pgs=3987 cs=0 l=1 rev1=1 crypto rx=0x7fde5c1cea20 tx=0x7fde54009850 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde63ac8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde63ac8640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c124000 0x7fde5c09e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde63ac8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde63ac8640 1 -- 172.21.15.98:0/943476806 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fde5c050830 con 0x7fde5c124000 2024-06-11T19:11:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde61864640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c124000 0x7fde5c09e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde61864640 10 monclient: get_auth_request con 0x7fde5c124000 auth_method 0 2024-06-11T19:11:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde61864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde61864640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde61864640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde61864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde61864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde61864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde61864640 10 monclient(hunting): handle_auth_done global_id 21557 payload 824 2024-06-11T19:11:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.369+0000 7fde61864640 1 -- 172.21.15.98:0/943476806 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fde5c0a92f0 con 0x7fde5c124000 2024-06-11T19:11:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde61864640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c124000 0x7fde5c09e9f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fde580119a0 tx=0x7fde580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 10 client.22868.objecter ms_handle_connect 0x7fde5c124000 2024-06-11T19:11:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 10 client.22868.objecter resend_mon_ops 2024-06-11T19:11:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde58014ac0 con 0x7fde5c124000 2024-06-11T19:11:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:11:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 10 monclient: dump: 2024-06-11T19:11:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.399 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.400 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.401 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fde5802c420 con 0x7fde5c124000 2024-06-11T19:11:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c124000 msgr2=0x7fde5c09e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c124000 0x7fde5c09e9f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fde580119a0 tx=0x7fde580046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 1 -- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c124000 msgr2=0x7fde5c09e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:35.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde527fc640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde5c124000 0x7fde5c09e9f0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.402 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 145603, 2024-06-11T19:11:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:35.402 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-06-11T19:11:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.405 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:35.406 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.407 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.408 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.410 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.412 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:35.414 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:35.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.416 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.418 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.419 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:35.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:35.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:35.425 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.426 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:35.427 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.429 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:11:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.430 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.431 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:35.431 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 10 librados: watch_flush enter 2024-06-11T19:11:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 10 librados: watch_flush exit 2024-06-11T19:11:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 20 client.22868.objecter shutdown clearing up homeless session... 2024-06-11T19:11:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 10 client.22868.objecter successfully canceled tick 2024-06-11T19:11:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 1 -- 172.21.15.98:0/943476806 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fde4003fdb0 msgr2=0x7fde40042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fde4003fdb0 0x7fde40042270 secure :-1 s=READY pgs=3987 cs=0 l=1 rev1=1 crypto rx=0x7fde5c1cea20 tx=0x7fde54009850 comp rx=0 tx=0).stop 2024-06-11T19:11:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 10 monclient: shutdown 2024-06-11T19:11:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 1 -- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a4920 msgr2=0x7fde5c137730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a4920 0x7fde5c137730 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fde4c000c00 tx=0x7fde4c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde62065640 1 -- 172.21.15.98:0/943476806 reap_dead start 2024-06-11T19:11:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 1 -- 172.21.15.98:0/943476806 shutdown_connections 2024-06-11T19:11:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 1 --2- 172.21.15.98:0/943476806 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde5c0a4920 0x7fde5c137730 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.370+0000 7fde63ac8640 1 -- 172.21.15.98:0/943476806 >> 172.21.15.98:0/943476806 conn(0x7fde5c0ab850 msgr2=0x7fde5c1bffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.371+0000 7fde63ac8640 1 -- 172.21.15.98:0/943476806 shutdown_connections 2024-06-11T19:11:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.371+0000 7fde63ac8640 1 -- 172.21.15.98:0/943476806 wait complete. 2024-06-11T19:11:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.371+0000 7fde63ac8640 1 librados: shutdown 2024-06-11T19:11:35.435 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:35.435 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:11:35.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:11:35.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.536+0000 7f43f8703640 10 monclient: get_monmap_and_config 2024-06-11T19:11:35.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.536+0000 7f43f8703640 10 monclient: build_initial_monmap 2024-06-11T19:11:35.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.536+0000 7f43f8703640 10 monclient: monmap: 2024-06-11T19:11:35.536 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:35.536 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.536 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:35.537267+0000 2024-06-11T19:11:35.536 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:35.537267+0000 2024-06-11T19:11:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:35.537 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:35.538 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:35.538 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.536+0000 7f43f8703640 1 Processor -- start 2024-06-11T19:11:35.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.536+0000 7f43f8703640 1 -- start start 2024-06-11T19:11:35.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.536+0000 7f43f8703640 10 monclient: init 2024-06-11T19:11:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 10 monclient: _add_conns ranks=[9,6,10,1,4,5,0,7,8,2,3] 2024-06-11T19:11:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43f01347f0 0x7f43f009e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 10 monclient(hunting): picked mon.noname-a con 0x7f43f01347f0 addr v2:172.21.15.98:3300/0 2024-06-11T19:11:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f43f0134bd0 0x7f43f00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 10 monclient(hunting): picked mon.noname-d con 0x7f43f0134bd0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f43f0135010 0x7f43f00a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 10 monclient(hunting): picked mon.noname-f con 0x7f43f0135010 addr v2:172.21.15.117:3300/0 2024-06-11T19:11:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f43f00687f0 con 0x7f43f01347f0 2024-06-11T19:11:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f43f009d410 con 0x7f43f0135010 2024-06-11T19:11:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f43f009d6f0 con 0x7f43f0134bd0 2024-06-11T19:11:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f8703640 10 monclient(hunting): authenticate will time out at 905.344515s 2024-06-11T19:11:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43f01347f0 0x7f43f009e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f5c9e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f43f0134bd0 0x7f43f00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43f01347f0 0x7f43f009e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43510/0 (socket says 172.21.15.98:43510) 2024-06-11T19:11:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 1 -- 172.21.15.98:0/3506820476 learned_addr learned my addr 172.21.15.98:0/3506820476 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 10 monclient(hunting): get_auth_request con 0x7f43f01347f0 auth_method 0 2024-06-11T19:11:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f5c9e640 10 monclient(hunting): get_auth_request con 0x7f43f0134bd0 auth_method 0 2024-06-11T19:11:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f5c9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f5c9e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f5c9e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f5c9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f5c9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f5c9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 10 monclient(hunting): handle_auth_done global_id 31032 payload 274 2024-06-11T19:11:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 1 -- 172.21.15.98:0/3506820476 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f43f0134bd0 msgr2=0x7f43f00a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 1 --2- 172.21.15.98:0/3506820476 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f43f0134bd0 0x7f43f00a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 1 -- 172.21.15.98:0/3506820476 >> v2:172.21.15.117:3300/0 conn(0x7f43f0135010 msgr2=0x7f43f00a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f6ca0640 1 --2- 172.21.15.98:0/3506820476 >> v2:172.21.15.117:3300/0 conn(0x7f43f0135010 0x7f43f00a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 1 --2- 172.21.15.98:0/3506820476 >> v2:172.21.15.117:3300/0 conn(0x7f43f0135010 0x7f43f00a3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 10 monclient: _finish_hunting 0 2024-06-11T19:11:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 1 monclient: found mon.noname-a 2024-06-11T19:11:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:11:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.537+0000 7f43f649f640 1 -- 172.21.15.98:0/3506820476 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f009d9d0 con 0x7f43f01347f0 2024-06-11T19:11:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f649f640 1 --2- 172.21.15.98:0/3506820476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f01347f0 0x7f43f009e6b0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f43f006cea0 tx=0x7f43e802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a87b3b67c62629 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 1 -- 172.21.15.98:0/3506820476 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43e802e070 con 0x7f43f01347f0 2024-06-11T19:11:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:11:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 10 monclient: dump: 2024-06-11T19:11:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.554 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 10 monclient: _finish_auth 0 2024-06-11T19:11:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 10 monclient: _check_auth_tickets 2024-06-11T19:11:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 1 -- 172.21.15.98:0/3506820476 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e8037070 con 0x7f43f01347f0 2024-06-11T19:11:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f8703640 5 monclient: authenticate success, global_id 31032 2024-06-11T19:11:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f8703640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f8703640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 1 -- 172.21.15.98:0/3506820476 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43e8032470 con 0x7f43f01347f0 2024-06-11T19:11:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f8703640 4 set_mon_vals no callback set 2024-06-11T19:11:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.538+0000 7f43f549d640 10 monclient: dump: 2024-06-11T19:11:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.557 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.539+0000 7f43f8703640 10 monclient: shutdown 2024-06-11T19:11:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.539+0000 7f43f8703640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.539+0000 7f43f8703640 1 -- 172.21.15.98:0/3506820476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f01347f0 msgr2=0x7f43f009e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.539+0000 7f43f8703640 1 --2- 172.21.15.98:0/3506820476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f01347f0 0x7f43f009e6b0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f43f006cea0 tx=0x7f43e802b880 comp rx=0 tx=0).stop 2024-06-11T19:11:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.539+0000 7f43f8703640 1 -- 172.21.15.98:0/3506820476 shutdown_connections 2024-06-11T19:11:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.539+0000 7f43f8703640 1 --2- 172.21.15.98:0/3506820476 >> v2:172.21.15.117:3300/0 conn(0x7f43f0135010 0x7f43f00a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.539+0000 7f43f8703640 1 --2- 172.21.15.98:0/3506820476 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f43f0134bd0 0x7f43f00a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.539+0000 7f43f8703640 1 --2- 172.21.15.98:0/3506820476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f01347f0 0x7f43f009e6b0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.539+0000 7f43f8703640 1 -- 172.21.15.98:0/3506820476 >> 172.21.15.98:0/3506820476 conn(0x7f43f00a4780 msgr2=0x7f43f00a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.539+0000 7f43f8703640 1 -- 172.21.15.98:0/3506820476 shutdown_connections 2024-06-11T19:11:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.539+0000 7f43f8703640 1 -- 172.21.15.98:0/3506820476 wait complete. 2024-06-11T19:11:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.539+0000 7f43f8703640 10 monclient: build_initial_monmap 2024-06-11T19:11:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 10 monclient: monmap: 2024-06-11T19:11:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:35.560 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 1 librados: starting msgr at 2024-06-11T19:11:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 1 librados: starting objecter 2024-06-11T19:11:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 1 Processor -- start 2024-06-11T19:11:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 1 -- start start 2024-06-11T19:11:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 1 librados: setting wanted keys 2024-06-11T19:11:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 1 librados: calling monclient init 2024-06-11T19:11:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 10 monclient: init 2024-06-11T19:11:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 10 monclient: _add_conns ranks=[4,2,0,1,3,8,5,6,7] 2024-06-11T19:11:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f01347f0 0x7f43f01cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 10 monclient(hunting): picked mon.noname-c con 0x7f43f01347f0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f0134bd0 0x7f43f01ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 10 monclient(hunting): picked mon.noname-b con 0x7f43f0134bd0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f0135010 0x7f43f011ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 10 monclient(hunting): picked mon.noname-a con 0x7f43f0135010 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f43f0068c40 con 0x7f43f0135010 2024-06-11T19:11:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f43f0068ac0 con 0x7f43f0134bd0 2024-06-11T19:11:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f43f00a4110 con 0x7f43f01347f0 2024-06-11T19:11:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f8703640 10 monclient(hunting): authenticate will time out at 905.347810s 2024-06-11T19:11:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f649f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f01347f0 0x7f43f01cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f5c9e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f0134bd0 0x7f43f01ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f6ca0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f0135010 0x7f43f011ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f649f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f01347f0 0x7f43f01cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:49960/0 (socket says 172.21.15.98:49960) 2024-06-11T19:11:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f5c9e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f0134bd0 0x7f43f01ce080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45710/0 (socket says 172.21.15.98:45710) 2024-06-11T19:11:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f649f640 1 -- 172.21.15.98:0/818843728 learned_addr learned my addr 172.21.15.98:0/818843728 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f6ca0640 10 monclient(hunting): get_auth_request con 0x7f43f0135010 auth_method 0 2024-06-11T19:11:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f6ca0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f6ca0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f6ca0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f649f640 10 monclient(hunting): get_auth_request con 0x7f43f01347f0 auth_method 0 2024-06-11T19:11:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f649f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f649f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f649f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f5c9e640 10 monclient(hunting): get_auth_request con 0x7f43f0134bd0 auth_method 0 2024-06-11T19:11:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f5c9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f5c9e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.540+0000 7f43f5c9e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f5c9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f5c9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f5c9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f649f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f649f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f649f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 10 monclient(hunting): handle_auth_done global_id 31041 payload 824 2024-06-11T19:11:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 1 -- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f01347f0 msgr2=0x7f43f01cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f01347f0 0x7f43f01cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 1 -- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f0134bd0 msgr2=0x7f43f01ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f0134bd0 0x7f43f01ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 10 monclient: _finish_hunting 0 2024-06-11T19:11:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 1 monclient: found mon.noname-a 2024-06-11T19:11:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:11:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 1 -- 172.21.15.98:0/818843728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f0121260 con 0x7f43f0135010 2024-06-11T19:11:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f5c9e640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43f0134bd0 0x7f43f01ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f6ca0640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f0135010 0x7f43f011ea30 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f43f00a4750 tx=0x7f43e000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 10 client.?.objecter ms_handle_connect 0x7f43f0135010 2024-06-11T19:11:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43e000ac70 con 0x7f43f0135010 2024-06-11T19:11:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:11:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 10 monclient: dump: 2024-06-11T19:11:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.565 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.567 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 10 monclient: _finish_auth 0 2024-06-11T19:11:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f8703640 5 monclient: authenticate success, global_id 31041 2024-06-11T19:11:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e000ae30 con 0x7f43f0135010 2024-06-11T19:11:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f8703640 10 monclient: _renew_subs 2024-06-11T19:11:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f8703640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f8703640 1 -- 172.21.15.98:0/818843728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f01229f0 con 0x7f43f0135010 2024-06-11T19:11:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43e000fcd0 con 0x7f43f0135010 2024-06-11T19:11:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f8703640 10 client.31041.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f4c9c640 4 set_mon_vals no callback set 2024-06-11T19:11:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43eeffd640 10 monclient: dump: 2024-06-11T19:11:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.569 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f8703640 10 monclient: _renew_subs 2024-06-11T19:11:35.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f8703640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:35.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f8703640 1 -- 172.21.15.98:0/818843728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f0135be0 con 0x7f43f0135010 2024-06-11T19:11:35.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.541+0000 7f43f8703640 1 librados: init done 2024-06-11T19:11:35.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f8703640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:35.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f8703640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f0122390 0x7f43f011ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f8703640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f8703640 1 -- 172.21.15.98:0/818843728 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f43f0050830 con 0x7f43f0122390 2024-06-11T19:11:35.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f43e000f3f0 con 0x7f43f0135010 2024-06-11T19:11:35.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f649f640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f0122390 0x7f43f011ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f649f640 10 monclient: get_auth_request con 0x7f43f0122390 auth_method 0 2024-06-11T19:11:35.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f649f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f649f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f649f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f649f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f649f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f649f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43eeffd640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f43dc03fdb0 0x7f43dc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f649f640 10 monclient(hunting): handle_auth_done global_id 22864 payload 824 2024-06-11T19:11:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f649f640 1 -- 172.21.15.98:0/818843728 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43f009e6b0 con 0x7f43f0122390 2024-06-11T19:11:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8218+0+0 (secure 0 0 0) 0x7f43e0009030 con 0x7f43f0135010 2024-06-11T19:11:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43eeffd640 10 client.31041.objecter ms_dispatch 0x7f43f0000b90 osd_map(79..79 src has 1..79) 2024-06-11T19:11:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43eeffd640 3 client.31041.objecter handle_osd_map got epochs [79,79] > 0 2024-06-11T19:11:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43eeffd640 3 client.31041.objecter handle_osd_map decoding full epoch 79 2024-06-11T19:11:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43eeffd640 20 client.31041.objecter dump_active .. 0 homeless 2024-06-11T19:11:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43eeffd640 10 client.31041.objecter ms_handle_connect 0x7f43f0122390 2024-06-11T19:11:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43eeffd640 10 client.31041.objecter resend_mon_ops 2024-06-11T19:11:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.542+0000 7f43f649f640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f0122390 0x7f43f011ef70 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f43e80097c0 tx=0x7f43e8004730 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43f5c9e640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f43dc03fdb0 0x7f43dc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43f5c9e640 10 monclient: get_auth_request con 0x7f43dc03fdb0 auth_method 0 2024-06-11T19:11:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43e804a070 con 0x7f43f0122390 2024-06-11T19:11:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43eeffd640 10 monclient: dump: 2024-06-11T19:11:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.572 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.573 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.573 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.573 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.573 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.573 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.573 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.573 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.573 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.573 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.573 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.573 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.574 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.574 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.574 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f43e8037040 con 0x7f43f0122390 2024-06-11T19:11:35.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:35.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43eeffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:35.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f0122390 msgr2=0x7f43f011ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43eeffd640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f0122390 0x7f43f011ef70 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f43e80097c0 tx=0x7f43e8004730 comp rx=0 tx=0).stop 2024-06-11T19:11:35.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f0122390 msgr2=0x7f43f011ef70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:35.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43eeffd640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f0122390 0x7f43f011ef70 secure :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f43e80097c0 tx=0x7f43e8004730 comp rx=0 tx=0).stop 2024-06-11T19:11:35.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43eeffd640 10 client.31041.objecter ms_handle_connect 0x7f43dc03fdb0 2024-06-11T19:11:35.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.543+0000 7f43f5c9e640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f43dc03fdb0 0x7f43dc042270 secure :-1 s=READY pgs=3988 cs=0 l=1 rev1=1 crypto rx=0x7f43f009d0a0 tx=0x7f43e4009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.549+0000 7f43f8703640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:35.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.549+0000 7f43f8703640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f011f4b0 0x7f43f011f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.549+0000 7f43f8703640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.549+0000 7f43f8703640 1 -- 172.21.15.98:0/818843728 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f43f011fe00 con 0x7f43f011f4b0 2024-06-11T19:11:35.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.549+0000 7f43f649f640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f011f4b0 0x7f43f011f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.549+0000 7f43f649f640 10 monclient: get_auth_request con 0x7f43f011f4b0 auth_method 0 2024-06-11T19:11:35.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.549+0000 7f43f649f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.549+0000 7f43f649f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.549+0000 7f43f649f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.549+0000 7f43f649f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.549+0000 7f43f649f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43f649f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43f649f640 10 monclient(hunting): handle_auth_done global_id 22873 payload 824 2024-06-11T19:11:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43f649f640 1 -- 172.21.15.98:0/818843728 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f43f0120fb0 con 0x7f43f011f4b0 2024-06-11T19:11:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 10 client.31041.objecter ms_handle_connect 0x7f43f011f4b0 2024-06-11T19:11:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43f649f640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f011f4b0 0x7f43f011f8c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f43e802b8c0 tx=0x7f43e8031000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 10 client.31041.objecter resend_mon_ops 2024-06-11T19:11:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43e8053070 con 0x7f43f011f4b0 2024-06-11T19:11:35.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 10 monclient: dump: 2024-06-11T19:11:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.580 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f43e8031c60 con 0x7f43f011f4b0 2024-06-11T19:11:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f011f4b0 msgr2=0x7f43f011f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f011f4b0 0x7f43f011f8c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f43e802b8c0 tx=0x7f43e8031000 comp rx=0 tx=0).stop 2024-06-11T19:11:35.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 1 -- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f011f4b0 msgr2=0x7f43f011f8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43eeffd640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f43f011f4b0 0x7f43f011f8c0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.581 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 145768, 2024-06-11T19:11:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:11:35.582 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.583 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.584 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:35.585 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:35.586 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.587 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:35.588 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.589 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.591 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.593 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.595 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.599 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.605 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.606 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.607 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:35.607 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43f8703640 10 librados: watch_flush enter 2024-06-11T19:11:35.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.550+0000 7f43f8703640 10 librados: watch_flush exit 2024-06-11T19:11:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 20 client.31041.objecter shutdown clearing up homeless session... 2024-06-11T19:11:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 10 client.31041.objecter successfully canceled tick 2024-06-11T19:11:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 1 -- 172.21.15.98:0/818843728 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f43dc03fdb0 msgr2=0x7f43dc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f43dc03fdb0 0x7f43dc042270 secure :-1 s=READY pgs=3988 cs=0 l=1 rev1=1 crypto rx=0x7f43f009d0a0 tx=0x7f43e4009190 comp rx=0 tx=0).stop 2024-06-11T19:11:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 10 monclient: shutdown 2024-06-11T19:11:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 1 -- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f0135010 msgr2=0x7f43f011ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 1 --2- 172.21.15.98:0/818843728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43f0135010 0x7f43f011ea30 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f43f00a4750 tx=0x7f43e000a5c0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f6ca0640 1 -- 172.21.15.98:0/818843728 reap_dead start 2024-06-11T19:11:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f6ca0640 1 -- 172.21.15.98:0/818843728 reap_dead start 2024-06-11T19:11:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 1 -- 172.21.15.98:0/818843728 shutdown_connections 2024-06-11T19:11:35.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 1 -- 172.21.15.98:0/818843728 >> 172.21.15.98:0/818843728 conn(0x7f43f00a4780 msgr2=0x7f43f01375a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:35.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 1 -- 172.21.15.98:0/818843728 shutdown_connections 2024-06-11T19:11:35.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 1 -- 172.21.15.98:0/818843728 wait complete. 2024-06-11T19:11:35.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.551+0000 7f43f8703640 1 librados: shutdown 2024-06-11T19:11:35.611 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:35.611 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:11:35.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:11:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.711+0000 7ff3769a1640 10 monclient: get_monmap_and_config 2024-06-11T19:11:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.711+0000 7ff3769a1640 10 monclient: build_initial_monmap 2024-06-11T19:11:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.711+0000 7ff3769a1640 10 monclient: monmap: 2024-06-11T19:11:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:35.712476+0000 2024-06-11T19:11:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:35.712476+0000 2024-06-11T19:11:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:35.712 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 1 Processor -- start 2024-06-11T19:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 1 -- start start 2024-06-11T19:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 10 monclient: init 2024-06-11T19:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 10 monclient: _add_conns ranks=[1,8,10,6,9,4,0,7,3,5,2] 2024-06-11T19:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff3700a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff3700a2530 0x7ff37013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 10 monclient(hunting): picked mon.noname-e con 0x7ff3700a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff37013e1a0 0x7ff370140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 10 monclient(hunting): picked mon.noname-f con 0x7ff37013e1a0 addr v2:172.21.15.117:3300/0 2024-06-11T19:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3700a7140 con 0x7ff3700a4920 2024-06-11T19:11:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7ff3700a2b50 con 0x7ff37013e1a0 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff3700a2e30 con 0x7ff3700a2530 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff3769a1640 10 monclient(hunting): authenticate will time out at 905.519829s 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff36f7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff3700a2530 0x7ff37013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff36f7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff3700a2530 0x7ff37013dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:45550/0 (socket says 172.21.15.98:45550) 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff36f7fe640 1 -- 172.21.15.98:0/1882525806 learned_addr learned my addr 172.21.15.98:0/1882525806 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff36f7fe640 10 monclient(hunting): get_auth_request con 0x7ff3700a2530 auth_method 0 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff36f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff36f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.712+0000 7ff36f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff374f3e640 1 --2- 172.21.15.98:0/1882525806 >> v2:172.21.15.117:3300/0 conn(0x7ff37013e1a0 0x7ff370140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff374f3e640 10 monclient(hunting): get_auth_request con 0x7ff37013e1a0 auth_method 0 2024-06-11T19:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff374f3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff374f3e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff374f3e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff374f3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff374f3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff374f3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 10 monclient(hunting): handle_auth_done global_id 22877 payload 274 2024-06-11T19:11:35.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 1 -- 172.21.15.98:0/1882525806 >> v2:172.21.15.117:3300/0 conn(0x7ff37013e1a0 msgr2=0x7ff370140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 1 --2- 172.21.15.98:0/1882525806 >> v2:172.21.15.117:3300/0 conn(0x7ff37013e1a0 0x7ff370140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 1 -- 172.21.15.98:0/1882525806 >> v1:172.21.15.117:6789/0 conn(0x7ff3700a4920 legacy=0x7ff3700a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:11:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 1 monclient: found mon.noname-e 2024-06-11T19:11:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 1 -- 172.21.15.98:0/1882525806 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3700a3110 con 0x7ff3700a2530 2024-06-11T19:11:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.713+0000 7ff36f7fe640 1 --2- 172.21.15.98:0/1882525806 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff3700a2530 0x7ff37013dc60 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7ff364009870 tx=0x7ff36402b860 comp rx=0 tx=0).ready entity=mon.8 client_cookie=de84d6804c2d3762 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 1 -- 172.21.15.98:0/1882525806 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff364031a30 con 0x7ff3700a2530 2024-06-11T19:11:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:35.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 10 monclient: dump: 2024-06-11T19:11:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.728 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 10 monclient: _finish_auth 0 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 1 -- 172.21.15.98:0/1882525806 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff364037070 con 0x7ff3700a2530 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 5 monclient: authenticate success, global_id 22877 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:35.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 1 -- 172.21.15.98:0/1882525806 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff364031c30 con 0x7ff3700a2530 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 4 set_mon_vals no callback set 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff36effd640 10 monclient: dump: 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.730 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 10 monclient: shutdown 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 1 -- 172.21.15.98:0/1882525806 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff3700a2530 msgr2=0x7ff37013dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 1 --2- 172.21.15.98:0/1882525806 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff3700a2530 0x7ff37013dc60 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7ff364009870 tx=0x7ff36402b860 comp rx=0 tx=0).stop 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 1 -- 172.21.15.98:0/1882525806 shutdown_connections 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 1 --2- 172.21.15.98:0/1882525806 >> v2:172.21.15.117:3300/0 conn(0x7ff37013e1a0 0x7ff370140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 1 --2- 172.21.15.98:0/1882525806 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff3700a2530 0x7ff37013dc60 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 1 -- 172.21.15.98:0/1882525806 >> 172.21.15.98:0/1882525806 conn(0x7ff3700ab850 msgr2=0x7ff3700a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 1 -- 172.21.15.98:0/1882525806 shutdown_connections 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.714+0000 7ff3769a1640 1 -- 172.21.15.98:0/1882525806 wait complete. 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.715+0000 7ff3769a1640 10 monclient: build_initial_monmap 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.715+0000 7ff3769a1640 10 monclient: monmap: 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:35.732 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.715+0000 7ff3769a1640 1 librados: starting msgr at 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.715+0000 7ff3769a1640 1 librados: starting objecter 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.715+0000 7ff3769a1640 1 Processor -- start 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.715+0000 7ff3769a1640 1 -- start start 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.715+0000 7ff3769a1640 1 librados: setting wanted keys 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.715+0000 7ff3769a1640 1 librados: calling monclient init 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.715+0000 7ff3769a1640 10 monclient: init 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:35.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 10 monclient: _add_conns ranks=[0,3,2,8,6,7,1,4,5] 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3700a2530 0x7ff3700a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 10 monclient(hunting): picked mon.noname-a con 0x7ff3700a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3700a4920 0x7ff3701cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 10 monclient(hunting): picked mon.noname-g con 0x7ff3700a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff37013e1a0 0x7ff3701cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 10 monclient(hunting): picked mon.noname-b con 0x7ff37013e1a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff370138e50 con 0x7ff3700a2530 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff3700a84b0 con 0x7ff37013e1a0 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ff3700a6da0 con 0x7ff3700a4920 2024-06-11T19:11:35.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 10 monclient(hunting): authenticate will time out at 905.523203s 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff37013e1a0 0x7ff3701cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3700a2530 0x7ff3700a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff37013e1a0 0x7ff3701cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45726/0 (socket says 172.21.15.98:45726) 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 1 -- 172.21.15.98:0/1346561152 learned_addr learned my addr 172.21.15.98:0/1346561152 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 10 monclient(hunting): get_auth_request con 0x7ff37013e1a0 auth_method 0 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3700a2530 0x7ff3700a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43536/0 (socket says 172.21.15.98:43536) 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36ffff640 10 monclient(hunting): get_auth_request con 0x7ff3700a2530 auth_method 0 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36f7fe640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3700a4920 0x7ff3701cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36f7fe640 10 monclient(hunting): get_auth_request con 0x7ff3700a4920 auth_method 0 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 10 monclient(hunting): handle_auth_done global_id 21575 payload 824 2024-06-11T19:11:35.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 1 -- 172.21.15.98:0/1346561152 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3700a4920 msgr2=0x7ff3701cd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3700a4920 0x7ff3701cd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 1 -- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3700a2530 msgr2=0x7ff3700a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3700a2530 0x7ff3700a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 10 monclient: _finish_hunting 0 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 1 monclient: found mon.noname-b 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 1 -- 172.21.15.98:0/1346561152 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3700a08d0 con 0x7ff37013e1a0 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36ffff640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3700a2530 0x7ff3700a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff374f3e640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff37013e1a0 0x7ff3701cdcd0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7ff36800efc0 tx=0x7ff36800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 10 client.?.objecter ms_handle_connect 0x7ff37013e1a0 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff368018020 con 0x7ff37013e1a0 2024-06-11T19:11:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 10 monclient: dump: 2024-06-11T19:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 10 monclient: _finish_auth 0 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 10 monclient: _check_auth_tickets 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 5 monclient: authenticate success, global_id 21575 2024-06-11T19:11:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff36800cd30 con 0x7ff37013e1a0 2024-06-11T19:11:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 10 monclient: _renew_subs 2024-06-11T19:11:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff368014d60 con 0x7ff37013e1a0 2024-06-11T19:11:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff3769a1640 1 -- 172.21.15.98:0/1346561152 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3700a0bb0 con 0x7ff37013e1a0 2024-06-11T19:11:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff36cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:11:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.716+0000 7ff36e7fc640 4 set_mon_vals no callback set 2024-06-11T19:11:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff36cff9640 10 monclient: dump: 2024-06-11T19:11:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.740 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff3769a1640 10 client.21575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff3769a1640 10 monclient: _renew_subs 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff3769a1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff3769a1640 1 -- 172.21.15.98:0/1346561152 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff37013da40 con 0x7ff37013e1a0 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff3769a1640 1 librados: init done 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff3769a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff3769a1640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff3701ce210 0x7ff3701337a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff3769a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff3769a1640 1 -- 172.21.15.98:0/1346561152 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff370050830 con 0x7ff3701ce210 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff368005d10 con 0x7ff37013e1a0 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff36ffff640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff3701ce210 0x7ff3701337a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff36ffff640 10 monclient: get_auth_request con 0x7ff3701ce210 auth_method 0 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff36ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff36ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff36ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff36ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.717+0000 7ff36ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff35c03fde0 0x7ff35c0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36ffff640 10 monclient(hunting): handle_auth_done global_id 54213 payload 824 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36ffff640 1 -- 172.21.15.98:0/1346561152 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3700a4290 con 0x7ff3701ce210 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(79..79 src has 1..79) ==== 8218+0+0 (secure 0 0 0) 0x7ff368052c00 con 0x7ff37013e1a0 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 10 client.21575.objecter ms_dispatch 0x7ff370000b90 osd_map(79..79 src has 1..79) 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 3 client.21575.objecter handle_osd_map got epochs [79,79] > 0 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 3 client.21575.objecter handle_osd_map decoding full epoch 79 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 20 client.21575.objecter dump_active .. 0 homeless 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36ffff640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff3701ce210 0x7ff3701337a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff36000fb10 tx=0x7ff360001d30 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 10 client.21575.objecter ms_handle_connect 0x7ff3701ce210 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 10 client.21575.objecter resend_mon_ops 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36f7fe640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff35c03fde0 0x7ff35c0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36f7fe640 10 monclient: get_auth_request con 0x7ff35c03fde0 auth_method 0 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff3600052e0 con 0x7ff3701ce210 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 10 monclient: dump: 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff360018070 con 0x7ff3701ce210 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff3701ce210 msgr2=0x7ff3701337a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff3701ce210 0x7ff3701337a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff36000fb10 tx=0x7ff360001d30 comp rx=0 tx=0).stop 2024-06-11T19:11:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff3701ce210 msgr2=0x7ff3701337a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff3701ce210 0x7ff3701337a0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff36000fb10 tx=0x7ff360001d30 comp rx=0 tx=0).stop 2024-06-11T19:11:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36cff9640 10 client.21575.objecter ms_handle_connect 0x7ff35c03fde0 2024-06-11T19:11:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.718+0000 7ff36f7fe640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff35c03fde0 0x7ff35c0422a0 secure :-1 s=READY pgs=3989 cs=0 l=1 rev1=1 crypto rx=0x7ff364036000 tx=0x7ff364008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff3769a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff3769a1640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff370133ce0 0x7ff3701340f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff3769a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff3769a1640 1 -- 172.21.15.98:0/1346561152 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff370134630 con 0x7ff370133ce0 2024-06-11T19:11:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36ffff640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff370133ce0 0x7ff3701340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36ffff640 10 monclient: get_auth_request con 0x7ff370133ce0 auth_method 0 2024-06-11T19:11:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36ffff640 10 monclient(hunting): handle_auth_done global_id 54222 payload 824 2024-06-11T19:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36ffff640 1 -- 172.21.15.98:0/1346561152 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff3700a4d00 con 0x7ff370133ce0 2024-06-11T19:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 10 client.21575.objecter ms_handle_connect 0x7ff370133ce0 2024-06-11T19:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36ffff640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff370133ce0 0x7ff3701340f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff36001d450 tx=0x7ff360000ed0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 10 client.21575.objecter resend_mon_ops 2024-06-11T19:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff3600035f0 con 0x7ff370133ce0 2024-06-11T19:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 10 monclient: dump: 2024-06-11T19:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9271 (secure 0 0 0) 0x7ff36000dac0 con 0x7ff370133ce0 2024-06-11T19:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff370133ce0 msgr2=0x7ff3701340f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff370133ce0 0x7ff3701340f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff36001d450 tx=0x7ff360000ed0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 1 -- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff370133ce0 msgr2=0x7ff3701340f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.725+0000 7ff36cff9640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff370133ce0 0x7ff3701340f0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff36001d450 tx=0x7ff360000ed0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 8481, 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:35.752 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.753 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.754 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.754 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.754 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.754 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.754 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.754 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.754 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.754 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:35.754 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:35.754 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.754 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:35.755 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:35.756 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.757 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:35.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.758 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:35.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.758 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:35.758 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:35.758 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.759 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:35.759 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.759 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.759 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:35.759 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.759 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.759 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:35.760 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:35.763 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.763 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:35.763 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:35.763 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.763 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.763 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:35.764 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:35.767 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:35.768 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.768 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.768 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:35.768 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:35.769 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.769 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.769 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.769 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:35.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:35.769 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.769 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.770 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:35.771 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:35.771 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.771 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.771 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.771 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:35.771 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:35.771 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.771 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.772 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:35.772 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.772 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:35.772 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.772 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.772 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.772 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:35.772 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:35.773 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.773 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.773 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.773 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:35.773 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:35.773 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:35.774 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:35.774 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.774 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.774 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:35.774 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:35.780 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 10 librados: watch_flush enter 2024-06-11T19:11:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 10 librados: watch_flush exit 2024-06-11T19:11:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 20 client.21575.objecter shutdown clearing up homeless session... 2024-06-11T19:11:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 10 client.21575.objecter successfully canceled tick 2024-06-11T19:11:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 1 -- 172.21.15.98:0/1346561152 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff35c03fde0 msgr2=0x7ff35c0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff35c03fde0 0x7ff35c0422a0 secure :-1 s=READY pgs=3989 cs=0 l=1 rev1=1 crypto rx=0x7ff364036000 tx=0x7ff364008030 comp rx=0 tx=0).stop 2024-06-11T19:11:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 10 monclient: shutdown 2024-06-11T19:11:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 1 -- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff37013e1a0 msgr2=0x7ff3701cdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 1 --2- 172.21.15.98:0/1346561152 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff37013e1a0 0x7ff3701cdcd0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7ff36800efc0 tx=0x7ff36800c490 comp rx=0 tx=0).stop 2024-06-11T19:11:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff374f3e640 1 -- 172.21.15.98:0/1346561152 reap_dead start 2024-06-11T19:11:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff374f3e640 1 -- 172.21.15.98:0/1346561152 reap_dead start 2024-06-11T19:11:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 1 -- 172.21.15.98:0/1346561152 shutdown_connections 2024-06-11T19:11:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 1 -- 172.21.15.98:0/1346561152 >> 172.21.15.98:0/1346561152 conn(0x7ff3700ab850 msgr2=0x7ff3700a61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 1 -- 172.21.15.98:0/1346561152 shutdown_connections 2024-06-11T19:11:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 1 -- 172.21.15.98:0/1346561152 wait complete. 2024-06-11T19:11:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.726+0000 7ff3769a1640 1 librados: shutdown 2024-06-11T19:11:35.784 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:35.784 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:11:35.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:11:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.897+0000 7f17c74e1640 10 monclient: get_monmap_and_config 2024-06-11T19:11:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.897+0000 7f17c74e1640 10 monclient: build_initial_monmap 2024-06-11T19:11:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.897+0000 7f17c74e1640 10 monclient: monmap: 2024-06-11T19:11:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:35.898512+0000 2024-06-11T19:11:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:35.898512+0000 2024-06-11T19:11:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:35.899 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 1 Processor -- start 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 1 -- start start 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 10 monclient: init 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 10 monclient: _add_conns ranks=[6,1,3,2,5,10,7,8,0,9,4] 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c00a4920 0x7f17c00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 10 monclient(hunting): picked mon.noname-d con 0x7f17c00a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f17c00a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17c00a2530 0x7f17c00a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 10 monclient(hunting): picked mon.noname-g con 0x7f17c00a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17c00a7140 con 0x7f17c00a5240 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f17c00a2b50 con 0x7f17c00a2530 2024-06-11T19:11:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f17c00a2e30 con 0x7f17c00a4920 2024-06-11T19:11:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c74e1640 10 monclient(hunting): authenticate will time out at 905.705746s 2024-06-11T19:11:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c527d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c00a4920 0x7f17c00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c527d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c00a4920 0x7f17c00a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39808/0 (socket says 172.21.15.98:39808) 2024-06-11T19:11:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c527d640 1 -- 172.21.15.98:0/2723694011 learned_addr learned my addr 172.21.15.98:0/2723694011 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c527d640 10 monclient(hunting): get_auth_request con 0x7f17c00a4920 auth_method 0 2024-06-11T19:11:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c527d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c527d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c527d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c527d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.898+0000 7f17c527d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c527d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c5a7e640 1 --2- 172.21.15.98:0/2723694011 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17c00a2530 0x7f17c00a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c5a7e640 10 monclient(hunting): get_auth_request con 0x7f17c00a2530 auth_method 0 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c5a7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c5a7e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c5a7e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c527d640 10 monclient(hunting): handle_auth_done global_id 54231 payload 274 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c527d640 1 -- 172.21.15.98:0/2723694011 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17c00a2530 msgr2=0x7f17c00a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c527d640 1 --2- 172.21.15.98:0/2723694011 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17c00a2530 0x7f17c00a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c527d640 1 -- 172.21.15.98:0/2723694011 >> v1:172.21.15.117:6789/0 conn(0x7f17c00a5240 legacy=0x7f17c00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c527d640 10 monclient: _finish_hunting 0 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c527d640 1 monclient: found mon.noname-d 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c527d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c527d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c527d640 1 -- 172.21.15.98:0/2723694011 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c00a3110 con 0x7f17c00a4920 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.899+0000 7f17c527d640 1 --2- 172.21.15.98:0/2723694011 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c00a4920 0x7f17c00a4d00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0098a0 tx=0x7f17bc02b840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=cc2e3811ac108b7c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 1 -- 172.21.15.98:0/2723694011 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17bc031a10 con 0x7f17c00a4920 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 10 monclient: dump: 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 10 monclient: _finish_auth 0 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 1 -- 172.21.15.98:0/2723694011 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17bc037070 con 0x7f17c00a4920 2024-06-11T19:11:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 5 monclient: authenticate success, global_id 54231 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 4 set_mon_vals no callback set 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 1 -- 172.21.15.98:0/2723694011 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17bc031c10 con 0x7f17c00a4920 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17b7fff640 10 monclient: dump: 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 10 monclient: shutdown 2024-06-11T19:11:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 1 -- 172.21.15.98:0/2723694011 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c00a4920 msgr2=0x7f17c00a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 1 --2- 172.21.15.98:0/2723694011 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c00a4920 0x7f17c00a4d00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0098a0 tx=0x7f17bc02b840 comp rx=0 tx=0).stop 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 1 -- 172.21.15.98:0/2723694011 shutdown_connections 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 1 --2- 172.21.15.98:0/2723694011 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17c00a2530 0x7f17c00a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 1 --2- 172.21.15.98:0/2723694011 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c00a4920 0x7f17c00a4d00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 1 -- 172.21.15.98:0/2723694011 >> 172.21.15.98:0/2723694011 conn(0x7f17c00ab850 msgr2=0x7f17c00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 1 -- 172.21.15.98:0/2723694011 shutdown_connections 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.900+0000 7f17c74e1640 1 -- 172.21.15.98:0/2723694011 wait complete. 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.901+0000 7f17c74e1640 10 monclient: build_initial_monmap 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.901+0000 7f17c74e1640 10 monclient: monmap: 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.901+0000 7f17c74e1640 1 librados: starting msgr at 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.901+0000 7f17c74e1640 1 librados: starting objecter 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.901+0000 7f17c74e1640 1 Processor -- start 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.901+0000 7f17c74e1640 1 -- start start 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.901+0000 7f17c74e1640 1 librados: setting wanted keys 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.901+0000 7f17c74e1640 1 librados: calling monclient init 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.901+0000 7f17c74e1640 10 monclient: init 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 10 monclient: _add_conns ranks=[7,4,2,5,8,1,3,0,6] 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f17c00a2530 0x7f17c01cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 10 monclient(hunting): picked mon.noname-i con 0x7f17c00a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17c00a4920 0x7f17c01ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 10 monclient(hunting): picked mon.noname-c con 0x7f17c00a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c00a5240 0x7f17c0123120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 10 monclient(hunting): picked mon.noname-b con 0x7f17c00a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f17c00a6da0 con 0x7f17c00a5240 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f17c0138e50 con 0x7f17c00a4920 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f17c00a65f0 con 0x7f17c00a2530 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 10 monclient(hunting): authenticate will time out at 905.709095s 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c00a5240 0x7f17c0123120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c4a7c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17c00a4920 0x7f17c01ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c00a5240 0x7f17c0123120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45730/0 (socket says 172.21.15.98:45730) 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c4a7c640 10 monclient(hunting): get_auth_request con 0x7f17c00a4920 auth_method 0 2024-06-11T19:11:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 1 -- 172.21.15.98:0/2499376478 learned_addr learned my addr 172.21.15.98:0/2499376478 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c4a7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c4a7c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c4a7c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 10 monclient(hunting): get_auth_request con 0x7f17c00a5240 auth_method 0 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c527d640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f17c00a2530 0x7f17c01cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c4a7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c4a7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c4a7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c527d640 10 monclient(hunting): get_auth_request con 0x7f17c00a2530 auth_method 0 2024-06-11T19:11:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c527d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c527d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c527d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c527d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c527d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c527d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 10 monclient(hunting): handle_auth_done global_id 21593 payload 824 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 1 -- 172.21.15.98:0/2499376478 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f17c00a2530 msgr2=0x7f17c01cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f17c00a2530 0x7f17c01cdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 1 -- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17c00a4920 msgr2=0x7f17c01ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17c00a4920 0x7f17c01ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 10 monclient: _finish_hunting 0 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 1 monclient: found mon.noname-b 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 1 -- 172.21.15.98:0/2499376478 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c0126cb0 con 0x7f17c00a5240 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c4a7c640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17c00a4920 0x7f17c01ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c5a7e640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c00a5240 0x7f17c0123120 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f17b8007e40 tx=0x7f17b800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 10 client.?.objecter ms_handle_connect 0x7f17c00a5240 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17b8014890 con 0x7f17c00a5240 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:11:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 10 monclient: dump: 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.923 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 10 monclient: _finish_auth 0 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 5 monclient: authenticate success, global_id 21593 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17b8019070 con 0x7f17c00a5240 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17b8013380 con 0x7f17c00a5240 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 10 monclient: _renew_subs 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:35.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17c74e1640 1 -- 172.21.15.98:0/2499376478 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c00a92f0 con 0x7f17c00a5240 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b77fe640 4 set_mon_vals no callback set 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.902+0000 7f17b5ffb640 10 monclient: dump: 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.925 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.903+0000 7f17c74e1640 10 client.21593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.903+0000 7f17c74e1640 10 monclient: _renew_subs 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.903+0000 7f17c74e1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.903+0000 7f17c74e1640 1 -- 172.21.15.98:0/2499376478 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c0124c30 con 0x7f17c00a5240 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.903+0000 7f17c74e1640 1 librados: init done 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.903+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f17b8014420 con 0x7f17c00a5240 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.903+0000 7f17c74e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:35.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.903+0000 7f17c74e1640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123660 0x7f17c0123a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.903+0000 7f17c74e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.903+0000 7f17c74e1640 1 -- 172.21.15.98:0/2499376478 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f17c0050830 con 0x7f17c0123660 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c527d640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123660 0x7f17c0123a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c527d640 10 monclient: get_auth_request con 0x7f17c0123660 auth_method 0 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c527d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c527d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c527d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f17a403fde0 0x7f17a40422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c527d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c527d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c527d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(80..80 src has 1..80) ==== 7956+0+0 (secure 0 0 0) 0x7f17b8052940 con 0x7f17c00a5240 2024-06-11T19:11:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 10 client.21593.objecter ms_dispatch 0x7f17c0000b90 osd_map(80..80 src has 1..80) 2024-06-11T19:11:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 3 client.21593.objecter handle_osd_map got epochs [80,80] > 0 2024-06-11T19:11:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 3 client.21593.objecter handle_osd_map decoding full epoch 80 2024-06-11T19:11:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 20 client.21593.objecter dump_active .. 0 homeless 2024-06-11T19:11:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c527d640 10 monclient(hunting): handle_auth_done global_id 22895 payload 824 2024-06-11T19:11:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c527d640 1 -- 172.21.15.98:0/2499376478 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17c01257f0 con 0x7f17c0123660 2024-06-11T19:11:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c4a7c640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f17a403fde0 0x7f17a40422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c4a7c640 10 monclient: get_auth_request con 0x7f17a403fde0 auth_method 0 2024-06-11T19:11:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 10 client.21593.objecter ms_handle_connect 0x7f17c0123660 2024-06-11T19:11:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 10 client.21593.objecter resend_mon_ops 2024-06-11T19:11:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c527d640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123660 0x7f17c0123a40 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f17c00a8420 tx=0x7f17bc004730 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17bc04a070 con 0x7f17c0123660 2024-06-11T19:11:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 10 monclient: dump: 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.930 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.930 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.930 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.930 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.930 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 10 client.21593.objecter ms_handle_connect 0x7f17a403fde0 2024-06-11T19:11:35.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17c4a7c640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f17a403fde0 0x7f17a40422a0 secure :-1 s=READY pgs=3991 cs=0 l=1 rev1=1 crypto rx=0x7f17c00a5620 tx=0x7f17b0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f17bc037040 con 0x7f17c0123660 2024-06-11T19:11:35.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:35.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:35.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123660 msgr2=0x7f17c0123a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123660 0x7f17c0123a40 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f17c00a8420 tx=0x7f17bc004730 comp rx=0 tx=0).stop 2024-06-11T19:11:35.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123660 msgr2=0x7f17c0123a40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:35.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.904+0000 7f17b5ffb640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123660 0x7f17c0123a40 secure :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f17c00a8420 tx=0x7f17bc004730 comp rx=0 tx=0).stop 2024-06-11T19:11:35.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c74e1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c74e1640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123f80 0x7f17c009e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c74e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c74e1640 1 -- 172.21.15.98:0/2499376478 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f17c00a4d00 con 0x7f17c0123f80 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c527d640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123f80 0x7f17c009e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c527d640 10 monclient: get_auth_request con 0x7f17c0123f80 auth_method 0 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c527d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c527d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c527d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c527d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c527d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c527d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c527d640 10 monclient(hunting): handle_auth_done global_id 22904 payload 824 2024-06-11T19:11:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c527d640 1 -- 172.21.15.98:0/2499376478 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f17c01279c0 con 0x7f17c0123f80 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17b5ffb640 10 client.21593.objecter ms_handle_connect 0x7f17c0123f80 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17b5ffb640 10 client.21593.objecter resend_mon_ops 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17c527d640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123f80 0x7f17c009e9f0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0097c0 tx=0x7f17bc031000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17bc04a040 con 0x7f17c0123f80 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17b5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.911+0000 7f17b5ffb640 10 monclient: dump: 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f17bc031bd0 con 0x7f17c0123f80 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17b5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17b5ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123f80 msgr2=0x7f17c009e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17b5ffb640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123f80 0x7f17c009e9f0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0097c0 tx=0x7f17bc031000 comp rx=0 tx=0).stop 2024-06-11T19:11:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17b5ffb640 1 -- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123f80 msgr2=0x7f17c009e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17b5ffb640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c0123f80 0x7f17c009e9f0 secure :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0097c0 tx=0x7f17bc031000 comp rx=0 tx=0).stop 2024-06-11T19:11:35.936 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 146148, 2024-06-11T19:11:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:11:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:35.938 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:35.938 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.938 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.938 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.938 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.938 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.938 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.938 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.938 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.938 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.938 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.938 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:35.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:35.953 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.955 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.956 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.956 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:35.956 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:35.956 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:35.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:35.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:35.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:35.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:35.957 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:35.958 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:35.959 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:35.959 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:35.959 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:35.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:35.959 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:35.959 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 10 librados: watch_flush enter 2024-06-11T19:11:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 10 librados: watch_flush exit 2024-06-11T19:11:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 20 client.21593.objecter shutdown clearing up homeless session... 2024-06-11T19:11:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 10 client.21593.objecter successfully canceled tick 2024-06-11T19:11:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 1 -- 172.21.15.98:0/2499376478 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f17a403fde0 msgr2=0x7f17a40422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f17a403fde0 0x7f17a40422a0 secure :-1 s=READY pgs=3991 cs=0 l=1 rev1=1 crypto rx=0x7f17c00a5620 tx=0x7f17b0009040 comp rx=0 tx=0).stop 2024-06-11T19:11:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 10 monclient: shutdown 2024-06-11T19:11:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 1 -- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c00a5240 msgr2=0x7f17c0123120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c5a7e640 1 -- 172.21.15.98:0/2499376478 reap_dead start 2024-06-11T19:11:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c00a5240 0x7f17c0123120 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f17b8007e40 tx=0x7f17b800e4b0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 1 -- 172.21.15.98:0/2499376478 shutdown_connections 2024-06-11T19:11:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 1 --2- 172.21.15.98:0/2499376478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c00a5240 0x7f17c0123120 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 1 -- 172.21.15.98:0/2499376478 >> 172.21.15.98:0/2499376478 conn(0x7f17c00ab850 msgr2=0x7f17c01be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 1 -- 172.21.15.98:0/2499376478 shutdown_connections 2024-06-11T19:11:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 1 -- 172.21.15.98:0/2499376478 wait complete. 2024-06-11T19:11:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:35.912+0000 7f17c74e1640 1 librados: shutdown 2024-06-11T19:11:35.962 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:35.962 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:11:35.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:11:36.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.063+0000 7f550348b640 10 monclient: get_monmap_and_config 2024-06-11T19:11:36.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.063+0000 7f550348b640 10 monclient: build_initial_monmap 2024-06-11T19:11:36.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.063+0000 7f550348b640 10 monclient: monmap: 2024-06-11T19:11:36.063 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:36.064973+0000 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:36.064973+0000 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:36.064 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:36.065 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:36.065 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:36.065 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.064+0000 7f550348b640 1 Processor -- start 2024-06-11T19:11:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.064+0000 7f550348b640 1 -- start start 2024-06-11T19:11:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.064+0000 7f550348b640 10 monclient: init 2024-06-11T19:11:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f550348b640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f550348b640 10 monclient: _add_conns ranks=[4,0,1,5,8,9,10,2,6,7,3] 2024-06-11T19:11:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f550348b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f54fc0a52a0 0x7f54fc0a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f550348b640 10 monclient(hunting): picked mon.noname-c con 0x7f54fc0a52a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f550348b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54fc09d520 addr v1:172.21.15.98:6789/0 2024-06-11T19:11:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f550348b640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f54fc073830 addr v1:172.21.15.117:6789/0 2024-06-11T19:11:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f550348b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54fc09d2e0 con 0x7f54fc09d520 2024-06-11T19:11:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f550348b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54fc0a5680 con 0x7f54fc073830 2024-06-11T19:11:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f550348b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f550348b640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f54fc09daa0 con 0x7f54fc0a52a0 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f550348b640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f550348b640 10 monclient(hunting): authenticate will time out at 905.872235s 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5500a26640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f54fc09d520 0x7f54fc0a3690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51586/0 (socket says 172.21.15.98:51586) 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5500a26640 1 -- 172.21.15.98:0/812850498 learned_addr learned my addr 172.21.15.98:0/812850498 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 1 --2- 172.21.15.98:0/812850498 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f54fc0a52a0 0x7f54fc0a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 10 monclient(hunting): get_auth_request con 0x7f54fc0a52a0 auth_method 0 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f54f3fff640 1 -- 172.21.15.98:0/812850498 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1910359248 0 0) 0x7f54fc09d2e0 con 0x7f54fc09d520 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f54f3fff640 10 monclient(hunting): _negotiate 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f54f3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f54f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f54f3fff640 10 monclient(hunting): my global_id is 31077 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f54f3fff640 1 -- 172.21.15.98:0/812850498 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54e8003520 con 0x7f54fc09d520 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 10 monclient(hunting): handle_auth_done global_id 22891 payload 274 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 1 -- 172.21.15.98:0/812850498 >> v1:172.21.15.117:6789/0 conn(0x7f54fc073830 legacy=0x7f54fc0a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 1 -- 172.21.15.98:0/812850498 >> v1:172.21.15.98:6789/0 conn(0x7f54fc09d520 legacy=0x7f54fc0a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 10 monclient: _finish_hunting 0 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 1 monclient: found mon.noname-c 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:11:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.065+0000 7f5501227640 1 -- 172.21.15.98:0/812850498 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54fc0a44b0 con 0x7f54fc0a52a0 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f5501227640 1 --2- 172.21.15.98:0/812850498 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f54fc0a52a0 0x7f54fc0a1910 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f54f80097d0 tx=0x7f54f802b810 comp rx=0 tx=0).ready entity=mon.4 client_cookie=d9db67053f27080c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 1 -- 172.21.15.98:0/812850498 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f54f80319e0 con 0x7f54fc0a52a0 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 10 monclient: dump: 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 10 monclient: _finish_auth 0 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 10 monclient: _check_auth_tickets 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f550348b640 5 monclient: authenticate success, global_id 22891 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 1 -- 172.21.15.98:0/812850498 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54f8037070 con 0x7f54fc0a52a0 2024-06-11T19:11:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f550348b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f550348b640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f550348b640 4 set_mon_vals no callback set 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 1 -- 172.21.15.98:0/812850498 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f54f8031ba0 con 0x7f54fc0a52a0 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.066+0000 7f54f3fff640 10 monclient: dump: 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.067+0000 7f550348b640 10 monclient: shutdown 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.067+0000 7f550348b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.067+0000 7f550348b640 1 -- 172.21.15.98:0/812850498 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f54fc0a52a0 msgr2=0x7f54fc0a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.067+0000 7f550348b640 1 --2- 172.21.15.98:0/812850498 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f54fc0a52a0 0x7f54fc0a1910 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f54f80097d0 tx=0x7f54f802b810 comp rx=0 tx=0).stop 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.067+0000 7f550348b640 1 -- 172.21.15.98:0/812850498 shutdown_connections 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.067+0000 7f550348b640 1 --2- 172.21.15.98:0/812850498 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f54fc0a52a0 0x7f54fc0a1910 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.067+0000 7f550348b640 1 -- 172.21.15.98:0/812850498 >> 172.21.15.98:0/812850498 conn(0x7f54fc0a4780 msgr2=0x7f54fc0a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.067+0000 7f550348b640 1 -- 172.21.15.98:0/812850498 shutdown_connections 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.067+0000 7f550348b640 1 -- 172.21.15.98:0/812850498 wait complete. 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.067+0000 7f550348b640 10 monclient: build_initial_monmap 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.067+0000 7f550348b640 10 monclient: monmap: 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:36.082 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 1 librados: starting msgr at 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 1 librados: starting objecter 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 1 Processor -- start 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 1 -- start start 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 1 librados: setting wanted keys 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 1 librados: calling monclient init 2024-06-11T19:11:36.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 10 monclient: init 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 10 monclient: _add_conns ranks=[4,6,8,1,3,2,0,5,7] 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f54fc073830 0x7f54fc11eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 10 monclient(hunting): picked mon.noname-c con 0x7f54fc073830 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f54fc09d520 0x7f54fc11f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 10 monclient(hunting): picked mon.noname-d con 0x7f54fc09d520 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f54fc0a52a0 0x7f54fc11f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 10 monclient(hunting): picked mon.noname-e con 0x7f54fc0a52a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f54fc135bd0 con 0x7f54fc073830 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f54fc0a1e50 con 0x7f54fc09d520 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f54fc0a3f80 con 0x7f54fc0a52a0 2024-06-11T19:11:36.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f550348b640 10 monclient(hunting): authenticate will time out at 905.875645s 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501227640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f54fc073830 0x7f54fc11eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501227640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f54fc073830 0x7f54fc11eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:49998/0 (socket says 172.21.15.98:49998) 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501227640 1 -- 172.21.15.98:0/4137068339 learned_addr learned my addr 172.21.15.98:0/4137068339 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501227640 10 monclient(hunting): get_auth_request con 0x7f54fc073830 auth_method 0 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501227640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501227640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501227640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5500a26640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f54fc09d520 0x7f54fc11f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5500a26640 10 monclient(hunting): get_auth_request con 0x7f54fc09d520 auth_method 0 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501a28640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f54fc0a52a0 0x7f54fc11f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5500a26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5500a26640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5500a26640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501a28640 10 monclient(hunting): get_auth_request con 0x7f54fc0a52a0 auth_method 0 2024-06-11T19:11:36.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501a28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501a28640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501a28640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501227640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501227640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501227640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5500a26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5500a26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5500a26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501a28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501a28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.068+0000 7f5501a28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f5501227640 10 monclient(hunting): handle_auth_done global_id 22900 payload 824 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f5501227640 1 -- 172.21.15.98:0/4137068339 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f54fc0a52a0 msgr2=0x7f54fc11f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f5501227640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f54fc0a52a0 0x7f54fc11f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f5501227640 1 -- 172.21.15.98:0/4137068339 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f54fc09d520 msgr2=0x7f54fc11f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f5501227640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f54fc09d520 0x7f54fc11f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f5501227640 10 monclient: _finish_hunting 0 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f5501227640 1 monclient: found mon.noname-c 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f5501227640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f5501227640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f5501227640 1 -- 172.21.15.98:0/4137068339 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54fc1208e0 con 0x7f54fc073830 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f5500a26640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f54fc09d520 0x7f54fc11f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f5501227640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f54fc073830 0x7f54fc11eba0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f54f80098f0 tx=0x7f54f80046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 10 client.?.objecter ms_handle_connect 0x7f54fc073830 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f54f8006e10 con 0x7f54fc073830 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 10 monclient: dump: 2024-06-11T19:11:36.087 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 10 monclient: _finish_auth 0 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f550348b640 5 monclient: authenticate success, global_id 22900 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54f80319e0 con 0x7f54fc073830 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f550348b640 10 monclient: _renew_subs 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f550348b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f550348b640 1 -- 172.21.15.98:0/4137068339 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54fc0689e0 con 0x7f54fc073830 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f54f8047480 con 0x7f54fc073830 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f1ffb640 10 monclient: dump: 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f550348b640 10 client.22900.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f550348b640 10 monclient: _renew_subs 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f550348b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f550348b640 1 -- 172.21.15.98:0/4137068339 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f54fc120370 con 0x7f54fc073830 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f550348b640 1 librados: init done 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.069+0000 7f54f37fe640 4 set_mon_vals no callback set 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f54f8031070 con 0x7f54fc073830 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f550348b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f550348b640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cd630 0x7f54fc1cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f550348b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f550348b640 1 -- 172.21.15.98:0/4137068339 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f54fc135d50 con 0x7f54fc1cd630 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f54f1ffb640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f54e003fdb0 0x7f54e0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(80..80 src has 1..80) ==== 7956+0+0 (secure 0 0 0) 0x7f54f8074100 con 0x7f54fc073830 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f54f1ffb640 10 client.22900.objecter ms_dispatch 0x7f54fc000b90 osd_map(80..80 src has 1..80) 2024-06-11T19:11:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f54f1ffb640 3 client.22900.objecter handle_osd_map got epochs [80,80] > 0 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f54f1ffb640 3 client.22900.objecter handle_osd_map decoding full epoch 80 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f54f1ffb640 20 client.22900.objecter dump_active .. 0 homeless 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f5500a26640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cd630 0x7f54fc1cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f5500a26640 10 monclient: get_auth_request con 0x7f54fc1cd630 auth_method 0 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f5500a26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f5500a26640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f5500a26640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f5501a28640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f54e003fdb0 0x7f54e0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.070+0000 7f5501a28640 10 monclient: get_auth_request con 0x7f54e003fdb0 auth_method 0 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f5500a26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f5500a26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f5500a26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f5500a26640 10 monclient(hunting): handle_auth_done global_id 30826 payload 824 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f5500a26640 1 -- 172.21.15.98:0/4137068339 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54fc0a1910 con 0x7f54fc1cd630 2024-06-11T19:11:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 10 client.22900.objecter ms_handle_connect 0x7f54e003fdb0 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f5501a28640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f54e003fdb0 0x7f54e0042270 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 crypto rx=0x7f54fc068470 tx=0x7f54f4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 10 client.22900.objecter ms_handle_connect 0x7f54fc1cd630 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 10 client.22900.objecter resend_mon_ops 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f5500a26640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cd630 0x7f54fc1cda10 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f54ec005880 tx=0x7f54ec0076e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f54ec006050 con 0x7f54fc1cd630 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 10 monclient: dump: 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f54ec018070 con 0x7f54fc1cd630 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cd630 msgr2=0x7f54fc1cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cd630 0x7f54fc1cda10 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f54ec005880 tx=0x7f54ec0076e0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cd630 msgr2=0x7f54fc1cda10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.071+0000 7f54f1ffb640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cd630 0x7f54fc1cda10 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f550348b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f550348b640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cdf50 0x7f54fc06f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f550348b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f550348b640 1 -- 172.21.15.98:0/4137068339 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f54fc050830 con 0x7f54fc1cdf50 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f5500a26640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cdf50 0x7f54fc06f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f5500a26640 10 monclient: get_auth_request con 0x7f54fc1cdf50 auth_method 0 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f5500a26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f5500a26640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f5500a26640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f5500a26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f5500a26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f5500a26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f5500a26640 10 monclient(hunting): handle_auth_done global_id 30835 payload 824 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f5500a26640 1 -- 172.21.15.98:0/4137068339 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f54fc06d540 con 0x7f54fc1cdf50 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f5500a26640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cdf50 0x7f54fc06f510 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f54fc067d20 tx=0x7f54ec0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f54f1ffb640 10 client.22900.objecter ms_handle_connect 0x7f54fc1cdf50 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.078+0000 7f54f1ffb640 10 client.22900.objecter resend_mon_ops 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f54ec00faf0 con 0x7f54fc1cdf50 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f54f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f54f1ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f54f1ffb640 10 monclient: dump: 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f54ec00f680 con 0x7f54fc1cdf50 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f54f1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f54f1ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cdf50 msgr2=0x7f54fc06f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f54f1ffb640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cdf50 0x7f54fc06f510 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f54fc067d20 tx=0x7f54ec0057f0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f54f1ffb640 1 -- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cdf50 msgr2=0x7f54fc06f510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f54f1ffb640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54fc1cdf50 0x7f54fc06f510 secure :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f54fc067d20 tx=0x7f54ec0057f0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.100 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:36.100 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 146298, 2024-06-11T19:11:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.101 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:36.102 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:36.103 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:36.105 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:36.123 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:36.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 10 librados: watch_flush enter 2024-06-11T19:11:36.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 10 librados: watch_flush exit 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 20 client.22900.objecter shutdown clearing up homeless session... 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 10 client.22900.objecter successfully canceled tick 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 1 -- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f54e003fdb0 msgr2=0x7f54e0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f54e003fdb0 0x7f54e0042270 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 crypto rx=0x7f54fc068470 tx=0x7f54f4009040 comp rx=0 tx=0).stop 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 10 monclient: shutdown 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 1 -- 172.21.15.98:0/4137068339 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f54fc073830 msgr2=0x7f54fc11eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 1 --2- 172.21.15.98:0/4137068339 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f54fc073830 0x7f54fc11eba0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f54f80098f0 tx=0x7f54f80046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f5501a28640 1 -- 172.21.15.98:0/4137068339 reap_dead start 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f5501a28640 1 -- 172.21.15.98:0/4137068339 reap_dead start 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 1 -- 172.21.15.98:0/4137068339 shutdown_connections 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 1 -- 172.21.15.98:0/4137068339 >> 172.21.15.98:0/4137068339 conn(0x7f54fc0a4780 msgr2=0x7f54fc1bb410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 1 -- 172.21.15.98:0/4137068339 shutdown_connections 2024-06-11T19:11:36.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 1 -- 172.21.15.98:0/4137068339 wait complete. 2024-06-11T19:11:36.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.079+0000 7f550348b640 1 librados: shutdown 2024-06-11T19:11:36.126 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:36.126 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:11:36.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:11:36.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.226+0000 7f48e53f3640 10 monclient: get_monmap_and_config 2024-06-11T19:11:36.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.226+0000 7f48e53f3640 10 monclient: build_initial_monmap 2024-06-11T19:11:36.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.226+0000 7f48e53f3640 10 monclient: monmap: 2024-06-11T19:11:36.226 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:36.226 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:36.227843+0000 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:36.227843+0000 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:36.227 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:36.228 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:36.228 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 1 Processor -- start 2024-06-11T19:11:36.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 1 -- start start 2024-06-11T19:11:36.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 10 monclient: init 2024-06-11T19:11:36.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:36.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 10 monclient: _add_conns ranks=[8,7,1,0,2,6,10,5,4,3,9] 2024-06-11T19:11:36.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48e00a4920 0x7f48e00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 10 monclient(hunting): picked mon.noname-e con 0x7f48e00a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:36.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f48e00a5240 0x7f48e00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 10 monclient(hunting): picked mon.noname-i con 0x7f48e00a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:36.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f48e00a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:11:36.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48e00a7140 con 0x7f48e00a2530 2024-06-11T19:11:36.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f48e00a2b50 con 0x7f48e00a5240 2024-06-11T19:11:36.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f48e00a2e30 con 0x7f48e00a4920 2024-06-11T19:11:36.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:36.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.227+0000 7f48e53f3640 10 monclient(hunting): authenticate will time out at 906.034954s 2024-06-11T19:11:36.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48e00a4920 0x7f48e00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48e00a4920 0x7f48e00a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:45584/0 (socket says 172.21.15.98:45584) 2024-06-11T19:11:36.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 1 -- 172.21.15.98:0/3178135541 learned_addr learned my addr 172.21.15.98:0/3178135541 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:36.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 10 monclient(hunting): get_auth_request con 0x7f48e00a4920 auth_method 0 2024-06-11T19:11:36.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48de7fc640 1 --2- 172.21.15.98:0/3178135541 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f48e00a5240 0x7f48e00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48de7fc640 10 monclient(hunting): get_auth_request con 0x7f48e00a5240 auth_method 0 2024-06-11T19:11:36.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48de7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48de7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48de7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48de7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48de7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 10 monclient(hunting): handle_auth_done global_id 22922 payload 274 2024-06-11T19:11:36.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 1 -- 172.21.15.98:0/3178135541 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f48e00a5240 msgr2=0x7f48e00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 1 --2- 172.21.15.98:0/3178135541 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f48e00a5240 0x7f48e00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 1 -- 172.21.15.98:0/3178135541 >> v1:172.21.15.117:6789/0 conn(0x7f48e00a2530 legacy=0x7f48e00a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 10 monclient: _finish_hunting 0 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 1 monclient: found mon.noname-e 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.228+0000 7f48deffd640 1 -- 172.21.15.98:0/3178135541 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e00a3110 con 0x7f48e00a4920 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48deffd640 1 --2- 172.21.15.98:0/3178135541 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48e00a4920 0x7f48e00a4d00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009e80 tx=0x7f48d002be10 comp rx=0 tx=0).ready entity=mon.8 client_cookie=d902813782dc2fa1 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 1 -- 172.21.15.98:0/3178135541 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48d002cc20 con 0x7f48e00a4920 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 10 monclient: dump: 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.243 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.244 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 10 monclient: _finish_auth 0 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 10 monclient: _check_auth_tickets 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 1 -- 172.21.15.98:0/3178135541 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d002cde0 con 0x7f48e00a4920 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 5 monclient: authenticate success, global_id 22922 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 1 -- 172.21.15.98:0/3178135541 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48d0034860 con 0x7f48e00a4920 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 4 set_mon_vals no callback set 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:36.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48ddffb640 10 monclient: dump: 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.246 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 10 monclient: shutdown 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 1 -- 172.21.15.98:0/3178135541 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48e00a4920 msgr2=0x7f48e00a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 1 --2- 172.21.15.98:0/3178135541 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48e00a4920 0x7f48e00a4d00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009e80 tx=0x7f48d002be10 comp rx=0 tx=0).stop 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 1 -- 172.21.15.98:0/3178135541 shutdown_connections 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 1 --2- 172.21.15.98:0/3178135541 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f48e00a5240 0x7f48e00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 1 --2- 172.21.15.98:0/3178135541 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48e00a4920 0x7f48e00a4d00 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 1 -- 172.21.15.98:0/3178135541 >> 172.21.15.98:0/3178135541 conn(0x7f48e00ab850 msgr2=0x7f48e00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 1 -- 172.21.15.98:0/3178135541 shutdown_connections 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.229+0000 7f48e53f3640 1 -- 172.21.15.98:0/3178135541 wait complete. 2024-06-11T19:11:36.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.230+0000 7f48e53f3640 10 monclient: build_initial_monmap 2024-06-11T19:11:36.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.230+0000 7f48e53f3640 10 monclient: monmap: 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.230+0000 7f48e53f3640 1 librados: starting msgr at 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.230+0000 7f48e53f3640 1 librados: starting objecter 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.230+0000 7f48e53f3640 1 Processor -- start 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.230+0000 7f48e53f3640 1 -- start start 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.230+0000 7f48e53f3640 1 librados: setting wanted keys 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.230+0000 7f48e53f3640 1 librados: calling monclient init 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.230+0000 7f48e53f3640 10 monclient: init 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 10 monclient: _add_conns ranks=[4,6,0,3,2,7,1,8,5] 2024-06-11T19:11:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f48e00a2530 0x7f48e01cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 10 monclient(hunting): picked mon.noname-c con 0x7f48e00a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48e00a4920 0x7f48e01ce330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 10 monclient(hunting): picked mon.noname-d con 0x7f48e00a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48e00a5240 0x7f48e0125170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 10 monclient(hunting): picked mon.noname-a con 0x7f48e00a5240 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48e00a6da0 con 0x7f48e00a5240 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f48e0138e50 con 0x7f48e00a2530 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f48e00a65f0 con 0x7f48e00a4920 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:36.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48e53f3640 10 monclient(hunting): authenticate will time out at 906.038377s 2024-06-11T19:11:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48deffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f48e00a2530 0x7f48e01cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48e00a4920 0x7f48e01ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48e00a4920 0x7f48e01ce330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39834/0 (socket says 172.21.15.98:39834) 2024-06-11T19:11:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48deffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f48e00a2530 0x7f48e01cddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:50004/0 (socket says 172.21.15.98:50004) 2024-06-11T19:11:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 1 -- 172.21.15.98:0/2358568393 learned_addr learned my addr 172.21.15.98:0/2358568393 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 10 monclient(hunting): get_auth_request con 0x7f48e00a4920 auth_method 0 2024-06-11T19:11:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48deffd640 10 monclient(hunting): get_auth_request con 0x7f48e00a2530 auth_method 0 2024-06-11T19:11:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48deffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48deffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48df7fe640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48e00a5240 0x7f48e0125170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48df7fe640 10 monclient(hunting): get_auth_request con 0x7f48e00a5240 auth_method 0 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48df7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 10 monclient(hunting): handle_auth_done global_id 54249 payload 824 2024-06-11T19:11:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 1 -- 172.21.15.98:0/2358568393 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f48e00a2530 msgr2=0x7f48e01cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f48e00a2530 0x7f48e01cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 1 -- 172.21.15.98:0/2358568393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48e00a5240 msgr2=0x7f48e0125170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48e00a5240 0x7f48e0125170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:11:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 1 monclient: found mon.noname-d 2024-06-11T19:11:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 1 -- 172.21.15.98:0/2358568393 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e01272d0 con 0x7f48e00a4920 2024-06-11T19:11:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48deffd640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f48e00a2530 0x7f48e01cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48de7fc640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48e00a4920 0x7f48e01ce330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f48d400e8d0 tx=0x7f48d400ed90 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48c77fe640 10 client.?.objecter ms_handle_connect 0x7f48e00a4920 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.231+0000 7f48c77fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48d4017070 con 0x7f48e00a4920 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 10 monclient: dump: 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 10 monclient: _finish_auth 0 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 10 monclient: _check_auth_tickets 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d400cdb0 con 0x7f48e00a4920 2024-06-11T19:11:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48e53f3640 5 monclient: authenticate success, global_id 54249 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48e53f3640 10 monclient: _renew_subs 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48e53f3640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48e53f3640 1 -- 172.21.15.98:0/2358568393 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48e0128b30 con 0x7f48e00a4920 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48d40138e0 con 0x7f48e00a4920 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48e53f3640 10 client.54249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48c77fe640 10 monclient: dump: 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48e53f3640 10 monclient: _renew_subs 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48e53f3640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48dd7fa640 4 set_mon_vals no callback set 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48e53f3640 1 -- 172.21.15.98:0/2358568393 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f48e00a9030 con 0x7f48e00a4920 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.232+0000 7f48e53f3640 1 librados: init done 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48e53f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48e53f3640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e01256b0 0x7f48e0125a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48e53f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48e53f3640 1 -- 172.21.15.98:0/2358568393 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f48e0050830 con 0x7f48e01256b0 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f48d401a070 con 0x7f48e00a4920 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48deffd640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e01256b0 0x7f48e0125a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48deffd640 10 monclient: get_auth_request con 0x7f48e01256b0 auth_method 0 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48deffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48deffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48c77fe640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f48cc03fde0 0x7f48cc0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(80..80 src has 1..80) ==== 7956+0+0 (secure 0 0 0) 0x7f48d400f040 con 0x7f48e00a4920 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48c77fe640 10 client.54249.objecter ms_dispatch 0x7f48e0000b90 osd_map(80..80 src has 1..80) 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48c77fe640 3 client.54249.objecter handle_osd_map got epochs [80,80] > 0 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48c77fe640 3 client.54249.objecter handle_osd_map decoding full epoch 80 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48c77fe640 20 client.54249.objecter dump_active .. 0 homeless 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48df7fe640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f48cc03fde0 0x7f48cc0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48deffd640 10 monclient(hunting): handle_auth_done global_id 21657 payload 824 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48deffd640 1 -- 172.21.15.98:0/2358568393 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f48e00a4d00 con 0x7f48e01256b0 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48df7fe640 10 monclient: get_auth_request con 0x7f48cc03fde0 auth_method 0 2024-06-11T19:11:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48deffd640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e01256b0 0x7f48e0125a90 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f48e00abf80 tx=0x7f48d0004a40 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48c77fe640 10 client.54249.objecter ms_handle_connect 0x7f48e01256b0 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.233+0000 7f48c77fe640 10 client.54249.objecter resend_mon_ops 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48d003e8a0 con 0x7f48e01256b0 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48df7fe640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f48cc03fde0 0x7f48cc0422a0 secure :-1 s=READY pgs=3993 cs=0 l=1 rev1=1 crypto rx=0x7f48d8000f60 tx=0x7f48d8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48c77fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48c77fe640 10 monclient: dump: 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48c77fe640 10 client.54249.objecter ms_handle_connect 0x7f48cc03fde0 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f48d003e430 con 0x7f48e01256b0 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48c77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48c77fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e01256b0 msgr2=0x7f48e0125a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48c77fe640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e01256b0 0x7f48e0125a90 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f48e00abf80 tx=0x7f48d0004a40 comp rx=0 tx=0).stop 2024-06-11T19:11:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e01256b0 msgr2=0x7f48e0125a90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.234+0000 7f48c77fe640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e01256b0 0x7f48e0125a90 secure :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f48e00abf80 tx=0x7f48d0004a40 comp rx=0 tx=0).stop 2024-06-11T19:11:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.240+0000 7f48e53f3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.240+0000 7f48e53f3640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e0125fd0 0x7f48e01215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.240+0000 7f48e53f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.240+0000 7f48e53f3640 1 -- 172.21.15.98:0/2358568393 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f48e006c300 con 0x7f48e0125fd0 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48deffd640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e0125fd0 0x7f48e01215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48deffd640 10 monclient: get_auth_request con 0x7f48e0125fd0 auth_method 0 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48deffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48deffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48deffd640 10 monclient(hunting): handle_auth_done global_id 21666 payload 824 2024-06-11T19:11:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48deffd640 1 -- 172.21.15.98:0/2358568393 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f48e01263e0 con 0x7f48e0125fd0 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48deffd640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e0125fd0 0x7f48e01215f0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f48d002be50 tx=0x7f48d0009810 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 10 client.54249.objecter ms_handle_connect 0x7f48e0125fd0 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 10 client.54249.objecter resend_mon_ops 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48d0049dd0 con 0x7f48e0125fd0 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 10 monclient: dump: 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f48d0049960 con 0x7f48e0125fd0 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e0125fd0 msgr2=0x7f48e01215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e0125fd0 0x7f48e01215f0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f48d002be50 tx=0x7f48d0009810 comp rx=0 tx=0).stop 2024-06-11T19:11:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 1 -- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e0125fd0 msgr2=0x7f48e01215f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.241+0000 7f48c77fe640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48e0125fd0 0x7f48e01215f0 secure :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f48d002be50 tx=0x7f48d0009810 comp rx=0 tx=0).stop 2024-06-11T19:11:36.268 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 146459, 2024-06-11T19:11:36.268 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:36.269 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.270 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:36.272 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:36.273 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:36.274 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.276 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:36.278 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:36.280 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:36.281 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:36.282 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:36.284 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:36.285 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:36.287 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:11:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:36.290 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:36.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 10 librados: watch_flush enter 2024-06-11T19:11:36.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 10 librados: watch_flush exit 2024-06-11T19:11:36.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 20 client.54249.objecter shutdown clearing up homeless session... 2024-06-11T19:11:36.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 10 client.54249.objecter successfully canceled tick 2024-06-11T19:11:36.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 1 -- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f48cc03fde0 msgr2=0x7f48cc0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f48cc03fde0 0x7f48cc0422a0 secure :-1 s=READY pgs=3993 cs=0 l=1 rev1=1 crypto rx=0x7f48d8000f60 tx=0x7f48d8009040 comp rx=0 tx=0).stop 2024-06-11T19:11:36.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 10 monclient: shutdown 2024-06-11T19:11:36.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:36.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 1 -- 172.21.15.98:0/2358568393 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48e00a4920 msgr2=0x7f48e01ce330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 1 --2- 172.21.15.98:0/2358568393 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48e00a4920 0x7f48e01ce330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f48d400e8d0 tx=0x7f48d400ed90 comp rx=0 tx=0).stop 2024-06-11T19:11:36.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48df7fe640 1 -- 172.21.15.98:0/2358568393 reap_dead start 2024-06-11T19:11:36.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48df7fe640 1 -- 172.21.15.98:0/2358568393 reap_dead start 2024-06-11T19:11:36.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 1 -- 172.21.15.98:0/2358568393 shutdown_connections 2024-06-11T19:11:36.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 1 -- 172.21.15.98:0/2358568393 >> 172.21.15.98:0/2358568393 conn(0x7f48e00ab850 msgr2=0x7f48e009c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:36.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 1 -- 172.21.15.98:0/2358568393 shutdown_connections 2024-06-11T19:11:36.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 1 -- 172.21.15.98:0/2358568393 wait complete. 2024-06-11T19:11:36.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.242+0000 7f48e53f3640 1 librados: shutdown 2024-06-11T19:11:36.293 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:36.293 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:11:36.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:11:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.395+0000 7fb660880640 10 monclient: get_monmap_and_config 2024-06-11T19:11:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.395+0000 7fb660880640 10 monclient: build_initial_monmap 2024-06-11T19:11:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.395+0000 7fb660880640 10 monclient: monmap: 2024-06-11T19:11:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:36.396293+0000 2024-06-11T19:11:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:36.396293+0000 2024-06-11T19:11:36.395 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:36.396 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.395+0000 7fb660880640 1 Processor -- start 2024-06-11T19:11:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.395+0000 7fb660880640 1 -- start start 2024-06-11T19:11:36.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.395+0000 7fb660880640 10 monclient: init 2024-06-11T19:11:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 10 monclient: _add_conns ranks=[4,1,8,7,6,9,3,0,2,5,10] 2024-06-11T19:11:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb65c09d560 0x7fb65c09d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 10 monclient(hunting): picked mon.noname-c con 0x7fb65c09d560 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fb65c136c80 addr v1:172.21.15.117:6789/0 2024-06-11T19:11:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb65c0a29e0 0x7fb65c13eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 10 monclient(hunting): picked mon.noname-e con 0x7fb65c0a29e0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb65c09b840 con 0x7fb65c136c80 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb65c1376d0 con 0x7fb65c09d560 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb65c09be00 con 0x7fb65c0a29e0 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb660880640 10 monclient(hunting): authenticate will time out at 906.203788s 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.396+0000 7fb65ad9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb65c0a29e0 0x7fb65c13eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb65c0a29e0 0x7fb65c13eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:45598/0 (socket says 172.21.15.98:45598) 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 1 -- 172.21.15.98:0/2256342572 learned_addr learned my addr 172.21.15.98:0/2256342572 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 10 monclient(hunting): get_auth_request con 0x7fb65c0a29e0 auth_method 0 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65a59c640 1 --2- 172.21.15.98:0/2256342572 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb65c09d560 0x7fb65c09d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65a59c640 10 monclient(hunting): get_auth_request con 0x7fb65c09d560 auth_method 0 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 10 monclient(hunting): handle_auth_done global_id 22931 payload 274 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 1 -- 172.21.15.98:0/2256342572 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb65c09d560 msgr2=0x7fb65c09d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 1 --2- 172.21.15.98:0/2256342572 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb65c09d560 0x7fb65c09d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 1 -- 172.21.15.98:0/2256342572 >> v1:172.21.15.117:6789/0 conn(0x7fb65c136c80 legacy=0x7fb65c0a22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:11:36.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 1 monclient: found mon.noname-e 2024-06-11T19:11:36.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:36.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:11:36.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.397+0000 7fb65ad9d640 1 -- 172.21.15.98:0/2256342572 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65c09e180 con 0x7fb65c0a29e0 2024-06-11T19:11:36.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.398+0000 7fb65ad9d640 1 --2- 172.21.15.98:0/2256342572 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb65c0a29e0 0x7fb65c13eab0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fb654009eb0 tx=0x7fb65402be00 comp rx=0 tx=0).ready entity=mon.8 client_cookie=e373d1a903c7d14c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 1 -- 172.21.15.98:0/2256342572 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb65402cc10 con 0x7fb65c0a29e0 2024-06-11T19:11:36.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:11:36.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 10 monclient: dump: 2024-06-11T19:11:36.410 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 10 monclient: _finish_auth 0 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 10 monclient: _check_auth_tickets 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 1 -- 172.21.15.98:0/2256342572 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb65402cdd0 con 0x7fb65c0a29e0 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb660880640 5 monclient: authenticate success, global_id 22931 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb660880640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb660880640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb660880640 4 set_mon_vals no callback set 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 1 -- 172.21.15.98:0/2256342572 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb654034860 con 0x7fb65c0a29e0 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:11:36.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb65959a640 10 monclient: dump: 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.413 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb660880640 10 monclient: shutdown 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb660880640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb660880640 1 -- 172.21.15.98:0/2256342572 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb65c0a29e0 msgr2=0x7fb65c13eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.399+0000 7fb660880640 1 --2- 172.21.15.98:0/2256342572 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb65c0a29e0 0x7fb65c13eab0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fb654009eb0 tx=0x7fb65402be00 comp rx=0 tx=0).stop 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.400+0000 7fb660880640 1 -- 172.21.15.98:0/2256342572 shutdown_connections 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.400+0000 7fb660880640 1 --2- 172.21.15.98:0/2256342572 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb65c0a29e0 0x7fb65c13eab0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.400+0000 7fb660880640 1 --2- 172.21.15.98:0/2256342572 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb65c09d560 0x7fb65c09d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.400+0000 7fb660880640 1 -- 172.21.15.98:0/2256342572 >> 172.21.15.98:0/2256342572 conn(0x7fb65c0a7850 msgr2=0x7fb65c0a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.400+0000 7fb660880640 1 -- 172.21.15.98:0/2256342572 shutdown_connections 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.400+0000 7fb660880640 1 -- 172.21.15.98:0/2256342572 wait complete. 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.400+0000 7fb660880640 10 monclient: build_initial_monmap 2024-06-11T19:11:36.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.400+0000 7fb660880640 10 monclient: monmap: 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:36.415 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 1 librados: starting msgr at 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 1 librados: starting objecter 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 1 Processor -- start 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 1 -- start start 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 1 librados: setting wanted keys 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 1 librados: calling monclient init 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 10 monclient: init 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 10 monclient: _add_conns ranks=[0,1,6,5,3,8,4,2,7] 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb65c09d560 0x7fb65c1c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 10 monclient(hunting): picked mon.noname-a con 0x7fb65c09d560 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb65c0a29e0 0x7fb65c1c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 10 monclient(hunting): picked mon.noname-f con 0x7fb65c0a29e0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:11:36.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb65c136c80 0x7fb65c12dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 10 monclient(hunting): picked mon.noname-d con 0x7fb65c136c80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c1370a0 con 0x7fb65c09d560 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c09bb20 con 0x7fb65c0a29e0 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb65c0769a0 con 0x7fb65c136c80 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb660880640 10 monclient(hunting): authenticate will time out at 906.208741s 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb65c09d560 0x7fb65c1c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb65c09d560 0x7fb65c1c9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43548/0 (socket says 172.21.15.98:43548) 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65a59c640 1 -- 172.21.15.98:0/3264079469 learned_addr learned my addr 172.21.15.98:0/3264079469 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65a59c640 10 monclient(hunting): get_auth_request con 0x7fb65c09d560 auth_method 0 2024-06-11T19:11:36.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65ad9d640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb65c136c80 0x7fb65c12dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65ad9d640 10 monclient(hunting): get_auth_request con 0x7fb65c136c80 auth_method 0 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.401+0000 7fb65ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb659d9b640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb65c0a29e0 0x7fb65c1c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb659d9b640 10 monclient(hunting): get_auth_request con 0x7fb65c0a29e0 auth_method 0 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb659d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb659d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb659d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb65a59c640 10 monclient(hunting): handle_auth_done global_id 31095 payload 824 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb65a59c640 1 -- 172.21.15.98:0/3264079469 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb65c136c80 msgr2=0x7fb65c12dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb65a59c640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb65c136c80 0x7fb65c12dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb65a59c640 1 -- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb65c0a29e0 msgr2=0x7fb65c1c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb65a59c640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb65c0a29e0 0x7fb65c1c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb65a59c640 10 monclient: _finish_hunting 0 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb65a59c640 1 monclient: found mon.noname-a 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb65a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb65a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:11:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb65a59c640 1 -- 172.21.15.98:0/3264079469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65c1328c0 con 0x7fb65c09d560 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb65ad9d640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb65c136c80 0x7fb65c12dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb65a59c640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb65c09d560 0x7fb65c1c9970 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fb64c009ca0 tx=0x7fb64c0134a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 10 client.?.objecter ms_handle_connect 0x7fb65c09d560 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb64c020070 con 0x7fb65c09d560 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 10 monclient: dump: 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.420 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 10 monclient: _finish_auth 0 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb64c01a3a0 con 0x7fb65c09d560 2024-06-11T19:11:36.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb660880640 5 monclient: authenticate success, global_id 31095 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb660880640 10 monclient: _renew_subs 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb660880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb660880640 1 -- 172.21.15.98:0/3264079469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb65c0a5760 con 0x7fb65c09d560 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb64c01a9c0 con 0x7fb65c09d560 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb64affd640 10 monclient: dump: 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.422 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb660880640 10 client.31095.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb660880640 10 monclient: _renew_subs 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb660880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:11:36.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb660880640 1 -- 172.21.15.98:0/3264079469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb65c0a5310 con 0x7fb65c09d560 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb660880640 1 librados: init done 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.402+0000 7fb658d99640 4 set_mon_vals no callback set 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb660880640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb660880640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12e530 0x7fb65c12e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb660880640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb660880640 1 -- 172.21.15.98:0/3264079469 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fb65c050830 con 0x7fb65c12e530 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb659d9b640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12e530 0x7fb65c12e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb659d9b640 10 monclient: get_auth_request con 0x7fb65c12e530 auth_method 0 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb659d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb659d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb659d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fb64c01a550 con 0x7fb65c09d560 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb659d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb659d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb659d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb64affd640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb63c03fdb0 0x7fb63c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb659d9b640 10 monclient(hunting): handle_auth_done global_id 22082 payload 824 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb659d9b640 1 -- 172.21.15.98:0/3264079469 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb65c0a2e60 con 0x7fb65c12e530 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7956+0+0 (secure 0 0 0) 0x7fb64c055e90 con 0x7fb65c09d560 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb64affd640 10 client.31095.objecter ms_dispatch 0x7fb65c000b90 osd_map(80..80 src has 1..80) 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb64affd640 3 client.31095.objecter handle_osd_map got epochs [80,80] > 0 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb64affd640 3 client.31095.objecter handle_osd_map decoding full epoch 80 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.403+0000 7fb64affd640 20 client.31095.objecter dump_active .. 0 homeless 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb65ad9d640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb63c03fdb0 0x7fb63c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb65ad9d640 10 monclient: get_auth_request con 0x7fb63c03fdb0 auth_method 0 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb659d9b640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12e530 0x7fb65c12e910 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0a1af0 tx=0x7fb65000e800 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 10 client.31095.objecter ms_handle_connect 0x7fb65c12e530 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 10 client.31095.objecter resend_mon_ops 2024-06-11T19:11:36.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb6500052e0 con 0x7fb65c12e530 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 10 monclient: dump: 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.426 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 10 client.31095.objecter ms_handle_connect 0x7fb63c03fdb0 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb65ad9d640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb63c03fdb0 0x7fb63c042270 secure :-1 s=READY pgs=3994 cs=0 l=1 rev1=1 crypto rx=0x7fb654035000 tx=0x7fb654030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb6500052e0 con 0x7fb65c12e530 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12e530 msgr2=0x7fb65c12e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12e530 0x7fb65c12e910 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0a1af0 tx=0x7fb65000e800 comp rx=0 tx=0).stop 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12e530 msgr2=0x7fb65c12e910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:36.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.404+0000 7fb64affd640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12e530 0x7fb65c12e910 secure :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0a1af0 tx=0x7fb65000e800 comp rx=0 tx=0).stop 2024-06-11T19:11:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.410+0000 7fb660880640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.410+0000 7fb660880640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12ee50 0x7fb65c072480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.410+0000 7fb660880640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.410+0000 7fb660880640 1 -- 172.21.15.98:0/3264079469 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fb65c130750 con 0x7fb65c12ee50 2024-06-11T19:11:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb659d9b640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12ee50 0x7fb65c072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb659d9b640 10 monclient: get_auth_request con 0x7fb65c12ee50 auth_method 0 2024-06-11T19:11:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb659d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb659d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb659d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb659d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb659d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb659d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb659d9b640 10 monclient(hunting): handle_auth_done global_id 22091 payload 824 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb659d9b640 1 -- 172.21.15.98:0/3264079469 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb65c0a0bf0 con 0x7fb65c12ee50 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb659d9b640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12ee50 0x7fb65c072480 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fb650001000 tx=0x7fb650000f20 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 10 client.31095.objecter ms_handle_connect 0x7fb65c12ee50 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 10 client.31095.objecter resend_mon_ops 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb650003ab0 con 0x7fb65c12ee50 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:11:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 10 monclient: dump: 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.432 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7fb650003640 con 0x7fb65c12ee50 2024-06-11T19:11:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12ee50 msgr2=0x7fb65c072480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12ee50 0x7fb65c072480 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fb650001000 tx=0x7fb650000f20 comp rx=0 tx=0).stop 2024-06-11T19:11:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 1 -- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12ee50 msgr2=0x7fb65c072480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.411+0000 7fb64affd640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb65c12ee50 0x7fb65c072480 secure :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fb650001000 tx=0x7fb650000f20 comp rx=0 tx=0).stop 2024-06-11T19:11:36.432 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:36.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 146606, 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.433 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:36.434 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:36.435 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:36.436 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:36.437 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:36.439 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:36.446 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.449 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.454 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:11:36.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:11:36.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.457 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:36.457 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 10 librados: watch_flush enter 2024-06-11T19:11:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 10 librados: watch_flush exit 2024-06-11T19:11:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 20 client.31095.objecter shutdown clearing up homeless session... 2024-06-11T19:11:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 10 client.31095.objecter successfully canceled tick 2024-06-11T19:11:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 1 -- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb63c03fdb0 msgr2=0x7fb63c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb63c03fdb0 0x7fb63c042270 secure :-1 s=READY pgs=3994 cs=0 l=1 rev1=1 crypto rx=0x7fb654035000 tx=0x7fb654030000 comp rx=0 tx=0).stop 2024-06-11T19:11:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 10 monclient: shutdown 2024-06-11T19:11:36.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 1 -- 172.21.15.98:0/3264079469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb65c09d560 msgr2=0x7fb65c1c9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 1 --2- 172.21.15.98:0/3264079469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb65c09d560 0x7fb65c1c9970 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fb64c009ca0 tx=0x7fb64c0134a0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb65ad9d640 1 -- 172.21.15.98:0/3264079469 reap_dead start 2024-06-11T19:11:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb65ad9d640 1 -- 172.21.15.98:0/3264079469 reap_dead start 2024-06-11T19:11:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 1 -- 172.21.15.98:0/3264079469 shutdown_connections 2024-06-11T19:11:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 1 -- 172.21.15.98:0/3264079469 >> 172.21.15.98:0/3264079469 conn(0x7fb65c0a7850 msgr2=0x7fb65c09e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 1 -- 172.21.15.98:0/3264079469 shutdown_connections 2024-06-11T19:11:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 1 -- 172.21.15.98:0/3264079469 wait complete. 2024-06-11T19:11:36.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.412+0000 7fb660880640 1 librados: shutdown 2024-06-11T19:11:36.460 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:36.461 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:11:36.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:11:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.561+0000 7f71cd426640 10 monclient: get_monmap_and_config 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.561+0000 7f71cd426640 10 monclient: build_initial_monmap 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.561+0000 7f71cd426640 10 monclient: monmap: 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:36.562220+0000 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:36.562220+0000 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:36.561 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:36.562 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:36.562 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:36.562 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:36.562 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:36.562 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:36.562 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:36.562 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.561+0000 7f71cd426640 1 Processor -- start 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.561+0000 7f71cd426640 1 -- start start 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.561+0000 7f71cd426640 10 monclient: init 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 10 monclient: _add_conns ranks=[3,6,5,10,2,4,0,8,7,1,9] 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71c80a2f40 0x7f71c80a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 10 monclient(hunting): picked mon.noname-g con 0x7f71c80a2f40 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f71c80a4df0 0x7f71c80a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 10 monclient(hunting): picked mon.noname-d con 0x7f71c80a4df0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f71c81366d0 0x7f71c8136af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 10 monclient(hunting): picked mon.noname-h con 0x7f71c81366d0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f71c80a5710 con 0x7f71c80a2f40 2024-06-11T19:11:36.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f71c80a3320 con 0x7f71c81366d0 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f71c80a2460 con 0x7f71c80a4df0 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71cd426640 10 monclient(hunting): authenticate will time out at 906.369399s 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f71c80a4df0 0x7f71c80a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f71c80a4df0 0x7f71c80a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39854/0 (socket says 172.21.15.98:39854) 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 1 -- 172.21.15.98:0/4171358690 learned_addr learned my addr 172.21.15.98:0/4171358690 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 10 monclient(hunting): get_auth_request con 0x7f71c80a4df0 auth_method 0 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c6ffd640 1 --2- 172.21.15.98:0/4171358690 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71c80a2f40 0x7f71c80a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 10 monclient(hunting): handle_auth_done global_id 54267 payload 274 2024-06-11T19:11:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 1 -- 172.21.15.98:0/4171358690 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f71c81366d0 msgr2=0x7f71c8136af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c77fe640 1 --2- 172.21.15.98:0/4171358690 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f71c81366d0 0x7f71c8136af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 1 --2- 172.21.15.98:0/4171358690 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f71c81366d0 0x7f71c8136af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 1 -- 172.21.15.98:0/4171358690 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71c80a2f40 msgr2=0x7f71c80a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 1 --2- 172.21.15.98:0/4171358690 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71c80a2f40 0x7f71c80a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 10 monclient: _finish_hunting 0 2024-06-11T19:11:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 1 monclient: found mon.noname-d 2024-06-11T19:11:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:36.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c67fc640 1 -- 172.21.15.98:0/4171358690 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c80a0ed0 con 0x7f71c80a4df0 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c77fe640 10 monclient: get_auth_request con 0x7f71c81366d0 auth_method 0 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c6ffd640 10 monclient: get_auth_request con 0x7f71c80a2f40 auth_method 0 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c77fe640 1 --2- 172.21.15.98:0/4171358690 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f71c81366d0 0x7f71c8136af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.562+0000 7f71c6ffd640 1 --2- 172.21.15.98:0/4171358690 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71c80a2f40 0x7f71c80a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c67fc640 1 --2- 172.21.15.98:0/4171358690 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f71c80a4df0 0x7f71c80a51d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f71bc0098d0 tx=0x7f71bc02b790 comp rx=0 tx=0).ready entity=mon.6 client_cookie=934048e1f5e319ea server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 1 -- 172.21.15.98:0/4171358690 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71bc039070 con 0x7f71c80a4df0 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 10 monclient: dump: 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.575 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 10 monclient: _finish_auth 0 2024-06-11T19:11:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 1 -- 172.21.15.98:0/4171358690 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71bc031960 con 0x7f71c80a4df0 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71cd426640 5 monclient: authenticate success, global_id 54267 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71cd426640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71cd426640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 1 -- 172.21.15.98:0/4171358690 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71bc032380 con 0x7f71c80a4df0 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71cd426640 4 set_mon_vals no callback set 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.563+0000 7f71c5ffb640 10 monclient: dump: 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.579 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 10 monclient: shutdown 2024-06-11T19:11:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 1 -- 172.21.15.98:0/4171358690 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f71c80a4df0 msgr2=0x7f71c80a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 1 --2- 172.21.15.98:0/4171358690 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f71c80a4df0 0x7f71c80a51d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f71bc0098d0 tx=0x7f71bc02b790 comp rx=0 tx=0).stop 2024-06-11T19:11:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 1 -- 172.21.15.98:0/4171358690 shutdown_connections 2024-06-11T19:11:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 1 --2- 172.21.15.98:0/4171358690 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f71c81366d0 0x7f71c8136af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 1 --2- 172.21.15.98:0/4171358690 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f71c80a4df0 0x7f71c80a51d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 1 --2- 172.21.15.98:0/4171358690 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71c80a2f40 0x7f71c80a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 1 -- 172.21.15.98:0/4171358690 >> 172.21.15.98:0/4171358690 conn(0x7f71c80a95b0 msgr2=0x7f71c80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 1 -- 172.21.15.98:0/4171358690 shutdown_connections 2024-06-11T19:11:36.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 1 -- 172.21.15.98:0/4171358690 wait complete. 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 10 monclient: build_initial_monmap 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.564+0000 7f71cd426640 10 monclient: monmap: 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 1 librados: starting msgr at 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 1 librados: starting objecter 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 1 Processor -- start 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 1 -- start start 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 1 librados: setting wanted keys 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 1 librados: calling monclient init 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 10 monclient: init 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 10 monclient: _add_conns ranks=[5,4,7,6,0,2,8,3,1] 2024-06-11T19:11:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f71c80a2f40 0x7f71c81cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 10 monclient(hunting): picked mon.noname-h con 0x7f71c80a2f40 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71c80a4df0 0x7f71c81cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 10 monclient(hunting): picked mon.noname-c con 0x7f71c80a4df0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c81366d0 0x7f71c8122190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 10 monclient(hunting): picked mon.noname-i con 0x7f71c81366d0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f71c80a1250 con 0x7f71c80a4df0 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f71c8137030 con 0x7f71c80a2f40 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f71c8136db0 con 0x7f71c81366d0 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71cd426640 10 monclient(hunting): authenticate will time out at 906.372817s 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71c67fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71c80a4df0 0x7f71c81cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71c67fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71c80a4df0 0x7f71c81cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:50022/0 (socket says 172.21.15.98:50022) 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71c67fc640 1 -- 172.21.15.98:0/1988742604 learned_addr learned my addr 172.21.15.98:0/1988742604 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71c67fc640 10 monclient(hunting): get_auth_request con 0x7f71c80a4df0 auth_method 0 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71c67fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.565+0000 7f71c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c6ffd640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f71c80a2f40 0x7f71c81cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c6ffd640 10 monclient(hunting): get_auth_request con 0x7f71c80a2f40 auth_method 0 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c77fe640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c81366d0 0x7f71c8122190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c77fe640 10 monclient(hunting): get_auth_request con 0x7f71c81366d0 auth_method 0 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 10 monclient(hunting): handle_auth_done global_id 22927 payload 824 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 1 -- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c81366d0 msgr2=0x7f71c8122190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c81366d0 0x7f71c8122190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 1 -- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f71c80a2f40 msgr2=0x7f71c81cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f71c80a2f40 0x7f71c81cb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 10 monclient: _finish_hunting 0 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 1 monclient: found mon.noname-c 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 1 -- 172.21.15.98:0/1988742604 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c8126ae0 con 0x7f71c80a4df0 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c67fc640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71c80a4df0 0x7f71c81cbb80 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f71bc0097c0 tx=0x7f71bc0046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 10 client.?.objecter ms_handle_connect 0x7f71c80a4df0 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71bc043020 con 0x7f71c80a4df0 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 10 monclient: dump: 2024-06-11T19:11:36.585 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 10 monclient: _finish_auth 0 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71cd426640 5 monclient: authenticate success, global_id 22927 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71bc0044b0 con 0x7f71c80a4df0 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71cd426640 10 monclient: _renew_subs 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71cd426640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71cd426640 1 -- 172.21.15.98:0/1988742604 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71c8125e20 con 0x7f71c80a4df0 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71bc039040 con 0x7f71c80a4df0 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71cd426640 10 client.22927.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71af7fe640 10 monclient: dump: 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71cd426640 10 monclient: _renew_subs 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71cd426640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71cd426640 1 -- 172.21.15.98:0/1988742604 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c8126100 con 0x7f71c80a4df0 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71cd426640 1 librados: init done 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.566+0000 7f71c57fa640 4 set_mon_vals no callback set 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71cd426640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71cd426640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c81226d0 0x7f71c8122ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71cd426640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71cd426640 1 -- 172.21.15.98:0/1988742604 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f71c80a13d0 con 0x7f71c81226d0 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f71bc046020 con 0x7f71c80a4df0 2024-06-11T19:11:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71c6ffd640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c81226d0 0x7f71c8122ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71c6ffd640 10 monclient: get_auth_request con 0x7f71c81226d0 auth_method 0 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71c6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71af7fe640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f71b403fde0 0x7f71b40422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(80..80 src has 1..80) ==== 7956+0+0 (secure 0 0 0) 0x7f71bc073b30 con 0x7f71c80a4df0 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71af7fe640 10 client.22927.objecter ms_dispatch 0x7f71c8000b90 osd_map(80..80 src has 1..80) 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71af7fe640 3 client.22927.objecter handle_osd_map got epochs [80,80] > 0 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71af7fe640 3 client.22927.objecter handle_osd_map decoding full epoch 80 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71c77fe640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f71b403fde0 0x7f71b40422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71af7fe640 20 client.22927.objecter dump_active .. 0 homeless 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71c77fe640 10 monclient: get_auth_request con 0x7f71b403fde0 auth_method 0 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71c6ffd640 10 monclient(hunting): handle_auth_done global_id 21994 payload 824 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.567+0000 7f71c6ffd640 1 -- 172.21.15.98:0/1988742604 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71c81358e0 con 0x7f71c81226d0 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 10 client.22927.objecter ms_handle_connect 0x7f71c81226d0 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71c6ffd640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c81226d0 0x7f71c8122ab0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f71b800c450 tx=0x7f71b800c910 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 10 client.22927.objecter resend_mon_ops 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 10 client.22927.objecter ms_handle_connect 0x7f71b403fde0 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71c77fe640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f71b403fde0 0x7f71b40422a0 secure :-1 s=READY pgs=3995 cs=0 l=1 rev1=1 crypto rx=0x7f71c80a69c0 tx=0x7f71c0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71b800bbb0 con 0x7f71c81226d0 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 10 monclient: dump: 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f71b800b740 con 0x7f71c81226d0 2024-06-11T19:11:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c81226d0 msgr2=0x7f71c8122ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c81226d0 0x7f71c8122ab0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f71b800c450 tx=0x7f71b800c910 comp rx=0 tx=0).stop 2024-06-11T19:11:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c81226d0 msgr2=0x7f71c8122ab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.568+0000 7f71af7fe640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c81226d0 0x7f71c8122ab0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.574+0000 7f71cd426640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.574+0000 7f71cd426640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c8122ff0 0x7f71c809d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.574+0000 7f71cd426640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.574+0000 7f71cd426640 1 -- 172.21.15.98:0/1988742604 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f71c8050830 con 0x7f71c8122ff0 2024-06-11T19:11:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71c6ffd640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c8122ff0 0x7f71c809d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71c6ffd640 10 monclient: get_auth_request con 0x7f71c8122ff0 auth_method 0 2024-06-11T19:11:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71c6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71c6ffd640 10 monclient(hunting): handle_auth_done global_id 22003 payload 824 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71c6ffd640 1 -- 172.21.15.98:0/1988742604 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f71c81373c0 con 0x7f71c8122ff0 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71c6ffd640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c8122ff0 0x7f71c809d530 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f71b800c450 tx=0x7f71b8004a40 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71af7fe640 10 client.22927.objecter ms_handle_connect 0x7f71c8122ff0 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71af7fe640 10 client.22927.objecter resend_mon_ops 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71b800a8b0 con 0x7f71c8122ff0 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71af7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.575+0000 7f71af7fe640 10 monclient: dump: 2024-06-11T19:11:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.597 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.598 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.598 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f71b800a440 con 0x7f71c8122ff0 2024-06-11T19:11:36.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71af7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:36.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71af7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:36.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c8122ff0 msgr2=0x7f71c809d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71af7fe640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c8122ff0 0x7f71c809d530 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f71b800c450 tx=0x7f71b8004a40 comp rx=0 tx=0).stop 2024-06-11T19:11:36.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71af7fe640 1 -- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c8122ff0 msgr2=0x7f71c809d530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:36.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71af7fe640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f71c8122ff0 0x7f71c809d530 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.598 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 12, 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 146793, 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:11:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:11:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 9, 2024-06-11T19:11:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:36.621 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:36.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 10 librados: watch_flush enter 2024-06-11T19:11:36.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 10 librados: watch_flush exit 2024-06-11T19:11:36.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 20 client.22927.objecter shutdown clearing up homeless session... 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 10 client.22927.objecter successfully canceled tick 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 1 -- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f71b403fde0 msgr2=0x7f71b40422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f71b403fde0 0x7f71b40422a0 secure :-1 s=READY pgs=3995 cs=0 l=1 rev1=1 crypto rx=0x7f71c80a69c0 tx=0x7f71c0008040 comp rx=0 tx=0).stop 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 10 monclient: shutdown 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 1 -- 172.21.15.98:0/1988742604 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71c80a4df0 msgr2=0x7f71c81cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71c80a4df0 0x7f71c81cbb80 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f71bc0097c0 tx=0x7f71bc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71c77fe640 1 -- 172.21.15.98:0/1988742604 reap_dead start 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 1 -- 172.21.15.98:0/1988742604 shutdown_connections 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 1 --2- 172.21.15.98:0/1988742604 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71c80a4df0 0x7f71c81cbb80 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 1 -- 172.21.15.98:0/1988742604 >> 172.21.15.98:0/1988742604 conn(0x7f71c80a95b0 msgr2=0x7f71c80a7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 1 -- 172.21.15.98:0/1988742604 shutdown_connections 2024-06-11T19:11:36.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 1 -- 172.21.15.98:0/1988742604 wait complete. 2024-06-11T19:11:36.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.576+0000 7f71cd426640 1 librados: shutdown 2024-06-11T19:11:36.625 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:11:36.625 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['e', 'a'] 2024-06-11T19:11:36.625 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-06-11T19:11:36.625 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-06-11T19:11:36.625 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-06-11T19:11:36.626 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-06-11T19:11:36.626 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:11:36.637 INFO:tasks.ceph.mon.e:Stopped 2024-06-11T19:11:36.637 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-06-11T19:11:36.637 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-06-11T19:11:36.637 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-06-11T19:11:36.637 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:11:36.690 INFO:tasks.ceph.mon.a:Stopped 2024-06-11T19:11:36.690 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 7 2024-06-11T19:11:36.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:11:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.827+0000 7fd77c05c640 10 monclient: get_monmap_and_config 2024-06-11T19:11:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.827+0000 7fd77c05c640 10 monclient: build_initial_monmap 2024-06-11T19:11:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.827+0000 7fd77c05c640 10 monclient: monmap: 2024-06-11T19:11:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:36.828806+0000 2024-06-11T19:11:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:36.828806+0000 2024-06-11T19:11:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:36.829 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.828+0000 7fd77c05c640 1 Processor -- start 2024-06-11T19:11:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.828+0000 7fd77c05c640 1 -- start start 2024-06-11T19:11:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.828+0000 7fd77c05c640 10 monclient: init 2024-06-11T19:11:36.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:36.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 10 monclient: _add_conns ranks=[3,9,6,2,5,8,10,7,4,0,1] 2024-06-11T19:11:36.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7740a4920 0x7fd7740a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 10 monclient(hunting): picked mon.noname-g con 0x7fd7740a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:36.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd7740a5240 0x7fd7740a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 10 monclient(hunting): picked mon.noname-a con 0x7fd7740a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:11:36.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a2530 0x7fd7740a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 10 monclient(hunting): picked mon.noname-d con 0x7fd7740a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:36.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd7740a7140 con 0x7fd7740a5240 2024-06-11T19:11:36.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd7740a2b50 con 0x7fd7740a4920 2024-06-11T19:11:36.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd7740a2e30 con 0x7fd7740a2530 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77c05c640 10 monclient(hunting): authenticate will time out at 906.636129s 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd7795f7640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7fd7740a5240 msgr2=0x7fd7740a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd7795f7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd7740a5240 0x7fd7740a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a2530 0x7fd7740a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a2530 0x7fd7740a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39856/0 (socket says 172.21.15.98:39856) 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 1 -- 172.21.15.98:0/794364257 learned_addr learned my addr 172.21.15.98:0/794364257 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 10 monclient(hunting): get_auth_request con 0x7fd7740a2530 auth_method 0 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd779df8640 1 --2- 172.21.15.98:0/794364257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7740a4920 0x7fd7740a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd779df8640 10 monclient(hunting): get_auth_request con 0x7fd7740a4920 auth_method 0 2024-06-11T19:11:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd779df8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd779df8640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd779df8640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 10 monclient(hunting): handle_auth_done global_id 54276 payload 274 2024-06-11T19:11:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 1 -- 172.21.15.98:0/794364257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7740a4920 msgr2=0x7fd7740a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 1 --2- 172.21.15.98:0/794364257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7740a4920 0x7fd7740a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 1 -- 172.21.15.98:0/794364257 >> v2:172.21.15.98:3300/0 conn(0x7fd7740a5240 msgr2=0x7fd7740a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 1 --2- 172.21.15.98:0/794364257 >> v2:172.21.15.98:3300/0 conn(0x7fd7740a5240 0x7fd7740a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 10 monclient: _finish_hunting 0 2024-06-11T19:11:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 1 monclient: found mon.noname-d 2024-06-11T19:11:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.829+0000 7fd77a5f9640 1 -- 172.21.15.98:0/794364257 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7740a3110 con 0x7fd7740a2530 2024-06-11T19:11:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77a5f9640 1 --2- 172.21.15.98:0/794364257 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a2530 0x7fd7740a5ee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd764009870 tx=0x7fd76402b7e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=193d4784a75016fd server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 1 -- 172.21.15.98:0/794364257 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd764038070 con 0x7fd7740a2530 2024-06-11T19:11:36.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:36.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 10 monclient: dump: 2024-06-11T19:11:36.847 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.847 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.847 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.847 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.847 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.847 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.848 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.848 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.848 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.848 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.848 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.848 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.848 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.849 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 10 monclient: _finish_auth 0 2024-06-11T19:11:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 10 monclient: _check_auth_tickets 2024-06-11T19:11:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 1 -- 172.21.15.98:0/794364257 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7640319b0 con 0x7fd7740a2530 2024-06-11T19:11:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 5 monclient: authenticate success, global_id 54276 2024-06-11T19:11:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 1 -- 172.21.15.98:0/794364257 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd76402f050 con 0x7fd7740a2530 2024-06-11T19:11:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 4 set_mon_vals no callback set 2024-06-11T19:11:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd778df6640 10 monclient: dump: 2024-06-11T19:11:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.852 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 10 monclient: shutdown 2024-06-11T19:11:36.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:36.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 1 -- 172.21.15.98:0/794364257 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a2530 msgr2=0x7fd7740a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 1 --2- 172.21.15.98:0/794364257 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a2530 0x7fd7740a5ee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd764009870 tx=0x7fd76402b7e0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 1 -- 172.21.15.98:0/794364257 shutdown_connections 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 1 --2- 172.21.15.98:0/794364257 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a2530 0x7fd7740a5ee0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 1 --2- 172.21.15.98:0/794364257 >> v2:172.21.15.98:3300/0 conn(0x7fd7740a5240 0x7fd7740a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 1 --2- 172.21.15.98:0/794364257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7740a4920 0x7fd7740a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.830+0000 7fd77c05c640 1 -- 172.21.15.98:0/794364257 >> 172.21.15.98:0/794364257 conn(0x7fd7740ab850 msgr2=0x7fd7740a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.831+0000 7fd77c05c640 1 -- 172.21.15.98:0/794364257 shutdown_connections 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.831+0000 7fd77c05c640 1 -- 172.21.15.98:0/794364257 wait complete. 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.831+0000 7fd77c05c640 10 monclient: build_initial_monmap 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.831+0000 7fd77c05c640 10 monclient: monmap: 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.831+0000 7fd77c05c640 1 librados: starting msgr at 2024-06-11T19:11:36.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.831+0000 7fd77c05c640 1 librados: starting objecter 2024-06-11T19:11:36.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.831+0000 7fd77c05c640 1 Processor -- start 2024-06-11T19:11:36.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.831+0000 7fd77c05c640 1 -- start start 2024-06-11T19:11:36.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.831+0000 7fd77c05c640 1 librados: setting wanted keys 2024-06-11T19:11:36.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.831+0000 7fd77c05c640 1 librados: calling monclient init 2024-06-11T19:11:36.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.831+0000 7fd77c05c640 10 monclient: init 2024-06-11T19:11:36.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:36.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 10 monclient: _add_conns ranks=[4,5,6,7,3,1,8,2,0] 2024-06-11T19:11:36.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7740a2530 0x7fd7741cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 10 monclient(hunting): picked mon.noname-c con 0x7fd7740a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:36.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd7740a4920 0x7fd7741ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 10 monclient(hunting): picked mon.noname-h con 0x7fd7740a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:11:36.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a5240 0x7fd774122e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 10 monclient(hunting): picked mon.noname-d con 0x7fd7740a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:36.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd7740a6da0 con 0x7fd7740a2530 2024-06-11T19:11:36.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd774138e50 con 0x7fd7740a4920 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd7740a6420 con 0x7fd7740a5240 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77c05c640 10 monclient(hunting): authenticate will time out at 906.639388s 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd779df8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7740a2530 0x7fd7741cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a5240 0x7fd774122e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd779df8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7740a2530 0x7fd7741cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:50024/0 (socket says 172.21.15.98:50024) 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a5240 0x7fd774122e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39864/0 (socket says 172.21.15.98:39864) 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd779df8640 1 -- 172.21.15.98:0/83847724 learned_addr learned my addr 172.21.15.98:0/83847724 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd779df8640 10 monclient(hunting): get_auth_request con 0x7fd7740a2530 auth_method 0 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd779df8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd779df8640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd779df8640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 10 monclient(hunting): get_auth_request con 0x7fd7740a5240 auth_method 0 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd7795f7640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd7740a4920 0x7fd7741ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd7795f7640 10 monclient(hunting): get_auth_request con 0x7fd7740a4920 auth_method 0 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd7795f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd7795f7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd7795f7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd779df8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd779df8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd779df8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd7795f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd7795f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd7795f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 10 monclient(hunting): handle_auth_done global_id 54285 payload 824 2024-06-11T19:11:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 1 -- 172.21.15.98:0/83847724 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd7740a4920 msgr2=0x7fd7741ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd7740a4920 0x7fd7741ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 1 -- 172.21.15.98:0/83847724 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7740a2530 msgr2=0x7fd7741cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7740a2530 0x7fd7741cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 10 monclient: _finish_hunting 0 2024-06-11T19:11:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 1 monclient: found mon.noname-d 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd77a5f9640 1 -- 172.21.15.98:0/83847724 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd774126d80 con 0x7fd7740a5240 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.832+0000 7fd779df8640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7740a2530 0x7fd7741cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd77a5f9640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a5240 0x7fd774122e40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd764033000 tx=0x7fd7640046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 10 client.?.objecter ms_handle_connect 0x7fd7740a5240 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 1 -- 172.21.15.98:0/83847724 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd764042020 con 0x7fd7740a5240 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 10 monclient: dump: 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.864 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 1 -- 172.21.15.98:0/83847724 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd764004470 con 0x7fd7740a5240 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd77c05c640 5 monclient: authenticate success, global_id 54285 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 1 -- 172.21.15.98:0/83847724 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd764038040 con 0x7fd7740a5240 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd77c05c640 10 monclient: _renew_subs 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd77c05c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd77c05c640 1 -- 172.21.15.98:0/83847724 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7740aac80 con 0x7fd7740a5240 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76a7fc640 10 monclient: dump: 2024-06-11T19:11:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd76bfff640 4 set_mon_vals no callback set 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd77c05c640 10 client.54285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd77c05c640 10 monclient: _renew_subs 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd77c05c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd77c05c640 1 -- 172.21.15.98:0/83847724 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7741267c0 con 0x7fd7740a5240 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.833+0000 7fd77c05c640 1 librados: init done 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd76a7fc640 1 -- 172.21.15.98:0/83847724 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd764045020 con 0x7fd7740a5240 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd77c05c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd77c05c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd77c05c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd77c05c640 1 -- 172.21.15.98:0/83847724 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7740a6750 con 0x7fd7740a5240 2024-06-11T19:11:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd76a7fc640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd76003fdb0 0x7fd760042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd76a7fc640 1 -- 172.21.15.98:0/83847724 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(80..80 src has 1..80) ==== 7956+0+0 (secure 0 0 0) 0x7fd764073ad0 con 0x7fd7740a5240 2024-06-11T19:11:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd76a7fc640 10 client.54285.objecter ms_dispatch 0x7fd774000b90 osd_map(80..80 src has 1..80) 2024-06-11T19:11:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd76a7fc640 3 client.54285.objecter handle_osd_map got epochs [80,80] > 0 2024-06-11T19:11:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd76a7fc640 3 client.54285.objecter handle_osd_map decoding full epoch 80 2024-06-11T19:11:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd76a7fc640 20 client.54285.objecter dump_active .. 0 homeless 2024-06-11T19:11:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd779df8640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd76003fdb0 0x7fd760042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.834+0000 7fd779df8640 10 monclient: get_auth_request con 0x7fd76003fdb0 auth_method 0 2024-06-11T19:11:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.835+0000 7fd76a7fc640 10 client.54285.objecter ms_handle_connect 0x7fd76003fdb0 2024-06-11T19:11:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.835+0000 7fd779df8640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd76003fdb0 0x7fd760042270 secure :-1 s=READY pgs=3996 cs=0 l=1 rev1=1 crypto rx=0x7fd76c003fc0 tx=0x7fd76c009970 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.838+0000 7fd76a7fc640 1 -- 172.21.15.98:0/83847724 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd764032de0 con 0x7fd7740a5240 2024-06-11T19:11:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.838+0000 7fd76a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:36.838+0000 7fd76a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.091+0000 7fd77c05c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:11:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.092+0000 7fd77c05c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:11:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.092+0000 7fd77c05c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.092+0000 7fd77c05c640 1 -- 172.21.15.98:0/83847724 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd774125300 con 0x7fd7740a5240 2024-06-11T19:11:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.092+0000 7fd76a7fc640 1 -- 172.21.15.98:0/83847724 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fd764003610 con 0x7fd7740a5240 2024-06-11T19:11:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.092+0000 7fd76a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:11:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.092+0000 7fd76a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:37.092 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":12,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:11:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 10 librados: watch_flush enter 2024-06-11T19:11:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 10 librados: watch_flush exit 2024-06-11T19:11:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 20 client.54285.objecter shutdown clearing up homeless session... 2024-06-11T19:11:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 10 client.54285.objecter successfully canceled tick 2024-06-11T19:11:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 -- 172.21.15.98:0/83847724 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd76003fdb0 msgr2=0x7fd760042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd76003fdb0 0x7fd760042270 secure :-1 s=READY pgs=3996 cs=0 l=1 rev1=1 crypto rx=0x7fd76c003fc0 tx=0x7fd76c009970 comp rx=0 tx=0).stop 2024-06-11T19:11:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 10 monclient: shutdown 2024-06-11T19:11:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 -- 172.21.15.98:0/83847724 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a5240 msgr2=0x7fd774122e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a5240 0x7fd774122e40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd764033000 tx=0x7fd7640046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 -- 172.21.15.98:0/83847724 shutdown_connections 2024-06-11T19:11:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd76003fdb0 0x7fd760042270 unknown :-1 s=CLOSED pgs=3996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7740a5240 0x7fd774122e40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd7740a4920 0x7fd7741ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 --2- 172.21.15.98:0/83847724 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7740a2530 0x7fd7741cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 -- 172.21.15.98:0/83847724 >> 172.21.15.98:0/83847724 conn(0x7fd7740ab850 msgr2=0x7fd7741bf010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 -- 172.21.15.98:0/83847724 shutdown_connections 2024-06-11T19:11:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 -- 172.21.15.98:0/83847724 wait complete. 2024-06-11T19:11:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:37.094+0000 7fd77c05c640 1 librados: shutdown 2024-06-11T19:11:40.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.213+0000 7f9c9c79c640 10 monclient: get_monmap_and_config 2024-06-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.213+0000 7f9c9c79c640 10 monclient: build_initial_monmap 2024-06-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.213+0000 7f9c9c79c640 10 monclient: monmap: 2024-06-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:40.213 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:40.214348+0000 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:40.214348+0000 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.213+0000 7f9c9c79c640 1 Processor -- start 2024-06-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.213+0000 7f9c9c79c640 1 -- start start 2024-06-11T19:11:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.213+0000 7f9c9c79c640 10 monclient: init 2024-06-11T19:11:40.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:40.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 10 monclient: _add_conns ranks=[1,4,7,0,2,9,5,6,8,3,10] 2024-06-11T19:11:40.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f9c940a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:11:40.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a2530 0x7f9c9413dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:40.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 10 monclient(hunting): picked mon.noname-c con 0x7f9c940a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:40.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9413e1a0 0x7f9c94140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 10 monclient(hunting): picked mon.noname-i con 0x7f9c9413e1a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c940a7140 con 0x7f9c940a4920 2024-06-11T19:11:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f9c940a2b50 con 0x7f9c940a2530 2024-06-11T19:11:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f9c940a2e30 con 0x7f9c9413e1a0 2024-06-11T19:11:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9c79c640 10 monclient(hunting): authenticate will time out at 910.021635s 2024-06-11T19:11:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c99d37640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a2530 0x7f9c9413dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c99d37640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a2530 0x7f9c9413dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:50026/0 (socket says 172.21.15.98:50026) 2024-06-11T19:11:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c99d37640 1 -- 172.21.15.98:0/264937384 learned_addr learned my addr 172.21.15.98:0/264937384 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:40.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c99d37640 10 monclient(hunting): get_auth_request con 0x7f9c940a2530 auth_method 0 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c99d37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c99d37640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c99d37640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c99d37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c99d37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c99d37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9ad39640 1 --2- 172.21.15.98:0/264937384 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9413e1a0 0x7f9c94140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9ad39640 10 monclient(hunting): get_auth_request con 0x7f9c9413e1a0 auth_method 0 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9ad39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9ad39640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c9ad39640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.214+0000 7f9c99d37640 10 monclient(hunting): handle_auth_done global_id 22945 payload 274 2024-06-11T19:11:40.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99d37640 1 -- 172.21.15.98:0/264937384 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9413e1a0 msgr2=0x7f9c94140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99d37640 1 --2- 172.21.15.98:0/264937384 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9413e1a0 0x7f9c94140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99d37640 1 -- 172.21.15.98:0/264937384 >> v1:172.21.15.117:6789/0 conn(0x7f9c940a4920 legacy=0x7f9c940a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99d37640 10 monclient: _finish_hunting 0 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99d37640 1 monclient: found mon.noname-c 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99d37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99d37640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99d37640 1 -- 172.21.15.98:0/264937384 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c940a3110 con 0x7f9c940a2530 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99d37640 1 --2- 172.21.15.98:0/264937384 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a2530 0x7f9c9413dc60 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f9c840098a0 tx=0x7f9c8402b840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=5f47bd0040754bfa server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 1 -- 172.21.15.98:0/264937384 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c84031a10 con 0x7f9c940a2530 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 10 monclient: dump: 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:40.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 10 monclient: _finish_auth 0 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 10 monclient: _check_auth_tickets 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 1 -- 172.21.15.98:0/264937384 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c84037070 con 0x7f9c940a2530 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c9c79c640 5 monclient: authenticate success, global_id 22945 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c9c79c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c9c79c640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 1 -- 172.21.15.98:0/264937384 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c84031c10 con 0x7f9c940a2530 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c9c79c640 4 set_mon_vals no callback set 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.215+0000 7f9c99536640 10 monclient: dump: 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:40.230 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.216+0000 7f9c9c79c640 10 monclient: shutdown 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.216+0000 7f9c9c79c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.216+0000 7f9c9c79c640 1 -- 172.21.15.98:0/264937384 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a2530 msgr2=0x7f9c9413dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.216+0000 7f9c9c79c640 1 --2- 172.21.15.98:0/264937384 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a2530 0x7f9c9413dc60 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f9c840098a0 tx=0x7f9c8402b840 comp rx=0 tx=0).stop 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.216+0000 7f9c9c79c640 1 -- 172.21.15.98:0/264937384 shutdown_connections 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.216+0000 7f9c9c79c640 1 --2- 172.21.15.98:0/264937384 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9413e1a0 0x7f9c94140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.216+0000 7f9c9c79c640 1 --2- 172.21.15.98:0/264937384 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a2530 0x7f9c9413dc60 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.216+0000 7f9c9c79c640 1 -- 172.21.15.98:0/264937384 >> 172.21.15.98:0/264937384 conn(0x7f9c940ab850 msgr2=0x7f9c940a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.216+0000 7f9c9c79c640 1 -- 172.21.15.98:0/264937384 shutdown_connections 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.216+0000 7f9c9c79c640 1 -- 172.21.15.98:0/264937384 wait complete. 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient: build_initial_monmap 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient: monmap: 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 1 librados: starting msgr at 2024-06-11T19:11:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 1 librados: starting objecter 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 1 Processor -- start 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 1 -- start start 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 1 librados: setting wanted keys 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 1 librados: calling monclient init 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient: init 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient: _add_conns ranks=[2,8,7,5,3,1,4,0,6] 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 0x7f9c94136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient(hunting): picked mon.noname-b con 0x7f9c940a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a4920 0x7f9c94137200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient(hunting): picked mon.noname-e con 0x7f9c940a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9413e1a0 0x7f9c9409fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient(hunting): picked mon.noname-i con 0x7f9c9413e1a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9c94138e50 con 0x7f9c940a2530 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f9c940a84b0 con 0x7f9c9413e1a0 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9c940a6da0 con 0x7f9c940a4920 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c9c79c640 10 monclient(hunting): authenticate will time out at 910.024929s 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c99d37640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a4920 msgr2=0x7f9c94137200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.217+0000 7f9c99d37640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a4920 0x7f9c94137200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 0x7f9c94136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 0x7f9c94136cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45742/0 (socket says 172.21.15.98:45742) 2024-06-11T19:11:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 1 -- 172.21.15.98:0/1809847734 learned_addr learned my addr 172.21.15.98:0/1809847734 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 10 monclient(hunting): get_auth_request con 0x7f9c940a2530 auth_method 0 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9ad39640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9413e1a0 0x7f9c9409fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9ad39640 10 monclient(hunting): get_auth_request con 0x7f9c9413e1a0 auth_method 0 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9ad39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9ad39640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9ad39640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9ad39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9ad39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9ad39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 10 monclient(hunting): handle_auth_done global_id 21602 payload 824 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 1 -- 172.21.15.98:0/1809847734 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a4920 msgr2=0x7f9c94137200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a4920 0x7f9c94137200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 1 -- 172.21.15.98:0/1809847734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9413e1a0 msgr2=0x7f9c9409fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9413e1a0 0x7f9c9409fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 10 monclient: _finish_hunting 0 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 1 monclient: found mon.noname-b 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 1 -- 172.21.15.98:0/1809847734 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c940aa8b0 con 0x7f9c940a2530 2024-06-11T19:11:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9a538640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 0x7f9c94136cc0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000c480 tx=0x7f9c9000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 10 client.?.objecter ms_handle_connect 0x7f9c940a2530 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 1 -- 172.21.15.98:0/1809847734 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c90001180 con 0x7f9c940a2530 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 10 monclient: dump: 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 10 monclient: _finish_auth 0 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9c79c640 5 monclient: authenticate success, global_id 21602 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 1 -- 172.21.15.98:0/1809847734 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c90012630 con 0x7f9c940a2530 2024-06-11T19:11:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 1 -- 172.21.15.98:0/1809847734 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c90012c50 con 0x7f9c940a2530 2024-06-11T19:11:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9c79c640 10 monclient: _renew_subs 2024-06-11T19:11:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9c79c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9c79c640 1 -- 172.21.15.98:0/1809847734 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c9413dcd0 con 0x7f9c940a2530 2024-06-11T19:11:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:11:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c98d35640 4 set_mon_vals no callback set 2024-06-11T19:11:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c8affd640 10 monclient: dump: 2024-06-11T19:11:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:40.240 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9c79c640 10 client.21602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9c79c640 10 monclient: _renew_subs 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9c79c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9c79c640 1 -- 172.21.15.98:0/1809847734 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c94123670 con 0x7f9c940a2530 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.218+0000 7f9c9c79c640 1 librados: init done 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.219+0000 7f9c8affd640 1 -- 172.21.15.98:0/1809847734 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f9c900127e0 con 0x7f9c940a2530 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.219+0000 7f9c9c79c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.219+0000 7f9c9c79c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.219+0000 7f9c9c79c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.219+0000 7f9c9c79c640 1 -- 172.21.15.98:0/1809847734 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c940a4d00 con 0x7f9c940a2530 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.219+0000 7f9c8affd640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c7803fdb0 0x7f9c78042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.219+0000 7f9c8affd640 1 -- 172.21.15.98:0/1809847734 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(80..80 src has 1..80) ==== 7956+0+0 (secure 0 0 0) 0x7f9c90051c60 con 0x7f9c940a2530 2024-06-11T19:11:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.219+0000 7f9c8affd640 10 client.21602.objecter ms_dispatch 0x7f9c94000b90 osd_map(80..80 src has 1..80) 2024-06-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.219+0000 7f9c8affd640 3 client.21602.objecter handle_osd_map got epochs [80,80] > 0 2024-06-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.219+0000 7f9c8affd640 3 client.21602.objecter handle_osd_map decoding full epoch 80 2024-06-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.220+0000 7f9c8affd640 20 client.21602.objecter dump_active .. 0 homeless 2024-06-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.220+0000 7f9c99d37640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c7803fdb0 0x7f9c78042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.220+0000 7f9c99d37640 10 monclient: get_auth_request con 0x7f9c7803fdb0 auth_method 0 2024-06-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.220+0000 7f9c8affd640 10 client.21602.objecter ms_handle_connect 0x7f9c7803fdb0 2024-06-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.220+0000 7f9c99d37640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c7803fdb0 0x7f9c78042270 secure :-1 s=READY pgs=3997 cs=0 l=1 rev1=1 crypto rx=0x7f9c84036000 tx=0x7f9c84008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.224+0000 7f9c8affd640 1 -- 172.21.15.98:0/1809847734 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9c90003d80 con 0x7f9c940a2530 2024-06-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.224+0000 7f9c8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.224+0000 7f9c8affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:40.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.480+0000 7f9c9c79c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:11:40.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.480+0000 7f9c9c79c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:11:40.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.480+0000 7f9c9c79c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:40.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.480+0000 7f9c9c79c640 1 -- 172.21.15.98:0/1809847734 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9c94121bf0 con 0x7f9c940a2530 2024-06-11T19:11:40.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.480+0000 7f9c8affd640 1 -- 172.21.15.98:0/1809847734 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f9c90003970 con 0x7f9c940a2530 2024-06-11T19:11:40.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.480+0000 7f9c8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:11:40.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.480+0000 7f9c8affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:40.481 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":12,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:11:40.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 10 librados: watch_flush enter 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 10 librados: watch_flush exit 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 20 client.21602.objecter shutdown clearing up homeless session... 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 10 client.21602.objecter successfully canceled tick 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 -- 172.21.15.98:0/1809847734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c7803fdb0 msgr2=0x7f9c78042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c7803fdb0 0x7f9c78042270 secure :-1 s=READY pgs=3997 cs=0 l=1 rev1=1 crypto rx=0x7f9c84036000 tx=0x7f9c84008030 comp rx=0 tx=0).stop 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 10 monclient: shutdown 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 -- 172.21.15.98:0/1809847734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 msgr2=0x7f9c94136cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 0x7f9c94136cc0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000c480 tx=0x7f9c9000c940 comp rx=0 tx=0).stop 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 -- 172.21.15.98:0/1809847734 shutdown_connections 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c7803fdb0 0x7f9c78042270 unknown :-1 s=CLOSED pgs=3997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9413e1a0 0x7f9c9409fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:40.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a4920 0x7f9c94137200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:40.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 --2- 172.21.15.98:0/1809847734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 0x7f9c94136cc0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:40.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 -- 172.21.15.98:0/1809847734 >> 172.21.15.98:0/1809847734 conn(0x7f9c940ab850 msgr2=0x7f9c94137a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:40.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 -- 172.21.15.98:0/1809847734 shutdown_connections 2024-06-11T19:11:40.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 -- 172.21.15.98:0/1809847734 wait complete. 2024-06-11T19:11:40.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:40.482+0000 7f9c9c79c640 1 librados: shutdown 2024-06-11T19:11:43.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:11:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.603+0000 7f3166b0b640 10 monclient: get_monmap_and_config 2024-06-11T19:11:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.603+0000 7f3166b0b640 10 monclient: build_initial_monmap 2024-06-11T19:11:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.603+0000 7f3166b0b640 10 monclient: monmap: 2024-06-11T19:11:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:43.604583+0000 2024-06-11T19:11:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:43.604583+0000 2024-06-11T19:11:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:43.604 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:43.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 1 Processor -- start 2024-06-11T19:11:43.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 1 -- start start 2024-06-11T19:11:43.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 10 monclient: init 2024-06-11T19:11:43.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:43.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 10 monclient: _add_conns ranks=[6,5,3,9,1,10,0,4,8,7,2] 2024-06-11T19:11:43.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a4920 0x7f31600a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:43.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 10 monclient(hunting): picked mon.noname-d con 0x7f31600a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:43.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f31600a5240 0x7f31600a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:43.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 10 monclient(hunting): picked mon.noname-h con 0x7f31600a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31600a2530 0x7f31600a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 10 monclient(hunting): picked mon.noname-g con 0x7f31600a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f31600a7140 con 0x7f31600a2530 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f31600a2b50 con 0x7f31600a5240 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f31600a2e30 con 0x7f31600a4920 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.604+0000 7f3166b0b640 10 monclient(hunting): authenticate will time out at 913.411990s 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a4920 0x7f31600a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a4920 0x7f31600a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54088/0 (socket says 172.21.15.98:54088) 2024-06-11T19:11:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 1 -- 172.21.15.98:0/716656140 learned_addr learned my addr 172.21.15.98:0/716656140 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 10 monclient(hunting): get_auth_request con 0x7f31600a4920 auth_method 0 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31650a8640 1 --2- 172.21.15.98:0/716656140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31600a2530 0x7f31600a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f3157fff640 1 --2- 172.21.15.98:0/716656140 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f31600a5240 0x7f31600a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31650a8640 10 monclient(hunting): get_auth_request con 0x7f31600a2530 auth_method 0 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31650a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31650a8640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:43.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31650a8640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:43.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f3157fff640 10 monclient(hunting): get_auth_request con 0x7f31600a5240 auth_method 0 2024-06-11T19:11:43.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f3157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:43.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f3157fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:43.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f3157fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:43.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31650a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:43.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31650a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:43.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31650a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:43.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f3157fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:43.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f3157fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:43.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f3157fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:43.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 10 monclient(hunting): handle_auth_done global_id 54294 payload 274 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 1 -- 172.21.15.98:0/716656140 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f31600a5240 msgr2=0x7f31600a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 1 --2- 172.21.15.98:0/716656140 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f31600a5240 0x7f31600a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 1 -- 172.21.15.98:0/716656140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31600a2530 msgr2=0x7f31600a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 1 --2- 172.21.15.98:0/716656140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31600a2530 0x7f31600a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 10 monclient: _finish_hunting 0 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 1 monclient: found mon.noname-d 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.605+0000 7f31648a7640 1 -- 172.21.15.98:0/716656140 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31600a3110 con 0x7f31600a4920 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31648a7640 1 --2- 172.21.15.98:0/716656140 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a4920 0x7f31600a4d00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f315c0098a0 tx=0x7f315c02b7a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=44c3a281fcb6ccb6 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 1 -- 172.21.15.98:0/716656140 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f315c038070 con 0x7f31600a4920 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:43.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 10 monclient: dump: 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:43.618 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 10 monclient: _finish_auth 0 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 10 monclient: _check_auth_tickets 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 5 monclient: authenticate success, global_id 54294 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 1 -- 172.21.15.98:0/716656140 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f315c031970 con 0x7f31600a4920 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 4 set_mon_vals no callback set 2024-06-11T19:11:43.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 1 -- 172.21.15.98:0/716656140 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f315c031ba0 con 0x7f31600a4920 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f31577fe640 10 monclient: dump: 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:43.620 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 10 monclient: shutdown 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 1 -- 172.21.15.98:0/716656140 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a4920 msgr2=0x7f31600a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 1 --2- 172.21.15.98:0/716656140 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a4920 0x7f31600a4d00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f315c0098a0 tx=0x7f315c02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 1 -- 172.21.15.98:0/716656140 shutdown_connections 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 1 --2- 172.21.15.98:0/716656140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31600a2530 0x7f31600a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 1 --2- 172.21.15.98:0/716656140 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f31600a5240 0x7f31600a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 1 --2- 172.21.15.98:0/716656140 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a4920 0x7f31600a4d00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.606+0000 7f3166b0b640 1 -- 172.21.15.98:0/716656140 >> 172.21.15.98:0/716656140 conn(0x7f31600ab850 msgr2=0x7f31600a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.607+0000 7f3166b0b640 1 -- 172.21.15.98:0/716656140 shutdown_connections 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.607+0000 7f3166b0b640 1 -- 172.21.15.98:0/716656140 wait complete. 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.607+0000 7f3166b0b640 10 monclient: build_initial_monmap 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.607+0000 7f3166b0b640 10 monclient: monmap: 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:43.622 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.607+0000 7f3166b0b640 1 librados: starting msgr at 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.607+0000 7f3166b0b640 1 librados: starting objecter 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 1 Processor -- start 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 1 -- start start 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 1 librados: setting wanted keys 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 1 librados: calling monclient init 2024-06-11T19:11:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 10 monclient: init 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 10 monclient: _add_conns ranks=[6,8,2,1,3,5,0,4,7] 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a2530 0x7f3160137740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 10 monclient(hunting): picked mon.noname-d con 0x7f31600a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31600a4920 0x7f3160137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 10 monclient(hunting): picked mon.noname-e con 0x7f31600a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31600a5240 0x7f31601232a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 10 monclient(hunting): picked mon.noname-b con 0x7f31600a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f31600a6da0 con 0x7f31600a5240 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3160138e50 con 0x7f31600a2530 2024-06-11T19:11:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f31600a6420 con 0x7f31600a4920 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3166b0b640 10 monclient(hunting): authenticate will time out at 913.415803s 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3157fff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31600a4920 msgr2=0x7f3160137c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f3157fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31600a4920 0x7f3160137c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31648a7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a2530 0x7f3160137740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31648a7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a2530 0x7f3160137740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54090/0 (socket says 172.21.15.98:54090) 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31648a7640 1 -- 172.21.15.98:0/549949723 learned_addr learned my addr 172.21.15.98:0/549949723 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31648a7640 10 monclient(hunting): get_auth_request con 0x7f31600a2530 auth_method 0 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31648a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31648a7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31648a7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31650a8640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31600a5240 0x7f31601232a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:43.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31650a8640 10 monclient(hunting): get_auth_request con 0x7f31600a5240 auth_method 0 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31650a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31650a8640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31650a8640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31648a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31648a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.608+0000 7f31648a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31650a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31650a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31650a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31648a7640 10 monclient(hunting): handle_auth_done global_id 54303 payload 824 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31648a7640 1 -- 172.21.15.98:0/549949723 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31600a4920 msgr2=0x7f3160137c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31648a7640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31600a4920 0x7f3160137c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31648a7640 1 -- 172.21.15.98:0/549949723 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31600a5240 msgr2=0x7f31601232a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:43.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31648a7640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31600a5240 0x7f31601232a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31648a7640 10 monclient: _finish_hunting 0 2024-06-11T19:11:43.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31648a7640 1 monclient: found mon.noname-d 2024-06-11T19:11:43.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31648a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:43.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31648a7640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:43.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31648a7640 1 -- 172.21.15.98:0/549949723 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31601c0d10 con 0x7f31600a2530 2024-06-11T19:11:43.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31650a8640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31600a5240 0x7f31601232a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:43.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31648a7640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a2530 0x7f3160137740 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f315c000c00 tx=0x7f315c0046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:43.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 10 client.?.objecter ms_handle_connect 0x7f31600a2530 2024-06-11T19:11:43.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:43.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 1 -- 172.21.15.98:0/549949723 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f315c042020 con 0x7f31600a2530 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 10 monclient: dump: 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:43.628 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:43.629 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:43.629 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:43.629 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:43.629 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:43.629 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:43.629 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:43.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 10 monclient: _finish_auth 0 2024-06-11T19:11:43.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 10 monclient: _check_auth_tickets 2024-06-11T19:11:43.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 1 -- 172.21.15.98:0/549949723 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f315c031ea0 con 0x7f31600a2530 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f3166b0b640 5 monclient: authenticate success, global_id 54303 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f3166b0b640 10 monclient: _renew_subs 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f3166b0b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f3166b0b640 1 -- 172.21.15.98:0/549949723 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3160126b80 con 0x7f31600a2530 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 1 -- 172.21.15.98:0/549949723 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f315c038040 con 0x7f31600a2530 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f31557fa640 10 monclient: dump: 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f3156ffd640 4 set_mon_vals no callback set 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f3166b0b640 10 client.54303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f3166b0b640 10 monclient: _renew_subs 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f3166b0b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f3166b0b640 1 -- 172.21.15.98:0/549949723 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f31600a92f0 con 0x7f31600a2530 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.609+0000 7f3166b0b640 1 librados: init done 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.610+0000 7f3166b0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.610+0000 7f3166b0b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.610+0000 7f3166b0b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.610+0000 7f3166b0b640 1 -- 172.21.15.98:0/549949723 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3160073550 con 0x7f31600a2530 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.610+0000 7f31557fa640 1 -- 172.21.15.98:0/549949723 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f315c045020 con 0x7f31600a2530 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.610+0000 7f31557fa640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f314c03fdb0 0x7f314c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.611+0000 7f31557fa640 1 -- 172.21.15.98:0/549949723 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(80..80 src has 1..80) ==== 7956+0+0 (secure 0 0 0) 0x7f315c073860 con 0x7f31600a2530 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.611+0000 7f31557fa640 10 client.54303.objecter ms_dispatch 0x7f3160000b90 osd_map(80..80 src has 1..80) 2024-06-11T19:11:43.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.611+0000 7f31557fa640 3 client.54303.objecter handle_osd_map got epochs [80,80] > 0 2024-06-11T19:11:43.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.611+0000 7f31557fa640 3 client.54303.objecter handle_osd_map decoding full epoch 80 2024-06-11T19:11:43.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.611+0000 7f31557fa640 20 client.54303.objecter dump_active .. 0 homeless 2024-06-11T19:11:43.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.611+0000 7f3157fff640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f314c03fdb0 0x7f314c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:43.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.611+0000 7f3157fff640 10 monclient: get_auth_request con 0x7f314c03fdb0 auth_method 0 2024-06-11T19:11:43.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.611+0000 7f31557fa640 10 client.54303.objecter ms_handle_connect 0x7f314c03fdb0 2024-06-11T19:11:43.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.611+0000 7f3157fff640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f314c03fdb0 0x7f314c042270 secure :-1 s=READY pgs=3998 cs=0 l=1 rev1=1 crypto rx=0x7f31600a5620 tx=0x7f3150009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:43.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.615+0000 7f31557fa640 1 -- 172.21.15.98:0/549949723 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3160073550 con 0x7f31600a2530 2024-06-11T19:11:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.615+0000 7f31557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.615+0000 7f31557fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:43.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.871+0000 7f3166b0b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:11:43.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.871+0000 7f3166b0b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:11:43.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.871+0000 7f3166b0b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:43.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.871+0000 7f3166b0b640 1 -- 172.21.15.98:0/549949723 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f31600a66f0 con 0x7f31600a2530 2024-06-11T19:11:43.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.872+0000 7f31557fa640 1 -- 172.21.15.98:0/549949723 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f315c04c030 con 0x7f31600a2530 2024-06-11T19:11:43.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.872+0000 7f31557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:11:43.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.872+0000 7f31557fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:43.872 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":12,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:11:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.873+0000 7f3166b0b640 10 librados: watch_flush enter 2024-06-11T19:11:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.873+0000 7f3166b0b640 10 librados: watch_flush exit 2024-06-11T19:11:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.873+0000 7f3166b0b640 20 client.54303.objecter shutdown clearing up homeless session... 2024-06-11T19:11:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.873+0000 7f3166b0b640 10 client.54303.objecter successfully canceled tick 2024-06-11T19:11:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.873+0000 7f3166b0b640 1 -- 172.21.15.98:0/549949723 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f314c03fdb0 msgr2=0x7f314c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.873+0000 7f3166b0b640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f314c03fdb0 0x7f314c042270 secure :-1 s=READY pgs=3998 cs=0 l=1 rev1=1 crypto rx=0x7f31600a5620 tx=0x7f3150009190 comp rx=0 tx=0).stop 2024-06-11T19:11:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.873+0000 7f3166b0b640 10 monclient: shutdown 2024-06-11T19:11:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.873+0000 7f3166b0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.873+0000 7f3166b0b640 1 -- 172.21.15.98:0/549949723 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a2530 msgr2=0x7f3160137740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.873+0000 7f3166b0b640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a2530 0x7f3160137740 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f315c000c00 tx=0x7f315c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.874+0000 7f3166b0b640 1 -- 172.21.15.98:0/549949723 shutdown_connections 2024-06-11T19:11:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.874+0000 7f3166b0b640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f314c03fdb0 0x7f314c042270 unknown :-1 s=CLOSED pgs=3998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.874+0000 7f3166b0b640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31600a5240 0x7f31601232a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.874+0000 7f3166b0b640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31600a4920 0x7f3160137c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.874+0000 7f3166b0b640 1 --2- 172.21.15.98:0/549949723 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31600a2530 0x7f3160137740 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.874+0000 7f3166b0b640 1 -- 172.21.15.98:0/549949723 >> 172.21.15.98:0/549949723 conn(0x7f31600ab850 msgr2=0x7f31600a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.874+0000 7f3166b0b640 1 -- 172.21.15.98:0/549949723 shutdown_connections 2024-06-11T19:11:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.874+0000 7f3166b0b640 1 -- 172.21.15.98:0/549949723 wait complete. 2024-06-11T19:11:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:43.874+0000 7f3166b0b640 1 librados: shutdown 2024-06-11T19:11:46.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:11:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.995+0000 7f4e92e92640 10 monclient: get_monmap_and_config 2024-06-11T19:11:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.995+0000 7f4e92e92640 10 monclient: build_initial_monmap 2024-06-11T19:11:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.995+0000 7f4e92e92640 10 monclient: monmap: 2024-06-11T19:11:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:46.996656+0000 2024-06-11T19:11:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:46.996656+0000 2024-06-11T19:11:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:46.996 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:46.996 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:46.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.996+0000 7f4e92e92640 1 Processor -- start 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.996+0000 7f4e92e92640 1 -- start start 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.996+0000 7f4e92e92640 10 monclient: init 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.996+0000 7f4e92e92640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.996+0000 7f4e92e92640 10 monclient: _add_conns ranks=[0,10,4,5,3,7,6,1,2,8,9] 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.996+0000 7f4e92e92640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e8c0a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.996+0000 7f4e92e92640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e92e92640 10 monclient(hunting): picked mon.noname-f con 0x7f4e8c0a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e92e92640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e92e92640 10 monclient(hunting): picked mon.noname-c con 0x7f4e8c13e1a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e92e92640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e8c0a7140 con 0x7f4e8c0a4920 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e92e92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e92e92640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f4e8c0a2b50 con 0x7f4e8c0a2530 2024-06-11T19:11:46.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e92e92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e92e92640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f4e8c0a2e30 con 0x7f4e8c13e1a0 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e92e92640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e92e92640 10 monclient(hunting): authenticate will time out at 916.804051s 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e90c2e640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4e8c0a4920 legacy=0x7f4e8c0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e9142f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e9142f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34866/0 (socket says 172.21.15.98:34866) 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 learned_addr learned my addr 172.21.15.98:0/2668371637 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e9142f640 10 monclient(hunting): get_auth_request con 0x7f4e8c13e1a0 auth_method 0 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e9142f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e9142f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e9142f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 msgr2=0x7f4e8c140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 msgr2=0x7f4e8c140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:47.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:47.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:47.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request con 0x7f4e8c0a2530 auth_method 0 2024-06-11T19:11:47.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:47.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e8bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:47.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:47.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 msgr2=0x7f4e8c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:11:47.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 msgr2=0x7f4e8c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:47.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:47.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:46.997+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:47.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> v1:172.21.15.98:6789/0 conn(0x7f4e8c0a4920 legacy=0x7f4e8c0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-06-11T19:11:47.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:47.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e9142f640 10 monclient(hunting): get_auth_request con 0x7f4e8c13e1a0 auth_method 0 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e9142f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e9142f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e9142f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 msgr2=0x7f4e8c140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 msgr2=0x7f4e8c140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request con 0x7f4e8c0a2530 auth_method 0 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e8bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 msgr2=0x7f4e8c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:11:47.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 msgr2=0x7f4e8c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:47.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:47.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.198+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:11:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> v1:172.21.15.98:6789/0 conn(0x7f4e8c0a4920 legacy=0x7f4e8c0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-06-11T19:11:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e9142f640 10 monclient(hunting): get_auth_request con 0x7f4e8c13e1a0 auth_method 0 2024-06-11T19:11:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e9142f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e9142f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e9142f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 msgr2=0x7f4e8c140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:11:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 msgr2=0x7f4e8c140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:11:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request con 0x7f4e8c0a2530 auth_method 0 2024-06-11T19:11:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e8bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 msgr2=0x7f4e8c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:11:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 msgr2=0x7f4e8c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:47.599+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:11:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> v1:172.21.15.98:6789/0 conn(0x7f4e8c0a4920 legacy=0x7f4e8c0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-06-11T19:11:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e9142f640 10 monclient(hunting): get_auth_request con 0x7f4e8c13e1a0 auth_method 0 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e9142f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e9142f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e9142f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 msgr2=0x7f4e8c140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 msgr2=0x7f4e8c140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request con 0x7f4e8c0a2530 auth_method 0 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e8bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 msgr2=0x7f4e8c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:11:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 msgr2=0x7f4e8c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:48.400+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:11:49.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 10 monclient(hunting): tick 2024-06-11T19:11:49.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:11:49.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 1 monclient(hunting): continuing hunt 2024-06-11T19:11:49.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:11:49.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 msgr2=0x7f4e8c140590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:49.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e8c13e1a0 0x7f4e8c140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 msgr2=0x7f4e8c13dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.117:3300/0 conn(0x7f4e8c0a2530 0x7f4e8c13dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 >> v1:172.21.15.98:6789/0 conn(0x7f4e8c0a4920 legacy=0x7f4e8c0a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 10 monclient: _add_conns ranks=[6,1,7,5,2,3,9,8] 2024-06-11T19:11:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 10 monclient(hunting): picked mon.noname-d con 0x7f4e700037d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f4e70006200 addr v1:172.21.15.117:6789/0 2024-06-11T19:11:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 10 monclient(hunting): picked mon.noname-i con 0x7f4e70009ed0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e8c0a3110 con 0x7f4e70006200 2024-06-11T19:11:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:49.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4e8c0a2b50 con 0x7f4e700037d0 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f4e8c0a2e30 con 0x7f4e70009ed0 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e8a7fc640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request con 0x7f4e700037d0 auth_method 0 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e90c2e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e90c2e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 msgr2=0x7f4e70005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 msgr2=0x7f4e70005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:50.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:50.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e9142f640 10 monclient(hunting): get_auth_request con 0x7f4e70009ed0 auth_method 0 2024-06-11T19:11:50.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e9142f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:50.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e9142f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:50.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e9142f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:50.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:11:50.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:50.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:50.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:49.997+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:50.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request con 0x7f4e700037d0 auth_method 0 2024-06-11T19:11:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e90c2e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e90c2e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 msgr2=0x7f4e70005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:11:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 msgr2=0x7f4e70005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:11:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e9142f640 10 monclient(hunting): get_auth_request con 0x7f4e70009ed0 auth_method 0 2024-06-11T19:11:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e9142f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e9142f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e9142f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:11:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.198+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:11:50.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request con 0x7f4e700037d0 auth_method 0 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e90c2e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e90c2e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 msgr2=0x7f4e70005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 msgr2=0x7f4e70005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e9142f640 10 monclient(hunting): get_auth_request con 0x7f4e70009ed0 auth_method 0 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e9142f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e9142f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e9142f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:11:50.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:50.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:50.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:50.599+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:11:51.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request con 0x7f4e700037d0 auth_method 0 2024-06-11T19:11:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e90c2e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e90c2e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 msgr2=0x7f4e70005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:11:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 msgr2=0x7f4e70005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:11:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e9142f640 10 monclient(hunting): get_auth_request con 0x7f4e70009ed0 auth_method 0 2024-06-11T19:11:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e9142f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e9142f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.400+0000 7f4e9142f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.401+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:11:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.401+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.401+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:51.401+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:11:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 10 monclient(hunting): tick 2024-06-11T19:11:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:11:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 1 monclient(hunting): continuing hunt 2024-06-11T19:11:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e70009ed0 0x7f4e7000c390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 msgr2=0x7f4e70005c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e700037d0 0x7f4e70005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 >> v1:172.21.15.117:6789/0 conn(0x7f4e70006200 legacy=0x7f4e70008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 reap_dead start 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 10 monclient: _add_conns ranks=[2,5,8,3,9] 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f4e7000cd90 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 10 monclient(hunting): picked mon.noname-h con 0x7f4e7000d210 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e70008e60 0x7f4e70008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 10 monclient(hunting): picked mon.noname-e con 0x7f4e70008e60 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8c0a2b50 con 0x7f4e7000cd90 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f4e8c0a2e30 con 0x7f4e7000d210 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4e7000d7b0 con 0x7f4e70008e60 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8a7fc640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e70008e60 msgr2=0x7f4e70008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e70008e60 0x7f4e70008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request con 0x7f4e7000cd90 auth_method 0 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e90c2e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e90c2e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request con 0x7f4e7000d210 auth_method 0 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.997+0000 7f4e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.998+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.998+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 msgr2=0x7f4e70005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.998+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 msgr2=0x7f4e70005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.998+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:52.998+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:53.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.198+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e70008e60 msgr2=0x7f4e70008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:11:53.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.198+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e70008e60 0x7f4e70008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:11:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request con 0x7f4e7000cd90 auth_method 0 2024-06-11T19:11:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:53.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e90c2e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e90c2e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request con 0x7f4e7000d210 auth_method 0 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e8bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 msgr2=0x7f4e70005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 msgr2=0x7f4e70005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:53.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.199+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:11:53.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.599+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e70008e60 msgr2=0x7f4e70008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:11:53.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.599+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e70008e60 0x7f4e70008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:11:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request con 0x7f4e7000cd90 auth_method 0 2024-06-11T19:11:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e90c2e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e90c2e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request con 0x7f4e7000d210 auth_method 0 2024-06-11T19:11:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e8bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:11:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:11:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 msgr2=0x7f4e70005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:11:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 msgr2=0x7f4e70005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:53.600+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:11:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.400+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e70008e60 msgr2=0x7f4e70008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:11:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.400+0000 7f4e9142f640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e70008e60 0x7f4e70008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:11:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request con 0x7f4e7000cd90 auth_method 0 2024-06-11T19:11:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e90c2e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e90c2e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request con 0x7f4e7000d210 auth_method 0 2024-06-11T19:11:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e8bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e8bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:11:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 msgr2=0x7f4e70005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:11:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 msgr2=0x7f4e70005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:11:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:54.401+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:11:55.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 10 monclient(hunting): tick 2024-06-11T19:11:55.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:11:55.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 1 monclient(hunting): continuing hunt 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e70008e60 msgr2=0x7f4e70008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e70008e60 0x7f4e70008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 msgr2=0x7f4e70005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e7000d210 0x7f4e70005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 msgr2=0x7f4e7000c390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e7000cd90 0x7f4e7000c390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 10 monclient: _add_conns ranks=[3,9] 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 0x7f4e700097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 10 monclient(hunting): picked mon.noname-g con 0x7f4e700093c0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.98:3300/0 conn(0x7f4e7000fd90 0x7f4e70012180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f4e7000fd90 addr v2:172.21.15.98:3300/0 2024-06-11T19:11:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4e8c0a2b50 con 0x7f4e7000fd90 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 1 -- 172.21.15.98:0/2668371637 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8c0a2e30 con 0x7f4e700093c0 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8a7fc640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.98:3300/0 conn(0x7f4e7000fd90 msgr2=0x7f4e70012180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.997+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.98:3300/0 conn(0x7f4e7000fd90 0x7f4e70012180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.998+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 0x7f4e700097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.998+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request con 0x7f4e700093c0 auth_method 0 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.998+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.998+0000 7f4e90c2e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.998+0000 7f4e90c2e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.998+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 msgr2=0x7f4e700097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:11:56.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.998+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 msgr2=0x7f4e700097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:56.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.998+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 0x7f4e700097a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:56.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:55.998+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 0x7f4e700097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.199+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.98:3300/0 conn(0x7f4e7000fd90 msgr2=0x7f4e70012180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-06-11T19:11:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.199+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.98:3300/0 conn(0x7f4e7000fd90 0x7f4e70012180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:11:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.199+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 0x7f4e700097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.199+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request con 0x7f4e700093c0 auth_method 0 2024-06-11T19:11:56.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.199+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:56.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.199+0000 7f4e90c2e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:56.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.199+0000 7f4e90c2e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:56.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.199+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 msgr2=0x7f4e700097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:11:56.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.199+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 msgr2=0x7f4e700097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:11:56.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.199+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 0x7f4e700097a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:11:56.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.199+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 0x7f4e700097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:11:56.489 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: Running main() from gmock_main.cc 2024-06-11T19:11:56.489 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-06-11T19:11:56.489 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [----------] Global test environment set-up. 2024-06-11T19:11:56.490 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-06-11T19:11:56.490 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-06-11T19:11:56.490 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (4408 ms) 2024-06-11T19:11:56.490 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-06-11T19:11:56.490 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5044 ms) 2024-06-11T19:11:56.490 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-06-11T19:11:56.490 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (3874 ms) 2024-06-11T19:11:56.490 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-06-11T19:11:56.490 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (4992 ms) 2024-06-11T19:11:56.490 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-06-11T19:11:56.490 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (6981 ms) 2024-06-11T19:11:56.490 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (25300 ms total) 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5022 ms) 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (5998 ms) 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8003 ms) 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (5980 ms) 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4524 ms) 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-06-11T19:11:56.491 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi117-37845-11. 2024-06-11T19:11:56.492 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: Waiting for snaps to purge. 2024-06-11T19:11:56.492 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (27155 ms) 2024-06-11T19:11:56.492 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (56682 ms total) 2024-06-11T19:11:56.492 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: 2024-06-11T19:11:56.492 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [----------] Global test environment tear-down 2024-06-11T19:11:56.492 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (81982 ms total) 2024-06-11T19:11:56.492 INFO:tasks.workunit.client.0.smithi117.stdout: snapshots: [ PASSED ] 11 tests. 2024-06-11T19:11:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.98:3300/0 conn(0x7f4e7000fd90 msgr2=0x7f4e70012180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-06-11T19:11:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.98:3300/0 conn(0x7f4e7000fd90 0x7f4e70012180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:11:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 0x7f4e700097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request con 0x7f4e700093c0 auth_method 0 2024-06-11T19:11:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:11:56.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 10 monclient(hunting): handle_auth_done global_id 21702 payload 274 2024-06-11T19:11:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 >> v2:172.21.15.98:3300/0 conn(0x7f4e7000fd90 msgr2=0x7f4e70012180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> v2:172.21.15.98:3300/0 conn(0x7f4e7000fd90 0x7f4e70012180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 10 monclient: _finish_hunting 0 2024-06-11T19:11:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 1 monclient: found mon.noname-g 2024-06-11T19:11:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:11:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.600+0000 7f4e90c2e640 1 -- 172.21.15.98:0/2668371637 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8c0a7140 con 0x7f4e700093c0 2024-06-11T19:11:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 0x7f4e700097a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f4e70003680 tx=0x7f4e80005040 comp rx=0 tx=0).ready entity=mon.3 client_cookie=4fc5ed0cd785852f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:56.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 1 -- 172.21.15.98:0/2668371637 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4e80038070 con 0x7f4e700093c0 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 10 monclient: dump: 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:56.604 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 1 -- 172.21.15.98:0/2668371637 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e80033ae0 con 0x7f4e700093c0 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e92e92640 5 monclient: authenticate success, global_id 21702 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e92e92640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e92e92640 4 set_mon_vals no callback set 2024-06-11T19:11:56.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 1 -- 172.21.15.98:0/2668371637 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4e80033c90 con 0x7f4e700093c0 2024-06-11T19:11:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:11:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.601+0000 7f4e8b7fe640 10 monclient: dump: 2024-06-11T19:11:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:56.606 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.602+0000 7f4e92e92640 10 monclient: shutdown 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.602+0000 7f4e92e92640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.602+0000 7f4e92e92640 1 -- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 msgr2=0x7f4e700097a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.602+0000 7f4e92e92640 1 --2- 172.21.15.98:0/2668371637 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e700093c0 0x7f4e700097a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f4e70003680 tx=0x7f4e80005040 comp rx=0 tx=0).stop 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.602+0000 7f4e9142f640 1 -- 172.21.15.98:0/2668371637 reap_dead start 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.602+0000 7f4e92e92640 1 -- 172.21.15.98:0/2668371637 shutdown_connections 2024-06-11T19:11:56.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.602+0000 7f4e92e92640 1 -- 172.21.15.98:0/2668371637 >> 172.21.15.98:0/2668371637 conn(0x7f4e8c0ab850 msgr2=0x7f4e8c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.602+0000 7f4e92e92640 1 -- 172.21.15.98:0/2668371637 shutdown_connections 2024-06-11T19:11:56.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.602+0000 7f4e92e92640 1 -- 172.21.15.98:0/2668371637 wait complete. 2024-06-11T19:11:56.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.603+0000 7f4e92e92640 10 monclient: build_initial_monmap 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.603+0000 7f4e92e92640 10 monclient: monmap: 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:56.613 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:56.614 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.603+0000 7f4e92e92640 1 librados: starting msgr at 2024-06-11T19:11:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.603+0000 7f4e92e92640 1 librados: starting objecter 2024-06-11T19:11:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.603+0000 7f4e92e92640 1 Processor -- start 2024-06-11T19:11:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.603+0000 7f4e92e92640 1 -- start start 2024-06-11T19:11:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.603+0000 7f4e92e92640 1 librados: setting wanted keys 2024-06-11T19:11:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.603+0000 7f4e92e92640 1 librados: calling monclient init 2024-06-11T19:11:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.603+0000 7f4e92e92640 10 monclient: init 2024-06-11T19:11:56.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 10 monclient: _add_conns ranks=[0,2,6,3,7,4,5,8,1] 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e700093c0 0x7f4e8c1cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 10 monclient(hunting): picked mon.noname-a con 0x7f4e700093c0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e8c136970 0x7f4e8c136d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 10 monclient(hunting): picked mon.noname-b con 0x7f4e8c136970 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e8c09f8e0 0x7f4e8c09fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 10 monclient(hunting): picked mon.noname-d con 0x7f4e8c09f8e0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c0a6da0 con 0x7f4e700093c0 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8c138e50 con 0x7f4e8c136970 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:56.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4e8c0a4d00 con 0x7f4e8c09f8e0 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e90c2e640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e700093c0 msgr2=0x7f4e8c1cd4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e92e92640 10 monclient(hunting): authenticate will time out at 926.411346s 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e90c2e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e700093c0 0x7f4e8c1cd4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e9142f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e8c09f8e0 0x7f4e8c09fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e8c136970 0x7f4e8c136d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e8c136970 0x7f4e8c136d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:37344/0 (socket says 172.21.15.98:37344) 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e9142f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e8c09f8e0 0x7f4e8c09fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54606/0 (socket says 172.21.15.98:54606) 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2576582467 learned_addr learned my addr 172.21.15.98:0/2576582467 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request con 0x7f4e8c136970 auth_method 0 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:56.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e9142f640 10 monclient(hunting): get_auth_request con 0x7f4e8c09f8e0 auth_method 0 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e9142f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e9142f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e9142f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e9142f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e9142f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e9142f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 10 monclient(hunting): handle_auth_done global_id 21620 payload 824 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2576582467 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e8c09f8e0 msgr2=0x7f4e8c09fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2576582467 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e8c09f8e0 0x7f4e8c09fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:56.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2576582467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e700093c0 msgr2=0x7f4e8c1cd4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2576582467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e700093c0 0x7f4e8c1cd4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 1 monclient: found mon.noname-b 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 1 -- 172.21.15.98:0/2576582467 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8c137290 con 0x7f4e8c136970 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e8bfff640 1 --2- 172.21.15.98:0/2576582467 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e8c136970 0x7f4e8c136d50 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f4e780179e0 tx=0x7f4e78017ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e897fa640 10 client.?.objecter ms_handle_connect 0x7f4e8c136970 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.604+0000 7f4e897fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 1 -- 172.21.15.98:0/2576582467 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4e78027070 con 0x7f4e8c136970 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:11:56.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 10 monclient: dump: 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:56.619 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 10 monclient: _finish_auth 0 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 10 monclient: _check_auth_tickets 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 1 -- 172.21.15.98:0/2576582467 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e78016c80 con 0x7f4e8c136970 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e92e92640 5 monclient: authenticate success, global_id 21620 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e92e92640 10 monclient: _renew_subs 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e92e92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e92e92640 1 -- 172.21.15.98:0/2576582467 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e8c123040 con 0x7f4e8c136970 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 1 -- 172.21.15.98:0/2576582467 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4e78016e30 con 0x7f4e8c136970 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:56.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e897fa640 10 monclient: dump: 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:56.621 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e92e92640 10 client.21620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e8affd640 4 set_mon_vals no callback set 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e92e92640 10 monclient: _renew_subs 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e92e92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e92e92640 1 -- 172.21.15.98:0/2576582467 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e8c122d60 con 0x7f4e8c136970 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.605+0000 7f4e92e92640 1 librados: init done 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e897fa640 1 -- 172.21.15.98:0/2576582467 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f4e78022470 con 0x7f4e8c136970 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e92e92640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e92e92640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e92e92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:56.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e92e92640 1 -- 172.21.15.98:0/2576582467 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e8c122b60 con 0x7f4e8c136970 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e897fa640 1 --2- 172.21.15.98:0/2576582467 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4e700424e0 0x7f4e700449a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e897fa640 1 -- 172.21.15.98:0/2576582467 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(81..81 src has 1..81) ==== 8134+0+0 (secure 0 0 0) 0x7f4e7801e030 con 0x7f4e8c136970 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e897fa640 10 client.21620.objecter ms_dispatch 0x7f4e8c000b90 osd_map(81..81 src has 1..81) 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e897fa640 3 client.21620.objecter handle_osd_map got epochs [81,81] > 0 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e897fa640 3 client.21620.objecter handle_osd_map decoding full epoch 81 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e897fa640 20 client.21620.objecter dump_active .. 0 homeless 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2576582467 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4e700424e0 0x7f4e700449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.606+0000 7f4e90c2e640 10 monclient: get_auth_request con 0x7f4e700424e0 auth_method 0 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.607+0000 7f4e897fa640 10 client.21620.objecter ms_handle_connect 0x7f4e700424e0 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.607+0000 7f4e90c2e640 1 --2- 172.21.15.98:0/2576582467 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4e700424e0 0x7f4e700449a0 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 crypto rx=0x7f4e70003680 tx=0x7f4e8003f040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.610+0000 7f4e897fa640 1 -- 172.21.15.98:0/2576582467 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4e8c137290 con 0x7f4e8c136970 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.610+0000 7f4e897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:56.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.610+0000 7f4e897fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.865+0000 7f4e92e92640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:11:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.865+0000 7f4e92e92640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:11:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.865+0000 7f4e92e92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.865+0000 7f4e92e92640 1 -- 172.21.15.98:0/2576582467 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4e8c0a5000 con 0x7f4e8c136970 2024-06-11T19:11:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.866+0000 7f4e897fa640 1 -- 172.21.15.98:0/2576582467 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3180 (secure 0 0 0) 0x7f4e78014d60 con 0x7f4e8c136970 2024-06-11T19:11:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.866+0000 7f4e897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:11:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.866+0000 7f4e897fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:56.866 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":18,"quorum":[1,2,3,4,5,6,7],"quorum_names":["f","b","g","c","h","d","i"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:11:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 10 librados: watch_flush enter 2024-06-11T19:11:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 10 librados: watch_flush exit 2024-06-11T19:11:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 20 client.21620.objecter shutdown clearing up homeless session... 2024-06-11T19:11:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 10 client.21620.objecter successfully canceled tick 2024-06-11T19:11:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 -- 172.21.15.98:0/2576582467 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4e700424e0 msgr2=0x7f4e700449a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 --2- 172.21.15.98:0/2576582467 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4e700424e0 0x7f4e700449a0 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 crypto rx=0x7f4e70003680 tx=0x7f4e8003f040 comp rx=0 tx=0).stop 2024-06-11T19:11:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 10 monclient: shutdown 2024-06-11T19:11:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 -- 172.21.15.98:0/2576582467 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e8c136970 msgr2=0x7f4e8c136d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 --2- 172.21.15.98:0/2576582467 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e8c136970 0x7f4e8c136d50 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f4e780179e0 tx=0x7f4e78017ea0 comp rx=0 tx=0).stop 2024-06-11T19:11:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 -- 172.21.15.98:0/2576582467 shutdown_connections 2024-06-11T19:11:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 --2- 172.21.15.98:0/2576582467 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4e700424e0 0x7f4e700449a0 unknown :-1 s=CLOSED pgs=4000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:56.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 --2- 172.21.15.98:0/2576582467 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e8c09f8e0 0x7f4e8c09fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:56.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 --2- 172.21.15.98:0/2576582467 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e8c136970 0x7f4e8c136d50 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:56.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 --2- 172.21.15.98:0/2576582467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e700093c0 0x7f4e8c1cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:56.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 -- 172.21.15.98:0/2576582467 >> 172.21.15.98:0/2576582467 conn(0x7f4e8c0ab850 msgr2=0x7f4e8c0a9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:56.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 -- 172.21.15.98:0/2576582467 shutdown_connections 2024-06-11T19:11:56.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 -- 172.21.15.98:0/2576582467 wait complete. 2024-06-11T19:11:56.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.868+0000 7f4e92e92640 1 librados: shutdown 2024-06-11T19:11:56.879 INFO:tasks.mon_thrash.ceph_manager:quorum is size 7 2024-06-11T19:11:56.879 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon b 2024-06-11T19:11:56.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:11:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.979+0000 7f74ca669640 10 monclient: get_monmap_and_config 2024-06-11T19:11:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.979+0000 7f74ca669640 10 monclient: build_initial_monmap 2024-06-11T19:11:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.979+0000 7f74ca669640 10 monclient: monmap: 2024-06-11T19:11:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:56.980276+0000 2024-06-11T19:11:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:56.980276+0000 2024-06-11T19:11:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:56.980 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:56.980 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:56.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.979+0000 7f74ca669640 1 Processor -- start 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 1 -- start start 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 10 monclient: init 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 10 monclient: _add_conns ranks=[10,6,7,3,8,2,4,0,5,1,9] 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f74c40a0cd0 0x7f74c40a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 10 monclient(hunting): picked mon.noname-f con 0x7f74c40a0cd0 addr v2:172.21.15.117:3300/0 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 0x7f74c4139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 10 monclient(hunting): picked mon.noname-d con 0x7f74c4138c80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74c40a2250 0x7f74c40a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 10 monclient(hunting): picked mon.noname-i con 0x7f74c40a2250 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f74c40739b0 con 0x7f74c40a0cd0 2024-06-11T19:11:56.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f74c409ecf0 con 0x7f74c4138c80 2024-06-11T19:11:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f74c409efd0 con 0x7f74c40a2250 2024-06-11T19:11:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74ca669640 10 monclient(hunting): authenticate will time out at 926.787545s 2024-06-11T19:11:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 0x7f74c4139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 0x7f74c4139060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54614/0 (socket says 172.21.15.98:54614) 2024-06-11T19:11:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 1 -- 172.21.15.98:0/4022900004 learned_addr learned my addr 172.21.15.98:0/4022900004 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 10 monclient(hunting): get_auth_request con 0x7f74c4138c80 auth_method 0 2024-06-11T19:11:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:56.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c3fff640 1 --2- 172.21.15.98:0/4022900004 >> v2:172.21.15.117:3300/0 conn(0x7f74c40a0cd0 0x7f74c40a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c8c06640 1 --2- 172.21.15.98:0/4022900004 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74c40a2250 0x7f74c40a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c3fff640 10 monclient(hunting): get_auth_request con 0x7f74c40a0cd0 auth_method 0 2024-06-11T19:11:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c8c06640 10 monclient(hunting): get_auth_request con 0x7f74c40a2250 auth_method 0 2024-06-11T19:11:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c8c06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:56.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c8c06640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c8c06640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 10 monclient(hunting): handle_auth_done global_id 54321 payload 274 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 1 -- 172.21.15.98:0/4022900004 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74c40a2250 msgr2=0x7f74c40a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 1 --2- 172.21.15.98:0/4022900004 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74c40a2250 0x7f74c40a26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 1 -- 172.21.15.98:0/4022900004 >> v2:172.21.15.117:3300/0 conn(0x7f74c40a0cd0 msgr2=0x7f74c40a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 1 --2- 172.21.15.98:0/4022900004 >> v2:172.21.15.117:3300/0 conn(0x7f74c40a0cd0 0x7f74c40a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 10 monclient: _finish_hunting 0 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 1 monclient: found mon.noname-d 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c37fe640 1 -- 172.21.15.98:0/4022900004 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c409f2b0 con 0x7f74c4138c80 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.980+0000 7f74c3fff640 1 --2- 172.21.15.98:0/4022900004 >> v2:172.21.15.117:3300/0 conn(0x7f74c40a0cd0 0x7f74c40a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c8c06640 1 --2- 172.21.15.98:0/4022900004 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74c40a2250 0x7f74c40a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:11:56.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c37fe640 1 --2- 172.21.15.98:0/4022900004 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 0x7f74c4139060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009880 tx=0x7f74ac02b7a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=abb017dcbd1f36f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 1 -- 172.21.15.98:0/4022900004 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f74ac038070 con 0x7f74c4138c80 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 10 monclient: dump: 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:56.996 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 10 monclient: _finish_auth 0 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 1 -- 172.21.15.98:0/4022900004 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74ac0044a0 con 0x7f74c4138c80 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74ca669640 5 monclient: authenticate success, global_id 54321 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74ca669640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:56.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 1 -- 172.21.15.98:0/4022900004 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f74ac003a80 con 0x7f74c4138c80 2024-06-11T19:11:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74ca669640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74ca669640 4 set_mon_vals no callback set 2024-06-11T19:11:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.981+0000 7f74c2ffd640 10 monclient: dump: 2024-06-11T19:11:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:56.998 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:56.999 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:56.999 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:56.999 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:56.999 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:56.999 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:56.999 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:56.999 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:56.999 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:56.999 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:56.999 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:56.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.982+0000 7f74ca669640 10 monclient: shutdown 2024-06-11T19:11:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.982+0000 7f74ca669640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.982+0000 7f74ca669640 1 -- 172.21.15.98:0/4022900004 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 msgr2=0x7f74c4139060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.982+0000 7f74ca669640 1 --2- 172.21.15.98:0/4022900004 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 0x7f74c4139060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009880 tx=0x7f74ac02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.982+0000 7f74ca669640 1 -- 172.21.15.98:0/4022900004 shutdown_connections 2024-06-11T19:11:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.982+0000 7f74ca669640 1 --2- 172.21.15.98:0/4022900004 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74c40a2250 0x7f74c40a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.982+0000 7f74ca669640 1 --2- 172.21.15.98:0/4022900004 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 0x7f74c4139060 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.982+0000 7f74ca669640 1 --2- 172.21.15.98:0/4022900004 >> v2:172.21.15.117:3300/0 conn(0x7f74c40a0cd0 0x7f74c40a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.982+0000 7f74ca669640 1 -- 172.21.15.98:0/4022900004 >> 172.21.15.98:0/4022900004 conn(0x7f74c40a7850 msgr2=0x7f74c40a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.982+0000 7f74ca669640 1 -- 172.21.15.98:0/4022900004 shutdown_connections 2024-06-11T19:11:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.982+0000 7f74ca669640 1 -- 172.21.15.98:0/4022900004 wait complete. 2024-06-11T19:11:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.983+0000 7f74ca669640 10 monclient: build_initial_monmap 2024-06-11T19:11:57.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.983+0000 7f74ca669640 10 monclient: monmap: 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:57.001 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:57.002 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:57.002 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:57.002 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.983+0000 7f74ca669640 1 librados: starting msgr at 2024-06-11T19:11:57.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.983+0000 7f74ca669640 1 librados: starting objecter 2024-06-11T19:11:57.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.983+0000 7f74ca669640 1 Processor -- start 2024-06-11T19:11:57.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.983+0000 7f74ca669640 1 -- start start 2024-06-11T19:11:57.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.983+0000 7f74ca669640 1 librados: setting wanted keys 2024-06-11T19:11:57.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.983+0000 7f74ca669640 1 librados: calling monclient init 2024-06-11T19:11:57.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.983+0000 7f74ca669640 10 monclient: init 2024-06-11T19:11:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.983+0000 7f74ca669640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.983+0000 7f74ca669640 10 monclient: _add_conns ranks=[7,2,6,3,8,4,5,0,1] 2024-06-11T19:11:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74c40a0cd0 0x7f74c41ce240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 monclient(hunting): picked mon.noname-i con 0x7f74c40a0cd0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c40a2250 0x7f74c41ce780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 monclient(hunting): picked mon.noname-b con 0x7f74c40a2250 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 0x7f74c412f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 monclient(hunting): picked mon.noname-d con 0x7f74c4138c80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f74c40a1770 con 0x7f74c40a2250 2024-06-11T19:11:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f74c40a15f0 con 0x7f74c4138c80 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f74c40a2c20 con 0x7f74c40a0cd0 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 monclient(hunting): authenticate will time out at 926.791053s 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c8c06640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 0x7f74c412f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c40a2250 0x7f74c41ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c8c06640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 0x7f74c412f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54628/0 (socket says 172.21.15.98:54628) 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c8c06640 1 -- 172.21.15.98:0/3513235762 learned_addr learned my addr 172.21.15.98:0/3513235762 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 10 monclient(hunting): get_auth_request con 0x7f74c40a2250 auth_method 0 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c8c06640 10 monclient(hunting): get_auth_request con 0x7f74c4138c80 auth_method 0 2024-06-11T19:11:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c8c06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c8c06640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c8c06640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c8c06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c8c06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c8c06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c3fff640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74c40a0cd0 0x7f74c41ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c3fff640 10 monclient(hunting): get_auth_request con 0x7f74c40a0cd0 auth_method 0 2024-06-11T19:11:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 10 monclient(hunting): handle_auth_done global_id 21629 payload 824 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 1 -- 172.21.15.98:0/3513235762 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74c40a0cd0 msgr2=0x7f74c41ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74c40a0cd0 0x7f74c41ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 1 -- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 msgr2=0x7f74c412f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 0x7f74c412f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 10 monclient: _finish_hunting 0 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 1 monclient: found mon.noname-b 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 1 -- 172.21.15.98:0/3513235762 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c4130e70 con 0x7f74c40a2250 2024-06-11T19:11:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c8c06640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f74c4138c80 0x7f74c412f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c3fff640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74c40a0cd0 0x7f74c41ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c37fe640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c40a2250 0x7f74c41ce780 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f74c40a38e0 tx=0x7f74ac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 10 client.?.objecter ms_handle_connect 0x7f74c40a2250 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f74ac043020 con 0x7f74c40a2250 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 10 monclient: dump: 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 10 monclient: _finish_auth 0 2024-06-11T19:11:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:11:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 5 monclient: authenticate success, global_id 21629 2024-06-11T19:11:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74ac039040 con 0x7f74c40a2250 2024-06-11T19:11:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 monclient: _renew_subs 2024-06-11T19:11:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 1 -- 172.21.15.98:0/3513235762 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c4139240 con 0x7f74c40a2250 2024-06-11T19:11:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f74ac038050 con 0x7f74c40a2250 2024-06-11T19:11:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:11:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c0ff9640 10 monclient: dump: 2024-06-11T19:11:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 client.21629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74c27fc640 4 set_mon_vals no callback set 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 monclient: _renew_subs 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 1 -- 172.21.15.98:0/3513235762 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f74c4132910 con 0x7f74c40a2250 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.984+0000 7f74ca669640 1 librados: init done 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74ca669640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74ca669640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c41319b0 0x7f74c412f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74ca669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74ca669640 1 -- 172.21.15.98:0/3513235762 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f74c4050830 con 0x7f74c41319b0 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f74ac0039e0 con 0x7f74c40a2250 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74c3fff640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c41319b0 0x7f74c412f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74c3fff640 10 monclient: get_auth_request con 0x7f74c41319b0 auth_method 0 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74c3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.985+0000 7f74c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c3fff640 10 monclient(hunting): handle_auth_done global_id 21638 payload 824 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c3fff640 1 -- 172.21.15.98:0/3513235762 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f74c40a46c0 con 0x7f74c41319b0 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f74a403fdb0 0x7f74a4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(81..81 src has 1..81) ==== 8134+0+0 (secure 0 0 0) 0x7f74ac073450 con 0x7f74c40a2250 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 10 client.21629.objecter ms_dispatch 0x7f74c4000b90 osd_map(81..81 src has 1..81) 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 3 client.21629.objecter handle_osd_map got epochs [81,81] > 0 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 3 client.21629.objecter handle_osd_map decoding full epoch 81 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c3fff640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c41319b0 0x7f74c412f9c0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f74b4010710 tx=0x7f74b4010bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 20 client.21629.objecter dump_active .. 0 homeless 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 10 client.21629.objecter ms_handle_connect 0x7f74c41319b0 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 10 client.21629.objecter resend_mon_ops 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c8c06640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f74a403fdb0 0x7f74a4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f74b401a020 con 0x7f74c41319b0 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 10 monclient: dump: 2024-06-11T19:11:57.013 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c8c06640 10 monclient: get_auth_request con 0x7f74a403fdb0 auth_method 0 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f74b400e7c0 con 0x7f74c41319b0 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c41319b0 msgr2=0x7f74c412f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c41319b0 0x7f74c412f9c0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f74b4010710 tx=0x7f74b4010bd0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c41319b0 msgr2=0x7f74c412f9c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c41319b0 0x7f74c412f9c0 secure :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f74b4010710 tx=0x7f74b4010bd0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c0ff9640 10 client.21629.objecter ms_handle_connect 0x7f74a403fdb0 2024-06-11T19:11:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.986+0000 7f74c8c06640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f74a403fdb0 0x7f74a4042270 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 crypto rx=0x7f74bc003f40 tx=0x7f74bc009890 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.992+0000 7f74ca669640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:57.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74ca669640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c412ff00 0x7f74c4130310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74ca669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74ca669640 1 -- 172.21.15.98:0/3513235762 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f74c40692c0 con 0x7f74c412ff00 2024-06-11T19:11:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c3fff640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c412ff00 0x7f74c4130310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c3fff640 10 monclient: get_auth_request con 0x7f74c412ff00 auth_method 0 2024-06-11T19:11:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c3fff640 10 monclient(hunting): handle_auth_done global_id 21647 payload 824 2024-06-11T19:11:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c3fff640 1 -- 172.21.15.98:0/3513235762 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f74c40a2f80 con 0x7f74c412ff00 2024-06-11T19:11:57.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 10 client.21629.objecter ms_handle_connect 0x7f74c412ff00 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 10 client.21629.objecter resend_mon_ops 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c3fff640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c412ff00 0x7f74c4130310 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f74b401b000 tx=0x7f74b40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f74b401a040 con 0x7f74c412ff00 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 10 monclient: dump: 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.019 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9421 (secure 0 0 0) 0x7f74b4015470 con 0x7f74c412ff00 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c412ff00 msgr2=0x7f74c4130310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c412ff00 0x7f74c4130310 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f74b401b000 tx=0x7f74b40046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 1 -- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c412ff00 msgr2=0x7f74c4130310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:57.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.993+0000 7f74c0ff9640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c412ff00 0x7f74c4130310 secure :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f74b401b000 tx=0x7f74b40046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.021 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:57.021 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:57.021 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:57.021 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:57.021 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 18, 2024-06-11T19:11:57.021 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 167226, 2024-06-11T19:11:57.021 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:57.021 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:57.022 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:57.022 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:57.022 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:57.022 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:57.022 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:57.022 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:11:57.022 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.022 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:11:57.022 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:57.022 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:57.022 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:57.022 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:57.023 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:57.025 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.026 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.028 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:57.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.030 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:57.030 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:57.030 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.030 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.030 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:57.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:57.035 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:57.039 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.041 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:57.042 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.043 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.044 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.045 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:57.045 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:57.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 10 librados: watch_flush enter 2024-06-11T19:11:57.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 10 librados: watch_flush exit 2024-06-11T19:11:57.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 20 client.21629.objecter shutdown clearing up homeless session... 2024-06-11T19:11:57.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 10 client.21629.objecter successfully canceled tick 2024-06-11T19:11:57.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 1 -- 172.21.15.98:0/3513235762 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f74a403fdb0 msgr2=0x7f74a4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f74a403fdb0 0x7f74a4042270 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 crypto rx=0x7f74bc003f40 tx=0x7f74bc009890 comp rx=0 tx=0).stop 2024-06-11T19:11:57.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 10 monclient: shutdown 2024-06-11T19:11:57.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:57.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 1 -- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c40a2250 msgr2=0x7f74c41ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c40a2250 0x7f74c41ce780 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f74c40a38e0 tx=0x7f74ac0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74c8c06640 1 -- 172.21.15.98:0/3513235762 reap_dead start 2024-06-11T19:11:57.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 1 -- 172.21.15.98:0/3513235762 shutdown_connections 2024-06-11T19:11:57.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 1 --2- 172.21.15.98:0/3513235762 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f74c40a2250 0x7f74c41ce780 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 1 -- 172.21.15.98:0/3513235762 >> 172.21.15.98:0/3513235762 conn(0x7f74c40a7850 msgr2=0x7f74c41bf820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:57.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 1 -- 172.21.15.98:0/3513235762 shutdown_connections 2024-06-11T19:11:57.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 1 -- 172.21.15.98:0/3513235762 wait complete. 2024-06-11T19:11:57.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:56.994+0000 7f74ca669640 1 librados: shutdown 2024-06-11T19:11:57.048 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:11:57.048 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon c 2024-06-11T19:11:57.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:11:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.148+0000 7f0a62143640 10 monclient: get_monmap_and_config 2024-06-11T19:11:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.148+0000 7f0a62143640 10 monclient: build_initial_monmap 2024-06-11T19:11:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.148+0000 7f0a62143640 10 monclient: monmap: 2024-06-11T19:11:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:57.149638+0000 2024-06-11T19:11:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:57.149638+0000 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:57.149 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.149+0000 7f0a62143640 1 Processor -- start 2024-06-11T19:11:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.149+0000 7f0a62143640 1 -- start start 2024-06-11T19:11:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.149+0000 7f0a62143640 10 monclient: init 2024-06-11T19:11:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.149+0000 7f0a62143640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.149+0000 7f0a62143640 10 monclient: _add_conns ranks=[3,4,7,8,6,2,1,5,0,9,10] 2024-06-11T19:11:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.149+0000 7f0a62143640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a5c0a4920 0x7f0a5c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.149+0000 7f0a62143640 10 monclient(hunting): picked mon.noname-g con 0x7f0a5c0a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.149+0000 7f0a62143640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c0a5240 0x7f0a5c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.149+0000 7f0a62143640 10 monclient(hunting): picked mon.noname-c con 0x7f0a5c0a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.149+0000 7f0a62143640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0a5c0a2530 0x7f0a5c0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a62143640 10 monclient(hunting): picked mon.noname-i con 0x7f0a5c0a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a62143640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a62143640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f0a5c0a7140 con 0x7f0a5c0a4920 2024-06-11T19:11:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a62143640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a62143640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0a5c0a2b50 con 0x7f0a5c0a5240 2024-06-11T19:11:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a62143640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a62143640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f0a5c0a2e30 con 0x7f0a5c0a2530 2024-06-11T19:11:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a62143640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a62143640 10 monclient(hunting): authenticate will time out at 926.957037s 2024-06-11T19:11:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c0a5240 0x7f0a5c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c0a5240 0x7f0a5c0a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55696/0 (socket says 172.21.15.98:55696) 2024-06-11T19:11:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 1 -- 172.21.15.98:0/3146168123 learned_addr learned my addr 172.21.15.98:0/3146168123 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 10 monclient(hunting): get_auth_request con 0x7f0a5c0a5240 auth_method 0 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5b7fe640 1 --2- 172.21.15.98:0/3146168123 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a5c0a4920 0x7f0a5c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5b7fe640 10 monclient(hunting): get_auth_request con 0x7f0a5c0a4920 auth_method 0 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5bfff640 1 --2- 172.21.15.98:0/3146168123 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0a5c0a2530 0x7f0a5c0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5bfff640 10 monclient(hunting): get_auth_request con 0x7f0a5c0a2530 auth_method 0 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 10 monclient(hunting): handle_auth_done global_id 22954 payload 274 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 1 -- 172.21.15.98:0/3146168123 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0a5c0a2530 msgr2=0x7f0a5c0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 1 --2- 172.21.15.98:0/3146168123 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0a5c0a2530 0x7f0a5c0a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 1 -- 172.21.15.98:0/3146168123 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a5c0a4920 msgr2=0x7f0a5c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 1 --2- 172.21.15.98:0/3146168123 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a5c0a4920 0x7f0a5c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 10 monclient: _finish_hunting 0 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 1 monclient: found mon.noname-c 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.150+0000 7f0a5affd640 1 -- 172.21.15.98:0/3146168123 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5c0a3110 con 0x7f0a5c0a5240 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5affd640 1 --2- 172.21.15.98:0/3146168123 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c0a5240 0x7f0a5c0a59a0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f0a500098a0 tx=0x7f0a5002b7a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=8d206d7ee36a0004 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 1 -- 172.21.15.98:0/3146168123 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a50038070 con 0x7f0a5c0a5240 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 10 monclient: dump: 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a62143640 5 monclient: authenticate success, global_id 22954 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 1 -- 172.21.15.98:0/3146168123 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a50031970 con 0x7f0a5c0a5240 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a62143640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a62143640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a62143640 4 set_mon_vals no callback set 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 1 -- 172.21.15.98:0/3146168123 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a50031ba0 con 0x7f0a5c0a5240 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a5a7fc640 10 monclient: dump: 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a62143640 10 monclient: shutdown 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a62143640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a62143640 1 -- 172.21.15.98:0/3146168123 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c0a5240 msgr2=0x7f0a5c0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a62143640 1 --2- 172.21.15.98:0/3146168123 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c0a5240 0x7f0a5c0a59a0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f0a500098a0 tx=0x7f0a5002b7a0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a62143640 1 -- 172.21.15.98:0/3146168123 shutdown_connections 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.151+0000 7f0a62143640 1 --2- 172.21.15.98:0/3146168123 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0a5c0a2530 0x7f0a5c0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.152+0000 7f0a62143640 1 --2- 172.21.15.98:0/3146168123 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c0a5240 0x7f0a5c0a59a0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.152+0000 7f0a62143640 1 --2- 172.21.15.98:0/3146168123 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a5c0a4920 0x7f0a5c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.152+0000 7f0a62143640 1 -- 172.21.15.98:0/3146168123 >> 172.21.15.98:0/3146168123 conn(0x7f0a5c0ab850 msgr2=0x7f0a5c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.152+0000 7f0a62143640 1 -- 172.21.15.98:0/3146168123 shutdown_connections 2024-06-11T19:11:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.152+0000 7f0a62143640 1 -- 172.21.15.98:0/3146168123 wait complete. 2024-06-11T19:11:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.152+0000 7f0a62143640 10 monclient: build_initial_monmap 2024-06-11T19:11:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.152+0000 7f0a62143640 10 monclient: monmap: 2024-06-11T19:11:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.152+0000 7f0a62143640 1 librados: starting msgr at 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.152+0000 7f0a62143640 1 librados: starting objecter 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.152+0000 7f0a62143640 1 Processor -- start 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 1 -- start start 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 1 librados: setting wanted keys 2024-06-11T19:11:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 1 librados: calling monclient init 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 10 monclient: init 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 10 monclient: _add_conns ranks=[7,6,5,4,3,2,0,1,8] 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0a5c0a2530 0x7f0a5c1cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 10 monclient(hunting): picked mon.noname-i con 0x7f0a5c0a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0a5c0a4920 0x7f0a5c1ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 10 monclient(hunting): picked mon.noname-d con 0x7f0a5c0a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0a5c0a5240 0x7f0a5c124fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 10 monclient(hunting): picked mon.noname-h con 0x7f0a5c0a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f0a5c0a6da0 con 0x7f0a5c0a5240 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0a5c138e50 con 0x7f0a5c0a4920 2024-06-11T19:11:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f0a5c0a6420 con 0x7f0a5c0a2530 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a62143640 10 monclient(hunting): authenticate will time out at 926.960553s 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0a5c0a4920 0x7f0a5c1ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0a5c0a4920 0x7f0a5c1ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54636/0 (socket says 172.21.15.98:54636) 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 1 -- 172.21.15.98:0/2131038138 learned_addr learned my addr 172.21.15.98:0/2131038138 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 10 monclient(hunting): get_auth_request con 0x7f0a5c0a4920 auth_method 0 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5b7fe640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0a5c0a2530 0x7f0a5c1cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5b7fe640 10 monclient(hunting): get_auth_request con 0x7f0a5c0a2530 auth_method 0 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5bfff640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0a5c0a5240 0x7f0a5c124fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5bfff640 10 monclient(hunting): get_auth_request con 0x7f0a5c0a5240 auth_method 0 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 10 monclient(hunting): handle_auth_done global_id 54339 payload 824 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 1 -- 172.21.15.98:0/2131038138 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0a5c0a2530 msgr2=0x7f0a5c1cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0a5c0a2530 0x7f0a5c1cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 1 -- 172.21.15.98:0/2131038138 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0a5c0a5240 msgr2=0x7f0a5c124fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0a5c0a5240 0x7f0a5c124fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 10 monclient: _finish_hunting 0 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 1 monclient: found mon.noname-d 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.153+0000 7f0a5affd640 1 -- 172.21.15.98:0/2131038138 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5c127770 con 0x7f0a5c0a4920 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a5affd640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0a5c0a4920 0x7f0a5c1ce010 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0a50000c00 tx=0x7f0a500046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 10 client.?.objecter ms_handle_connect 0x7f0a5c0a4920 2024-06-11T19:11:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a50038050 con 0x7f0a5c0a4920 2024-06-11T19:11:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 10 monclient: dump: 2024-06-11T19:11:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.177 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 10 monclient: _finish_auth 0 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 10 monclient: _check_auth_tickets 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a50044070 con 0x7f0a5c0a4920 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a62143640 5 monclient: authenticate success, global_id 54339 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a62143640 10 monclient: _renew_subs 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a62143640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a62143640 1 -- 172.21.15.98:0/2131038138 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a5c128f00 con 0x7f0a5c0a4920 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a50004350 con 0x7f0a5c0a4920 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a62143640 10 client.54339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a43fff640 10 monclient: dump: 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a59ffb640 4 set_mon_vals no callback set 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a62143640 10 monclient: _renew_subs 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a62143640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a62143640 1 -- 172.21.15.98:0/2131038138 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a5c1387f0 con 0x7f0a5c0a4920 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.154+0000 7f0a62143640 1 librados: init done 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a62143640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a62143640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128740 0x7f0a5c125520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a62143640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a62143640 1 -- 172.21.15.98:0/2131038138 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0a5c128330 con 0x7f0a5c128740 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5b7fe640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128740 0x7f0a5c125520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5b7fe640 10 monclient: get_auth_request con 0x7f0a5c128740 auth_method 0 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f0a50003610 con 0x7f0a5c0a4920 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_done global_id 22963 payload 824 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5b7fe640 1 -- 172.21.15.98:0/2131038138 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a5c0a66f0 con 0x7f0a5c128740 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0a4803fdb0 0x7f0a48042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(81..81 src has 1..81) ==== 8134+0+0 (secure 0 0 0) 0x7f0a50073e20 con 0x7f0a5c0a4920 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 10 client.54339.objecter ms_dispatch 0x7f0a5c000b90 osd_map(81..81 src has 1..81) 2024-06-11T19:11:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 3 client.54339.objecter handle_osd_map got epochs [81,81] > 0 2024-06-11T19:11:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 3 client.54339.objecter handle_osd_map decoding full epoch 81 2024-06-11T19:11:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5b7fe640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128740 0x7f0a5c125520 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c011860 tx=0x7f0a4c011d20 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5bfff640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0a4803fdb0 0x7f0a48042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a5bfff640 10 monclient: get_auth_request con 0x7f0a4803fdb0 auth_method 0 2024-06-11T19:11:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 20 client.54339.objecter dump_active .. 0 homeless 2024-06-11T19:11:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 10 client.54339.objecter ms_handle_connect 0x7f0a5c128740 2024-06-11T19:11:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 10 client.54339.objecter resend_mon_ops 2024-06-11T19:11:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a4c019070 con 0x7f0a5c128740 2024-06-11T19:11:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.155+0000 7f0a43fff640 10 monclient: dump: 2024-06-11T19:11:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.184 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.156+0000 7f0a43fff640 10 client.54339.objecter ms_handle_connect 0x7f0a4803fdb0 2024-06-11T19:11:57.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.156+0000 7f0a5bfff640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0a4803fdb0 0x7f0a48042270 secure :-1 s=READY pgs=4002 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c127d90 tx=0x7f0a5400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.156+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0a4c004240 con 0x7f0a5c128740 2024-06-11T19:11:57.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.156+0000 7f0a43fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:57.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.156+0000 7f0a43fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:57.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.156+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128740 msgr2=0x7f0a5c125520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.156+0000 7f0a43fff640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128740 0x7f0a5c125520 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c011860 tx=0x7f0a4c011d20 comp rx=0 tx=0).stop 2024-06-11T19:11:57.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.156+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128740 msgr2=0x7f0a5c125520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:57.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.156+0000 7f0a43fff640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128740 0x7f0a5c125520 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a62143640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a62143640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128b20 0x7f0a5c125a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a62143640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a62143640 1 -- 172.21.15.98:0/2131038138 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0a5c0a6140 con 0x7f0a5c128b20 2024-06-11T19:11:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a5b7fe640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128b20 0x7f0a5c125a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a5b7fe640 10 monclient: get_auth_request con 0x7f0a5c128b20 auth_method 0 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a5b7fe640 10 monclient(hunting): handle_auth_done global_id 22972 payload 824 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.162+0000 7f0a5b7fe640 1 -- 172.21.15.98:0/2131038138 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0a5c06c300 con 0x7f0a5c128b20 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a5b7fe640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128b20 0x7f0a5c125a60 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c000c00 tx=0x7f0a4c0046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 10 client.54339.objecter ms_handle_connect 0x7f0a5c128b20 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 10 client.54339.objecter resend_mon_ops 2024-06-11T19:11:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a4c019040 con 0x7f0a5c128b20 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 10 monclient: dump: 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.188 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.189 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.189 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.189 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.189 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.189 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9251 (secure 0 0 0) 0x7f0a4c01e450 con 0x7f0a5c128b20 2024-06-11T19:11:57.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:57.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:57.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128b20 msgr2=0x7f0a5c125a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128b20 0x7f0a5c125a60 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c000c00 tx=0x7f0a4c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 1 -- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128b20 msgr2=0x7f0a5c125a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:57.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a43fff640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0a5c128b20 0x7f0a5c125a60 secure :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c000c00 tx=0x7f0a4c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.190 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 18, 2024-06-11T19:11:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 167381, 2024-06-11T19:11:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:57.194 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.195 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.197 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.198 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.199 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:57.200 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:57.201 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:57.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:57.207 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:57.211 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.213 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:57.213 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:57.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a62143640 10 librados: watch_flush enter 2024-06-11T19:11:57.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a62143640 10 librados: watch_flush exit 2024-06-11T19:11:57.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a62143640 20 client.54339.objecter shutdown clearing up homeless session... 2024-06-11T19:11:57.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a62143640 10 client.54339.objecter successfully canceled tick 2024-06-11T19:11:57.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a62143640 1 -- 172.21.15.98:0/2131038138 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0a4803fdb0 msgr2=0x7f0a48042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a62143640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0a4803fdb0 0x7f0a48042270 secure :-1 s=READY pgs=4002 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c127d90 tx=0x7f0a5400b040 comp rx=0 tx=0).stop 2024-06-11T19:11:57.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a62143640 10 monclient: shutdown 2024-06-11T19:11:57.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a62143640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:57.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a62143640 1 -- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0a5c0a4920 msgr2=0x7f0a5c1ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a62143640 1 --2- 172.21.15.98:0/2131038138 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0a5c0a4920 0x7f0a5c1ce010 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0a50000c00 tx=0x7f0a500046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a5bfff640 1 -- 172.21.15.98:0/2131038138 reap_dead start 2024-06-11T19:11:57.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a5bfff640 1 -- 172.21.15.98:0/2131038138 reap_dead start 2024-06-11T19:11:57.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a62143640 1 -- 172.21.15.98:0/2131038138 shutdown_connections 2024-06-11T19:11:57.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.163+0000 7f0a62143640 1 -- 172.21.15.98:0/2131038138 >> 172.21.15.98:0/2131038138 conn(0x7f0a5c0ab850 msgr2=0x7f0a5c09c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:57.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.164+0000 7f0a62143640 1 -- 172.21.15.98:0/2131038138 shutdown_connections 2024-06-11T19:11:57.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.164+0000 7f0a62143640 1 -- 172.21.15.98:0/2131038138 wait complete. 2024-06-11T19:11:57.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.164+0000 7f0a62143640 1 librados: shutdown 2024-06-11T19:11:57.216 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:11:57.216 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon d 2024-06-11T19:11:57.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:11:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.316+0000 7f9cae346640 10 monclient: get_monmap_and_config 2024-06-11T19:11:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.316+0000 7f9cae346640 10 monclient: build_initial_monmap 2024-06-11T19:11:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.316+0000 7f9cae346640 10 monclient: monmap: 2024-06-11T19:11:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:57.317763+0000 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:57.317763+0000 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:57.318 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:57.318 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:57.318 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.317+0000 7f9cae346640 1 Processor -- start 2024-06-11T19:11:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.317+0000 7f9cae346640 1 -- start start 2024-06-11T19:11:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.317+0000 7f9cae346640 10 monclient: init 2024-06-11T19:11:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.317+0000 7f9cae346640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.317+0000 7f9cae346640 10 monclient: _add_conns ranks=[2,0,4,10,7,1,8,3,6,9,5] 2024-06-11T19:11:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.317+0000 7f9cae346640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca80a4920 0x7f9ca80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.317+0000 7f9cae346640 10 monclient(hunting): picked mon.noname-b con 0x7f9ca80a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.317+0000 7f9cae346640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ca80a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cae346640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9ca80a2530 0x7f9ca80a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cae346640 10 monclient(hunting): picked mon.noname-c con 0x7f9ca80a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cae346640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ca80a7140 con 0x7f9ca80a5240 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cae346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cae346640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca80a2b50 con 0x7f9ca80a4920 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cae346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cae346640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f9ca80a2e30 con 0x7f9ca80a2530 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cae346640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cae346640 10 monclient(hunting): authenticate will time out at 927.125040s 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9ca77fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9ca80a5240 legacy=0x7f9ca80a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9ca80a2530 0x7f9ca80a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9ca7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca80a4920 0x7f9ca80a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9ca80a2530 0x7f9ca80a60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55710/0 (socket says 172.21.15.98:55710) 2024-06-11T19:11:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 1 -- 172.21.15.98:0/1806198081 learned_addr learned my addr 172.21.15.98:0/1806198081 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9ca7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca80a4920 0x7f9ca80a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:37370/0 (socket says 172.21.15.98:37370) 2024-06-11T19:11:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 10 monclient(hunting): get_auth_request con 0x7f9ca80a2530 auth_method 0 2024-06-11T19:11:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9ca7fff640 10 monclient(hunting): get_auth_request con 0x7f9ca80a4920 auth_method 0 2024-06-11T19:11:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9ca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9ca7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9ca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9ca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9ca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9ca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 10 monclient(hunting): handle_auth_done global_id 22981 payload 274 2024-06-11T19:11:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 1 -- 172.21.15.98:0/1806198081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca80a4920 msgr2=0x7f9ca80a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 1 --2- 172.21.15.98:0/1806198081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca80a4920 0x7f9ca80a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 1 -- 172.21.15.98:0/1806198081 >> v1:172.21.15.98:6789/0 conn(0x7f9ca80a5240 legacy=0x7f9ca80a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 10 monclient: _finish_hunting 0 2024-06-11T19:11:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 1 monclient: found mon.noname-c 2024-06-11T19:11:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:11:57.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9cac8e3640 1 -- 172.21.15.98:0/1806198081 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca80a3110 con 0x7f9ca80a2530 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.318+0000 7f9ca7fff640 1 --2- 172.21.15.98:0/1806198081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca80a4920 0x7f9ca80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cac8e3640 1 --2- 172.21.15.98:0/1806198081 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9ca80a2530 0x7f9ca80a60b0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0009e80 tx=0x7f9ca002bd70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=851da48425c36e3e server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 1 -- 172.21.15.98:0/1806198081 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ca002cb80 con 0x7f9ca80a2530 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 10 monclient: dump: 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 10 monclient: _finish_auth 0 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 1 -- 172.21.15.98:0/1806198081 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ca002cd40 con 0x7f9ca80a2530 2024-06-11T19:11:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cae346640 5 monclient: authenticate success, global_id 22981 2024-06-11T19:11:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cae346640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cae346640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 1 -- 172.21.15.98:0/1806198081 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ca0034860 con 0x7f9ca80a2530 2024-06-11T19:11:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cae346640 4 set_mon_vals no callback set 2024-06-11T19:11:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9ca6ffd640 10 monclient: dump: 2024-06-11T19:11:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.337 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cae346640 10 monclient: shutdown 2024-06-11T19:11:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cae346640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cae346640 1 -- 172.21.15.98:0/1806198081 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9ca80a2530 msgr2=0x7f9ca80a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cae346640 1 --2- 172.21.15.98:0/1806198081 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9ca80a2530 0x7f9ca80a60b0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0009e80 tx=0x7f9ca002bd70 comp rx=0 tx=0).stop 2024-06-11T19:11:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cae346640 1 -- 172.21.15.98:0/1806198081 shutdown_connections 2024-06-11T19:11:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cae346640 1 --2- 172.21.15.98:0/1806198081 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9ca80a2530 0x7f9ca80a60b0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cae346640 1 --2- 172.21.15.98:0/1806198081 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9ca80a4920 0x7f9ca80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.319+0000 7f9cae346640 1 -- 172.21.15.98:0/1806198081 >> 172.21.15.98:0/1806198081 conn(0x7f9ca80ab850 msgr2=0x7f9ca80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.320+0000 7f9cae346640 1 -- 172.21.15.98:0/1806198081 shutdown_connections 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.320+0000 7f9cae346640 1 -- 172.21.15.98:0/1806198081 wait complete. 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.320+0000 7f9cae346640 10 monclient: build_initial_monmap 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.320+0000 7f9cae346640 10 monclient: monmap: 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.320+0000 7f9cae346640 1 librados: starting msgr at 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.320+0000 7f9cae346640 1 librados: starting objecter 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.320+0000 7f9cae346640 1 Processor -- start 2024-06-11T19:11:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 1 -- start start 2024-06-11T19:11:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 1 librados: setting wanted keys 2024-06-11T19:11:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 1 librados: calling monclient init 2024-06-11T19:11:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 10 monclient: init 2024-06-11T19:11:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 10 monclient: _add_conns ranks=[6,0,8,3,2,7,5,1,4] 2024-06-11T19:11:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca80a2530 0x7f9ca81cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 10 monclient(hunting): picked mon.noname-d con 0x7f9ca80a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ca80a4920 0x7f9ca81ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 10 monclient(hunting): picked mon.noname-a con 0x7f9ca80a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9ca80a5240 0x7f9ca8122d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 10 monclient(hunting): picked mon.noname-e con 0x7f9ca80a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca80a6da0 con 0x7f9ca80a4920 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f9ca8138e50 con 0x7f9ca80a2530 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9ca80a65f0 con 0x7f9ca80a5240 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cae346640 10 monclient(hunting): authenticate will time out at 927.128520s 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca77fe640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ca80a4920 msgr2=0x7f9ca81ce010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cac8e3640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9ca80a5240 msgr2=0x7f9ca8122d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ca80a4920 0x7f9ca81ce010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9cac8e3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9ca80a5240 0x7f9ca8122d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca80a2530 0x7f9ca81cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca80a2530 0x7f9ca81cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54640/0 (socket says 172.21.15.98:54640) 2024-06-11T19:11:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 1 -- 172.21.15.98:0/2952339325 learned_addr learned my addr 172.21.15.98:0/2952339325 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 10 monclient(hunting): get_auth_request con 0x7f9ca80a2530 auth_method 0 2024-06-11T19:11:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 10 monclient(hunting): handle_auth_done global_id 54348 payload 824 2024-06-11T19:11:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 1 -- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9ca80a5240 msgr2=0x7f9ca8122d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:11:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9ca80a5240 0x7f9ca8122d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 1 -- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ca80a4920 msgr2=0x7f9ca81ce010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:11:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ca80a4920 0x7f9ca81ce010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:11:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 1 monclient: found mon.noname-d 2024-06-11T19:11:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:11:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 1 -- 172.21.15.98:0/2952339325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca8126cb0 con 0x7f9ca80a2530 2024-06-11T19:11:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca7fff640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca80a2530 0x7f9ca81cdad0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800ebe0 tx=0x7f9c9800c520 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca4ff9640 10 client.?.objecter ms_handle_connect 0x7f9ca80a2530 2024-06-11T19:11:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.321+0000 7f9ca4ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c98017070 con 0x7f9ca80a2530 2024-06-11T19:11:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:11:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 10 monclient: dump: 2024-06-11T19:11:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 10 monclient: _finish_auth 0 2024-06-11T19:11:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:11:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9cae346640 5 monclient: authenticate success, global_id 54348 2024-06-11T19:11:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c9800cc30 con 0x7f9ca80a2530 2024-06-11T19:11:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c9800cde0 con 0x7f9ca80a2530 2024-06-11T19:11:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca4ff9640 10 monclient: dump: 2024-06-11T19:11:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9ca67fc640 4 set_mon_vals no callback set 2024-06-11T19:11:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9cae346640 10 monclient: _renew_subs 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9cae346640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9cae346640 1 -- 172.21.15.98:0/2952339325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca80a9110 con 0x7f9ca80a2530 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9cae346640 10 client.54348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9cae346640 10 monclient: _renew_subs 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9cae346640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9cae346640 1 -- 172.21.15.98:0/2952339325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca81269d0 con 0x7f9ca80a2530 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.322+0000 7f9cae346640 1 librados: init done 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9cae346640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9cae346640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123a40 0x7f9ca802cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9cae346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9cae346640 1 -- 172.21.15.98:0/2952339325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f9ca8126650 con 0x7f9ca8123a40 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f9c9801a070 con 0x7f9ca80a2530 2024-06-11T19:11:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca77fe640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123a40 0x7f9ca802cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca77fe640 10 monclient: get_auth_request con 0x7f9ca8123a40 auth_method 0 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c9403fde0 0x7f9c940422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca77fe640 10 monclient(hunting): handle_auth_done global_id 54357 payload 824 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca77fe640 1 -- 172.21.15.98:0/2952339325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ca80a4d00 con 0x7f9ca8123a40 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(81..81 src has 1..81) ==== 8134+0+0 (secure 0 0 0) 0x7f9c9800f040 con 0x7f9ca80a2530 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 10 client.54348.objecter ms_dispatch 0x7f9ca8000b90 osd_map(81..81 src has 1..81) 2024-06-11T19:11:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 3 client.54348.objecter handle_osd_map got epochs [81,81] > 0 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 3 client.54348.objecter handle_osd_map decoding full epoch 81 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca77fe640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123a40 0x7f9ca802cb40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9ca81247f0 tx=0x7f9c9c00b830 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 20 client.54348.objecter dump_active .. 0 homeless 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 10 client.54348.objecter ms_handle_connect 0x7f9ca8123a40 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 10 client.54348.objecter resend_mon_ops 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9cac8e3640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c9403fde0 0x7f9c940422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9cac8e3640 10 monclient: get_auth_request con 0x7f9c9403fde0 auth_method 0 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c9c016070 con 0x7f9ca8123a40 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.323+0000 7f9ca4ff9640 10 monclient: dump: 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.352 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.324+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9c9c003bc0 con 0x7f9ca8123a40 2024-06-11T19:11:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.324+0000 7f9ca4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.324+0000 7f9ca4ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.324+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123a40 msgr2=0x7f9ca802cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.324+0000 7f9ca4ff9640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123a40 0x7f9ca802cb40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9ca81247f0 tx=0x7f9c9c00b830 comp rx=0 tx=0).stop 2024-06-11T19:11:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.324+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123a40 msgr2=0x7f9ca802cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.324+0000 7f9ca4ff9640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123a40 0x7f9ca802cb40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.324+0000 7f9ca4ff9640 10 client.54348.objecter ms_handle_connect 0x7f9c9403fde0 2024-06-11T19:11:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.324+0000 7f9cac8e3640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c9403fde0 0x7f9c940422a0 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0035000 tx=0x7f9ca0030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9cae346640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9cae346640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123e20 0x7f9ca802d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9cae346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9cae346640 1 -- 172.21.15.98:0/2952339325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f9ca81249c0 con 0x7f9ca8123e20 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca77fe640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123e20 0x7f9ca802d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca77fe640 10 monclient: get_auth_request con 0x7f9ca8123e20 auth_method 0 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca77fe640 10 monclient(hunting): handle_auth_done global_id 54366 payload 824 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca77fe640 1 -- 172.21.15.98:0/2952339325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9ca80a0100 con 0x7f9ca8123e20 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca4ff9640 10 client.54348.objecter ms_handle_connect 0x7f9ca8123e20 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca77fe640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123e20 0x7f9ca802d080 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c013000 tx=0x7f9c9c0037f0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.330+0000 7f9ca4ff9640 10 client.54348.objecter resend_mon_ops 2024-06-11T19:11:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c9c016040 con 0x7f9ca8123e20 2024-06-11T19:11:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9ca4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9ca4ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:11:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9ca4ff9640 10 monclient: dump: 2024-06-11T19:11:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9250 (secure 0 0 0) 0x7f9c9c011400 con 0x7f9ca8123e20 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9ca4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9ca4ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123e20 msgr2=0x7f9ca802d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9ca4ff9640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123e20 0x7f9ca802d080 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c013000 tx=0x7f9c9c0037f0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9ca4ff9640 1 -- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123e20 msgr2=0x7f9ca802d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:57.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9ca4ff9640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca8123e20 0x7f9ca802d080 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c013000 tx=0x7f9c9c0037f0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.358 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:57.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:57.358 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:57.358 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:57.358 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 18, 2024-06-11T19:11:57.358 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 30087, 2024-06-11T19:11:57.358 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:57.358 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.360 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.360 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.360 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.360 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.360 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.360 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.360 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.360 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.360 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:57.360 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:57.360 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:57.381 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:57.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 10 librados: watch_flush enter 2024-06-11T19:11:57.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 10 librados: watch_flush exit 2024-06-11T19:11:57.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 20 client.54348.objecter shutdown clearing up homeless session... 2024-06-11T19:11:57.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 10 client.54348.objecter successfully canceled tick 2024-06-11T19:11:57.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 1 -- 172.21.15.98:0/2952339325 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c9403fde0 msgr2=0x7f9c940422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c9403fde0 0x7f9c940422a0 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0035000 tx=0x7f9ca0030000 comp rx=0 tx=0).stop 2024-06-11T19:11:57.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 10 monclient: shutdown 2024-06-11T19:11:57.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:57.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 1 -- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca80a2530 msgr2=0x7f9ca81cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 1 --2- 172.21.15.98:0/2952339325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9ca80a2530 0x7f9ca81cdad0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800ebe0 tx=0x7f9c9800c520 comp rx=0 tx=0).stop 2024-06-11T19:11:57.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cac8e3640 1 -- 172.21.15.98:0/2952339325 reap_dead start 2024-06-11T19:11:57.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cac8e3640 1 -- 172.21.15.98:0/2952339325 reap_dead start 2024-06-11T19:11:57.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 1 -- 172.21.15.98:0/2952339325 shutdown_connections 2024-06-11T19:11:57.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 1 -- 172.21.15.98:0/2952339325 >> 172.21.15.98:0/2952339325 conn(0x7f9ca80ab850 msgr2=0x7f9ca81bef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:57.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 1 -- 172.21.15.98:0/2952339325 shutdown_connections 2024-06-11T19:11:57.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 1 -- 172.21.15.98:0/2952339325 wait complete. 2024-06-11T19:11:57.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.331+0000 7f9cae346640 1 librados: shutdown 2024-06-11T19:11:57.385 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:11:57.385 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon f 2024-06-11T19:11:57.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:11:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.487+0000 7ff892c20640 10 monclient: get_monmap_and_config 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.487+0000 7ff892c20640 10 monclient: build_initial_monmap 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.487+0000 7ff892c20640 10 monclient: monmap: 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:57.488256+0000 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:57.488256+0000 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:57.488 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:57.488 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:57.488 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:57.488 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:57.488 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:57.488 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:57.488 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 1 Processor -- start 2024-06-11T19:11:57.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 1 -- start start 2024-06-11T19:11:57.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 10 monclient: init 2024-06-11T19:11:57.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 10 monclient: _add_conns ranks=[5,3,10,8,2,9,0,1,7,4,6] 2024-06-11T19:11:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff88c09d330 0x7ff88c09d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 10 monclient(hunting): picked mon.noname-h con 0x7ff88c09d330 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:11:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff88c09dc50 0x7ff88c136770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 10 monclient(hunting): picked mon.noname-g con 0x7ff88c09dc50 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff88c0a2850 0x7ff88c136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 10 monclient(hunting): picked mon.noname-f con 0x7ff88c0a2850 addr v2:172.21.15.117:3300/0 2024-06-11T19:11:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7ff88c09bb20 con 0x7ff88c0a2850 2024-06-11T19:11:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ff88c09b560 con 0x7ff88c09dc50 2024-06-11T19:11:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ff88c09b840 con 0x7ff88c09d330 2024-06-11T19:11:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.488+0000 7ff892c20640 10 monclient(hunting): authenticate will time out at 927.295977s 2024-06-11T19:11:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8909bc640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff88c09d330 0x7ff88c09d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8909bc640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff88c09d330 0x7ff88c09d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:58346/0 (socket says 172.21.15.98:58346) 2024-06-11T19:11:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8909bc640 1 -- 172.21.15.98:0/2060686632 learned_addr learned my addr 172.21.15.98:0/2060686632 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8909bc640 10 monclient(hunting): get_auth_request con 0x7ff88c09d330 auth_method 0 2024-06-11T19:11:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8909bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8909bc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8909bc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8911bd640 1 --2- 172.21.15.98:0/2060686632 >> v2:172.21.15.117:3300/0 conn(0x7ff88c0a2850 0x7ff88c136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 1 --2- 172.21.15.98:0/2060686632 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff88c09dc50 0x7ff88c136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 10 monclient(hunting): get_auth_request con 0x7ff88c09dc50 auth_method 0 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8911bd640 10 monclient(hunting): get_auth_request con 0x7ff88c0a2850 auth_method 0 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8911bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8911bd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8911bd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8909bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8909bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8909bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8911bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8911bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8911bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 10 monclient(hunting): handle_auth_done global_id 21720 payload 274 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 1 -- 172.21.15.98:0/2060686632 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff88c09d330 msgr2=0x7ff88c09d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 1 --2- 172.21.15.98:0/2060686632 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff88c09d330 0x7ff88c09d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 1 -- 172.21.15.98:0/2060686632 >> v2:172.21.15.117:3300/0 conn(0x7ff88c0a2850 msgr2=0x7ff88c136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 1 --2- 172.21.15.98:0/2060686632 >> v2:172.21.15.117:3300/0 conn(0x7ff88c0a2850 0x7ff88c136cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 1 monclient: found mon.noname-g 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:57.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff88bfff640 1 -- 172.21.15.98:0/2060686632 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c0769a0 con 0x7ff88c09dc50 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8911bd640 1 --2- 172.21.15.98:0/2060686632 >> v2:172.21.15.117:3300/0 conn(0x7ff88c0a2850 0x7ff88c136cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.489+0000 7ff8909bc640 1 --2- 172.21.15.98:0/2060686632 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff88c09d330 0x7ff88c09d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88bfff640 1 --2- 172.21.15.98:0/2060686632 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff88c09dc50 0x7ff88c136770 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7ff878009790 tx=0x7ff87802b6e0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=89a7944d0b569e00 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 1 -- 172.21.15.98:0/2060686632 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff878037070 con 0x7ff88c09dc50 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 10 monclient: dump: 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.497 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:11:57.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:11:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff892c20640 5 monclient: authenticate success, global_id 21720 2024-06-11T19:11:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff892c20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 1 -- 172.21.15.98:0/2060686632 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff878030980 con 0x7ff88c09dc50 2024-06-11T19:11:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff892c20640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff892c20640 4 set_mon_vals no callback set 2024-06-11T19:11:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 1 -- 172.21.15.98:0/2060686632 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff878030bb0 con 0x7ff88c09dc50 2024-06-11T19:11:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:11:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.490+0000 7ff88b7fe640 10 monclient: dump: 2024-06-11T19:11:57.499 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.500 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.491+0000 7ff892c20640 10 monclient: shutdown 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.491+0000 7ff892c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.491+0000 7ff892c20640 1 -- 172.21.15.98:0/2060686632 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff88c09dc50 msgr2=0x7ff88c136770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.491+0000 7ff892c20640 1 --2- 172.21.15.98:0/2060686632 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff88c09dc50 0x7ff88c136770 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7ff878009790 tx=0x7ff87802b6e0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.491+0000 7ff892c20640 1 -- 172.21.15.98:0/2060686632 shutdown_connections 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.491+0000 7ff892c20640 1 --2- 172.21.15.98:0/2060686632 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff88c09dc50 0x7ff88c136770 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.491+0000 7ff892c20640 1 --2- 172.21.15.98:0/2060686632 >> v2:172.21.15.117:3300/0 conn(0x7ff88c0a2850 0x7ff88c136cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.491+0000 7ff892c20640 1 --2- 172.21.15.98:0/2060686632 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff88c09d330 0x7ff88c09d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.491+0000 7ff892c20640 1 -- 172.21.15.98:0/2060686632 >> 172.21.15.98:0/2060686632 conn(0x7ff88c0a7850 msgr2=0x7ff88c0a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.491+0000 7ff892c20640 1 -- 172.21.15.98:0/2060686632 shutdown_connections 2024-06-11T19:11:57.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.491+0000 7ff892c20640 1 -- 172.21.15.98:0/2060686632 wait complete. 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.492+0000 7ff892c20640 10 monclient: build_initial_monmap 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.492+0000 7ff892c20640 10 monclient: monmap: 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:57.502 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:57.503 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.492+0000 7ff892c20640 1 librados: starting msgr at 2024-06-11T19:11:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.492+0000 7ff892c20640 1 librados: starting objecter 2024-06-11T19:11:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.492+0000 7ff892c20640 1 Processor -- start 2024-06-11T19:11:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.493+0000 7ff892c20640 1 -- start start 2024-06-11T19:11:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.493+0000 7ff892c20640 1 librados: setting wanted keys 2024-06-11T19:11:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.493+0000 7ff892c20640 1 librados: calling monclient init 2024-06-11T19:11:57.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.493+0000 7ff892c20640 10 monclient: init 2024-06-11T19:11:57.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:57.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 10 monclient: _add_conns ranks=[8,6,1,5,4,0,7,3,2] 2024-06-11T19:11:57.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff88c09d330 0x7ff88c1328b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 10 monclient(hunting): picked mon.noname-e con 0x7ff88c09d330 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:11:57.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff88c0a2850 0x7ff88c132df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 10 monclient(hunting): picked mon.noname-d con 0x7ff88c0a2850 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c133330 0x7ff88c1cd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 10 monclient(hunting): picked mon.noname-f con 0x7ff88c133330 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff8909bc640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff88c09d330 msgr2=0x7ff88c1328b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c1371f0 con 0x7ff88c133330 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff8909bc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff88c09d330 0x7ff88c1328b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff88c09e270 con 0x7ff88c0a2850 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff88c137370 con 0x7ff88c09d330 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:57.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff892c20640 10 monclient(hunting): authenticate will time out at 927.301867s 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.494+0000 7ff8911bd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c133330 0x7ff88c1cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c133330 0x7ff88c1cd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:55064/0 (socket says 172.21.15.98:55064) 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 1 -- 172.21.15.98:0/8906109 learned_addr learned my addr 172.21.15.98:0/8906109 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 10 monclient(hunting): get_auth_request con 0x7ff88c133330 auth_method 0 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 10 monclient(hunting): handle_auth_done global_id 30907 payload 824 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 1 -- 172.21.15.98:0/8906109 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff88c09d330 msgr2=0x7ff88c1328b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:11:57.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff88c09d330 0x7ff88c1328b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 1 -- 172.21.15.98:0/8906109 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff88c0a2850 msgr2=0x7ff88c132df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff88c0a2850 0x7ff88c132df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 10 monclient: _finish_hunting 0 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 1 monclient: found mon.noname-f 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.495+0000 7ff8911bd640 1 -- 172.21.15.98:0/8906109 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c0a71e0 con 0x7ff88c133330 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.496+0000 7ff8911bd640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c133330 0x7ff88c1cd920 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7ff884009e10 tx=0x7ff88400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.496+0000 7ff8897fa640 10 client.?.objecter ms_handle_connect 0x7ff88c133330 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.496+0000 7ff8897fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.497+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff884018020 con 0x7ff88c133330 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.497+0000 7ff8897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff8897fa640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff8897fa640 10 monclient: dump: 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.514 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff8897fa640 10 monclient: _finish_auth 0 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff8897fa640 10 monclient: _check_auth_tickets 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff8897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff892c20640 5 monclient: authenticate success, global_id 30907 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff892c20640 10 monclient: _renew_subs 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff892c20640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff892c20640 1 -- 172.21.15.98:0/8906109 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff88c0a6e20 con 0x7ff88c133330 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff892c20640 10 client.30907.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:57.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff892c20640 10 monclient: _renew_subs 2024-06-11T19:11:57.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff892c20640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:11:57.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff892c20640 1 -- 172.21.15.98:0/8906109 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff88c134d20 con 0x7ff88c133330 2024-06-11T19:11:57.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.498+0000 7ff892c20640 1 librados: init done 2024-06-11T19:11:57.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.500+0000 7ff892c20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:57.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.500+0000 7ff892c20640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c1ce620 0x7ff88c02cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.500+0000 7ff892c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.500+0000 7ff892c20640 1 -- 172.21.15.98:0/8906109 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c133770 con 0x7ff88c1ce620 2024-06-11T19:11:57.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.500+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff88400cd90 con 0x7ff88c133330 2024-06-11T19:11:57.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.500+0000 7ff8897fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:57.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.500+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff8840148f0 con 0x7ff88c133330 2024-06-11T19:11:57.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.500+0000 7ff8897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.501+0000 7ff8897fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.501+0000 7ff8897fa640 10 monclient: dump: 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.517 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.501+0000 7ff88affd640 4 set_mon_vals no callback set 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.501+0000 7ff8909bc640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c1ce620 0x7ff88c02cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.501+0000 7ff8909bc640 10 monclient: get_auth_request con 0x7ff88c1ce620 auth_method 0 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.501+0000 7ff8909bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.501+0000 7ff8909bc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.501+0000 7ff8909bc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.502+0000 7ff8909bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.502+0000 7ff8909bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.502+0000 7ff8909bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.502+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff884005d10 con 0x7ff88c133330 2024-06-11T19:11:57.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.503+0000 7ff8897fa640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff87003fdb0 0x7ff870042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.503+0000 7ff8909bc640 10 monclient(hunting): handle_auth_done global_id 30916 payload 824 2024-06-11T19:11:57.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.503+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8529+0+0 (secure 0 0 0) 0x7ff8840530d0 con 0x7ff88c133330 2024-06-11T19:11:57.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.503+0000 7ff8897fa640 10 client.30907.objecter ms_dispatch 0x7ff88c000b90 osd_map(82..82 src has 1..82) 2024-06-11T19:11:57.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.503+0000 7ff8897fa640 3 client.30907.objecter handle_osd_map got epochs [82,82] > 0 2024-06-11T19:11:57.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.503+0000 7ff8897fa640 3 client.30907.objecter handle_osd_map decoding full epoch 82 2024-06-11T19:11:57.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.503+0000 7ff8909bc640 1 -- 172.21.15.98:0/8906109 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff88c135da0 con 0x7ff88c1ce620 2024-06-11T19:11:57.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.503+0000 7ff8897fa640 20 client.30907.objecter dump_active .. 0 homeless 2024-06-11T19:11:57.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.504+0000 7ff8909bc640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c1ce620 0x7ff88c02cb40 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7ff880006c80 tx=0x7ff88000ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.504+0000 7ff8897fa640 10 client.30907.objecter ms_handle_connect 0x7ff88c1ce620 2024-06-11T19:11:57.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.504+0000 7ff8897fa640 10 client.30907.objecter resend_mon_ops 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.504+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff880017070 con 0x7ff88c1ce620 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.504+0000 7ff8897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.504+0000 7ff8897fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.504+0000 7ff8897fa640 10 monclient: dump: 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.522 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.505+0000 7ff88bfff640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff87003fdb0 0x7ff870042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.505+0000 7ff88bfff640 10 monclient: get_auth_request con 0x7ff87003fdb0 auth_method 0 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.505+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff88000ee50 con 0x7ff88c1ce620 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.505+0000 7ff8897fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.505+0000 7ff8897fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:57.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.505+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c1ce620 msgr2=0x7ff88c02cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.505+0000 7ff8897fa640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c1ce620 0x7ff88c02cb40 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7ff880006c80 tx=0x7ff88000ec00 comp rx=0 tx=0).stop 2024-06-11T19:11:57.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.505+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c1ce620 msgr2=0x7ff88c02cb40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:57.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.505+0000 7ff8897fa640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c1ce620 0x7ff88c02cb40 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.508+0000 7ff8897fa640 10 client.30907.objecter ms_handle_connect 0x7ff87003fdb0 2024-06-11T19:11:57.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.508+0000 7ff88bfff640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff87003fdb0 0x7ff870042270 secure :-1 s=READY pgs=4004 cs=0 l=1 rev1=1 crypto rx=0x7ff878034000 tx=0x7ff87802f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.518+0000 7ff892c20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:57.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.518+0000 7ff892c20640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c09dbd0 0x7ff88c02d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.518+0000 7ff892c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.518+0000 7ff892c20640 1 -- 172.21.15.98:0/8906109 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c050830 con 0x7ff88c09dbd0 2024-06-11T19:11:57.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.518+0000 7ff8909bc640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c09dbd0 0x7ff88c02d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.518+0000 7ff8909bc640 10 monclient: get_auth_request con 0x7ff88c09dbd0 auth_method 0 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.518+0000 7ff8909bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.518+0000 7ff8909bc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.518+0000 7ff8909bc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.519+0000 7ff8909bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.519+0000 7ff8909bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.519+0000 7ff8909bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.519+0000 7ff8909bc640 10 monclient(hunting): handle_auth_done global_id 30925 payload 824 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.519+0000 7ff8909bc640 1 -- 172.21.15.98:0/8906109 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff88c0a6c00 con 0x7ff88c09dbd0 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.520+0000 7ff8909bc640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c09dbd0 0x7ff88c02d080 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7ff880000c00 tx=0x7ff880004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.520+0000 7ff8897fa640 10 client.30907.objecter ms_handle_connect 0x7ff88c09dbd0 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.520+0000 7ff8897fa640 10 client.30907.objecter resend_mon_ops 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.521+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff880017070 con 0x7ff88c09dbd0 2024-06-11T19:11:57.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.521+0000 7ff8897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.521+0000 7ff8897fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.521+0000 7ff8897fa640 10 monclient: dump: 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.528 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.529 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.521+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9253 (secure 0 0 0) 0x7ff88000ce10 con 0x7ff88c09dbd0 2024-06-11T19:11:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.521+0000 7ff8897fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.521+0000 7ff8897fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.521+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c09dbd0 msgr2=0x7ff88c02d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.521+0000 7ff8897fa640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c09dbd0 0x7ff88c02d080 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7ff880000c00 tx=0x7ff880004730 comp rx=0 tx=0).stop 2024-06-11T19:11:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.521+0000 7ff8897fa640 1 -- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c09dbd0 msgr2=0x7ff88c02d080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.521+0000 7ff8897fa640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c09dbd0 0x7ff88c02d080 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.530 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:57.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:57.530 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:57.530 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:11:57.530 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 18, 2024-06-11T19:11:57.530 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 167739, 2024-06-11T19:11:57.530 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:57.530 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:57.530 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:57.530 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.531 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.535 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:57.538 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:57.539 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.542 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.544 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.550 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:57.550 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:57.550 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.550 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.550 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:57.550 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.550 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:57.551 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.553 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:57.553 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 10 librados: watch_flush enter 2024-06-11T19:11:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 10 librados: watch_flush exit 2024-06-11T19:11:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 20 client.30907.objecter shutdown clearing up homeless session... 2024-06-11T19:11:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 10 client.30907.objecter successfully canceled tick 2024-06-11T19:11:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 1 -- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff87003fdb0 msgr2=0x7ff870042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff87003fdb0 0x7ff870042270 secure :-1 s=READY pgs=4004 cs=0 l=1 rev1=1 crypto rx=0x7ff878034000 tx=0x7ff87802f000 comp rx=0 tx=0).stop 2024-06-11T19:11:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 10 monclient: shutdown 2024-06-11T19:11:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 1 -- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c133330 msgr2=0x7ff88c1cd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c133330 0x7ff88c1cd920 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7ff884009e10 tx=0x7ff88400c490 comp rx=0 tx=0).stop 2024-06-11T19:11:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff8911bd640 1 -- 172.21.15.98:0/8906109 reap_dead start 2024-06-11T19:11:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 1 -- 172.21.15.98:0/8906109 shutdown_connections 2024-06-11T19:11:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 1 --2- 172.21.15.98:0/8906109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff88c133330 0x7ff88c1cd920 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 1 -- 172.21.15.98:0/8906109 >> 172.21.15.98:0/8906109 conn(0x7ff88c0a7850 msgr2=0x7ff88c09fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 1 -- 172.21.15.98:0/8906109 shutdown_connections 2024-06-11T19:11:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 1 -- 172.21.15.98:0/8906109 wait complete. 2024-06-11T19:11:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.522+0000 7ff892c20640 1 librados: shutdown 2024-06-11T19:11:57.556 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:11:57.556 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon g 2024-06-11T19:11:57.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:11:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.663+0000 7f5900da8640 10 monclient: get_monmap_and_config 2024-06-11T19:11:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.663+0000 7f5900da8640 10 monclient: build_initial_monmap 2024-06-11T19:11:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.663+0000 7f5900da8640 10 monclient: monmap: 2024-06-11T19:11:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:57.664138+0000 2024-06-11T19:11:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:57.664138+0000 2024-06-11T19:11:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:57.664 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.663+0000 7f5900da8640 1 Processor -- start 2024-06-11T19:11:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.663+0000 7f5900da8640 1 -- start start 2024-06-11T19:11:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.663+0000 7f5900da8640 10 monclient: init 2024-06-11T19:11:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 10 monclient: _add_conns ranks=[7,0,4,9,6,2,5,1,10,3,8] 2024-06-11T19:11:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58fc0a2f40 0x7f58fc0a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 10 monclient(hunting): picked mon.noname-i con 0x7f58fc0a2f40 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58fc0a4df0 addr v1:172.21.15.98:6789/0 2024-06-11T19:11:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f58fc0a51d0 0x7f58fc136de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 10 monclient(hunting): picked mon.noname-c con 0x7f58fc0a51d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58fc0a0ed0 con 0x7f58fc0a4df0 2024-06-11T19:11:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f58fc0a3320 con 0x7f58fc0a51d0 2024-06-11T19:11:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f58fc0a2460 con 0x7f58fc0a2f40 2024-06-11T19:11:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f5900da8640 10 monclient(hunting): authenticate will time out at 927.471347s 2024-06-11T19:11:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58f1d9b640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f58fc0a4df0 legacy=0x7f58fc1366d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-06-11T19:11:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f58fc0a51d0 0x7f58fc136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f58fc0a51d0 0x7f58fc136de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55724/0 (socket says 172.21.15.98:55724) 2024-06-11T19:11:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 1 -- 172.21.15.98:0/2291768375 learned_addr learned my addr 172.21.15.98:0/2291768375 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 10 monclient(hunting): get_auth_request con 0x7f58fc0a51d0 auth_method 0 2024-06-11T19:11:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fa59c640 1 --2- 172.21.15.98:0/2291768375 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58fc0a2f40 0x7f58fc0a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fa59c640 10 monclient(hunting): get_auth_request con 0x7f58fc0a2f40 auth_method 0 2024-06-11T19:11:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fa59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 10 monclient(hunting): handle_auth_done global_id 22990 payload 274 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 1 -- 172.21.15.98:0/2291768375 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58fc0a2f40 msgr2=0x7f58fc0a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 1 --2- 172.21.15.98:0/2291768375 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58fc0a2f40 0x7f58fc0a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 1 -- 172.21.15.98:0/2291768375 >> v1:172.21.15.98:6789/0 conn(0x7f58fc0a4df0 legacy=0x7f58fc1366d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 1 monclient: found mon.noname-c 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.664+0000 7f58fad9d640 1 -- 172.21.15.98:0/2291768375 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58fc0a2180 con 0x7f58fc0a51d0 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58fad9d640 1 --2- 172.21.15.98:0/2291768375 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f58fc0a51d0 0x7f58fc136de0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009eb0 tx=0x7f58f402bd70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=d1fccaff0ef69298 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 1 -- 172.21.15.98:0/2291768375 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f58f402cb80 con 0x7f58fc0a51d0 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 10 monclient: dump: 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 10 monclient: _finish_auth 0 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 1 -- 172.21.15.98:0/2291768375 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58f402cd40 con 0x7f58fc0a51d0 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f5900da8640 5 monclient: authenticate success, global_id 22990 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f5900da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f5900da8640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 1 -- 172.21.15.98:0/2291768375 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f58f4034860 con 0x7f58fc0a51d0 2024-06-11T19:11:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f5900da8640 4 set_mon_vals no callback set 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.665+0000 7f58f9d9b640 10 monclient: dump: 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.666+0000 7f5900da8640 10 monclient: shutdown 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.666+0000 7f5900da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.666+0000 7f5900da8640 1 -- 172.21.15.98:0/2291768375 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f58fc0a51d0 msgr2=0x7f58fc136de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.666+0000 7f5900da8640 1 --2- 172.21.15.98:0/2291768375 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f58fc0a51d0 0x7f58fc136de0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009eb0 tx=0x7f58f402bd70 comp rx=0 tx=0).stop 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.666+0000 7f5900da8640 1 -- 172.21.15.98:0/2291768375 shutdown_connections 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.666+0000 7f5900da8640 1 --2- 172.21.15.98:0/2291768375 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f58fc0a51d0 0x7f58fc136de0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.666+0000 7f5900da8640 1 --2- 172.21.15.98:0/2291768375 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58fc0a2f40 0x7f58fc0a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.666+0000 7f5900da8640 1 -- 172.21.15.98:0/2291768375 >> 172.21.15.98:0/2291768375 conn(0x7f58fc0a95b0 msgr2=0x7f58fc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.666+0000 7f5900da8640 1 -- 172.21.15.98:0/2291768375 shutdown_connections 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.666+0000 7f5900da8640 1 -- 172.21.15.98:0/2291768375 wait complete. 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.666+0000 7f5900da8640 10 monclient: build_initial_monmap 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.666+0000 7f5900da8640 10 monclient: monmap: 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 1 librados: starting msgr at 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 1 librados: starting objecter 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 1 Processor -- start 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 1 -- start start 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 1 librados: setting wanted keys 2024-06-11T19:11:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 1 librados: calling monclient init 2024-06-11T19:11:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 10 monclient: init 2024-06-11T19:11:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 10 monclient: _add_conns ranks=[3,0,1,4,5,8,2,7,6] 2024-06-11T19:11:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc0a2f40 0x7f58fc1cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 10 monclient(hunting): picked mon.noname-g con 0x7f58fc0a2f40 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc0a4df0 0x7f58fc1cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 10 monclient(hunting): picked mon.noname-a con 0x7f58fc0a4df0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58fc0a51d0 0x7f58fc121200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 10 monclient(hunting): picked mon.noname-f con 0x7f58fc0a51d0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:11:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc0a1250 con 0x7f58fc0a4df0 2024-06-11T19:11:57.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc137320 con 0x7f58fc0a51d0 2024-06-11T19:11:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f58fc122d90 con 0x7f58fc0a2f40 2024-06-11T19:11:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f5900da8640 10 monclient(hunting): authenticate will time out at 927.474807s 2024-06-11T19:11:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f58f1d9b640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc0a4df0 msgr2=0x7f58fc1cbbd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-06-11T19:11:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.667+0000 7f58f1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc0a4df0 0x7f58fc1cbbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc0a2f40 0x7f58fc1cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58fc0a51d0 0x7f58fc121200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc0a2f40 0x7f58fc1cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:57966/0 (socket says 172.21.15.98:57966) 2024-06-11T19:11:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 1 -- 172.21.15.98:0/1309221764 learned_addr learned my addr 172.21.15.98:0/1309221764 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 10 monclient(hunting): get_auth_request con 0x7f58fc0a2f40 auth_method 0 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fad9d640 10 monclient(hunting): get_auth_request con 0x7f58fc0a51d0 auth_method 0 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 10 monclient(hunting): handle_auth_done global_id 21729 payload 824 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 1 -- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58fc0a51d0 msgr2=0x7f58fc121200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58fc0a51d0 0x7f58fc121200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 1 -- 172.21.15.98:0/1309221764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc0a4df0 msgr2=0x7f58fc1cbbd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58fc0a4df0 0x7f58fc1cbbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 10 monclient: _finish_hunting 0 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 1 monclient: found mon.noname-g 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 1 -- 172.21.15.98:0/1309221764 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58fc1247c0 con 0x7f58fc0a2f40 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fad9d640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58fc0a51d0 0x7f58fc121200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58fa59c640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc0a2f40 0x7f58fc1cb690 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f58ec00d800 tx=0x7f58ec00dcc0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 10 client.?.objecter ms_handle_connect 0x7f58fc0a2f40 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f58ec00bb30 con 0x7f58fc0a2f40 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 10 monclient: dump: 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 10 monclient: _finish_auth 0 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 10 monclient: _check_auth_tickets 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58ec019070 con 0x7f58fc0a2f40 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f5900da8640 5 monclient: authenticate success, global_id 21729 2024-06-11T19:11:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f58ec00bd30 con 0x7f58fc0a2f40 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f959a640 4 set_mon_vals no callback set 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f58f37fe640 10 monclient: dump: 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.691 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f5900da8640 10 monclient: _renew_subs 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f5900da8640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f5900da8640 1 -- 172.21.15.98:0/1309221764 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58fc125060 con 0x7f58fc0a2f40 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f5900da8640 10 client.21729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f5900da8640 10 monclient: _renew_subs 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f5900da8640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:11:57.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f5900da8640 1 -- 172.21.15.98:0/1309221764 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f58fc09f520 con 0x7f58fc0a2f40 2024-06-11T19:11:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.668+0000 7f5900da8640 1 librados: init done 2024-06-11T19:11:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f5900da8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f5900da8640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc124a00 0x7f58fc121740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f5900da8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f5900da8640 1 -- 172.21.15.98:0/1309221764 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f58fc069750 con 0x7f58fc124a00 2024-06-11T19:11:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 <== mon.3 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f58ec00b6c0 con 0x7f58fc0a2f40 2024-06-11T19:11:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f58f1d9b640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc124a00 0x7f58fc121740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f58f1d9b640 10 monclient: get_auth_request con 0x7f58fc124a00 auth_method 0 2024-06-11T19:11:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f58f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f58f1d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f58f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f58f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f58f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f58f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.669+0000 7f58f37fe640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f58dc03fdb0 0x7f58dc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 <== mon.3 v2:172.21.15.117:3301/0 5 ==== osd_map(82..82 src has 1..82) ==== 8529+0+0 (secure 0 0 0) 0x7f58ec00f040 con 0x7f58fc0a2f40 2024-06-11T19:11:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 10 client.21729.objecter ms_dispatch 0x7f58fc000b90 osd_map(82..82 src has 1..82) 2024-06-11T19:11:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 3 client.21729.objecter handle_osd_map got epochs [82,82] > 0 2024-06-11T19:11:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 3 client.21729.objecter handle_osd_map decoding full epoch 82 2024-06-11T19:11:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f1d9b640 10 monclient(hunting): handle_auth_done global_id 21738 payload 824 2024-06-11T19:11:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f1d9b640 1 -- 172.21.15.98:0/1309221764 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f58fc0a48b0 con 0x7f58fc124a00 2024-06-11T19:11:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 20 client.21729.objecter dump_active .. 0 homeless 2024-06-11T19:11:57.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58fad9d640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f58dc03fdb0 0x7f58dc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58fad9d640 10 monclient: get_auth_request con 0x7f58dc03fdb0 auth_method 0 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 10 client.21729.objecter ms_handle_connect 0x7f58fc124a00 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 10 client.21729.objecter resend_mon_ops 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f1d9b640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc124a00 0x7f58fc121740 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f58e4003980 tx=0x7f58e400b830 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f58e4016070 con 0x7f58fc124a00 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 10 monclient: dump: 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.695 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 10 client.21729.objecter ms_handle_connect 0x7f58dc03fdb0 2024-06-11T19:11:57.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58fad9d640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f58dc03fdb0 0x7f58dc042270 secure :-1 s=READY pgs=4005 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009eb0 tx=0x7f58f4030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f58e4003bc0 con 0x7f58fc124a00 2024-06-11T19:11:57.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:57.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:57.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc124a00 msgr2=0x7f58fc121740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc124a00 0x7f58fc121740 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f58e4003980 tx=0x7f58e400b830 comp rx=0 tx=0).stop 2024-06-11T19:11:57.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc124a00 msgr2=0x7f58fc121740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:57.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.670+0000 7f58f37fe640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc124a00 0x7f58fc121740 secure :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f58e4003980 tx=0x7f58e400b830 comp rx=0 tx=0).stop 2024-06-11T19:11:57.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.676+0000 7f5900da8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:57.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.676+0000 7f5900da8640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc121c80 0x7f58fc122090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.676+0000 7f5900da8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.676+0000 7f5900da8640 1 -- 172.21.15.98:0/1309221764 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f58fc0a13d0 con 0x7f58fc121c80 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f1d9b640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc121c80 0x7f58fc122090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f1d9b640 10 monclient: get_auth_request con 0x7f58fc121c80 auth_method 0 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f1d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f1d9b640 10 monclient(hunting): handle_auth_done global_id 21747 payload 824 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f1d9b640 1 -- 172.21.15.98:0/1309221764 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f58fc1225d0 con 0x7f58fc121c80 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f37fe640 10 client.21729.objecter ms_handle_connect 0x7f58fc121c80 2024-06-11T19:11:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f37fe640 10 client.21729.objecter resend_mon_ops 2024-06-11T19:11:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f1d9b640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc121c80 0x7f58fc122090 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f58e400bfd0 tx=0x7f58e40037f0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f58e4016040 con 0x7f58fc121c80 2024-06-11T19:11:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f37fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:11:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.677+0000 7f58f37fe640 10 monclient: dump: 2024-06-11T19:11:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9251 (secure 0 0 0) 0x7f58e4011400 con 0x7f58fc121c80 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f58f37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f58f37fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc121c80 msgr2=0x7f58fc122090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f58f37fe640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc121c80 0x7f58fc122090 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f58e400bfd0 tx=0x7f58e40037f0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f58f37fe640 1 -- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc121c80 msgr2=0x7f58fc122090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f58f37fe640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc121c80 0x7f58fc122090 secure :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f58e400bfd0 tx=0x7f58e40037f0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.703 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 18, 2024-06-11T19:11:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 167894, 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:57.714 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.714 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:57.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:57.714 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.714 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:57.716 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.716 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.716 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:57.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:57.716 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.716 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.716 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.718 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:57.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:57.721 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:57.722 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:57.722 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.722 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:57.724 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:57.725 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.726 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.727 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:57.727 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:57.727 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:57.727 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.728 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.729 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.730 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2024-06-11T19:11:57.730 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.730 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.730 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:57.730 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 10 librados: watch_flush enter 2024-06-11T19:11:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 10 librados: watch_flush exit 2024-06-11T19:11:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 20 client.21729.objecter shutdown clearing up homeless session... 2024-06-11T19:11:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 10 client.21729.objecter successfully canceled tick 2024-06-11T19:11:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 1 -- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f58dc03fdb0 msgr2=0x7f58dc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f58dc03fdb0 0x7f58dc042270 secure :-1 s=READY pgs=4005 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009eb0 tx=0x7f58f4030000 comp rx=0 tx=0).stop 2024-06-11T19:11:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 10 monclient: shutdown 2024-06-11T19:11:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 1 -- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc0a2f40 msgr2=0x7f58fc1cb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 1 --2- 172.21.15.98:0/1309221764 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58fc0a2f40 0x7f58fc1cb690 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f58ec00d800 tx=0x7f58ec00dcc0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f58fad9d640 1 -- 172.21.15.98:0/1309221764 reap_dead start 2024-06-11T19:11:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f58fad9d640 1 -- 172.21.15.98:0/1309221764 reap_dead start 2024-06-11T19:11:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 1 -- 172.21.15.98:0/1309221764 shutdown_connections 2024-06-11T19:11:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 1 -- 172.21.15.98:0/1309221764 >> 172.21.15.98:0/1309221764 conn(0x7f58fc0a95b0 msgr2=0x7f58fc09ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 1 -- 172.21.15.98:0/1309221764 shutdown_connections 2024-06-11T19:11:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 1 -- 172.21.15.98:0/1309221764 wait complete. 2024-06-11T19:11:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.678+0000 7f5900da8640 1 librados: shutdown 2024-06-11T19:11:57.733 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:11:57.734 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon h 2024-06-11T19:11:57.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:11:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.833+0000 7fd25000a640 10 monclient: get_monmap_and_config 2024-06-11T19:11:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.833+0000 7fd25000a640 10 monclient: build_initial_monmap 2024-06-11T19:11:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.833+0000 7fd25000a640 10 monclient: monmap: 2024-06-11T19:11:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:57.834598+0000 2024-06-11T19:11:57.833 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:57.834598+0000 2024-06-11T19:11:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:57.834 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:57.835 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:57.835 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:57.835 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 1 Processor -- start 2024-06-11T19:11:57.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 1 -- start start 2024-06-11T19:11:57.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 10 monclient: init 2024-06-11T19:11:57.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:57.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 10 monclient: _add_conns ranks=[4,0,2,1,3,5,7,10,9,6,8] 2024-06-11T19:11:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 0x7fd2480a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 10 monclient(hunting): picked mon.noname-c con 0x7fd2480a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2480a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:11:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2480a2530 0x7fd2480a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 10 monclient(hunting): picked mon.noname-b con 0x7fd2480a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2480a7140 con 0x7fd2480a5240 2024-06-11T19:11:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd2480a2b50 con 0x7fd2480a2530 2024-06-11T19:11:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd2480a2e30 con 0x7fd2480a4920 2024-06-11T19:11:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd25000a640 10 monclient(hunting): authenticate will time out at 927.641789s 2024-06-11T19:11:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24d5a5640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fd2480a5240 legacy=0x7fd2480a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-06-11T19:11:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24dda6640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 0x7fd2480a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24e5a7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2480a2530 0x7fd2480a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24e5a7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2480a2530 0x7fd2480a60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:37382/0 (socket says 172.21.15.98:37382) 2024-06-11T19:11:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24dda6640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 0x7fd2480a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55734/0 (socket says 172.21.15.98:55734) 2024-06-11T19:11:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24e5a7640 1 -- 172.21.15.98:0/2498393983 learned_addr learned my addr 172.21.15.98:0/2498393983 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24e5a7640 10 monclient(hunting): get_auth_request con 0x7fd2480a2530 auth_method 0 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24e5a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24e5a7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24e5a7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24dda6640 10 monclient(hunting): get_auth_request con 0x7fd2480a4920 auth_method 0 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24dda6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24dda6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.834+0000 7fd24dda6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24e5a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24e5a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24e5a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 10 monclient(hunting): handle_auth_done global_id 22999 payload 274 2024-06-11T19:11:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 1 -- 172.21.15.98:0/2498393983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2480a2530 msgr2=0x7fd2480a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 1 --2- 172.21.15.98:0/2498393983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2480a2530 0x7fd2480a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 1 -- 172.21.15.98:0/2498393983 >> v1:172.21.15.98:6789/0 conn(0x7fd2480a5240 legacy=0x7fd2480a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:11:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 10 monclient: _finish_hunting 0 2024-06-11T19:11:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 1 monclient: found mon.noname-c 2024-06-11T19:11:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:11:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 1 -- 172.21.15.98:0/2498393983 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2480a3110 con 0x7fd2480a4920 2024-06-11T19:11:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24e5a7640 1 --2- 172.21.15.98:0/2498393983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2480a2530 0x7fd2480a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:11:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.835+0000 7fd24dda6640 1 --2- 172.21.15.98:0/2498393983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 0x7fd2480a4d00 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fd2440097d0 tx=0x7fd24402b6d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ebe755b011bb98fd server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 1 -- 172.21.15.98:0/2498393983 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd244037070 con 0x7fd2480a4920 2024-06-11T19:11:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 10 monclient: dump: 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.854 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.855 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 10 monclient: _finish_auth 0 2024-06-11T19:11:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 10 monclient: _check_auth_tickets 2024-06-11T19:11:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 5 monclient: authenticate success, global_id 22999 2024-06-11T19:11:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 1 -- 172.21.15.98:0/2498393983 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd244030980 con 0x7fd2480a4920 2024-06-11T19:11:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:57.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 1 -- 172.21.15.98:0/2498393983 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd244030bb0 con 0x7fd2480a4920 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 4 set_mon_vals no callback set 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd24cda4640 10 monclient: dump: 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.856 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.857 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.857 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.857 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.857 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.857 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.857 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.857 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.857 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.857 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 10 monclient: shutdown 2024-06-11T19:11:57.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:57.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 1 -- 172.21.15.98:0/2498393983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 msgr2=0x7fd2480a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 1 --2- 172.21.15.98:0/2498393983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 0x7fd2480a4d00 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fd2440097d0 tx=0x7fd24402b6d0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 1 -- 172.21.15.98:0/2498393983 shutdown_connections 2024-06-11T19:11:57.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 1 --2- 172.21.15.98:0/2498393983 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2480a2530 0x7fd2480a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 1 --2- 172.21.15.98:0/2498393983 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 0x7fd2480a4d00 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 1 -- 172.21.15.98:0/2498393983 >> 172.21.15.98:0/2498393983 conn(0x7fd2480ab850 msgr2=0x7fd2480a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:57.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 1 -- 172.21.15.98:0/2498393983 shutdown_connections 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.836+0000 7fd25000a640 1 -- 172.21.15.98:0/2498393983 wait complete. 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.837+0000 7fd25000a640 10 monclient: build_initial_monmap 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.837+0000 7fd25000a640 10 monclient: monmap: 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:57.859 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.837+0000 7fd25000a640 1 librados: starting msgr at 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.837+0000 7fd25000a640 1 librados: starting objecter 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.837+0000 7fd25000a640 1 Processor -- start 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.837+0000 7fd25000a640 1 -- start start 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.837+0000 7fd25000a640 1 librados: setting wanted keys 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.837+0000 7fd25000a640 1 librados: calling monclient init 2024-06-11T19:11:57.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.837+0000 7fd25000a640 10 monclient: init 2024-06-11T19:11:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient: _add_conns ranks=[1,4,2,0,7,3,6,5,8] 2024-06-11T19:11:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2480a2530 0x7fd248136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient(hunting): picked mon.noname-f con 0x7fd2480a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:11:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 0x7fd248137410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient(hunting): picked mon.noname-c con 0x7fd2480a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2480a5240 0x7fd2481228f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient(hunting): picked mon.noname-b con 0x7fd2480a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:11:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd2480a6da0 con 0x7fd2480a2530 2024-06-11T19:11:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd248138e50 con 0x7fd2480a5240 2024-06-11T19:11:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd2480a65f0 con 0x7fd2480a4920 2024-06-11T19:11:57.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient(hunting): authenticate will time out at 927.645114s 2024-06-11T19:11:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 0x7fd248137410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24e5a7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2480a5240 0x7fd2481228f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 0x7fd248137410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55740/0 (socket says 172.21.15.98:55740) 2024-06-11T19:11:57.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 1 -- 172.21.15.98:0/4259827434 learned_addr learned my addr 172.21.15.98:0/4259827434 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 10 monclient(hunting): get_auth_request con 0x7fd2480a4920 auth_method 0 2024-06-11T19:11:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24e5a7640 10 monclient(hunting): get_auth_request con 0x7fd2480a5240 auth_method 0 2024-06-11T19:11:57.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24e5a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24e5a7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24e5a7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24e5a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24e5a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24e5a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24dda6640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2480a2530 0x7fd248136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24dda6640 10 monclient(hunting): get_auth_request con 0x7fd2480a2530 auth_method 0 2024-06-11T19:11:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24dda6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24dda6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24dda6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 10 monclient(hunting): handle_auth_done global_id 23008 payload 824 2024-06-11T19:11:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 1 -- 172.21.15.98:0/4259827434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2480a5240 msgr2=0x7fd2481228f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd2480a5240 0x7fd2481228f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 1 -- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2480a2530 msgr2=0x7fd248136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2480a2530 0x7fd248136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 10 monclient: _finish_hunting 0 2024-06-11T19:11:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 1 monclient: found mon.noname-c 2024-06-11T19:11:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:11:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 1 -- 172.21.15.98:0/4259827434 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2481262f0 con 0x7fd2480a4920 2024-06-11T19:11:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24dda6640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2480a2530 0x7fd248136ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:11:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd24d5a5640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 0x7fd248137410 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fd23800b450 tx=0x7fd23800b910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 10 client.?.objecter ms_handle_connect 0x7fd2480a4920 2024-06-11T19:11:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd238003bc0 con 0x7fd2480a4920 2024-06-11T19:11:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:11:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 10 monclient: dump: 2024-06-11T19:11:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.871 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 10 monclient: _finish_auth 0 2024-06-11T19:11:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:11:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 5 monclient: authenticate success, global_id 23008 2024-06-11T19:11:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd238015070 con 0x7fd2480a4920 2024-06-11T19:11:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd238003da0 con 0x7fd2480a4920 2024-06-11T19:11:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23ffff640 4 set_mon_vals no callback set 2024-06-11T19:11:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd23e7fc640 10 monclient: dump: 2024-06-11T19:11:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.876 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient: _renew_subs 2024-06-11T19:11:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:11:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 1 -- 172.21.15.98:0/4259827434 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2481c0c40 con 0x7fd2480a4920 2024-06-11T19:11:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 client.23008.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient: _renew_subs 2024-06-11T19:11:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:11:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 1 -- 172.21.15.98:0/4259827434 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fd248124e70 con 0x7fd2480a4920 2024-06-11T19:11:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.838+0000 7fd25000a640 1 librados: init done 2024-06-11T19:11:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.839+0000 7fd25000a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.839+0000 7fd25000a640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248122e30 0x7fd248123210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.839+0000 7fd25000a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.839+0000 7fd25000a640 1 -- 172.21.15.98:0/4259827434 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd248125190 con 0x7fd248122e30 2024-06-11T19:11:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.839+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd23800f070 con 0x7fd2480a4920 2024-06-11T19:11:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24dda6640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248122e30 0x7fd248123210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd22c03fde0 0x7fd22c0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24dda6640 10 monclient: get_auth_request con 0x7fd248122e30 auth_method 0 2024-06-11T19:11:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24dda6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24dda6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24dda6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(82..82 src has 1..82) ==== 8529+0+0 (secure 0 0 0) 0x7fd238051b60 con 0x7fd2480a4920 2024-06-11T19:11:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 10 client.23008.objecter ms_dispatch 0x7fd248000b90 osd_map(82..82 src has 1..82) 2024-06-11T19:11:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 3 client.23008.objecter handle_osd_map got epochs [82,82] > 0 2024-06-11T19:11:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 3 client.23008.objecter handle_osd_map decoding full epoch 82 2024-06-11T19:11:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24dda6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24dda6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24dda6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 20 client.23008.objecter dump_active .. 0 homeless 2024-06-11T19:11:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24e5a7640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd22c03fde0 0x7fd22c0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24e5a7640 10 monclient: get_auth_request con 0x7fd22c03fde0 auth_method 0 2024-06-11T19:11:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24dda6640 10 monclient(hunting): handle_auth_done global_id 22145 payload 824 2024-06-11T19:11:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24dda6640 1 -- 172.21.15.98:0/4259827434 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2481245c0 con 0x7fd248122e30 2024-06-11T19:11:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 10 client.23008.objecter ms_handle_connect 0x7fd248122e30 2024-06-11T19:11:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 10 client.23008.objecter resend_mon_ops 2024-06-11T19:11:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24dda6640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248122e30 0x7fd248123210 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd2480a8420 tx=0x7fd2440046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 10 client.23008.objecter ms_handle_connect 0x7fd22c03fde0 2024-06-11T19:11:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd24e5a7640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd22c03fde0 0x7fd22c0422a0 secure :-1 s=READY pgs=4006 cs=0 l=1 rev1=1 crypto rx=0x7fd240009c90 tx=0x7fd24000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd244037040 con 0x7fd248122e30 2024-06-11T19:11:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:11:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 10 monclient: dump: 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd244030eb0 con 0x7fd248122e30 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248122e30 msgr2=0x7fd248123210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248122e30 0x7fd248123210 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd2480a8420 tx=0x7fd2440046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248122e30 msgr2=0x7fd248123210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.840+0000 7fd23e7fc640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248122e30 0x7fd248123210 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd25000a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd25000a640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248123750 0x7fd24809e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd25000a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd25000a640 1 -- 172.21.15.98:0/4259827434 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd248050830 con 0x7fd248123750 2024-06-11T19:11:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd24dda6640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248123750 0x7fd24809e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd24dda6640 10 monclient: get_auth_request con 0x7fd248123750 auth_method 0 2024-06-11T19:11:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd24dda6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd24dda6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd24dda6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd24dda6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd24dda6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd24dda6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd24dda6640 10 monclient(hunting): handle_auth_done global_id 22154 payload 824 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd24dda6640 1 -- 172.21.15.98:0/4259827434 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd2480a4d00 con 0x7fd248123750 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd23e7fc640 10 client.23008.objecter ms_handle_connect 0x7fd248123750 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd23e7fc640 10 client.23008.objecter resend_mon_ops 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd24dda6640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248123750 0x7fd24809e9f0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fd24402b710 tx=0x7fd244030000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd244006e10 con 0x7fd248123750 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.847+0000 7fd23e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd23e7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd23e7fc640 10 monclient: dump: 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9591 (secure 0 0 0) 0x7fd244003e00 con 0x7fd248123750 2024-06-11T19:11:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd23e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd23e7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248123750 msgr2=0x7fd24809e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd23e7fc640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248123750 0x7fd24809e9f0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fd24402b710 tx=0x7fd244030000 comp rx=0 tx=0).stop 2024-06-11T19:11:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd23e7fc640 1 -- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248123750 msgr2=0x7fd24809e9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd23e7fc640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd248123750 0x7fd24809e9f0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:57.885 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:57.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:57.885 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:57.886 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:57.886 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 18, 2024-06-11T19:11:57.886 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 168042, 2024-06-11T19:11:57.886 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:57.886 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:57.886 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:57.886 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:57.886 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:57.886 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:57.886 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:57.886 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:11:57.886 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.887 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.888 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:11:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:11:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:57.910 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 10 librados: watch_flush enter 2024-06-11T19:11:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 10 librados: watch_flush exit 2024-06-11T19:11:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 20 client.23008.objecter shutdown clearing up homeless session... 2024-06-11T19:11:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 10 client.23008.objecter successfully canceled tick 2024-06-11T19:11:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 1 -- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd22c03fde0 msgr2=0x7fd22c0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd22c03fde0 0x7fd22c0422a0 secure :-1 s=READY pgs=4006 cs=0 l=1 rev1=1 crypto rx=0x7fd240009c90 tx=0x7fd24000b040 comp rx=0 tx=0).stop 2024-06-11T19:11:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 10 monclient: shutdown 2024-06-11T19:11:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 1 -- 172.21.15.98:0/4259827434 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 msgr2=0x7fd248137410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 1 --2- 172.21.15.98:0/4259827434 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd2480a4920 0x7fd248137410 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fd23800b450 tx=0x7fd23800b910 comp rx=0 tx=0).stop 2024-06-11T19:11:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd24e5a7640 1 -- 172.21.15.98:0/4259827434 reap_dead start 2024-06-11T19:11:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd24e5a7640 1 -- 172.21.15.98:0/4259827434 reap_dead start 2024-06-11T19:11:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 1 -- 172.21.15.98:0/4259827434 shutdown_connections 2024-06-11T19:11:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 1 -- 172.21.15.98:0/4259827434 >> 172.21.15.98:0/4259827434 conn(0x7fd2480ab850 msgr2=0x7fd2480726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 1 -- 172.21.15.98:0/4259827434 shutdown_connections 2024-06-11T19:11:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 1 -- 172.21.15.98:0/4259827434 wait complete. 2024-06-11T19:11:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:57.848+0000 7fd25000a640 1 librados: shutdown 2024-06-11T19:11:57.914 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:11:57.914 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon i 2024-06-11T19:11:57.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:11:58.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.013+0000 7fa3bfcca640 10 monclient: get_monmap_and_config 2024-06-11T19:11:58.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.013+0000 7fa3bfcca640 10 monclient: build_initial_monmap 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.013+0000 7fa3bfcca640 10 monclient: monmap: 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:11:58.014969+0000 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:11:58.014969+0000 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:58.014 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:58.015 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:58.015 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:58.015 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:58.015 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:11:58.015 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:11:58.015 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:58.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.014+0000 7fa3bfcca640 1 Processor -- start 2024-06-11T19:11:58.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.014+0000 7fa3bfcca640 1 -- start start 2024-06-11T19:11:58.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.014+0000 7fa3bfcca640 10 monclient: init 2024-06-11T19:11:58.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:58.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 10 monclient: _add_conns ranks=[7,5,1,10,0,3,9,8,4,2,6] 2024-06-11T19:11:58.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b80a4920 0x7fa3b80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:58.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 10 monclient(hunting): picked mon.noname-i con 0x7fa3b80a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa3b80a5240 0x7fa3b80a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 10 monclient(hunting): picked mon.noname-h con 0x7fa3b80a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa3b80a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3b80a7140 con 0x7fa3b80a2530 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fa3b80a2b50 con 0x7fa3b80a5240 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa3b80a2e30 con 0x7fa3b80a4920 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bfcca640 10 monclient(hunting): authenticate will time out at 927.822217s 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b80a4920 0x7fa3b80a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bd265640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa3b80a5240 0x7fa3b80a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:58.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3be267640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fa3b80a2530 0x7fa3b80a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:59818/0 (socket says 172.21.15.98:59818) 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3be267640 1 -- 172.21.15.98:0/530085222 learned_addr learned my addr 172.21.15.98:0/530085222 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 10 monclient(hunting): get_auth_request con 0x7fa3b80a4920 auth_method 0 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bd265640 10 monclient(hunting): get_auth_request con 0x7fa3b80a5240 auth_method 0 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bd265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bd265640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bd265640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bd265640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:58.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bd265640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:58.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bd265640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:58.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 10 monclient(hunting): handle_auth_done global_id 22084 payload 274 2024-06-11T19:11:58.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 1 -- 172.21.15.98:0/530085222 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa3b80a5240 msgr2=0x7fa3b80a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:11:58.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 1 --2- 172.21.15.98:0/530085222 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa3b80a5240 0x7fa3b80a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:58.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 1 -- 172.21.15.98:0/530085222 >> v1:172.21.15.117:6789/0 conn(0x7fa3b80a2530 legacy=0x7fa3b80a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:58.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 10 monclient: _finish_hunting 0 2024-06-11T19:11:58.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 1 monclient: found mon.noname-i 2024-06-11T19:11:58.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bca64640 1 -- 172.21.15.98:0/530085222 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4293612600 0 0) 0x7fa3b80a7140 con 0x7fa3b80a2530 2024-06-11T19:11:58.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:58.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bda66640 1 -- 172.21.15.98:0/530085222 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b80a3110 con 0x7fa3b80a4920 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.015+0000 7fa3bca64640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bda66640 1 --2- 172.21.15.98:0/530085222 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b80a4920 0x7fa3b80a4d00 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00097d0 tx=0x7fa3b002b770 comp rx=0 tx=0).ready entity=mon.7 client_cookie=463af19f76afc53e server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 1 -- 172.21.15.98:0/530085222 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa3b0039070 con 0x7fa3b80a4920 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 10 monclient: dump: 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 10 monclient: _finish_auth 0 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 10 monclient: _check_auth_tickets 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 1 -- 172.21.15.98:0/530085222 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3b0030070 con 0x7fa3b80a4920 2024-06-11T19:11:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bfcca640 5 monclient: authenticate success, global_id 22084 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bfcca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 1 -- 172.21.15.98:0/530085222 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa3b0031db0 con 0x7fa3b80a4920 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bfcca640 10 monclient: get_monmap_and_config success 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bfcca640 4 set_mon_vals no callback set 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.016+0000 7fa3bca64640 10 monclient: dump: 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.017+0000 7fa3bfcca640 10 monclient: shutdown 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.017+0000 7fa3bfcca640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.017+0000 7fa3bfcca640 1 -- 172.21.15.98:0/530085222 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b80a4920 msgr2=0x7fa3b80a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.017+0000 7fa3bfcca640 1 --2- 172.21.15.98:0/530085222 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b80a4920 0x7fa3b80a4d00 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00097d0 tx=0x7fa3b002b770 comp rx=0 tx=0).stop 2024-06-11T19:11:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.017+0000 7fa3bfcca640 1 -- 172.21.15.98:0/530085222 shutdown_connections 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.017+0000 7fa3bfcca640 1 --2- 172.21.15.98:0/530085222 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa3b80a5240 0x7fa3b80a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.017+0000 7fa3bfcca640 1 --2- 172.21.15.98:0/530085222 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b80a4920 0x7fa3b80a4d00 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.017+0000 7fa3bfcca640 1 -- 172.21.15.98:0/530085222 >> 172.21.15.98:0/530085222 conn(0x7fa3b80ab850 msgr2=0x7fa3b80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.017+0000 7fa3bfcca640 1 -- 172.21.15.98:0/530085222 shutdown_connections 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.017+0000 7fa3bfcca640 1 -- 172.21.15.98:0/530085222 wait complete. 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.017+0000 7fa3bfcca640 10 monclient: build_initial_monmap 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.017+0000 7fa3bfcca640 10 monclient: monmap: 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:11:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 1 librados: starting msgr at 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 1 librados: starting objecter 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 1 Processor -- start 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 1 -- start start 2024-06-11T19:11:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 1 librados: setting wanted keys 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 1 librados: calling monclient init 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 10 monclient: init 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 10 monclient: _reopen_session rank -1 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 10 monclient: _add_conns ranks=[0,3,4,7,2,8,5,1,6] 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b80a2530 0x7fa3b81cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 10 monclient(hunting): picked mon.noname-a con 0x7fa3b80a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3b80a4920 0x7fa3b81ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 10 monclient(hunting): picked mon.noname-g con 0x7fa3b80a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa3b80a5240 0x7fa3b8125030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bda66640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b80a2530 msgr2=0x7fa3b81cdb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 10 monclient(hunting): picked mon.noname-c con 0x7fa3b80a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b80a6da0 con 0x7fa3b80a2530 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bda66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b80a2530 0x7fa3b81cdb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:11:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fa3b8138e50 con 0x7fa3b80a4920 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa3b80a65f0 con 0x7fa3b80a5240 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 10 monclient(hunting): _renew_subs 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bfcca640 10 monclient(hunting): authenticate will time out at 927.825738s 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3be267640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa3b80a5240 0x7fa3b8125030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3be267640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa3b80a5240 0x7fa3b8125030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55746/0 (socket says 172.21.15.98:55746) 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3be267640 1 -- 172.21.15.98:0/391076330 learned_addr learned my addr 172.21.15.98:0/391076330 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3be267640 10 monclient(hunting): get_auth_request con 0x7fa3b80a5240 auth_method 0 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3be267640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3be267640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3be267640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bd265640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3b80a4920 0x7fa3b81ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bd265640 10 monclient(hunting): get_auth_request con 0x7fa3b80a4920 auth_method 0 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bd265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bd265640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bd265640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3be267640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3be267640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3be267640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bd265640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bd265640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.018+0000 7fa3bd265640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3be267640 10 monclient(hunting): handle_auth_done global_id 23017 payload 824 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3be267640 1 -- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3b80a4920 msgr2=0x7fa3b81ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3be267640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3b80a4920 0x7fa3b81ce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3be267640 1 -- 172.21.15.98:0/391076330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b80a2530 msgr2=0x7fa3b81cdb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3be267640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3b80a2530 0x7fa3b81cdb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3be267640 10 monclient: _finish_hunting 0 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3be267640 1 monclient: found mon.noname-c 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3be267640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3be267640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3be267640 1 -- 172.21.15.98:0/391076330 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b80a8f10 con 0x7fa3b80a5240 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3be267640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa3b80a5240 0x7fa3b8125030 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fa3a800c3a0 tx=0x7fa3a800c860 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 10 client.?.objecter ms_handle_connect 0x7fa3b80a5240 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa3a80047e0 con 0x7fa3b80a5240 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 10 monclient: dump: 2024-06-11T19:11:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 10 monclient: _finish_auth 0 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 10 monclient: _check_auth_tickets 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a80049e0 con 0x7fa3b80a5240 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3bfcca640 5 monclient: authenticate success, global_id 23017 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa3a8004bd0 con 0x7fa3b80a5240 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3bfcca640 10 monclient: _renew_subs 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3bfcca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3bfcca640 1 -- 172.21.15.98:0/391076330 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3b80aabe0 con 0x7fa3b80a5240 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:11:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b67fc640 10 monclient: dump: 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3b7fff640 4 set_mon_vals no callback set 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3bfcca640 10 client.23017.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3bfcca640 10 monclient: _renew_subs 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3bfcca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3bfcca640 1 -- 172.21.15.98:0/391076330 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3b8128c40 con 0x7fa3b80a5240 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.019+0000 7fa3bfcca640 1 librados: init done 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bfcca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bfcca640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125570 0x7fa3b8125950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bfcca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bfcca640 1 -- 172.21.15.98:0/391076330 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa3b8050830 con 0x7fa3b8125570 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fa3a8011070 con 0x7fa3b80a5240 2024-06-11T19:11:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bda66640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125570 0x7fa3b8125950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bda66640 10 monclient: get_auth_request con 0x7fa3b8125570 auth_method 0 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bda66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bda66640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bda66640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bda66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bda66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bda66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3b67fc640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa3a403fdb0 0x7fa3a4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bda66640 10 monclient(hunting): handle_auth_done global_id 22093 payload 824 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bda66640 1 -- 172.21.15.98:0/391076330 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3b80a4d00 con 0x7fa3b8125570 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(82..82 src has 1..82) ==== 8529+0+0 (secure 0 0 0) 0x7fa3a8051e90 con 0x7fa3b80a5240 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3b67fc640 10 client.23017.objecter ms_dispatch 0x7fa3b8000b90 osd_map(82..82 src has 1..82) 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3b67fc640 3 client.23017.objecter handle_osd_map got epochs [82,82] > 0 2024-06-11T19:11:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3b67fc640 3 client.23017.objecter handle_osd_map decoding full epoch 82 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bd265640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa3a403fdb0 0x7fa3a4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.020+0000 7fa3bd265640 10 monclient: get_auth_request con 0x7fa3a403fdb0 auth_method 0 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 20 client.23017.objecter dump_active .. 0 homeless 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3bda66640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125570 0x7fa3b8125950 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80abf80 tx=0x7fa3b00046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 10 client.23017.objecter ms_handle_connect 0x7fa3b8125570 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 10 client.23017.objecter resend_mon_ops 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa3b0043030 con 0x7fa3b8125570 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 10 monclient: dump: 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 10 client.23017.objecter ms_handle_connect 0x7fa3a403fdb0 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3bd265640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa3a403fdb0 0x7fa3a4042270 secure :-1 s=READY pgs=4007 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8126790 tx=0x7fa3ac00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa3b0031940 con 0x7fa3b8125570 2024-06-11T19:11:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:11:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:11:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125570 msgr2=0x7fa3b8125950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125570 0x7fa3b8125950 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80abf80 tx=0x7fa3b00046c0 comp rx=0 tx=0).stop 2024-06-11T19:11:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125570 msgr2=0x7fa3b8125950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.021+0000 7fa3b67fc640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125570 0x7fa3b8125950 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bfcca640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:11:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bfcca640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125e90 0x7fa3b81215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:11:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bfcca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:11:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bfcca640 1 -- 172.21.15.98:0/391076330 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa3b8069740 con 0x7fa3b8125e90 2024-06-11T19:11:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bda66640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125e90 0x7fa3b81215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bda66640 10 monclient: get_auth_request con 0x7fa3b8125e90 auth_method 0 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bda66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bda66640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bda66640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bda66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bda66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bda66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bda66640 10 monclient(hunting): handle_auth_done global_id 22102 payload 824 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.028+0000 7fa3bda66640 1 -- 172.21.15.98:0/391076330 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa3b8128680 con 0x7fa3b8125e90 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 10 client.23017.objecter ms_handle_connect 0x7fa3b8125e90 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 10 client.23017.objecter resend_mon_ops 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bda66640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125e90 0x7fa3b81215f0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0000c00 tx=0x7fa3b0009190 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa3b0043040 con 0x7fa3b8125e90 2024-06-11T19:11:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:11:58.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:11:58.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 10 monclient: dump: 2024-06-11T19:11:58.050 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:11:58.050 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:11:58.050 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:58.050 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:11:58.050 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:11:58.050 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:11:58.050 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:11:58.050 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:11:58.050 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:11:58.050 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9421 (secure 0 0 0) 0x7fa3b0003b70 con 0x7fa3b8125e90 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125e90 msgr2=0x7fa3b81215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125e90 0x7fa3b81215f0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0000c00 tx=0x7fa3b0009190 comp rx=0 tx=0).stop 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 1 -- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125e90 msgr2=0x7fa3b81215f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:11:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3b67fc640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa3b8125e90 0x7fa3b81215f0 secure :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0000c00 tx=0x7fa3b0009190 comp rx=0 tx=0).stop 2024-06-11T19:11:58.052 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:11:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:11:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 18, 2024-06-11T19:11:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 168245, 2024-06-11T19:11:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:11:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:11:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:11:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:11:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:11:58.052 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:58.053 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:58.054 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:11:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:11:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:11:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:11:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:11:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:11:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:11:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:11:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:11:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:11:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:11:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:11:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:11:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:11:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:11:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:11:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:11:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:11:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:11:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:11:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:11:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:11:58.074 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:11:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 10 librados: watch_flush enter 2024-06-11T19:11:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 10 librados: watch_flush exit 2024-06-11T19:11:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 20 client.23017.objecter shutdown clearing up homeless session... 2024-06-11T19:11:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 10 client.23017.objecter successfully canceled tick 2024-06-11T19:11:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 1 -- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa3a403fdb0 msgr2=0x7fa3a4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa3a403fdb0 0x7fa3a4042270 secure :-1 s=READY pgs=4007 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8126790 tx=0x7fa3ac00b040 comp rx=0 tx=0).stop 2024-06-11T19:11:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 10 monclient: shutdown 2024-06-11T19:11:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:11:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 1 -- 172.21.15.98:0/391076330 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa3b80a5240 msgr2=0x7fa3b8125030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:11:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa3b80a5240 0x7fa3b8125030 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fa3a800c3a0 tx=0x7fa3a800c860 comp rx=0 tx=0).stop 2024-06-11T19:11:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3be267640 1 -- 172.21.15.98:0/391076330 reap_dead start 2024-06-11T19:11:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 1 -- 172.21.15.98:0/391076330 shutdown_connections 2024-06-11T19:11:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 1 --2- 172.21.15.98:0/391076330 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa3b80a5240 0x7fa3b8125030 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:11:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 1 -- 172.21.15.98:0/391076330 >> 172.21.15.98:0/391076330 conn(0x7fa3b80ab850 msgr2=0x7fa3b80a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:11:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 1 -- 172.21.15.98:0/391076330 shutdown_connections 2024-06-11T19:11:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 1 -- 172.21.15.98:0/391076330 wait complete. 2024-06-11T19:11:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:11:58.029+0000 7fa3bfcca640 1 librados: shutdown 2024-06-11T19:11:58.078 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:11:58.078 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-06-11T19:12:00.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-06-11T19:12:00.826 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: seed 36868 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2030 ms) 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2020 ms) 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2017 ms) 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2003 ms) 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-06-11T19:12:00.827 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3860 ms) 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11930 ms total) 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4075 ms) 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3959 ms) 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5956 ms) 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4043 ms) 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2008 ms) 2024-06-11T19:12:00.828 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-4216-g37bbe1d8/rpm/el9/BUILD/ceph-19.0.0-4216-g37bbe1d8/src/test/librados/snapshots_cxx.cc:460: Skipped 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi117-36868-7 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: waiting for snaps to purge 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17971 ms) 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (38012 ms total) 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (12 ms) 2024-06-11T19:12:00.829 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-06-11T19:12:00.830 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5046 ms) 2024-06-11T19:12:00.830 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5058 ms total) 2024-06-11T19:12:00.830 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: 2024-06-11T19:12:00.830 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-06-11T19:12:00.830 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-06-11T19:12:00.830 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3027 ms) 2024-06-11T19:12:00.830 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-06-11T19:12:00.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRados api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-06-11T19:12:00.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-06-11T19:12:00.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-06-11T19:12:00.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-06-11T19:12:00.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-06-11T19:12:00.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-06-11T19:12:00.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-06-11T19:12:00.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-06-11T19:12:00.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: seed 36810 2024-06-11T19:12:00.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-06-11T19:12:00.831 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-06-11T19:12:00.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-06-11T19:12:00.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (270 ms) 2024-06-11T19:12:00.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-06-11T19:12:00.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (23 ms) 2024-06-11T19:12:00.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-06-11T19:12:00.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (14 ms) 2024-06-11T19:12:00.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-06-11T19:12:00.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (17 ms) 2024-06-11T19:12:00.832 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-06-11T19:12:00.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2024-06-11T19:12:00.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-06-11T19:12:00.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-06-11T19:12:00.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-06-11T19:12:00.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (9 ms) 2024-06-11T19:12:00.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-06-11T19:12:00.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (6 ms) 2024-06-11T19:12:00.833 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (19 ms) 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (6 ms) 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (14 ms) 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (21 ms) 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: osd_max_attr_size = 0 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2601 ms) 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-06-11T19:12:00.834 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (381 ms) 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: done waiting, doing copies 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: waiting for final deep scrubs... 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: done waiting 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60363 ms) 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (8 ms) 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (3 ms) 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (20181 ms) 2024-06-11T19:12:00.835 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-06-11T19:12:00.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-06-11T19:12:00.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-06-11T19:12:00.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-06-11T19:12:00.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-06-11T19:12:00.836 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-06-11T19:12:17.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 22 tests from LibRadosMis api_aio: Running main() from gmock_main.cc 2024-06-11T19:12:17.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-06-11T19:12:17.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [----------] Global test environment set-up. 2024-06-11T19:12:17.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-06-11T19:12:17.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-06-11T19:12:17.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2841 ms) 2024-06-11T19:12:17.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-06-11T19:12:17.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3052 ms) 2024-06-11T19:12:17.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-06-11T19:12:17.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2992 ms) 2024-06-11T19:12:17.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-06-11T19:12:17.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2864 ms) 2024-06-11T19:12:17.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-06-11T19:12:17.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3008 ms) 2024-06-11T19:12:17.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-06-11T19:12:17.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3020 ms) 2024-06-11T19:12:17.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-06-11T19:12:17.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (2981 ms) 2024-06-11T19:12:17.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-06-11T19:12:17.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3044 ms) 2024-06-11T19:12:17.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-06-11T19:12:17.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2957 ms) 2024-06-11T19:12:17.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-06-11T19:12:17.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3056 ms) 2024-06-11T19:12:17.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-06-11T19:12:17.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3004 ms) 2024-06-11T19:12:17.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-06-11T19:12:17.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2951 ms) 2024-06-11T19:12:17.855 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3002 ms) 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2988 ms) 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.Flush (3006 ms) 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2995 ms) 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2986 ms) 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3521 ms) 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3014 ms) 2024-06-11T19:12:17.856 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3012 ms) 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (22130 ms) 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3030 ms) 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2318 ms) 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2994 ms) 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3007 ms) 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (2994 ms) 2024-06-11T19:12:17.857 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [----------] 26 tests from LibRadosAio (96767 ms total) 2024-06-11T19:12:17.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: 2024-06-11T19:12:17.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-06-11T19:12:17.858 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-06-11T19:12:25.852 INFO:tasks.workunit.client.0.smithi117.stdout:SnapshotsECPP.SnapRemovePP (1317 ms) 2024-06-11T19:12:25.852 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2011 ms) 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1986 ms) 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8341 ms total) 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4011 ms) 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3991 ms) 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3995 ms) 2024-06-11T19:12:25.853 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11997 ms total) 2024-06-11T19:12:25.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: 2024-06-11T19:12:25.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-06-11T19:12:25.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (111666 ms total) 2024-06-11T19:12:25.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-06-11T19:12:25.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-06-11T19:12:25.854 INFO:tasks.workunit.client.0.smithi117.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-06-11T19:12:26.860 INFO:tasks.workunit.client.0.smithi117.stdout:cPP (83943 ms total) 2024-06-11T19:12:26.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-06-11T19:12:26.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-06-11T19:12:26.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-06-11T19:12:26.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (82 ms) 2024-06-11T19:12:26.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (82 ms total) 2024-06-11T19:12:26.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-06-11T19:12:26.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-06-11T19:12:26.860 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (23 ms) 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (4 ms) 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (27 ms total) 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (27 ms) 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (7 ms) 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (34 ms total) 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-06-11T19:12:26.861 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (24 ms) 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (4 ms) 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (28 ms total) 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1032 ms) 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1032 ms total) 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (112669 ms total) 2024-06-11T19:12:26.862 INFO:tasks.workunit.client.0.smithi117.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-06-11T19:12:33.844 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-06-11T19:12:33.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-06-11T19:12:33.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-06-11T19:12:33.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-06-11T19:12:33.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-06-11T19:12:33.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (2845 ms) 2024-06-11T19:12:33.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-06-11T19:12:33.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (19917 ms) 2024-06-11T19:12:33.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-06-11T19:12:33.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6022 ms) 2024-06-11T19:12:33.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-06-11T19:12:33.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2953 ms) 2024-06-11T19:12:33.845 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-06-11T19:12:33.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3021 ms) 2024-06-11T19:12:33.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-06-11T19:12:33.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3005 ms) 2024-06-11T19:12:33.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-06-11T19:12:33.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2988 ms) 2024-06-11T19:12:33.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-06-11T19:12:33.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3004 ms) 2024-06-11T19:12:33.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-06-11T19:12:33.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2997 ms) 2024-06-11T19:12:33.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-06-11T19:12:33.846 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3006 ms) 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2992 ms) 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3525 ms) 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3006 ms) 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2472 ms) 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (22692 ms) 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2322 ms) 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3013 ms) 2024-06-11T19:12:33.847 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2988 ms) 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3001 ms) 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3004 ms) 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2995 ms) 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3009 ms) 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3000 ms) 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3002 ms) 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-06-11T19:12:33.848 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3002 ms) 2024-06-11T19:12:33.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-06-11T19:12:33.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2987 ms) 2024-06-11T19:12:33.849 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-06-11T19:13:00.659 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAited=5 2024-06-11T19:13:00.659 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:89d3ae78:::11:head 2024-06-11T19:13:00.659 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:89d3ae78:::11:head expected=10:89d3ae78:::11:head 2024-06-11T19:13:00.659 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:89d3ae78:::11:head -> 11 2024-06-11T19:13:00.659 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=11 expected=11 2024-06-11T19:13:00.659 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:863748b0:::15:head 2024-06-11T19:13:00.659 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:863748b0:::15:head expected=10:863748b0:::15:head 2024-06-11T19:13:00.659 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:863748b0:::15:head -> 15 2024-06-11T19:13:00.659 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=15 expected=15 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:6cac518f:::0:head 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:6cac518f:::0:head expected=10:6cac518f:::0:head 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:6cac518f:::0:head -> 0 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=0 expected=0 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:52ea6a34:::10:head 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:52ea6a34:::10:head expected=10:52ea6a34:::10:head 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:52ea6a34:::10:head -> 10 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=10 expected=10 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:02547ec2:::1:head 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:02547ec2:::1:head expected=10:02547ec2:::1:head 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:02547ec2:::1:head -> 1 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=1 expected=1 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:62a1935d:::14:head 2024-06-11T19:13:00.660 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:62a1935d:::14:head expected=10:62a1935d:::14:head 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:62a1935d:::14:head -> 14 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=14 expected=14 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:5c6b0b28:::7:head 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:5c6b0b28:::7:head expected=10:5c6b0b28:::7:head 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:5c6b0b28:::7:head -> 7 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=7 expected=7 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : seek to 10:566253c9:::13:head 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : cursor()=10:566253c9:::13:head expected=10:566253c9:::13:head 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: > 10:566253c9:::13:head -> 13 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: : entry=13 expected=13 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (258 ms) 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-06-11T19:13:00.661 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (110197 ms) 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: split 0/5 -> MIN 10:33333333::::head 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: split 1/5 -> 10:33333333::::head 10:66666666::::head 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: split 2/5 -> 10:66666666::::head 10:99999999::::head 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: split 3/5 -> 10:99999999::::head 10:cccccccc::::head 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: split 4/5 -> 10:cccccccc::::head MAX 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (28028 ms) 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] 7 tests from LibRadosList (139052 ms total) 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1730 ms) 2024-06-11T19:13:00.662 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset foo1,foo2,foo3 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo1 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo2 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo3 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset foo1,foo4,foo5 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo4 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo5 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo1 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset foo6,foo7 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo7 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: foo6 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-06-11T19:13:00.663 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns1:foo4 2024-06-11T19:13:00.664 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns1:foo5 2024-06-11T19:13:00.664 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns2:foo7 2024-06-11T19:13:00.664 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns2:foo6 2024-06-11T19:13:00.664 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: ns1:foo1 2024-06-11T19:13:00.664 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: :foo1 2024-06-11T19:13:00.664 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: :foo2 2024-06-11T19:13:00.664 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: :foo3 2024-06-11T19:13:00.664 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (51 ms) 2024-06-11T19:13:05.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-06-11T19:13:05.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 1 0 2024-06-11T19:13:05.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 10 0 2024-06-11T19:13:05.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 13 0 2024-06-11T19:13:05.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 7 0 2024-06-11T19:13:05.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 14 0 2024-06-11T19:13:05.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 0 0 2024-06-11T19:13:05.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 15 0 2024-06-11T19:13:05.906 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 11 0 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 5 0 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 8 0 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 6 0 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 3 0 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 4 0 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 12 0 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 9 0 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 2 0 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (90 ms) 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] 3 tests from LibRadosListEC (1871 ms total) 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-06-11T19:13:05.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-06-11T19:13:05.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3039 ms) 2024-06-11T19:13:05.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] 1 test from LibRadosListNP (3039 ms total) 2024-06-11T19:13:05.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: 2024-06-11T19:13:05.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [----------] Global test environment tear-down 2024-06-11T19:13:05.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [==========] 11 tests from 3 test suites ran. (151793 ms total) 2024-06-11T19:13:05.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_list: [ PASSED ] 11 tests. 2024-06-11T19:13:28.078 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-06-11T19:13:28.079 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-06-11T19:13:28.079 INFO:tasks.ceph.mon.e:Restarting daemon 2024-06-11T19:13:28.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-06-11T19:13:28.083 INFO:tasks.ceph.mon.e:Started 2024-06-11T19:13:28.083 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-06-11T19:13:28.083 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-06-11T19:13:28.083 INFO:tasks.ceph.mon.a:Restarting daemon 2024-06-11T19:13:28.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-06-11T19:13:28.085 INFO:tasks.ceph.mon.a:Started 2024-06-11T19:13:28.085 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:13:28.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:13:28.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.194+0000 7fca4ec54640 10 monclient: get_monmap_and_config 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.194+0000 7fca4ec54640 10 monclient: build_initial_monmap 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.194+0000 7fca4ec54640 10 monclient: monmap: 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:28.195203+0000 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:28.195203+0000 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:28.195 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:28.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.194+0000 7fca4ec54640 1 Processor -- start 2024-06-11T19:13:28.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 1 -- start start 2024-06-11T19:13:28.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 10 monclient: init 2024-06-11T19:13:28.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:28.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 10 monclient: _add_conns ranks=[2,8,0,3,4,9,5,1,6,7,10] 2024-06-11T19:13:28.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca480a2f40 0x7fca480a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 10 monclient(hunting): picked mon.noname-b con 0x7fca480a2f40 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:28.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca480a4df0 0x7fca480a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 10 monclient(hunting): picked mon.noname-e con 0x7fca480a4df0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca481366d0 addr v1:172.21.15.98:6789/0 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca480a0ed0 con 0x7fca481366d0 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fca480a5710 con 0x7fca480a2f40 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca47fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca480a4df0 0x7fca480a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fca480a3320 con 0x7fca480a4df0 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca47fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca480a4df0 0x7fca480a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:52582/0 (socket says 172.21.15.98:52582) 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4d1f1640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fca481366d0 0x7fca48136b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43058/0 (socket says 172.21.15.98:43058) 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca47fff640 1 -- 172.21.15.98:0/1831898179 learned_addr learned my addr 172.21.15.98:0/1831898179 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:28.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca4ec54640 10 monclient(hunting): authenticate will time out at 1018.002718s 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca47fff640 10 monclient(hunting): get_auth_request con 0x7fca480a4df0 auth_method 0 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca47fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca47fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca47fff640 1 -- 172.21.15.98:0/1831898179 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca480a4df0 msgr2=0x7fca480a51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca47fff640 1 -- 172.21.15.98:0/1831898179 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca480a4df0 msgr2=0x7fca480a51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca47fff640 1 --2- 172.21.15.98:0/1831898179 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca480a4df0 0x7fca480a51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.195+0000 7fca47fff640 1 --2- 172.21.15.98:0/1831898179 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca480a4df0 0x7fca480a51d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 1 --2- 172.21.15.98:0/1831898179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca480a2f40 0x7fca480a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 10 monclient(hunting): get_auth_request con 0x7fca480a2f40 auth_method 0 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:28.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 10 monclient(hunting): handle_auth_done global_id 21989 payload 274 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 1 -- 172.21.15.98:0/1831898179 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca480a4df0 msgr2=0x7fca480a51d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 1 --2- 172.21.15.98:0/1831898179 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca480a4df0 0x7fca480a51d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 1 -- 172.21.15.98:0/1831898179 >> v1:172.21.15.98:6789/0 conn(0x7fca481366d0 legacy=0x7fca48136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 10 monclient: _finish_hunting 0 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 1 monclient: found mon.noname-b 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.196+0000 7fca4c9f0640 1 -- 172.21.15.98:0/1831898179 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca480a2180 con 0x7fca480a2f40 2024-06-11T19:13:28.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4c9f0640 1 --2- 172.21.15.98:0/1831898179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca480a2f40 0x7fca480a48b0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fca40009800 tx=0x7fca4002b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5fd3aaf103d3b0a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 1 -- 172.21.15.98:0/1831898179 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca40037070 con 0x7fca480a2f40 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 10 monclient: dump: 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:28.211 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 10 monclient: _finish_auth 0 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 10 monclient: _check_auth_tickets 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 1 -- 172.21.15.98:0/1831898179 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca400044a0 con 0x7fca480a2f40 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 5 monclient: authenticate success, global_id 21989 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:28.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 1 -- 172.21.15.98:0/1831898179 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca40003a80 con 0x7fca480a2f40 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 4 set_mon_vals no callback set 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca477fe640 10 monclient: dump: 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:28.213 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 10 monclient: shutdown 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 1 -- 172.21.15.98:0/1831898179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca480a2f40 msgr2=0x7fca480a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 1 --2- 172.21.15.98:0/1831898179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca480a2f40 0x7fca480a48b0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fca40009800 tx=0x7fca4002b760 comp rx=0 tx=0).stop 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 1 -- 172.21.15.98:0/1831898179 shutdown_connections 2024-06-11T19:13:28.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 1 --2- 172.21.15.98:0/1831898179 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca480a4df0 0x7fca480a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 1 --2- 172.21.15.98:0/1831898179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fca480a2f40 0x7fca480a48b0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 1 -- 172.21.15.98:0/1831898179 >> 172.21.15.98:0/1831898179 conn(0x7fca480a95b0 msgr2=0x7fca480a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 1 -- 172.21.15.98:0/1831898179 shutdown_connections 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.197+0000 7fca4ec54640 1 -- 172.21.15.98:0/1831898179 wait complete. 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.198+0000 7fca4ec54640 10 monclient: build_initial_monmap 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.198+0000 7fca4ec54640 10 monclient: monmap: 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:28.215 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.198+0000 7fca4ec54640 1 librados: starting msgr at 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.198+0000 7fca4ec54640 1 librados: starting objecter 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.198+0000 7fca4ec54640 1 Processor -- start 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 1 -- start start 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 1 librados: setting wanted keys 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 1 librados: calling monclient init 2024-06-11T19:13:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 10 monclient: init 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 10 monclient: _add_conns ranks=[7,6,8,0,2,3,1,4,5] 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fca480a2f40 0x7fca48134e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 10 monclient(hunting): picked mon.noname-i con 0x7fca480a2f40 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca480a4df0 0x7fca48135340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 10 monclient(hunting): picked mon.noname-d con 0x7fca480a4df0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca481366d0 0x7fca48120e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 10 monclient(hunting): picked mon.noname-e con 0x7fca481366d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fca48137230 con 0x7fca480a4df0 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fca480a2460 con 0x7fca480a2f40 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca47fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca480a4df0 0x7fca48135340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fca481373b0 con 0x7fca481366d0 2024-06-11T19:13:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca47fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca480a4df0 0x7fca48135340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:37128/0 (socket says 172.21.15.98:37128) 2024-06-11T19:13:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca47fff640 1 -- 172.21.15.98:0/988124043 learned_addr learned my addr 172.21.15.98:0/988124043 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.199+0000 7fca4ec54640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4ec54640 10 monclient(hunting): authenticate will time out at 1018.007020s 2024-06-11T19:13:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 10 monclient(hunting): get_auth_request con 0x7fca480a4df0 auth_method 0 2024-06-11T19:13:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4d1f1640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca481366d0 0x7fca48120e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4c9f0640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fca480a2f40 0x7fca48134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4d1f1640 10 monclient(hunting): get_auth_request con 0x7fca481366d0 auth_method 0 2024-06-11T19:13:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4d1f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4d1f1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4d1f1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4c9f0640 10 monclient(hunting): get_auth_request con 0x7fca480a2f40 auth_method 0 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4c9f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4c9f0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4c9f0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4d1f1640 1 -- 172.21.15.98:0/988124043 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca481366d0 msgr2=0x7fca48120e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4d1f1640 1 -- 172.21.15.98:0/988124043 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca481366d0 msgr2=0x7fca48120e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4d1f1640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca481366d0 0x7fca48120e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4d1f1640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca481366d0 0x7fca48120e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4c9f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4c9f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4c9f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 10 monclient(hunting): handle_auth_done global_id 54591 payload 824 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 1 -- 172.21.15.98:0/988124043 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca481366d0 msgr2=0x7fca48120e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca481366d0 0x7fca48120e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 1 -- 172.21.15.98:0/988124043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fca480a2f40 msgr2=0x7fca48134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fca480a2f40 0x7fca48134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 10 monclient: _finish_hunting 0 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 1 monclient: found mon.noname-d 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 1 -- 172.21.15.98:0/988124043 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca48075380 con 0x7fca480a4df0 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4c9f0640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fca480a2f40 0x7fca48134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca47fff640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca480a4df0 0x7fca48135340 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fca38007bd0 tx=0x7fca3800d4a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 10 client.?.objecter ms_handle_connect 0x7fca480a4df0 2024-06-11T19:13:28.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 1 -- 172.21.15.98:0/988124043 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca3800dbb0 con 0x7fca480a4df0 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 10 monclient: dump: 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 10 monclient: _finish_auth 0 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 10 monclient: _check_auth_tickets 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 1 -- 172.21.15.98:0/988124043 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca38018070 con 0x7fca480a4df0 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 1 -- 172.21.15.98:0/988124043 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca3800ddb0 con 0x7fca480a4df0 2024-06-11T19:13:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca46ffd640 4 set_mon_vals no callback set 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca457fa640 10 monclient: dump: 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:28.223 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4ec54640 5 monclient: authenticate success, global_id 54591 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4ec54640 10 monclient: _renew_subs 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4ec54640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4ec54640 1 -- 172.21.15.98:0/988124043 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca48124b00 con 0x7fca480a4df0 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4ec54640 10 client.54591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4ec54640 10 monclient: _renew_subs 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4ec54640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4ec54640 1 -- 172.21.15.98:0/988124043 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fca48124210 con 0x7fca480a4df0 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.200+0000 7fca4ec54640 1 librados: init done 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca457fa640 1 -- 172.21.15.98:0/988124043 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fca3801d460 con 0x7fca480a4df0 2024-06-11T19:13:28.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca457fa640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca2803fdb0 0x7fca28042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca457fa640 1 -- 172.21.15.98:0/988124043 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(173..173 src has 1..173) ==== 6622+0+0 (secure 0 0 0) 0x7fca38051fd0 con 0x7fca480a4df0 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca457fa640 10 client.54591.objecter ms_dispatch 0x7fca48000b90 osd_map(173..173 src has 1..173) 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca457fa640 3 client.54591.objecter handle_osd_map got epochs [173,173] > 0 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca457fa640 3 client.54591.objecter handle_osd_map decoding full epoch 173 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca4ec54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca4ec54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca4ec54640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca457fa640 20 client.54591.objecter dump_active .. 0 homeless 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca4ec54640 1 -- 172.21.15.98:0/988124043 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca48124540 con 0x7fca480a4df0 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca4c9f0640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca2803fdb0 0x7fca28042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.202+0000 7fca4c9f0640 10 monclient: get_auth_request con 0x7fca2803fdb0 auth_method 0 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.203+0000 7fca457fa640 10 client.54591.objecter ms_handle_connect 0x7fca2803fdb0 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.203+0000 7fca4c9f0640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca2803fdb0 0x7fca28042270 secure :-1 s=READY pgs=4063 cs=0 l=1 rev1=1 crypto rx=0x7fca40034000 tx=0x7fca4003c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:28.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.207+0000 7fca457fa640 1 -- 172.21.15.98:0/988124043 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fca38010070 con 0x7fca480a4df0 2024-06-11T19:13:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.207+0000 7fca457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:28.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.207+0000 7fca457fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:28.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.483+0000 7fca4ec54640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:13:28.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.483+0000 7fca4ec54640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:13:28.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.483+0000 7fca4ec54640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:28.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.483+0000 7fca4ec54640 1 -- 172.21.15.98:0/988124043 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fca480a48b0 con 0x7fca480a4df0 2024-06-11T19:13:28.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.484+0000 7fca457fa640 1 -- 172.21.15.98:0/988124043 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fca3801bc60 con 0x7fca480a4df0 2024-06-11T19:13:28.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.484+0000 7fca457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:13:28.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.484+0000 7fca457fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:28.484 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":20,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:13:28.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 10 librados: watch_flush enter 2024-06-11T19:13:28.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 10 librados: watch_flush exit 2024-06-11T19:13:28.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 20 client.54591.objecter shutdown clearing up homeless session... 2024-06-11T19:13:28.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 10 client.54591.objecter successfully canceled tick 2024-06-11T19:13:28.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 -- 172.21.15.98:0/988124043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca2803fdb0 msgr2=0x7fca28042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:28.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca2803fdb0 0x7fca28042270 secure :-1 s=READY pgs=4063 cs=0 l=1 rev1=1 crypto rx=0x7fca40034000 tx=0x7fca4003c040 comp rx=0 tx=0).stop 2024-06-11T19:13:28.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 10 monclient: shutdown 2024-06-11T19:13:28.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:28.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 -- 172.21.15.98:0/988124043 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca480a4df0 msgr2=0x7fca48135340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:28.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca480a4df0 0x7fca48135340 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fca38007bd0 tx=0x7fca3800d4a0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 -- 172.21.15.98:0/988124043 shutdown_connections 2024-06-11T19:13:28.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fca2803fdb0 0x7fca28042270 unknown :-1 s=CLOSED pgs=4063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fca481366d0 0x7fca48120e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fca480a4df0 0x7fca48135340 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 --2- 172.21.15.98:0/988124043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fca480a2f40 0x7fca48134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 -- 172.21.15.98:0/988124043 >> 172.21.15.98:0/988124043 conn(0x7fca480a95b0 msgr2=0x7fca480a8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:28.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 -- 172.21.15.98:0/988124043 shutdown_connections 2024-06-11T19:13:28.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 -- 172.21.15.98:0/988124043 wait complete. 2024-06-11T19:13:28.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.486+0000 7fca4ec54640 1 librados: shutdown 2024-06-11T19:13:28.496 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:13:28.496 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:13:28.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:13:28.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.596+0000 7f864fc15640 10 monclient: get_monmap_and_config 2024-06-11T19:13:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.596+0000 7f864fc15640 10 monclient: build_initial_monmap 2024-06-11T19:13:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.596+0000 7f864fc15640 10 monclient: monmap: 2024-06-11T19:13:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:28.597560+0000 2024-06-11T19:13:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:28.597560+0000 2024-06-11T19:13:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:28.596 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:28.597 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:28.597 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:28.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 1 Processor -- start 2024-06-11T19:13:28.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 1 -- start start 2024-06-11T19:13:28.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 10 monclient: init 2024-06-11T19:13:28.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:28.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 10 monclient: _add_conns ranks=[5,1,8,0,10,3,7,9,4,2,6] 2024-06-11T19:13:28.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f86480a4920 0x7f86480a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 10 monclient(hunting): picked mon.noname-h con 0x7f86480a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:28.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f86480a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:13:28.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f86480a2530 0x7f86480a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 10 monclient(hunting): picked mon.noname-e con 0x7f86480a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:28.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86480a7140 con 0x7f86480a5240 2024-06-11T19:13:28.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:28.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f86480a2b50 con 0x7f86480a4920 2024-06-11T19:13:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f86480a2e30 con 0x7f86480a2530 2024-06-11T19:13:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864fc15640 10 monclient(hunting): authenticate will time out at 1018.404808s 2024-06-11T19:13:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864e1b2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f86480a2530 0x7f86480a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864e1b2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f86480a2530 0x7f86480a60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:52596/0 (socket says 172.21.15.98:52596) 2024-06-11T19:13:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864e1b2640 1 -- 172.21.15.98:0/803720391 learned_addr learned my addr 172.21.15.98:0/803720391 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864e1b2640 10 monclient(hunting): get_auth_request con 0x7f86480a2530 auth_method 0 2024-06-11T19:13:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864e1b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864e1b2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.597+0000 7f864e1b2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864e1b2640 1 -- 172.21.15.98:0/803720391 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f86480a2530 msgr2=0x7f86480a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:13:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864e1b2640 1 -- 172.21.15.98:0/803720391 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f86480a2530 msgr2=0x7f86480a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864e1b2640 1 --2- 172.21.15.98:0/803720391 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f86480a2530 0x7f86480a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864e1b2640 1 --2- 172.21.15.98:0/803720391 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f86480a2530 0x7f86480a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 1 --2- 172.21.15.98:0/803720391 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f86480a4920 0x7f86480a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 10 monclient(hunting): get_auth_request con 0x7f86480a4920 auth_method 0 2024-06-11T19:13:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 10 monclient(hunting): handle_auth_done global_id 22406 payload 274 2024-06-11T19:13:28.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864c9af640 1 -- 172.21.15.98:0/803720391 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1941800937 0 0) 0x7f86480a7140 con 0x7f86480a5240 2024-06-11T19:13:28.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 1 -- 172.21.15.98:0/803720391 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f86480a2530 msgr2=0x7f86480a60b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:28.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 1 --2- 172.21.15.98:0/803720391 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f86480a2530 0x7f86480a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 1 -- 172.21.15.98:0/803720391 >> v1:172.21.15.117:6789/0 conn(0x7f86480a5240 legacy=0x7f86480a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:28.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 10 monclient: _finish_hunting 0 2024-06-11T19:13:28.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 1 monclient: found mon.noname-h 2024-06-11T19:13:28.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:28.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:13:28.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864d9b1640 1 -- 172.21.15.98:0/803720391 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86480a3110 con 0x7f86480a4920 2024-06-11T19:13:28.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.598+0000 7f864c9af640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:13:28.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864d9b1640 1 --2- 172.21.15.98:0/803720391 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f86480a4920 0x7f86480a4d00 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f86440097d0 tx=0x7f864402b770 comp rx=0 tx=0).ready entity=mon.5 client_cookie=ce5f2937d99cf65d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:28.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 1 -- 172.21.15.98:0/803720391 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8644039070 con 0x7f86480a4920 2024-06-11T19:13:28.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:28.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:13:28.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 10 monclient: dump: 2024-06-11T19:13:28.612 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:28.612 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:28.612 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.612 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.612 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:28.612 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:28.612 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:28.612 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:28.612 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:28.613 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:28.613 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:28.613 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:28.613 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:28.613 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:28.613 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:28.613 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:28.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 10 monclient: _finish_auth 0 2024-06-11T19:13:28.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 10 monclient: _check_auth_tickets 2024-06-11T19:13:28.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:28.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 1 -- 172.21.15.98:0/803720391 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8644030070 con 0x7f86480a4920 2024-06-11T19:13:28.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864fc15640 5 monclient: authenticate success, global_id 22406 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864fc15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864fc15640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864fc15640 4 set_mon_vals no callback set 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 1 -- 172.21.15.98:0/803720391 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8644031940 con 0x7f86480a4920 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864c9af640 10 monclient: dump: 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864fc15640 10 monclient: shutdown 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864fc15640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:28.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864fc15640 1 -- 172.21.15.98:0/803720391 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f86480a4920 msgr2=0x7f86480a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864fc15640 1 --2- 172.21.15.98:0/803720391 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f86480a4920 0x7f86480a4d00 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f86440097d0 tx=0x7f864402b770 comp rx=0 tx=0).stop 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864fc15640 1 -- 172.21.15.98:0/803720391 shutdown_connections 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864fc15640 1 --2- 172.21.15.98:0/803720391 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f86480a2530 0x7f86480a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864fc15640 1 --2- 172.21.15.98:0/803720391 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f86480a4920 0x7f86480a4d00 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.599+0000 7f864fc15640 1 -- 172.21.15.98:0/803720391 >> 172.21.15.98:0/803720391 conn(0x7f86480ab850 msgr2=0x7f86480a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.600+0000 7f864fc15640 1 -- 172.21.15.98:0/803720391 shutdown_connections 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.600+0000 7f864fc15640 1 -- 172.21.15.98:0/803720391 wait complete. 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.600+0000 7f864fc15640 10 monclient: build_initial_monmap 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.600+0000 7f864fc15640 10 monclient: monmap: 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.600+0000 7f864fc15640 1 librados: starting msgr at 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.600+0000 7f864fc15640 1 librados: starting objecter 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.600+0000 7f864fc15640 1 Processor -- start 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.600+0000 7f864fc15640 1 -- start start 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.600+0000 7f864fc15640 1 librados: setting wanted keys 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.600+0000 7f864fc15640 1 librados: calling monclient init 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.600+0000 7f864fc15640 10 monclient: init 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 10 monclient: _add_conns ranks=[1,7,0,6,8,3,5,2,4] 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86480a2530 0x7f86481cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 10 monclient(hunting): picked mon.noname-f con 0x7f86480a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f86480a4920 0x7f86481cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 10 monclient(hunting): picked mon.noname-i con 0x7f86480a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:28.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480a5240 0x7f86481251f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 10 monclient(hunting): picked mon.noname-a con 0x7f86480a5240 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86480a6da0 con 0x7f86480a5240 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8648138e50 con 0x7f86480a2530 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f86480a65f0 con 0x7f86480a4920 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864fc15640 10 monclient(hunting): authenticate will time out at 1018.408339s 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86480a2530 0x7f86481cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864e1b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480a5240 0x7f86481251f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864e1b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480a5240 0x7f86481251f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45248/0 (socket says 172.21.15.98:45248) 2024-06-11T19:13:28.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864e1b2640 1 -- 172.21.15.98:0/897215609 learned_addr learned my addr 172.21.15.98:0/897215609 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:28.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86480a2530 0x7f86481cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:40600/0 (socket says 172.21.15.98:40600) 2024-06-11T19:13:28.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864e1b2640 10 monclient(hunting): get_auth_request con 0x7f86480a5240 auth_method 0 2024-06-11T19:13:28.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864e1b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:28.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864e1b2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:28.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864e1b2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:28.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 10 monclient(hunting): get_auth_request con 0x7f86480a2530 auth_method 0 2024-06-11T19:13:28.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:28.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:28.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:28.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f86480a4920 0x7f86481cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d1b0640 10 monclient(hunting): get_auth_request con 0x7f86480a4920 auth_method 0 2024-06-11T19:13:28.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d1b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d1b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d1b0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864e1b2640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480a5240 msgr2=0x7f86481251f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864e1b2640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480a5240 msgr2=0x7f86481251f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864e1b2640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480a5240 0x7f86481251f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864e1b2640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480a5240 0x7f86481251f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d1b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d1b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d1b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:28.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 10 monclient(hunting): handle_auth_done global_id 31393 payload 824 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f86480a4920 msgr2=0x7f86481cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f86480a4920 0x7f86481cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480a5240 msgr2=0x7f86481251f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480a5240 0x7f86481251f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 10 monclient: _finish_hunting 0 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 1 monclient: found mon.noname-f 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d9b1640 1 -- 172.21.15.98:0/897215609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86481387f0 con 0x7f86480a2530 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.601+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f86480a4920 0x7f86481cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f864d9b1640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86480a2530 0x7f86481cda70 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f8644033000 tx=0x7f86440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 10 client.?.objecter ms_handle_connect 0x7f86480a2530 2024-06-11T19:13:28.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8644042020 con 0x7f86480a2530 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 10 monclient: dump: 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:28.623 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 10 monclient: _finish_auth 0 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f864403a040 con 0x7f86480a2530 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f864fc15640 5 monclient: authenticate success, global_id 31393 2024-06-11T19:13:28.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f864fc15640 10 monclient: _renew_subs 2024-06-11T19:13:28.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f864fc15640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:13:28.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f864fc15640 1 -- 172.21.15.98:0/897215609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86480a8f30 con 0x7f86480a2530 2024-06-11T19:13:28.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:28.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8644039070 con 0x7f86480a2530 2024-06-11T19:13:28.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:28.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:13:28.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f864fc15640 10 client.31393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:28.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863e7fc640 10 monclient: dump: 2024-06-11T19:13:28.625 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:28.625 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:28.625 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.625 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:28.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f863ffff640 4 set_mon_vals no callback set 2024-06-11T19:13:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f864fc15640 10 monclient: _renew_subs 2024-06-11T19:13:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f864fc15640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:13:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f864fc15640 1 -- 172.21.15.98:0/897215609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8648128b70 con 0x7f86480a2530 2024-06-11T19:13:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.602+0000 7f864fc15640 1 librados: init done 2024-06-11T19:13:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864fc15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864fc15640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864fc15640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864fc15640 1 -- 172.21.15.98:0/897215609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86480aaac0 con 0x7f8648128d40 2024-06-11T19:13:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8644045020 con 0x7f86480a2530 2024-06-11T19:13:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864d1b0640 10 monclient: get_auth_request con 0x7f8648128d40 auth_method 0 2024-06-11T19:13:28.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864d1b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864d1b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864d1b0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 msgr2=0x7f8648063180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-06-11T19:13:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 msgr2=0x7f8648063180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:13:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f863e7fc640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f862c03fdb0 0x7f862c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 6622+0+0 (secure 0 0 0) 0x7f86440732c0 con 0x7f86480a2530 2024-06-11T19:13:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f863e7fc640 10 client.31393.objecter ms_dispatch 0x7f8648000b90 osd_map(173..173 src has 1..173) 2024-06-11T19:13:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f863e7fc640 3 client.31393.objecter handle_osd_map got epochs [173,173] > 0 2024-06-11T19:13:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f863e7fc640 3 client.31393.objecter handle_osd_map decoding full epoch 173 2024-06-11T19:13:28.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f863e7fc640 20 client.31393.objecter dump_active .. 0 homeless 2024-06-11T19:13:28.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864e1b2640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f862c03fdb0 0x7f862c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.603+0000 7f864e1b2640 10 monclient: get_auth_request con 0x7f862c03fdb0 auth_method 0 2024-06-11T19:13:28.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.604+0000 7f863e7fc640 10 client.31393.objecter ms_handle_connect 0x7f862c03fdb0 2024-06-11T19:13:28.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.604+0000 7f864e1b2640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f862c03fdb0 0x7f862c042270 secure :-1 s=READY pgs=4064 cs=0 l=1 rev1=1 crypto rx=0x7f86481269d0 tx=0x7f8640009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.804+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.804+0000 7f864d1b0640 10 monclient: get_auth_request con 0x7f8648128d40 auth_method 0 2024-06-11T19:13:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.804+0000 7f864d1b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.804+0000 7f864d1b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.804+0000 7f864d1b0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.804+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 msgr2=0x7f8648063180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:13:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.804+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 msgr2=0x7f8648063180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:13:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.804+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:28.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:28.804+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:13:29.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:29.205+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:29.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:29.205+0000 7f864d1b0640 10 monclient: get_auth_request con 0x7f8648128d40 auth_method 0 2024-06-11T19:13:29.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:29.205+0000 7f864d1b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:29.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:29.205+0000 7f864d1b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:29.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:29.205+0000 7f864d1b0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:29.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:29.205+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 msgr2=0x7f8648063180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:13:29.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:29.205+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 msgr2=0x7f8648063180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:13:29.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:29.205+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:29.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:29.205+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:13:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:30.006+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:30.006+0000 7f864d1b0640 10 monclient: get_auth_request con 0x7f8648128d40 auth_method 0 2024-06-11T19:13:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:30.006+0000 7f864d1b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:30.006+0000 7f864d1b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:30.006+0000 7f864d1b0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:30.006+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 msgr2=0x7f8648063180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:13:30.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:30.006+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 msgr2=0x7f8648063180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:13:30.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:30.006+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:30.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:30.006+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:13:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.602+0000 7f863d7fa640 10 monclient: tick 2024-06-11T19:13:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.602+0000 7f863d7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:13:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.602+0000 7f863d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.602+0000 7f863d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.608+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.608+0000 7f864d1b0640 10 monclient: get_auth_request con 0x7f8648128d40 auth_method 0 2024-06-11T19:13:31.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.608+0000 7f864d1b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.608+0000 7f864d1b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.608+0000 7f864d1b0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.608+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 msgr2=0x7f8648063180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:13:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.608+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 msgr2=0x7f8648063180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:13:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.608+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:31.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:31.608+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-06-11T19:13:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f863d7fa640 10 monclient: tick 2024-06-11T19:13:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f863d7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:13:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f863d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f863d7fa640 5 monclient: _check_tell_commands timeout tell command 1 2024-06-11T19:13:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f863d7fa640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 msgr2=0x7f8648063180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f863d7fa640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f863d7fa640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 0x7f8624005a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f863d7fa640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 msgr2=0x7f8648063180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f863d7fa640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8648128d40 0x7f8648063180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f863d7fa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:32.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f863d7fa640 1 -- 172.21.15.98:0/897215609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86240060b0 con 0x7f8624003540 2024-06-11T19:13:32.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 0x7f8624005a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:32.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f864d1b0640 10 monclient: get_auth_request con 0x7f8624003540 auth_method 0 2024-06-11T19:13:32.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f864d1b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:32.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f864d1b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:32.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f864d1b0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:32.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 msgr2=0x7f8624005a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:13:32.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 msgr2=0x7f8624005a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:13:32.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 0x7f8624005a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:32.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.602+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 0x7f8624005a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.803+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 0x7f8624005a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.803+0000 7f864d1b0640 10 monclient: get_auth_request con 0x7f8624003540 auth_method 0 2024-06-11T19:13:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.803+0000 7f864d1b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.803+0000 7f864d1b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.803+0000 7f864d1b0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.803+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 msgr2=0x7f8624005a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:13:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.803+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 msgr2=0x7f8624005a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:13:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.803+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 0x7f8624005a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:32.803+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 0x7f8624005a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:13:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 0x7f8624005a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f864d1b0640 10 monclient: get_auth_request con 0x7f8624003540 auth_method 0 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f864d1b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f864d1b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f864d1b0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f864d1b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f864d1b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f864d1b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f864d1b0640 10 monclient(hunting): handle_auth_done global_id 64098 payload 824 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8624006260 con 0x7f8624003540 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 0x7f8624005a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8638000f60 tx=0x7f8638004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f863e7fc640 10 client.31393.objecter ms_handle_connect 0x7f8624003540 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f863e7fc640 10 client.31393.objecter resend_mon_ops 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8638018d80 con 0x7f8624003540 2024-06-11T19:13:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f863e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f863e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.204+0000 7f863e7fc640 10 monclient: dump: 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.207 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.205+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8638018910 con 0x7f8624003540 2024-06-11T19:13:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.205+0000 7f863e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.205+0000 7f863e7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.205+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 msgr2=0x7f8624005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.205+0000 7f863e7fc640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 0x7f8624005a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8638000f60 tx=0x7f8638004cc0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.205+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 msgr2=0x7f8624005a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.205+0000 7f863e7fc640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8624003540 0x7f8624005a00 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8638000f60 tx=0x7f8638004cc0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864fc15640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864fc15640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480636c0 0x7f864802cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864fc15640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864fc15640 1 -- 172.21.15.98:0/897215609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8648126dd0 con 0x7f86480636c0 2024-06-11T19:13:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480636c0 0x7f864802cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864d1b0640 10 monclient: get_auth_request con 0x7f86480636c0 auth_method 0 2024-06-11T19:13:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864d1b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864d1b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864d1b0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864d1b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864d1b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864d1b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864d1b0640 10 monclient(hunting): handle_auth_done global_id 64107 payload 824 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864d1b0640 1 -- 172.21.15.98:0/897215609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f86480a0c90 con 0x7f86480636c0 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f864d1b0640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480636c0 0x7f864802cb40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8638021000 tx=0x7f8638007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f863e7fc640 10 client.31393.objecter ms_handle_connect 0x7f86480636c0 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f863e7fc640 10 client.31393.objecter resend_mon_ops 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8638027bf0 con 0x7f86480636c0 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f863e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f863e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.212+0000 7f863e7fc640 10 monclient: dump: 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9271 (secure 0 0 0) 0x7f8638027780 con 0x7f86480636c0 2024-06-11T19:13:33.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f863e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:33.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f863e7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:33.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480636c0 msgr2=0x7f864802cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f863e7fc640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480636c0 0x7f864802cb40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8638021000 tx=0x7f8638007e30 comp rx=0 tx=0).stop 2024-06-11T19:13:33.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f863e7fc640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480636c0 msgr2=0x7f864802cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:33.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f863e7fc640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480636c0 0x7f864802cb40 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8638021000 tx=0x7f8638007e30 comp rx=0 tx=0).stop 2024-06-11T19:13:33.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864e1b2640 1 -- 172.21.15.98:0/897215609 reap_dead start 2024-06-11T19:13:33.216 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 5040, 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:13:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:33.238 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 10 librados: watch_flush enter 2024-06-11T19:13:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 10 librados: watch_flush exit 2024-06-11T19:13:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 20 client.31393.objecter shutdown clearing up homeless session... 2024-06-11T19:13:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 10 client.31393.objecter successfully canceled tick 2024-06-11T19:13:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f862c03fdb0 msgr2=0x7f862c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f862c03fdb0 0x7f862c042270 secure :-1 s=READY pgs=4064 cs=0 l=1 rev1=1 crypto rx=0x7f86481269d0 tx=0x7f8640009040 comp rx=0 tx=0).stop 2024-06-11T19:13:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 10 monclient: shutdown 2024-06-11T19:13:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 1 -- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86480a2530 msgr2=0x7f86481cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86480a2530 0x7f86481cda70 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f8644033000 tx=0x7f86440046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 1 -- 172.21.15.98:0/897215609 shutdown_connections 2024-06-11T19:13:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f862c03fdb0 0x7f862c042270 unknown :-1 s=CLOSED pgs=4064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 1 --2- 172.21.15.98:0/897215609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86480a2530 0x7f86481cda70 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 1 -- 172.21.15.98:0/897215609 >> 172.21.15.98:0/897215609 conn(0x7f86480ab850 msgr2=0x7f864809c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 1 -- 172.21.15.98:0/897215609 shutdown_connections 2024-06-11T19:13:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 1 -- 172.21.15.98:0/897215609 wait complete. 2024-06-11T19:13:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.213+0000 7f864fc15640 1 librados: shutdown 2024-06-11T19:13:33.241 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:33.242 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:13:33.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:13:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.347+0000 7f9e06333640 10 monclient: get_monmap_and_config 2024-06-11T19:13:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.347+0000 7f9e06333640 10 monclient: build_initial_monmap 2024-06-11T19:13:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.347+0000 7f9e06333640 10 monclient: monmap: 2024-06-11T19:13:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:33.348108+0000 2024-06-11T19:13:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:33.348108+0000 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:33.347 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:33.348 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.347+0000 7f9e06333640 1 Processor -- start 2024-06-11T19:13:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.347+0000 7f9e06333640 1 -- start start 2024-06-11T19:13:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.347+0000 7f9e06333640 10 monclient: init 2024-06-11T19:13:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 10 monclient: _add_conns ranks=[5,9,3,6,4,7,0,1,2,10,8] 2024-06-11T19:13:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9e00136c90 0x7f9e00137070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 10 monclient(hunting): picked mon.noname-h con 0x7f9e00136c90 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e001367d0 0x7f9e000a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 10 monclient(hunting): picked mon.noname-a con 0x7f9e001367d0 addr v2:172.21.15.98:3300/0 2024-06-11T19:13:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e000a3bd0 0x7f9e000a4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 10 monclient(hunting): picked mon.noname-g con 0x7f9e000a3bd0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9e00068180 con 0x7f9e001367d0 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f9e000a0270 con 0x7f9e000a3bd0 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f9e0009fc20 con 0x7f9e00136c90 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e06333640 10 monclient(hunting): authenticate will time out at 1023.155426s 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e001367d0 0x7f9e000a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e001367d0 0x7f9e000a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49796/0 (socket says 172.21.15.98:49796) 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 1 -- 172.21.15.98:0/3001516171 learned_addr learned my addr 172.21.15.98:0/3001516171 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 10 monclient(hunting): get_auth_request con 0x7f9e001367d0 auth_method 0 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 10 monclient(hunting): handle_auth_done global_id 64116 payload 274 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dfffff640 1 --2- 172.21.15.98:0/3001516171 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9e00136c90 0x7f9e00137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9e048d0640 1 --2- 172.21.15.98:0/3001516171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e000a3bd0 0x7f9e000a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 1 -- 172.21.15.98:0/3001516171 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9e00136c90 msgr2=0x7f9e00137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 1 --2- 172.21.15.98:0/3001516171 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9e00136c90 0x7f9e00137070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 1 -- 172.21.15.98:0/3001516171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e000a3bd0 msgr2=0x7f9e000a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 1 --2- 172.21.15.98:0/3001516171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e000a3bd0 0x7f9e000a4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:13:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 1 monclient: found mon.noname-a 2024-06-11T19:13:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:13:33.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.348+0000 7f9dff7fe640 1 -- 172.21.15.98:0/3001516171 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0009ff00 con 0x7f9e001367d0 2024-06-11T19:13:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dff7fe640 1 --2- 172.21.15.98:0/3001516171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e001367d0 0x7f9e000a3690 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009870 tx=0x7f9df402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2668b976b3fe41 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 1 -- 172.21.15.98:0/3001516171 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9df4031a30 con 0x7f9e001367d0 2024-06-11T19:13:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:13:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 10 monclient: dump: 2024-06-11T19:13:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.361 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.362 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 10 monclient: _finish_auth 0 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 10 monclient: _check_auth_tickets 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 1 -- 172.21.15.98:0/3001516171 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df4037070 con 0x7f9e001367d0 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9e06333640 5 monclient: authenticate success, global_id 64116 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9e06333640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9e06333640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 1 -- 172.21.15.98:0/3001516171 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9df4031bf0 con 0x7f9e001367d0 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9e06333640 4 set_mon_vals no callback set 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:33.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.349+0000 7f9dfeffd640 10 monclient: dump: 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.364 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 10 monclient: shutdown 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 1 -- 172.21.15.98:0/3001516171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e001367d0 msgr2=0x7f9e000a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 1 --2- 172.21.15.98:0/3001516171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e001367d0 0x7f9e000a3690 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009870 tx=0x7f9df402b860 comp rx=0 tx=0).stop 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 1 -- 172.21.15.98:0/3001516171 shutdown_connections 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 1 --2- 172.21.15.98:0/3001516171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e000a3bd0 0x7f9e000a4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 1 --2- 172.21.15.98:0/3001516171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e001367d0 0x7f9e000a3690 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 1 --2- 172.21.15.98:0/3001516171 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9e00136c90 0x7f9e00137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 1 -- 172.21.15.98:0/3001516171 >> 172.21.15.98:0/3001516171 conn(0x7f9e000a4780 msgr2=0x7f9e000a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 1 -- 172.21.15.98:0/3001516171 shutdown_connections 2024-06-11T19:13:33.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 1 -- 172.21.15.98:0/3001516171 wait complete. 2024-06-11T19:13:33.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 10 monclient: build_initial_monmap 2024-06-11T19:13:33.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.350+0000 7f9e06333640 10 monclient: monmap: 2024-06-11T19:13:33.366 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:33.366 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:33.366 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:33.366 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:33.366 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:33.366 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:33.366 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:33.366 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:33.366 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:33.366 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 1 librados: starting msgr at 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 1 librados: starting objecter 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 1 Processor -- start 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 1 -- start start 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 1 librados: setting wanted keys 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 1 librados: calling monclient init 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 10 monclient: init 2024-06-11T19:13:33.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 10 monclient: _add_conns ranks=[5,3,2,7,1,6,0,8,4] 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9e000a3bd0 0x7f9e001cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 10 monclient(hunting): picked mon.noname-h con 0x7f9e000a3bd0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e001367d0 0x7f9e001cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 10 monclient(hunting): picked mon.noname-g con 0x7f9e001367d0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00136c90 0x7f9e00123080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 10 monclient(hunting): picked mon.noname-b con 0x7f9e00136c90 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9e001375b0 con 0x7f9e00136c90 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f9e00068560 con 0x7f9e001367d0 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f9e00127a20 con 0x7f9e000a3bd0 2024-06-11T19:13:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e06333640 10 monclient(hunting): authenticate will time out at 1023.158824s 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e048d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00136c90 0x7f9e00123080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e048d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00136c90 0x7f9e00123080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55418/0 (socket says 172.21.15.98:55418) 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e048d0640 1 -- 172.21.15.98:0/4093112917 learned_addr learned my addr 172.21.15.98:0/4093112917 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e048d0640 10 monclient(hunting): get_auth_request con 0x7f9e00136c90 auth_method 0 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e048d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e048d0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.351+0000 7f9e048d0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfffff640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9e000a3bd0 0x7f9e001cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dff7fe640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e001367d0 0x7f9e001cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfffff640 10 monclient(hunting): get_auth_request con 0x7f9e000a3bd0 auth_method 0 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dff7fe640 10 monclient(hunting): get_auth_request con 0x7f9e001367d0 auth_method 0 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dff7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 10 monclient(hunting): handle_auth_done global_id 21998 payload 824 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 1 -- 172.21.15.98:0/4093112917 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9e000a3bd0 msgr2=0x7f9e001cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9e000a3bd0 0x7f9e001cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 1 -- 172.21.15.98:0/4093112917 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e001367d0 msgr2=0x7f9e001cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e001367d0 0x7f9e001cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 10 monclient: _finish_hunting 0 2024-06-11T19:13:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 1 monclient: found mon.noname-b 2024-06-11T19:13:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:13:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 1 -- 172.21.15.98:0/4093112917 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e001251a0 con 0x7f9e00136c90 2024-06-11T19:13:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e048d0640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00136c90 0x7f9e00123080 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f9e00125370 tx=0x7f9df800a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 10 client.?.objecter ms_handle_connect 0x7f9e00136c90 2024-06-11T19:13:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9df8016890 con 0x7f9e00136c90 2024-06-11T19:13:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:13:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 10 monclient: dump: 2024-06-11T19:13:33.371 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.372 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 10 monclient: _finish_auth 0 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 10 monclient: _check_auth_tickets 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df8016a70 con 0x7f9e00136c90 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e06333640 5 monclient: authenticate success, global_id 21998 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e06333640 10 monclient: _renew_subs 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e06333640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e06333640 1 -- 172.21.15.98:0/4093112917 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e00126ff0 con 0x7f9e00136c90 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9df8016c20 con 0x7f9e00136c90 2024-06-11T19:13:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfcff9640 10 monclient: dump: 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9dfe7fc640 4 set_mon_vals no callback set 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e06333640 10 client.21998.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e06333640 10 monclient: _renew_subs 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e06333640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e06333640 1 -- 172.21.15.98:0/4093112917 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e00126a30 con 0x7f9e00136c90 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.352+0000 7f9e06333640 1 librados: init done 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9e06333640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9e06333640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e001235c0 0x7f9e001239a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9e06333640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9e06333640 1 -- 172.21.15.98:0/4093112917 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9e00137730 con 0x7f9e001235c0 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfffff640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e001235c0 0x7f9e001239a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfffff640 10 monclient: get_auth_request con 0x7f9e001235c0 auth_method 0 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f9df801d020 con 0x7f9e00136c90 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfffff640 10 monclient(hunting): handle_auth_done global_id 22007 payload 824 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfffff640 1 -- 172.21.15.98:0/4093112917 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e00126750 con 0x7f9e001235c0 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfcff9640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9dec03fdb0 0x7f9dec042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(178..178 src has 1..178) ==== 7467+0+0 (secure 0 0 0) 0x7f9df8052740 con 0x7f9e00136c90 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfcff9640 10 client.21998.objecter ms_dispatch 0x7f9e00000b90 osd_map(178..178 src has 1..178) 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfcff9640 3 client.21998.objecter handle_osd_map got epochs [178,178] > 0 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfcff9640 3 client.21998.objecter handle_osd_map decoding full epoch 178 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.353+0000 7f9dfffff640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e001235c0 0x7f9e001239a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e570 tx=0x7f9df000ea30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 20 client.21998.objecter dump_active .. 0 homeless 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 10 client.21998.objecter ms_handle_connect 0x7f9e001235c0 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 10 client.21998.objecter resend_mon_ops 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9df0004240 con 0x7f9e001235c0 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dff7fe640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9dec03fdb0 0x7f9dec042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 10 monclient: dump: 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.379 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dff7fe640 10 monclient: get_auth_request con 0x7f9dec03fdb0 auth_method 0 2024-06-11T19:13:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9df001a070 con 0x7f9e001235c0 2024-06-11T19:13:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e001235c0 msgr2=0x7f9e001239a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e001235c0 0x7f9e001239a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e570 tx=0x7f9df000ea30 comp rx=0 tx=0).stop 2024-06-11T19:13:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e001235c0 msgr2=0x7f9e001239a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e001235c0 0x7f9e001239a0 secure :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e570 tx=0x7f9df000ea30 comp rx=0 tx=0).stop 2024-06-11T19:13:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dfcff9640 10 client.21998.objecter ms_handle_connect 0x7f9dec03fdb0 2024-06-11T19:13:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.354+0000 7f9dff7fe640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9dec03fdb0 0x7f9dec042270 secure :-1 s=READY pgs=4066 cs=0 l=1 rev1=1 crypto rx=0x7f9e000a8d40 tx=0x7f9df4008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.360+0000 7f9e06333640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.360+0000 7f9e06333640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00123ee0 0x7f9e0006f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.360+0000 7f9e06333640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.360+0000 7f9e06333640 1 -- 172.21.15.98:0/4093112917 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9e00050830 con 0x7f9e00123ee0 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.360+0000 7f9dfffff640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00123ee0 0x7f9e0006f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.360+0000 7f9dfffff640 10 monclient: get_auth_request con 0x7f9e00123ee0 auth_method 0 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.360+0000 7f9dfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.360+0000 7f9dfffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.360+0000 7f9dfffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfffff640 10 monclient(hunting): handle_auth_done global_id 22016 payload 824 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfffff640 1 -- 172.21.15.98:0/4093112917 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9e0009f2a0 con 0x7f9e00123ee0 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfffff640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00123ee0 0x7f9e0006f530 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e540 tx=0x7f9df00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 10 client.21998.objecter ms_handle_connect 0x7f9e00123ee0 2024-06-11T19:13:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 10 client.21998.objecter resend_mon_ops 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9df001f260 con 0x7f9e00123ee0 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 10 monclient: dump: 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f9df001f260 con 0x7f9e00123ee0 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00123ee0 msgr2=0x7f9e0006f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00123ee0 0x7f9e0006f530 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e540 tx=0x7f9df00046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 1 -- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00123ee0 msgr2=0x7f9e0006f530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:33.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9dfcff9640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00123ee0 0x7f9e0006f530 secure :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e540 tx=0x7f9df00046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.385 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 263594, 2024-06-11T19:13:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:33.385 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.386 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.387 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:33.388 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:33.389 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.390 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.392 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:33.393 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:33.399 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:13:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:33.408 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:33.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9e06333640 10 librados: watch_flush enter 2024-06-11T19:13:33.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9e06333640 10 librados: watch_flush exit 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9e06333640 20 client.21998.objecter shutdown clearing up homeless session... 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9e06333640 10 client.21998.objecter successfully canceled tick 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9e06333640 1 -- 172.21.15.98:0/4093112917 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9dec03fdb0 msgr2=0x7f9dec042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9e06333640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9dec03fdb0 0x7f9dec042270 secure :-1 s=READY pgs=4066 cs=0 l=1 rev1=1 crypto rx=0x7f9e000a8d40 tx=0x7f9df4008030 comp rx=0 tx=0).stop 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9e06333640 10 monclient: shutdown 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9e06333640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.361+0000 7f9e06333640 1 -- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00136c90 msgr2=0x7f9e00123080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.362+0000 7f9e048d0640 1 -- 172.21.15.98:0/4093112917 reap_dead start 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.362+0000 7f9e06333640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00136c90 0x7f9e00123080 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f9e00125370 tx=0x7f9df800a660 comp rx=0 tx=0).stop 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.362+0000 7f9e06333640 1 -- 172.21.15.98:0/4093112917 shutdown_connections 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.362+0000 7f9e06333640 1 --2- 172.21.15.98:0/4093112917 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9e00136c90 0x7f9e00123080 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.362+0000 7f9e06333640 1 -- 172.21.15.98:0/4093112917 >> 172.21.15.98:0/4093112917 conn(0x7f9e000a4780 msgr2=0x7f9e001bd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:33.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.362+0000 7f9e06333640 1 -- 172.21.15.98:0/4093112917 shutdown_connections 2024-06-11T19:13:33.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.362+0000 7f9e06333640 1 -- 172.21.15.98:0/4093112917 wait complete. 2024-06-11T19:13:33.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.362+0000 7f9e06333640 1 librados: shutdown 2024-06-11T19:13:33.411 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:33.411 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:13:33.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:13:33.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.511+0000 7fc2c37f0640 10 monclient: get_monmap_and_config 2024-06-11T19:13:33.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.511+0000 7fc2c37f0640 10 monclient: build_initial_monmap 2024-06-11T19:13:33.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.511+0000 7fc2c37f0640 10 monclient: monmap: 2024-06-11T19:13:33.511 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:33.511 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.511 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:33.512941+0000 2024-06-11T19:13:33.511 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:33.512941+0000 2024-06-11T19:13:33.511 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:33.512 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.512+0000 7fc2c37f0640 1 Processor -- start 2024-06-11T19:13:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.512+0000 7fc2c37f0640 1 -- start start 2024-06-11T19:13:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.512+0000 7fc2c37f0640 10 monclient: init 2024-06-11T19:13:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 10 monclient: _add_conns ranks=[7,10,4,3,2,1,0,9,6,5,8] 2024-06-11T19:13:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc2bc0a4920 0x7fc2bc0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 10 monclient(hunting): picked mon.noname-i con 0x7fc2bc0a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc2bc0a5240 0x7fc2bc0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 10 monclient(hunting): picked mon.noname-f con 0x7fc2bc0a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a2530 0x7fc2bc0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 10 monclient(hunting): picked mon.noname-c con 0x7fc2bc0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fc2bc0a7140 con 0x7fc2bc0a5240 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc2bc0a2b50 con 0x7fc2bc0a2530 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fc2bc0a2e30 con 0x7fc2bc0a4920 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c37f0640 10 monclient(hunting): authenticate will time out at 1023.320268s 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a2530 0x7fc2bc0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a2530 0x7fc2bc0a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52194/0 (socket says 172.21.15.98:52194) 2024-06-11T19:13:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 1 -- 172.21.15.98:0/2470230123 learned_addr learned my addr 172.21.15.98:0/2470230123 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 10 monclient(hunting): get_auth_request con 0x7fc2bc0a2530 auth_method 0 2024-06-11T19:13:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c158c640 1 --2- 172.21.15.98:0/2470230123 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc2bc0a4920 0x7fc2bc0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c0d8b640 1 --2- 172.21.15.98:0/2470230123 >> v2:172.21.15.117:3300/0 conn(0x7fc2bc0a5240 0x7fc2bc0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c158c640 10 monclient(hunting): get_auth_request con 0x7fc2bc0a4920 auth_method 0 2024-06-11T19:13:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c158c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c158c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c158c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 10 monclient(hunting): handle_auth_done global_id 23332 payload 274 2024-06-11T19:13:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 1 -- 172.21.15.98:0/2470230123 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc2bc0a4920 msgr2=0x7fc2bc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 1 --2- 172.21.15.98:0/2470230123 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc2bc0a4920 0x7fc2bc0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 1 -- 172.21.15.98:0/2470230123 >> v2:172.21.15.117:3300/0 conn(0x7fc2bc0a5240 msgr2=0x7fc2bc0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 1 --2- 172.21.15.98:0/2470230123 >> v2:172.21.15.117:3300/0 conn(0x7fc2bc0a5240 0x7fc2bc0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 10 monclient: _finish_hunting 0 2024-06-11T19:13:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 1 monclient: found mon.noname-c 2024-06-11T19:13:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c1d8d640 1 -- 172.21.15.98:0/2470230123 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2bc0a3110 con 0x7fc2bc0a2530 2024-06-11T19:13:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c0d8b640 10 monclient: get_auth_request con 0x7fc2bc0a5240 auth_method 0 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.513+0000 7fc2c0d8b640 1 --2- 172.21.15.98:0/2470230123 >> v2:172.21.15.117:3300/0 conn(0x7fc2bc0a5240 0x7fc2bc0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2c1d8d640 1 --2- 172.21.15.98:0/2470230123 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a2530 0x7fc2bc0a5ee0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009e50 tx=0x7fc2b402bdb0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=7dd9cf3556000f4d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 1 -- 172.21.15.98:0/2470230123 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2b402cbc0 con 0x7fc2bc0a2530 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 10 monclient: dump: 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.527 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 10 monclient: _finish_auth 0 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 10 monclient: _check_auth_tickets 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2c37f0640 5 monclient: authenticate success, global_id 23332 2024-06-11T19:13:33.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2c37f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 1 -- 172.21.15.98:0/2470230123 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2b402cd80 con 0x7fc2bc0a2530 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2c37f0640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2c37f0640 4 set_mon_vals no callback set 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 1 -- 172.21.15.98:0/2470230123 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2b4034860 con 0x7fc2bc0a2530 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.514+0000 7fc2b3fff640 10 monclient: dump: 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.529 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.530 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.530 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.530 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.530 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.530 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.530 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.530 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 10 monclient: shutdown 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2470230123 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a2530 msgr2=0x7fc2bc0a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 1 --2- 172.21.15.98:0/2470230123 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a2530 0x7fc2bc0a5ee0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009e50 tx=0x7fc2b402bdb0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2470230123 shutdown_connections 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 1 --2- 172.21.15.98:0/2470230123 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a2530 0x7fc2bc0a5ee0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 1 --2- 172.21.15.98:0/2470230123 >> v2:172.21.15.117:3300/0 conn(0x7fc2bc0a5240 0x7fc2bc0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 1 --2- 172.21.15.98:0/2470230123 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc2bc0a4920 0x7fc2bc0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2470230123 >> 172.21.15.98:0/2470230123 conn(0x7fc2bc0ab850 msgr2=0x7fc2bc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2470230123 shutdown_connections 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2470230123 wait complete. 2024-06-11T19:13:33.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 10 monclient: build_initial_monmap 2024-06-11T19:13:33.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.515+0000 7fc2c37f0640 10 monclient: monmap: 2024-06-11T19:13:33.532 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:33.532 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:33.532 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:33.532 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:33.532 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:33.532 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:33.532 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:33.533 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:33.533 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:33.533 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:33.533 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:33.533 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:33.533 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:33.533 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:33.533 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:33.534 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 1 librados: starting msgr at 2024-06-11T19:13:33.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 1 librados: starting objecter 2024-06-11T19:13:33.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 1 Processor -- start 2024-06-11T19:13:33.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 1 -- start start 2024-06-11T19:13:33.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 1 librados: setting wanted keys 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 1 librados: calling monclient init 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 10 monclient: init 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 10 monclient: _add_conns ranks=[2,1,4,8,7,3,5,6,0] 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2bc0a2530 0x7fc2bc1371c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 10 monclient(hunting): picked mon.noname-b con 0x7fc2bc0a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2bc0a4920 0x7fc2bc137700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 10 monclient(hunting): picked mon.noname-f con 0x7fc2bc0a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a5240 0x7fc2bc1231a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 10 monclient(hunting): picked mon.noname-c con 0x7fc2bc0a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc0a6da0 con 0x7fc2bc0a4920 2024-06-11T19:13:33.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc2bc138e50 con 0x7fc2bc0a2530 2024-06-11T19:13:33.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc2bc0a6420 con 0x7fc2bc0a5240 2024-06-11T19:13:33.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:33.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c37f0640 10 monclient(hunting): authenticate will time out at 1023.323757s 2024-06-11T19:13:33.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c158c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2bc0a2530 0x7fc2bc1371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c1d8d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a5240 0x7fc2bc1231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c158c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2bc0a2530 0x7fc2bc1371c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55444/0 (socket says 172.21.15.98:55444) 2024-06-11T19:13:33.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c1d8d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a5240 0x7fc2bc1231a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52202/0 (socket says 172.21.15.98:52202) 2024-06-11T19:13:33.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c158c640 1 -- 172.21.15.98:0/2309319320 learned_addr learned my addr 172.21.15.98:0/2309319320 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:33.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c158c640 10 monclient(hunting): get_auth_request con 0x7fc2bc0a2530 auth_method 0 2024-06-11T19:13:33.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c158c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c158c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c158c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c1d8d640 10 monclient(hunting): get_auth_request con 0x7fc2bc0a5240 auth_method 0 2024-06-11T19:13:33.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c1d8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c1d8d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c1d8d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c1d8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c1d8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c1d8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c158c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c158c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.516+0000 7fc2c158c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c0d8b640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2bc0a4920 0x7fc2bc137700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c0d8b640 10 monclient(hunting): get_auth_request con 0x7fc2bc0a4920 auth_method 0 2024-06-11T19:13:33.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c0d8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c0d8b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c0d8b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c1d8d640 10 monclient(hunting): handle_auth_done global_id 23341 payload 824 2024-06-11T19:13:33.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c1d8d640 1 -- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2bc0a2530 msgr2=0x7fc2bc1371c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c1d8d640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2bc0a2530 0x7fc2bc1371c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c1d8d640 1 -- 172.21.15.98:0/2309319320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2bc0a4920 msgr2=0x7fc2bc137700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c1d8d640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2bc0a4920 0x7fc2bc137700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c1d8d640 10 monclient: _finish_hunting 0 2024-06-11T19:13:33.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c1d8d640 1 monclient: found mon.noname-c 2024-06-11T19:13:33.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c1d8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:33.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c1d8d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:33.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c1d8d640 1 -- 172.21.15.98:0/2309319320 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2bc126b20 con 0x7fc2bc0a5240 2024-06-11T19:13:33.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c158c640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc2bc0a2530 0x7fc2bc1371c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:33.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c1d8d640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a5240 0x7fc2bc1231a0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4032000 tx=0x7fc2b4004a40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 10 client.?.objecter ms_handle_connect 0x7fc2bc0a5240 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2b4042020 con 0x7fc2bc0a5240 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 10 monclient: dump: 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.541 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.542 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 10 monclient: _finish_auth 0 2024-06-11T19:13:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:13:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2b40046c0 con 0x7fc2bc0a5240 2024-06-11T19:13:33.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c37f0640 5 monclient: authenticate success, global_id 23341 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c37f0640 10 monclient: _renew_subs 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c37f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2309319320 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2bc0a9080 con 0x7fc2bc0a5240 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2b403e8c0 con 0x7fc2bc0a5240 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b1ffb640 10 monclient: dump: 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.543 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.544 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.544 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.544 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.544 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.544 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.544 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.544 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.544 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.545 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.545 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c37f0640 10 client.23341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:33.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c37f0640 10 monclient: _renew_subs 2024-06-11T19:13:33.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c37f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:33.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2309319320 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2bc126e00 con 0x7fc2bc0a5240 2024-06-11T19:13:33.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2c37f0640 1 librados: init done 2024-06-11T19:13:33.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.517+0000 7fc2b37fe640 4 set_mon_vals no callback set 2024-06-11T19:13:33.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c37f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:33.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c37f0640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc1236e0 0x7fc2bc123ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c37f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2309319320 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc2bc050830 con 0x7fc2bc1236e0 2024-06-11T19:13:33.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fc2b4045020 con 0x7fc2bc0a5240 2024-06-11T19:13:33.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c158c640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc1236e0 0x7fc2bc123ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c158c640 10 monclient: get_auth_request con 0x7fc2bc1236e0 auth_method 0 2024-06-11T19:13:33.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c158c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c158c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c158c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c158c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c158c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c158c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2b1ffb640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc2a003fde0 0x7fc2a00422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c158c640 10 monclient(hunting): handle_auth_done global_id 23350 payload 824 2024-06-11T19:13:33.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(178..178 src has 1..178) ==== 7467+0+0 (secure 0 0 0) 0x7fc2b403a040 con 0x7fc2bc0a5240 2024-06-11T19:13:33.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2b1ffb640 10 client.23341.objecter ms_dispatch 0x7fc2bc000b90 osd_map(178..178 src has 1..178) 2024-06-11T19:13:33.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c158c640 1 -- 172.21.15.98:0/2309319320 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2bc0aaa30 con 0x7fc2bc1236e0 2024-06-11T19:13:33.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2b1ffb640 3 client.23341.objecter handle_osd_map got epochs [178,178] > 0 2024-06-11T19:13:33.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2b1ffb640 3 client.23341.objecter handle_osd_map decoding full epoch 178 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2b1ffb640 20 client.23341.objecter dump_active .. 0 homeless 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2b1ffb640 10 client.23341.objecter ms_handle_connect 0x7fc2bc1236e0 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2b1ffb640 10 client.23341.objecter resend_mon_ops 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c158c640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc1236e0 0x7fc2bc123ac0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8010c80 tx=0x7fc2b800e5a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.518+0000 7fc2c0d8b640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc2a003fde0 0x7fc2a00422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2c0d8b640 10 monclient: get_auth_request con 0x7fc2a003fde0 auth_method 0 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2b8018070 con 0x7fc2bc1236e0 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2b1ffb640 10 monclient: dump: 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.548 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc2b801d450 con 0x7fc2bc1236e0 2024-06-11T19:13:33.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2b1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:33.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2b1ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:33.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc1236e0 msgr2=0x7fc2bc123ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2b1ffb640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc1236e0 0x7fc2bc123ac0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8010c80 tx=0x7fc2b800e5a0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc1236e0 msgr2=0x7fc2bc123ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:33.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2b1ffb640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc1236e0 0x7fc2bc123ac0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2b1ffb640 10 client.23341.objecter ms_handle_connect 0x7fc2a003fde0 2024-06-11T19:13:33.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.519+0000 7fc2c0d8b640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc2a003fde0 0x7fc2a00422a0 secure :-1 s=READY pgs=4067 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0a5620 tx=0x7fc2ac008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.525+0000 7fc2c37f0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:33.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.525+0000 7fc2c37f0640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc124000 0x7fc2bc09e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.525+0000 7fc2c37f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.525+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2309319320 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc2bc069740 con 0x7fc2bc124000 2024-06-11T19:13:33.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.525+0000 7fc2c158c640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc124000 0x7fc2bc09e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.525+0000 7fc2c158c640 10 monclient: get_auth_request con 0x7fc2bc124000 auth_method 0 2024-06-11T19:13:33.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.525+0000 7fc2c158c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.525+0000 7fc2c158c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.525+0000 7fc2c158c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.525+0000 7fc2c158c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.525+0000 7fc2c158c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2c158c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2c158c640 10 monclient(hunting): handle_auth_done global_id 23359 payload 824 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2c158c640 1 -- 172.21.15.98:0/2309319320 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc2bc0a9e80 con 0x7fc2bc124000 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2c158c640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc124000 0x7fc2bc09e9f0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8010c80 tx=0x7fc2b80046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 10 client.23341.objecter ms_handle_connect 0x7fc2bc124000 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 10 client.23341.objecter resend_mon_ops 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2b801f2e0 con 0x7fc2bc124000 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:33.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 10 monclient: dump: 2024-06-11T19:13:33.554 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.554 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.554 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.554 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.554 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.554 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.554 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.554 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.554 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.555 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.555 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.555 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.555 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.555 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.555 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.555 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fc2b801f2e0 con 0x7fc2bc124000 2024-06-11T19:13:33.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:33.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:33.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc124000 msgr2=0x7fc2bc09e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc124000 0x7fc2bc09e9f0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8010c80 tx=0x7fc2b80046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 1 -- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc124000 msgr2=0x7fc2bc09e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:33.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2b1ffb640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc124000 0x7fc2bc09e9f0 secure :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8010c80 tx=0x7fc2b80046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.556 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 263744, 2024-06-11T19:13:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:13:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:33.569 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.570 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.572 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.576 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.577 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:33.579 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.580 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:33.581 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2c37f0640 10 librados: watch_flush enter 2024-06-11T19:13:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2c37f0640 10 librados: watch_flush exit 2024-06-11T19:13:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2c37f0640 20 client.23341.objecter shutdown clearing up homeless session... 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.526+0000 7fc2c37f0640 10 client.23341.objecter successfully canceled tick 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2309319320 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc2a003fde0 msgr2=0x7fc2a00422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c37f0640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc2a003fde0 0x7fc2a00422a0 secure :-1 s=READY pgs=4067 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0a5620 tx=0x7fc2ac008040 comp rx=0 tx=0).stop 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c37f0640 10 monclient: shutdown 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c37f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a5240 msgr2=0x7fc2bc1231a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c1d8d640 1 -- 172.21.15.98:0/2309319320 reap_dead start 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c37f0640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a5240 0x7fc2bc1231a0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4032000 tx=0x7fc2b4004a40 comp rx=0 tx=0).stop 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2309319320 shutdown_connections 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c37f0640 1 --2- 172.21.15.98:0/2309319320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2bc0a5240 0x7fc2bc1231a0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2309319320 >> 172.21.15.98:0/2309319320 conn(0x7fc2bc0ab850 msgr2=0x7fc2bc1bee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2309319320 shutdown_connections 2024-06-11T19:13:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c37f0640 1 -- 172.21.15.98:0/2309319320 wait complete. 2024-06-11T19:13:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.527+0000 7fc2c37f0640 1 librados: shutdown 2024-06-11T19:13:33.585 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:33.585 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:13:33.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:13:33.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.685+0000 7f3a9fe35640 10 monclient: get_monmap_and_config 2024-06-11T19:13:33.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.685+0000 7f3a9fe35640 10 monclient: build_initial_monmap 2024-06-11T19:13:33.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.685+0000 7f3a9fe35640 10 monclient: monmap: 2024-06-11T19:13:33.685 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:33.685 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.685 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:33.686596+0000 2024-06-11T19:13:33.685 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:33.686596+0000 2024-06-11T19:13:33.685 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:33.685 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.685 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:33.685 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:33.685 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:33.686 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:33.686 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:33.686 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:33.686 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:33.686 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:33.686 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:33.686 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:33.686 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:33.686 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 1 Processor -- start 2024-06-11T19:13:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 1 -- start start 2024-06-11T19:13:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 10 monclient: init 2024-06-11T19:13:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 10 monclient: _add_conns ranks=[0,4,10,8,9,2,6,7,1,5,3] 2024-06-11T19:13:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a980a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:13:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 0x7f3a9813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 10 monclient(hunting): picked mon.noname-c con 0x7f3a980a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3a9813e1a0 0x7f3a98140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 10 monclient(hunting): picked mon.noname-f con 0x7f3a9813e1a0 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a980a7140 con 0x7f3a980a4920 2024-06-11T19:13:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f3a980a2b50 con 0x7f3a9813e1a0 2024-06-11T19:13:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3a980a2e30 con 0x7f3a980a2530 2024-06-11T19:13:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9fe35640 10 monclient(hunting): authenticate will time out at 1023.493851s 2024-06-11T19:13:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9d3d0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 0x7f3a9813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9dbd1640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3a980a4920 0x7f3a980a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50116/0 (socket says 172.21.15.98:50116) 2024-06-11T19:13:33.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.686+0000 7f3a9dbd1640 1 -- 172.21.15.98:0/1686902276 learned_addr learned my addr 172.21.15.98:0/1686902276 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 10 monclient(hunting): get_auth_request con 0x7f3a980a2530 auth_method 0 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9e3d2640 1 --2- 172.21.15.98:0/1686902276 >> v2:172.21.15.117:3300/0 conn(0x7f3a9813e1a0 0x7f3a98140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9cbcf640 1 -- 172.21.15.98:0/1686902276 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1339403499 0 0) 0x7f3a980a7140 con 0x7f3a980a4920 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9cbcf640 10 monclient(hunting): _negotiate 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9cbcf640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9cbcf640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9cbcf640 10 monclient(hunting): my global_id is 64125 2024-06-11T19:13:33.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9cbcf640 1 -- 172.21.15.98:0/1686902276 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a84003520 con 0x7f3a980a4920 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9e3d2640 10 monclient(hunting): get_auth_request con 0x7f3a9813e1a0 auth_method 0 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9e3d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9e3d2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9e3d2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_done global_id 23368 payload 274 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 1 -- 172.21.15.98:0/1686902276 >> v2:172.21.15.117:3300/0 conn(0x7f3a9813e1a0 msgr2=0x7f3a98140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 1 --2- 172.21.15.98:0/1686902276 >> v2:172.21.15.117:3300/0 conn(0x7f3a9813e1a0 0x7f3a98140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 1 -- 172.21.15.98:0/1686902276 >> v1:172.21.15.98:6789/0 conn(0x7f3a980a4920 legacy=0x7f3a980a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 10 monclient: _finish_hunting 0 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 1 monclient: found mon.noname-c 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.687+0000 7f3a9d3d0640 1 -- 172.21.15.98:0/1686902276 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a980a3110 con 0x7f3a980a2530 2024-06-11T19:13:33.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9d3d0640 1 --2- 172.21.15.98:0/1686902276 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 0x7f3a9813dc60 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f3a880097a0 tx=0x7f3a8802b7a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1f36b76505aa5583 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 1 -- 172.21.15.98:0/1686902276 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a88038070 con 0x7f3a980a2530 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 10 monclient: dump: 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.700 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.701 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.701 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.701 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.701 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.701 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 10 monclient: _finish_auth 0 2024-06-11T19:13:33.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 10 monclient: _check_auth_tickets 2024-06-11T19:13:33.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:33.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 5 monclient: authenticate success, global_id 23368 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 1 -- 172.21.15.98:0/1686902276 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a88031970 con 0x7f3a980a2530 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 4 set_mon_vals no callback set 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 1 -- 172.21.15.98:0/1686902276 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a88032390 con 0x7f3a980a2530 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9cbcf640 10 monclient: dump: 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.702 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 10 monclient: shutdown 2024-06-11T19:13:33.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 1 -- 172.21.15.98:0/1686902276 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 msgr2=0x7f3a9813dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 1 --2- 172.21.15.98:0/1686902276 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 0x7f3a9813dc60 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f3a880097a0 tx=0x7f3a8802b7a0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 1 -- 172.21.15.98:0/1686902276 shutdown_connections 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 1 --2- 172.21.15.98:0/1686902276 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 0x7f3a9813dc60 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 1 --2- 172.21.15.98:0/1686902276 >> v2:172.21.15.117:3300/0 conn(0x7f3a9813e1a0 0x7f3a98140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 1 -- 172.21.15.98:0/1686902276 >> 172.21.15.98:0/1686902276 conn(0x7f3a980ab850 msgr2=0x7f3a980a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 1 -- 172.21.15.98:0/1686902276 shutdown_connections 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.688+0000 7f3a9fe35640 1 -- 172.21.15.98:0/1686902276 wait complete. 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.689+0000 7f3a9fe35640 10 monclient: build_initial_monmap 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.689+0000 7f3a9fe35640 10 monclient: monmap: 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:33.704 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.689+0000 7f3a9fe35640 1 librados: starting msgr at 2024-06-11T19:13:33.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.689+0000 7f3a9fe35640 1 librados: starting objecter 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.689+0000 7f3a9fe35640 1 Processor -- start 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.689+0000 7f3a9fe35640 1 -- start start 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.689+0000 7f3a9fe35640 1 librados: setting wanted keys 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.689+0000 7f3a9fe35640 1 librados: calling monclient init 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.689+0000 7f3a9fe35640 10 monclient: init 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 10 monclient: _add_conns ranks=[4,6,5,7,3,0,2,1,8] 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 0x7f3a981cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 10 monclient(hunting): picked mon.noname-c con 0x7f3a980a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a980a4920 0x7f3a981369c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 10 monclient(hunting): picked mon.noname-d con 0x7f3a980a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a9813e1a0 0x7f3a98136f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 10 monclient(hunting): picked mon.noname-h con 0x7f3a9813e1a0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3a98138e50 con 0x7f3a980a2530 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f3a980a84b0 con 0x7f3a9813e1a0 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3a980a6da0 con 0x7f3a980a4920 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 10 monclient(hunting): authenticate will time out at 1023.497242s 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 0x7f3a981cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 0x7f3a981cd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52232/0 (socket says 172.21.15.98:52232) 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 1 -- 172.21.15.98:0/2030758633 learned_addr learned my addr 172.21.15.98:0/2030758633 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 10 monclient(hunting): get_auth_request con 0x7f3a980a2530 auth_method 0 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9d3d0640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a980a4920 0x7f3a981369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9d3d0640 10 monclient(hunting): get_auth_request con 0x7f3a980a4920 auth_method 0 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9d3d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9d3d0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9d3d0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9e3d2640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a9813e1a0 0x7f3a98136f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9e3d2640 10 monclient(hunting): get_auth_request con 0x7f3a9813e1a0 auth_method 0 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9e3d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9e3d2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9e3d2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9e3d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9e3d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9e3d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 10 monclient(hunting): handle_auth_done global_id 23377 payload 824 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 1 -- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a980a4920 msgr2=0x7f3a981369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a980a4920 0x7f3a981369c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 1 -- 172.21.15.98:0/2030758633 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a9813e1a0 msgr2=0x7f3a98136f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a9813e1a0 0x7f3a98136f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 10 monclient: _finish_hunting 0 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 1 monclient: found mon.noname-c 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:33.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 1 -- 172.21.15.98:0/2030758633 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a98123370 con 0x7f3a980a2530 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9d3d0640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a980a4920 0x7f3a981369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9dbd1640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 0x7f3a981cd510 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400c500 tx=0x7f3a9400c9c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 10 client.?.objecter ms_handle_connect 0x7f3a980a2530 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a94007070 con 0x7f3a980a2530 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 10 monclient: dump: 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.710 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.711 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.711 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.711 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.711 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.711 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.711 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.711 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.711 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.711 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.711 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 10 monclient: _finish_auth 0 2024-06-11T19:13:33.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 5 monclient: authenticate success, global_id 23377 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a94007250 con 0x7f3a980a2530 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 10 monclient: _renew_subs 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a9fe35640 1 -- 172.21.15.98:0/2030758633 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a981ce420 con 0x7f3a980a2530 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a94012400 con 0x7f3a980a2530 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8e7fc640 10 monclient: dump: 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.712 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.713 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.690+0000 7f3a8ffff640 4 set_mon_vals no callback set 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9fe35640 10 client.23377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9fe35640 10 monclient: _renew_subs 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9fe35640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9fe35640 1 -- 172.21.15.98:0/2030758633 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a980a15c0 con 0x7f3a980a2530 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9fe35640 1 librados: init done 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9fe35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9fe35640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a9809f990 0x7f3a9809fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9fe35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9fe35640 1 -- 172.21.15.98:0/2030758633 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3a98050830 con 0x7f3a9809f990 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3a94007440 con 0x7f3a980a2530 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9d3d0640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a9809f990 0x7f3a9809fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9d3d0640 10 monclient: get_auth_request con 0x7f3a9809f990 auth_method 0 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9d3d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9d3d0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.691+0000 7f3a9d3d0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_done global_id 54618 payload 824 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a9d3d0640 1 -- 172.21.15.98:0/2030758633 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a98122c60 con 0x7f3a9809f990 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3a7c03fdb0 0x7f3a7c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(178..178 src has 1..178) ==== 7467+0+0 (secure 0 0 0) 0x7f3a94051ae0 con 0x7f3a980a2530 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 10 client.23377.objecter ms_dispatch 0x7f3a98000b90 osd_map(178..178 src has 1..178) 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 3 client.23377.objecter handle_osd_map got epochs [178,178] > 0 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 3 client.23377.objecter handle_osd_map decoding full epoch 178 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a9d3d0640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a9809f990 0x7f3a9809fd70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3a98073de0 tx=0x7f3a880046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a9e3d2640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3a7c03fdb0 0x7f3a7c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a9e3d2640 10 monclient: get_auth_request con 0x7f3a7c03fdb0 auth_method 0 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 20 client.23377.objecter dump_active .. 0 homeless 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 10 client.23377.objecter ms_handle_connect 0x7f3a9809f990 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 10 client.23377.objecter resend_mon_ops 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a88038040 con 0x7f3a9809f990 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 10 monclient: dump: 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.716 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3a88039040 con 0x7f3a9809f990 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a9809f990 msgr2=0x7f3a9809fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a9809f990 0x7f3a9809fd70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3a98073de0 tx=0x7f3a880046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a9809f990 msgr2=0x7f3a9809fd70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:33.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a9809f990 0x7f3a9809fd70 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3a98073de0 tx=0x7f3a880046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a8e7fc640 10 client.23377.objecter ms_handle_connect 0x7f3a7c03fdb0 2024-06-11T19:13:33.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.692+0000 7f3a9e3d2640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3a7c03fdb0 0x7f3a7c042270 secure :-1 s=READY pgs=4068 cs=0 l=1 rev1=1 crypto rx=0x7f3a98123860 tx=0x7f3a90009780 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9fe35640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:33.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9fe35640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a980a02b0 0x7f3a9813bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9fe35640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9fe35640 1 -- 172.21.15.98:0/2030758633 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3a98121940 con 0x7f3a980a02b0 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9d3d0640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a980a02b0 0x7f3a9813bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9d3d0640 10 monclient: get_auth_request con 0x7f3a980a02b0 auth_method 0 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9d3d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9d3d0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9d3d0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9d3d0640 10 monclient(hunting): handle_auth_done global_id 54627 payload 824 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9d3d0640 1 -- 172.21.15.98:0/2030758633 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3a980aa620 con 0x7f3a980a02b0 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a9d3d0640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a980a02b0 0x7f3a9813bbc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3a880097a0 tx=0x7f3a88031000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a8e7fc640 10 client.23377.objecter ms_handle_connect 0x7f3a980a02b0 2024-06-11T19:13:33.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a8e7fc640 10 client.23377.objecter resend_mon_ops 2024-06-11T19:13:33.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a88003bf0 con 0x7f3a980a02b0 2024-06-11T19:13:33.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a8e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:33.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.699+0000 7f3a8e7fc640 10 monclient: dump: 2024-06-11T19:13:33.721 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.721 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.721 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.721 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.721 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.721 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.721 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.722 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.722 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.722 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.722 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.722 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.722 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.722 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.722 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.722 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f3a88003bf0 con 0x7f3a980a02b0 2024-06-11T19:13:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a8e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a8e7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a980a02b0 msgr2=0x7f3a9813bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a8e7fc640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a980a02b0 0x7f3a9813bbc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3a880097a0 tx=0x7f3a88031000 comp rx=0 tx=0).stop 2024-06-11T19:13:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a8e7fc640 1 -- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a980a02b0 msgr2=0x7f3a9813bbc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:33.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a8e7fc640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a980a02b0 0x7f3a9813bbc0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3a880097a0 tx=0x7f3a88031000 comp rx=0 tx=0).stop 2024-06-11T19:13:33.723 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:33.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:33.723 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:33.723 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:33.724 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:33.724 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 126455, 2024-06-11T19:13:33.724 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:33.724 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:33.724 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:33.725 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:33.725 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:33.725 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:33.725 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:33.725 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:33.726 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:33.726 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:33.726 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.726 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:13:33.726 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:33.727 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:33.727 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:33.727 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.727 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.727 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.727 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.727 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.728 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.728 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.728 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.728 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.728 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.729 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.729 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.729 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:33.729 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:33.729 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.729 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.729 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.729 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.729 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.729 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.729 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.729 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:33.730 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.731 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:33.733 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.734 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.734 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.734 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:33.734 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:33.734 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.734 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.734 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:33.734 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.736 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.738 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:33.739 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.741 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.743 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.746 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:33.746 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:33.746 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.746 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.746 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:33.746 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.747 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.747 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:33.747 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.747 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:33.748 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:33.749 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:33.749 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.749 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.749 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:33.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.749 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:33.749 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:33.749 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:33.749 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.749 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.750 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:33.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.750 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:33.750 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:33.750 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:33.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.750 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.750 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:33.750 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 10 librados: watch_flush enter 2024-06-11T19:13:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 10 librados: watch_flush exit 2024-06-11T19:13:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 20 client.23377.objecter shutdown clearing up homeless session... 2024-06-11T19:13:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 10 client.23377.objecter successfully canceled tick 2024-06-11T19:13:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 1 -- 172.21.15.98:0/2030758633 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3a7c03fdb0 msgr2=0x7f3a7c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3a7c03fdb0 0x7f3a7c042270 secure :-1 s=READY pgs=4068 cs=0 l=1 rev1=1 crypto rx=0x7f3a98123860 tx=0x7f3a90009780 comp rx=0 tx=0).stop 2024-06-11T19:13:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 10 monclient: shutdown 2024-06-11T19:13:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 1 -- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 msgr2=0x7f3a981cd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9e3d2640 1 -- 172.21.15.98:0/2030758633 reap_dead start 2024-06-11T19:13:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 0x7f3a981cd510 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400c500 tx=0x7f3a9400c9c0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 1 -- 172.21.15.98:0/2030758633 shutdown_connections 2024-06-11T19:13:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 1 --2- 172.21.15.98:0/2030758633 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a980a2530 0x7f3a981cd510 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 1 -- 172.21.15.98:0/2030758633 >> 172.21.15.98:0/2030758633 conn(0x7f3a980ab850 msgr2=0x7f3a98137960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 1 -- 172.21.15.98:0/2030758633 shutdown_connections 2024-06-11T19:13:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 1 -- 172.21.15.98:0/2030758633 wait complete. 2024-06-11T19:13:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.700+0000 7f3a9fe35640 1 librados: shutdown 2024-06-11T19:13:33.754 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:33.754 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:13:33.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:13:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.854+0000 7f558bdd2640 10 monclient: get_monmap_and_config 2024-06-11T19:13:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.854+0000 7f558bdd2640 10 monclient: build_initial_monmap 2024-06-11T19:13:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.854+0000 7f558bdd2640 10 monclient: monmap: 2024-06-11T19:13:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:33.855632+0000 2024-06-11T19:13:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:33.855632+0000 2024-06-11T19:13:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:33.854 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:33.855 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:33.855 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 1 Processor -- start 2024-06-11T19:13:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 1 -- start start 2024-06-11T19:13:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 10 monclient: init 2024-06-11T19:13:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:33.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 10 monclient: _add_conns ranks=[8,2,4,10,9,3,0,6,5,1,7] 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841347f0 0x7f558409e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 10 monclient(hunting): picked mon.noname-e con 0x7f55841347f0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5584134bd0 0x7f55840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 10 monclient(hunting): picked mon.noname-b con 0x7f5584134bd0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5584135010 0x7f55840a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 10 monclient(hunting): picked mon.noname-c con 0x7f5584135010 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f55840687f0 con 0x7f5584134bd0 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f558409d410 con 0x7f5584135010 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f558409d6f0 con 0x7f55841347f0 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.855+0000 7f558bdd2640 10 monclient(hunting): authenticate will time out at 1023.662933s 2024-06-11T19:13:33.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5584134bd0 0x7f55840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f5589b6e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841347f0 0x7f558409e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5584134bd0 0x7f55840a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55452/0 (socket says 172.21.15.98:55452) 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 1 -- 172.21.15.98:0/3324265967 learned_addr learned my addr 172.21.15.98:0/3324265967 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 10 monclient(hunting): get_auth_request con 0x7f5584134bd0 auth_method 0 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f5589b6e640 1 --2- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841347f0 0x7f558409e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35324/0 (socket says 172.21.15.98:35324) 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f5589b6e640 10 monclient(hunting): get_auth_request con 0x7f55841347f0 auth_method 0 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f5589b6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558a36f640 1 --2- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5584135010 0x7f55840a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f5589b6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f5589b6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558a36f640 10 monclient(hunting): get_auth_request con 0x7f5584135010 auth_method 0 2024-06-11T19:13:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558a36f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558a36f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558a36f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f5589b6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f5589b6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f5589b6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558a36f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558a36f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558a36f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 10 monclient(hunting): handle_auth_done global_id 22034 payload 274 2024-06-11T19:13:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 1 -- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841347f0 msgr2=0x7f558409e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 1 --2- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841347f0 0x7f558409e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 1 -- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5584135010 msgr2=0x7f55840a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 1 --2- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5584135010 0x7f55840a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 10 monclient: _finish_hunting 0 2024-06-11T19:13:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 1 monclient: found mon.noname-b 2024-06-11T19:13:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:13:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f558936d640 1 -- 172.21.15.98:0/3324265967 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558409d9d0 con 0x7f5584134bd0 2024-06-11T19:13:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.856+0000 7f5589b6e640 1 --2- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841347f0 0x7f558409e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.857+0000 7f558936d640 1 --2- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5584134bd0 0x7f55840a3690 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f55740098a0 tx=0x7f557402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75c18a9c13e1a620 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 1 -- 172.21.15.98:0/3324265967 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5574031a10 con 0x7f5584134bd0 2024-06-11T19:13:33.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:13:33.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 10 monclient: dump: 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.865 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.866 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 10 monclient: _finish_auth 0 2024-06-11T19:13:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 10 monclient: _check_auth_tickets 2024-06-11T19:13:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 1 -- 172.21.15.98:0/3324265967 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5574037070 con 0x7f5584134bd0 2024-06-11T19:13:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 1 -- 172.21.15.98:0/3324265967 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5574031c10 con 0x7f5584134bd0 2024-06-11T19:13:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:33.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.860+0000 7f5588b6c640 10 monclient: dump: 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.867 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.861+0000 7f558bdd2640 5 monclient: authenticate success, global_id 22034 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.861+0000 7f558bdd2640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.861+0000 7f558bdd2640 4 set_mon_vals no callback set 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.862+0000 7f558bdd2640 10 monclient: shutdown 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.862+0000 7f558bdd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.862+0000 7f558bdd2640 1 -- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5584134bd0 msgr2=0x7f55840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.862+0000 7f558bdd2640 1 --2- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5584134bd0 0x7f55840a3690 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f55740098a0 tx=0x7f557402b840 comp rx=0 tx=0).stop 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.863+0000 7f558bdd2640 1 -- 172.21.15.98:0/3324265967 shutdown_connections 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.863+0000 7f558bdd2640 1 --2- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5584135010 0x7f55840a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.863+0000 7f558bdd2640 1 --2- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5584134bd0 0x7f55840a3690 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.863+0000 7f558bdd2640 1 --2- 172.21.15.98:0/3324265967 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841347f0 0x7f558409e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.863+0000 7f558bdd2640 1 -- 172.21.15.98:0/3324265967 >> 172.21.15.98:0/3324265967 conn(0x7f55840a4780 msgr2=0x7f55840a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.864+0000 7f558bdd2640 1 -- 172.21.15.98:0/3324265967 shutdown_connections 2024-06-11T19:13:33.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.864+0000 7f558bdd2640 1 -- 172.21.15.98:0/3324265967 wait complete. 2024-06-11T19:13:33.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.865+0000 7f558bdd2640 10 monclient: build_initial_monmap 2024-06-11T19:13:33.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.865+0000 7f558bdd2640 10 monclient: monmap: 2024-06-11T19:13:33.876 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:33.876 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:33.876 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:33.876 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:33.876 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:33.876 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.866+0000 7f558bdd2640 1 librados: starting msgr at 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.866+0000 7f558bdd2640 1 librados: starting objecter 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.866+0000 7f558bdd2640 1 Processor -- start 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.866+0000 7f558bdd2640 1 -- start start 2024-06-11T19:13:33.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.866+0000 7f558bdd2640 1 librados: setting wanted keys 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.866+0000 7f558bdd2640 1 librados: calling monclient init 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 10 monclient: init 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 10 monclient: _add_conns ranks=[5,7,6,2,0,8,1,4,3] 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f55841347f0 0x7f5584133660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 10 monclient(hunting): picked mon.noname-h con 0x7f55841347f0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5584134bd0 0x7f558411ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 10 monclient(hunting): picked mon.noname-i con 0x7f5584134bd0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5584135010 0x7f558411f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 10 monclient(hunting): picked mon.noname-d con 0x7f5584135010 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f5584068c40 con 0x7f55841347f0 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5584068ac0 con 0x7f5584135010 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f55840a4110 con 0x7f5584134bd0 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.867+0000 7f558bdd2640 10 monclient(hunting): authenticate will time out at 1023.674873s 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558936d640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5584134bd0 0x7f558411ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558936d640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5584134bd0 0x7f558411ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:46616/0 (socket says 172.21.15.98:46616) 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558936d640 1 -- 172.21.15.98:0/1027665817 learned_addr learned my addr 172.21.15.98:0/1027665817 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558936d640 10 monclient(hunting): get_auth_request con 0x7f5584134bd0 auth_method 0 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558936d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558936d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558936d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5584135010 0x7f558411f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 10 monclient(hunting): get_auth_request con 0x7f5584135010 auth_method 0 2024-06-11T19:13:33.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f5589b6e640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f55841347f0 0x7f5584133660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558936d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558936d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558936d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f5589b6e640 10 monclient(hunting): get_auth_request con 0x7f55841347f0 auth_method 0 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f5589b6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f5589b6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f5589b6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 10 monclient(hunting): handle_auth_done global_id 54636 payload 824 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 1 -- 172.21.15.98:0/1027665817 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5584134bd0 msgr2=0x7f558411ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5584134bd0 0x7f558411ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 1 -- 172.21.15.98:0/1027665817 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f55841347f0 msgr2=0x7f5584133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f55841347f0 0x7f5584133660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 10 monclient: _finish_hunting 0 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 1 monclient: found mon.noname-d 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.868+0000 7f558a36f640 1 -- 172.21.15.98:0/1027665817 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55841ce9d0 con 0x7f5584135010 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f558a36f640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5584135010 0x7f558411f320 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5584073990 tx=0x7f557c00c490 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 10 client.?.objecter ms_handle_connect 0x7f5584135010 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f557c018020 con 0x7f5584135010 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 10 monclient: dump: 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.882 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f558bdd2640 5 monclient: authenticate success, global_id 54636 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f557c014480 con 0x7f5584135010 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557bfff640 4 set_mon_vals no callback set 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f557c014aa0 con 0x7f5584135010 2024-06-11T19:13:33.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f558bdd2640 10 monclient: _renew_subs 2024-06-11T19:13:33.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f558bdd2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:33.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f558bdd2640 1 -- 172.21.15.98:0/1027665817 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5584120ab0 con 0x7f5584135010 2024-06-11T19:13:33.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f558bdd2640 10 client.54636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:33.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f558bdd2640 10 monclient: _renew_subs 2024-06-11T19:13:33.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f558bdd2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:33.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f558bdd2640 1 -- 172.21.15.98:0/1027665817 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f5584135d20 con 0x7f5584135010 2024-06-11T19:13:33.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f558bdd2640 1 librados: init done 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.869+0000 7f557a7fc640 10 monclient: dump: 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.885 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.886 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.886 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.886 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.886 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.886 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f558bdd2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:33.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f557c005d10 con 0x7f5584135010 2024-06-11T19:13:33.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f558bdd2640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841c0d00 0x7f558411f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f558bdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f558bdd2640 1 -- 172.21.15.98:0/1027665817 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5584050830 con 0x7f55841c0d00 2024-06-11T19:13:33.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f5589b6e640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841c0d00 0x7f558411f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f557a7fc640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f556803fdb0 0x7f5568042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f5589b6e640 10 monclient: get_auth_request con 0x7f55841c0d00 auth_method 0 2024-06-11T19:13:33.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f5589b6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f5589b6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f5589b6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(179..179 src has 1..179) ==== 6757+0+0 (secure 0 0 0) 0x7f557c0527e0 con 0x7f5584135010 2024-06-11T19:13:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f557a7fc640 10 client.54636.objecter ms_dispatch 0x7f5584000b90 osd_map(179..179 src has 1..179) 2024-06-11T19:13:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f557a7fc640 3 client.54636.objecter handle_osd_map got epochs [179,179] > 0 2024-06-11T19:13:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f557a7fc640 3 client.54636.objecter handle_osd_map decoding full epoch 179 2024-06-11T19:13:33.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f5589b6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f5589b6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f5589b6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f557a7fc640 20 client.54636.objecter dump_active .. 0 homeless 2024-06-11T19:13:33.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f5589b6e640 10 monclient(hunting): handle_auth_done global_id 64115 payload 824 2024-06-11T19:13:33.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f5589b6e640 1 -- 172.21.15.98:0/1027665817 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5584075170 con 0x7f55841c0d00 2024-06-11T19:13:33.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f558936d640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f556803fdb0 0x7f5568042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.871+0000 7f558936d640 10 monclient: get_auth_request con 0x7f556803fdb0 auth_method 0 2024-06-11T19:13:33.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f5589b6e640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841c0d00 0x7f558411f860 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5580011590 tx=0x7f5580011a50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 10 client.54636.objecter ms_handle_connect 0x7f55841c0d00 2024-06-11T19:13:33.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 10 client.54636.objecter resend_mon_ops 2024-06-11T19:13:33.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f558000fb80 con 0x7f55841c0d00 2024-06-11T19:13:33.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:13:33.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 10 monclient: dump: 2024-06-11T19:13:33.890 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.890 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.890 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 10 client.54636.objecter ms_handle_connect 0x7f556803fdb0 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f558000f710 con 0x7f55841c0d00 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841c0d00 msgr2=0x7f558411f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841c0d00 0x7f558411f860 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5580011590 tx=0x7f5580011a50 comp rx=0 tx=0).stop 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f558936d640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f556803fdb0 0x7f5568042270 secure :-1 s=READY pgs=4069 cs=0 l=1 rev1=1 crypto rx=0x7f5574034040 tx=0x7f557403d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841c0d00 msgr2=0x7f558411f860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:33.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.872+0000 7f557a7fc640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f55841c0d00 0x7f558411f860 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5580011590 tx=0x7f5580011a50 comp rx=0 tx=0).stop 2024-06-11T19:13:33.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.880+0000 7f558bdd2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.880+0000 7f558bdd2640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5584122900 0x7f558411fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.880+0000 7f558bdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.880+0000 7f558bdd2640 1 -- 172.21.15.98:0/1027665817 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f558409e6b0 con 0x7f5584122900 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.880+0000 7f5589b6e640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5584122900 0x7f558411fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.880+0000 7f5589b6e640 10 monclient: get_auth_request con 0x7f5584122900 auth_method 0 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.880+0000 7f5589b6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.880+0000 7f5589b6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.880+0000 7f5589b6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.880+0000 7f5589b6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.880+0000 7f5589b6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.880+0000 7f5589b6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f5589b6e640 10 monclient(hunting): handle_auth_done global_id 64124 payload 824 2024-06-11T19:13:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f5589b6e640 1 -- 172.21.15.98:0/1027665817 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f558406d540 con 0x7f5584122900 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f5589b6e640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5584122900 0x7f558411fda0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5580011590 tx=0x7f55800046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 10 client.54636.objecter ms_handle_connect 0x7f5584122900 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 10 client.54636.objecter resend_mon_ops 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f55800064b0 con 0x7f5584122900 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 10 monclient: dump: 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9271 (secure 0 0 0) 0x7f5580005f60 con 0x7f5584122900 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5584122900 msgr2=0x7f558411fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5584122900 0x7f558411fda0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5580011590 tx=0x7f55800046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 1 -- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5584122900 msgr2=0x7f558411fda0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:33.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f557a7fc640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5584122900 0x7f558411fda0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:33.898 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:33.898 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:33.898 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:33.898 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:33.898 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:33.898 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 5708, 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:33.899 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:33.900 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.901 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:33.902 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:33.902 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:33.902 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:33.902 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:33.902 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:33.902 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:33.902 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:33.902 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:33.902 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:33.902 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:33.902 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:33.902 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:33.906 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.908 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:33.910 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:33.913 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.913 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:33.913 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:33.913 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.913 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:33.915 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:33.916 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:33.919 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:33.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:33.920 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:33.920 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:33.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f558bdd2640 10 librados: watch_flush enter 2024-06-11T19:13:33.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f558bdd2640 10 librados: watch_flush exit 2024-06-11T19:13:33.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f558bdd2640 20 client.54636.objecter shutdown clearing up homeless session... 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.881+0000 7f558bdd2640 10 client.54636.objecter successfully canceled tick 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558bdd2640 1 -- 172.21.15.98:0/1027665817 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f556803fdb0 msgr2=0x7f5568042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558bdd2640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f556803fdb0 0x7f5568042270 secure :-1 s=READY pgs=4069 cs=0 l=1 rev1=1 crypto rx=0x7f5574034040 tx=0x7f557403d040 comp rx=0 tx=0).stop 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558bdd2640 10 monclient: shutdown 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558bdd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558bdd2640 1 -- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5584135010 msgr2=0x7f558411f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558bdd2640 1 --2- 172.21.15.98:0/1027665817 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5584135010 0x7f558411f320 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5584073990 tx=0x7f557c00c490 comp rx=0 tx=0).stop 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558a36f640 1 -- 172.21.15.98:0/1027665817 reap_dead start 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558a36f640 1 -- 172.21.15.98:0/1027665817 reap_dead start 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558bdd2640 1 -- 172.21.15.98:0/1027665817 shutdown_connections 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558bdd2640 1 -- 172.21.15.98:0/1027665817 >> 172.21.15.98:0/1027665817 conn(0x7f55840a4780 msgr2=0x7f55841bf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558bdd2640 1 -- 172.21.15.98:0/1027665817 shutdown_connections 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558bdd2640 1 -- 172.21.15.98:0/1027665817 wait complete. 2024-06-11T19:13:33.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:33.882+0000 7f558bdd2640 1 librados: shutdown 2024-06-11T19:13:33.923 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:33.923 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:13:33.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:13:34.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.023+0000 7fbbf31d2640 10 monclient: get_monmap_and_config 2024-06-11T19:13:34.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.023+0000 7fbbf31d2640 10 monclient: build_initial_monmap 2024-06-11T19:13:34.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.023+0000 7fbbf31d2640 10 monclient: monmap: 2024-06-11T19:13:34.023 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:34.023 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.023 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:34.024724+0000 2024-06-11T19:13:34.023 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:34.024724+0000 2024-06-11T19:13:34.023 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:34.023 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:34.024 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.024+0000 7fbbf31d2640 1 Processor -- start 2024-06-11T19:13:34.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.024+0000 7fbbf31d2640 1 -- start start 2024-06-11T19:13:34.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.024+0000 7fbbf31d2640 10 monclient: init 2024-06-11T19:13:34.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.024+0000 7fbbf31d2640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:34.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.024+0000 7fbbf31d2640 10 monclient: _add_conns ranks=[4,3,10,5,9,6,7,8,2,0,1] 2024-06-11T19:13:34.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.024+0000 7fbbf31d2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbec0a4920 0x7fbbec0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.024+0000 7fbbf31d2640 10 monclient(hunting): picked mon.noname-c con 0x7fbbec0a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:34.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.024+0000 7fbbf31d2640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbec0a5240 0x7fbbec0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.024+0000 7fbbf31d2640 10 monclient(hunting): picked mon.noname-g con 0x7fbbec0a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:34.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.024+0000 7fbbf31d2640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fbbec0a2530 0x7fbbec0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf31d2640 10 monclient(hunting): picked mon.noname-f con 0x7fbbec0a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:34.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf31d2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf31d2640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fbbec0a7140 con 0x7fbbec0a2530 2024-06-11T19:13:34.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf31d2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf31d2640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fbbec0a2b50 con 0x7fbbec0a5240 2024-06-11T19:13:34.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf31d2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf31d2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fbbec0a2e30 con 0x7fbbec0a4920 2024-06-11T19:13:34.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf31d2640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:34.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf31d2640 10 monclient(hunting): authenticate will time out at 1023.832038s 2024-06-11T19:13:34.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbec0a4920 0x7fbbec0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbec0a4920 0x7fbbec0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52254/0 (socket says 172.21.15.98:52254) 2024-06-11T19:13:34.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 1 -- 172.21.15.98:0/3539831490 learned_addr learned my addr 172.21.15.98:0/3539831490 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:34.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 10 monclient(hunting): get_auth_request con 0x7fbbec0a4920 auth_method 0 2024-06-11T19:13:34.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbe3fff640 1 --2- 172.21.15.98:0/3539831490 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbec0a5240 0x7fbbec0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf176f640 1 --2- 172.21.15.98:0/3539831490 >> v2:172.21.15.117:3300/0 conn(0x7fbbec0a2530 0x7fbbec0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_done global_id 23395 payload 274 2024-06-11T19:13:34.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 1 -- 172.21.15.98:0/3539831490 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbec0a5240 msgr2=0x7fbbec0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:34.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 1 --2- 172.21.15.98:0/3539831490 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbec0a5240 0x7fbbec0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 1 -- 172.21.15.98:0/3539831490 >> v2:172.21.15.117:3300/0 conn(0x7fbbec0a2530 msgr2=0x7fbbec0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 1 --2- 172.21.15.98:0/3539831490 >> v2:172.21.15.117:3300/0 conn(0x7fbbec0a2530 0x7fbbec0a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 10 monclient: _finish_hunting 0 2024-06-11T19:13:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 1 monclient: found mon.noname-c 2024-06-11T19:13:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf0f6e640 1 -- 172.21.15.98:0/3539831490 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbec0a3110 con 0x7fbbec0a4920 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbe3fff640 10 monclient: get_auth_request con 0x7fbbec0a5240 auth_method 0 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbe3fff640 1 --2- 172.21.15.98:0/3539831490 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbec0a5240 0x7fbbec0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf176f640 10 monclient: get_auth_request con 0x7fbbec0a2530 auth_method 0 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.025+0000 7fbbf176f640 1 --2- 172.21.15.98:0/3539831490 >> v2:172.21.15.117:3300/0 conn(0x7fbbec0a2530 0x7fbbec0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf0f6e640 1 --2- 172.21.15.98:0/3539831490 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbec0a4920 0x7fbbec0a4d00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009870 tx=0x7fbbe802b7c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=cfa4a41245904a54 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 1 -- 172.21.15.98:0/3539831490 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbbe8038070 con 0x7fbbec0a4920 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 10 monclient: dump: 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 10 monclient: _finish_auth 0 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 10 monclient: _check_auth_tickets 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 5 monclient: authenticate success, global_id 23395 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 1 -- 172.21.15.98:0/3539831490 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe8031990 con 0x7fbbec0a4920 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 4 set_mon_vals no callback set 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 1 -- 172.21.15.98:0/3539831490 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbbe80323b0 con 0x7fbbec0a4920 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbe37fe640 10 monclient: dump: 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.044 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.045 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 10 monclient: shutdown 2024-06-11T19:13:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 1 -- 172.21.15.98:0/3539831490 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbec0a4920 msgr2=0x7fbbec0a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 1 --2- 172.21.15.98:0/3539831490 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbec0a4920 0x7fbbec0a4d00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009870 tx=0x7fbbe802b7c0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 1 -- 172.21.15.98:0/3539831490 shutdown_connections 2024-06-11T19:13:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 1 --2- 172.21.15.98:0/3539831490 >> v2:172.21.15.117:3300/0 conn(0x7fbbec0a2530 0x7fbbec0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 1 --2- 172.21.15.98:0/3539831490 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbec0a5240 0x7fbbec0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 1 --2- 172.21.15.98:0/3539831490 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbec0a4920 0x7fbbec0a4d00 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 1 -- 172.21.15.98:0/3539831490 >> 172.21.15.98:0/3539831490 conn(0x7fbbec0ab850 msgr2=0x7fbbec0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 1 -- 172.21.15.98:0/3539831490 shutdown_connections 2024-06-11T19:13:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.026+0000 7fbbf31d2640 1 -- 172.21.15.98:0/3539831490 wait complete. 2024-06-11T19:13:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.027+0000 7fbbf31d2640 10 monclient: build_initial_monmap 2024-06-11T19:13:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.027+0000 7fbbf31d2640 10 monclient: monmap: 2024-06-11T19:13:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.027+0000 7fbbf31d2640 1 librados: starting msgr at 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.027+0000 7fbbf31d2640 1 librados: starting objecter 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.027+0000 7fbbf31d2640 1 Processor -- start 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.027+0000 7fbbf31d2640 1 -- start start 2024-06-11T19:13:34.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.027+0000 7fbbf31d2640 1 librados: setting wanted keys 2024-06-11T19:13:34.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.027+0000 7fbbf31d2640 1 librados: calling monclient init 2024-06-11T19:13:34.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.027+0000 7fbbf31d2640 10 monclient: init 2024-06-11T19:13:34.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:34.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 10 monclient: _add_conns ranks=[8,7,2,1,5,6,3,0,4] 2024-06-11T19:13:34.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbbec0a2530 0x7fbbec1cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 10 monclient(hunting): picked mon.noname-e con 0x7fbbec0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:34.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbbec0a4920 0x7fbbec1ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 10 monclient(hunting): picked mon.noname-i con 0x7fbbec0a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:34.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbec0a5240 0x7fbbec1223e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 10 monclient(hunting): picked mon.noname-b con 0x7fbbec0a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:34.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbbec0a6da0 con 0x7fbbec0a5240 2024-06-11T19:13:34.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fbbec138e50 con 0x7fbbec0a4920 2024-06-11T19:13:34.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbbec0a6420 con 0x7fbbec0a2530 2024-06-11T19:13:34.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:34.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf31d2640 10 monclient(hunting): authenticate will time out at 1023.835433s 2024-06-11T19:13:34.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf0f6e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbbec0a2530 0x7fbbec1cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbec0a5240 0x7fbbec1223e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf0f6e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbbec0a2530 0x7fbbec1cdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35348/0 (socket says 172.21.15.98:35348) 2024-06-11T19:13:34.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf0f6e640 1 -- 172.21.15.98:0/442201245 learned_addr learned my addr 172.21.15.98:0/442201245 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:34.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf0f6e640 10 monclient(hunting): get_auth_request con 0x7fbbec0a2530 auth_method 0 2024-06-11T19:13:34.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf0f6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf0f6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf0f6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 10 monclient(hunting): get_auth_request con 0x7fbbec0a5240 auth_method 0 2024-06-11T19:13:34.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbe3fff640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbbec0a4920 0x7fbbec1ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbe3fff640 10 monclient(hunting): get_auth_request con 0x7fbbec0a4920 auth_method 0 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbe3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbe3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 10 monclient(hunting): handle_auth_done global_id 22043 payload 824 2024-06-11T19:13:34.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 1 -- 172.21.15.98:0/442201245 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbbec0a2530 msgr2=0x7fbbec1cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbbec0a2530 0x7fbbec1cdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 1 -- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbbec0a4920 msgr2=0x7fbbec1ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbbec0a4920 0x7fbbec1ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 10 monclient: _finish_hunting 0 2024-06-11T19:13:34.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 1 monclient: found mon.noname-b 2024-06-11T19:13:34.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:34.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:13:34.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 1 -- 172.21.15.98:0/442201245 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbec126070 con 0x7fbbec0a5240 2024-06-11T19:13:34.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf0f6e640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbbec0a2530 0x7fbbec1cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:34.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbf176f640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbec0a5240 0x7fbbec1223e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400cdc0 tx=0x7fbbe400a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbe17fa640 10 client.?.objecter ms_handle_connect 0x7fbbec0a5240 2024-06-11T19:13:34.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.028+0000 7fbbe17fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbbe401a070 con 0x7fbbec0a5240 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 10 monclient: dump: 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.054 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 10 monclient: _finish_auth 0 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 10 monclient: _check_auth_tickets 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe400ae20 con 0x7fbbec0a5240 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbf31d2640 5 monclient: authenticate success, global_id 22043 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbbe4013340 con 0x7fbbec0a5240 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbf31d2640 10 monclient: _renew_subs 2024-06-11T19:13:34.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbf31d2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbf31d2640 1 -- 172.21.15.98:0/442201245 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbec124bc0 con 0x7fbbec0a5240 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe17fa640 10 monclient: dump: 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.056 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbe2ffd640 4 set_mon_vals no callback set 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbf31d2640 10 client.22043.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbf31d2640 10 monclient: _renew_subs 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbf31d2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbf31d2640 1 -- 172.21.15.98:0/442201245 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbec0a9410 con 0x7fbbec0a5240 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.029+0000 7fbbf31d2640 1 librados: init done 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf31d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf31d2640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec125c80 0x7fbbec122920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf31d2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf31d2640 1 -- 172.21.15.98:0/442201245 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec069750 con 0x7fbbec125c80 2024-06-11T19:13:34.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fbbe400d640 con 0x7fbbec0a5240 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf0f6e640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec125c80 0x7fbbec122920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf0f6e640 10 monclient: get_auth_request con 0x7fbbec125c80 auth_method 0 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf0f6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf0f6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf0f6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbe17fa640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbbd803fdb0 0x7fbbd8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(179..179 src has 1..179) ==== 6757+0+0 (secure 0 0 0) 0x7fbbe4052350 con 0x7fbbec0a5240 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbe17fa640 10 client.22043.objecter ms_dispatch 0x7fbbec000b90 osd_map(179..179 src has 1..179) 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbe17fa640 3 client.22043.objecter handle_osd_map got epochs [179,179] > 0 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbe17fa640 3 client.22043.objecter handle_osd_map decoding full epoch 179 2024-06-11T19:13:34.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbe17fa640 20 client.22043.objecter dump_active .. 0 homeless 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbe3fff640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbbd803fdb0 0x7fbbd8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbe3fff640 10 monclient: get_auth_request con 0x7fbbd803fdb0 auth_method 0 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_done global_id 31420 payload 824 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf0f6e640 1 -- 172.21.15.98:0/442201245 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbbec0a8ec0 con 0x7fbbec125c80 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbe17fa640 10 client.22043.objecter ms_handle_connect 0x7fbbec125c80 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbe17fa640 10 client.22043.objecter resend_mon_ops 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.030+0000 7fbbf0f6e640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec125c80 0x7fbbec122920 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8000c00 tx=0x7fbbe80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe17fa640 10 client.22043.objecter ms_handle_connect 0x7fbbd803fdb0 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe3fff640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbbd803fdb0 0x7fbbd8042270 secure :-1 s=READY pgs=4070 cs=0 l=1 rev1=1 crypto rx=0x7fbbec123b40 tx=0x7fbbdc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbbe8044070 con 0x7fbbec125c80 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe17fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe17fa640 10 monclient: dump: 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.060 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.061 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbbe8031e60 con 0x7fbbec125c80 2024-06-11T19:13:34.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:34.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe17fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:34.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec125c80 msgr2=0x7fbbec122920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe17fa640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec125c80 0x7fbbec122920 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8000c00 tx=0x7fbbe80046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec125c80 msgr2=0x7fbbec122920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:34.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.031+0000 7fbbe17fa640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec125c80 0x7fbbec122920 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.037+0000 7fbbf31d2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:34.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.037+0000 7fbbf31d2640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec122e60 0x7fbbec123270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.037+0000 7fbbf31d2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.037+0000 7fbbf31d2640 1 -- 172.21.15.98:0/442201245 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec050830 con 0x7fbbec122e60 2024-06-11T19:13:34.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbf0f6e640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec122e60 0x7fbbec123270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbf0f6e640 10 monclient: get_auth_request con 0x7fbbec122e60 auth_method 0 2024-06-11T19:13:34.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbf0f6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbf0f6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbf0f6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbf0f6e640 10 monclient(hunting): handle_auth_done global_id 31429 payload 824 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbf0f6e640 1 -- 172.21.15.98:0/442201245 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbbec0a6720 con 0x7fbbec122e60 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbe17fa640 10 client.22043.objecter ms_handle_connect 0x7fbbec122e60 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbe17fa640 10 client.22043.objecter resend_mon_ops 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.038+0000 7fbbf0f6e640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec122e60 0x7fbbec123270 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fbbe802b800 tx=0x7fbbe8009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbbe8038040 con 0x7fbbec122e60 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbe17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbe17fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbe17fa640 10 monclient: dump: 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.064 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9275 (secure 0 0 0) 0x7fbbe8049d30 con 0x7fbbec122e60 2024-06-11T19:13:34.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbe17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:34.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbe17fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:34.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec122e60 msgr2=0x7fbbec123270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbe17fa640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec122e60 0x7fbbec123270 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fbbe802b800 tx=0x7fbbe8009240 comp rx=0 tx=0).stop 2024-06-11T19:13:34.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbe17fa640 1 -- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec122e60 msgr2=0x7fbbec123270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:34.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbe17fa640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec122e60 0x7fbbec123270 secure :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fbbe802b800 tx=0x7fbbe8009240 comp rx=0 tx=0).stop 2024-06-11T19:13:34.066 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:34.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:34.066 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:34.066 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:13:34.066 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 264257, 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:34.067 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:34.068 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:34.069 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:34.070 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.071 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.072 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.074 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:34.075 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:34.077 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:34.079 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.079 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:34.079 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:34.079 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.079 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.079 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:34.080 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:34.082 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:34.084 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:34.086 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:34.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.087 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.087 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:34.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.087 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:34.087 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:34.087 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:13:34.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.087 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.087 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:34.087 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:34.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbf31d2640 10 librados: watch_flush enter 2024-06-11T19:13:34.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbf31d2640 10 librados: watch_flush exit 2024-06-11T19:13:34.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbf31d2640 20 client.22043.objecter shutdown clearing up homeless session... 2024-06-11T19:13:34.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.039+0000 7fbbf31d2640 10 client.22043.objecter successfully canceled tick 2024-06-11T19:13:34.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf31d2640 1 -- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbbd803fdb0 msgr2=0x7fbbd8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf31d2640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbbd803fdb0 0x7fbbd8042270 secure :-1 s=READY pgs=4070 cs=0 l=1 rev1=1 crypto rx=0x7fbbec123b40 tx=0x7fbbdc008040 comp rx=0 tx=0).stop 2024-06-11T19:13:34.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf31d2640 10 monclient: shutdown 2024-06-11T19:13:34.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf31d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:34.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf31d2640 1 -- 172.21.15.98:0/442201245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbec0a5240 msgr2=0x7fbbec1223e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf31d2640 1 --2- 172.21.15.98:0/442201245 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbec0a5240 0x7fbbec1223e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400cdc0 tx=0x7fbbe400a710 comp rx=0 tx=0).stop 2024-06-11T19:13:34.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf176f640 1 -- 172.21.15.98:0/442201245 reap_dead start 2024-06-11T19:13:34.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf176f640 1 -- 172.21.15.98:0/442201245 reap_dead start 2024-06-11T19:13:34.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf31d2640 1 -- 172.21.15.98:0/442201245 shutdown_connections 2024-06-11T19:13:34.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf31d2640 1 -- 172.21.15.98:0/442201245 >> 172.21.15.98:0/442201245 conn(0x7fbbec0ab850 msgr2=0x7fbbec1381c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:34.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf31d2640 1 -- 172.21.15.98:0/442201245 shutdown_connections 2024-06-11T19:13:34.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf31d2640 1 -- 172.21.15.98:0/442201245 wait complete. 2024-06-11T19:13:34.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.040+0000 7fbbf31d2640 1 librados: shutdown 2024-06-11T19:13:34.090 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:34.090 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:13:34.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:13:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.191+0000 7fed51b3f640 10 monclient: get_monmap_and_config 2024-06-11T19:13:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.191+0000 7fed51b3f640 10 monclient: build_initial_monmap 2024-06-11T19:13:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.191+0000 7fed51b3f640 10 monclient: monmap: 2024-06-11T19:13:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:34.192560+0000 2024-06-11T19:13:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:34.192560+0000 2024-06-11T19:13:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:34.191 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:34.192 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:34.192 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:34.192 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:34.192 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:34.192 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:34.192 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:34.192 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:34.192 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:34.192 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.192+0000 7fed51b3f640 1 Processor -- start 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.192+0000 7fed51b3f640 1 -- start start 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.192+0000 7fed51b3f640 10 monclient: init 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 10 monclient: _add_conns ranks=[5,2,1,6,10,9,3,4,8,0,7] 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fed4c0a4920 0x7fed4c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 10 monclient(hunting): picked mon.noname-h con 0x7fed4c0a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a5240 0x7fed4c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 10 monclient(hunting): picked mon.noname-b con 0x7fed4c0a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fed4c0a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed4c0a7140 con 0x7fed4c0a2530 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fed4c0a2b50 con 0x7fed4c0a5240 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fed4c0a2e30 con 0x7fed4c0a4920 2024-06-11T19:13:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed51b3f640 10 monclient(hunting): authenticate will time out at 1024.000469s 2024-06-11T19:13:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a5240 0x7fed4c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4bfff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fed4c0a2530 0x7fed4c0a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:57650/0 (socket says 172.21.15.98:57650) 2024-06-11T19:13:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4bfff640 1 -- 172.21.15.98:0/949419768 learned_addr learned my addr 172.21.15.98:0/949419768 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:13:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4affd640 10 monclient(hunting): get_auth_request con 0x7fed4c0a5240 auth_method 0 2024-06-11T19:13:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4b7fe640 1 --2- 172.21.15.98:0/949419768 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fed4c0a4920 0x7fed4c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4b7fe640 10 monclient(hunting): get_auth_request con 0x7fed4c0a4920 auth_method 0 2024-06-11T19:13:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.193+0000 7fed4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4affd640 10 monclient(hunting): handle_auth_done global_id 22052 payload 274 2024-06-11T19:13:34.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4affd640 1 -- 172.21.15.98:0/949419768 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fed4c0a4920 msgr2=0x7fed4c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:34.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4affd640 1 --2- 172.21.15.98:0/949419768 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fed4c0a4920 0x7fed4c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4affd640 1 -- 172.21.15.98:0/949419768 >> v1:172.21.15.117:6789/0 conn(0x7fed4c0a2530 legacy=0x7fed4c0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4affd640 10 monclient: _finish_hunting 0 2024-06-11T19:13:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4affd640 1 monclient: found mon.noname-b 2024-06-11T19:13:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:13:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4affd640 1 -- 172.21.15.98:0/949419768 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4c0a3110 con 0x7fed4c0a5240 2024-06-11T19:13:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.194+0000 7fed4affd640 1 --2- 172.21.15.98:0/949419768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a5240 0x7fed4c0a59a0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fed400097d0 tx=0x7fed4002b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b801c79619cc242 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 1 -- 172.21.15.98:0/949419768 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed40039070 con 0x7fed4c0a5240 2024-06-11T19:13:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:13:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 10 monclient: dump: 2024-06-11T19:13:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.203 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.203 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.203 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.203 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.203 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.203 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.203 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.203 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.204 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:13:34.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:13:34.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:34.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 5 monclient: authenticate success, global_id 22052 2024-06-11T19:13:34.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 1 -- 172.21.15.98:0/949419768 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed40030070 con 0x7fed4c0a5240 2024-06-11T19:13:34.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:34.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:34.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:34.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 4 set_mon_vals no callback set 2024-06-11T19:13:34.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 1 -- 172.21.15.98:0/949419768 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed40031db0 con 0x7fed4c0a5240 2024-06-11T19:13:34.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:34.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed4a7fc640 10 monclient: dump: 2024-06-11T19:13:34.205 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.205 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.206 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.206 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.206 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.206 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.206 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.206 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.206 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.206 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.207 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.207 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.207 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.207 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.207 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.207 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 10 monclient: shutdown 2024-06-11T19:13:34.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:34.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 1 -- 172.21.15.98:0/949419768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a5240 msgr2=0x7fed4c0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 1 --2- 172.21.15.98:0/949419768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a5240 0x7fed4c0a59a0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fed400097d0 tx=0x7fed4002b770 comp rx=0 tx=0).stop 2024-06-11T19:13:34.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 1 -- 172.21.15.98:0/949419768 shutdown_connections 2024-06-11T19:13:34.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 1 --2- 172.21.15.98:0/949419768 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a5240 0x7fed4c0a59a0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 1 --2- 172.21.15.98:0/949419768 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fed4c0a4920 0x7fed4c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 1 -- 172.21.15.98:0/949419768 >> 172.21.15.98:0/949419768 conn(0x7fed4c0ab850 msgr2=0x7fed4c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:34.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 1 -- 172.21.15.98:0/949419768 shutdown_connections 2024-06-11T19:13:34.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.195+0000 7fed51b3f640 1 -- 172.21.15.98:0/949419768 wait complete. 2024-06-11T19:13:34.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.196+0000 7fed51b3f640 10 monclient: build_initial_monmap 2024-06-11T19:13:34.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.196+0000 7fed51b3f640 10 monclient: monmap: 2024-06-11T19:13:34.209 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:34.209 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:34.209 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:34.209 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:34.209 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:34.209 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:34.209 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:34.209 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:34.210 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:34.210 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:34.210 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:34.210 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:34.210 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:34.210 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:34.210 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:34.210 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.196+0000 7fed51b3f640 1 librados: starting msgr at 2024-06-11T19:13:34.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.196+0000 7fed51b3f640 1 librados: starting objecter 2024-06-11T19:13:34.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.196+0000 7fed51b3f640 1 Processor -- start 2024-06-11T19:13:34.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 1 -- start start 2024-06-11T19:13:34.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 1 librados: setting wanted keys 2024-06-11T19:13:34.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 1 librados: calling monclient init 2024-06-11T19:13:34.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 10 monclient: init 2024-06-11T19:13:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 10 monclient: _add_conns ranks=[4,2,1,5,6,0,8,7,3] 2024-06-11T19:13:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed4c0a2530 0x7fed4c1377f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 10 monclient(hunting): picked mon.noname-c con 0x7fed4c0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a4920 0x7fed4c137d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 10 monclient(hunting): picked mon.noname-b con 0x7fed4c0a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed4c0a5240 0x7fed4c123210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 10 monclient(hunting): picked mon.noname-f con 0x7fed4c0a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:13:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c0a6da0 con 0x7fed4c0a5240 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fed4c138e50 con 0x7fed4c0a4920 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fed4c0a65f0 con 0x7fed4c0a2530 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed51b3f640 10 monclient(hunting): authenticate will time out at 1024.004792s 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.197+0000 7fed4affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a4920 0x7fed4c137d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a4920 0x7fed4c137d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55472/0 (socket says 172.21.15.98:55472) 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 1 -- 172.21.15.98:0/925904177 learned_addr learned my addr 172.21.15.98:0/925904177 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 10 monclient(hunting): get_auth_request con 0x7fed4c0a4920 auth_method 0 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4b7fe640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed4c0a2530 0x7fed4c1377f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4b7fe640 10 monclient(hunting): get_auth_request con 0x7fed4c0a2530 auth_method 0 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4bfff640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed4c0a5240 0x7fed4c123210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4bfff640 10 monclient(hunting): get_auth_request con 0x7fed4c0a5240 auth_method 0 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 10 monclient(hunting): handle_auth_done global_id 22061 payload 824 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 1 -- 172.21.15.98:0/925904177 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed4c0a2530 msgr2=0x7fed4c1377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed4c0a2530 0x7fed4c1377f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 1 -- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed4c0a5240 msgr2=0x7fed4c123210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed4c0a5240 0x7fed4c123210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 10 monclient: _finish_hunting 0 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 1 monclient: found mon.noname-b 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 1 -- 172.21.15.98:0/925904177 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4c0a9160 con 0x7fed4c0a4920 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4b7fe640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed4c0a2530 0x7fed4c1377f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed4affd640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a4920 0x7fed4c137d30 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fed400096f0 tx=0x7fed400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed33fff640 10 client.?.objecter ms_handle_connect 0x7fed4c0a4920 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.198+0000 7fed33fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed40042020 con 0x7fed4c0a4920 2024-06-11T19:13:34.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 10 monclient: dump: 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.222 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 10 monclient: _finish_auth 0 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 10 monclient: _check_auth_tickets 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed4003a040 con 0x7fed4c0a4920 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed40039070 con 0x7fed4c0a4920 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed51b3f640 5 monclient: authenticate success, global_id 22061 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed51b3f640 10 monclient: _renew_subs 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed51b3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:34.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed51b3f640 1 -- 172.21.15.98:0/925904177 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4c126ef0 con 0x7fed4c0a4920 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed49ffb640 4 set_mon_vals no callback set 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed33fff640 10 monclient: dump: 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.224 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed51b3f640 10 client.22061.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed51b3f640 10 monclient: _renew_subs 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed51b3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed51b3f640 1 -- 172.21.15.98:0/925904177 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4c0a8da0 con 0x7fed4c0a4920 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.200+0000 7fed51b3f640 1 librados: init done 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fed40045020 con 0x7fed4c0a4920 2024-06-11T19:13:34.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed51b3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed51b3f640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c1c0c20 0x7fed4c123750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed51b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed51b3f640 1 -- 172.21.15.98:0/925904177 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fed4c050830 con 0x7fed4c1c0c20 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed33fff640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fed3803fdb0 0x7fed38042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(179..179 src has 1..179) ==== 6757+0+0 (secure 0 0 0) 0x7fed400730e0 con 0x7fed4c0a4920 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed33fff640 10 client.22061.objecter ms_dispatch 0x7fed4c000b90 osd_map(179..179 src has 1..179) 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed33fff640 3 client.22061.objecter handle_osd_map got epochs [179,179] > 0 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed33fff640 3 client.22061.objecter handle_osd_map decoding full epoch 179 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed33fff640 20 client.22061.objecter dump_active .. 0 homeless 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed4b7fe640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c1c0c20 0x7fed4c123750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed4b7fe640 10 monclient: get_auth_request con 0x7fed4c1c0c20 auth_method 0 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed4b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed4bfff640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fed3803fdb0 0x7fed38042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.201+0000 7fed4bfff640 10 monclient: get_auth_request con 0x7fed3803fdb0 auth_method 0 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed4b7fe640 10 monclient(hunting): handle_auth_done global_id 21981 payload 824 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed4b7fe640 1 -- 172.21.15.98:0/925904177 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed4c075360 con 0x7fed4c1c0c20 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed33fff640 10 client.22061.objecter ms_handle_connect 0x7fed3803fdb0 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed4bfff640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fed3803fdb0 0x7fed38042270 secure :-1 s=READY pgs=4071 cs=0 l=1 rev1=1 crypto rx=0x7fed440012e0 tx=0x7fed44009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed33fff640 10 client.22061.objecter ms_handle_connect 0x7fed4c1c0c20 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed33fff640 10 client.22061.objecter resend_mon_ops 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed4b7fe640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c1c0c20 0x7fed4c123750 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fed3c011830 tx=0x7fed3c011cf0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed3c00fca0 con 0x7fed4c1c0c20 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed33fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:13:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.202+0000 7fed33fff640 10 monclient: dump: 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.203+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fed3c00f830 con 0x7fed4c1c0c20 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.203+0000 7fed33fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.203+0000 7fed33fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.203+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c1c0c20 msgr2=0x7fed4c123750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.203+0000 7fed33fff640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c1c0c20 0x7fed4c123750 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fed3c011830 tx=0x7fed3c011cf0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.203+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c1c0c20 msgr2=0x7fed4c123750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.203+0000 7fed33fff640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c1c0c20 0x7fed4c123750 secure :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fed3c011830 tx=0x7fed3c011cf0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.209+0000 7fed51b3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.209+0000 7fed51b3f640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c123c90 0x7fed4c1240a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.209+0000 7fed51b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.209+0000 7fed51b3f640 1 -- 172.21.15.98:0/925904177 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fed4c0a4d00 con 0x7fed4c123c90 2024-06-11T19:13:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.209+0000 7fed4b7fe640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c123c90 0x7fed4c1240a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.209+0000 7fed4b7fe640 10 monclient: get_auth_request con 0x7fed4c123c90 auth_method 0 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.209+0000 7fed4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.209+0000 7fed4b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.209+0000 7fed4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.209+0000 7fed4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.209+0000 7fed4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.209+0000 7fed4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed4b7fe640 10 monclient(hunting): handle_auth_done global_id 21990 payload 824 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed4b7fe640 1 -- 172.21.15.98:0/925904177 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fed4c1245e0 con 0x7fed4c123c90 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed4b7fe640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c123c90 0x7fed4c1240a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fed3c000c00 tx=0x7fed3c004730 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 10 client.22061.objecter ms_handle_connect 0x7fed4c123c90 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 10 client.22061.objecter resend_mon_ops 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed3c006780 con 0x7fed4c123c90 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 10 monclient: dump: 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.232 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fed3c006310 con 0x7fed4c123c90 2024-06-11T19:13:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c123c90 msgr2=0x7fed4c1240a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c123c90 0x7fed4c1240a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fed3c000c00 tx=0x7fed3c004730 comp rx=0 tx=0).stop 2024-06-11T19:13:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 1 -- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c123c90 msgr2=0x7fed4c1240a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed33fff640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed4c123c90 0x7fed4c1240a0 secure :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fed3c000c00 tx=0x7fed3c004730 comp rx=0 tx=0).stop 2024-06-11T19:13:34.232 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:34.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:34.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:34.233 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:34.233 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:34.233 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 264427, 2024-06-11T19:13:34.233 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:34.233 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:34.233 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:34.233 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:34.233 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:34.233 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:13:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:34.234 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:34.236 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:34.237 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:34.237 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:34.237 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:34.237 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:34.237 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:34.237 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:34.237 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:34.237 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:34.237 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:34.237 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:34.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:34.237 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:34.238 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:34.238 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:34.238 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:34.238 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:34.238 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:34.238 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:34.238 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:34.238 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:34.238 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:34.238 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:34.238 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:34.239 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.241 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:34.242 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:34.242 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.242 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.242 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:34.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:34.242 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:34.243 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:34.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.246 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:34.247 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:34.247 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.247 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.247 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:34.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:34.247 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:34.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:34.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.248 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:34.250 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:34.250 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.250 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.250 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.252 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:34.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:34.253 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:34.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:34.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.254 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:34.255 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.256 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.256 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:34.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.256 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:34.256 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:34.256 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:13:34.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.256 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.256 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:34.256 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed51b3f640 10 librados: watch_flush enter 2024-06-11T19:13:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed51b3f640 10 librados: watch_flush exit 2024-06-11T19:13:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed51b3f640 20 client.22061.objecter shutdown clearing up homeless session... 2024-06-11T19:13:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.210+0000 7fed51b3f640 10 client.22061.objecter successfully canceled tick 2024-06-11T19:13:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed51b3f640 1 -- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fed3803fdb0 msgr2=0x7fed38042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed51b3f640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fed3803fdb0 0x7fed38042270 secure :-1 s=READY pgs=4071 cs=0 l=1 rev1=1 crypto rx=0x7fed440012e0 tx=0x7fed44009040 comp rx=0 tx=0).stop 2024-06-11T19:13:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed51b3f640 10 monclient: shutdown 2024-06-11T19:13:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed51b3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:34.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed51b3f640 1 -- 172.21.15.98:0/925904177 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a4920 msgr2=0x7fed4c137d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed4bfff640 1 -- 172.21.15.98:0/925904177 reap_dead start 2024-06-11T19:13:34.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed51b3f640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a4920 0x7fed4c137d30 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fed400096f0 tx=0x7fed400046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed51b3f640 1 -- 172.21.15.98:0/925904177 shutdown_connections 2024-06-11T19:13:34.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed51b3f640 1 --2- 172.21.15.98:0/925904177 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed4c0a4920 0x7fed4c137d30 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed51b3f640 1 -- 172.21.15.98:0/925904177 >> 172.21.15.98:0/925904177 conn(0x7fed4c0ab850 msgr2=0x7fed4c0a9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:34.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed51b3f640 1 -- 172.21.15.98:0/925904177 shutdown_connections 2024-06-11T19:13:34.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed51b3f640 1 -- 172.21.15.98:0/925904177 wait complete. 2024-06-11T19:13:34.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.211+0000 7fed51b3f640 1 librados: shutdown 2024-06-11T19:13:34.259 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:34.259 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:13:34.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:13:34.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.359+0000 7f536533f640 10 monclient: get_monmap_and_config 2024-06-11T19:13:34.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.359+0000 7f536533f640 10 monclient: build_initial_monmap 2024-06-11T19:13:34.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.360+0000 7f536533f640 10 monclient: monmap: 2024-06-11T19:13:34.359 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:34.359 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:34.361007+0000 2024-06-11T19:13:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:34.361007+0000 2024-06-11T19:13:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:34.361 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:34.361 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:34.361 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:34.361 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:34.361 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.360+0000 7f536533f640 1 Processor -- start 2024-06-11T19:13:34.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.360+0000 7f536533f640 1 -- start start 2024-06-11T19:13:34.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.360+0000 7f536533f640 10 monclient: init 2024-06-11T19:13:34.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:34.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 10 monclient: _add_conns ranks=[4,9,0,2,7,10,8,6,1,3,5] 2024-06-11T19:13:34.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53600a4920 0x7f53600a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 10 monclient(hunting): picked mon.noname-c con 0x7f53600a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:34.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f53600a5240 0x7f53600a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 10 monclient(hunting): picked mon.noname-a con 0x7f53600a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:13:34.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53600a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:13:34.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53600a7140 con 0x7f53600a2530 2024-06-11T19:13:34.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f53600a2b50 con 0x7f53600a5240 2024-06-11T19:13:34.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f53600a2e30 con 0x7f53600a4920 2024-06-11T19:13:34.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:34.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f536533f640 10 monclient(hunting): authenticate will time out at 1024.168175s 2024-06-11T19:13:34.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53600a4920 0x7f53600a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535e7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f53600a5240 0x7f53600a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53600a4920 0x7f53600a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52266/0 (socket says 172.21.15.98:52266) 2024-06-11T19:13:34.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 1 -- 172.21.15.98:0/2233325454 learned_addr learned my addr 172.21.15.98:0/2233325454 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:34.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 10 monclient(hunting): get_auth_request con 0x7f53600a4920 auth_method 0 2024-06-11T19:13:34.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535e7fc640 10 monclient(hunting): get_auth_request con 0x7f53600a5240 auth_method 0 2024-06-11T19:13:34.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535e7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 10 monclient(hunting): handle_auth_done global_id 23413 payload 274 2024-06-11T19:13:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 1 -- 172.21.15.98:0/2233325454 >> v2:172.21.15.98:3300/0 conn(0x7f53600a5240 msgr2=0x7f53600a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 1 --2- 172.21.15.98:0/2233325454 >> v2:172.21.15.98:3300/0 conn(0x7f53600a5240 0x7f53600a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 1 -- 172.21.15.98:0/2233325454 >> v1:172.21.15.98:6789/0 conn(0x7f53600a2530 legacy=0x7f53600a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 10 monclient: _finish_hunting 0 2024-06-11T19:13:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 1 monclient: found mon.noname-c 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535dffb640 1 -- 172.21.15.98:0/2233325454 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1566426191 0 0) 0x7f53600a7140 con 0x7f53600a2530 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535effd640 1 -- 172.21.15.98:0/2233325454 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53600a3110 con 0x7f53600a4920 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535dffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.361+0000 7f535e7fc640 1 --2- 172.21.15.98:0/2233325454 >> v2:172.21.15.98:3300/0 conn(0x7f53600a5240 0x7f53600a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535effd640 1 --2- 172.21.15.98:0/2233325454 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53600a4920 0x7f53600a4d00 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f5350009e50 tx=0x7f535002be40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=22aa4a68a94da290 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 1 -- 172.21.15.98:0/2233325454 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f535002cc50 con 0x7f53600a4920 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 10 monclient: dump: 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.376 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 10 monclient: _finish_auth 0 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 10 monclient: _check_auth_tickets 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 1 -- 172.21.15.98:0/2233325454 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f535002ce10 con 0x7f53600a4920 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f536533f640 5 monclient: authenticate success, global_id 23413 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f536533f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f536533f640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 1 -- 172.21.15.98:0/2233325454 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5350034860 con 0x7f53600a4920 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f536533f640 4 set_mon_vals no callback set 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:34.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.362+0000 7f535dffb640 10 monclient: dump: 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.378 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 10 monclient: shutdown 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 1 -- 172.21.15.98:0/2233325454 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53600a4920 msgr2=0x7f53600a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 1 --2- 172.21.15.98:0/2233325454 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53600a4920 0x7f53600a4d00 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f5350009e50 tx=0x7f535002be40 comp rx=0 tx=0).stop 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 1 -- 172.21.15.98:0/2233325454 shutdown_connections 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 1 --2- 172.21.15.98:0/2233325454 >> v2:172.21.15.98:3300/0 conn(0x7f53600a5240 0x7f53600a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 1 --2- 172.21.15.98:0/2233325454 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53600a4920 0x7f53600a4d00 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 1 -- 172.21.15.98:0/2233325454 >> 172.21.15.98:0/2233325454 conn(0x7f53600ab850 msgr2=0x7f53600a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 1 -- 172.21.15.98:0/2233325454 shutdown_connections 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 1 -- 172.21.15.98:0/2233325454 wait complete. 2024-06-11T19:13:34.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 10 monclient: build_initial_monmap 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 10 monclient: monmap: 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:34.380 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:34.381 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:34.381 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:34.381 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:34.381 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 1 librados: starting msgr at 2024-06-11T19:13:34.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.363+0000 7f536533f640 1 librados: starting objecter 2024-06-11T19:13:34.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 1 Processor -- start 2024-06-11T19:13:34.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 1 -- start start 2024-06-11T19:13:34.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 1 librados: setting wanted keys 2024-06-11T19:13:34.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 1 librados: calling monclient init 2024-06-11T19:13:34.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 10 monclient: init 2024-06-11T19:13:34.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:34.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 10 monclient: _add_conns ranks=[3,1,7,5,8,6,4,2,0] 2024-06-11T19:13:34.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f53600a2530 0x7f53601cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 10 monclient(hunting): picked mon.noname-g con 0x7f53600a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:34.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53600a4920 0x7f53601ce050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 10 monclient(hunting): picked mon.noname-f con 0x7f53600a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:13:34.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f53600a5240 0x7f5360124f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 10 monclient(hunting): picked mon.noname-i con 0x7f53600a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:34.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f53600a6da0 con 0x7f53600a4920 2024-06-11T19:13:34.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f5360138e50 con 0x7f53600a2530 2024-06-11T19:13:34.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f53600a65f0 con 0x7f53600a5240 2024-06-11T19:13:34.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f536533f640 10 monclient(hunting): authenticate will time out at 1024.171631s 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535effd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f53600a2530 0x7f53601cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53600a4920 0x7f53601ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535effd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f53600a2530 0x7f53601cdb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:59128/0 (socket says 172.21.15.98:59128) 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535effd640 1 -- 172.21.15.98:0/3704040399 learned_addr learned my addr 172.21.15.98:0/3704040399 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535effd640 10 monclient(hunting): get_auth_request con 0x7f53600a2530 auth_method 0 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535f7fe640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f53600a5240 0x7f5360124f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535e7fc640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53600a4920 0x7f53601ce050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:40416/0 (socket says 172.21.15.98:40416) 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535e7fc640 10 monclient(hunting): get_auth_request con 0x7f53600a4920 auth_method 0 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535e7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535f7fe640 10 monclient(hunting): get_auth_request con 0x7f53600a5240 auth_method 0 2024-06-11T19:13:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.364+0000 7f535e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535effd640 10 monclient(hunting): handle_auth_done global_id 21999 payload 824 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535effd640 1 -- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f53600a5240 msgr2=0x7f5360124f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535effd640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f53600a5240 0x7f5360124f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535effd640 1 -- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53600a4920 msgr2=0x7f53601ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535effd640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53600a4920 0x7f53601ce050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535effd640 10 monclient: _finish_hunting 0 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535effd640 1 monclient: found mon.noname-g 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535effd640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:13:34.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535effd640 1 -- 172.21.15.98:0/3704040399 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53601298a0 con 0x7f53600a2530 2024-06-11T19:13:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535f7fe640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f53600a5240 0x7f5360124f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535effd640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f53600a2530 0x7f53601cdb10 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f5350000f80 tx=0x7f5350004a40 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 10 client.?.objecter ms_handle_connect 0x7f53600a2530 2024-06-11T19:13:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5350043020 con 0x7f53600a2530 2024-06-11T19:13:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:13:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 10 monclient: dump: 2024-06-11T19:13:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.390 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 10 monclient: _finish_auth 0 2024-06-11T19:13:34.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 10 monclient: _check_auth_tickets 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53500046c0 con 0x7f53600a2530 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f536533f640 5 monclient: authenticate success, global_id 21999 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f536533f640 10 monclient: _renew_subs 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f536533f640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f536533f640 1 -- 172.21.15.98:0/3704040399 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5360128be0 con 0x7f53600a2530 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f535003e8c0 con 0x7f53600a2530 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f53477fe640 10 monclient: dump: 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.392 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f535d7fa640 4 set_mon_vals no callback set 2024-06-11T19:13:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f536533f640 10 client.21999.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f536533f640 10 monclient: _renew_subs 2024-06-11T19:13:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f536533f640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:13:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f536533f640 1 -- 172.21.15.98:0/3704040399 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5360128ec0 con 0x7f53600a2530 2024-06-11T19:13:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.365+0000 7f536533f640 1 librados: init done 2024-06-11T19:13:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f536533f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f536533f640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125490 0x7f5360125870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f536533f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f536533f640 1 -- 172.21.15.98:0/3704040399 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f5360050830 con 0x7f5360125490 2024-06-11T19:13:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 <== mon.3 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f5350046020 con 0x7f53600a2530 2024-06-11T19:13:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f535e7fc640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125490 0x7f5360125870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f535e7fc640 10 monclient: get_auth_request con 0x7f5360125490 auth_method 0 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f535e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f535e7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f535e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f535e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f535e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f535e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f53477fe640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f534c03fdb0 0x7f534c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f535e7fc640 10 monclient(hunting): handle_auth_done global_id 22460 payload 824 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 <== mon.3 v2:172.21.15.117:3301/0 5 ==== osd_map(179..179 src has 1..179) ==== 6757+0+0 (secure 0 0 0) 0x7f535003a040 con 0x7f53600a2530 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f535e7fc640 1 -- 172.21.15.98:0/3704040399 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5360138760 con 0x7f5360125490 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f53477fe640 10 client.21999.objecter ms_dispatch 0x7f5360000b90 osd_map(179..179 src has 1..179) 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f53477fe640 3 client.21999.objecter handle_osd_map got epochs [179,179] > 0 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f53477fe640 3 client.21999.objecter handle_osd_map decoding full epoch 179 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f535f7fe640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f534c03fdb0 0x7f534c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f535f7fe640 10 monclient: get_auth_request con 0x7f534c03fdb0 auth_method 0 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.366+0000 7f53477fe640 20 client.21999.objecter dump_active .. 0 homeless 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 10 client.21999.objecter ms_handle_connect 0x7f5360125490 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f535e7fc640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125490 0x7f5360125870 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f53600a5620 tx=0x7f5354011cf0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 10 client.21999.objecter resend_mon_ops 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5354014440 con 0x7f5360125490 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 10 monclient: dump: 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.396 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 10 client.21999.objecter ms_handle_connect 0x7f534c03fdb0 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f535f7fe640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f534c03fdb0 0x7f534c042270 secure :-1 s=READY pgs=4072 cs=0 l=1 rev1=1 crypto rx=0x7f53601293b0 tx=0x7f5358009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f535400fa70 con 0x7f5360125490 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:34.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125490 msgr2=0x7f5360125870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125490 0x7f5360125870 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f53600a5620 tx=0x7f5354011cf0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125490 msgr2=0x7f5360125870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:34.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.367+0000 7f53477fe640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125490 0x7f5360125870 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.373+0000 7f536533f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:34.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f536533f640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125db0 0x7f53601215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f536533f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f536533f640 1 -- 172.21.15.98:0/3704040399 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f53600aaa10 con 0x7f5360125db0 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f535e7fc640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125db0 0x7f53601215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f535e7fc640 10 monclient: get_auth_request con 0x7f5360125db0 auth_method 0 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f535e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f535e7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f535e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f535e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f535e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f535e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f535e7fc640 10 monclient(hunting): handle_auth_done global_id 22469 payload 824 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f535e7fc640 1 -- 172.21.15.98:0/3704040399 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f53600a8e80 con 0x7f5360125db0 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f535e7fc640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125db0 0x7f53601215f0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f53600a5620 tx=0x7f53540046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f53477fe640 10 client.21999.objecter ms_handle_connect 0x7f5360125db0 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f53477fe640 10 client.21999.objecter resend_mon_ops 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53540229e0 con 0x7f5360125db0 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f53477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f53477fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.374+0000 7f53477fe640 10 monclient: dump: 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.401 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.402 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.402 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.402 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.402 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.402 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.402 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.402 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f535402c420 con 0x7f5360125db0 2024-06-11T19:13:34.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f53477fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:34.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f53477fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:34.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125db0 msgr2=0x7f53601215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f53477fe640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125db0 0x7f53601215f0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f53600a5620 tx=0x7f53540046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f53477fe640 1 -- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125db0 msgr2=0x7f53601215f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:34.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f53477fe640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5360125db0 0x7f53601215f0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.403 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:34.403 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 264569, 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:34.404 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:34.405 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:34.406 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:34.407 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:34.408 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:34.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.412 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.414 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:34.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:34.416 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:34.418 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.425 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.427 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:34.427 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:34.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 10 librados: watch_flush enter 2024-06-11T19:13:34.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 10 librados: watch_flush exit 2024-06-11T19:13:34.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 20 client.21999.objecter shutdown clearing up homeless session... 2024-06-11T19:13:34.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 10 client.21999.objecter successfully canceled tick 2024-06-11T19:13:34.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 1 -- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f534c03fdb0 msgr2=0x7f534c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f534c03fdb0 0x7f534c042270 secure :-1 s=READY pgs=4072 cs=0 l=1 rev1=1 crypto rx=0x7f53601293b0 tx=0x7f5358009040 comp rx=0 tx=0).stop 2024-06-11T19:13:34.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 10 monclient: shutdown 2024-06-11T19:13:34.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:34.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 1 -- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f53600a2530 msgr2=0x7f53601cdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 1 --2- 172.21.15.98:0/3704040399 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f53600a2530 0x7f53601cdb10 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f5350000f80 tx=0x7f5350004a40 comp rx=0 tx=0).stop 2024-06-11T19:13:34.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f535f7fe640 1 -- 172.21.15.98:0/3704040399 reap_dead start 2024-06-11T19:13:34.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f535f7fe640 1 -- 172.21.15.98:0/3704040399 reap_dead start 2024-06-11T19:13:34.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 1 -- 172.21.15.98:0/3704040399 shutdown_connections 2024-06-11T19:13:34.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 1 -- 172.21.15.98:0/3704040399 >> 172.21.15.98:0/3704040399 conn(0x7f53600ab850 msgr2=0x7f536009c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:34.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 1 -- 172.21.15.98:0/3704040399 shutdown_connections 2024-06-11T19:13:34.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 1 -- 172.21.15.98:0/3704040399 wait complete. 2024-06-11T19:13:34.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.375+0000 7f536533f640 1 librados: shutdown 2024-06-11T19:13:34.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:34.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:13:34.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:13:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.531+0000 7f140a5a1640 10 monclient: get_monmap_and_config 2024-06-11T19:13:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.531+0000 7f140a5a1640 10 monclient: build_initial_monmap 2024-06-11T19:13:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.531+0000 7f140a5a1640 10 monclient: monmap: 2024-06-11T19:13:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:34.532303+0000 2024-06-11T19:13:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:34.532303+0000 2024-06-11T19:13:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:34.532 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 1 Processor -- start 2024-06-11T19:13:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 1 -- start start 2024-06-11T19:13:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 10 monclient: init 2024-06-11T19:13:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 10 monclient: _add_conns ranks=[5,2,9,8,6,1,3,10,4,0,7] 2024-06-11T19:13:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f14040a2f40 0x7f14040a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 10 monclient(hunting): picked mon.noname-h con 0x7f14040a2f40 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14040a4df0 0x7f14040a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 10 monclient(hunting): picked mon.noname-b con 0x7f14040a4df0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f14041366d0 0x7f1404136b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 10 monclient(hunting): picked mon.noname-a con 0x7f14041366d0 addr v2:172.21.15.98:3300/0 2024-06-11T19:13:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f14040a5710 con 0x7f14041366d0 2024-06-11T19:13:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f14040a3320 con 0x7f14040a4df0 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f14040a2460 con 0x7f14040a2f40 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f140a5a1640 10 monclient(hunting): authenticate will time out at 1024.339554s 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f14041366d0 0x7f1404136b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f14037fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14040a4df0 0x7f14040a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f14041366d0 0x7f1404136b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49820/0 (socket says 172.21.15.98:49820) 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 1 -- 172.21.15.98:0/202664182 learned_addr learned my addr 172.21.15.98:0/202664182 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f14037fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14040a4df0 0x7f14040a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55488/0 (socket says 172.21.15.98:55488) 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 10 monclient(hunting): get_auth_request con 0x7f14041366d0 auth_method 0 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f14037fe640 10 monclient(hunting): get_auth_request con 0x7f14040a4df0 auth_method 0 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f14037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f14037fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f14037fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f14037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f14037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f14037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 10 monclient(hunting): handle_auth_done global_id 64152 payload 274 2024-06-11T19:13:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1403fff640 1 --2- 172.21.15.98:0/202664182 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f14040a2f40 0x7f14040a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 1 -- 172.21.15.98:0/202664182 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f14040a2f40 msgr2=0x7f14040a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 1 --2- 172.21.15.98:0/202664182 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f14040a2f40 0x7f14040a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 1 -- 172.21.15.98:0/202664182 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14040a4df0 msgr2=0x7f14040a51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 1 --2- 172.21.15.98:0/202664182 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14040a4df0 0x7f14040a51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 10 monclient: _finish_hunting 0 2024-06-11T19:13:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 1 monclient: found mon.noname-a 2024-06-11T19:13:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:13:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.532+0000 7f1408b3e640 1 -- 172.21.15.98:0/202664182 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14040a0ed0 con 0x7f14041366d0 2024-06-11T19:13:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f14037fe640 1 --2- 172.21.15.98:0/202664182 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14040a4df0 0x7f14040a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1408b3e640 1 --2- 172.21.15.98:0/202664182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14041366d0 0x7f1404136b20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f13fc009eb0 tx=0x7f13fc02bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45698f4794d64ef7 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 1 -- 172.21.15.98:0/202664182 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f13fc02cb80 con 0x7f14041366d0 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 10 monclient: dump: 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.550 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 10 monclient: _finish_auth 0 2024-06-11T19:13:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:13:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f140a5a1640 5 monclient: authenticate success, global_id 64152 2024-06-11T19:13:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 1 -- 172.21.15.98:0/202664182 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13fc02cd40 con 0x7f14041366d0 2024-06-11T19:13:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f140a5a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f140a5a1640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f140a5a1640 4 set_mon_vals no callback set 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 1 -- 172.21.15.98:0/202664182 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f13fc034860 con 0x7f14041366d0 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.533+0000 7f1402ffd640 10 monclient: dump: 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.534+0000 7f140a5a1640 10 monclient: shutdown 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.534+0000 7f140a5a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.534+0000 7f140a5a1640 1 -- 172.21.15.98:0/202664182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14041366d0 msgr2=0x7f1404136b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.534+0000 7f140a5a1640 1 --2- 172.21.15.98:0/202664182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14041366d0 0x7f1404136b20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f13fc009eb0 tx=0x7f13fc02bd70 comp rx=0 tx=0).stop 2024-06-11T19:13:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.534+0000 7f140a5a1640 1 -- 172.21.15.98:0/202664182 shutdown_connections 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.534+0000 7f140a5a1640 1 --2- 172.21.15.98:0/202664182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14041366d0 0x7f1404136b20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.534+0000 7f140a5a1640 1 --2- 172.21.15.98:0/202664182 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f14040a4df0 0x7f14040a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.534+0000 7f140a5a1640 1 --2- 172.21.15.98:0/202664182 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f14040a2f40 0x7f14040a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.534+0000 7f140a5a1640 1 -- 172.21.15.98:0/202664182 >> 172.21.15.98:0/202664182 conn(0x7f14040a95b0 msgr2=0x7f14040a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.534+0000 7f140a5a1640 1 -- 172.21.15.98:0/202664182 shutdown_connections 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.534+0000 7f140a5a1640 1 -- 172.21.15.98:0/202664182 wait complete. 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.535+0000 7f140a5a1640 10 monclient: build_initial_monmap 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.535+0000 7f140a5a1640 10 monclient: monmap: 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:34.554 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.535+0000 7f140a5a1640 1 librados: starting msgr at 2024-06-11T19:13:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.535+0000 7f140a5a1640 1 librados: starting objecter 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.535+0000 7f140a5a1640 1 Processor -- start 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.535+0000 7f140a5a1640 1 -- start start 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.535+0000 7f140a5a1640 1 librados: setting wanted keys 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.535+0000 7f140a5a1640 1 librados: calling monclient init 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.535+0000 7f140a5a1640 10 monclient: init 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.535+0000 7f140a5a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.535+0000 7f140a5a1640 10 monclient: _add_conns ranks=[3,7,6,1,5,2,0,4,8] 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14040a2f40 0x7f14041354f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 monclient(hunting): picked mon.noname-g con 0x7f14040a2f40 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f14040a4df0 0x7f1404135a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 monclient(hunting): picked mon.noname-i con 0x7f14040a4df0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f14041366d0 0x7f1404120f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 monclient(hunting): picked mon.noname-d con 0x7f14041366d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f14040a1250 con 0x7f14040a2f40 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1404137060 con 0x7f14041366d0 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f14041bc770 con 0x7f14040a4df0 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 monclient(hunting): authenticate will time out at 1024.343081s 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f14041366d0 0x7f1404120f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f14041366d0 0x7f1404120f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:53210/0 (socket says 172.21.15.98:53210) 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1403fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14040a2f40 0x7f14041354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 1 -- 172.21.15.98:0/1509038669 learned_addr learned my addr 172.21.15.98:0/1509038669 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 10 monclient(hunting): get_auth_request con 0x7f14041366d0 auth_method 0 2024-06-11T19:13:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1403fff640 10 monclient(hunting): get_auth_request con 0x7f14040a2f40 auth_method 0 2024-06-11T19:13:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1403fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1403fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1403fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f14037fe640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f14040a4df0 0x7f1404135a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f14037fe640 10 monclient(hunting): get_auth_request con 0x7f14040a4df0 auth_method 0 2024-06-11T19:13:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f14037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f14037fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f14037fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1403fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1403fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1403fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f14037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f14037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f14037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 10 monclient(hunting): handle_auth_done global_id 54645 payload 824 2024-06-11T19:13:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 1 -- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f14040a4df0 msgr2=0x7f1404135a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f14040a4df0 0x7f1404135a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 1 -- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14040a2f40 msgr2=0x7f14041354f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14040a2f40 0x7f14041354f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 10 monclient: _finish_hunting 0 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 1 monclient: found mon.noname-d 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 1 -- 172.21.15.98:0/1509038669 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1404124bf0 con 0x7f14041366d0 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1403fff640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14040a2f40 0x7f14041354f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1408b3e640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f14041366d0 0x7f1404120f60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f13fc02c380 tx=0x7f13fc004a40 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 10 client.?.objecter ms_handle_connect 0x7f14041366d0 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f13fc042070 con 0x7f14041366d0 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 10 monclient: dump: 2024-06-11T19:13:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.560 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 10 monclient: _finish_auth 0 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13fc0046c0 con 0x7f14041366d0 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 5 monclient: authenticate success, global_id 54645 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 monclient: _renew_subs 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 1 -- 172.21.15.98:0/1509038669 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1404123740 con 0x7f14041366d0 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f13fc009510 con 0x7f14041366d0 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f1400ff9640 10 monclient: dump: 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 client.54645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f14027fc640 4 set_mon_vals no callback set 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 monclient: _renew_subs 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 1 -- 172.21.15.98:0/1509038669 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f14040a3ff0 con 0x7f14041366d0 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.536+0000 7f140a5a1640 1 librados: init done 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.537+0000 7f140a5a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.537+0000 7f140a5a1640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f14041214a0 0x7f1404121880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.537+0000 7f140a5a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.537+0000 7f140a5a1640 1 -- 172.21.15.98:0/1509038669 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f14040a13d0 con 0x7f14041214a0 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.537+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f13fc005d10 con 0x7f14041366d0 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.537+0000 7f1403fff640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f14041214a0 0x7f1404121880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.537+0000 7f1403fff640 10 monclient: get_auth_request con 0x7f14041214a0 auth_method 0 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.537+0000 7f1403fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.537+0000 7f1403fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.537+0000 7f1403fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.537+0000 7f1400ff9640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f13e403fdb0 0x7f13e4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1403fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1403fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(179..179 src has 1..179) ==== 6757+0+0 (secure 0 0 0) 0x7f13fc074220 con 0x7f14041366d0 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1403fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 10 client.54645.objecter ms_dispatch 0x7f1404000b90 osd_map(179..179 src has 1..179) 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 3 client.54645.objecter handle_osd_map got epochs [179,179] > 0 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 3 client.54645.objecter handle_osd_map decoding full epoch 179 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 20 client.54645.objecter dump_active .. 0 homeless 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f14037fe640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f13e403fdb0 0x7f13e4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f14037fe640 10 monclient: get_auth_request con 0x7f13e403fdb0 auth_method 0 2024-06-11T19:13:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1403fff640 10 monclient(hunting): handle_auth_done global_id 22543 payload 824 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1403fff640 1 -- 172.21.15.98:0/1509038669 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1404122c10 con 0x7f14041214a0 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 10 client.54645.objecter ms_handle_connect 0x7f14041214a0 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1403fff640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f14041214a0 0x7f1404121880 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f13f400fbd0 tx=0x7f13f400d590 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 10 client.54645.objecter resend_mon_ops 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 10 client.54645.objecter ms_handle_connect 0x7f13e403fdb0 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f14037fe640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f13e403fdb0 0x7f13e4042270 secure :-1 s=READY pgs=4073 cs=0 l=1 rev1=1 crypto rx=0x7f14041226c0 tx=0x7f13ec00f040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f13f400fcc0 con 0x7f14041214a0 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 10 monclient: dump: 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.567 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f13f400fcc0 con 0x7f14041214a0 2024-06-11T19:13:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f14041214a0 msgr2=0x7f1404121880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f14041214a0 0x7f1404121880 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f13f400fbd0 tx=0x7f13f400d590 comp rx=0 tx=0).stop 2024-06-11T19:13:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f14041214a0 msgr2=0x7f1404121880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.538+0000 7f1400ff9640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f14041214a0 0x7f1404121880 secure :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f13f400fbd0 tx=0x7f13f400d590 comp rx=0 tx=0).stop 2024-06-11T19:13:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f140a5a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f140a5a1640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1404121dc0 0x7f14040705e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f140a5a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f140a5a1640 1 -- 172.21.15.98:0/1509038669 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f1404050830 con 0x7f1404121dc0 2024-06-11T19:13:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1403fff640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1404121dc0 0x7f14040705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1403fff640 10 monclient: get_auth_request con 0x7f1404121dc0 auth_method 0 2024-06-11T19:13:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1403fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1403fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1403fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1403fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1403fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1403fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1403fff640 10 monclient(hunting): handle_auth_done global_id 22552 payload 824 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1403fff640 1 -- 172.21.15.98:0/1509038669 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1404136ae0 con 0x7f1404121dc0 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1403fff640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1404121dc0 0x7f14040705e0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f13f4008010 tx=0x7f13f40046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1400ff9640 10 client.54645.objecter ms_handle_connect 0x7f1404121dc0 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.545+0000 7f1400ff9640 10 client.54645.objecter resend_mon_ops 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f13f40044a0 con 0x7f1404121dc0 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f1400ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f1400ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f1400ff9640 10 monclient: dump: 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7f13f4004030 con 0x7f1404121dc0 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f1400ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f1400ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:34.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1404121dc0 msgr2=0x7f14040705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f1400ff9640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1404121dc0 0x7f14040705e0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f13f4008010 tx=0x7f13f40046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f1400ff9640 1 -- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1404121dc0 msgr2=0x7f14040705e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:34.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f1400ff9640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1404121dc0 0x7f14040705e0 secure :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f13f4008010 tx=0x7f13f40046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.572 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 264762, 2024-06-11T19:13:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:34.574 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:34.575 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:34.576 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:34.577 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.579 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.580 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.583 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.584 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.586 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.587 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:34.588 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:34.591 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:34.592 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.593 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:34.594 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:34.595 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:34.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 10 librados: watch_flush enter 2024-06-11T19:13:34.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 10 librados: watch_flush exit 2024-06-11T19:13:34.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 20 client.54645.objecter shutdown clearing up homeless session... 2024-06-11T19:13:34.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 10 client.54645.objecter successfully canceled tick 2024-06-11T19:13:34.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 1 -- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f13e403fdb0 msgr2=0x7f13e4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f13e403fdb0 0x7f13e4042270 secure :-1 s=READY pgs=4073 cs=0 l=1 rev1=1 crypto rx=0x7f14041226c0 tx=0x7f13ec00f040 comp rx=0 tx=0).stop 2024-06-11T19:13:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 10 monclient: shutdown 2024-06-11T19:13:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 1 -- 172.21.15.98:0/1509038669 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f14041366d0 msgr2=0x7f1404120f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f1408b3e640 1 -- 172.21.15.98:0/1509038669 reap_dead start 2024-06-11T19:13:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f14041366d0 0x7f1404120f60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f13fc02c380 tx=0x7f13fc004a40 comp rx=0 tx=0).stop 2024-06-11T19:13:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 1 -- 172.21.15.98:0/1509038669 shutdown_connections 2024-06-11T19:13:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 1 --2- 172.21.15.98:0/1509038669 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f14041366d0 0x7f1404120f60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 1 -- 172.21.15.98:0/1509038669 >> 172.21.15.98:0/1509038669 conn(0x7f14040a95b0 msgr2=0x7f140409f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 1 -- 172.21.15.98:0/1509038669 shutdown_connections 2024-06-11T19:13:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 1 -- 172.21.15.98:0/1509038669 wait complete. 2024-06-11T19:13:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.546+0000 7f140a5a1640 1 librados: shutdown 2024-06-11T19:13:34.597 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:34.598 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-06-11T19:13:34.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-06-11T19:13:34.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.697+0000 7f2280654640 10 monclient: get_monmap_and_config 2024-06-11T19:13:34.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.697+0000 7f2280654640 10 monclient: build_initial_monmap 2024-06-11T19:13:34.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.697+0000 7f2280654640 10 monclient: monmap: 2024-06-11T19:13:34.696 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:34.697 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.697 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:34.698296+0000 2024-06-11T19:13:34.697 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:34.698296+0000 2024-06-11T19:13:34.697 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:34.697 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.697 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:34.697 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:34.697 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:34.697 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:34.697 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:34.697 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:34.698 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.697+0000 7f2280654640 1 Processor -- start 2024-06-11T19:13:34.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.697+0000 7f2280654640 1 -- start start 2024-06-11T19:13:34.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.697+0000 7f2280654640 10 monclient: init 2024-06-11T19:13:34.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:34.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 10 monclient: _add_conns ranks=[0,10,5,7,8,9,1,4,6,3,2] 2024-06-11T19:13:34.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22780a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:13:34.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f22780a2530 0x7f227813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 10 monclient(hunting): picked mon.noname-f con 0x7f22780a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:34.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f227813e1a0 0x7f2278140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 10 monclient(hunting): picked mon.noname-h con 0x7f227813e1a0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22780a7140 con 0x7f22780a4920 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f22780a2b50 con 0x7f22780a2530 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f22780a2e30 con 0x7f227813e1a0 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f2280654640 10 monclient(hunting): authenticate will time out at 1024.505623s 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227e3f0640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f22780a4920 0x7f22780a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50134/0 (socket says 172.21.15.98:50134) 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227e3f0640 1 -- 172.21.15.98:0/3779098013 learned_addr learned my addr 172.21.15.98:0/3779098013 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227ebf1640 1 --2- 172.21.15.98:0/3779098013 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f227813e1a0 0x7f2278140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227dbef640 1 --2- 172.21.15.98:0/3779098013 >> v2:172.21.15.117:3300/0 conn(0x7f22780a2530 0x7f227813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227ebf1640 10 monclient(hunting): get_auth_request con 0x7f227813e1a0 auth_method 0 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227ebf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227ebf1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227ebf1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227dbef640 10 monclient(hunting): get_auth_request con 0x7f22780a2530 auth_method 0 2024-06-11T19:13:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227dbef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227dbef640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227dbef640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.698+0000 7f227d3ee640 1 -- 172.21.15.98:0/3779098013 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1692596842 0 0) 0x7f22780a7140 con 0x7f22780a4920 2024-06-11T19:13:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227d3ee640 10 monclient(hunting): _negotiate 2024-06-11T19:13:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227d3ee640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227d3ee640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227d3ee640 10 monclient(hunting): my global_id is 64161 2024-06-11T19:13:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227d3ee640 1 -- 172.21.15.98:0/3779098013 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2264003520 con 0x7f22780a4920 2024-06-11T19:13:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227ebf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227ebf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227ebf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227dbef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227dbef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227dbef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227ebf1640 10 monclient(hunting): handle_auth_done global_id 22478 payload 274 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227d3ee640 1 -- 172.21.15.98:0/3779098013 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 519554350 0 0) 0x7f2264003520 con 0x7f22780a4920 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227ebf1640 1 -- 172.21.15.98:0/3779098013 >> v2:172.21.15.117:3300/0 conn(0x7f22780a2530 msgr2=0x7f227813dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227ebf1640 1 --2- 172.21.15.98:0/3779098013 >> v2:172.21.15.117:3300/0 conn(0x7f22780a2530 0x7f227813dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227ebf1640 1 -- 172.21.15.98:0/3779098013 >> v1:172.21.15.98:6789/0 conn(0x7f22780a4920 legacy=0x7f22780a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227ebf1640 10 monclient: _finish_hunting 0 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227ebf1640 1 monclient: found mon.noname-h 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227ebf1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227ebf1640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:13:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227ebf1640 1 -- 172.21.15.98:0/3779098013 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22780a3110 con 0x7f227813e1a0 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227d3ee640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.699+0000 7f227dbef640 1 --2- 172.21.15.98:0/3779098013 >> v2:172.21.15.117:3300/0 conn(0x7f22780a2530 0x7f227813dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227ebf1640 1 --2- 172.21.15.98:0/3779098013 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f227813e1a0 0x7f2278140590 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f2270009d90 tx=0x7f227002bc50 comp rx=0 tx=0).ready entity=mon.5 client_cookie=c3726d9f8d2fa793 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 1 -- 172.21.15.98:0/3779098013 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2270039070 con 0x7f227813e1a0 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 10 monclient: dump: 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 10 monclient: _finish_auth 0 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 10 monclient: _check_auth_tickets 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 1 -- 172.21.15.98:0/3779098013 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f227002cbf0 con 0x7f227813e1a0 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 5 monclient: authenticate success, global_id 22478 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 1 -- 172.21.15.98:0/3779098013 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2268001960 con 0x7f227813e1a0 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 4 set_mon_vals no callback set 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f227d3ee640 10 monclient: dump: 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 10 monclient: shutdown 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 1 -- 172.21.15.98:0/3779098013 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f227813e1a0 msgr2=0x7f2278140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 1 --2- 172.21.15.98:0/3779098013 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f227813e1a0 0x7f2278140590 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f2270009d90 tx=0x7f227002bc50 comp rx=0 tx=0).stop 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 1 -- 172.21.15.98:0/3779098013 shutdown_connections 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 1 --2- 172.21.15.98:0/3779098013 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f227813e1a0 0x7f2278140590 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 1 --2- 172.21.15.98:0/3779098013 >> v2:172.21.15.117:3300/0 conn(0x7f22780a2530 0x7f227813dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 1 -- 172.21.15.98:0/3779098013 >> 172.21.15.98:0/3779098013 conn(0x7f22780ab850 msgr2=0x7f22780a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 1 -- 172.21.15.98:0/3779098013 shutdown_connections 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.700+0000 7f2280654640 1 -- 172.21.15.98:0/3779098013 wait complete. 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.701+0000 7f2280654640 10 monclient: build_initial_monmap 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.701+0000 7f2280654640 10 monclient: monmap: 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.701+0000 7f2280654640 1 librados: starting msgr at 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.701+0000 7f2280654640 1 librados: starting objecter 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.701+0000 7f2280654640 1 Processor -- start 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.701+0000 7f2280654640 1 -- start start 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.701+0000 7f2280654640 1 librados: setting wanted keys 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.701+0000 7f2280654640 1 librados: calling monclient init 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.701+0000 7f2280654640 10 monclient: init 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 10 monclient: _add_conns ranks=[0,6,4,8,7,5,3,2,1] 2024-06-11T19:13:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22780a2530 0x7f22781cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 10 monclient(hunting): picked mon.noname-a con 0x7f22780a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22780a4920 0x7f2278135b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 10 monclient(hunting): picked mon.noname-d con 0x7f22780a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f227813e1a0 0x7f227809ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 10 monclient(hunting): picked mon.noname-c con 0x7f227813e1a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2278138e50 con 0x7f22780a2530 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f22780a84b0 con 0x7f227813e1a0 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f22780a6da0 con 0x7f22780a4920 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f2280654640 10 monclient(hunting): authenticate will time out at 1024.509396s 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22780a2530 0x7f22781cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22780a2530 0x7f22781cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49830/0 (socket says 172.21.15.98:49830) 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227dbef640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22780a4920 0x7f2278135b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 1 -- 172.21.15.98:0/1448761816 learned_addr learned my addr 172.21.15.98:0/1448761816 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 10 monclient(hunting): get_auth_request con 0x7f22780a2530 auth_method 0 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227ebf1640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f227813e1a0 0x7f227809ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227dbef640 10 monclient(hunting): get_auth_request con 0x7f22780a4920 auth_method 0 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227dbef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227dbef640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227dbef640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227ebf1640 10 monclient(hunting): get_auth_request con 0x7f227813e1a0 auth_method 0 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227ebf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227ebf1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227ebf1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227dbef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227dbef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227dbef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227ebf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227ebf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227ebf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 10 monclient(hunting): handle_auth_done global_id 64170 payload 824 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 1 -- 172.21.15.98:0/1448761816 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22780a4920 msgr2=0x7f2278135b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22780a4920 0x7f2278135b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 1 -- 172.21.15.98:0/1448761816 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f227813e1a0 msgr2=0x7f227809ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f227813e1a0 0x7f227809ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 10 monclient: _finish_hunting 0 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 1 monclient: found mon.noname-a 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 1 -- 172.21.15.98:0/1448761816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22781ce5d0 con 0x7f22780a2530 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227dbef640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22780a4920 0x7f2278135b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f227e3f0640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22780a2530 0x7f22781cd6b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f226800c700 tx=0x7f226800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f226effd640 10 client.?.objecter ms_handle_connect 0x7f22780a2530 2024-06-11T19:13:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.702+0000 7f226effd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 1 -- 172.21.15.98:0/1448761816 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2268005d40 con 0x7f22780a2530 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 10 monclient: dump: 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 10 monclient: _finish_auth 0 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 10 monclient: _check_auth_tickets 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 1 -- 172.21.15.98:0/1448761816 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2268007070 con 0x7f22780a2530 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f2280654640 5 monclient: authenticate success, global_id 64170 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 1 -- 172.21.15.98:0/1448761816 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2268007710 con 0x7f22780a2530 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f2280654640 10 monclient: _renew_subs 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f2280654640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:13:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f2280654640 1 -- 172.21.15.98:0/1448761816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22780a07c0 con 0x7f22780a2530 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f227cbed640 4 set_mon_vals no callback set 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f226effd640 10 monclient: dump: 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f2280654640 10 client.64170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f2280654640 10 monclient: _renew_subs 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f2280654640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f2280654640 1 -- 172.21.15.98:0/1448761816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22780aa560 con 0x7f22780a2530 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.703+0000 7f2280654640 1 librados: init done 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f2280654640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f226effd640 1 -- 172.21.15.98:0/1448761816 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f22680072a0 con 0x7f22780a2530 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f2280654640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f2280654640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:13:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f2280654640 1 -- 172.21.15.98:0/1448761816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22780730f0 con 0x7f22780a2530 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f226effd640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f225c03fdb0 0x7f225c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f226effd640 1 -- 172.21.15.98:0/1448761816 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 6757+0+0 (secure 0 0 0) 0x7f2268051a20 con 0x7f22780a2530 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f226effd640 10 client.64170.objecter ms_dispatch 0x7f2278000b90 osd_map(179..179 src has 1..179) 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f226effd640 3 client.64170.objecter handle_osd_map got epochs [179,179] > 0 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f226effd640 3 client.64170.objecter handle_osd_map decoding full epoch 179 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f226effd640 20 client.64170.objecter dump_active .. 0 homeless 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f227dbef640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f225c03fdb0 0x7f225c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f227dbef640 10 monclient: get_auth_request con 0x7f225c03fdb0 auth_method 0 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f226effd640 10 client.64170.objecter ms_handle_connect 0x7f225c03fdb0 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.704+0000 7f227dbef640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f225c03fdb0 0x7f225c042270 secure :-1 s=READY pgs=4074 cs=0 l=1 rev1=1 crypto rx=0x7f2274009560 tx=0x7f2274009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.708+0000 7f226effd640 1 -- 172.21.15.98:0/1448761816 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2268005ab0 con 0x7f22780a2530 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.708+0000 7f226effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.708+0000 7f226effd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:34.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.970+0000 7f2280654640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-06-11T19:13:34.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.970+0000 7f2280654640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-06-11T19:13:34.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.970+0000 7f2280654640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:13:34.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.970+0000 7f2280654640 1 -- 172.21.15.98:0/1448761816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f22781215f0 con 0x7f22780a2530 2024-06-11T19:13:34.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.972+0000 7f226effd640 1 -- 172.21.15.98:0/1448761816 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f2268019300 con 0x7f22780a2530 2024-06-11T19:13:34.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.972+0000 7f226effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-06-11T19:13:34.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.972+0000 7f226effd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:34.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 10 librados: watch_flush enter 2024-06-11T19:13:34.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 10 librados: watch_flush exit 2024-06-11T19:13:34.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 20 client.64170.objecter shutdown clearing up homeless session... 2024-06-11T19:13:34.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 10 client.64170.objecter successfully canceled tick 2024-06-11T19:13:34.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 1 -- 172.21.15.98:0/1448761816 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f225c03fdb0 msgr2=0x7f225c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f225c03fdb0 0x7f225c042270 secure :-1 s=READY pgs=4074 cs=0 l=1 rev1=1 crypto rx=0x7f2274009560 tx=0x7f2274009190 comp rx=0 tx=0).stop 2024-06-11T19:13:34.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 10 monclient: shutdown 2024-06-11T19:13:34.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:34.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 1 -- 172.21.15.98:0/1448761816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22780a2530 msgr2=0x7f22781cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:34.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22780a2530 0x7f22781cd6b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f226800c700 tx=0x7f226800cbc0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 1 -- 172.21.15.98:0/1448761816 shutdown_connections 2024-06-11T19:13:34.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f225c03fdb0 0x7f225c042270 unknown :-1 s=CLOSED pgs=4074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f227813e1a0 0x7f227809ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22780a4920 0x7f2278135b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 1 --2- 172.21.15.98:0/1448761816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22780a2530 0x7f22781cd6b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 1 -- 172.21.15.98:0/1448761816 >> 172.21.15.98:0/1448761816 conn(0x7f22780ab850 msgr2=0x7f22780726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 1 -- 172.21.15.98:0/1448761816 shutdown_connections 2024-06-11T19:13:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.975+0000 7f2280654640 1 -- 172.21.15.98:0/1448761816 wait complete. 2024-06-11T19:13:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:34.976+0000 7f2280654640 1 librados: shutdown 2024-06-11T19:13:34.992 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-06-11T19:13:35.034 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.034 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=635154577,config=692619927,health=920097043,kv=2025760297,logm=412134247}) 2024-06-11T19:13:35.034 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=1,health=12,kv=8,logm=65} crc {auth=635154577,config=3678701417,health=663884318,kv=708759692,logm=766972264}) 2024-06-11T19:13:35.034 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.034 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=635154577,config=692619927,health=920097043,kv=2025760297,logm=412134247}) 2024-06-11T19:13:35.034 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=12,kv=10,logm=64} crc {auth=862446774,config=692619927,health=841574241,kv=2000936593,logm=1782276871}) 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=635154577,config=692619927,health=920097043,kv=2025760297,logm=412134247}) 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=14,config=2,health=11,kv=10,logm=63} crc {auth=635154577,config=692619927,health=920097043,kv=2000936593,logm=4170972444}) 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=635154577,config=692619927,health=920097043,kv=2025760297,logm=412134247}) 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=12,config=2,health=11,kv=9,logm=66} crc {auth=1024761455,config=692619927,health=816214272,kv=1471832156,logm=875045748}) 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=635154577,config=692619927,health=920097043,kv=2025760297,logm=412134247}) 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=14,config=1,health=12,kv=10,logm=63} crc {auth=635154577,config=3678701417,health=663884318,kv=2000936593,logm=929727956}) 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=635154577,config=692619927,health=920097043,kv=2025760297,logm=412134247}) 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=13,config=2,health=12,kv=10,logm=63} crc {auth=3279380942,config=692619927,health=663884318,kv=2000936593,logm=3408299443}) 2024-06-11T19:13:35.035 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=635154577,config=692619927,health=920097043,kv=2025760297,logm=412134247}) 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=13,config=2,health=12,kv=9,logm=64} crc {auth=3279380942,config=692619927,health=663884318,kv=1471832156,logm=3007572202}) 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=635154577,config=692619927,health=920097043,kv=2025760297,logm=412134247}) 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.973+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=635154577,config=692619927,health=663884318,kv=1820642019,logm=3461717688}) 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500715728}) 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=928208294}) 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500715728}) 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1585479509}) 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500715728}) 2024-06-11T19:13:35.036 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2490636596}) 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500715728}) 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2874053257}) 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500715728}) 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4217890960}) 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500715728}) 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=28692404}) 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500715728}) 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=504035900}) 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.037 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500715728}) 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.975+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2356844467}) 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3834738929}) 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384456274}) 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3834738929}) 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3150085277}) 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3834738929}) 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1634656055}) 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3834738929}) 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2401478732}) 2024-06-11T19:13:35.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3834738929}) 2024-06-11T19:13:35.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2417446941}) 2024-06-11T19:13:35.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3834738929}) 2024-06-11T19:13:35.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1463687601}) 2024-06-11T19:13:35.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3834738929}) 2024-06-11T19:13:35.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=654673374}) 2024-06-11T19:13:35.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3834738929}) 2024-06-11T19:13:35.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.977+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1478396890}) 2024-06-11T19:13:35.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=883691139}) 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2317695266}) 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=883691139}) 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1389168176}) 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=883691139}) 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=991615261}) 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=883691139}) 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1978595823}) 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=883691139}) 2024-06-11T19:13:35.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2759033427}) 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=883691139}) 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2315743866}) 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=883691139}) 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3419424077}) 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=883691139}) 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.978+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1563971723}) 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873805366}) 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466070393}) 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873805366}) 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4163569876}) 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873805366}) 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1822536568}) 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873805366}) 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=571338755}) 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873805366}) 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3646146300}) 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873805366}) 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1857163320}) 2024-06-11T19:13:35.042 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873805366}) 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1256473507}) 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873805366}) 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.980+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2138172309}) 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522796767}) 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=319051661}) 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522796767}) 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3647055189}) 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.043 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522796767}) 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3235579923}) 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522796767}) 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3956395911}) 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522796767}) 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1389561455}) 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522796767}) 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1203060273}) 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522796767}) 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1748824494}) 2024-06-11T19:13:35.044 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522796767}) 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.981+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1637422443}) 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.982+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.982+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388176673}) 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.982+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2499972786}) 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.982+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388176673}) 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251383685}) 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388176673}) 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1817995924}) 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.045 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388176673}) 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3171579941}) 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388176673}) 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1927940895}) 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388176673}) 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3822166520}) 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388176673}) 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3468590972}) 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388176673}) 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.983+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3677242274}) 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.046 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667782554}) 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979886843}) 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667782554}) 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1506774054}) 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667782554}) 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=835184019}) 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667782554}) 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3118222942}) 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667782554}) 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=154869938}) 2024-06-11T19:13:35.047 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667782554}) 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=674797448}) 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667782554}) 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=760750287}) 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667782554}) 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.984+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=907639480}) 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589429492}) 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2939379470}) 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589429492}) 2024-06-11T19:13:35.048 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3674212002}) 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589429492}) 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4159186286}) 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589429492}) 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=72553316}) 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589429492}) 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1194669778}) 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589429492}) 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=682190216}) 2024-06-11T19:13:35.049 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589429492}) 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=393409180}) 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.985+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589429492}) 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.986+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1894876723}) 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750468266}) 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1048323303}) 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750468266}) 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3824066926}) 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750468266}) 2024-06-11T19:13:35.050 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2122171787}) 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750468266}) 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3729920537}) 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750468266}) 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4152652687}) 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750468266}) 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2553765686}) 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750468266}) 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=761314359}) 2024-06-11T19:13:35.051 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750468266}) 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.987+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=464007885}) 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124565696}) 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434115907}) 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124565696}) 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221076952}) 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124565696}) 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3144646314}) 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124565696}) 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2495571247}) 2024-06-11T19:13:35.052 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124565696}) 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=746699632}) 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124565696}) 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4213026543}) 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124565696}) 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.988+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=615496123}) 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.989+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.989+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124565696}) 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.989+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3989131423}) 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.053 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350617189}) 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195110798}) 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350617189}) 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3997514551}) 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350617189}) 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2145233975}) 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350617189}) 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3435376383}) 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350617189}) 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=495682538}) 2024-06-11T19:13:35.054 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350617189}) 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2585337399}) 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350617189}) 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=545479677}) 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350617189}) 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.990+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4034799955}) 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462392109}) 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2482269173}) 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462392109}) 2024-06-11T19:13:35.055 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=55053863}) 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462392109}) 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1608857849}) 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462392109}) 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3733293407}) 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462392109}) 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1654410477}) 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462392109}) 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3055679546}) 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.056 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462392109}) 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1729971661}) 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462392109}) 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.991+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3167076277}) 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947259287}) 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205238869}) 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947259287}) 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3129510138}) 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947259287}) 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1657146688}) 2024-06-11T19:13:35.057 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947259287}) 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3919959116}) 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947259287}) 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4264455771}) 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947259287}) 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3494933475}) 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947259287}) 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3522688101}) 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947259287}) 2024-06-11T19:13:35.058 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.992+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1955078211}) 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921308484}) 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487927104}) 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921308484}) 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2342038748}) 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921308484}) 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1920170797}) 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921308484}) 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=928199867}) 2024-06-11T19:13:35.059 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921308484}) 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=734106376}) 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921308484}) 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=861718383}) 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921308484}) 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3675896201}) 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921308484}) 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.994+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2388180355}) 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524617686}) 2024-06-11T19:13:35.060 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814506793}) 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524617686}) 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3839154099}) 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524617686}) 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1790365604}) 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524617686}) 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3822248849}) 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524617686}) 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3723025263}) 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.061 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524617686}) 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1305496786}) 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524617686}) 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1960505509}) 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524617686}) 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.995+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2232497149}) 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.996+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.996+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833455453}) 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.996+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264395931}) 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.996+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.996+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833455453}) 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.996+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4066750530}) 2024-06-11T19:13:35.062 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.996+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.996+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833455453}) 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.996+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=656759778}) 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.996+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833455453}) 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=883248290}) 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833455453}) 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4187431550}) 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833455453}) 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1737513148}) 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.063 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833455453}) 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4057431598}) 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833455453}) 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.997+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2486822492}) 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=677286957}) 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=821639148}) 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=677286957}) 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2298536246}) 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=677286957}) 2024-06-11T19:13:35.064 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4051021121}) 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=677286957}) 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=363974267}) 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=677286957}) 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2924672438}) 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=677286957}) 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=884170663}) 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=677286957}) 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2490900256}) 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.065 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=677286957}) 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.998+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1949417902}) 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=980095297}) 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3533644847}) 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=980095297}) 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4221021419}) 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=980095297}) 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4269175319}) 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=980095297}) 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3202049434}) 2024-06-11T19:13:35.066 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=980095297}) 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3551369109}) 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=980095297}) 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1152378363}) 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=980095297}) 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3269489688}) 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=980095297}) 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:34.999+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2779442827}) 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956915362}) 2024-06-11T19:13:35.067 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364126054}) 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956915362}) 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010234723}) 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956915362}) 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2387437819}) 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956915362}) 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1490760652}) 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956915362}) 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1009763852}) 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.068 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956915362}) 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4254876938}) 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956915362}) 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3631450564}) 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956915362}) 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.000+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2825628508}) 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543343160}) 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372103557}) 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543343160}) 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4130553096}) 2024-06-11T19:13:35.069 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543343160}) 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1655532886}) 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543343160}) 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4225801379}) 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543343160}) 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2819856}) 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543343160}) 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1810390194}) 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.001+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543343160}) 2024-06-11T19:13:35.070 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.002+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4181674770}) 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.002+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.002+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543343160}) 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.002+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=709831661}) 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=193747298}) 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3315112052}) 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=193747298}) 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3567260459}) 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=193747298}) 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2125685352}) 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=193747298}) 2024-06-11T19:13:35.071 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1756155408}) 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=193747298}) 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=144416006}) 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=193747298}) 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3545926708}) 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=193747298}) 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=579302329}) 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=193747298}) 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.003+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3627017372}) 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.072 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=162023664}) 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374060923}) 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=162023664}) 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2752969529}) 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=162023664}) 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4187055287}) 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=162023664}) 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2978337278}) 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=162023664}) 2024-06-11T19:13:35.073 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1984737728}) 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=162023664}) 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=600929766}) 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=162023664}) 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1549703286}) 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=162023664}) 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.004+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=701779717}) 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068362655}) 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2488313293}) 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.074 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068362655}) 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3341411943}) 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068362655}) 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1621797561}) 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068362655}) 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=925425453}) 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068362655}) 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2639050423}) 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068362655}) 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2738320852}) 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.075 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068362655}) 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1211882996}) 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068362655}) 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.006+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3110501128}) 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101949206}) 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1079473695}) 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101949206}) 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185712140}) 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101949206}) 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1609991368}) 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.076 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101949206}) 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2341087418}) 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101949206}) 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=333887534}) 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101949206}) 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1217870892}) 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101949206}) 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4264848255}) 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101949206}) 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.007+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4182739451}) 2024-06-11T19:13:35.077 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.078 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692680146}) 2024-06-11T19:13:35.078 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2210081715}) 2024-06-11T19:13:35.078 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.078 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692680146}) 2024-06-11T19:13:35.078 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3904080623}) 2024-06-11T19:13:35.078 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.078 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692680146}) 2024-06-11T19:13:35.078 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=496993855}) 2024-06-11T19:13:35.078 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.078 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692680146}) 2024-06-11T19:13:35.078 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1012333351}) 2024-06-11T19:13:35.078 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692680146}) 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3397927978}) 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.008+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.009+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692680146}) 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.009+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1917778872}) 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.009+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.009+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692680146}) 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.009+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3721912273}) 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.009+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.009+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692680146}) 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.009+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=258561997}) 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408883243}) 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421696190}) 2024-06-11T19:13:35.079 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408883243}) 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3707656230}) 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408883243}) 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1557605288}) 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408883243}) 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2167606913}) 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408883243}) 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3463927688}) 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408883243}) 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3906756786}) 2024-06-11T19:13:35.080 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408883243}) 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3788405272}) 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408883243}) 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.010+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3687891007}) 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197643247}) 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=855662031}) 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197643247}) 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3199826782}) 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197643247}) 2024-06-11T19:13:35.081 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1438666887}) 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197643247}) 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2400095869}) 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197643247}) 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2732952262}) 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197643247}) 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1900131253}) 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197643247}) 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3563289595}) 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.082 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197643247}) 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.011+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2933264180}) 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6} crc {logm=388595978,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgrstat=1156990293,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398}) 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=59,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1} crc {logm=958740785,mdsmap=1401145748,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1156990293,mon_config_key=124724172,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=249489454,osd_snap=1508070444}) 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6} crc {logm=388595978,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgrstat=1156990293,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398}) 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=56,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=5} crc {logm=236434891,mdsmap=2840952586,mgr=3850381706,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1902696833,mon_config_key=2314039359,monmap=1460791231,osd_metadata=728168835,osd_pg_creating=249489454,osd_snap=1426522944}) 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6} crc {logm=388595978,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgrstat=1156990293,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398}) 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=62,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=9,monmap=3,osd_metadata=5} crc {logm=3751443686,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1156990293,mon_config_key=2764689593,monmap=506305176,osd_metadata=728168835}) 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6} crc {logm=388595978,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgrstat=1156990293,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398}) 2024-06-11T19:13:35.083 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=5} crc {logm=1725356459,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1156990293,mon_config_key=2314039359,monmap=506305176,osd_metadata=728168835}) 2024-06-11T19:13:35.084 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.084 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6} crc {logm=388595978,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgrstat=1156990293,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398}) 2024-06-11T19:13:35.084 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=57,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=5,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=5} crc {logm=3479282545,mdsmap=2840952586,mgr=2102212137,mgr_metadata=1826097544,mgrstat=3484320305,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=249489454,osd_snap=1426522944}) 2024-06-11T19:13:35.084 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.084 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6} crc {logm=388595978,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgrstat=1156990293,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398}) 2024-06-11T19:13:35.084 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=62,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=11,monmap=3,osd_metadata=4} crc {logm=60054177,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2900511152,mon_config_key=2314039359,monmap=506305176,osd_metadata=3351267617}) 2024-06-11T19:13:35.084 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.084 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6} crc {logm=388595978,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgrstat=1156990293,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398}) 2024-06-11T19:13:35.084 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=61,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=4} crc {logm=2454638819,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1156990293,mon_config_key=2314039359,monmap=506305176,osd_metadata=3351267617}) 2024-06-11T19:13:35.084 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.084 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6} crc {logm=388595978,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgrstat=1156990293,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398}) 2024-06-11T19:13:35.084 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.013+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=64,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,osd_metadata=3} crc {logm=3965512987,mdsmap=2840952586,mgr=1864858522,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1156990293,mon_config_key=657769230,monmap=506305176,osd_metadata=1912569348}) 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_pg_creating=1,osd_snap=20,osdmap=79} crc {osd_pg_creating=249489454,osd_snap=510004664,osdmap=4136004014}) 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_pg_creating=1,osd_snap=20,osdmap=79} crc {osd_pg_creating=249489454,osd_snap=2942837642,osdmap=299769428}) 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_pg_creating=1,osd_snap=20,osdmap=79} crc {osd_pg_creating=249489454,osd_snap=510004664,osdmap=4136004014}) 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_pg_creating=1,osd_snap=23,osdmap=76} crc {osd_pg_creating=249489454,osd_snap=1978319181,osdmap=1828757832}) 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_pg_creating=1,osd_snap=20,osdmap=79} crc {osd_pg_creating=249489454,osd_snap=510004664,osdmap=4136004014}) 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_pg_creating=1,osd_snap=22,osdmap=77} crc {osd_pg_creating=249489454,osd_snap=1621787997,osdmap=3936767826}) 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_pg_creating=1,osd_snap=20,osdmap=79} crc {osd_pg_creating=249489454,osd_snap=510004664,osdmap=4136004014}) 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_pg_creating=1,osd_snap=21,osdmap=78} crc {osd_pg_creating=249489454,osd_snap=436838999,osdmap=2318015017}) 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_pg_creating=1,osd_snap=20,osdmap=79} crc {osd_pg_creating=249489454,osd_snap=510004664,osdmap=4136004014}) 2024-06-11T19:13:35.085 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osd_pg_creating=1,osd_snap=23,osdmap=76} crc {osd_pg_creating=249489454,osd_snap=1978319181,osdmap=4229866301}) 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_pg_creating=1,osd_snap=20,osdmap=79} crc {osd_pg_creating=249489454,osd_snap=510004664,osdmap=4136004014}) 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osd_pg_creating=1,osd_snap=22,osdmap=77} crc {osd_pg_creating=249489454,osd_snap=1035223998,osdmap=2879207008}) 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_pg_creating=1,osd_snap=20,osdmap=79} crc {osd_pg_creating=249489454,osd_snap=510004664,osdmap=4136004014}) 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osd_pg_creating=1,osd_snap=21,osdmap=78} crc {osd_pg_creating=249489454,osd_snap=928534933,osdmap=646866979}) 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_pg_creating=1,osd_snap=20,osdmap=79} crc {osd_pg_creating=249489454,osd_snap=510004664,osdmap=4136004014}) 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.014+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_pg_creating=1,osd_snap=23,osdmap=76} crc {osd_pg_creating=249489454,osd_snap=1978319181,osdmap=2359342942}) 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2300672548}) 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1379825960}) 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.086 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2300672548}) 2024-06-11T19:13:35.087 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=759868350}) 2024-06-11T19:13:35.087 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.087 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2300672548}) 2024-06-11T19:13:35.087 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=150835397}) 2024-06-11T19:13:35.087 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.087 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2300672548}) 2024-06-11T19:13:35.087 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2429732341}) 2024-06-11T19:13:35.087 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.087 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2300672548}) 2024-06-11T19:13:35.087 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4075483095}) 2024-06-11T19:13:35.087 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.087 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2300672548}) 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=424495058}) 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2300672548}) 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1739739062}) 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2300672548}) 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.016+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3876493494}) 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=96} crc {osdmap=233267793}) 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=98} crc {osdmap=2681105288}) 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=96} crc {osdmap=233267793}) 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=97} crc {osdmap=3020312106}) 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.088 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=96} crc {osdmap=233267793}) 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1150268114}) 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=96} crc {osdmap=233267793}) 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=98} crc {osdmap=2093180509}) 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=96} crc {osdmap=233267793}) 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=94} crc {osdmap=2229718982}) 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=96} crc {osdmap=233267793}) 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3654242995}) 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=96} crc {osdmap=233267793}) 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=97} crc {osdmap=4266273804}) 2024-06-11T19:13:35.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:13:35.090 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=96} crc {osdmap=233267793}) 2024-06-11T19:13:35.090 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:13:35.026+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=97} crc {osdmap=1876280941}) 2024-06-11T19:13:35.993 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:13:35.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:13:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.095+0000 7faf2e3b5640 10 monclient: get_monmap_and_config 2024-06-11T19:13:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.095+0000 7faf2e3b5640 10 monclient: build_initial_monmap 2024-06-11T19:13:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.095+0000 7faf2e3b5640 10 monclient: monmap: 2024-06-11T19:13:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:36.096318+0000 2024-06-11T19:13:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:36.096318+0000 2024-06-11T19:13:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:36.096 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 1 Processor -- start 2024-06-11T19:13:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 1 -- start start 2024-06-11T19:13:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 10 monclient: init 2024-06-11T19:13:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 10 monclient: _add_conns ranks=[1,7,6,10,3,9,4,8,5,2,0] 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7faf280a2f40 addr v1:172.21.15.117:6789/0 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7faf280a4fc0 0x7faf2813b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 10 monclient(hunting): picked mon.noname-i con 0x7faf280a4fc0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faf280a53a0 0x7faf2813df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 10 monclient(hunting): picked mon.noname-d con 0x7faf280a53a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf280a0ed0 con 0x7faf280a2f40 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7faf280a3320 con 0x7faf280a53a0 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7faf280a2460 con 0x7faf280a4fc0 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2e3b5640 10 monclient(hunting): authenticate will time out at 1025.903662s 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2c952640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faf280a53a0 0x7faf2813df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2c952640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faf280a53a0 0x7faf2813df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:53238/0 (socket says 172.21.15.98:53238) 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2c952640 1 -- 172.21.15.98:0/3768753819 learned_addr learned my addr 172.21.15.98:0/3768753819 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2c952640 10 monclient(hunting): get_auth_request con 0x7faf280a53a0 auth_method 0 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2c952640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2c952640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2c952640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2c952640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2c952640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf2c952640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf2c952640 10 monclient(hunting): handle_auth_done global_id 54672 payload 274 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.096+0000 7faf277fe640 1 --2- 172.21.15.98:0/3768753819 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7faf280a4fc0 0x7faf2813b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf2c952640 1 -- 172.21.15.98:0/3768753819 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7faf280a4fc0 msgr2=0x7faf2813b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf2c952640 1 --2- 172.21.15.98:0/3768753819 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7faf280a4fc0 0x7faf2813b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf2c952640 1 -- 172.21.15.98:0/3768753819 >> v1:172.21.15.117:6789/0 conn(0x7faf280a2f40 legacy=0x7faf280a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf2c952640 10 monclient: _finish_hunting 0 2024-06-11T19:13:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf2c952640 1 monclient: found mon.noname-d 2024-06-11T19:13:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf2c952640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf2c952640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:13:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf2c952640 1 -- 172.21.15.98:0/3768753819 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf280a2180 con 0x7faf280a53a0 2024-06-11T19:13:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf277fe640 10 monclient: get_auth_request con 0x7faf280a4fc0 auth_method 0 2024-06-11T19:13:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf277fe640 1 --2- 172.21.15.98:0/3768753819 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7faf280a4fc0 0x7faf2813b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:13:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf2c952640 1 --2- 172.21.15.98:0/3768753819 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faf280a53a0 0x7faf2813df40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faf20009eb0 tx=0x7faf2002be00 comp rx=0 tx=0).ready entity=mon.6 client_cookie=5408a9113906305b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf26ffd640 1 -- 172.21.15.98:0/3768753819 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faf2002cc10 con 0x7faf280a53a0 2024-06-11T19:13:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.097+0000 7faf26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf26ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:13:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf26ffd640 10 monclient: dump: 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.110 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf26ffd640 10 monclient: _finish_auth 0 2024-06-11T19:13:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf26ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:13:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf26ffd640 1 -- 172.21.15.98:0/3768753819 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf2002cdd0 con 0x7faf280a53a0 2024-06-11T19:13:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 5 monclient: authenticate success, global_id 54672 2024-06-11T19:13:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf26ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf26ffd640 1 -- 172.21.15.98:0/3768753819 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faf20034860 con 0x7faf280a53a0 2024-06-11T19:13:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:36.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 4 set_mon_vals no callback set 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf26ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf26ffd640 10 monclient: dump: 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 10 monclient: shutdown 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 1 -- 172.21.15.98:0/3768753819 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faf280a53a0 msgr2=0x7faf2813df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 1 --2- 172.21.15.98:0/3768753819 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faf280a53a0 0x7faf2813df40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faf20009eb0 tx=0x7faf2002be00 comp rx=0 tx=0).stop 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 1 -- 172.21.15.98:0/3768753819 shutdown_connections 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 1 --2- 172.21.15.98:0/3768753819 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faf280a53a0 0x7faf2813df40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 1 --2- 172.21.15.98:0/3768753819 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7faf280a4fc0 0x7faf2813b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 1 -- 172.21.15.98:0/3768753819 >> 172.21.15.98:0/3768753819 conn(0x7faf280a95b0 msgr2=0x7faf280a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 1 -- 172.21.15.98:0/3768753819 shutdown_connections 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.098+0000 7faf2e3b5640 1 -- 172.21.15.98:0/3768753819 wait complete. 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.099+0000 7faf2e3b5640 10 monclient: build_initial_monmap 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.099+0000 7faf2e3b5640 10 monclient: monmap: 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.099+0000 7faf2e3b5640 1 librados: starting msgr at 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.099+0000 7faf2e3b5640 1 librados: starting objecter 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.099+0000 7faf2e3b5640 1 Processor -- start 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.099+0000 7faf2e3b5640 1 -- start start 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.099+0000 7faf2e3b5640 1 librados: setting wanted keys 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.099+0000 7faf2e3b5640 1 librados: calling monclient init 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.099+0000 7faf2e3b5640 10 monclient: init 2024-06-11T19:13:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient: _add_conns ranks=[8,1,0,2,6,5,7,3,4] 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faf280a2f40 0x7faf28135d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient(hunting): picked mon.noname-e con 0x7faf280a2f40 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faf280a4fc0 0x7faf281362b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient(hunting): picked mon.noname-f con 0x7faf280a4fc0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf280a53a0 0x7faf281367f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient(hunting): picked mon.noname-a con 0x7faf280a53a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf2813e600 con 0x7faf280a53a0 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7faf2813e480 con 0x7faf280a4fc0 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7faf2813e780 con 0x7faf280a2f40 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient(hunting): authenticate will time out at 1025.907105s 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faf280a2f40 0x7faf28135d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faf280a2f40 0x7faf28135d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35362/0 (socket says 172.21.15.98:35362) 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 1 -- 172.21.15.98:0/885710255 learned_addr learned my addr 172.21.15.98:0/885710255 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2c952640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf280a53a0 0x7faf281367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 10 monclient(hunting): get_auth_request con 0x7faf280a2f40 auth_method 0 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2c952640 10 monclient(hunting): get_auth_request con 0x7faf280a53a0 auth_method 0 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2c952640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2c952640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2c952640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2c952640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2c952640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2c952640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf277fe640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faf280a4fc0 0x7faf281362b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf277fe640 10 monclient(hunting): get_auth_request con 0x7faf280a4fc0 auth_method 0 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf277fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf277fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 10 monclient(hunting): handle_auth_done global_id 64142 payload 824 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 1 -- 172.21.15.98:0/885710255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faf280a4fc0 msgr2=0x7faf281362b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faf280a4fc0 0x7faf281362b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 1 -- 172.21.15.98:0/885710255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf280a53a0 msgr2=0x7faf281367f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf280a53a0 0x7faf281367f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 10 monclient: _finish_hunting 0 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 1 monclient: found mon.noname-e 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 1 -- 172.21.15.98:0/885710255 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf280a8980 con 0x7faf280a2f40 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2c952640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf280a53a0 0x7faf281367f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf27fff640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faf280a2f40 0x7faf28135d70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faf1800c450 tx=0x7faf1800c910 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 10 client.?.objecter ms_handle_connect 0x7faf280a2f40 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 1 -- 172.21.15.98:0/885710255 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faf18001180 con 0x7faf280a2f40 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 10 monclient: dump: 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.119 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 10 monclient: _finish_auth 0 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 1 -- 172.21.15.98:0/885710255 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf18012630 con 0x7faf280a2f40 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 5 monclient: authenticate success, global_id 64142 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient: _renew_subs 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 1 -- 172.21.15.98:0/885710255 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2813cb40 con 0x7faf280a2f40 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 1 -- 172.21.15.98:0/885710255 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faf18012c50 con 0x7faf280a2f40 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf24ff9640 10 monclient: dump: 2024-06-11T19:13:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.122 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 client.64142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:36.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient: _renew_subs 2024-06-11T19:13:36.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:13:36.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 1 -- 172.21.15.98:0/885710255 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7faf2806dbb0 con 0x7faf280a2f40 2024-06-11T19:13:36.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf2e3b5640 1 librados: init done 2024-06-11T19:13:36.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.100+0000 7faf267fc640 4 set_mon_vals no callback set 2024-06-11T19:13:36.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.101+0000 7faf24ff9640 1 -- 172.21.15.98:0/885710255 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7faf180127e0 con 0x7faf280a2f40 2024-06-11T19:13:36.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.101+0000 7faf2e3b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:36.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.101+0000 7faf2e3b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:36.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.101+0000 7faf2e3b5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:13:36.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.101+0000 7faf2e3b5640 1 -- 172.21.15.98:0/885710255 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf280a48b0 con 0x7faf280a2f40 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.101+0000 7faf24ff9640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7faf1403fdb0 0x7faf14042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.102+0000 7faf24ff9640 1 -- 172.21.15.98:0/885710255 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(181..181 src has 1..181) ==== 6621+0+0 (secure 0 0 0) 0x7faf180516d0 con 0x7faf280a2f40 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.102+0000 7faf24ff9640 10 client.64142.objecter ms_dispatch 0x7faf28000b90 osd_map(181..181 src has 1..181) 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.102+0000 7faf24ff9640 3 client.64142.objecter handle_osd_map got epochs [181,181] > 0 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.102+0000 7faf24ff9640 3 client.64142.objecter handle_osd_map decoding full epoch 181 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.102+0000 7faf24ff9640 20 client.64142.objecter dump_active .. 0 homeless 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.102+0000 7faf277fe640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7faf1403fdb0 0x7faf14042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.102+0000 7faf277fe640 10 monclient: get_auth_request con 0x7faf1403fdb0 auth_method 0 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.102+0000 7faf24ff9640 10 client.64142.objecter ms_handle_connect 0x7faf1403fdb0 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.102+0000 7faf277fe640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7faf1403fdb0 0x7faf14042270 secure :-1 s=READY pgs=4076 cs=0 l=1 rev1=1 crypto rx=0x7faf2806d7b0 tx=0x7faf1c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.106+0000 7faf24ff9640 1 -- 172.21.15.98:0/885710255 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faf280a8980 con 0x7faf280a2f40 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.106+0000 7faf24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:36.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.106+0000 7faf24ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.360+0000 7faf2e3b5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:13:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.360+0000 7faf2e3b5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:13:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.360+0000 7faf2e3b5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:13:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.360+0000 7faf2e3b5640 1 -- 172.21.15.98:0/885710255 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faf2813c260 con 0x7faf280a2f40 2024-06-11T19:13:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.360+0000 7faf24ff9640 1 -- 172.21.15.98:0/885710255 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7faf18003d80 con 0x7faf280a2f40 2024-06-11T19:13:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.361+0000 7faf24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:13:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.361+0000 7faf24ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:36.361 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":20,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:13:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.362+0000 7faf2e3b5640 10 librados: watch_flush enter 2024-06-11T19:13:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.362+0000 7faf2e3b5640 10 librados: watch_flush exit 2024-06-11T19:13:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.362+0000 7faf2e3b5640 20 client.64142.objecter shutdown clearing up homeless session... 2024-06-11T19:13:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.362+0000 7faf2e3b5640 10 client.64142.objecter successfully canceled tick 2024-06-11T19:13:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.362+0000 7faf2e3b5640 1 -- 172.21.15.98:0/885710255 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7faf1403fdb0 msgr2=0x7faf14042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.362+0000 7faf2e3b5640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7faf1403fdb0 0x7faf14042270 secure :-1 s=READY pgs=4076 cs=0 l=1 rev1=1 crypto rx=0x7faf2806d7b0 tx=0x7faf1c008040 comp rx=0 tx=0).stop 2024-06-11T19:13:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 10 monclient: shutdown 2024-06-11T19:13:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 1 -- 172.21.15.98:0/885710255 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faf280a2f40 msgr2=0x7faf28135d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faf280a2f40 0x7faf28135d70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faf1800c450 tx=0x7faf1800c910 comp rx=0 tx=0).stop 2024-06-11T19:13:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 1 -- 172.21.15.98:0/885710255 shutdown_connections 2024-06-11T19:13:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7faf1403fdb0 0x7faf14042270 unknown :-1 s=CLOSED pgs=4076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf280a53a0 0x7faf281367f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faf280a4fc0 0x7faf281362b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 1 --2- 172.21.15.98:0/885710255 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faf280a2f40 0x7faf28135d70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 1 -- 172.21.15.98:0/885710255 >> 172.21.15.98:0/885710255 conn(0x7faf280a95b0 msgr2=0x7faf2809e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 1 -- 172.21.15.98:0/885710255 shutdown_connections 2024-06-11T19:13:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 1 -- 172.21.15.98:0/885710255 wait complete. 2024-06-11T19:13:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.363+0000 7faf2e3b5640 1 librados: shutdown 2024-06-11T19:13:36.372 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:13:36.373 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-06-11T19:13:36.373 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:13:36.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:13:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.472+0000 7fe0e0f51640 10 monclient: get_monmap_and_config 2024-06-11T19:13:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.472+0000 7fe0e0f51640 10 monclient: build_initial_monmap 2024-06-11T19:13:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.472+0000 7fe0e0f51640 10 monclient: monmap: 2024-06-11T19:13:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:36.473606+0000 2024-06-11T19:13:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:36.473606+0000 2024-06-11T19:13:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:36.473 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 1 Processor -- start 2024-06-11T19:13:36.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 1 -- start start 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 10 monclient: init 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 10 monclient: _add_conns ranks=[4,8,10,9,0,2,1,7,6,5,3] 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe0dc0a4920 0x7fe0dc0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 10 monclient(hunting): picked mon.noname-c con 0x7fe0dc0a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe0dc0a5240 0x7fe0dc0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 10 monclient(hunting): picked mon.noname-e con 0x7fe0dc0a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe0dc0a2530 0x7fe0dc0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 10 monclient(hunting): picked mon.noname-f con 0x7fe0dc0a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fe0dc0a7140 con 0x7fe0dc0a2530 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe0dc0a2b50 con 0x7fe0dc0a4920 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe0dc0a2e30 con 0x7fe0dc0a5240 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0e0f51640 10 monclient(hunting): authenticate will time out at 1026.280834s 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0da59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe0dc0a4920 0x7fe0dc0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0d9d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe0dc0a5240 0x7fe0dc0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0d9d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe0dc0a5240 0x7fe0dc0a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35368/0 (socket says 172.21.15.98:35368) 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0d9d9b640 1 -- 172.21.15.98:0/1117851484 learned_addr learned my addr 172.21.15.98:0/1117851484 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0da59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe0dc0a4920 0x7fe0dc0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52290/0 (socket says 172.21.15.98:52290) 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0d9d9b640 10 monclient(hunting): get_auth_request con 0x7fe0dc0a5240 auth_method 0 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0d9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.473+0000 7fe0d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0da59c640 10 monclient(hunting): get_auth_request con 0x7fe0dc0a4920 auth_method 0 2024-06-11T19:13:36.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0da59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0da59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0dad9d640 1 --2- 172.21.15.98:0/1117851484 >> v2:172.21.15.117:3300/0 conn(0x7fe0dc0a2530 0x7fe0dc0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0dad9d640 10 monclient(hunting): get_auth_request con 0x7fe0dc0a2530 auth_method 0 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0dad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_done global_id 64151 payload 274 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 1 -- 172.21.15.98:0/1117851484 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe0dc0a4920 msgr2=0x7fe0dc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 1 --2- 172.21.15.98:0/1117851484 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe0dc0a4920 0x7fe0dc0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 1 -- 172.21.15.98:0/1117851484 >> v2:172.21.15.117:3300/0 conn(0x7fe0dc0a2530 msgr2=0x7fe0dc0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 1 --2- 172.21.15.98:0/1117851484 >> v2:172.21.15.117:3300/0 conn(0x7fe0dc0a2530 0x7fe0dc0a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 1 monclient: found mon.noname-e 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0d9d9b640 1 -- 172.21.15.98:0/1117851484 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0dc0a3110 con 0x7fe0dc0a5240 2024-06-11T19:13:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.474+0000 7fe0da59c640 1 --2- 172.21.15.98:0/1117851484 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe0dc0a4920 0x7fe0dc0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d9d9b640 1 --2- 172.21.15.98:0/1117851484 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe0dc0a5240 0x7fe0dc0a59a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0009870 tx=0x7fe0d002b7c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=408cf0ccc7f7896c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 1 -- 172.21.15.98:0/1117851484 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe0d0038070 con 0x7fe0dc0a5240 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 10 monclient: dump: 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 10 monclient: _finish_auth 0 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 10 monclient: _check_auth_tickets 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 5 monclient: authenticate success, global_id 64151 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 1 -- 172.21.15.98:0/1117851484 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0d0031990 con 0x7fe0dc0a5240 2024-06-11T19:13:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 4 set_mon_vals no callback set 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 1 -- 172.21.15.98:0/1117851484 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe0d00323b0 con 0x7fe0dc0a5240 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0d959a640 10 monclient: dump: 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 10 monclient: shutdown 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 1 -- 172.21.15.98:0/1117851484 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe0dc0a5240 msgr2=0x7fe0dc0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 1 --2- 172.21.15.98:0/1117851484 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe0dc0a5240 0x7fe0dc0a59a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0009870 tx=0x7fe0d002b7c0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 1 -- 172.21.15.98:0/1117851484 shutdown_connections 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 1 --2- 172.21.15.98:0/1117851484 >> v2:172.21.15.117:3300/0 conn(0x7fe0dc0a2530 0x7fe0dc0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 1 --2- 172.21.15.98:0/1117851484 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe0dc0a5240 0x7fe0dc0a59a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 1 --2- 172.21.15.98:0/1117851484 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe0dc0a4920 0x7fe0dc0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 1 -- 172.21.15.98:0/1117851484 >> 172.21.15.98:0/1117851484 conn(0x7fe0dc0ab850 msgr2=0x7fe0dc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 1 -- 172.21.15.98:0/1117851484 shutdown_connections 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.475+0000 7fe0e0f51640 1 -- 172.21.15.98:0/1117851484 wait complete. 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.476+0000 7fe0e0f51640 10 monclient: build_initial_monmap 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.476+0000 7fe0e0f51640 10 monclient: monmap: 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.476+0000 7fe0e0f51640 1 librados: starting msgr at 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.476+0000 7fe0e0f51640 1 librados: starting objecter 2024-06-11T19:13:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.476+0000 7fe0e0f51640 1 Processor -- start 2024-06-11T19:13:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.476+0000 7fe0e0f51640 1 -- start start 2024-06-11T19:13:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.476+0000 7fe0e0f51640 1 librados: setting wanted keys 2024-06-11T19:13:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.476+0000 7fe0e0f51640 1 librados: calling monclient init 2024-06-11T19:13:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.476+0000 7fe0e0f51640 10 monclient: init 2024-06-11T19:13:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 10 monclient: _add_conns ranks=[6,5,2,7,4,8,1,0,3] 2024-06-11T19:13:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe0dc0a2530 0x7fe0dc1cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 10 monclient(hunting): picked mon.noname-d con 0x7fe0dc0a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe0dc0a4920 0x7fe0dc1ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 10 monclient(hunting): picked mon.noname-h con 0x7fe0dc0a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0dc0a5240 0x7fe0dc123030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 10 monclient(hunting): picked mon.noname-b con 0x7fe0dc0a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe0dc0a6da0 con 0x7fe0dc0a5240 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fe0dc138e50 con 0x7fe0dc0a4920 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe0dc0a6420 con 0x7fe0dc0a2530 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0e0f51640 10 monclient(hunting): authenticate will time out at 1026.284491s 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe0dc0a2530 0x7fe0dc1cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe0dc0a2530 0x7fe0dc1cdcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:53246/0 (socket says 172.21.15.98:53246) 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0dad9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0dc0a5240 0x7fe0dc123030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 1 -- 172.21.15.98:0/334022880 learned_addr learned my addr 172.21.15.98:0/334022880 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 10 monclient(hunting): get_auth_request con 0x7fe0dc0a2530 auth_method 0 2024-06-11T19:13:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0dad9d640 10 monclient(hunting): get_auth_request con 0x7fe0dc0a5240 auth_method 0 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0dad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0d9d9b640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe0dc0a4920 0x7fe0dc1ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0d9d9b640 10 monclient(hunting): get_auth_request con 0x7fe0dc0a4920 auth_method 0 2024-06-11T19:13:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0d9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 10 monclient(hunting): handle_auth_done global_id 54681 payload 824 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 1 -- 172.21.15.98:0/334022880 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe0dc0a4920 msgr2=0x7fe0dc1ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe0dc0a4920 0x7fe0dc1ce1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 1 -- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0dc0a5240 msgr2=0x7fe0dc123030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0dc0a5240 0x7fe0dc123030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 10 monclient: _finish_hunting 0 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 1 monclient: found mon.noname-d 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0da59c640 1 -- 172.21.15.98:0/334022880 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0dc126920 con 0x7fe0dc0a2530 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.477+0000 7fe0d9d9b640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe0dc0a4920 0x7fe0dc1ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0da59c640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe0dc0a2530 0x7fe0dc1cdcb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00ebe0 tx=0x7fe0cc00c520 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 10 client.?.objecter ms_handle_connect 0x7fe0dc0a2530 2024-06-11T19:13:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe0cc017070 con 0x7fe0dc0a2530 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 10 monclient: dump: 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 10 monclient: _finish_auth 0 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 10 monclient: _check_auth_tickets 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0e0f51640 5 monclient: authenticate success, global_id 54681 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0cc00cc30 con 0x7fe0dc0a2530 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0e0f51640 10 monclient: _renew_subs 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0e0f51640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0e0f51640 1 -- 172.21.15.98:0/334022880 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0dc126c00 con 0x7fe0dc0a2530 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe0cc00cde0 con 0x7fe0dc0a2530 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0e0f51640 10 client.54681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0d8d99640 4 set_mon_vals no callback set 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0caffd640 10 monclient: dump: 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0e0f51640 10 monclient: _renew_subs 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0e0f51640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0e0f51640 1 -- 172.21.15.98:0/334022880 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0dc0a8da0 con 0x7fe0dc0a2530 2024-06-11T19:13:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.478+0000 7fe0e0f51640 1 librados: init done 2024-06-11T19:13:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0e0f51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0e0f51640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123570 0x7fe0dc123950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0e0f51640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0e0f51640 1 -- 172.21.15.98:0/334022880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0dc069750 con 0x7fe0dc123570 2024-06-11T19:13:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0d9d9b640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123570 0x7fe0dc123950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0d9d9b640 10 monclient: get_auth_request con 0x7fe0dc123570 auth_method 0 2024-06-11T19:13:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0d9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fe0cc01a070 con 0x7fe0dc0a2530 2024-06-11T19:13:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_done global_id 64206 payload 824 2024-06-11T19:13:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0d9d9b640 1 -- 172.21.15.98:0/334022880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe0dc127000 con 0x7fe0dc123570 2024-06-11T19:13:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe0bc03fdb0 0x7fe0bc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(181..181 src has 1..181) ==== 6621+0+0 (secure 0 0 0) 0x7fe0cc00f040 con 0x7fe0dc0a2530 2024-06-11T19:13:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 10 client.54681.objecter ms_dispatch 0x7fe0dc000b90 osd_map(181..181 src has 1..181) 2024-06-11T19:13:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 3 client.54681.objecter handle_osd_map got epochs [181,181] > 0 2024-06-11T19:13:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 3 client.54681.objecter handle_osd_map decoding full epoch 181 2024-06-11T19:13:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0d9d9b640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123570 0x7fe0dc123950 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0abf80 tx=0x7fe0d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 20 client.54681.objecter dump_active .. 0 homeless 2024-06-11T19:13:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 10 client.54681.objecter ms_handle_connect 0x7fe0dc123570 2024-06-11T19:13:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 10 client.54681.objecter resend_mon_ops 2024-06-11T19:13:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe0d0043070 con 0x7fe0dc123570 2024-06-11T19:13:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0caffd640 10 monclient: dump: 2024-06-11T19:13:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0dad9d640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe0bc03fdb0 0x7fe0bc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.479+0000 7fe0dad9d640 10 monclient: get_auth_request con 0x7fe0bc03fdb0 auth_method 0 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.480+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe0d0003b20 con 0x7fe0dc123570 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.480+0000 7fe0caffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.480+0000 7fe0caffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.480+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123570 msgr2=0x7fe0dc123950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.480+0000 7fe0caffd640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123570 0x7fe0dc123950 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0abf80 tx=0x7fe0d00046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.480+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123570 msgr2=0x7fe0dc123950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.480+0000 7fe0caffd640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123570 0x7fe0dc123950 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0abf80 tx=0x7fe0d00046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.480+0000 7fe0caffd640 10 client.54681.objecter ms_handle_connect 0x7fe0bc03fdb0 2024-06-11T19:13:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.480+0000 7fe0dad9d640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe0bc03fdb0 0x7fe0bc042270 secure :-1 s=READY pgs=4077 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc124750 tx=0x7fe0d4009820 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0e0f51640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0e0f51640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123e90 0x7fe0dc09e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0e0f51640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0e0f51640 1 -- 172.21.15.98:0/334022880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe0dc050830 con 0x7fe0dc123e90 2024-06-11T19:13:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0d9d9b640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123e90 0x7fe0dc09e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0d9d9b640 10 monclient: get_auth_request con 0x7fe0dc123e90 auth_method 0 2024-06-11T19:13:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0d9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.486+0000 7fe0d9d9b640 10 monclient(hunting): handle_auth_done global_id 64215 payload 824 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0d9d9b640 1 -- 172.21.15.98:0/334022880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe0dc0a6720 con 0x7fe0dc123e90 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0d9d9b640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123e90 0x7fe0dc09e9f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0000c00 tx=0x7fe0d0008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 10 client.54681.objecter ms_handle_connect 0x7fe0dc123e90 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 10 client.54681.objecter resend_mon_ops 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe0d0038040 con 0x7fe0dc123e90 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 10 monclient: dump: 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9271 (secure 0 0 0) 0x7fe0d003e440 con 0x7fe0dc123e90 2024-06-11T19:13:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:36.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:36.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123e90 msgr2=0x7fe0dc09e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123e90 0x7fe0dc09e9f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0000c00 tx=0x7fe0d0008030 comp rx=0 tx=0).stop 2024-06-11T19:13:36.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 1 -- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123e90 msgr2=0x7fe0dc09e9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:36.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0caffd640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe0dc123e90 0x7fe0dc09e9f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.511 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 8314, 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:36.529 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:36.530 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.531 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.532 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.533 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:36.534 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:36.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 10 librados: watch_flush enter 2024-06-11T19:13:36.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 10 librados: watch_flush exit 2024-06-11T19:13:36.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 20 client.54681.objecter shutdown clearing up homeless session... 2024-06-11T19:13:36.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 10 client.54681.objecter successfully canceled tick 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 1 -- 172.21.15.98:0/334022880 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe0bc03fdb0 msgr2=0x7fe0bc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe0bc03fdb0 0x7fe0bc042270 secure :-1 s=READY pgs=4077 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc124750 tx=0x7fe0d4009820 comp rx=0 tx=0).stop 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 10 monclient: shutdown 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 1 -- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe0dc0a2530 msgr2=0x7fe0dc1cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 1 --2- 172.21.15.98:0/334022880 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe0dc0a2530 0x7fe0dc1cdcb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00ebe0 tx=0x7fe0cc00c520 comp rx=0 tx=0).stop 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0dad9d640 1 -- 172.21.15.98:0/334022880 reap_dead start 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0dad9d640 1 -- 172.21.15.98:0/334022880 reap_dead start 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 1 -- 172.21.15.98:0/334022880 shutdown_connections 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 1 -- 172.21.15.98:0/334022880 >> 172.21.15.98:0/334022880 conn(0x7fe0dc0ab850 msgr2=0x7fe0dc0a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.487+0000 7fe0e0f51640 1 -- 172.21.15.98:0/334022880 shutdown_connections 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.488+0000 7fe0e0f51640 1 -- 172.21.15.98:0/334022880 wait complete. 2024-06-11T19:13:36.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.488+0000 7fe0e0f51640 1 librados: shutdown 2024-06-11T19:13:36.537 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:36.537 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:13:36.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:13:36.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.636+0000 7fd33a9bc640 10 monclient: get_monmap_and_config 2024-06-11T19:13:36.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.636+0000 7fd33a9bc640 10 monclient: build_initial_monmap 2024-06-11T19:13:36.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.636+0000 7fd33a9bc640 10 monclient: monmap: 2024-06-11T19:13:36.635 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:36.635 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:36.637092+0000 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:36.637092+0000 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:36.636 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:36.637 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:36.637 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:36.637 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.636+0000 7fd33a9bc640 1 Processor -- start 2024-06-11T19:13:36.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.636+0000 7fd33a9bc640 1 -- start start 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.636+0000 7fd33a9bc640 10 monclient: init 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 10 monclient: _add_conns ranks=[2,1,10,5,4,9,6,7,0,8,3] 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a4920 0x7fd3340a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 10 monclient(hunting): picked mon.noname-b con 0x7fd3340a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd3340a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd3340a2530 0x7fd3340a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 10 monclient(hunting): picked mon.noname-f con 0x7fd3340a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3340a7140 con 0x7fd3340a5240 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fd3340a2b50 con 0x7fd3340a2530 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd3340a2e30 con 0x7fd3340a4920 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd33a9bc640 10 monclient(hunting): authenticate will time out at 1026.444436s 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a4920 0x7fd3340a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a4920 0x7fd3340a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55496/0 (socket says 172.21.15.98:55496) 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 1 -- 172.21.15.98:0/4242473133 learned_addr learned my addr 172.21.15.98:0/4242473133 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 10 monclient(hunting): get_auth_request con 0x7fd3340a4920 auth_method 0 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd338f59640 1 --2- 172.21.15.98:0/4242473133 >> v2:172.21.15.117:3300/0 conn(0x7fd3340a2530 0x7fd3340a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd338f59640 10 monclient(hunting): get_auth_request con 0x7fd3340a2530 auth_method 0 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd338f59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd338f59640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd338f59640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 10 monclient(hunting): handle_auth_done global_id 22088 payload 274 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 1 -- 172.21.15.98:0/4242473133 >> v2:172.21.15.117:3300/0 conn(0x7fd3340a2530 msgr2=0x7fd3340a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 1 --2- 172.21.15.98:0/4242473133 >> v2:172.21.15.117:3300/0 conn(0x7fd3340a2530 0x7fd3340a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 1 -- 172.21.15.98:0/4242473133 >> v1:172.21.15.117:6789/0 conn(0x7fd3340a5240 legacy=0x7fd3340a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 10 monclient: _finish_hunting 0 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 1 monclient: found mon.noname-b 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:13:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.637+0000 7fd333fff640 1 -- 172.21.15.98:0/4242473133 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3340a3110 con 0x7fd3340a4920 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd333fff640 1 --2- 172.21.15.98:0/4242473133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a4920 0x7fd3340a4d00 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fd324009870 tx=0x7fd32402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f31b24eb29dc28d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 1 -- 172.21.15.98:0/4242473133 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd324031a30 con 0x7fd3340a4920 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 10 monclient: dump: 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 10 monclient: _finish_auth 0 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd33a9bc640 5 monclient: authenticate success, global_id 22088 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 1 -- 172.21.15.98:0/4242473133 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd324037070 con 0x7fd3340a4920 2024-06-11T19:13:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd33a9bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 1 -- 172.21.15.98:0/4242473133 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd324031c30 con 0x7fd3340a4920 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd33a9bc640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd33a9bc640 4 set_mon_vals no callback set 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.638+0000 7fd332ffd640 10 monclient: dump: 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.652 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.639+0000 7fd33a9bc640 10 monclient: shutdown 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.639+0000 7fd33a9bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.639+0000 7fd33a9bc640 1 -- 172.21.15.98:0/4242473133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a4920 msgr2=0x7fd3340a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.639+0000 7fd33a9bc640 1 --2- 172.21.15.98:0/4242473133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a4920 0x7fd3340a4d00 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fd324009870 tx=0x7fd32402b860 comp rx=0 tx=0).stop 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.639+0000 7fd33a9bc640 1 -- 172.21.15.98:0/4242473133 shutdown_connections 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.639+0000 7fd33a9bc640 1 --2- 172.21.15.98:0/4242473133 >> v2:172.21.15.117:3300/0 conn(0x7fd3340a2530 0x7fd3340a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.639+0000 7fd33a9bc640 1 --2- 172.21.15.98:0/4242473133 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a4920 0x7fd3340a4d00 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.639+0000 7fd33a9bc640 1 -- 172.21.15.98:0/4242473133 >> 172.21.15.98:0/4242473133 conn(0x7fd3340ab850 msgr2=0x7fd3340a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.639+0000 7fd33a9bc640 1 -- 172.21.15.98:0/4242473133 shutdown_connections 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.639+0000 7fd33a9bc640 1 -- 172.21.15.98:0/4242473133 wait complete. 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.639+0000 7fd33a9bc640 10 monclient: build_initial_monmap 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.639+0000 7fd33a9bc640 10 monclient: monmap: 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:36.654 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:36.655 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 1 librados: starting msgr at 2024-06-11T19:13:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 1 librados: starting objecter 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 1 Processor -- start 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 1 -- start start 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 1 librados: setting wanted keys 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 1 librados: calling monclient init 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 10 monclient: init 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 10 monclient: _add_conns ranks=[2,4,8,6,5,0,3,1,7] 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a2530 0x7fd3341cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 10 monclient(hunting): picked mon.noname-b con 0x7fd3340a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd3340a4920 0x7fd3341ce160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 10 monclient(hunting): picked mon.noname-c con 0x7fd3340a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3340a5240 0x7fd334122f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 10 monclient(hunting): picked mon.noname-e con 0x7fd3340a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd3340a6da0 con 0x7fd3340a2530 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd334138e50 con 0x7fd3340a4920 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd3340a65f0 con 0x7fd3340a5240 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.640+0000 7fd33a9bc640 10 monclient(hunting): authenticate will time out at 1026.447966s 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a2530 0x7fd3341cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a2530 0x7fd3341cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55502/0 (socket says 172.21.15.98:55502) 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 1 -- 172.21.15.98:0/3166576093 learned_addr learned my addr 172.21.15.98:0/3166576093 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 10 monclient(hunting): get_auth_request con 0x7fd3340a2530 auth_method 0 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd338f59640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3340a5240 0x7fd334122f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd3337fe640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd3340a4920 0x7fd3341ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd3337fe640 10 monclient(hunting): get_auth_request con 0x7fd3340a4920 auth_method 0 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd3337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd3337fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd3337fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd338f59640 10 monclient(hunting): get_auth_request con 0x7fd3340a5240 auth_method 0 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd338f59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd338f59640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd338f59640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd3337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd3337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd3337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd338f59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd338f59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd338f59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 10 monclient(hunting): handle_auth_done global_id 22097 payload 824 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 1 -- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3340a5240 msgr2=0x7fd334122f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3340a5240 0x7fd334122f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 1 -- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd3340a4920 msgr2=0x7fd3341ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd3340a4920 0x7fd3341ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 10 monclient: _finish_hunting 0 2024-06-11T19:13:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 1 monclient: found mon.noname-b 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 1 -- 172.21.15.98:0/3166576093 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd334125680 con 0x7fd3340a2530 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd333fff640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a2530 0x7fd3341cdc20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fd3240097c0 tx=0x7fd3240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 10 client.?.objecter ms_handle_connect 0x7fd3340a2530 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd324041070 con 0x7fd3340a2530 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 10 monclient: dump: 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 10 monclient: _finish_auth 0 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd33a9bc640 5 monclient: authenticate success, global_id 22097 2024-06-11T19:13:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd324003c50 con 0x7fd3340a2530 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd33a9bc640 10 monclient: _renew_subs 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd33a9bc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd33a9bc640 1 -- 172.21.15.98:0/3166576093 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd334126e10 con 0x7fd3340a2530 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3240044e0 con 0x7fd3340a2530 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd33a9bc640 10 client.22097.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd330ff9640 10 monclient: dump: 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd3327fc640 4 set_mon_vals no callback set 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd33a9bc640 10 monclient: _renew_subs 2024-06-11T19:13:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd33a9bc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd33a9bc640 1 -- 172.21.15.98:0/3166576093 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd334126850 con 0x7fd3340a2530 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.641+0000 7fd33a9bc640 1 librados: init done 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd33a9bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd33a9bc640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123450 0x7fd334123830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd33a9bc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd33a9bc640 1 -- 172.21.15.98:0/3166576093 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd334127840 con 0x7fd334123450 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd32404c060 con 0x7fd3340a2530 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd3337fe640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123450 0x7fd334123830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd3337fe640 10 monclient: get_auth_request con 0x7fd334123450 auth_method 0 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd3337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd3337fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd3337fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd3337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd3337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd3337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.642+0000 7fd330ff9640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd32003fdb0 0x7fd320042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd3337fe640 10 monclient(hunting): handle_auth_done global_id 22106 payload 824 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(181..181 src has 1..181) ==== 6621+0+0 (secure 0 0 0) 0x7fd324072f60 con 0x7fd3340a2530 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 10 client.22097.objecter ms_dispatch 0x7fd334000b90 osd_map(181..181 src has 1..181) 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 3 client.22097.objecter handle_osd_map got epochs [181,181] > 0 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 3 client.22097.objecter handle_osd_map decoding full epoch 181 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd3337fe640 1 -- 172.21.15.98:0/3166576093 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3340a4d00 con 0x7fd334123450 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 20 client.22097.objecter dump_active .. 0 homeless 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd338f59640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd32003fdb0 0x7fd320042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd338f59640 10 monclient: get_auth_request con 0x7fd32003fdb0 auth_method 0 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 10 client.22097.objecter ms_handle_connect 0x7fd334123450 2024-06-11T19:13:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 10 client.22097.objecter resend_mon_ops 2024-06-11T19:13:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd3337fe640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123450 0x7fd334123830 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fd334073fc0 tx=0x7fd32800f9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3280052e0 con 0x7fd334123450 2024-06-11T19:13:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 10 monclient: dump: 2024-06-11T19:13:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 10 client.22097.objecter ms_handle_connect 0x7fd32003fdb0 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd338f59640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd32003fdb0 0x7fd320042270 secure :-1 s=READY pgs=4078 cs=0 l=1 rev1=1 crypto rx=0x7fd3341245f0 tx=0x7fd32c009820 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd328018070 con 0x7fd334123450 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:36.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:36.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123450 msgr2=0x7fd334123830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123450 0x7fd334123830 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fd334073fc0 tx=0x7fd32800f9d0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123450 msgr2=0x7fd334123830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:36.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.643+0000 7fd330ff9640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123450 0x7fd334123830 secure :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fd334073fc0 tx=0x7fd32800f9d0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.649+0000 7fd33a9bc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.649+0000 7fd33a9bc640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123d70 0x7fd33409e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.649+0000 7fd33a9bc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.649+0000 7fd33a9bc640 1 -- 172.21.15.98:0/3166576093 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd334050830 con 0x7fd334123d70 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd3337fe640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123d70 0x7fd33409e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd3337fe640 10 monclient: get_auth_request con 0x7fd334123d70 auth_method 0 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd3337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd3337fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd3337fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd3337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd3337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd3337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd3337fe640 10 monclient(hunting): handle_auth_done global_id 22115 payload 824 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd3337fe640 1 -- 172.21.15.98:0/3166576093 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd3341253d0 con 0x7fd334123d70 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 10 client.22097.objecter ms_handle_connect 0x7fd334123d70 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 10 client.22097.objecter resend_mon_ops 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd3337fe640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123d70 0x7fd33409e9f0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fd334073fc0 tx=0x7fd328000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd328003d40 con 0x7fd334123d70 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 10 monclient: dump: 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.671 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fd3280038d0 con 0x7fd334123d70 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123d70 msgr2=0x7fd33409e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123d70 0x7fd33409e9f0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fd334073fc0 tx=0x7fd328000ed0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 1 -- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123d70 msgr2=0x7fd33409e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd330ff9640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd334123d70 0x7fd33409e9f0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.673 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 266883, 2024-06-11T19:13:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:36.673 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:36.674 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:36.675 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:36.676 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:36.677 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:36.678 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:36.679 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:36.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.685 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:36.687 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:13:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:36.696 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:36.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd33a9bc640 10 librados: watch_flush enter 2024-06-11T19:13:36.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.650+0000 7fd33a9bc640 10 librados: watch_flush exit 2024-06-11T19:13:36.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 20 client.22097.objecter shutdown clearing up homeless session... 2024-06-11T19:13:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 10 client.22097.objecter successfully canceled tick 2024-06-11T19:13:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 1 -- 172.21.15.98:0/3166576093 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd32003fdb0 msgr2=0x7fd320042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd32003fdb0 0x7fd320042270 secure :-1 s=READY pgs=4078 cs=0 l=1 rev1=1 crypto rx=0x7fd3341245f0 tx=0x7fd32c009820 comp rx=0 tx=0).stop 2024-06-11T19:13:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 10 monclient: shutdown 2024-06-11T19:13:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 1 -- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a2530 msgr2=0x7fd3341cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd338f59640 1 -- 172.21.15.98:0/3166576093 reap_dead start 2024-06-11T19:13:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a2530 0x7fd3341cdc20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fd3240097c0 tx=0x7fd3240046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 1 -- 172.21.15.98:0/3166576093 shutdown_connections 2024-06-11T19:13:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 1 --2- 172.21.15.98:0/3166576093 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3340a2530 0x7fd3341cdc20 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 1 -- 172.21.15.98:0/3166576093 >> 172.21.15.98:0/3166576093 conn(0x7fd3340ab850 msgr2=0x7fd3341be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:36.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 1 -- 172.21.15.98:0/3166576093 shutdown_connections 2024-06-11T19:13:36.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 1 -- 172.21.15.98:0/3166576093 wait complete. 2024-06-11T19:13:36.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.651+0000 7fd33a9bc640 1 librados: shutdown 2024-06-11T19:13:36.699 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:36.699 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:13:36.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:13:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.799+0000 7f30010f1640 10 monclient: get_monmap_and_config 2024-06-11T19:13:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.799+0000 7f30010f1640 10 monclient: build_initial_monmap 2024-06-11T19:13:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.799+0000 7f30010f1640 10 monclient: monmap: 2024-06-11T19:13:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:36.800699+0000 2024-06-11T19:13:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:36.800699+0000 2024-06-11T19:13:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:36.800 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 1 Processor -- start 2024-06-11T19:13:36.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 1 -- start start 2024-06-11T19:13:36.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 10 monclient: init 2024-06-11T19:13:36.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:36.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 10 monclient: _add_conns ranks=[6,3,10,0,4,5,8,7,1,2,9] 2024-06-11T19:13:36.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ffc0a4920 0x7f2ffc0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 10 monclient(hunting): picked mon.noname-d con 0x7f2ffc0a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ffc0a5240 0x7f2ffc0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 10 monclient(hunting): picked mon.noname-g con 0x7f2ffc0a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2ffc0a2530 0x7f2ffc0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 10 monclient(hunting): picked mon.noname-f con 0x7f2ffc0a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.800+0000 7f30010f1640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f2ffc0a7140 con 0x7f2ffc0a2530 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f30010f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f30010f1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f2ffc0a2b50 con 0x7f2ffc0a5240 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f30010f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f30010f1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f2ffc0a2e30 con 0x7f2ffc0a4920 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f30010f1640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f30010f1640 10 monclient(hunting): authenticate will time out at 1026.608022s 2024-06-11T19:13:36.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ffc0a4920 0x7f2ffc0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ffc0a4920 0x7f2ffc0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:53248/0 (socket says 172.21.15.98:53248) 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 1 -- 172.21.15.98:0/2784881624 learned_addr learned my addr 172.21.15.98:0/2784881624 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 10 monclient(hunting): get_auth_request con 0x7f2ffc0a4920 auth_method 0 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffb59e640 1 --2- 172.21.15.98:0/2784881624 >> v2:172.21.15.117:3300/0 conn(0x7f2ffc0a2530 0x7f2ffc0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ff3fff640 1 --2- 172.21.15.98:0/2784881624 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ffc0a5240 0x7f2ffc0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffb59e640 10 monclient(hunting): get_auth_request con 0x7f2ffc0a2530 auth_method 0 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffb59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffb59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ff3fff640 10 monclient(hunting): get_auth_request con 0x7f2ffc0a5240 auth_method 0 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ff3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ff3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_done global_id 54690 payload 274 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 1 -- 172.21.15.98:0/2784881624 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ffc0a5240 msgr2=0x7f2ffc0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 1 --2- 172.21.15.98:0/2784881624 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ffc0a5240 0x7f2ffc0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 1 -- 172.21.15.98:0/2784881624 >> v2:172.21.15.117:3300/0 conn(0x7f2ffc0a2530 msgr2=0x7f2ffc0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 1 --2- 172.21.15.98:0/2784881624 >> v2:172.21.15.117:3300/0 conn(0x7f2ffc0a2530 0x7f2ffc0a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 1 monclient: found mon.noname-d 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:13:36.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffad9d640 1 -- 172.21.15.98:0/2784881624 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ffc0a3110 con 0x7f2ffc0a4920 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.801+0000 7f2ffb59e640 1 --2- 172.21.15.98:0/2784881624 >> v2:172.21.15.117:3300/0 conn(0x7f2ffc0a2530 0x7f2ffc0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffad9d640 1 --2- 172.21.15.98:0/2784881624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ffc0a4920 0x7f2ffc0a4d00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009870 tx=0x7f2fec02b7c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=20ddcd3b0025365 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 1 -- 172.21.15.98:0/2784881624 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2fec038070 con 0x7f2ffc0a4920 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 10 monclient: dump: 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.814 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 10 monclient: _finish_auth 0 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 10 monclient: _check_auth_tickets 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 1 -- 172.21.15.98:0/2784881624 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fec031990 con 0x7f2ffc0a4920 2024-06-11T19:13:36.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 5 monclient: authenticate success, global_id 54690 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 1 -- 172.21.15.98:0/2784881624 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2fec0323b0 con 0x7f2ffc0a4920 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 4 set_mon_vals no callback set 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f2ffa59c640 10 monclient: dump: 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 10 monclient: shutdown 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 1 -- 172.21.15.98:0/2784881624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ffc0a4920 msgr2=0x7f2ffc0a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 1 --2- 172.21.15.98:0/2784881624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ffc0a4920 0x7f2ffc0a4d00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009870 tx=0x7f2fec02b7c0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 1 -- 172.21.15.98:0/2784881624 shutdown_connections 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 1 --2- 172.21.15.98:0/2784881624 >> v2:172.21.15.117:3300/0 conn(0x7f2ffc0a2530 0x7f2ffc0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 1 --2- 172.21.15.98:0/2784881624 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ffc0a5240 0x7f2ffc0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 1 --2- 172.21.15.98:0/2784881624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ffc0a4920 0x7f2ffc0a4d00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 1 -- 172.21.15.98:0/2784881624 >> 172.21.15.98:0/2784881624 conn(0x7f2ffc0ab850 msgr2=0x7f2ffc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 1 -- 172.21.15.98:0/2784881624 shutdown_connections 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.802+0000 7f30010f1640 1 -- 172.21.15.98:0/2784881624 wait complete. 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.803+0000 7f30010f1640 10 monclient: build_initial_monmap 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.803+0000 7f30010f1640 10 monclient: monmap: 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:36.818 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.803+0000 7f30010f1640 1 librados: starting msgr at 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.803+0000 7f30010f1640 1 librados: starting objecter 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.803+0000 7f30010f1640 1 Processor -- start 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 1 -- start start 2024-06-11T19:13:36.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 1 librados: setting wanted keys 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 1 librados: calling monclient init 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 10 monclient: init 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 10 monclient: _add_conns ranks=[1,2,7,4,0,8,6,5,3] 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ffc0a2530 0x7f2ffc1cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 10 monclient(hunting): picked mon.noname-f con 0x7f2ffc0a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ffc0a4920 0x7f2ffc1cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 10 monclient(hunting): picked mon.noname-b con 0x7f2ffc0a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f2ffc0a5240 0x7f2ffc125110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 10 monclient(hunting): picked mon.noname-i con 0x7f2ffc0a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc0a6da0 con 0x7f2ffc0a2530 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2ffc138e50 con 0x7f2ffc0a4920 2024-06-11T19:13:36.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f2ffc0a6420 con 0x7f2ffc0a5240 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f30010f1640 10 monclient(hunting): authenticate will time out at 1026.611509s 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ffc0a4920 0x7f2ffc1cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ffc0a4920 0x7f2ffc1cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55524/0 (socket says 172.21.15.98:55524) 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 1 -- 172.21.15.98:0/427505742 learned_addr learned my addr 172.21.15.98:0/427505742 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 10 monclient(hunting): get_auth_request con 0x7f2ffc0a4920 auth_method 0 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ffb59e640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f2ffc0a5240 0x7f2ffc125110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ffad9d640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ffc0a2530 0x7f2ffc1cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 10 monclient(hunting): handle_auth_done global_id 22124 payload 824 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 1 -- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f2ffc0a5240 msgr2=0x7f2ffc125110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f2ffc0a5240 0x7f2ffc125110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 1 -- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ffc0a2530 msgr2=0x7f2ffc1cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ffc0a2530 0x7f2ffc1cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 10 monclient: _finish_hunting 0 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 1 monclient: found mon.noname-b 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ff3fff640 1 -- 172.21.15.98:0/427505742 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ffc128b10 con 0x7f2ffc0a4920 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ffb59e640 10 monclient: get_auth_request con 0x7f2ffc0a5240 auth_method 0 2024-06-11T19:13:36.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ffad9d640 10 monclient: get_auth_request con 0x7f2ffc0a2530 auth_method 0 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ffb59e640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f2ffc0a5240 0x7f2ffc125110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.804+0000 7f2ffad9d640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ffc0a2530 0x7f2ffc1cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff3fff640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ffc0a4920 0x7f2ffc1cdfe0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400d5d0 tx=0x7f2fe400da90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 10 client.?.objecter ms_handle_connect 0x7f2ffc0a4920 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2fe4012a60 con 0x7f2ffc0a4920 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 10 monclient: dump: 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.823 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 10 monclient: _finish_auth 0 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 10 monclient: _check_auth_tickets 2024-06-11T19:13:36.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fe4019070 con 0x7f2ffc0a4920 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f30010f1640 5 monclient: authenticate success, global_id 22124 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2fe4012c40 con 0x7f2ffc0a4920 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f30010f1640 10 monclient: _renew_subs 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f30010f1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f30010f1640 1 -- 172.21.15.98:0/427505742 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ffc127940 con 0x7f2ffc0a4920 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff9d9b640 4 set_mon_vals no callback set 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f2ff37fe640 10 monclient: dump: 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.825 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f30010f1640 10 client.22124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f30010f1640 10 monclient: _renew_subs 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f30010f1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f30010f1640 1 -- 172.21.15.98:0/427505742 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ffc138570 con 0x7f2ffc0a4920 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.805+0000 7f30010f1640 1 librados: init done 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f2fe40125f0 con 0x7f2ffc0a4920 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f30010f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f30010f1640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc125e10 0x7f2ffc02cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f30010f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f30010f1640 1 -- 172.21.15.98:0/427505742 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2ffc050830 con 0x7f2ffc125e10 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffad9d640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc125e10 0x7f2ffc02cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffad9d640 10 monclient: get_auth_request con 0x7f2ffc125e10 auth_method 0 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2fdc03fdb0 0x7f2fdc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(181..181 src has 1..181) ==== 6621+0+0 (secure 0 0 0) 0x7f2fe400f040 con 0x7f2ffc0a4920 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 10 client.22124.objecter ms_dispatch 0x7f2ffc000b90 osd_map(181..181 src has 1..181) 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 3 client.22124.objecter handle_osd_map got epochs [181,181] > 0 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 3 client.22124.objecter handle_osd_map decoding full epoch 181 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 20 client.22124.objecter dump_active .. 0 homeless 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_done global_id 23449 payload 824 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffad9d640 1 -- 172.21.15.98:0/427505742 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ffc0a90f0 con 0x7f2ffc125e10 2024-06-11T19:13:36.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffb59e640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2fdc03fdb0 0x7f2fdc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffb59e640 10 monclient: get_auth_request con 0x7f2fdc03fdb0 auth_method 0 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ffad9d640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc125e10 0x7f2ffc02cb40 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0abf80 tx=0x7f2fec0046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 10 client.22124.objecter ms_handle_connect 0x7f2ffc125e10 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 10 client.22124.objecter resend_mon_ops 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2fec042070 con 0x7f2ffc125e10 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.806+0000 7f2ff37fe640 10 monclient: dump: 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.829 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.807+0000 7f2ff37fe640 10 client.22124.objecter ms_handle_connect 0x7f2fdc03fdb0 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.807+0000 7f2ffb59e640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2fdc03fdb0 0x7f2fdc042270 secure :-1 s=READY pgs=4079 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40094c0 tx=0x7f2ff400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.807+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2fec003b20 con 0x7f2ffc125e10 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.807+0000 7f2ff37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.807+0000 7f2ff37fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:36.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.807+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc125e10 msgr2=0x7f2ffc02cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.807+0000 7f2ff37fe640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc125e10 0x7f2ffc02cb40 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0abf80 tx=0x7f2fec0046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.807+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc125e10 msgr2=0x7f2ffc02cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:36.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.807+0000 7f2ff37fe640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc125e10 0x7f2ffc02cb40 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f30010f1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f30010f1640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc1261f0 0x7f2ffc02d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f30010f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f30010f1640 1 -- 172.21.15.98:0/427505742 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2ffc0a8e30 con 0x7f2ffc1261f0 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f2ffad9d640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc1261f0 0x7f2ffc02d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f2ffad9d640 10 monclient: get_auth_request con 0x7f2ffc1261f0 auth_method 0 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f2ffad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f2ffad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f2ffad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_done global_id 23458 payload 824 2024-06-11T19:13:36.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.813+0000 7f2ffad9d640 1 -- 172.21.15.98:0/427505742 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ffc138830 con 0x7f2ffc1261f0 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 10 client.22124.objecter ms_handle_connect 0x7f2ffc1261f0 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ffad9d640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc1261f0 0x7f2ffc02d080 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0097c0 tx=0x7f2fec009240 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 10 client.22124.objecter resend_mon_ops 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2fec038040 con 0x7f2ffc1261f0 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 10 monclient: dump: 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.834 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f2fec03d670 con 0x7f2ffc1261f0 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc1261f0 msgr2=0x7f2ffc02d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc1261f0 0x7f2ffc02d080 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0097c0 tx=0x7f2fec009240 comp rx=0 tx=0).stop 2024-06-11T19:13:36.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 1 -- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc1261f0 msgr2=0x7f2ffc02d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:36.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ff37fe640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ffc1261f0 0x7f2ffc02d080 secure :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0097c0 tx=0x7f2fec009240 comp rx=0 tx=0).stop 2024-06-11T19:13:36.836 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 267032, 2024-06-11T19:13:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:36.836 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:36.837 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:36.838 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:36.839 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:36.840 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:36.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:36.843 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:36.847 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:36.849 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.850 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.851 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:36.853 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.854 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:36.855 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:36.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:36.857 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:36.858 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:36.858 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 10 librados: watch_flush enter 2024-06-11T19:13:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 10 librados: watch_flush exit 2024-06-11T19:13:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 20 client.22124.objecter shutdown clearing up homeless session... 2024-06-11T19:13:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 10 client.22124.objecter successfully canceled tick 2024-06-11T19:13:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 1 -- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2fdc03fdb0 msgr2=0x7f2fdc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2fdc03fdb0 0x7f2fdc042270 secure :-1 s=READY pgs=4079 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40094c0 tx=0x7f2ff400b040 comp rx=0 tx=0).stop 2024-06-11T19:13:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 10 monclient: shutdown 2024-06-11T19:13:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 1 -- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ffc0a4920 msgr2=0x7f2ffc1cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 1 --2- 172.21.15.98:0/427505742 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ffc0a4920 0x7f2ffc1cdfe0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400d5d0 tx=0x7f2fe400da90 comp rx=0 tx=0).stop 2024-06-11T19:13:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ffb59e640 1 -- 172.21.15.98:0/427505742 reap_dead start 2024-06-11T19:13:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f2ffb59e640 1 -- 172.21.15.98:0/427505742 reap_dead start 2024-06-11T19:13:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 1 -- 172.21.15.98:0/427505742 shutdown_connections 2024-06-11T19:13:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 1 -- 172.21.15.98:0/427505742 >> 172.21.15.98:0/427505742 conn(0x7f2ffc0ab850 msgr2=0x7f2ffc09c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 1 -- 172.21.15.98:0/427505742 shutdown_connections 2024-06-11T19:13:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 1 -- 172.21.15.98:0/427505742 wait complete. 2024-06-11T19:13:36.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.814+0000 7f30010f1640 1 librados: shutdown 2024-06-11T19:13:36.862 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:36.862 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:13:36.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:13:36.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.962+0000 7ff66a166640 10 monclient: get_monmap_and_config 2024-06-11T19:13:36.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.962+0000 7ff66a166640 10 monclient: build_initial_monmap 2024-06-11T19:13:36.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.962+0000 7ff66a166640 10 monclient: monmap: 2024-06-11T19:13:36.962 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:36.963956+0000 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:36.963956+0000 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:36.963 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:36.964 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:36.964 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:36.964 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:36.964 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.963+0000 7ff66a166640 1 Processor -- start 2024-06-11T19:13:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.963+0000 7ff66a166640 1 -- start start 2024-06-11T19:13:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.963+0000 7ff66a166640 10 monclient: init 2024-06-11T19:13:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 10 monclient: _add_conns ranks=[7,2,4,5,3,1,8,9,0,6,10] 2024-06-11T19:13:36.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6640a4920 0x7ff6640a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 10 monclient(hunting): picked mon.noname-i con 0x7ff6640a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a5240 0x7ff6640a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 10 monclient(hunting): picked mon.noname-b con 0x7ff6640a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff6640a2530 0x7ff6640a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 10 monclient(hunting): picked mon.noname-c con 0x7ff6640a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff6640a7140 con 0x7ff6640a5240 2024-06-11T19:13:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff6640a2b50 con 0x7ff6640a2530 2024-06-11T19:13:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff6640a2e30 con 0x7ff6640a4920 2024-06-11T19:13:36.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff66a166640 10 monclient(hunting): authenticate will time out at 1026.771435s 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff663fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff6640a2530 0x7ff6640a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a5240 0x7ff6640a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff663fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff6640a2530 0x7ff6640a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52338/0 (socket says 172.21.15.98:52338) 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a5240 0x7ff6640a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55528/0 (socket says 172.21.15.98:55528) 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff663fff640 1 -- 172.21.15.98:0/503170062 learned_addr learned my addr 172.21.15.98:0/503170062 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff663fff640 10 monclient(hunting): get_auth_request con 0x7ff6640a2530 auth_method 0 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff663fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff663fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff663fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 10 monclient(hunting): get_auth_request con 0x7ff6640a5240 auth_method 0 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff663fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff663fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff663fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff6637fe640 1 --2- 172.21.15.98:0/503170062 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6640a4920 0x7ff6640a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff6637fe640 10 monclient(hunting): get_auth_request con 0x7ff6640a4920 auth_method 0 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff6637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff6637fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff6637fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 10 monclient(hunting): handle_auth_done global_id 22133 payload 274 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 1 -- 172.21.15.98:0/503170062 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6640a4920 msgr2=0x7ff6640a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:36.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 1 --2- 172.21.15.98:0/503170062 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6640a4920 0x7ff6640a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 1 -- 172.21.15.98:0/503170062 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff6640a2530 msgr2=0x7ff6640a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 1 --2- 172.21.15.98:0/503170062 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff6640a2530 0x7ff6640a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 1 monclient: found mon.noname-b 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff65bfff640 1 -- 172.21.15.98:0/503170062 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6640a3110 con 0x7ff6640a5240 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.964+0000 7ff663fff640 1 --2- 172.21.15.98:0/503170062 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff6640a2530 0x7ff6640a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff65bfff640 1 --2- 172.21.15.98:0/503170062 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a5240 0x7ff6640a59a0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0097d0 tx=0x7ff64c02b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f123144ca79f510 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 1 -- 172.21.15.98:0/503170062 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff64c037070 con 0x7ff6640a5240 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:13:36.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 10 monclient: dump: 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.978 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 10 monclient: _finish_auth 0 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff66a166640 5 monclient: authenticate success, global_id 22133 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 1 -- 172.21.15.98:0/503170062 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff64c030980 con 0x7ff6640a5240 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 1 -- 172.21.15.98:0/503170062 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff64c030bb0 con 0x7ff6640a5240 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff662ffd640 10 monclient: dump: 2024-06-11T19:13:36.979 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.980 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.980 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.980 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.980 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.980 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.980 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.980 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.980 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.980 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.980 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.980 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.980 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff66a166640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.965+0000 7ff66a166640 4 set_mon_vals no callback set 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 10 monclient: shutdown 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 1 -- 172.21.15.98:0/503170062 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a5240 msgr2=0x7ff6640a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 1 --2- 172.21.15.98:0/503170062 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a5240 0x7ff6640a59a0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0097d0 tx=0x7ff64c02b6d0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 1 -- 172.21.15.98:0/503170062 shutdown_connections 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 1 --2- 172.21.15.98:0/503170062 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff6640a2530 0x7ff6640a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 1 --2- 172.21.15.98:0/503170062 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a5240 0x7ff6640a59a0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 1 --2- 172.21.15.98:0/503170062 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6640a4920 0x7ff6640a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 1 -- 172.21.15.98:0/503170062 >> 172.21.15.98:0/503170062 conn(0x7ff6640ab850 msgr2=0x7ff6640a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 1 -- 172.21.15.98:0/503170062 shutdown_connections 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 1 -- 172.21.15.98:0/503170062 wait complete. 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 10 monclient: build_initial_monmap 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.966+0000 7ff66a166640 10 monclient: monmap: 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:36.982 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 1 librados: starting msgr at 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 1 librados: starting objecter 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 1 Processor -- start 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 1 -- start start 2024-06-11T19:13:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 1 librados: setting wanted keys 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 1 librados: calling monclient init 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 10 monclient: init 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 10 monclient: _add_conns ranks=[2,3,0,6,1,5,8,7,4] 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a2530 0x7ff664137620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 10 monclient(hunting): picked mon.noname-b con 0x7ff6640a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6640a4920 0x7ff664137b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 10 monclient(hunting): picked mon.noname-g con 0x7ff6640a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6640a5240 0x7ff6641231a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 10 monclient(hunting): picked mon.noname-a con 0x7ff6640a5240 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6640a6da0 con 0x7ff6640a5240 2024-06-11T19:13:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff664138e50 con 0x7ff6640a2530 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ff6640a6420 con 0x7ff6640a4920 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff66a166640 10 monclient(hunting): authenticate will time out at 1026.774737s 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff6637fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a2530 0x7ff664137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff6637fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a2530 0x7ff664137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55540/0 (socket says 172.21.15.98:55540) 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff6637fe640 1 -- 172.21.15.98:0/1083934429 learned_addr learned my addr 172.21.15.98:0/1083934429 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff6637fe640 10 monclient(hunting): get_auth_request con 0x7ff6640a2530 auth_method 0 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff6637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff6637fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff6637fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff663fff640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6640a5240 0x7ff6641231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff663fff640 10 monclient(hunting): get_auth_request con 0x7ff6640a5240 auth_method 0 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff663fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff663fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff663fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff6637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff6637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff6637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff663fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff663fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.967+0000 7ff663fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff65bfff640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6640a4920 0x7ff664137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff65bfff640 10 monclient(hunting): get_auth_request con 0x7ff6640a4920 auth_method 0 2024-06-11T19:13:36.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff65bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff65bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff65bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff6637fe640 10 monclient(hunting): handle_auth_done global_id 22142 payload 824 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff6637fe640 1 -- 172.21.15.98:0/1083934429 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6640a4920 msgr2=0x7ff664137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff6637fe640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6640a4920 0x7ff664137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff6637fe640 1 -- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6640a5240 msgr2=0x7ff6641231a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff6637fe640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6640a5240 0x7ff6641231a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff6637fe640 10 monclient: _finish_hunting 0 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff6637fe640 1 monclient: found mon.noname-b 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff6637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff6637fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff6637fe640 1 -- 172.21.15.98:0/1083934429 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6641268c0 con 0x7ff6640a2530 2024-06-11T19:13:36.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff663fff640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6640a5240 0x7ff6641231a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff6637fe640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a2530 0x7ff664137620 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ff65400d800 tx=0x7ff65400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 10 client.?.objecter ms_handle_connect 0x7ff6640a2530 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff65400bad0 con 0x7ff6640a2530 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 10 monclient: dump: 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.988 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 10 monclient: _finish_auth 0 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff654019070 con 0x7ff6640a2530 2024-06-11T19:13:36.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff66a166640 5 monclient: authenticate success, global_id 22142 2024-06-11T19:13:36.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff66a166640 10 monclient: _renew_subs 2024-06-11T19:13:36.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff66a166640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:36.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff66a166640 1 -- 172.21.15.98:0/1083934429 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6640a90c0 con 0x7ff6640a2530 2024-06-11T19:13:36.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:36.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff65400bcb0 con 0x7ff6640a2530 2024-06-11T19:13:36.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:36.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:36.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff6627fc640 4 set_mon_vals no callback set 2024-06-11T19:13:36.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff660ff9640 10 monclient: dump: 2024-06-11T19:13:36.990 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:36.990 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:36.990 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:36.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff66a166640 10 client.22142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff66a166640 10 monclient: _renew_subs 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff66a166640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff66a166640 1 -- 172.21.15.98:0/1083934429 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff664126e80 con 0x7ff6640a2530 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.968+0000 7ff66a166640 1 librados: init done 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff66a166640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff66a166640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff6641236e0 0x7ff664123ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff66a166640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff66a166640 1 -- 172.21.15.98:0/1083934429 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff664069750 con 0x7ff6641236e0 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff65400b660 con 0x7ff6640a2530 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff65bfff640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff6641236e0 0x7ff664123ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff65bfff640 10 monclient: get_auth_request con 0x7ff6641236e0 auth_method 0 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff65bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff65bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:36.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff65bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff65bfff640 1 -- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff6641236e0 msgr2=0x7ff664123ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff65bfff640 1 -- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff6641236e0 msgr2=0x7ff664123ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff65bfff640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff6641236e0 0x7ff664123ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff65bfff640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff6641236e0 0x7ff664123ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff660ff9640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff64403fdb0 0x7ff644042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(182..182 src has 1..182) ==== 6628+0+0 (secure 0 0 0) 0x7ff65400f040 con 0x7ff6640a2530 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff660ff9640 10 client.22142.objecter ms_dispatch 0x7ff664000b90 osd_map(182..182 src has 1..182) 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff660ff9640 3 client.22142.objecter handle_osd_map got epochs [182,182] > 0 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff660ff9640 3 client.22142.objecter handle_osd_map decoding full epoch 182 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.969+0000 7ff660ff9640 20 client.22142.objecter dump_active .. 0 homeless 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.970+0000 7ff663fff640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff64403fdb0 0x7ff644042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.970+0000 7ff663fff640 10 monclient: get_auth_request con 0x7ff64403fdb0 auth_method 0 2024-06-11T19:13:36.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.970+0000 7ff660ff9640 10 client.22142.objecter ms_handle_connect 0x7ff64403fdb0 2024-06-11T19:13:36.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:36.970+0000 7ff663fff640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff64403fdb0 0x7ff644042270 secure :-1 s=READY pgs=4081 cs=0 l=1 rev1=1 crypto rx=0x7ff664124ca0 tx=0x7ff65c00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff65bfff640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff6641236e0 0x7ff664123ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff65bfff640 10 monclient: get_auth_request con 0x7ff6641236e0 auth_method 0 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff65bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff65bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff65bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff65bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff65bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff65bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff65bfff640 10 monclient(hunting): handle_auth_done global_id 54708 payload 824 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff65bfff640 1 -- 172.21.15.98:0/1083934429 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6640a92f0 con 0x7ff6641236e0 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff65bfff640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff6641236e0 0x7ff664123ac0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0097a0 tx=0x7ff64c0046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 10 client.22142.objecter ms_handle_connect 0x7ff6641236e0 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 10 client.22142.objecter resend_mon_ops 2024-06-11T19:13:37.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff64c042070 con 0x7ff6641236e0 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 10 monclient: dump: 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.172 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.173 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff64c009300 con 0x7ff6641236e0 2024-06-11T19:13:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff6641236e0 msgr2=0x7ff664123ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff6641236e0 0x7ff664123ac0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0097a0 tx=0x7ff64c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff6641236e0 msgr2=0x7ff664123ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.170+0000 7ff660ff9640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff6641236e0 0x7ff664123ac0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff66a166640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:37.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff66a166640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff664124000 0x7ff66409e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff66a166640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff66a166640 1 -- 172.21.15.98:0/1083934429 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff664050830 con 0x7ff664124000 2024-06-11T19:13:37.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff65bfff640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff664124000 0x7ff66409e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff65bfff640 10 monclient: get_auth_request con 0x7ff664124000 auth_method 0 2024-06-11T19:13:37.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff65bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff65bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff65bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff65bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff65bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff65bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff65bfff640 10 monclient(hunting): handle_auth_done global_id 54717 payload 824 2024-06-11T19:13:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.177+0000 7ff65bfff640 1 -- 172.21.15.98:0/1083934429 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff6640a66f0 con 0x7ff664124000 2024-06-11T19:13:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff65bfff640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff664124000 0x7ff66409e9f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0096f0 tx=0x7ff64c009190 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 10 client.22142.objecter ms_handle_connect 0x7ff664124000 2024-06-11T19:13:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 10 client.22142.objecter resend_mon_ops 2024-06-11T19:13:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff64c037040 con 0x7ff664124000 2024-06-11T19:13:37.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 10 monclient: dump: 2024-06-11T19:13:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7ff64c030720 con 0x7ff664124000 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff664124000 msgr2=0x7ff66409e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff664124000 0x7ff66409e9f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0096f0 tx=0x7ff64c009190 comp rx=0 tx=0).stop 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 1 -- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff664124000 msgr2=0x7ff66409e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff660ff9640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff664124000 0x7ff66409e9f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.182 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 129934, 2024-06-11T19:13:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:37.182 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.183 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.184 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.185 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:37.186 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.187 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:37.188 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.190 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:37.192 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.195 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.197 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:37.199 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:37.202 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.205 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.206 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.207 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.207 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:37.207 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.207 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:37.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.209 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:37.209 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 10 librados: watch_flush enter 2024-06-11T19:13:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 10 librados: watch_flush exit 2024-06-11T19:13:37.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 20 client.22142.objecter shutdown clearing up homeless session... 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 10 client.22142.objecter successfully canceled tick 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 1 -- 172.21.15.98:0/1083934429 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff64403fdb0 msgr2=0x7ff644042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff64403fdb0 0x7ff644042270 secure :-1 s=READY pgs=4081 cs=0 l=1 rev1=1 crypto rx=0x7ff664124ca0 tx=0x7ff65c00b040 comp rx=0 tx=0).stop 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 10 monclient: shutdown 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 1 -- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a2530 msgr2=0x7ff664137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a2530 0x7ff664137620 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ff65400d800 tx=0x7ff65400dcc0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff663fff640 1 -- 172.21.15.98:0/1083934429 reap_dead start 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 1 -- 172.21.15.98:0/1083934429 shutdown_connections 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 1 --2- 172.21.15.98:0/1083934429 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff6640a2530 0x7ff664137620 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 1 -- 172.21.15.98:0/1083934429 >> 172.21.15.98:0/1083934429 conn(0x7ff6640ab850 msgr2=0x7ff6641bffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 1 -- 172.21.15.98:0/1083934429 shutdown_connections 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 1 -- 172.21.15.98:0/1083934429 wait complete. 2024-06-11T19:13:37.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.178+0000 7ff66a166640 1 librados: shutdown 2024-06-11T19:13:37.213 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:37.213 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:13:37.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:13:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.313+0000 7fa49d2ff640 10 monclient: get_monmap_and_config 2024-06-11T19:13:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.313+0000 7fa49d2ff640 10 monclient: build_initial_monmap 2024-06-11T19:13:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.313+0000 7fa49d2ff640 10 monclient: monmap: 2024-06-11T19:13:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:37.314305+0000 2024-06-11T19:13:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:37.314305+0000 2024-06-11T19:13:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:37.315 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.313+0000 7fa49d2ff640 1 Processor -- start 2024-06-11T19:13:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.313+0000 7fa49d2ff640 1 -- start start 2024-06-11T19:13:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.313+0000 7fa49d2ff640 10 monclient: init 2024-06-11T19:13:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 10 monclient: _add_conns ranks=[10,1,4,6,5,9,3,2,0,8,7] 2024-06-11T19:13:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa4980a52a0 0x7fa4980a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 10 monclient(hunting): picked mon.noname-f con 0x7fa4980a52a0 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa49809d520 addr v1:172.21.15.117:6789/0 2024-06-11T19:13:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa498073830 0x7fa4980a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 10 monclient(hunting): picked mon.noname-c con 0x7fa498073830 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa49809d2e0 con 0x7fa49809d520 2024-06-11T19:13:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fa49809daa0 con 0x7fa4980a52a0 2024-06-11T19:13:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa4980a5680 con 0x7fa498073830 2024-06-11T19:13:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa49d2ff640 10 monclient(hunting): authenticate will time out at 1027.121398s 2024-06-11T19:13:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa498073830 0x7fa4980a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa498073830 0x7fa4980a3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52346/0 (socket says 172.21.15.98:52346) 2024-06-11T19:13:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 1 -- 172.21.15.98:0/3004738263 learned_addr learned my addr 172.21.15.98:0/3004738263 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 10 monclient(hunting): get_auth_request con 0x7fa498073830 auth_method 0 2024-06-11T19:13:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa496ffd640 1 --2- 172.21.15.98:0/3004738263 >> v2:172.21.15.117:3300/0 conn(0x7fa4980a52a0 0x7fa4980a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa496ffd640 10 monclient(hunting): get_auth_request con 0x7fa4980a52a0 auth_method 0 2024-06-11T19:13:37.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa496ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa496ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 10 monclient(hunting): handle_auth_done global_id 23485 payload 274 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 1 -- 172.21.15.98:0/3004738263 >> v2:172.21.15.117:3300/0 conn(0x7fa4980a52a0 msgr2=0x7fa4980a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 1 --2- 172.21.15.98:0/3004738263 >> v2:172.21.15.117:3300/0 conn(0x7fa4980a52a0 0x7fa4980a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 1 -- 172.21.15.98:0/3004738263 >> v1:172.21.15.117:6789/0 conn(0x7fa49809d520 legacy=0x7fa4980a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 10 monclient: _finish_hunting 0 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 1 monclient: found mon.noname-c 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.314+0000 7fa4977fe640 1 -- 172.21.15.98:0/3004738263 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49809e9d0 con 0x7fa498073830 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa4977fe640 1 --2- 172.21.15.98:0/3004738263 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa498073830 0x7fa4980a3da0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fa48800bb10 tx=0x7fa48802df90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=6b9568dc0dda9a31 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 1 -- 172.21.15.98:0/3004738263 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa48803b070 con 0x7fa498073830 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 10 monclient: dump: 2024-06-11T19:13:37.327 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 10 monclient: _finish_auth 0 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 1 -- 172.21.15.98:0/3004738263 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa488034340 con 0x7fa498073830 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa49d2ff640 5 monclient: authenticate success, global_id 23485 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa49d2ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa49d2ff640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 1 -- 172.21.15.98:0/3004738263 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4880349e0 con 0x7fa498073830 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa49d2ff640 4 set_mon_vals no callback set 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.315+0000 7fa495ffb640 10 monclient: dump: 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.329 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.316+0000 7fa49d2ff640 10 monclient: shutdown 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.316+0000 7fa49d2ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.316+0000 7fa49d2ff640 1 -- 172.21.15.98:0/3004738263 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa498073830 msgr2=0x7fa4980a3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.316+0000 7fa49d2ff640 1 --2- 172.21.15.98:0/3004738263 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa498073830 0x7fa4980a3da0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fa48800bb10 tx=0x7fa48802df90 comp rx=0 tx=0).stop 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.316+0000 7fa49d2ff640 1 -- 172.21.15.98:0/3004738263 shutdown_connections 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.316+0000 7fa49d2ff640 1 --2- 172.21.15.98:0/3004738263 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa498073830 0x7fa4980a3da0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.316+0000 7fa49d2ff640 1 --2- 172.21.15.98:0/3004738263 >> v2:172.21.15.117:3300/0 conn(0x7fa4980a52a0 0x7fa4980a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.316+0000 7fa49d2ff640 1 -- 172.21.15.98:0/3004738263 >> 172.21.15.98:0/3004738263 conn(0x7fa4980a4780 msgr2=0x7fa4980a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.316+0000 7fa49d2ff640 1 -- 172.21.15.98:0/3004738263 shutdown_connections 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.316+0000 7fa49d2ff640 1 -- 172.21.15.98:0/3004738263 wait complete. 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.316+0000 7fa49d2ff640 10 monclient: build_initial_monmap 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.316+0000 7fa49d2ff640 10 monclient: monmap: 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:37.331 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 1 librados: starting msgr at 2024-06-11T19:13:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 1 librados: starting objecter 2024-06-11T19:13:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 1 Processor -- start 2024-06-11T19:13:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 1 -- start start 2024-06-11T19:13:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 1 librados: setting wanted keys 2024-06-11T19:13:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 1 librados: calling monclient init 2024-06-11T19:13:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 10 monclient: init 2024-06-11T19:13:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 10 monclient: _add_conns ranks=[7,1,6,5,2,4,8,0,3] 2024-06-11T19:13:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa498073830 0x7fa4981c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 10 monclient(hunting): picked mon.noname-i con 0x7fa498073830 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:37.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa49809d520 0x7fa4981c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 10 monclient(hunting): picked mon.noname-f con 0x7fa49809d520 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa4980a52a0 0x7fa49811fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 10 monclient(hunting): picked mon.noname-d con 0x7fa4980a52a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa4980a4460 con 0x7fa49809d520 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa4980a42e0 con 0x7fa4980a52a0 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa4980a1e50 con 0x7fa498073830 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa49d2ff640 10 monclient(hunting): authenticate will time out at 1027.124855s 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa4977fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa4980a52a0 0x7fa49811fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa4977fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa4980a52a0 0x7fa49811fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:53280/0 (socket says 172.21.15.98:53280) 2024-06-11T19:13:37.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa4977fe640 1 -- 172.21.15.98:0/1749445069 learned_addr learned my addr 172.21.15.98:0/1749445069 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa4977fe640 10 monclient(hunting): get_auth_request con 0x7fa4980a52a0 auth_method 0 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa4977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa4977fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.317+0000 7fa4977fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa496ffd640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa498073830 0x7fa4981c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4967fc640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa49809d520 0x7fa4981c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa496ffd640 10 monclient(hunting): get_auth_request con 0x7fa498073830 auth_method 0 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa496ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa496ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4967fc640 10 monclient(hunting): get_auth_request con 0x7fa49809d520 auth_method 0 2024-06-11T19:13:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4967fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4967fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa496ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa496ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa496ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 10 monclient(hunting): handle_auth_done global_id 54726 payload 824 2024-06-11T19:13:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 1 -- 172.21.15.98:0/1749445069 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa498073830 msgr2=0x7fa4981c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa498073830 0x7fa4981c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 1 -- 172.21.15.98:0/1749445069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa49809d520 msgr2=0x7fa4981c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa49809d520 0x7fa4981c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 10 monclient: _finish_hunting 0 2024-06-11T19:13:37.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 1 monclient: found mon.noname-d 2024-06-11T19:13:37.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:37.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:13:37.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 1 -- 172.21.15.98:0/1749445069 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa498068760 con 0x7fa4980a52a0 2024-06-11T19:13:37.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4977fe640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa4980a52a0 0x7fa49811fc80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa48800bb10 tx=0x7fa4880046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 10 client.?.objecter ms_handle_connect 0x7fa4980a52a0 2024-06-11T19:13:37.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:37.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa48803b040 con 0x7fa4980a52a0 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 10 monclient: dump: 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa49d2ff640 5 monclient: authenticate success, global_id 54726 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa488046070 con 0x7fa4980a52a0 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4880344f0 con 0x7fa4980a52a0 2024-06-11T19:13:37.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa4957fa640 4 set_mon_vals no callback set 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa48f7fe640 10 monclient: dump: 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.340 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa49d2ff640 10 monclient: _renew_subs 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa49d2ff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa49d2ff640 1 -- 172.21.15.98:0/1749445069 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa498121900 con 0x7fa4980a52a0 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa49d2ff640 10 client.54726.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa49d2ff640 10 monclient: _renew_subs 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa49d2ff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa49d2ff640 1 -- 172.21.15.98:0/1749445069 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fa498123910 con 0x7fa4980a52a0 2024-06-11T19:13:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.318+0000 7fa49d2ff640 1 librados: init done 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa49d2ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa49d2ff640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa4981201c0 0x7fa4981205a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fa488034690 con 0x7fa4980a52a0 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa49d2ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa49d2ff640 1 -- 172.21.15.98:0/1749445069 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa498075170 con 0x7fa4981201c0 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa496ffd640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa4981201c0 0x7fa4981205a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa496ffd640 10 monclient: get_auth_request con 0x7fa4981201c0 auth_method 0 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa496ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa496ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa496ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa496ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa496ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa48f7fe640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa47803fdb0 0x7fa478042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(182..182 src has 1..182) ==== 6628+0+0 (secure 0 0 0) 0x7fa4880761b0 con 0x7fa4980a52a0 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa48f7fe640 10 client.54726.objecter ms_dispatch 0x7fa498000b90 osd_map(182..182 src has 1..182) 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa48f7fe640 3 client.54726.objecter handle_osd_map got epochs [182,182] > 0 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa48f7fe640 3 client.54726.objecter handle_osd_map decoding full epoch 182 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa496ffd640 10 monclient(hunting): handle_auth_done global_id 64178 payload 824 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa496ffd640 1 -- 172.21.15.98:0/1749445069 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa498122460 con 0x7fa4981201c0 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa48f7fe640 20 client.54726.objecter dump_active .. 0 homeless 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa4967fc640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa47803fdb0 0x7fa478042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa4967fc640 10 monclient: get_auth_request con 0x7fa47803fdb0 auth_method 0 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa48f7fe640 10 client.54726.objecter ms_handle_connect 0x7fa4981201c0 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa48f7fe640 10 client.54726.objecter resend_mon_ops 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.319+0000 7fa496ffd640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa4981201c0 0x7fa4981205a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa490004610 tx=0x7fa490007700 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa490015890 con 0x7fa4981201c0 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa48f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa48f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa48f7fe640 10 monclient: dump: 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.344 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa48f7fe640 10 client.54726.objecter ms_handle_connect 0x7fa47803fdb0 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa4967fc640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa47803fdb0 0x7fa478042270 secure :-1 s=READY pgs=4082 cs=0 l=1 rev1=1 crypto rx=0x7fa498121730 tx=0x7fa480007e30 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa49001a070 con 0x7fa4981201c0 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa48f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa48f7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa4981201c0 msgr2=0x7fa4981205a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa48f7fe640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa4981201c0 0x7fa4981205a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa490004610 tx=0x7fa490007700 comp rx=0 tx=0).stop 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa4981201c0 msgr2=0x7fa4981205a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.320+0000 7fa48f7fe640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa4981201c0 0x7fa4981205a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.326+0000 7fa49d2ff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.326+0000 7fa49d2ff640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa498120ae0 0x7fa498071540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.326+0000 7fa49d2ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.326+0000 7fa49d2ff640 1 -- 172.21.15.98:0/1749445069 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa498050830 con 0x7fa498120ae0 2024-06-11T19:13:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.326+0000 7fa496ffd640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa498120ae0 0x7fa498071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.326+0000 7fa496ffd640 10 monclient: get_auth_request con 0x7fa498120ae0 auth_method 0 2024-06-11T19:13:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.326+0000 7fa496ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.326+0000 7fa496ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.326+0000 7fa496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.326+0000 7fa496ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.326+0000 7fa496ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.326+0000 7fa496ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa496ffd640 10 monclient(hunting): handle_auth_done global_id 64187 payload 824 2024-06-11T19:13:37.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa496ffd640 1 -- 172.21.15.98:0/1749445069 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa4980a1910 con 0x7fa498120ae0 2024-06-11T19:13:37.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa496ffd640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa498120ae0 0x7fa498071540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa490004610 tx=0x7fa490004a40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 10 client.54726.objecter ms_handle_connect 0x7fa498120ae0 2024-06-11T19:13:37.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 10 client.54726.objecter resend_mon_ops 2024-06-11T19:13:37.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa490028410 con 0x7fa498120ae0 2024-06-11T19:13:37.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:13:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 10 monclient: dump: 2024-06-11T19:13:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9441 (secure 0 0 0) 0x7fa490015a00 con 0x7fa498120ae0 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa498120ae0 msgr2=0x7fa498071540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa498120ae0 0x7fa498071540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa490004610 tx=0x7fa490004a40 comp rx=0 tx=0).stop 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 1 -- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa498120ae0 msgr2=0x7fa498071540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa48f7fe640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa498120ae0 0x7fa498071540 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.352 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:37.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:37.352 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:37.352 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:37.352 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:37.352 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 9154, 2024-06-11T19:13:37.352 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:37.352 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:37.352 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:37.352 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.354 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.354 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.354 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.354 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.354 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.354 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.354 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.354 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.354 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.354 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:37.354 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:37.357 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:37.357 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:37.357 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:37.357 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:37.357 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.357 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.358 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.358 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.358 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.358 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.358 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.358 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.358 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.358 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.358 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.358 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:37.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:37.364 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:37.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:37.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.366 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.366 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:37.367 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.369 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:37.370 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:37.370 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.370 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.370 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.370 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:37.370 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:37.370 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:37.371 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:37.372 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.372 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.372 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.372 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:37.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:37.372 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.373 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:37.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:37.375 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:37.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa49d2ff640 10 librados: watch_flush enter 2024-06-11T19:13:37.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa49d2ff640 10 librados: watch_flush exit 2024-06-11T19:13:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa49d2ff640 20 client.54726.objecter shutdown clearing up homeless session... 2024-06-11T19:13:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa49d2ff640 10 client.54726.objecter successfully canceled tick 2024-06-11T19:13:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa49d2ff640 1 -- 172.21.15.98:0/1749445069 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa47803fdb0 msgr2=0x7fa478042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa49d2ff640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa47803fdb0 0x7fa478042270 secure :-1 s=READY pgs=4082 cs=0 l=1 rev1=1 crypto rx=0x7fa498121730 tx=0x7fa480007e30 comp rx=0 tx=0).stop 2024-06-11T19:13:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa49d2ff640 10 monclient: shutdown 2024-06-11T19:13:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa49d2ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa49d2ff640 1 -- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa4980a52a0 msgr2=0x7fa49811fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa49d2ff640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa4980a52a0 0x7fa49811fc80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa48800bb10 tx=0x7fa4880046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa4977fe640 1 -- 172.21.15.98:0/1749445069 reap_dead start 2024-06-11T19:13:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa49d2ff640 1 -- 172.21.15.98:0/1749445069 shutdown_connections 2024-06-11T19:13:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.327+0000 7fa49d2ff640 1 --2- 172.21.15.98:0/1749445069 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa4980a52a0 0x7fa49811fc80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.328+0000 7fa49d2ff640 1 -- 172.21.15.98:0/1749445069 >> 172.21.15.98:0/1749445069 conn(0x7fa4980a4780 msgr2=0x7fa49809f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:37.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.328+0000 7fa49d2ff640 1 -- 172.21.15.98:0/1749445069 shutdown_connections 2024-06-11T19:13:37.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.328+0000 7fa49d2ff640 1 -- 172.21.15.98:0/1749445069 wait complete. 2024-06-11T19:13:37.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.328+0000 7fa49d2ff640 1 librados: shutdown 2024-06-11T19:13:37.379 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:37.379 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:13:37.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.479+0000 7ff100f4e640 10 monclient: get_monmap_and_config 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.479+0000 7ff100f4e640 10 monclient: build_initial_monmap 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.479+0000 7ff100f4e640 10 monclient: monmap: 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:37.480645+0000 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:37.480645+0000 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:37.480 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 1 Processor -- start 2024-06-11T19:13:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 1 -- start start 2024-06-11T19:13:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 10 monclient: init 2024-06-11T19:13:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 10 monclient: _add_conns ranks=[7,1,8,4,3,5,9,6,10,2,0] 2024-06-11T19:13:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff0fc0a4920 0x7ff0fc0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 10 monclient(hunting): picked mon.noname-i con 0x7ff0fc0a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff0fc0a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff0fc0a2530 0x7ff0fc0a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 10 monclient(hunting): picked mon.noname-e con 0x7ff0fc0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0fc0a7140 con 0x7ff0fc0a5240 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff0fc0a2b50 con 0x7ff0fc0a4920 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff0fc0a2e30 con 0x7ff0fc0a2530 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff100f4e640 10 monclient(hunting): authenticate will time out at 1027.287850s 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff0fad9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff0fc0a2530 0x7ff0fc0a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.480+0000 7ff0fad9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff0fc0a2530 0x7ff0fc0a60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35402/0 (socket says 172.21.15.98:35402) 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 1 -- 172.21.15.98:0/876551335 learned_addr learned my addr 172.21.15.98:0/876551335 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 10 monclient(hunting): get_auth_request con 0x7ff0fc0a2530 auth_method 0 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fa59c640 1 --2- 172.21.15.98:0/876551335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff0fc0a4920 0x7ff0fc0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 10 monclient(hunting): handle_auth_done global_id 64196 payload 274 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 1 -- 172.21.15.98:0/876551335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff0fc0a4920 msgr2=0x7ff0fc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 1 --2- 172.21.15.98:0/876551335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff0fc0a4920 0x7ff0fc0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 1 -- 172.21.15.98:0/876551335 >> v1:172.21.15.117:6789/0 conn(0x7ff0fc0a5240 legacy=0x7ff0fc0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 1 monclient: found mon.noname-e 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fad9d640 1 -- 172.21.15.98:0/876551335 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc0a3110 con 0x7ff0fc0a2530 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fa59c640 10 monclient: get_auth_request con 0x7ff0fc0a4920 auth_method 0 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.481+0000 7ff0fa59c640 1 --2- 172.21.15.98:0/876551335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff0fc0a4920 0x7ff0fc0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0fad9d640 1 --2- 172.21.15.98:0/876551335 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff0fc0a2530 0x7ff0fc0a60b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009e80 tx=0x7ff0f402be10 comp rx=0 tx=0).ready entity=mon.8 client_cookie=3184eed856a68a2d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 1 -- 172.21.15.98:0/876551335 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0f402cc20 con 0x7ff0fc0a2530 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 10 monclient: dump: 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.492 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 10 monclient: _finish_auth 0 2024-06-11T19:13:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 10 monclient: _check_auth_tickets 2024-06-11T19:13:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 1 -- 172.21.15.98:0/876551335 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f402cde0 con 0x7ff0fc0a2530 2024-06-11T19:13:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 5 monclient: authenticate success, global_id 64196 2024-06-11T19:13:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 1 -- 172.21.15.98:0/876551335 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0f4034860 con 0x7ff0fc0a2530 2024-06-11T19:13:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 4 set_mon_vals no callback set 2024-06-11T19:13:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff0f959a640 10 monclient: dump: 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.495 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 10 monclient: shutdown 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 1 -- 172.21.15.98:0/876551335 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff0fc0a2530 msgr2=0x7ff0fc0a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 1 --2- 172.21.15.98:0/876551335 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff0fc0a2530 0x7ff0fc0a60b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009e80 tx=0x7ff0f402be10 comp rx=0 tx=0).stop 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 1 -- 172.21.15.98:0/876551335 shutdown_connections 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 1 --2- 172.21.15.98:0/876551335 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff0fc0a2530 0x7ff0fc0a60b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 1 --2- 172.21.15.98:0/876551335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff0fc0a4920 0x7ff0fc0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 1 -- 172.21.15.98:0/876551335 >> 172.21.15.98:0/876551335 conn(0x7ff0fc0ab850 msgr2=0x7ff0fc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 1 -- 172.21.15.98:0/876551335 shutdown_connections 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.482+0000 7ff100f4e640 1 -- 172.21.15.98:0/876551335 wait complete. 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.483+0000 7ff100f4e640 10 monclient: build_initial_monmap 2024-06-11T19:13:37.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.483+0000 7ff100f4e640 10 monclient: monmap: 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.483+0000 7ff100f4e640 1 librados: starting msgr at 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.483+0000 7ff100f4e640 1 librados: starting objecter 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.483+0000 7ff100f4e640 1 Processor -- start 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.483+0000 7ff100f4e640 1 -- start start 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.483+0000 7ff100f4e640 1 librados: setting wanted keys 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.483+0000 7ff100f4e640 1 librados: calling monclient init 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.483+0000 7ff100f4e640 10 monclient: init 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 10 monclient: _add_conns ranks=[3,2,5,8,1,0,7,4,6] 2024-06-11T19:13:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0fc0a2530 0x7ff0fc1cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 10 monclient(hunting): picked mon.noname-g con 0x7ff0fc0a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0fc0a4920 0x7ff0fc1cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 10 monclient(hunting): picked mon.noname-b con 0x7ff0fc0a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0fc0a5240 0x7ff0fc1221a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 10 monclient(hunting): picked mon.noname-h con 0x7ff0fc0a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc0a6da0 con 0x7ff0fc0a4920 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc138e50 con 0x7ff0fc0a2530 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ff0fc0a65f0 con 0x7ff0fc0a5240 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff100f4e640 10 monclient(hunting): authenticate will time out at 1027.291324s 2024-06-11T19:13:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0fc0a4920 0x7ff0fc1cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0fc0a4920 0x7ff0fc1cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55556/0 (socket says 172.21.15.98:55556) 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 1 -- 172.21.15.98:0/745409376 learned_addr learned my addr 172.21.15.98:0/745409376 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 10 monclient(hunting): get_auth_request con 0x7ff0fc0a4920 auth_method 0 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0fad9d640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0fc0a5240 0x7ff0fc1221a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0fa59c640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0fc0a2530 0x7ff0fc1cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0fad9d640 10 monclient(hunting): get_auth_request con 0x7ff0fc0a5240 auth_method 0 2024-06-11T19:13:37.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0fad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 10 monclient(hunting): handle_auth_done global_id 22151 payload 824 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 1 -- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0fc0a5240 msgr2=0x7ff0fc1221a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0fc0a5240 0x7ff0fc1221a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 1 -- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0fc0a2530 msgr2=0x7ff0fc1cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0fc0a2530 0x7ff0fc1cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 1 monclient: found mon.noname-b 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 1 -- 172.21.15.98:0/745409376 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc125a80 con 0x7ff0fc0a4920 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0fa59c640 10 monclient: get_auth_request con 0x7ff0fc0a2530 auth_method 0 2024-06-11T19:13:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0fa59c640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0fc0a2530 0x7ff0fc1cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0f9d9b640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0fc0a4920 0x7ff0fc1cdfb0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000c340 tx=0x7ff0f000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0eaffd640 10 client.?.objecter ms_handle_connect 0x7ff0fc0a4920 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.484+0000 7ff0eaffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0f0017070 con 0x7ff0fc0a4920 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 10 monclient: dump: 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 10 monclient: _finish_auth 0 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 10 monclient: _check_auth_tickets 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f00035f0 con 0x7ff0fc0a4920 2024-06-11T19:13:37.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff100f4e640 5 monclient: authenticate success, global_id 22151 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff100f4e640 10 monclient: _renew_subs 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff100f4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff100f4e640 1 -- 172.21.15.98:0/745409376 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0fc125d60 con 0x7ff0fc0a4920 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0f0003c50 con 0x7ff0fc0a4920 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff100f4e640 10 client.22151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0f8d99640 4 set_mon_vals no callback set 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 10 monclient: dump: 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.504 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff100f4e640 10 monclient: _renew_subs 2024-06-11T19:13:37.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff100f4e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:13:37.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff100f4e640 1 -- 172.21.15.98:0/745409376 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0fc1387f0 con 0x7ff0fc0a4920 2024-06-11T19:13:37.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff100f4e640 1 librados: init done 2024-06-11T19:13:37.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.485+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff0f00037e0 con 0x7ff0fc0a4920 2024-06-11T19:13:37.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff100f4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:37.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff100f4e640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc1226e0 0x7ff0fc122ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff100f4e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff100f4e640 1 -- 172.21.15.98:0/745409376 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc124260 con 0x7ff0fc1226e0 2024-06-11T19:13:37.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0eaffd640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0dc03fdb0 0x7ff0dc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(182..182 src has 1..182) ==== 6628+0+0 (secure 0 0 0) 0x7ff0f0050d80 con 0x7ff0fc0a4920 2024-06-11T19:13:37.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0eaffd640 10 client.22151.objecter ms_dispatch 0x7ff0fc000b90 osd_map(182..182 src has 1..182) 2024-06-11T19:13:37.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0eaffd640 3 client.22151.objecter handle_osd_map got epochs [182,182] > 0 2024-06-11T19:13:37.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0eaffd640 3 client.22151.objecter handle_osd_map decoding full epoch 182 2024-06-11T19:13:37.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fa59c640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc1226e0 0x7ff0fc122ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fa59c640 10 monclient: get_auth_request con 0x7ff0fc1226e0 auth_method 0 2024-06-11T19:13:37.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fa59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0eaffd640 20 client.22151.objecter dump_active .. 0 homeless 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fad9d640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0dc03fdb0 0x7ff0dc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fad9d640 10 monclient: get_auth_request con 0x7ff0dc03fdb0 auth_method 0 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fa59c640 10 monclient(hunting): handle_auth_done global_id 31555 payload 824 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fa59c640 1 -- 172.21.15.98:0/745409376 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0fc0a4d00 con 0x7ff0fc1226e0 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0eaffd640 10 client.22151.objecter ms_handle_connect 0x7ff0fc1226e0 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0eaffd640 10 client.22151.objecter resend_mon_ops 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fa59c640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc1226e0 0x7ff0fc122ac0 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec00d790 tx=0x7ff0ec00dc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0eaffd640 10 client.22151.objecter ms_handle_connect 0x7ff0dc03fdb0 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.486+0000 7ff0fad9d640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0dc03fdb0 0x7ff0dc042270 secure :-1 s=READY pgs=4083 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4035000 tx=0x7ff0f4030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.487+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0ec016070 con 0x7ff0fc1226e0 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.487+0000 7ff0eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.487+0000 7ff0eaffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.487+0000 7ff0eaffd640 10 monclient: dump: 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.509 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.487+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff0ec00de40 con 0x7ff0fc1226e0 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.487+0000 7ff0eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.487+0000 7ff0eaffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.487+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc1226e0 msgr2=0x7ff0fc122ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.487+0000 7ff0eaffd640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc1226e0 0x7ff0fc122ac0 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec00d790 tx=0x7ff0ec00dc50 comp rx=0 tx=0).stop 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.487+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc1226e0 msgr2=0x7ff0fc122ac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:37.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.487+0000 7ff0eaffd640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc1226e0 0x7ff0fc122ac0 secure :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec00d790 tx=0x7ff0ec00dc50 comp rx=0 tx=0).stop 2024-06-11T19:13:37.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.493+0000 7ff100f4e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:37.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.493+0000 7ff100f4e640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc123000 0x7ff0fc09e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.493+0000 7ff100f4e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.493+0000 7ff100f4e640 1 -- 172.21.15.98:0/745409376 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc050830 con 0x7ff0fc123000 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0fa59c640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc123000 0x7ff0fc09e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0fa59c640 10 monclient: get_auth_request con 0x7ff0fc123000 auth_method 0 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0fa59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0fa59c640 10 monclient(hunting): handle_auth_done global_id 31564 payload 824 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0fa59c640 1 -- 172.21.15.98:0/745409376 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff0fc06c300 con 0x7ff0fc123000 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0fa59c640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc123000 0x7ff0fc09e2b0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec000c00 tx=0x7ff0ec004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0eaffd640 10 client.22151.objecter ms_handle_connect 0x7ff0fc123000 2024-06-11T19:13:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0eaffd640 10 client.22151.objecter resend_mon_ops 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0ec012d40 con 0x7ff0fc123000 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0eaffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.494+0000 7ff0eaffd640 10 monclient: dump: 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9275 (secure 0 0 0) 0x7ff0ec012d40 con 0x7ff0fc123000 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff0eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff0eaffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc123000 msgr2=0x7ff0fc09e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff0eaffd640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc123000 0x7ff0fc09e2b0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec000c00 tx=0x7ff0ec004730 comp rx=0 tx=0).stop 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff0eaffd640 1 -- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc123000 msgr2=0x7ff0fc09e2b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:37.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff0eaffd640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc123000 0x7ff0fc09e2b0 secure :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec000c00 tx=0x7ff0ec004730 comp rx=0 tx=0).stop 2024-06-11T19:13:37.516 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:13:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 267712, 2024-06-11T19:13:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 9, 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:13:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:37.540 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:37.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 10 librados: watch_flush enter 2024-06-11T19:13:37.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 10 librados: watch_flush exit 2024-06-11T19:13:37.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 20 client.22151.objecter shutdown clearing up homeless session... 2024-06-11T19:13:37.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 10 client.22151.objecter successfully canceled tick 2024-06-11T19:13:37.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 1 -- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0dc03fdb0 msgr2=0x7ff0dc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0dc03fdb0 0x7ff0dc042270 secure :-1 s=READY pgs=4083 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4035000 tx=0x7ff0f4030000 comp rx=0 tx=0).stop 2024-06-11T19:13:37.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 10 monclient: shutdown 2024-06-11T19:13:37.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:37.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 1 -- 172.21.15.98:0/745409376 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0fc0a4920 msgr2=0x7ff0fc1cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 1 --2- 172.21.15.98:0/745409376 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0fc0a4920 0x7ff0fc1cdfb0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000c340 tx=0x7ff0f000c800 comp rx=0 tx=0).stop 2024-06-11T19:13:37.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff0fad9d640 1 -- 172.21.15.98:0/745409376 reap_dead start 2024-06-11T19:13:37.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff0fad9d640 1 -- 172.21.15.98:0/745409376 reap_dead start 2024-06-11T19:13:37.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 1 -- 172.21.15.98:0/745409376 shutdown_connections 2024-06-11T19:13:37.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 1 -- 172.21.15.98:0/745409376 >> 172.21.15.98:0/745409376 conn(0x7ff0fc0ab850 msgr2=0x7ff0fc0726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:37.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 1 -- 172.21.15.98:0/745409376 shutdown_connections 2024-06-11T19:13:37.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 1 -- 172.21.15.98:0/745409376 wait complete. 2024-06-11T19:13:37.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.495+0000 7ff100f4e640 1 librados: shutdown 2024-06-11T19:13:37.543 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:37.543 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:13:37.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:13:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.643+0000 7f3a1b3f0640 10 monclient: get_monmap_and_config 2024-06-11T19:13:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.643+0000 7f3a1b3f0640 10 monclient: build_initial_monmap 2024-06-11T19:13:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.644+0000 7f3a1b3f0640 10 monclient: monmap: 2024-06-11T19:13:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:37.645000+0000 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:37.645000+0000 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:37.645 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:37.645 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:37.645 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:37.645 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.644+0000 7f3a1b3f0640 1 Processor -- start 2024-06-11T19:13:37.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.644+0000 7f3a1b3f0640 1 -- start start 2024-06-11T19:13:37.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.644+0000 7f3a1b3f0640 10 monclient: init 2024-06-11T19:13:37.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:37.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 10 monclient: _add_conns ranks=[3,10,8,7,1,5,9,6,2,4,0] 2024-06-11T19:13:37.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409d9f0 0x7f3a140a1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 10 monclient(hunting): picked mon.noname-g con 0x7f3a1409d9f0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:37.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3a140a0690 0x7f3a140a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 10 monclient(hunting): picked mon.noname-f con 0x7f3a140a0690 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:37.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a140a2380 0x7f3a14136740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 10 monclient(hunting): picked mon.noname-e con 0x7f3a140a2380 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:37.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f3a140a1090 con 0x7f3a140a0690 2024-06-11T19:13:37.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f3a140a3190 con 0x7f3a1409d9f0 2024-06-11T19:13:37.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3a1409ddd0 con 0x7f3a140a2380 2024-06-11T19:13:37.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:37.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1b3f0640 10 monclient(hunting): authenticate will time out at 1027.452355s 2024-06-11T19:13:37.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a140a2380 0x7f3a14136740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a140a2380 0x7f3a14136740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35408/0 (socket says 172.21.15.98:35408) 2024-06-11T19:13:37.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 1 -- 172.21.15.98:0/186650841 learned_addr learned my addr 172.21.15.98:0/186650841 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:37.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 10 monclient(hunting): get_auth_request con 0x7f3a140a2380 auth_method 0 2024-06-11T19:13:37.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1898b640 1 --2- 172.21.15.98:0/186650841 >> v2:172.21.15.117:3300/0 conn(0x7f3a140a0690 0x7f3a140a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1918c640 1 --2- 172.21.15.98:0/186650841 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409d9f0 0x7f3a140a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 10 monclient(hunting): handle_auth_done global_id 64205 payload 274 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 1 -- 172.21.15.98:0/186650841 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409d9f0 msgr2=0x7f3a140a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 1 --2- 172.21.15.98:0/186650841 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409d9f0 0x7f3a140a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 1 -- 172.21.15.98:0/186650841 >> v2:172.21.15.117:3300/0 conn(0x7f3a140a0690 msgr2=0x7f3a140a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 1 --2- 172.21.15.98:0/186650841 >> v2:172.21.15.117:3300/0 conn(0x7f3a140a0690 0x7f3a140a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 10 monclient: _finish_hunting 0 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 1 monclient: found mon.noname-e 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.645+0000 7f3a1998d640 1 -- 172.21.15.98:0/186650841 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a140a2eb0 con 0x7f3a140a2380 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a1998d640 1 --2- 172.21.15.98:0/186650841 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a140a2380 0x7f3a14136740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009e80 tx=0x7f3a0c02bd80 comp rx=0 tx=0).ready entity=mon.8 client_cookie=94900a4c26ea41e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 1 -- 172.21.15.98:0/186650841 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a0c02cb90 con 0x7f3a140a2380 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:13:37.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 10 monclient: dump: 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.658 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.659 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.659 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 10 monclient: _finish_auth 0 2024-06-11T19:13:37.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:13:37.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:37.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 1 -- 172.21.15.98:0/186650841 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a0c02cd50 con 0x7f3a140a2380 2024-06-11T19:13:37.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a1b3f0640 5 monclient: authenticate success, global_id 64205 2024-06-11T19:13:37.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a1b3f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:37.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:37.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a1b3f0640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:37.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 1 -- 172.21.15.98:0/186650841 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a0c034860 con 0x7f3a140a2380 2024-06-11T19:13:37.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a1b3f0640 4 set_mon_vals no callback set 2024-06-11T19:13:37.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:13:37.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.646+0000 7f3a0bfff640 10 monclient: dump: 2024-06-11T19:13:37.660 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.660 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.660 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.660 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.660 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.660 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 10 monclient: shutdown 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:37.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/186650841 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a140a2380 msgr2=0x7f3a14136740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 1 --2- 172.21.15.98:0/186650841 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a140a2380 0x7f3a14136740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009e80 tx=0x7f3a0c02bd80 comp rx=0 tx=0).stop 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/186650841 shutdown_connections 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 1 --2- 172.21.15.98:0/186650841 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a140a2380 0x7f3a14136740 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 1 --2- 172.21.15.98:0/186650841 >> v2:172.21.15.117:3300/0 conn(0x7f3a140a0690 0x7f3a140a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 1 --2- 172.21.15.98:0/186650841 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409d9f0 0x7f3a140a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/186650841 >> 172.21.15.98:0/186650841 conn(0x7f3a140a73b0 msgr2=0x7f3a140a3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/186650841 shutdown_connections 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/186650841 wait complete. 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 10 monclient: build_initial_monmap 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.647+0000 7f3a1b3f0640 10 monclient: monmap: 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:37.662 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 1 librados: starting msgr at 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 1 librados: starting objecter 2024-06-11T19:13:37.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 1 Processor -- start 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 1 -- start start 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 1 librados: setting wanted keys 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 1 librados: calling monclient init 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 10 monclient: init 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 10 monclient: _add_conns ranks=[8,2,7,3,6,4,0,5,1] 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a1409d9f0 0x7f3a141cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 10 monclient(hunting): picked mon.noname-e con 0x7f3a1409d9f0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a140a0690 0x7f3a141cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 10 monclient(hunting): picked mon.noname-b con 0x7f3a140a0690 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a140a2380 0x7f3a1409e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 10 monclient(hunting): picked mon.noname-i con 0x7f3a140a2380 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3a14136c80 con 0x7f3a140a0690 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f3a140a29f0 con 0x7f3a140a2380 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3a14136e00 con 0x7f3a1409d9f0 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1b3f0640 10 monclient(hunting): authenticate will time out at 1027.455679s 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a1409d9f0 0x7f3a141cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a1409d9f0 0x7f3a141cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35418/0 (socket says 172.21.15.98:35418) 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 1 -- 172.21.15.98:0/158381879 learned_addr learned my addr 172.21.15.98:0/158381879 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 10 monclient(hunting): get_auth_request con 0x7f3a1409d9f0 auth_method 0 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1898b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a140a0690 0x7f3a141cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1898b640 10 monclient(hunting): get_auth_request con 0x7f3a140a0690 auth_method 0 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1898b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1898b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1898b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1898b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1898b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1898b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 10 monclient(hunting): handle_auth_done global_id 64214 payload 824 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 1 -- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a140a2380 msgr2=0x7f3a1409e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.648+0000 7f3a1918c640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a140a2380 0x7f3a1409e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1918c640 1 -- 172.21.15.98:0/158381879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a140a0690 msgr2=0x7f3a141cdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1918c640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a140a0690 0x7f3a141cdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1918c640 10 monclient: _finish_hunting 0 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1918c640 1 monclient: found mon.noname-e 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1918c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1918c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1918c640 1 -- 172.21.15.98:0/158381879 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1409fe80 con 0x7f3a1409d9f0 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1918c640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a1409d9f0 0x7f3a141cda10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3a1000c7f0 tx=0x7f3a1000ccb0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 10 client.?.objecter ms_handle_connect 0x7f3a1409d9f0 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a10015070 con 0x7f3a1409d9f0 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 10 monclient: dump: 2024-06-11T19:13:37.667 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 10 monclient: _finish_auth 0 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a10007a90 con 0x7f3a1409d9f0 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1b3f0640 5 monclient: authenticate success, global_id 64214 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1b3f0640 10 monclient: _renew_subs 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1b3f0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/158381879 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a14126660 con 0x7f3a1409d9f0 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a10007c40 con 0x7f3a1409d9f0 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a09ffb640 10 monclient: dump: 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1b3f0640 10 client.64214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a0b7fe640 4 set_mon_vals no callback set 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1b3f0640 10 monclient: _renew_subs 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1b3f0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/158381879 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a14125390 con 0x7f3a1409d9f0 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.649+0000 7f3a1b3f0640 1 librados: init done 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3a100075c0 con 0x7f3a1409d9f0 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1b3f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1b3f0640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409e7c0 0x7f3a1409eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1b3f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/158381879 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f3a140a6090 con 0x7f3a1409e7c0 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a09ffb640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f39f803fdb0 0x7f39f8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(182..182 src has 1..182) ==== 6628+0+0 (secure 0 0 0) 0x7f3a10051aa0 con 0x7f3a1409d9f0 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a09ffb640 10 client.64214.objecter ms_dispatch 0x7f3a14000b90 osd_map(182..182 src has 1..182) 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a09ffb640 3 client.64214.objecter handle_osd_map got epochs [182,182] > 0 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a09ffb640 3 client.64214.objecter handle_osd_map decoding full epoch 182 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a09ffb640 20 client.64214.objecter dump_active .. 0 homeless 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1898b640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409e7c0 0x7f3a1409eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1898b640 10 monclient: get_auth_request con 0x7f3a1409e7c0 auth_method 0 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1898b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1898b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1898b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1998d640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f39f803fdb0 0x7f39f8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1998d640 10 monclient: get_auth_request con 0x7f39f803fdb0 auth_method 0 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1898b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1898b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1898b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1898b640 10 monclient(hunting): handle_auth_done global_id 22053 payload 824 2024-06-11T19:13:37.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.650+0000 7f3a1898b640 1 -- 172.21.15.98:0/158381879 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a14068e20 con 0x7f3a1409e7c0 2024-06-11T19:13:37.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a1998d640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f39f803fdb0 0x7f39f8042270 secure :-1 s=READY pgs=4084 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009e80 tx=0x7f3a0c02e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 10 client.64214.objecter ms_handle_connect 0x7f39f803fdb0 2024-06-11T19:13:37.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 10 client.64214.objecter ms_handle_connect 0x7f3a1409e7c0 2024-06-11T19:13:37.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 10 client.64214.objecter resend_mon_ops 2024-06-11T19:13:37.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a1898b640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409e7c0 0x7f3a1409eba0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400f760 tx=0x7f3a0400fc20 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a0400de90 con 0x7f3a1409e7c0 2024-06-11T19:13:37.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 10 monclient: dump: 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.674 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.675 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.675 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.675 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.675 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3a0400fe70 con 0x7f3a1409e7c0 2024-06-11T19:13:37.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:37.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:37.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409e7c0 msgr2=0x7f3a1409eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409e7c0 0x7f3a1409eba0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400f760 tx=0x7f3a0400fc20 comp rx=0 tx=0).stop 2024-06-11T19:13:37.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409e7c0 msgr2=0x7f3a1409eba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:37.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.651+0000 7f3a09ffb640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409e7c0 0x7f3a1409eba0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.657+0000 7f3a1b3f0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:37.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.657+0000 7f3a1b3f0640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409f0e0 0x7f3a14135780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.657+0000 7f3a1b3f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.657+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/158381879 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f3a14050830 con 0x7f3a1409f0e0 2024-06-11T19:13:37.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a1898b640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409f0e0 0x7f3a14135780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a1898b640 10 monclient: get_auth_request con 0x7f3a1409f0e0 auth_method 0 2024-06-11T19:13:37.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a1898b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a1898b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a1898b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a1898b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a1898b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a1898b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a1898b640 10 monclient(hunting): handle_auth_done global_id 22062 payload 824 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a1898b640 1 -- 172.21.15.98:0/158381879 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3a140a1e40 con 0x7f3a1409f0e0 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a1898b640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409f0e0 0x7f3a14135780 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400f760 tx=0x7f3a04004730 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a09ffb640 10 client.64214.objecter ms_handle_connect 0x7f3a1409f0e0 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a09ffb640 10 client.64214.objecter resend_mon_ops 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a0401ba80 con 0x7f3a1409f0e0 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a09ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.658+0000 7f3a09ffb640 10 monclient: dump: 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.678 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f3a04025390 con 0x7f3a1409f0e0 2024-06-11T19:13:37.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a09ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:37.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a09ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:37.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409f0e0 msgr2=0x7f3a14135780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a09ffb640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409f0e0 0x7f3a14135780 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400f760 tx=0x7f3a04004730 comp rx=0 tx=0).stop 2024-06-11T19:13:37.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a09ffb640 1 -- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409f0e0 msgr2=0x7f3a14135780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:37.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a09ffb640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a1409f0e0 0x7f3a14135780 secure :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400f760 tx=0x7f3a04004730 comp rx=0 tx=0).stop 2024-06-11T19:13:37.680 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:37.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:37.680 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:37.680 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:37.680 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 267875, 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 9, 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:37.681 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.682 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.683 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:37.684 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.685 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.686 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.686 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.686 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.686 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:37.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.686 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:37.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.686 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:37.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:37.686 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:37.687 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:37.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:37.689 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.689 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.691 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.696 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.703 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:37.703 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 10 librados: watch_flush enter 2024-06-11T19:13:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 10 librados: watch_flush exit 2024-06-11T19:13:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 20 client.64214.objecter shutdown clearing up homeless session... 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 10 client.64214.objecter successfully canceled tick 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f39f803fdb0 msgr2=0x7f39f8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f39f803fdb0 0x7f39f8042270 secure :-1 s=READY pgs=4084 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009e80 tx=0x7f3a0c02e000 comp rx=0 tx=0).stop 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 10 monclient: shutdown 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/158381879 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a1409d9f0 msgr2=0x7f3a141cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 1 --2- 172.21.15.98:0/158381879 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a1409d9f0 0x7f3a141cda10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3a1000c7f0 tx=0x7f3a1000ccb0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1998d640 1 -- 172.21.15.98:0/158381879 reap_dead start 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1998d640 1 -- 172.21.15.98:0/158381879 reap_dead start 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/158381879 shutdown_connections 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/158381879 >> 172.21.15.98:0/158381879 conn(0x7f3a140a73b0 msgr2=0x7f3a140a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/158381879 shutdown_connections 2024-06-11T19:13:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 1 -- 172.21.15.98:0/158381879 wait complete. 2024-06-11T19:13:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.659+0000 7f3a1b3f0640 1 librados: shutdown 2024-06-11T19:13:37.707 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:37.707 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:13:37.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:13:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.808+0000 7fc2f0d84640 10 monclient: get_monmap_and_config 2024-06-11T19:13:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.808+0000 7fc2f0d84640 10 monclient: build_initial_monmap 2024-06-11T19:13:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.808+0000 7fc2f0d84640 10 monclient: monmap: 2024-06-11T19:13:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:37.809463+0000 2024-06-11T19:13:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:37.809463+0000 2024-06-11T19:13:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:37.809 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 1 Processor -- start 2024-06-11T19:13:37.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 1 -- start start 2024-06-11T19:13:37.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 10 monclient: init 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 10 monclient: _add_conns ranks=[1,3,9,10,6,5,2,8,4,7,0] 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc2ec0a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc2ec0a2530 0x7fc2ec13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 10 monclient(hunting): picked mon.noname-g con 0x7fc2ec0a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2ec13e1a0 0x7fc2ec140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 10 monclient(hunting): picked mon.noname-a con 0x7fc2ec13e1a0 addr v2:172.21.15.98:3300/0 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2ec0a7140 con 0x7fc2ec0a4920 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc2ec0a2b50 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fc2ec0a2e30 con 0x7fc2ec0a2530 2024-06-11T19:13:37.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2f0d84640 10 monclient(hunting): authenticate will time out at 1027.616758s 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2ead9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2ec13e1a0 0x7fc2ec140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2ead9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2ec13e1a0 0x7fc2ec140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49908/0 (socket says 172.21.15.98:49908) 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2ead9d640 1 -- 172.21.15.98:0/4294712053 learned_addr learned my addr 172.21.15.98:0/4294712053 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2ead9d640 10 monclient(hunting): get_auth_request con 0x7fc2ec13e1a0 auth_method 0 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2ead9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.809+0000 7fc2ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2e9d9b640 1 --2- 172.21.15.98:0/4294712053 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc2ec0a2530 0x7fc2ec13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2e9d9b640 10 monclient(hunting): get_auth_request con 0x7fc2ec0a2530 auth_method 0 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2e9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2ead9d640 10 monclient(hunting): handle_auth_done global_id 64233 payload 274 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2ead9d640 1 -- 172.21.15.98:0/4294712053 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc2ec0a2530 msgr2=0x7fc2ec13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2ead9d640 1 --2- 172.21.15.98:0/4294712053 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc2ec0a2530 0x7fc2ec13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2ead9d640 1 -- 172.21.15.98:0/4294712053 >> v1:172.21.15.117:6789/0 conn(0x7fc2ec0a4920 legacy=0x7fc2ec0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2ead9d640 10 monclient: _finish_hunting 0 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2ead9d640 1 monclient: found mon.noname-a 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2ead9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2ead9d640 1 -- 172.21.15.98:0/4294712053 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2ec0a3110 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.810+0000 7fc2ead9d640 1 --2- 172.21.15.98:0/4294712053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2ec13e1a0 0x7fc2ec140590 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009e50 tx=0x7fc2e402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67b02c371fc4381 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 1 -- 172.21.15.98:0/4294712053 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2e402cbc0 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:13:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 10 monclient: dump: 2024-06-11T19:13:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.825 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.825 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.825 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.825 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.825 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.825 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.825 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.825 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.825 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 10 monclient: _finish_auth 0 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 10 monclient: _check_auth_tickets 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 1 -- 172.21.15.98:0/4294712053 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e402cd80 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 1 -- 172.21.15.98:0/4294712053 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2e4034860 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 5 monclient: authenticate success, global_id 64233 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 4 set_mon_vals no callback set 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2e959a640 10 monclient: dump: 2024-06-11T19:13:37.826 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 10 monclient: shutdown 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 1 -- 172.21.15.98:0/4294712053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2ec13e1a0 msgr2=0x7fc2ec140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 1 --2- 172.21.15.98:0/4294712053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2ec13e1a0 0x7fc2ec140590 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009e50 tx=0x7fc2e402bdb0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 1 -- 172.21.15.98:0/4294712053 shutdown_connections 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 1 --2- 172.21.15.98:0/4294712053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2ec13e1a0 0x7fc2ec140590 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 1 --2- 172.21.15.98:0/4294712053 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc2ec0a2530 0x7fc2ec13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 1 -- 172.21.15.98:0/4294712053 >> 172.21.15.98:0/4294712053 conn(0x7fc2ec0ab850 msgr2=0x7fc2ec0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 1 -- 172.21.15.98:0/4294712053 shutdown_connections 2024-06-11T19:13:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.811+0000 7fc2f0d84640 1 -- 172.21.15.98:0/4294712053 wait complete. 2024-06-11T19:13:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.812+0000 7fc2f0d84640 10 monclient: build_initial_monmap 2024-06-11T19:13:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.812+0000 7fc2f0d84640 10 monclient: monmap: 2024-06-11T19:13:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.812+0000 7fc2f0d84640 1 librados: starting msgr at 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.812+0000 7fc2f0d84640 1 librados: starting objecter 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.812+0000 7fc2f0d84640 1 Processor -- start 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.812+0000 7fc2f0d84640 1 -- start start 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.812+0000 7fc2f0d84640 1 librados: setting wanted keys 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.812+0000 7fc2f0d84640 1 librados: calling monclient init 2024-06-11T19:13:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.812+0000 7fc2f0d84640 10 monclient: init 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 10 monclient: _add_conns ranks=[3,5,4,1,2,0,7,8,6] 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc2ec0a2530 0x7fc2ec133b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 10 monclient(hunting): picked mon.noname-g con 0x7fc2ec0a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec0a4920 0x7fc2ec1340a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 10 monclient(hunting): picked mon.noname-h con 0x7fc2ec0a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2ec13e1a0 0x7fc2ec1cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 10 monclient(hunting): picked mon.noname-c con 0x7fc2ec13e1a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fc2ec138e50 con 0x7fc2ec0a2530 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc2ec0a84b0 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc2ec0a6da0 con 0x7fc2ec0a4920 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 10 monclient(hunting): authenticate will time out at 1027.620200s 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2ec13e1a0 0x7fc2ec1cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2ec13e1a0 0x7fc2ec1cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52356/0 (socket says 172.21.15.98:52356) 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 1 -- 172.21.15.98:0/3922350401 learned_addr learned my addr 172.21.15.98:0/3922350401 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 10 monclient(hunting): get_auth_request con 0x7fc2ec13e1a0 auth_method 0 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ea59c640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc2ec0a2530 0x7fc2ec133b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2e9d9b640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec0a4920 0x7fc2ec1340a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ea59c640 10 monclient(hunting): get_auth_request con 0x7fc2ec0a2530 auth_method 0 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ea59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2e9d9b640 1 -- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec0a4920 msgr2=0x7fc2ec1340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2e9d9b640 1 -- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec0a4920 msgr2=0x7fc2ec1340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2e9d9b640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec0a4920 0x7fc2ec1340a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2e9d9b640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec0a4920 0x7fc2ec1340a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 10 monclient(hunting): handle_auth_done global_id 23494 payload 824 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 1 -- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec0a4920 msgr2=0x7fc2ec1340a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec0a4920 0x7fc2ec1340a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 1 -- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc2ec0a2530 msgr2=0x7fc2ec133b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc2ec0a2530 0x7fc2ec133b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 10 monclient: _finish_hunting 0 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 1 monclient: found mon.noname-c 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 1 -- 172.21.15.98:0/3922350401 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2ec0a0880 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2ead9d640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2ec13e1a0 0x7fc2ec1cd740 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009f70 tx=0x7fc2e4004a40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 10 client.?.objecter ms_handle_connect 0x7fc2ec13e1a0 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2e4042030 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 10 monclient: dump: 2024-06-11T19:13:37.835 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.835 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.835 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.835 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.835 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.835 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.835 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.835 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.835 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.835 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.836 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.836 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.836 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.836 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.836 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.836 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 10 monclient: _finish_auth 0 2024-06-11T19:13:37.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 10 monclient: _check_auth_tickets 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e40046c0 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 5 monclient: authenticate success, global_id 23494 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2e4009510 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 10 monclient: _renew_subs 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2f0d84640 1 -- 172.21.15.98:0/3922350401 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2ec13df90 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2daffd640 10 monclient: dump: 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.837 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.813+0000 7fc2e8d99640 4 set_mon_vals no callback set 2024-06-11T19:13:37.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.814+0000 7fc2f0d84640 10 client.23494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.814+0000 7fc2f0d84640 10 monclient: _renew_subs 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.814+0000 7fc2f0d84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.814+0000 7fc2f0d84640 1 -- 172.21.15.98:0/3922350401 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2ec0a05a0 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.814+0000 7fc2f0d84640 1 librados: init done 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.814+0000 7fc2f0d84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.814+0000 7fc2f0d84640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec1cdc80 0x7fc2ec1ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.814+0000 7fc2f0d84640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.814+0000 7fc2f0d84640 1 -- 172.21.15.98:0/3922350401 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc2ec09f200 con 0x7fc2ec1cdc80 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.814+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fc2e4045070 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc2cc03fdb0 0x7fc2cc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(182..182 src has 1..182) ==== 6628+0+0 (secure 0 0 0) 0x7fc2e4043070 con 0x7fc2ec13e1a0 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2ea59c640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec1cdc80 0x7fc2ec1ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 10 client.23494.objecter ms_dispatch 0x7fc2ec000b90 osd_map(182..182 src has 1..182) 2024-06-11T19:13:37.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 3 client.23494.objecter handle_osd_map got epochs [182,182] > 0 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 3 client.23494.objecter handle_osd_map decoding full epoch 182 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2ea59c640 10 monclient: get_auth_request con 0x7fc2ec1cdc80 auth_method 0 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2ea59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 20 client.23494.objecter dump_active .. 0 homeless 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2e9d9b640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc2cc03fdb0 0x7fc2cc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2e9d9b640 10 monclient: get_auth_request con 0x7fc2cc03fdb0 auth_method 0 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2ea59c640 10 monclient(hunting): handle_auth_done global_id 22514 payload 824 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2ea59c640 1 -- 172.21.15.98:0/3922350401 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2ec0a4240 con 0x7fc2ec1cdc80 2024-06-11T19:13:37.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 10 client.23494.objecter ms_handle_connect 0x7fc2ec1cdc80 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 10 client.23494.objecter resend_mon_ops 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2ea59c640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec1cdc80 0x7fc2ec1ce060 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00e4d0 tx=0x7fc2dc00e990 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2e9d9b640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc2cc03fdb0 0x7fc2cc042270 secure :-1 s=READY pgs=4085 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec09e350 tx=0x7fc2e0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 10 client.23494.objecter ms_handle_connect 0x7fc2cc03fdb0 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2dc0067d0 con 0x7fc2ec1cdc80 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.815+0000 7fc2daffd640 10 monclient: dump: 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.841 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.816+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc2dc007240 con 0x7fc2ec1cdc80 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.816+0000 7fc2daffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.816+0000 7fc2daffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:37.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.816+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec1cdc80 msgr2=0x7fc2ec1ce060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.816+0000 7fc2daffd640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec1cdc80 0x7fc2ec1ce060 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00e4d0 tx=0x7fc2dc00e990 comp rx=0 tx=0).stop 2024-06-11T19:13:37.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.816+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec1cdc80 msgr2=0x7fc2ec1ce060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:37.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.816+0000 7fc2daffd640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec1cdc80 0x7fc2ec1ce060 secure :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00e4d0 tx=0x7fc2dc00e990 comp rx=0 tx=0).stop 2024-06-11T19:13:37.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.822+0000 7fc2f0d84640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:37.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.822+0000 7fc2f0d84640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec139cd0 0x7fc2ec13a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.822+0000 7fc2f0d84640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.822+0000 7fc2f0d84640 1 -- 172.21.15.98:0/3922350401 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc2ec050830 con 0x7fc2ec139cd0 2024-06-11T19:13:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.822+0000 7fc2ea59c640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec139cd0 0x7fc2ec13a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.822+0000 7fc2ea59c640 10 monclient: get_auth_request con 0x7fc2ec139cd0 auth_method 0 2024-06-11T19:13:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.822+0000 7fc2ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.822+0000 7fc2ea59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.822+0000 7fc2ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.822+0000 7fc2ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.822+0000 7fc2ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.822+0000 7fc2ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2ea59c640 10 monclient(hunting): handle_auth_done global_id 22523 payload 824 2024-06-11T19:13:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2ea59c640 1 -- 172.21.15.98:0/3922350401 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc2ec1ce5a0 con 0x7fc2ec139cd0 2024-06-11T19:13:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2ea59c640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec139cd0 0x7fc2ec13a0e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00e4d0 tx=0x7fc2dc003760 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 10 client.23494.objecter ms_handle_connect 0x7fc2ec139cd0 2024-06-11T19:13:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 10 client.23494.objecter resend_mon_ops 2024-06-11T19:13:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2dc01ebd0 con 0x7fc2ec139cd0 2024-06-11T19:13:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:13:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 10 monclient: dump: 2024-06-11T19:13:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.847 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.848 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fc2dc01e760 con 0x7fc2ec139cd0 2024-06-11T19:13:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec139cd0 msgr2=0x7fc2ec13a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec139cd0 0x7fc2ec13a0e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00e4d0 tx=0x7fc2dc003760 comp rx=0 tx=0).stop 2024-06-11T19:13:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 1 -- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec139cd0 msgr2=0x7fc2ec13a0e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:37.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2daffd640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2ec139cd0 0x7fc2ec13a0e0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.848 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:37.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 268017, 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:37.849 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 9, 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.850 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.851 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:37.852 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:37.853 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:37.854 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:37.854 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:37.854 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:37.854 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:37.854 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.854 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:37.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.854 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:37.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.854 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:37.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:37.855 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.855 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.855 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:37.855 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.855 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:37.856 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.856 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.856 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:37.856 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:37.856 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.856 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.856 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.857 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:37.857 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:37.857 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.857 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.857 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:37.857 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.859 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:37.859 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:37.859 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:37.859 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:37.860 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:37.863 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.866 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:37.866 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:37.866 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:37.866 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:37.867 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.867 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.867 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:37.867 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:37.867 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.867 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.867 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.868 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:37.868 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:37.868 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:37.868 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:37.870 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.871 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:37.872 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:13:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:37.873 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2f0d84640 10 librados: watch_flush enter 2024-06-11T19:13:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2f0d84640 10 librados: watch_flush exit 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2f0d84640 20 client.23494.objecter shutdown clearing up homeless session... 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.823+0000 7fc2f0d84640 10 client.23494.objecter successfully canceled tick 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2f0d84640 1 -- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc2cc03fdb0 msgr2=0x7fc2cc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2f0d84640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc2cc03fdb0 0x7fc2cc042270 secure :-1 s=READY pgs=4085 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec09e350 tx=0x7fc2e0008040 comp rx=0 tx=0).stop 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2f0d84640 10 monclient: shutdown 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2f0d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2f0d84640 1 -- 172.21.15.98:0/3922350401 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2ec13e1a0 msgr2=0x7fc2ec1cd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2ead9d640 1 -- 172.21.15.98:0/3922350401 reap_dead start 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2f0d84640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2ec13e1a0 0x7fc2ec1cd740 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009f70 tx=0x7fc2e4004a40 comp rx=0 tx=0).stop 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2f0d84640 1 -- 172.21.15.98:0/3922350401 shutdown_connections 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2f0d84640 1 --2- 172.21.15.98:0/3922350401 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc2ec13e1a0 0x7fc2ec1cd740 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2f0d84640 1 -- 172.21.15.98:0/3922350401 >> 172.21.15.98:0/3922350401 conn(0x7fc2ec0ab850 msgr2=0x7fc2ec137c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2f0d84640 1 -- 172.21.15.98:0/3922350401 shutdown_connections 2024-06-11T19:13:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2f0d84640 1 -- 172.21.15.98:0/3922350401 wait complete. 2024-06-11T19:13:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.824+0000 7fc2f0d84640 1 librados: shutdown 2024-06-11T19:13:37.876 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:37.876 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:13:37.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:13:37.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.977+0000 7f1643c69640 10 monclient: get_monmap_and_config 2024-06-11T19:13:37.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.977+0000 7f1643c69640 10 monclient: build_initial_monmap 2024-06-11T19:13:37.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.977+0000 7f1643c69640 10 monclient: monmap: 2024-06-11T19:13:37.976 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:37.978136+0000 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:37.978136+0000 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:37.978 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.977+0000 7f1643c69640 1 Processor -- start 2024-06-11T19:13:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.977+0000 7f1643c69640 1 -- start start 2024-06-11T19:13:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.977+0000 7f1643c69640 10 monclient: init 2024-06-11T19:13:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 10 monclient: _add_conns ranks=[10,9,0,6,5,8,2,4,7,3,1] 2024-06-11T19:13:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f163c0a0d10 0x7f163c0a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 10 monclient(hunting): picked mon.noname-f con 0x7f163c0a0d10 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f163c138c80 0x7f163c139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 10 monclient(hunting): picked mon.noname-a con 0x7f163c138c80 addr v2:172.21.15.98:3300/0 2024-06-11T19:13:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f163c0a22d0 addr v1:172.21.15.98:6789/0 2024-06-11T19:13:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f163c076a00 con 0x7f163c0a22d0 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f163c09f010 con 0x7f163c138c80 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f163c09ed30 con 0x7f163c0a0d10 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1643c69640 10 monclient(hunting): authenticate will time out at 1027.785442s 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1642206640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f163c0a22d0 0x7f163c0a26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50144/0 (socket says 172.21.15.98:50144) 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1642206640 1 -- 172.21.15.98:0/2730734494 learned_addr learned my addr 172.21.15.98:0/2730734494 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641204640 1 --2- 172.21.15.98:0/2730734494 >> v2:172.21.15.98:3300/0 conn(0x7f163c138c80 0x7f163c139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641204640 10 monclient(hunting): get_auth_request con 0x7f163c138c80 auth_method 0 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641204640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641204640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641204640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641a05640 1 --2- 172.21.15.98:0/2730734494 >> v2:172.21.15.117:3300/0 conn(0x7f163c0a0d10 0x7f163c0a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641a05640 10 monclient(hunting): get_auth_request con 0x7f163c0a0d10 auth_method 0 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641a05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641a05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641a05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641204640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641204640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641204640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1640a03640 1 -- 172.21.15.98:0/2730734494 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 597354989 0 0) 0x7f163c076a00 con 0x7f163c0a22d0 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1640a03640 10 monclient(hunting): _negotiate 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1640a03640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1640a03640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:37.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1640a03640 10 monclient(hunting): my global_id is 64251 2024-06-11T19:13:37.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1640a03640 1 -- 172.21.15.98:0/2730734494 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1628003520 con 0x7f163c0a22d0 2024-06-11T19:13:37.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641a05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:37.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641a05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:37.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641a05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:37.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641204640 10 monclient(hunting): handle_auth_done global_id 64242 payload 274 2024-06-11T19:13:37.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.978+0000 7f1641204640 1 -- 172.21.15.98:0/2730734494 >> v2:172.21.15.117:3300/0 conn(0x7f163c0a0d10 msgr2=0x7f163c0a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:37.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1641204640 1 --2- 172.21.15.98:0/2730734494 >> v2:172.21.15.117:3300/0 conn(0x7f163c0a0d10 0x7f163c0a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1641204640 1 -- 172.21.15.98:0/2730734494 >> v1:172.21.15.98:6789/0 conn(0x7f163c0a22d0 legacy=0x7f163c0a26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1641204640 10 monclient: _finish_hunting 0 2024-06-11T19:13:37.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1641204640 1 monclient: found mon.noname-a 2024-06-11T19:13:37.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1641204640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:37.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1641204640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:13:37.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1641204640 1 -- 172.21.15.98:0/2730734494 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163c0a1630 con 0x7f163c138c80 2024-06-11T19:13:37.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1641204640 1 --2- 172.21.15.98:0/2730734494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f163c138c80 0x7f163c139060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f163c1399f0 tx=0x7f162c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fc6d69a6f85b0a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:37.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 1 -- 172.21.15.98:0/2730734494 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f162c038070 con 0x7f163c138c80 2024-06-11T19:13:37.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:13:37.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 10 monclient: dump: 2024-06-11T19:13:37.993 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.993 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.993 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.994 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 10 monclient: _finish_auth 0 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 10 monclient: _check_auth_tickets 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 1 -- 172.21.15.98:0/2730734494 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f162c031970 con 0x7f163c138c80 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1643c69640 5 monclient: authenticate success, global_id 64242 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1643c69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1643c69640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 1 -- 172.21.15.98:0/2730734494 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f162c032390 con 0x7f163c138c80 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1643c69640 4 set_mon_vals no callback set 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.979+0000 7f1640a03640 10 monclient: dump: 2024-06-11T19:13:37.995 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.980+0000 7f1643c69640 10 monclient: shutdown 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.980+0000 7f1643c69640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.980+0000 7f1643c69640 1 -- 172.21.15.98:0/2730734494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f163c138c80 msgr2=0x7f163c139060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.980+0000 7f1643c69640 1 --2- 172.21.15.98:0/2730734494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f163c138c80 0x7f163c139060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f163c1399f0 tx=0x7f162c02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.980+0000 7f1643c69640 1 -- 172.21.15.98:0/2730734494 shutdown_connections 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.980+0000 7f1643c69640 1 --2- 172.21.15.98:0/2730734494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f163c138c80 0x7f163c139060 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.980+0000 7f1643c69640 1 --2- 172.21.15.98:0/2730734494 >> v2:172.21.15.117:3300/0 conn(0x7f163c0a0d10 0x7f163c0a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.980+0000 7f1643c69640 1 -- 172.21.15.98:0/2730734494 >> 172.21.15.98:0/2730734494 conn(0x7f163c0a7850 msgr2=0x7f163c0a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.980+0000 7f1643c69640 1 -- 172.21.15.98:0/2730734494 shutdown_connections 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.980+0000 7f1643c69640 1 -- 172.21.15.98:0/2730734494 wait complete. 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient: build_initial_monmap 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient: monmap: 2024-06-11T19:13:37.997 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:37.998 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:37.999 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 1 librados: starting msgr at 2024-06-11T19:13:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 1 librados: starting objecter 2024-06-11T19:13:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 1 Processor -- start 2024-06-11T19:13:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 1 -- start start 2024-06-11T19:13:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 1 librados: setting wanted keys 2024-06-11T19:13:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 1 librados: calling monclient init 2024-06-11T19:13:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient: init 2024-06-11T19:13:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:37.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient: _add_conns ranks=[4,8,7,2,3,5,1,0,6] 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f163c0a0d10 0x7f163c1cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient(hunting): picked mon.noname-c con 0x7f163c0a0d10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f163c0a22d0 0x7f163c1ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient(hunting): picked mon.noname-e con 0x7f163c0a22d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c138c80 0x7f163c1302c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient(hunting): picked mon.noname-i con 0x7f163c138c80 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f163c0a2dc0 con 0x7f163c0a0d10 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f163c0a3d70 con 0x7f163c138c80 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f163c0a2f40 con 0x7f163c0a22d0 2024-06-11T19:13:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.981+0000 7f1643c69640 10 monclient(hunting): authenticate will time out at 1027.788934s 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f163c0a0d10 0x7f163c1cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641204640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f163c0a22d0 0x7f163c1ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f163c0a0d10 0x7f163c1cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52362/0 (socket says 172.21.15.98:52362) 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 1 -- 172.21.15.98:0/1362211371 learned_addr learned my addr 172.21.15.98:0/1362211371 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 10 monclient(hunting): get_auth_request con 0x7f163c0a0d10 auth_method 0 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641204640 10 monclient(hunting): get_auth_request con 0x7f163c0a22d0 auth_method 0 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641204640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641204640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641204640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641204640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641204640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641204640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1642206640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c138c80 0x7f163c1302c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1642206640 10 monclient(hunting): get_auth_request con 0x7f163c138c80 auth_method 0 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1642206640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1642206640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1642206640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1642206640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1642206640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1642206640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 10 monclient(hunting): handle_auth_done global_id 23503 payload 824 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 1 -- 172.21.15.98:0/1362211371 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f163c0a22d0 msgr2=0x7f163c1ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f163c0a22d0 0x7f163c1ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 1 -- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c138c80 msgr2=0x7f163c1302c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c138c80 0x7f163c1302c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 10 monclient: _finish_hunting 0 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 1 monclient: found mon.noname-c 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 1 -- 172.21.15.98:0/1362211371 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163c133880 con 0x7f163c0a0d10 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641204640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f163c0a22d0 0x7f163c1ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1641a05640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f163c0a0d10 0x7f163c1cdc60 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f163800e9b0 tx=0x7f163800ee70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:38.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 10 client.?.objecter ms_handle_connect 0x7f163c0a0d10 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1638017070 con 0x7f163c0a0d10 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 10 monclient: dump: 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 10 monclient: _finish_auth 0 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 10 monclient: _check_auth_tickets 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f163800ce10 con 0x7f163c0a0d10 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1643c69640 5 monclient: authenticate success, global_id 23503 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1643c69640 10 monclient: _renew_subs 2024-06-11T19:13:38.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1643c69640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1643c69640 1 -- 172.21.15.98:0/1362211371 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f163c133b60 con 0x7f163c0a0d10 2024-06-11T19:13:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16380138e0 con 0x7f163c0a0d10 2024-06-11T19:13:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f16327fc640 10 monclient: dump: 2024-06-11T19:13:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:38.006 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1633fff640 4 set_mon_vals no callback set 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1643c69640 10 client.23503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1643c69640 10 monclient: _renew_subs 2024-06-11T19:13:38.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1643c69640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1643c69640 1 -- 172.21.15.98:0/1362211371 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f163c09e360 con 0x7f163c0a0d10 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.982+0000 7f1643c69640 1 librados: init done 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f163801a070 con 0x7f163c0a0d10 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f1643c69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f1643c69640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c130800 0x7f163c130be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f1643c69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f1643c69640 1 -- 172.21.15.98:0/1362211371 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f163c050830 con 0x7f163c130800 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f16327fc640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f162003fdb0 0x7f1620042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(183..183 src has 1..183) ==== 7369+0+0 (secure 0 0 0) 0x7f163800f040 con 0x7f163c0a0d10 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f16327fc640 10 client.23503.objecter ms_dispatch 0x7f163c000b90 osd_map(183..183 src has 1..183) 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f16327fc640 3 client.23503.objecter handle_osd_map got epochs [183,183] > 0 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f16327fc640 3 client.23503.objecter handle_osd_map decoding full epoch 183 2024-06-11T19:13:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f1641204640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c130800 0x7f163c130be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:38.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f1641204640 10 monclient: get_auth_request con 0x7f163c130800 auth_method 0 2024-06-11T19:13:38.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f1641204640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:38.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f1641204640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:38.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.983+0000 7f1641204640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:38.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f1642206640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f162003fdb0 0x7f1620042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:38.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f1641204640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:38.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f1641204640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:38.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f1641204640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:38.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f1642206640 10 monclient: get_auth_request con 0x7f162003fdb0 auth_method 0 2024-06-11T19:13:38.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 20 client.23503.objecter dump_active .. 0 homeless 2024-06-11T19:13:38.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f1641204640 10 monclient(hunting): handle_auth_done global_id 22606 payload 824 2024-06-11T19:13:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f1641204640 1 -- 172.21.15.98:0/1362211371 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f163c0a5600 con 0x7f163c130800 2024-06-11T19:13:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 10 client.23503.objecter ms_handle_connect 0x7f163c130800 2024-06-11T19:13:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 10 client.23503.objecter resend_mon_ops 2024-06-11T19:13:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f1641204640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c130800 0x7f163c130be0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f163c0a3300 tx=0x7f162c0046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f1642206640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f162003fdb0 0x7f1620042270 secure :-1 s=READY pgs=4086 cs=0 l=1 rev1=1 crypto rx=0x7f163c1319e0 tx=0x7f1634009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 10 client.23503.objecter ms_handle_connect 0x7f162003fdb0 2024-06-11T19:13:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f162c044070 con 0x7f163c130800 2024-06-11T19:13:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:13:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 10 monclient: dump: 2024-06-11T19:13:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f162c039040 con 0x7f163c130800 2024-06-11T19:13:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c130800 msgr2=0x7f163c130be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c130800 0x7f163c130be0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f163c0a3300 tx=0x7f162c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c130800 msgr2=0x7f163c130be0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:38.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.984+0000 7f16327fc640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c130800 0x7f163c130be0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1643c69640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:38.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1643c69640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c131120 0x7f163c12c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1643c69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1643c69640 1 -- 172.21.15.98:0/1362211371 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f163c0a10f0 con 0x7f163c131120 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1641204640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c131120 0x7f163c12c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1641204640 10 monclient: get_auth_request con 0x7f163c131120 auth_method 0 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1641204640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1641204640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1641204640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1641204640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1641204640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1641204640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1641204640 10 monclient(hunting): handle_auth_done global_id 22615 payload 824 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.991+0000 7f1641204640 1 -- 172.21.15.98:0/1362211371 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f163c073920 con 0x7f163c131120 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1641204640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c131120 0x7f163c12c4f0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f162c000c00 tx=0x7f162c031000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:38.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 10 client.23503.objecter ms_handle_connect 0x7f163c131120 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 10 client.23503.objecter resend_mon_ops 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f162c038040 con 0x7f163c131120 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 10 monclient: dump: 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:38.016 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7f162c0498f0 con 0x7f163c131120 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c131120 msgr2=0x7f163c12c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c131120 0x7f163c12c4f0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f162c000c00 tx=0x7f162c031000 comp rx=0 tx=0).stop 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 1 -- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c131120 msgr2=0x7f163c12c4f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f16327fc640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f163c131120 0x7f163c12c4f0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.018 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:38.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:38.018 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:38.018 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:38.018 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 20, 2024-06-11T19:13:38.018 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 268208, 2024-06-11T19:13:38.018 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:38.018 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:38.018 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:38.018 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 9, 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:38.019 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:38.020 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:38.020 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:38.020 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:38.020 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:38.020 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:38.020 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:38.020 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:38.020 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:38.020 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:38.020 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:38.020 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.021 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:38.022 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:38.023 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.026 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:38.026 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:38.026 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:38.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:38.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.026 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:38.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:38.026 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.026 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:38.027 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:38.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:38.029 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:38.029 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:38.029 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:38.029 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:38.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.029 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:38.029 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:38.030 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:38.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:38.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:38.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:38.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:38.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:38.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:38.034 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:38.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:38.036 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:38.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:38.039 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:38.040 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.040 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:38.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.040 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:38.040 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:38.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.040 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:38.040 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:38.040 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:38.040 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.040 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:38.040 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.041 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:38.042 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:13:38.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:38.042 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:38.042 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:38.042 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:38.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:38.042 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:38.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:38.042 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:38.042 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:38.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 10 librados: watch_flush enter 2024-06-11T19:13:38.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 10 librados: watch_flush exit 2024-06-11T19:13:38.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 20 client.23503.objecter shutdown clearing up homeless session... 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 10 client.23503.objecter successfully canceled tick 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 1 -- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f162003fdb0 msgr2=0x7f1620042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f162003fdb0 0x7f1620042270 secure :-1 s=READY pgs=4086 cs=0 l=1 rev1=1 crypto rx=0x7f163c1319e0 tx=0x7f1634009040 comp rx=0 tx=0).stop 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 10 monclient: shutdown 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 1 -- 172.21.15.98:0/1362211371 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f163c0a0d10 msgr2=0x7f163c1cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 1 --2- 172.21.15.98:0/1362211371 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f163c0a0d10 0x7f163c1cdc60 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f163800e9b0 tx=0x7f163800ee70 comp rx=0 tx=0).stop 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1642206640 1 -- 172.21.15.98:0/1362211371 reap_dead start 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1642206640 1 -- 172.21.15.98:0/1362211371 reap_dead start 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 1 -- 172.21.15.98:0/1362211371 shutdown_connections 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 1 -- 172.21.15.98:0/1362211371 >> 172.21.15.98:0/1362211371 conn(0x7f163c0a7850 msgr2=0x7f163c09d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 1 -- 172.21.15.98:0/1362211371 shutdown_connections 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 1 -- 172.21.15.98:0/1362211371 wait complete. 2024-06-11T19:13:38.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:37.992+0000 7f1643c69640 1 librados: shutdown 2024-06-11T19:13:38.046 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:13:38.046 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b', 'g', 'e', 'i'] 2024-06-11T19:13:38.046 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-06-11T19:13:38.047 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-06-11T19:13:38.047 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-06-11T19:13:38.047 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-06-11T19:13:38.047 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:13:38.062 INFO:tasks.ceph.mon.b:Stopped 2024-06-11T19:13:38.062 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-06-11T19:13:38.062 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-06-11T19:13:38.063 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-06-11T19:13:38.063 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:13:38.077 INFO:tasks.ceph.mon.g:Stopped 2024-06-11T19:13:38.077 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-06-11T19:13:38.077 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-06-11T19:13:38.077 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-06-11T19:13:38.077 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:13:38.114 INFO:tasks.ceph.mon.e:Stopped 2024-06-11T19:13:38.114 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-06-11T19:13:38.114 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-06-11T19:13:38.114 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-06-11T19:13:38.114 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:13:38.133 INFO:tasks.ceph.mon.i:Stopped 2024-06-11T19:13:38.133 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-06-11T19:13:38.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:13:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.251+0000 7fc91d069640 10 monclient: get_monmap_and_config 2024-06-11T19:13:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.251+0000 7fc91d069640 10 monclient: build_initial_monmap 2024-06-11T19:13:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.251+0000 7fc91d069640 10 monclient: monmap: 2024-06-11T19:13:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:38.252251+0000 2024-06-11T19:13:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:38.252251+0000 2024-06-11T19:13:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.251+0000 7fc91d069640 1 Processor -- start 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.251+0000 7fc91d069640 1 -- start start 2024-06-11T19:13:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.251+0000 7fc91d069640 10 monclient: init 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 10 monclient: _add_conns ranks=[4,10,9,3,2,1,8,5,0,7,6] 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc918136c90 0x7fc918137070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 10 monclient(hunting): picked mon.noname-c con 0x7fc918136c90 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc9181367d0 0x7fc9180a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 10 monclient(hunting): picked mon.noname-f con 0x7fc9181367d0 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc9180a3bd0 0x7fc9180a4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 10 monclient(hunting): picked mon.noname-a con 0x7fc9180a3bd0 addr v2:172.21.15.98:3300/0 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc918068180 con 0x7fc9180a3bd0 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fc9180a0270 con 0x7fc9181367d0 2024-06-11T19:13:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc91809fc20 con 0x7fc918136c90 2024-06-11T19:13:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91d069640 10 monclient(hunting): authenticate will time out at 1028.059415s 2024-06-11T19:13:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc9180a3bd0 0x7fc9180a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc916d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc918136c90 0x7fc918137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc9180a3bd0 0x7fc9180a4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49938/0 (socket says 172.21.15.98:49938) 2024-06-11T19:13:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc916d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc918136c90 0x7fc918137070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52376/0 (socket says 172.21.15.98:52376) 2024-06-11T19:13:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 1 -- 172.21.15.98:0/3323792948 learned_addr learned my addr 172.21.15.98:0/3323792948 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 10 monclient(hunting): get_auth_request con 0x7fc9180a3bd0 auth_method 0 2024-06-11T19:13:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc916d9d640 10 monclient(hunting): get_auth_request con 0x7fc918136c90 auth_method 0 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc916d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc916d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc916d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc916d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc916d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc916d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 10 monclient(hunting): handle_auth_done global_id 64260 payload 274 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 1 -- 172.21.15.98:0/3323792948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc918136c90 msgr2=0x7fc918137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 1 --2- 172.21.15.98:0/3323792948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc918136c90 0x7fc918137070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 1 -- 172.21.15.98:0/3323792948 >> v2:172.21.15.117:3300/0 conn(0x7fc9181367d0 msgr2=0x7fc9180a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91659c640 1 --2- 172.21.15.98:0/3323792948 >> v2:172.21.15.117:3300/0 conn(0x7fc9181367d0 0x7fc9180a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 1 --2- 172.21.15.98:0/3323792948 >> v2:172.21.15.117:3300/0 conn(0x7fc9181367d0 0x7fc9180a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 10 monclient: _finish_hunting 0 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 1 monclient: found mon.noname-a 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91759e640 1 -- 172.21.15.98:0/3323792948 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc91809ff00 con 0x7fc9180a3bd0 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91659c640 10 monclient: get_auth_request con 0x7fc9181367d0 auth_method 0 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc916d9d640 1 --2- 172.21.15.98:0/3323792948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc918136c90 0x7fc918137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.252+0000 7fc91659c640 1 --2- 172.21.15.98:0/3323792948 >> v2:172.21.15.117:3300/0 conn(0x7fc9181367d0 0x7fc9180a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc91759e640 1 --2- 172.21.15.98:0/3323792948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9180a3bd0 0x7fc9180a4060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc9180a5570 tx=0x7fc90802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806990b6a3938784 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 1 -- 172.21.15.98:0/3323792948 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc90802e070 con 0x7fc9180a3bd0 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 10 monclient: dump: 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 10 monclient: _finish_auth 0 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc91d069640 5 monclient: authenticate success, global_id 64260 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 1 -- 172.21.15.98:0/3323792948 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc908037070 con 0x7fc9180a3bd0 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc91d069640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc91d069640 4 set_mon_vals no callback set 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 1 -- 172.21.15.98:0/3323792948 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc908032470 con 0x7fc9180a3bd0 2024-06-11T19:13:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.253+0000 7fc915d9b640 10 monclient: dump: 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.254+0000 7fc91d069640 10 monclient: shutdown 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.254+0000 7fc91d069640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.254+0000 7fc91d069640 1 -- 172.21.15.98:0/3323792948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9180a3bd0 msgr2=0x7fc9180a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.254+0000 7fc91d069640 1 --2- 172.21.15.98:0/3323792948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9180a3bd0 0x7fc9180a4060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc9180a5570 tx=0x7fc90802b880 comp rx=0 tx=0).stop 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.254+0000 7fc91d069640 1 -- 172.21.15.98:0/3323792948 shutdown_connections 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.254+0000 7fc91d069640 1 --2- 172.21.15.98:0/3323792948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9180a3bd0 0x7fc9180a4060 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.254+0000 7fc91d069640 1 --2- 172.21.15.98:0/3323792948 >> v2:172.21.15.117:3300/0 conn(0x7fc9181367d0 0x7fc9180a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.254+0000 7fc91d069640 1 --2- 172.21.15.98:0/3323792948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc918136c90 0x7fc918137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.254+0000 7fc91d069640 1 -- 172.21.15.98:0/3323792948 >> 172.21.15.98:0/3323792948 conn(0x7fc9180a4780 msgr2=0x7fc9180a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.254+0000 7fc91d069640 1 -- 172.21.15.98:0/3323792948 shutdown_connections 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.254+0000 7fc91d069640 1 -- 172.21.15.98:0/3323792948 wait complete. 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 10 monclient: build_initial_monmap 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 10 monclient: monmap: 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 1 librados: starting msgr at 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 1 librados: starting objecter 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 1 Processor -- start 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 1 -- start start 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 1 librados: setting wanted keys 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 1 librados: calling monclient init 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 10 monclient: init 2024-06-11T19:13:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 10 monclient: _add_conns ranks=[4,6,8,0,7,1,5,3,2] 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc9180a3bd0 0x7fc9181cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 10 monclient(hunting): picked mon.noname-c con 0x7fc9180a3bd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9181367d0 0x7fc9181cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 10 monclient(hunting): picked mon.noname-d con 0x7fc9181367d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc918136c90 0x7fc91809c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 10 monclient(hunting): picked mon.noname-e con 0x7fc918136c90 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.255+0000 7fc91d069640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc9181375b0 con 0x7fc9180a3bd0 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc918068560 con 0x7fc9181367d0 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc918128770 con 0x7fc918136c90 2024-06-11T19:13:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 10 monclient(hunting): authenticate will time out at 1028.063016s 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91759e640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc918136c90 msgr2=0x7fc91809c9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91759e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc918136c90 0x7fc91809c9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc9180a3bd0 0x7fc9181cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc9180a3bd0 0x7fc9181cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52392/0 (socket says 172.21.15.98:52392) 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91659c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9181367d0 0x7fc9181cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 1 -- 172.21.15.98:0/2873927438 learned_addr learned my addr 172.21.15.98:0/2873927438 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 10 monclient(hunting): get_auth_request con 0x7fc9180a3bd0 auth_method 0 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91659c640 10 monclient(hunting): get_auth_request con 0x7fc9181367d0 auth_method 0 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 10 monclient(hunting): handle_auth_done global_id 23521 payload 824 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 1 -- 172.21.15.98:0/2873927438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc918136c90 msgr2=0x7fc91809c9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc918136c90 0x7fc91809c9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 1 -- 172.21.15.98:0/2873927438 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9181367d0 msgr2=0x7fc9181cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9181367d0 0x7fc9181cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 1 monclient: found mon.noname-c 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 1 -- 172.21.15.98:0/2873927438 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc918128a50 con 0x7fc9180a3bd0 2024-06-11T19:13:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91659c640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9181367d0 0x7fc9181cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc916d9d640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc9180a3bd0 0x7fc9181cd9b0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fc91000efc0 tx=0x7fc91000c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 10 client.?.objecter ms_handle_connect 0x7fc9180a3bd0 2024-06-11T19:13:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 1 -- 172.21.15.98:0/2873927438 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc910018020 con 0x7fc9180a3bd0 2024-06-11T19:13:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 10 monclient: dump: 2024-06-11T19:13:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:13:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 1 -- 172.21.15.98:0/2873927438 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc91000ce70 con 0x7fc9180a3bd0 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 5 monclient: authenticate success, global_id 23521 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 1 -- 172.21.15.98:0/2873927438 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc910014d60 con 0x7fc9180a3bd0 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 10 monclient: _renew_subs 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 1 -- 172.21.15.98:0/2873927438 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9180a3960 con 0x7fc9180a3bd0 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91559a640 4 set_mon_vals no callback set 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 10 client.23521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc90f7fe640 10 monclient: dump: 2024-06-11T19:13:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 10 monclient: _renew_subs 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 1 -- 172.21.15.98:0/2873927438 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9180a7670 con 0x7fc9180a3bd0 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.256+0000 7fc91d069640 1 librados: init done 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.257+0000 7fc90f7fe640 1 -- 172.21.15.98:0/2873927438 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fc910005d10 con 0x7fc9180a3bd0 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.257+0000 7fc90f7fe640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc8f803fdb0 0x7fc8f8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.257+0000 7fc90f7fe640 1 -- 172.21.15.98:0/2873927438 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(183..183 src has 1..183) ==== 7369+0+0 (secure 0 0 0) 0x7fc910052560 con 0x7fc9180a3bd0 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.257+0000 7fc90f7fe640 10 client.23521.objecter ms_dispatch 0x7fc918000b90 osd_map(183..183 src has 1..183) 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.257+0000 7fc90f7fe640 3 client.23521.objecter handle_osd_map got epochs [183,183] > 0 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.257+0000 7fc90f7fe640 3 client.23521.objecter handle_osd_map decoding full epoch 183 2024-06-11T19:13:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.257+0000 7fc91d069640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.257+0000 7fc91d069640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.257+0000 7fc91d069640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.257+0000 7fc91d069640 1 -- 172.21.15.98:0/2873927438 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc91809f2a0 con 0x7fc9180a3bd0 2024-06-11T19:13:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.257+0000 7fc90f7fe640 20 client.23521.objecter dump_active .. 0 homeless 2024-06-11T19:13:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.258+0000 7fc91659c640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc8f803fdb0 0x7fc8f8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.258+0000 7fc91659c640 10 monclient: get_auth_request con 0x7fc8f803fdb0 auth_method 0 2024-06-11T19:13:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.258+0000 7fc90f7fe640 10 client.23521.objecter ms_handle_connect 0x7fc8f803fdb0 2024-06-11T19:13:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.258+0000 7fc91659c640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc8f803fdb0 0x7fc8f8042270 secure :-1 s=READY pgs=4087 cs=0 l=1 rev1=1 crypto rx=0x7fc91809e450 tx=0x7fc900008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.262+0000 7fc90f7fe640 1 -- 172.21.15.98:0/2873927438 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc910010040 con 0x7fc9180a3bd0 2024-06-11T19:13:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.262+0000 7fc90f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.262+0000 7fc90f7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:38.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.514+0000 7fc91d069640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:13:38.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.514+0000 7fc91d069640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:13:38.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.514+0000 7fc91d069640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:38.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.514+0000 7fc91d069640 1 -- 172.21.15.98:0/2873927438 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc9181275a0 con 0x7fc9180a3bd0 2024-06-11T19:13:38.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.515+0000 7fc90f7fe640 1 -- 172.21.15.98:0/2873927438 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7fc91001ccd0 con 0x7fc9180a3bd0 2024-06-11T19:13:38.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.515+0000 7fc90f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:13:38.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.515+0000 7fc90f7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:38.515 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":20,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:13:38.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.516+0000 7fc91d069640 10 librados: watch_flush enter 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.516+0000 7fc91d069640 10 librados: watch_flush exit 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.516+0000 7fc91d069640 20 client.23521.objecter shutdown clearing up homeless session... 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.516+0000 7fc91d069640 10 client.23521.objecter successfully canceled tick 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.516+0000 7fc91d069640 1 -- 172.21.15.98:0/2873927438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc8f803fdb0 msgr2=0x7fc8f8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc8f803fdb0 0x7fc8f8042270 secure :-1 s=READY pgs=4087 cs=0 l=1 rev1=1 crypto rx=0x7fc91809e450 tx=0x7fc900008040 comp rx=0 tx=0).stop 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 10 monclient: shutdown 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 1 -- 172.21.15.98:0/2873927438 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc9180a3bd0 msgr2=0x7fc9181cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc9180a3bd0 0x7fc9181cd9b0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fc91000efc0 tx=0x7fc91000c490 comp rx=0 tx=0).stop 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 1 -- 172.21.15.98:0/2873927438 shutdown_connections 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc8f803fdb0 0x7fc8f8042270 unknown :-1 s=CLOSED pgs=4087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc918136c90 0x7fc91809c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9181367d0 0x7fc9181cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 1 --2- 172.21.15.98:0/2873927438 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc9180a3bd0 0x7fc9181cd9b0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 1 -- 172.21.15.98:0/2873927438 >> 172.21.15.98:0/2873927438 conn(0x7fc9180a4780 msgr2=0x7fc9180721c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 1 -- 172.21.15.98:0/2873927438 shutdown_connections 2024-06-11T19:13:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 1 -- 172.21.15.98:0/2873927438 wait complete. 2024-06-11T19:13:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:38.517+0000 7fc91d069640 1 librados: shutdown 2024-06-11T19:13:41.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:13:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.636+0000 7f3cf1a71640 10 monclient: get_monmap_and_config 2024-06-11T19:13:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.636+0000 7f3cf1a71640 10 monclient: build_initial_monmap 2024-06-11T19:13:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.636+0000 7f3cf1a71640 10 monclient: monmap: 2024-06-11T19:13:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:41.637937+0000 2024-06-11T19:13:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:41.637937+0000 2024-06-11T19:13:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:41.638 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:41.638 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:41.638 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:41.638 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:41.638 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:41.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.637+0000 7f3cf1a71640 1 Processor -- start 2024-06-11T19:13:41.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 1 -- start start 2024-06-11T19:13:41.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 10 monclient: init 2024-06-11T19:13:41.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 10 monclient: _add_conns ranks=[9,0,4,6,10,2,7,8,1,5,3] 2024-06-11T19:13:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3cec0a4920 0x7f3cec0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 10 monclient(hunting): picked mon.noname-a con 0x7f3cec0a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:13:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3cec0a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:13:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3cec0a2530 0x7f3cec0a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 10 monclient(hunting): picked mon.noname-c con 0x7f3cec0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cec0a7140 con 0x7f3cec0a5240 2024-06-11T19:13:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3cec0a2b50 con 0x7f3cec0a4920 2024-06-11T19:13:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3cec0a2e30 con 0x7f3cec0a2530 2024-06-11T19:13:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cf1a71640 10 monclient(hunting): authenticate will time out at 1031.445599s 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3ceb7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3cec0a4920 0x7f3cec0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3ce3fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3cec0a5240 0x7f3cec0a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50158/0 (socket says 172.21.15.98:50158) 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3ce3fff640 1 -- 172.21.15.98:0/407631153 learned_addr learned my addr 172.21.15.98:0/407631153 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3ceb7fe640 10 monclient(hunting): get_auth_request con 0x7f3cec0a4920 auth_method 0 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3ceb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3ceb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3ceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cebfff640 1 --2- 172.21.15.98:0/407631153 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3cec0a2530 0x7f3cec0a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cebfff640 10 monclient(hunting): get_auth_request con 0x7f3cec0a2530 auth_method 0 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cebfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cebfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3ceb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3ceb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3ceb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.638+0000 7f3cebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceb7fe640 10 monclient(hunting): handle_auth_done global_id 64269 payload 274 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceb7fe640 1 -- 172.21.15.98:0/407631153 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3cec0a2530 msgr2=0x7f3cec0a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceb7fe640 1 --2- 172.21.15.98:0/407631153 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3cec0a2530 0x7f3cec0a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceb7fe640 1 -- 172.21.15.98:0/407631153 >> v1:172.21.15.98:6789/0 conn(0x7f3cec0a5240 legacy=0x7f3cec0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceb7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceb7fe640 1 monclient: found mon.noname-a 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:41.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:13:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceb7fe640 1 -- 172.21.15.98:0/407631153 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cec0a3110 con 0x7f3cec0a4920 2024-06-11T19:13:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3cebfff640 1 --2- 172.21.15.98:0/407631153 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3cec0a2530 0x7f3cec0a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceb7fe640 1 --2- 172.21.15.98:0/407631153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cec0a4920 0x7f3cec0a4d00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc0097a0 tx=0x7f3cdc02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316f72e152014fb1 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceaffd640 1 -- 172.21.15.98:0/407631153 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3cdc038070 con 0x7f3cec0a4920 2024-06-11T19:13:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:13:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.639+0000 7f3ceaffd640 10 monclient: dump: 2024-06-11T19:13:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3ceaffd640 10 monclient: _finish_auth 0 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3ceaffd640 10 monclient: _check_auth_tickets 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3ceaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 5 monclient: authenticate success, global_id 64269 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3ceaffd640 1 -- 172.21.15.98:0/407631153 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cdc031970 con 0x7f3cec0a4920 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3ceaffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 4 set_mon_vals no callback set 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3ceaffd640 1 -- 172.21.15.98:0/407631153 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3cdc031ba0 con 0x7f3cec0a4920 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3ceaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3ceaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3ceaffd640 10 monclient: dump: 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:41.653 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 10 monclient: shutdown 2024-06-11T19:13:41.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 1 -- 172.21.15.98:0/407631153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cec0a4920 msgr2=0x7f3cec0a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 1 --2- 172.21.15.98:0/407631153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cec0a4920 0x7f3cec0a4d00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc0097a0 tx=0x7f3cdc02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 1 -- 172.21.15.98:0/407631153 shutdown_connections 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 1 --2- 172.21.15.98:0/407631153 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3cec0a2530 0x7f3cec0a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 1 --2- 172.21.15.98:0/407631153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cec0a4920 0x7f3cec0a4d00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 1 -- 172.21.15.98:0/407631153 >> 172.21.15.98:0/407631153 conn(0x7f3cec0ab850 msgr2=0x7f3cec0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 1 -- 172.21.15.98:0/407631153 shutdown_connections 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.640+0000 7f3cf1a71640 1 -- 172.21.15.98:0/407631153 wait complete. 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.641+0000 7f3cf1a71640 10 monclient: build_initial_monmap 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.641+0000 7f3cf1a71640 10 monclient: monmap: 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:41.655 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.641+0000 7f3cf1a71640 1 librados: starting msgr at 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.641+0000 7f3cf1a71640 1 librados: starting objecter 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.641+0000 7f3cf1a71640 1 Processor -- start 2024-06-11T19:13:41.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.641+0000 7f3cf1a71640 1 -- start start 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.641+0000 7f3cf1a71640 1 librados: setting wanted keys 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.641+0000 7f3cf1a71640 1 librados: calling monclient init 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.641+0000 7f3cf1a71640 10 monclient: init 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 10 monclient: _add_conns ranks=[8,1,2,3,0,4,6,5,7] 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3cec0a2530 0x7f3cec137560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 10 monclient(hunting): picked mon.noname-e con 0x7f3cec0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cec0a4920 0x7f3cec137aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 10 monclient(hunting): picked mon.noname-f con 0x7f3cec0a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cec0a5240 0x7f3cec123020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 10 monclient(hunting): picked mon.noname-b con 0x7f3cec0a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ceb7fe640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3cec0a2530 msgr2=0x7f3cec137560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3cec0a6da0 con 0x7f3cec0a4920 2024-06-11T19:13:41.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3cec138e50 con 0x7f3cec0a5240 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3cec0a65f0 con 0x7f3cec0a2530 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ceb7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3cec0a2530 0x7f3cec137560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cf1a71640 10 monclient(hunting): authenticate will time out at 1031.449248s 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cebfff640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cec0a5240 msgr2=0x7f3cec123020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3cebfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cec0a5240 0x7f3cec123020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cec0a4920 0x7f3cec137aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cec0a4920 0x7f3cec137aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:40536/0 (socket says 172.21.15.98:40536) 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 1 -- 172.21.15.98:0/928747858 learned_addr learned my addr 172.21.15.98:0/928747858 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 10 monclient(hunting): get_auth_request con 0x7f3cec0a4920 auth_method 0 2024-06-11T19:13:41.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 10 monclient(hunting): handle_auth_done global_id 31582 payload 824 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 1 -- 172.21.15.98:0/928747858 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3cec0a2530 msgr2=0x7f3cec137560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 1 --2- 172.21.15.98:0/928747858 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3cec0a2530 0x7f3cec137560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 1 -- 172.21.15.98:0/928747858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cec0a5240 msgr2=0x7f3cec123020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 1 --2- 172.21.15.98:0/928747858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cec0a5240 0x7f3cec123020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 10 monclient: _finish_hunting 0 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 1 monclient: found mon.noname-f 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:41.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:13:41.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.642+0000 7f3ce3fff640 1 -- 172.21.15.98:0/928747858 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cec126c60 con 0x7f3cec0a4920 2024-06-11T19:13:41.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce3fff640 1 --2- 172.21.15.98:0/928747858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cec0a4920 0x7f3cec137aa0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c400 tx=0x7f3cd400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:41.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 10 client.?.objecter ms_handle_connect 0x7f3cec0a4920 2024-06-11T19:13:41.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:41.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 1 -- 172.21.15.98:0/928747858 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3cd4017070 con 0x7f3cec0a4920 2024-06-11T19:13:41.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:41.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:13:41.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 10 monclient: dump: 2024-06-11T19:13:41.660 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:41.660 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:41.660 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:41.660 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:41.661 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:41.661 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:41.661 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:41.661 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:41.661 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:41.661 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:41.661 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:41.661 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:41.661 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:41.661 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:41.661 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:41.661 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:41.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 10 monclient: _finish_auth 0 2024-06-11T19:13:41.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:13:41.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:41.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 1 -- 172.21.15.98:0/928747858 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd40070d0 con 0x7f3cec0a4920 2024-06-11T19:13:41.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3cf1a71640 5 monclient: authenticate success, global_id 31582 2024-06-11T19:13:41.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3cf1a71640 10 monclient: _renew_subs 2024-06-11T19:13:41.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3cf1a71640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:13:41.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3cf1a71640 1 -- 172.21.15.98:0/928747858 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cec1257b0 con 0x7f3cec0a4920 2024-06-11T19:13:41.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:41.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 1 -- 172.21.15.98:0/928747858 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3cd40076f0 con 0x7f3cec0a4920 2024-06-11T19:13:41.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:41.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:13:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3ce8ff9640 10 monclient: dump: 2024-06-11T19:13:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:41.663 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3cf1a71640 10 client.31582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3cf1a71640 10 monclient: _renew_subs 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3cf1a71640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3cf1a71640 1 -- 172.21.15.98:0/928747858 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cec0a91d0 con 0x7f3cec0a4920 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3cea7fc640 4 set_mon_vals no callback set 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.643+0000 7f3cf1a71640 1 librados: init done 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3cf1a71640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3cf1a71640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:41.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3cf1a71640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:13:41.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3cf1a71640 1 -- 172.21.15.98:0/928747858 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cec0a4d00 con 0x7f3cec0a4920 2024-06-11T19:13:41.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3ce8ff9640 1 -- 172.21.15.98:0/928747858 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3cd4007280 con 0x7f3cec0a4920 2024-06-11T19:13:41.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3ce8ff9640 1 --2- 172.21.15.98:0/928747858 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3ccc03fdb0 0x7f3ccc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:41.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3ce8ff9640 1 -- 172.21.15.98:0/928747858 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 7369+0+0 (secure 0 0 0) 0x7f3cd40510e0 con 0x7f3cec0a4920 2024-06-11T19:13:41.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3ce8ff9640 10 client.31582.objecter ms_dispatch 0x7f3cec000b90 osd_map(183..183 src has 1..183) 2024-06-11T19:13:41.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3ce8ff9640 3 client.31582.objecter handle_osd_map got epochs [183,183] > 0 2024-06-11T19:13:41.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3ce8ff9640 3 client.31582.objecter handle_osd_map decoding full epoch 183 2024-06-11T19:13:41.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3ce8ff9640 20 client.31582.objecter dump_active .. 0 homeless 2024-06-11T19:13:41.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3ceb7fe640 1 --2- 172.21.15.98:0/928747858 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3ccc03fdb0 0x7f3ccc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:41.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.644+0000 7f3ceb7fe640 10 monclient: get_auth_request con 0x7f3ccc03fdb0 auth_method 0 2024-06-11T19:13:41.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.645+0000 7f3ce8ff9640 10 client.31582.objecter ms_handle_connect 0x7f3ccc03fdb0 2024-06-11T19:13:41.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.645+0000 7f3ceb7fe640 1 --2- 172.21.15.98:0/928747858 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3ccc03fdb0 0x7f3ccc042270 secure :-1 s=READY pgs=4088 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc035000 tx=0x7f3cdc008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:41.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.649+0000 7f3ce8ff9640 1 -- 172.21.15.98:0/928747858 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3cd4019d60 con 0x7f3cec0a4920 2024-06-11T19:13:41.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.649+0000 7f3ce8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:41.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.649+0000 7f3ce8ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.902+0000 7f3cf1a71640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:13:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.902+0000 7f3cf1a71640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:13:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.902+0000 7f3cf1a71640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:13:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.902+0000 7f3cf1a71640 1 -- 172.21.15.98:0/928747858 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3cec125500 con 0x7f3cec0a4920 2024-06-11T19:13:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.903+0000 7f3ce8ff9640 1 -- 172.21.15.98:0/928747858 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f3cd401a360 con 0x7f3cec0a4920 2024-06-11T19:13:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.903+0000 7f3ce8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:13:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.903+0000 7f3ce8ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:41.903 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":20,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:13:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 10 librados: watch_flush enter 2024-06-11T19:13:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 10 librados: watch_flush exit 2024-06-11T19:13:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 20 client.31582.objecter shutdown clearing up homeless session... 2024-06-11T19:13:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 10 client.31582.objecter successfully canceled tick 2024-06-11T19:13:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 -- 172.21.15.98:0/928747858 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3ccc03fdb0 msgr2=0x7f3ccc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 --2- 172.21.15.98:0/928747858 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3ccc03fdb0 0x7f3ccc042270 secure :-1 s=READY pgs=4088 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc035000 tx=0x7f3cdc008030 comp rx=0 tx=0).stop 2024-06-11T19:13:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 10 monclient: shutdown 2024-06-11T19:13:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 -- 172.21.15.98:0/928747858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cec0a4920 msgr2=0x7f3cec137aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 --2- 172.21.15.98:0/928747858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cec0a4920 0x7f3cec137aa0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c400 tx=0x7f3cd400c8c0 comp rx=0 tx=0).stop 2024-06-11T19:13:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 -- 172.21.15.98:0/928747858 shutdown_connections 2024-06-11T19:13:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 --2- 172.21.15.98:0/928747858 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3ccc03fdb0 0x7f3ccc042270 unknown :-1 s=CLOSED pgs=4088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 --2- 172.21.15.98:0/928747858 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3cec0a5240 0x7f3cec123020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 --2- 172.21.15.98:0/928747858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cec0a4920 0x7f3cec137aa0 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 --2- 172.21.15.98:0/928747858 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3cec0a2530 0x7f3cec137560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 -- 172.21.15.98:0/928747858 >> 172.21.15.98:0/928747858 conn(0x7f3cec0ab850 msgr2=0x7f3cec1be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 -- 172.21.15.98:0/928747858 shutdown_connections 2024-06-11T19:13:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 -- 172.21.15.98:0/928747858 wait complete. 2024-06-11T19:13:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:41.905+0000 7f3cf1a71640 1 librados: shutdown 2024-06-11T19:13:44.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:13:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.026+0000 7f3a39aa4640 10 monclient: get_monmap_and_config 2024-06-11T19:13:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.026+0000 7f3a39aa4640 10 monclient: build_initial_monmap 2024-06-11T19:13:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.026+0000 7f3a39aa4640 10 monclient: monmap: 2024-06-11T19:13:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:45.027730+0000 2024-06-11T19:13:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:45.027730+0000 2024-06-11T19:13:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:45.027 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.027+0000 7f3a39aa4640 1 Processor -- start 2024-06-11T19:13:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.027+0000 7f3a39aa4640 1 -- start start 2024-06-11T19:13:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.027+0000 7f3a39aa4640 10 monclient: init 2024-06-11T19:13:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 10 monclient: _add_conns ranks=[7,2,8,10,3,4,9,5,1,0,6] 2024-06-11T19:13:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a340a4920 0x7f3a340a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 10 monclient(hunting): picked mon.noname-i con 0x7f3a340a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a340a5240 0x7f3a340a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 10 monclient(hunting): picked mon.noname-b con 0x7f3a340a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a340a2530 0x7f3a340a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 10 monclient(hunting): picked mon.noname-e con 0x7f3a340a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3a340a7140 con 0x7f3a340a5240 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f3a340a2b50 con 0x7f3a340a4920 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3a340a2e30 con 0x7f3a340a2530 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a39aa4640 10 monclient(hunting): authenticate will time out at 1034.835178s 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a32ffd640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a340a5240 msgr2=0x7f3a340a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-06-11T19:13:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a32ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a340a5240 0x7f3a340a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a33fff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a340a2530 msgr2=0x7f3a340a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:13:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a33fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a340a2530 0x7f3a340a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a337fe640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a340a4920 msgr2=0x7f3a340a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:13:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.028+0000 7f3a337fe640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a340a4920 0x7f3a340a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:45.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.229+0000 7f3a32ffd640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a340a5240 msgr2=0x7f3a340a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-06-11T19:13:45.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.229+0000 7f3a33fff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a340a2530 msgr2=0x7f3a340a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:13:45.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.229+0000 7f3a32ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a340a5240 0x7f3a340a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:13:45.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.229+0000 7f3a33fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a340a2530 0x7f3a340a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:13:45.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.229+0000 7f3a337fe640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a340a4920 msgr2=0x7f3a340a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:13:45.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.229+0000 7f3a337fe640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a340a4920 0x7f3a340a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:13:45.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.630+0000 7f3a32ffd640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a340a5240 msgr2=0x7f3a340a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-06-11T19:13:45.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.630+0000 7f3a33fff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a340a2530 msgr2=0x7f3a340a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:13:45.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.630+0000 7f3a32ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a340a5240 0x7f3a340a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:13:45.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.630+0000 7f3a33fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a340a2530 0x7f3a340a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:13:45.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.630+0000 7f3a337fe640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a340a4920 msgr2=0x7f3a340a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:13:45.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:45.630+0000 7f3a337fe640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a340a4920 0x7f3a340a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:13:46.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:46.431+0000 7f3a33fff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a340a2530 msgr2=0x7f3a340a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:13:46.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:46.431+0000 7f3a32ffd640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a340a5240 msgr2=0x7f3a340a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-06-11T19:13:46.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:46.431+0000 7f3a32ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a340a5240 0x7f3a340a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:13:46.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:46.431+0000 7f3a33fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a340a2530 0x7f3a340a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:13:46.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:46.431+0000 7f3a337fe640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a340a4920 msgr2=0x7f3a340a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:13:46.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:46.431+0000 7f3a337fe640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a340a4920 0x7f3a340a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:13:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 10 monclient(hunting): tick 2024-06-11T19:13:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 1 monclient(hunting): continuing hunt 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a340a2530 msgr2=0x7f3a340a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a340a2530 0x7f3a340a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a340a4920 msgr2=0x7f3a340a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3a340a4920 0x7f3a340a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a340a5240 msgr2=0x7f3a340a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a340a5240 0x7f3a340a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 10 monclient: _add_conns ranks=[3,0,1,4,5,9,10,6] 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a20003840 0x7f3a20005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 10 monclient(hunting): picked mon.noname-g con 0x7f3a20003840 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a20006270 addr v1:172.21.15.98:6789/0 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3a20009fa0 addr v1:172.21.15.117:6789/0 2024-06-11T19:13:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a340a3110 con 0x7f3a20006270 2024-06-11T19:13:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a2000ed40 con 0x7f3a20009fa0 2024-06-11T19:13:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f3a340a7140 con 0x7f3a20003840 2024-06-11T19:13:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a317fa640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a337fe640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a20003840 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:13:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a337fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a20003840 0x7f3a20005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a32ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3a20006270 0x7f3a20008700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55890/0 (socket says 172.21.15.98:55890) 2024-06-11T19:13:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.028+0000 7f3a32ffd640 1 -- 172.21.15.98:0/392453363 learned_addr learned my addr 172.21.15.98:0/392453363 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:13:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.229+0000 7f3a337fe640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a20003840 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:13:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.229+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a20003840 0x7f3a20005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:13:48.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.630+0000 7f3a337fe640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a20003840 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:13:48.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:48.630+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a20003840 0x7f3a20005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:13:49.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:49.431+0000 7f3a337fe640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a20003840 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:13:49.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:49.431+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a20003840 0x7f3a20005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:13:51.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 10 monclient(hunting): tick 2024-06-11T19:13:51.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:13:51.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 1 monclient(hunting): continuing hunt 2024-06-11T19:13:51.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:13:51.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a20003840 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:51.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a20003840 0x7f3a20005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:51.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 1 -- 172.21.15.98:0/392453363 >> v1:172.21.15.117:6789/0 conn(0x7f3a20009fa0 legacy=0x7f3a2000c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:51.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 1 -- 172.21.15.98:0/392453363 >> v1:172.21.15.98:6789/0 conn(0x7f3a20006270 legacy=0x7f3a20008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:51.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a33fff640 1 -- 172.21.15.98:0/392453363 reap_dead start 2024-06-11T19:13:51.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 10 monclient: _add_conns ranks=[5,6,4,10,9] 2024-06-11T19:13:51.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:51.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 10 monclient(hunting): picked mon.noname-h con 0x7f3a2000f050 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:51.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:51.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 10 monclient(hunting): picked mon.noname-d con 0x7f3a2000f620 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:51.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:51.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a33fff640 1 -- 172.21.15.98:0/392453363 reap_dead start 2024-06-11T19:13:51.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2000fac0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:51.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 1 -- 172.21.15.98:0/392453363 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3a340a7140 con 0x7f3a2000fac0 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 1 -- 172.21.15.98:0/392453363 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f3a340a2b50 con 0x7f3a2000f050 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 1 -- 172.21.15.98:0/392453363 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3a340a2e30 con 0x7f3a2000f620 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a317fa640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request con 0x7f3a2000f620 auth_method 0 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a32ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a33fff640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a33fff640 10 monclient(hunting): get_auth_request con 0x7f3a2000fac0 auth_method 0 2024-06-11T19:13:51.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a33fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a33fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a337fe640 10 monclient(hunting): get_auth_request con 0x7f3a2000f050 auth_method 0 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a337fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a32ffd640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 msgr2=0x7f3a2000dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a32ffd640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 msgr2=0x7f3a2000dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a33fff640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 msgr2=0x7f3a2000c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a33fff640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 msgr2=0x7f3a2000c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a33fff640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:51.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:51.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a33fff640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:51.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a337fe640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:13:51.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a337fe640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:51.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:51.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.029+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a33fff640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a33fff640 10 monclient(hunting): get_auth_request con 0x7f3a2000fac0 auth_method 0 2024-06-11T19:13:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a33fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a33fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request con 0x7f3a2000f620 auth_method 0 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a32ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a32ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a33fff640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 msgr2=0x7f3a2000c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a33fff640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 msgr2=0x7f3a2000c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a337fe640 10 monclient(hunting): get_auth_request con 0x7f3a2000f050 auth_method 0 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a33fff640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a337fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a337fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a32ffd640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 msgr2=0x7f3a2000dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:13:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a32ffd640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 msgr2=0x7f3a2000dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:51.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:51.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a33fff640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:13:51.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:13:51.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a337fe640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:13:51.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a337fe640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:51.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:51.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.230+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:13:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a33fff640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a33fff640 10 monclient(hunting): get_auth_request con 0x7f3a2000fac0 auth_method 0 2024-06-11T19:13:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a33fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a33fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request con 0x7f3a2000f620 auth_method 0 2024-06-11T19:13:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a32ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a32ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a33fff640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 msgr2=0x7f3a2000c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:13:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a33fff640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 msgr2=0x7f3a2000c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a33fff640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a32ffd640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 msgr2=0x7f3a2000dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a32ffd640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 msgr2=0x7f3a2000dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a33fff640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a337fe640 10 monclient(hunting): get_auth_request con 0x7f3a2000f050 auth_method 0 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a337fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a337fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a337fe640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a337fe640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:51.631+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:13:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a33fff640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a33fff640 10 monclient(hunting): get_auth_request con 0x7f3a2000fac0 auth_method 0 2024-06-11T19:13:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a33fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a33fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request con 0x7f3a2000f620 auth_method 0 2024-06-11T19:13:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a32ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a32ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a33fff640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 msgr2=0x7f3a2000c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a32ffd640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 msgr2=0x7f3a2000dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a33fff640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 msgr2=0x7f3a2000c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a32ffd640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 msgr2=0x7f3a2000dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a33fff640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a33fff640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a337fe640 10 monclient(hunting): get_auth_request con 0x7f3a2000f050 auth_method 0 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a337fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:52.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a337fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:13:52.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a337fe640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:13:52.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a337fe640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:13:52.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:13:52.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:52.431+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:13:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 10 monclient(hunting): tick 2024-06-11T19:13:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:13:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 1 monclient(hunting): continuing hunt 2024-06-11T19:13:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:13:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 msgr2=0x7f3a2000dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a2000f620 0x7f3a2000dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 msgr2=0x7f3a20005c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3a2000f050 0x7f3a20005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 msgr2=0x7f3a2000c460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a2000fac0 0x7f3a2000c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 10 monclient: _add_conns ranks=[10,9] 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 1 --2- 172.21.15.98:0/392453363 >> v2:172.21.15.117:3300/0 conn(0x7f3a2000e5b0 0x7f3a20008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 10 monclient(hunting): picked mon.noname-f con 0x7f3a2000e5b0 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.028+0000 7f3a317fa640 1 --2- 172.21.15.98:0/392453363 >> v2:172.21.15.98:3300/0 conn(0x7f3a20008eb0 0x7f3a200092c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a317fa640 10 monclient(hunting): picked mon.noname-a con 0x7f3a20008eb0 addr v2:172.21.15.98:3300/0 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a317fa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a317fa640 1 -- 172.21.15.98:0/392453363 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3a340a7140 con 0x7f3a20008eb0 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a317fa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a317fa640 1 -- 172.21.15.98:0/392453363 --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f3a340a2b50 con 0x7f3a2000e5b0 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a317fa640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> v2:172.21.15.98:3300/0 conn(0x7f3a20008eb0 0x7f3a200092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request con 0x7f3a20008eb0 auth_method 0 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a337fe640 1 --2- 172.21.15.98:0/392453363 >> v2:172.21.15.117:3300/0 conn(0x7f3a2000e5b0 0x7f3a20008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a337fe640 10 monclient(hunting): get_auth_request con 0x7f3a2000e5b0 auth_method 0 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a337fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 10 monclient(hunting): handle_auth_done global_id 64278 payload 274 2024-06-11T19:13:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 1 -- 172.21.15.98:0/392453363 >> v2:172.21.15.117:3300/0 conn(0x7f3a2000e5b0 msgr2=0x7f3a20008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> v2:172.21.15.117:3300/0 conn(0x7f3a2000e5b0 0x7f3a20008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 1 monclient: found mon.noname-a 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.029+0000 7f3a32ffd640 1 -- 172.21.15.98:0/392453363 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2000ffe0 con 0x7f3a20008eb0 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a20008eb0 0x7f3a200092c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000e580 tx=0x7f3a2802caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8eb5f2265baf258 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 1 -- 172.21.15.98:0/392453363 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a2802eb20 con 0x7f3a20008eb0 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 10 monclient: dump: 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 10 monclient: _finish_auth 0 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 10 monclient: _check_auth_tickets 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 1 -- 172.21.15.98:0/392453363 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a2802ed00 con 0x7f3a20008eb0 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 1 -- 172.21.15.98:0/392453363 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a2802f8f0 con 0x7f3a20008eb0 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a327fc640 10 monclient: dump: 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a39aa4640 5 monclient: authenticate success, global_id 64278 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a39aa4640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.030+0000 7f3a39aa4640 4 set_mon_vals no callback set 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.031+0000 7f3a39aa4640 10 monclient: shutdown 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.031+0000 7f3a39aa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.031+0000 7f3a39aa4640 1 -- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a20008eb0 msgr2=0x7f3a200092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.031+0000 7f3a39aa4640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a20008eb0 0x7f3a200092c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000e580 tx=0x7f3a2802caf0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.031+0000 7f3a33fff640 1 -- 172.21.15.98:0/392453363 reap_dead start 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.031+0000 7f3a39aa4640 1 -- 172.21.15.98:0/392453363 shutdown_connections 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.031+0000 7f3a39aa4640 1 --2- 172.21.15.98:0/392453363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a20008eb0 0x7f3a200092c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.031+0000 7f3a39aa4640 1 -- 172.21.15.98:0/392453363 >> 172.21.15.98:0/392453363 conn(0x7f3a340ab850 msgr2=0x7f3a340a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.031+0000 7f3a39aa4640 1 -- 172.21.15.98:0/392453363 shutdown_connections 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.031+0000 7f3a39aa4640 1 -- 172.21.15.98:0/392453363 wait complete. 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.032+0000 7f3a39aa4640 10 monclient: build_initial_monmap 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.032+0000 7f3a39aa4640 10 monclient: monmap: 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.032+0000 7f3a39aa4640 1 librados: starting msgr at 2024-06-11T19:13:54.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.032+0000 7f3a39aa4640 1 librados: starting objecter 2024-06-11T19:13:54.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.032+0000 7f3a39aa4640 1 Processor -- start 2024-06-11T19:13:54.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 1 -- start start 2024-06-11T19:13:54.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 1 librados: setting wanted keys 2024-06-11T19:13:54.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 1 librados: calling monclient init 2024-06-11T19:13:54.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 10 monclient: init 2024-06-11T19:13:54.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:54.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 10 monclient: _add_conns ranks=[1,4,2,6,7,0,5,8,3] 2024-06-11T19:13:54.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a20008eb0 0x7f3a341cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 10 monclient(hunting): picked mon.noname-f con 0x7f3a20008eb0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:13:54.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a341cdfe0 0x7f3a34124060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 10 monclient(hunting): picked mon.noname-c con 0x7f3a341cdfe0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:54.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a341ce3e0 0x7f3a341245a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 10 monclient(hunting): picked mon.noname-b con 0x7f3a341ce3e0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3a340a6da0 con 0x7f3a20008eb0 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3a34138e50 con 0x7f3a341ce3e0 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3a340a4d00 con 0x7f3a341cdfe0 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a39aa4640 10 monclient(hunting): authenticate will time out at 1043.840681s 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a33fff640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a341ce3e0 msgr2=0x7f3a341245a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a33fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a341ce3e0 0x7f3a341245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a32ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a341cdfe0 0x7f3a34124060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a337fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a20008eb0 0x7f3a341cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a32ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a341cdfe0 0x7f3a34124060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:35948/0 (socket says 172.21.15.98:35948) 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a337fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a20008eb0 0x7f3a341cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:36174/0 (socket says 172.21.15.98:36174) 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a32ffd640 1 -- 172.21.15.98:0/1765278389 learned_addr learned my addr 172.21.15.98:0/1765278389 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request con 0x7f3a341cdfe0 auth_method 0 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a32ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a337fe640 10 monclient(hunting): get_auth_request con 0x7f3a20008eb0 auth_method 0 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a337fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.033+0000 7f3a32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a32ffd640 10 monclient(hunting): handle_auth_done global_id 23539 payload 824 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a32ffd640 1 -- 172.21.15.98:0/1765278389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a341ce3e0 msgr2=0x7f3a341245a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/1765278389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a341ce3e0 0x7f3a341245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a32ffd640 1 -- 172.21.15.98:0/1765278389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a20008eb0 msgr2=0x7f3a341cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/1765278389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a20008eb0 0x7f3a341cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a32ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a32ffd640 1 monclient: found mon.noname-c 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a32ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a32ffd640 1 -- 172.21.15.98:0/1765278389 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a34127d40 con 0x7f3a341cdfe0 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a32ffd640 1 --2- 172.21.15.98:0/1765278389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a341cdfe0 0x7f3a34124060 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f3a28000c00 tx=0x7f3a28005cb0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 10 client.?.objecter ms_handle_connect 0x7f3a341cdfe0 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 1 -- 172.21.15.98:0/1765278389 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a28049070 con 0x7f3a341cdfe0 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 10 monclient: dump: 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 10 monclient: _finish_auth 0 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 1 -- 172.21.15.98:0/1765278389 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a2803e040 con 0x7f3a341cdfe0 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 1 -- 172.21.15.98:0/1765278389 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a28038870 con 0x7f3a341cdfe0 2024-06-11T19:13:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a39aa4640 5 monclient: authenticate success, global_id 23539 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a1bfff640 10 monclient: dump: 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a39aa4640 10 monclient: _renew_subs 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a31ffb640 4 set_mon_vals no callback set 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a39aa4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a39aa4640 1 -- 172.21.15.98:0/1765278389 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a34126890 con 0x7f3a341cdfe0 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a39aa4640 10 client.23539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a39aa4640 10 monclient: _renew_subs 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a39aa4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a39aa4640 1 -- 172.21.15.98:0/1765278389 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a340aa950 con 0x7f3a341cdfe0 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.034+0000 7f3a39aa4640 1 librados: init done 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.035+0000 7f3a1bfff640 1 -- 172.21.15.98:0/1765278389 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3a2802ed00 con 0x7f3a341cdfe0 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.035+0000 7f3a39aa4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.035+0000 7f3a39aa4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.035+0000 7f3a39aa4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.035+0000 7f3a39aa4640 1 -- 172.21.15.98:0/1765278389 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a340a4380 con 0x7f3a341cdfe0 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.036+0000 7f3a1bfff640 1 --2- 172.21.15.98:0/1765278389 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3a20042450 0x7f3a20044910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.036+0000 7f3a1bfff640 1 -- 172.21.15.98:0/1765278389 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(185..185 src has 1..185) ==== 7479+0+0 (secure 0 0 0) 0x7f3a28006e00 con 0x7f3a341cdfe0 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.036+0000 7f3a1bfff640 10 client.23539.objecter ms_dispatch 0x7f3a34000b90 osd_map(185..185 src has 1..185) 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.036+0000 7f3a1bfff640 3 client.23539.objecter handle_osd_map got epochs [185,185] > 0 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.036+0000 7f3a1bfff640 3 client.23539.objecter handle_osd_map decoding full epoch 185 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.036+0000 7f3a1bfff640 20 client.23539.objecter dump_active .. 0 homeless 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.036+0000 7f3a337fe640 1 --2- 172.21.15.98:0/1765278389 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3a20042450 0x7f3a20044910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.036+0000 7f3a337fe640 10 monclient: get_auth_request con 0x7f3a20042450 auth_method 0 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.036+0000 7f3a1bfff640 10 client.23539.objecter ms_handle_connect 0x7f3a20042450 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.036+0000 7f3a337fe640 1 --2- 172.21.15.98:0/1765278389 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3a20042450 0x7f3a20044910 secure :-1 s=READY pgs=4089 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009400 tx=0x7f3a2400e040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.040+0000 7f3a1bfff640 1 -- 172.21.15.98:0/1765278389 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3a28034030 con 0x7f3a341cdfe0 2024-06-11T19:13:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.040+0000 7f3a1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.040+0000 7f3a1bfff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:54.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.292+0000 7f3a39aa4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:13:54.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.292+0000 7f3a39aa4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:13:54.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.292+0000 7f3a39aa4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:54.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.292+0000 7f3a39aa4640 1 -- 172.21.15.98:0/1765278389 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3a340aac80 con 0x7f3a341cdfe0 2024-06-11T19:13:54.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.292+0000 7f3a1bfff640 1 -- 172.21.15.98:0/1765278389 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3168 (secure 0 0 0) 0x7f3a28009570 con 0x7f3a341cdfe0 2024-06-11T19:13:54.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.292+0000 7f3a1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:13:54.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.292+0000 7f3a1bfff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:54.293 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":24,"quorum":[0,1,4,5,6],"quorum_names":["a","f","c","h","d"],"quorum_leader_name":"f","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:13:54.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 10 librados: watch_flush enter 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 10 librados: watch_flush exit 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 20 client.23539.objecter shutdown clearing up homeless session... 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 10 client.23539.objecter successfully canceled tick 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 1 -- 172.21.15.98:0/1765278389 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3a20042450 msgr2=0x7f3a20044910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 1 --2- 172.21.15.98:0/1765278389 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3a20042450 0x7f3a20044910 secure :-1 s=READY pgs=4089 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009400 tx=0x7f3a2400e040 comp rx=0 tx=0).stop 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 10 monclient: shutdown 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 1 -- 172.21.15.98:0/1765278389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a341cdfe0 msgr2=0x7f3a34124060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 1 --2- 172.21.15.98:0/1765278389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a341cdfe0 0x7f3a34124060 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f3a28000c00 tx=0x7f3a28005cb0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 1 -- 172.21.15.98:0/1765278389 shutdown_connections 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 1 --2- 172.21.15.98:0/1765278389 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3a20042450 0x7f3a20044910 unknown :-1 s=CLOSED pgs=4089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 1 --2- 172.21.15.98:0/1765278389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a341cdfe0 0x7f3a34124060 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 1 --2- 172.21.15.98:0/1765278389 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a341ce3e0 0x7f3a341245a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.294+0000 7f3a39aa4640 1 --2- 172.21.15.98:0/1765278389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a20008eb0 0x7f3a341cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.295+0000 7f3a39aa4640 1 -- 172.21.15.98:0/1765278389 >> 172.21.15.98:0/1765278389 conn(0x7f3a340ab850 msgr2=0x7f3a34137fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.295+0000 7f3a39aa4640 1 -- 172.21.15.98:0/1765278389 shutdown_connections 2024-06-11T19:13:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.295+0000 7f3a39aa4640 1 -- 172.21.15.98:0/1765278389 wait complete. 2024-06-11T19:13:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.295+0000 7f3a39aa4640 1 librados: shutdown 2024-06-11T19:13:54.305 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-06-11T19:13:54.305 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon a 2024-06-11T19:13:54.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:13:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.405+0000 7f791d0d9640 10 monclient: get_monmap_and_config 2024-06-11T19:13:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.405+0000 7f791d0d9640 10 monclient: build_initial_monmap 2024-06-11T19:13:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.405+0000 7f791d0d9640 10 monclient: monmap: 2024-06-11T19:13:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:54.406352+0000 2024-06-11T19:13:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:54.406352+0000 2024-06-11T19:13:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:54.406 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 1 Processor -- start 2024-06-11T19:13:54.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 1 -- start start 2024-06-11T19:13:54.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 10 monclient: init 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 10 monclient: _add_conns ranks=[4,10,2,5,1,3,7,8,6,0,9] 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79180a4920 0x7f79180a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 10 monclient(hunting): picked mon.noname-c con 0x7f79180a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f79180a5240 0x7f79180a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 10 monclient(hunting): picked mon.noname-f con 0x7f79180a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79180a2530 0x7f79180a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 10 monclient(hunting): picked mon.noname-b con 0x7f79180a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f79180a7140 con 0x7f79180a5240 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f79180a2b50 con 0x7f79180a2530 2024-06-11T19:13:54.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f79180a2e30 con 0x7f79180a4920 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791d0d9640 10 monclient(hunting): authenticate will time out at 1044.213598s 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791759e640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79180a2530 msgr2=0x7f79180a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f791759e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79180a2530 0x7f79180a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79180a4920 0x7f79180a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79180a4920 0x7f79180a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:35950/0 (socket says 172.21.15.98:35950) 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 1 -- 172.21.15.98:0/1739806122 learned_addr learned my addr 172.21.15.98:0/1739806122 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 10 monclient(hunting): get_auth_request con 0x7f79180a4920 auth_method 0 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f790ffff640 1 --2- 172.21.15.98:0/1739806122 >> v2:172.21.15.117:3300/0 conn(0x7f79180a5240 0x7f79180a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f790ffff640 10 monclient(hunting): get_auth_request con 0x7f79180a5240 auth_method 0 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f790ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f790ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f790ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 10 monclient(hunting): handle_auth_done global_id 23548 payload 274 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 1 -- 172.21.15.98:0/1739806122 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79180a2530 msgr2=0x7f79180a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 1 --2- 172.21.15.98:0/1739806122 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79180a2530 0x7f79180a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 1 -- 172.21.15.98:0/1739806122 >> v2:172.21.15.117:3300/0 conn(0x7f79180a5240 msgr2=0x7f79180a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 1 --2- 172.21.15.98:0/1739806122 >> v2:172.21.15.117:3300/0 conn(0x7f79180a5240 0x7f79180a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:13:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 1 monclient: found mon.noname-c 2024-06-11T19:13:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.406+0000 7f7916d9d640 1 -- 172.21.15.98:0/1739806122 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79180a3110 con 0x7f79180a4920 2024-06-11T19:13:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f7916d9d640 1 --2- 172.21.15.98:0/1739806122 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79180a4920 0x7f79180a4d00 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f7908009870 tx=0x7f790802b7e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=7809ec2fc6bf6b7 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 1 -- 172.21.15.98:0/1739806122 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7908038070 con 0x7f79180a4920 2024-06-11T19:13:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 10 monclient: dump: 2024-06-11T19:13:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.424 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 10 monclient: _finish_auth 0 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 10 monclient: _check_auth_tickets 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 1 -- 172.21.15.98:0/1739806122 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79080319b0 con 0x7f79180a4920 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791d0d9640 5 monclient: authenticate success, global_id 23548 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791d0d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791d0d9640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 1 -- 172.21.15.98:0/1739806122 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f790802f050 con 0x7f79180a4920 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791d0d9640 4 set_mon_vals no callback set 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.407+0000 7f791659c640 10 monclient: dump: 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.426 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.408+0000 7f791d0d9640 10 monclient: shutdown 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.408+0000 7f791d0d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.408+0000 7f791d0d9640 1 -- 172.21.15.98:0/1739806122 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79180a4920 msgr2=0x7f79180a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.408+0000 7f791d0d9640 1 --2- 172.21.15.98:0/1739806122 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79180a4920 0x7f79180a4d00 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f7908009870 tx=0x7f790802b7e0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.408+0000 7f791d0d9640 1 -- 172.21.15.98:0/1739806122 shutdown_connections 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.408+0000 7f791d0d9640 1 --2- 172.21.15.98:0/1739806122 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79180a2530 0x7f79180a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.408+0000 7f791d0d9640 1 --2- 172.21.15.98:0/1739806122 >> v2:172.21.15.117:3300/0 conn(0x7f79180a5240 0x7f79180a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.408+0000 7f791d0d9640 1 --2- 172.21.15.98:0/1739806122 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79180a4920 0x7f79180a4d00 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.408+0000 7f791d0d9640 1 -- 172.21.15.98:0/1739806122 >> 172.21.15.98:0/1739806122 conn(0x7f79180ab850 msgr2=0x7f79180a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.408+0000 7f791d0d9640 1 -- 172.21.15.98:0/1739806122 shutdown_connections 2024-06-11T19:13:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.408+0000 7f791d0d9640 1 -- 172.21.15.98:0/1739806122 wait complete. 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient: build_initial_monmap 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient: monmap: 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:54.428 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:54.429 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 1 librados: starting msgr at 2024-06-11T19:13:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 1 librados: starting objecter 2024-06-11T19:13:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 1 Processor -- start 2024-06-11T19:13:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 1 -- start start 2024-06-11T19:13:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 1 librados: setting wanted keys 2024-06-11T19:13:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 1 librados: calling monclient init 2024-06-11T19:13:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient: init 2024-06-11T19:13:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient: _add_conns ranks=[7,0,5,2,4,1,8,6,3] 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f79180a2530 0x7f7918137670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient(hunting): picked mon.noname-i con 0x7f79180a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79180a4920 0x7f7918137bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient(hunting): picked mon.noname-a con 0x7f79180a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f79180a5240 0x7f7918123090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient(hunting): picked mon.noname-h con 0x7f79180a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79180a6da0 con 0x7f79180a4920 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f7918138e50 con 0x7f79180a5240 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f79180a6420 con 0x7f79180a2530 2024-06-11T19:13:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.409+0000 7f791d0d9640 10 monclient(hunting): authenticate will time out at 1044.216943s 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f7916d9d640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f79180a2530 msgr2=0x7f7918137670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f7916d9d640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f79180a2530 0x7f7918137670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79180a4920 0x7f7918137bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79180a4920 0x7f7918137bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55370/0 (socket says 172.21.15.98:55370) 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 1 -- 172.21.15.98:0/223092039 learned_addr learned my addr 172.21.15.98:0/223092039 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 10 monclient(hunting): get_auth_request con 0x7f79180a4920 auth_method 0 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791759e640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f79180a5240 0x7f7918123090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791759e640 10 monclient(hunting): get_auth_request con 0x7f79180a5240 auth_method 0 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791759e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791759e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 10 monclient(hunting): handle_auth_done global_id 64287 payload 824 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 1 -- 172.21.15.98:0/223092039 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f79180a2530 msgr2=0x7f7918137670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f79180a2530 0x7f7918137670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 1 -- 172.21.15.98:0/223092039 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f79180a5240 msgr2=0x7f7918123090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f79180a5240 0x7f7918123090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 1 monclient: found mon.noname-a 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:13:54.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 1 -- 172.21.15.98:0/223092039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79181c0bf0 con 0x7f79180a4920 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790ffff640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79180a4920 0x7f7918137bb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f790000d600 tx=0x7f790000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 10 client.?.objecter ms_handle_connect 0x7f79180a4920 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7900012a60 con 0x7f79180a4920 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 10 monclient: dump: 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7900019070 con 0x7f79180a4920 2024-06-11T19:13:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791d0d9640 5 monclient: authenticate success, global_id 64287 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7900012c20 con 0x7f79180a4920 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791d0d9640 10 monclient: _renew_subs 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791d0d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791d0d9640 1 -- 172.21.15.98:0/223092039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7918126d70 con 0x7f79180a4920 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f7915d9b640 4 set_mon_vals no callback set 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f790f7fe640 10 monclient: dump: 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791d0d9640 10 client.64287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791d0d9640 10 monclient: _renew_subs 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791d0d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791d0d9640 1 -- 172.21.15.98:0/223092039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7918127050 con 0x7f79180a4920 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.410+0000 7f791d0d9640 1 librados: init done 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f791d0d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f791d0d9640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79181235d0 0x7f79181239b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f791d0d9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f791d0d9640 1 -- 172.21.15.98:0/223092039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7918050830 con 0x7f79181235d0 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f79000125f0 con 0x7f79180a4920 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f7916d9d640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79181235d0 0x7f79181239b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f7916d9d640 10 monclient: get_auth_request con 0x7f79181235d0 auth_method 0 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f7916d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f7916d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f7916d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f7916d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f7916d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f7916d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f790f7fe640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f78f803fdb0 0x7f78f8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 7479+0+0 (secure 0 0 0) 0x7f790000f040 con 0x7f79180a4920 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f790f7fe640 10 client.64287.objecter ms_dispatch 0x7f7918000b90 osd_map(185..185 src has 1..185) 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f790f7fe640 3 client.64287.objecter handle_osd_map got epochs [185,185] > 0 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f790f7fe640 3 client.64287.objecter handle_osd_map decoding full epoch 185 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f7916d9d640 10 monclient(hunting): handle_auth_done global_id 64296 payload 824 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f7916d9d640 1 -- 172.21.15.98:0/223092039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f79180a93d0 con 0x7f79181235d0 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.411+0000 7f790f7fe640 20 client.64287.objecter dump_active .. 0 homeless 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 10 client.64287.objecter ms_handle_connect 0x7f79181235d0 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 10 client.64287.objecter resend_mon_ops 2024-06-11T19:13:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f7916d9d640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79181235d0 0x7f79181239b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f79180a8420 tx=0x7f79080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f791759e640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f78f803fdb0 0x7f78f8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f791759e640 10 monclient: get_auth_request con 0x7f78f803fdb0 auth_method 0 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7908041030 con 0x7f79181235d0 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 10 monclient: dump: 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7908039040 con 0x7f79181235d0 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79181235d0 msgr2=0x7f79181239b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79181235d0 0x7f79181239b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f79180a8420 tx=0x7f79080046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79181235d0 msgr2=0x7f79181239b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79181235d0 0x7f79181239b0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f79180a8420 tx=0x7f79080046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f790f7fe640 10 client.64287.objecter ms_handle_connect 0x7f78f803fdb0 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.412+0000 7f791759e640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f78f803fdb0 0x7f78f8042270 secure :-1 s=READY pgs=4090 cs=0 l=1 rev1=1 crypto rx=0x7f79181266e0 tx=0x7f7910009450 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.418+0000 7f791d0d9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.418+0000 7f791d0d9640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7918123ef0 0x7f791809e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.418+0000 7f791d0d9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.418+0000 7f791d0d9640 1 -- 172.21.15.98:0/223092039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7918124df0 con 0x7f7918123ef0 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.418+0000 7f7916d9d640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7918123ef0 0x7f791809e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f7916d9d640 10 monclient: get_auth_request con 0x7f7918123ef0 auth_method 0 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f7916d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f7916d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f7916d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f7916d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f7916d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f7916d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f7916d9d640 10 monclient(hunting): handle_auth_done global_id 64305 payload 824 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f7916d9d640 1 -- 172.21.15.98:0/223092039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f79180a6750 con 0x7f7918123ef0 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 10 client.64287.objecter ms_handle_connect 0x7f7918123ef0 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 10 client.64287.objecter resend_mon_ops 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f7916d9d640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7918123ef0 0x7f791809e9f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f79080097c0 tx=0x7f7908009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7908041070 con 0x7f7918123ef0 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 10 monclient: dump: 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9228 (secure 0 0 0) 0x7f7908047810 con 0x7f7918123ef0 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:54.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7918123ef0 msgr2=0x7f791809e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7918123ef0 0x7f791809e9f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f79080097c0 tx=0x7f7908009240 comp rx=0 tx=0).stop 2024-06-11T19:13:54.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 1 -- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7918123ef0 msgr2=0x7f791809e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:54.444 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:54.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:54.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:54.444 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:54.444 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 24, 2024-06-11T19:13:54.444 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 26247, 2024-06-11T19:13:54.444 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: 6 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:54.445 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:54.446 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:54.447 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:54.448 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:54.449 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:54.450 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:54.450 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.450 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.450 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.457 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:54.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:54.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:54.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.462 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:54.464 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:54.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.466 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:54.466 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:54.466 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:13:54.466 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.466 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.466 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:54.466 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f790f7fe640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7918123ef0 0x7f791809e9f0 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f79080097c0 tx=0x7f7908009240 comp rx=0 tx=0).stop 2024-06-11T19:13:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.419+0000 7f791d0d9640 10 librados: watch_flush enter 2024-06-11T19:13:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 10 librados: watch_flush exit 2024-06-11T19:13:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 20 client.64287.objecter shutdown clearing up homeless session... 2024-06-11T19:13:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 10 client.64287.objecter successfully canceled tick 2024-06-11T19:13:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 1 -- 172.21.15.98:0/223092039 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f78f803fdb0 msgr2=0x7f78f8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f78f803fdb0 0x7f78f8042270 secure :-1 s=READY pgs=4090 cs=0 l=1 rev1=1 crypto rx=0x7f79181266e0 tx=0x7f7910009450 comp rx=0 tx=0).stop 2024-06-11T19:13:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 10 monclient: shutdown 2024-06-11T19:13:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:54.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 1 -- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79180a4920 msgr2=0x7f7918137bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 1 --2- 172.21.15.98:0/223092039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79180a4920 0x7f7918137bb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f790000d600 tx=0x7f790000dac0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791759e640 1 -- 172.21.15.98:0/223092039 reap_dead start 2024-06-11T19:13:54.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791759e640 1 -- 172.21.15.98:0/223092039 reap_dead start 2024-06-11T19:13:54.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 1 -- 172.21.15.98:0/223092039 shutdown_connections 2024-06-11T19:13:54.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 1 -- 172.21.15.98:0/223092039 >> 172.21.15.98:0/223092039 conn(0x7f79180ab850 msgr2=0x7f79181bfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:54.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 1 -- 172.21.15.98:0/223092039 shutdown_connections 2024-06-11T19:13:54.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 1 -- 172.21.15.98:0/223092039 wait complete. 2024-06-11T19:13:54.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.420+0000 7f791d0d9640 1 librados: shutdown 2024-06-11T19:13:54.469 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-06-11T19:13:54.469 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon c 2024-06-11T19:13:54.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:13:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.569+0000 7f998826c640 10 monclient: get_monmap_and_config 2024-06-11T19:13:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.569+0000 7f998826c640 10 monclient: build_initial_monmap 2024-06-11T19:13:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.569+0000 7f998826c640 10 monclient: monmap: 2024-06-11T19:13:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:54.570826+0000 2024-06-11T19:13:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:54.570826+0000 2024-06-11T19:13:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:54.571 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.570+0000 7f998826c640 1 Processor -- start 2024-06-11T19:13:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.570+0000 7f998826c640 1 -- start start 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.570+0000 7f998826c640 10 monclient: init 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.570+0000 7f998826c640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.570+0000 7f998826c640 10 monclient: _add_conns ranks=[3,10,2,6,9,0,1,4,5,7,8] 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99800a4920 0x7f99800a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 10 monclient(hunting): picked mon.noname-g con 0x7f99800a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f99800a5240 0x7f99800a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 10 monclient(hunting): picked mon.noname-f con 0x7f99800a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800a2530 0x7f99800a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 10 monclient(hunting): picked mon.noname-b con 0x7f99800a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f99800a7140 con 0x7f99800a5240 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f99800a2b50 con 0x7f99800a2530 2024-06-11T19:13:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f99800a2e30 con 0x7f99800a4920 2024-06-11T19:13:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f998826c640 10 monclient(hunting): authenticate will time out at 1044.378058s 2024-06-11T19:13:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9986809640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800a2530 msgr2=0x7f99800a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-06-11T19:13:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9986809640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800a2530 0x7f99800a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9986008640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99800a4920 msgr2=0x7f99800a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:13:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9986008640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99800a4920 0x7f99800a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f99800a5240 0x7f99800a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f99800a5240 0x7f99800a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:36182/0 (socket says 172.21.15.98:36182) 2024-06-11T19:13:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 1 -- 172.21.15.98:0/1941110196 learned_addr learned my addr 172.21.15.98:0/1941110196 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 10 monclient(hunting): get_auth_request con 0x7f99800a5240 auth_method 0 2024-06-11T19:13:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 10 monclient(hunting): handle_auth_done global_id 31618 payload 274 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 1 -- 172.21.15.98:0/1941110196 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99800a4920 msgr2=0x7f99800a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 1 --2- 172.21.15.98:0/1941110196 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99800a4920 0x7f99800a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 1 -- 172.21.15.98:0/1941110196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800a2530 msgr2=0x7f99800a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 1 --2- 172.21.15.98:0/1941110196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800a2530 0x7f99800a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 10 monclient: _finish_hunting 0 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 1 monclient: found mon.noname-f 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:13:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.571+0000 7f9985807640 1 -- 172.21.15.98:0/1941110196 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99800a3110 con 0x7f99800a5240 2024-06-11T19:13:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985807640 1 --2- 172.21.15.98:0/1941110196 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99800a5240 0x7f99800a59a0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f9970009870 tx=0x7f997002b800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad01c241e8ecabbb server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 1 -- 172.21.15.98:0/1941110196 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99700319d0 con 0x7f99800a5240 2024-06-11T19:13:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:13:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 10 monclient: dump: 2024-06-11T19:13:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 10 monclient: _finish_auth 0 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 10 monclient: _check_auth_tickets 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 1 -- 172.21.15.98:0/1941110196 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9970037070 con 0x7f99800a5240 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f998826c640 5 monclient: authenticate success, global_id 31618 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f998826c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f998826c640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 1 -- 172.21.15.98:0/1941110196 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9970031bd0 con 0x7f99800a5240 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f998826c640 4 set_mon_vals no callback set 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.572+0000 7f9985006640 10 monclient: dump: 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 10 monclient: shutdown 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 1 -- 172.21.15.98:0/1941110196 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99800a5240 msgr2=0x7f99800a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 1 --2- 172.21.15.98:0/1941110196 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99800a5240 0x7f99800a59a0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f9970009870 tx=0x7f997002b800 comp rx=0 tx=0).stop 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 1 -- 172.21.15.98:0/1941110196 shutdown_connections 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 1 --2- 172.21.15.98:0/1941110196 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99800a2530 0x7f99800a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 1 --2- 172.21.15.98:0/1941110196 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99800a5240 0x7f99800a59a0 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 1 --2- 172.21.15.98:0/1941110196 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99800a4920 0x7f99800a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 1 -- 172.21.15.98:0/1941110196 >> 172.21.15.98:0/1941110196 conn(0x7f99800ab850 msgr2=0x7f99800a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 1 -- 172.21.15.98:0/1941110196 shutdown_connections 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 1 -- 172.21.15.98:0/1941110196 wait complete. 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 10 monclient: build_initial_monmap 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.573+0000 7f998826c640 10 monclient: monmap: 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:54.594 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 1 librados: starting msgr at 2024-06-11T19:13:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 1 librados: starting objecter 2024-06-11T19:13:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 1 Processor -- start 2024-06-11T19:13:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 1 -- start start 2024-06-11T19:13:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 1 librados: setting wanted keys 2024-06-11T19:13:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 1 librados: calling monclient init 2024-06-11T19:13:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 10 monclient: init 2024-06-11T19:13:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 10 monclient: _add_conns ranks=[0,3,4,7,1,2,8,6,5] 2024-06-11T19:13:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99800a2530 0x7f99801cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 10 monclient(hunting): picked mon.noname-a con 0x7f99800a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:13:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99800a4920 0x7f99801ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 10 monclient(hunting): picked mon.noname-g con 0x7f99800a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f99800a5240 0x7f9980122ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 10 monclient(hunting): picked mon.noname-c con 0x7f99800a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:13:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f99800a6da0 con 0x7f99800a2530 2024-06-11T19:13:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f9980138e50 con 0x7f99800a4920 2024-06-11T19:13:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f99800a6420 con 0x7f99800a5240 2024-06-11T19:13:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f998826c640 10 monclient(hunting): authenticate will time out at 1044.381661s 2024-06-11T19:13:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986008640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99800a2530 0x7f99801cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986008640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99800a2530 0x7f99801cdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55404/0 (socket says 172.21.15.98:55404) 2024-06-11T19:13:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9985807640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99800a4920 msgr2=0x7f99801ce0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:13:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986809640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f99800a5240 0x7f9980122ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986008640 1 -- 172.21.15.98:0/2808569214 learned_addr learned my addr 172.21.15.98:0/2808569214 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986008640 10 monclient(hunting): get_auth_request con 0x7f99800a2530 auth_method 0 2024-06-11T19:13:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986008640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986008640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986008640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9985807640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99800a4920 0x7f99801ce0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986809640 10 monclient(hunting): get_auth_request con 0x7f99800a5240 auth_method 0 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986809640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986809640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986809640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986008640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986008640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986008640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986809640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986809640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.574+0000 7f9986809640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9986809640 10 monclient(hunting): handle_auth_done global_id 23557 payload 824 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9986809640 1 -- 172.21.15.98:0/2808569214 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99800a4920 msgr2=0x7f99801ce0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9986809640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99800a4920 0x7f99801ce0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9986809640 1 -- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99800a2530 msgr2=0x7f99801cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9986809640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99800a2530 0x7f99801cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9986809640 10 monclient: _finish_hunting 0 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9986809640 1 monclient: found mon.noname-c 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9986809640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9986809640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9986809640 1 -- 172.21.15.98:0/2808569214 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9980126b50 con 0x7f99800a5240 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9986008640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99800a2530 0x7f99801cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9986809640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f99800a5240 0x7f9980122ec0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f997800bdd0 tx=0x7f9978009ed0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 10 client.?.objecter ms_handle_connect 0x7f99800a5240 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f997800cbf0 con 0x7f99800a5240 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 10 monclient: dump: 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 10 monclient: _finish_auth 0 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9978010030 con 0x7f99800a5240 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f998826c640 5 monclient: authenticate success, global_id 23557 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f998826c640 10 monclient: _renew_subs 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f998826c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f998826c640 1 -- 172.21.15.98:0/2808569214 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99801256a0 con 0x7f99800a5240 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f997800cdf0 con 0x7f99800a5240 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9976ffd640 10 monclient: dump: 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f998826c640 10 client.23557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f9984805640 4 set_mon_vals no callback set 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f998826c640 10 monclient: _renew_subs 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f998826c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f998826c640 1 -- 172.21.15.98:0/2808569214 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f99800a91a0 con 0x7f99800a5240 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.575+0000 7f998826c640 1 librados: init done 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f998826c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f998826c640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123400 0x7f99801237e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f998826c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f998826c640 1 -- 172.21.15.98:0/2808569214 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f9980050830 con 0x7f9980123400 2024-06-11T19:13:54.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9986008640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123400 0x7f99801237e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9986008640 10 monclient: get_auth_request con 0x7f9980123400 auth_method 0 2024-06-11T19:13:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9986008640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9986008640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f9978006150 con 0x7f99800a5240 2024-06-11T19:13:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9986008640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9986008640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9986008640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9986008640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f996403fdb0 0x7f9964042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9986008640 10 monclient(hunting): handle_auth_done global_id 23566 payload 824 2024-06-11T19:13:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9986008640 1 -- 172.21.15.98:0/2808569214 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f99800aad30 con 0x7f9980123400 2024-06-11T19:13:54.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(185..185 src has 1..185) ==== 7479+0+0 (secure 0 0 0) 0x7f9978052250 con 0x7f99800a5240 2024-06-11T19:13:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 10 client.23557.objecter ms_dispatch 0x7f9980000b90 osd_map(185..185 src has 1..185) 2024-06-11T19:13:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 3 client.23557.objecter handle_osd_map got epochs [185,185] > 0 2024-06-11T19:13:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 3 client.23557.objecter handle_osd_map decoding full epoch 185 2024-06-11T19:13:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 20 client.23557.objecter dump_active .. 0 homeless 2024-06-11T19:13:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9986008640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123400 0x7f99801237e0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f997c0053f0 tx=0x7f997c0058b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 10 client.23557.objecter ms_handle_connect 0x7f9980123400 2024-06-11T19:13:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 10 client.23557.objecter resend_mon_ops 2024-06-11T19:13:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9985807640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f996403fdb0 0x7f9964042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f997c016070 con 0x7f9980123400 2024-06-11T19:13:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:54.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9976ffd640 10 monclient: dump: 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.605 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.576+0000 7f9985807640 10 monclient: get_auth_request con 0x7f996403fdb0 auth_method 0 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.577+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f997c00eb30 con 0x7f9980123400 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.577+0000 7f9976ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.577+0000 7f9976ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.577+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123400 msgr2=0x7f99801237e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.577+0000 7f9976ffd640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123400 0x7f99801237e0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f997c0053f0 tx=0x7f997c0058b0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.577+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123400 msgr2=0x7f99801237e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.577+0000 7f9976ffd640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123400 0x7f99801237e0 secure :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f997c0053f0 tx=0x7f997c0058b0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.577+0000 7f9976ffd640 10 client.23557.objecter ms_handle_connect 0x7f996403fdb0 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.577+0000 7f9985807640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f996403fdb0 0x7f9964042270 secure :-1 s=READY pgs=4091 cs=0 l=1 rev1=1 crypto rx=0x7f9970036000 tx=0x7f9970008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f998826c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f998826c640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123d20 0x7f998009e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f998826c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f998826c640 1 -- 172.21.15.98:0/2808569214 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f99800a4d00 con 0x7f9980123d20 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f9986008640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123d20 0x7f998009e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f9986008640 10 monclient: get_auth_request con 0x7f9980123d20 auth_method 0 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f9986008640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f9986008640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f9986008640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f9986008640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f9986008640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f9986008640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f9986008640 10 monclient(hunting): handle_auth_done global_id 23575 payload 824 2024-06-11T19:13:54.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.583+0000 7f9986008640 1 -- 172.21.15.98:0/2808569214 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f99801253a0 con 0x7f9980123d20 2024-06-11T19:13:54.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 10 client.23557.objecter ms_handle_connect 0x7f9980123d20 2024-06-11T19:13:54.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9986008640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123d20 0x7f998009e9f0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f997c0053f0 tx=0x7f997c0059d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 10 client.23557.objecter resend_mon_ops 2024-06-11T19:13:54.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f997c016040 con 0x7f9980123d20 2024-06-11T19:13:54.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:13:54.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 10 monclient: dump: 2024-06-11T19:13:54.608 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.608 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.608 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.608 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.608 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.609 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.609 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.609 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.609 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.609 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.609 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.609 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.609 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.609 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.609 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.609 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9399 (secure 0 0 0) 0x7f997c01b450 con 0x7f9980123d20 2024-06-11T19:13:54.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:54.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:54.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123d20 msgr2=0x7f998009e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123d20 0x7f998009e9f0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f997c0053f0 tx=0x7f997c0059d0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 1 -- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123d20 msgr2=0x7f998009e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:54.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9976ffd640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9980123d20 0x7f998009e9f0 secure :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f997c0053f0 tx=0x7f997c0059d0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.610 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 24, 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 284802, 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: 6 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.627 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.631 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.632 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:54.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.632 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:54.632 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:54.632 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:13:54.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.632 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.632 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:54.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.632 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:54.632 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:54.632 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:13:54.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.633 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:54.633 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:54.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 10 librados: watch_flush enter 2024-06-11T19:13:54.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 10 librados: watch_flush exit 2024-06-11T19:13:54.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 20 client.23557.objecter shutdown clearing up homeless session... 2024-06-11T19:13:54.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 10 client.23557.objecter successfully canceled tick 2024-06-11T19:13:54.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 1 -- 172.21.15.98:0/2808569214 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f996403fdb0 msgr2=0x7f9964042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f996403fdb0 0x7f9964042270 secure :-1 s=READY pgs=4091 cs=0 l=1 rev1=1 crypto rx=0x7f9970036000 tx=0x7f9970008030 comp rx=0 tx=0).stop 2024-06-11T19:13:54.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 10 monclient: shutdown 2024-06-11T19:13:54.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:54.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 1 -- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f99800a5240 msgr2=0x7f9980122ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f9986809640 1 -- 172.21.15.98:0/2808569214 reap_dead start 2024-06-11T19:13:54.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f99800a5240 0x7f9980122ec0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f997800bdd0 tx=0x7f9978009ed0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 1 -- 172.21.15.98:0/2808569214 shutdown_connections 2024-06-11T19:13:54.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 1 --2- 172.21.15.98:0/2808569214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f99800a5240 0x7f9980122ec0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 1 -- 172.21.15.98:0/2808569214 >> 172.21.15.98:0/2808569214 conn(0x7f99800ab850 msgr2=0x7f99801be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:54.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 1 -- 172.21.15.98:0/2808569214 shutdown_connections 2024-06-11T19:13:54.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.584+0000 7f998826c640 1 -- 172.21.15.98:0/2808569214 wait complete. 2024-06-11T19:13:54.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.585+0000 7f998826c640 1 librados: shutdown 2024-06-11T19:13:54.636 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-06-11T19:13:54.637 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon d 2024-06-11T19:13:54.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:13:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.736+0000 7f403b985640 10 monclient: get_monmap_and_config 2024-06-11T19:13:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.736+0000 7f403b985640 10 monclient: build_initial_monmap 2024-06-11T19:13:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.736+0000 7f403b985640 10 monclient: monmap: 2024-06-11T19:13:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:54.737894+0000 2024-06-11T19:13:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:54.737894+0000 2024-06-11T19:13:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:54.737 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.737+0000 7f403b985640 1 Processor -- start 2024-06-11T19:13:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.737+0000 7f403b985640 1 -- start start 2024-06-11T19:13:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.737+0000 7f403b985640 10 monclient: init 2024-06-11T19:13:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 10 monclient: _add_conns ranks=[6,10,8,3,4,5,2,9,1,7,0] 2024-06-11T19:13:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40340a4920 0x7f40340a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 10 monclient(hunting): picked mon.noname-d con 0x7f40340a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f40340a5240 0x7f40340a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 10 monclient(hunting): picked mon.noname-f con 0x7f40340a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40340a2530 0x7f40340a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 10 monclient(hunting): picked mon.noname-e con 0x7f40340a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f40340a7140 con 0x7f40340a5240 2024-06-11T19:13:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f40340a2b50 con 0x7f40340a4920 2024-06-11T19:13:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f40340a2e30 con 0x7f40340a2530 2024-06-11T19:13:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f403b985640 10 monclient(hunting): authenticate will time out at 1044.545156s 2024-06-11T19:13:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039f22640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40340a2530 msgr2=0x7f40340a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:13:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039f22640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40340a2530 0x7f40340a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40340a4920 0x7f40340a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40340a4920 0x7f40340a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49456/0 (socket says 172.21.15.98:49456) 2024-06-11T19:13:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 1 -- 172.21.15.98:0/1295749998 learned_addr learned my addr 172.21.15.98:0/1295749998 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 10 monclient(hunting): get_auth_request con 0x7f40340a4920 auth_method 0 2024-06-11T19:13:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 10 monclient(hunting): handle_auth_done global_id 54744 payload 274 2024-06-11T19:13:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 1 -- 172.21.15.98:0/1295749998 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40340a2530 msgr2=0x7f40340a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:54.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 1 --2- 172.21.15.98:0/1295749998 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40340a2530 0x7f40340a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 1 -- 172.21.15.98:0/1295749998 >> v2:172.21.15.117:3300/0 conn(0x7f40340a5240 msgr2=0x7f40340a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 1 --2- 172.21.15.98:0/1295749998 >> v2:172.21.15.117:3300/0 conn(0x7f40340a5240 0x7f40340a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 10 monclient: _finish_hunting 0 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 1 monclient: found mon.noname-d 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.738+0000 7f4039721640 1 -- 172.21.15.98:0/1295749998 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40340a3110 con 0x7f40340a4920 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f4039721640 1 --2- 172.21.15.98:0/1295749998 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40340a4920 0x7f40340a4d00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4030009870 tx=0x7f403002b800 comp rx=0 tx=0).ready entity=mon.6 client_cookie=636068ac8f70358b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 1 -- 172.21.15.98:0/1295749998 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40300319d0 con 0x7f40340a4920 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 10 monclient: dump: 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.751 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.752 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 10 monclient: _finish_auth 0 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f403b985640 5 monclient: authenticate success, global_id 54744 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 1 -- 172.21.15.98:0/1295749998 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4030037070 con 0x7f40340a4920 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f403b985640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 1 -- 172.21.15.98:0/1295749998 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4030031bd0 con 0x7f40340a4920 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f403b985640 4 set_mon_vals no callback set 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f402bfff640 10 monclient: dump: 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f403b985640 10 monclient: shutdown 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f403b985640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f403b985640 1 -- 172.21.15.98:0/1295749998 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40340a4920 msgr2=0x7f40340a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f403b985640 1 --2- 172.21.15.98:0/1295749998 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40340a4920 0x7f40340a4d00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4030009870 tx=0x7f403002b800 comp rx=0 tx=0).stop 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f403b985640 1 -- 172.21.15.98:0/1295749998 shutdown_connections 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f403b985640 1 --2- 172.21.15.98:0/1295749998 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40340a2530 0x7f40340a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f403b985640 1 --2- 172.21.15.98:0/1295749998 >> v2:172.21.15.117:3300/0 conn(0x7f40340a5240 0x7f40340a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.739+0000 7f403b985640 1 --2- 172.21.15.98:0/1295749998 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40340a4920 0x7f40340a4d00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.740+0000 7f403b985640 1 -- 172.21.15.98:0/1295749998 >> 172.21.15.98:0/1295749998 conn(0x7f40340ab850 msgr2=0x7f40340a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.740+0000 7f403b985640 1 -- 172.21.15.98:0/1295749998 shutdown_connections 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.740+0000 7f403b985640 1 -- 172.21.15.98:0/1295749998 wait complete. 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.740+0000 7f403b985640 10 monclient: build_initial_monmap 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.740+0000 7f403b985640 10 monclient: monmap: 2024-06-11T19:13:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.740+0000 7f403b985640 1 librados: starting msgr at 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.740+0000 7f403b985640 1 librados: starting objecter 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.740+0000 7f403b985640 1 Processor -- start 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.740+0000 7f403b985640 1 -- start start 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.740+0000 7f403b985640 1 librados: setting wanted keys 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.740+0000 7f403b985640 1 librados: calling monclient init 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.740+0000 7f403b985640 10 monclient: init 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 10 monclient: _add_conns ranks=[0,7,5,4,3,8,1,6,2] 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40340a2530 0x7f40341375d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 10 monclient(hunting): picked mon.noname-a con 0x7f40340a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:13:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f40340a4920 0x7f4034137b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 10 monclient(hunting): picked mon.noname-i con 0x7f40340a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f40340a5240 0x7f4034123080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 10 monclient(hunting): picked mon.noname-h con 0x7f40340a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f40340a6da0 con 0x7f40340a2530 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f4034138e50 con 0x7f40340a5240 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f40340a6420 con 0x7f40340a4920 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f403b985640 10 monclient(hunting): authenticate will time out at 1044.548444s 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40340a2530 0x7f40341375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40340a2530 0x7f40341375d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55410/0 (socket says 172.21.15.98:55410) 2024-06-11T19:13:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 1 -- 172.21.15.98:0/310373985 learned_addr learned my addr 172.21.15.98:0/310373985 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 10 monclient(hunting): get_auth_request con 0x7f40340a2530 auth_method 0 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4038f20640 1 -- 172.21.15.98:0/310373985 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f40340a4920 msgr2=0x7f4034137b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4038f20640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f40340a4920 0x7f4034137b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 10 monclient(hunting): handle_auth_done global_id 64323 payload 824 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 1 -- 172.21.15.98:0/310373985 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f40340a4920 msgr2=0x7f4034137b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f40340a4920 0x7f4034137b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039f22640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f40340a5240 0x7f4034123080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 1 -- 172.21.15.98:0/310373985 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f40340a5240 msgr2=0x7f4034123080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f40340a5240 0x7f4034123080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 10 monclient: _finish_hunting 0 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 1 monclient: found mon.noname-a 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 1 -- 172.21.15.98:0/310373985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40341c0bf0 con 0x7f40340a2530 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4039721640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40340a2530 0x7f40341375d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f40300090b0 tx=0x7f40300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4029ffb640 10 client.?.objecter ms_handle_connect 0x7f40340a2530 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.741+0000 7f4029ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4030041070 con 0x7f40340a2530 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:13:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 10 monclient: dump: 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 10 monclient: _finish_auth 0 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f403b985640 5 monclient: authenticate success, global_id 64323 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4030031560 con 0x7f40340a2530 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4030031b80 con 0x7f40340a2530 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f402b7fe640 4 set_mon_vals no callback set 2024-06-11T19:13:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f4029ffb640 10 monclient: dump: 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.763 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f403b985640 10 monclient: _renew_subs 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f403b985640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f403b985640 1 -- 172.21.15.98:0/310373985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4034125820 con 0x7f40340a2530 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f403b985640 10 client.64323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f403b985640 10 monclient: _renew_subs 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f403b985640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f403b985640 1 -- 172.21.15.98:0/310373985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40340a9280 con 0x7f40340a2530 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.742+0000 7f403b985640 1 librados: init done 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f403b985640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:54.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f403004c060 con 0x7f40340a2530 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f403b985640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034126950 0x7f40341235c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f403b985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f403b985640 1 -- 172.21.15.98:0/310373985 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4034050830 con 0x7f4034126950 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4038f20640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034126950 0x7f40341235c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4038f20640 10 monclient: get_auth_request con 0x7f4034126950 auth_method 0 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4038f20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4038f20640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4038f20640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4038f20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4038f20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4038f20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f401803fdb0 0x7f4018042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 7479+0+0 (secure 0 0 0) 0x7f4030073240 con 0x7f40340a2530 2024-06-11T19:13:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 10 client.64323.objecter ms_dispatch 0x7f4034000b90 osd_map(185..185 src has 1..185) 2024-06-11T19:13:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 3 client.64323.objecter handle_osd_map got epochs [185,185] > 0 2024-06-11T19:13:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 3 client.64323.objecter handle_osd_map decoding full epoch 185 2024-06-11T19:13:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4038f20640 10 monclient(hunting): handle_auth_done global_id 54753 payload 824 2024-06-11T19:13:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4038f20640 1 -- 172.21.15.98:0/310373985 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4034069290 con 0x7f4034126950 2024-06-11T19:13:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 20 client.64323.objecter dump_active .. 0 homeless 2024-06-11T19:13:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 10 client.64323.objecter ms_handle_connect 0x7f4034126950 2024-06-11T19:13:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 10 client.64323.objecter resend_mon_ops 2024-06-11T19:13:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4038f20640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034126950 0x7f40341235c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4034073fc0 tx=0x7f402400c7a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4039f22640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f401803fdb0 0x7f4018042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4024017070 con 0x7f4034126950 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4029ffb640 10 monclient: dump: 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.743+0000 7f4039f22640 10 monclient: get_auth_request con 0x7f401803fdb0 auth_method 0 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.744+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f40240072e0 con 0x7f4034126950 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.744+0000 7f4029ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.744+0000 7f4029ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.744+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034126950 msgr2=0x7f40341235c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.744+0000 7f4029ffb640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034126950 0x7f40341235c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4034073fc0 tx=0x7f402400c7a0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.744+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034126950 msgr2=0x7f40341235c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.744+0000 7f4029ffb640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034126950 0x7f40341235c0 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4034073fc0 tx=0x7f402400c7a0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.744+0000 7f4029ffb640 10 client.64323.objecter ms_handle_connect 0x7f401803fdb0 2024-06-11T19:13:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.744+0000 7f4039f22640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f401803fdb0 0x7f4018042270 secure :-1 s=READY pgs=4092 cs=0 l=1 rev1=1 crypto rx=0x7f4034125300 tx=0x7f402c0071c0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f403b985640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f403b985640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034123b00 0x7f4034123f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f403b985640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f403b985640 1 -- 172.21.15.98:0/310373985 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f40340a4d00 con 0x7f4034123b00 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f4038f20640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034123b00 0x7f4034123f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f4038f20640 10 monclient: get_auth_request con 0x7f4034123b00 auth_method 0 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f4038f20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f4038f20640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f4038f20640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f4038f20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f4038f20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f4038f20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f4038f20640 10 monclient(hunting): handle_auth_done global_id 54762 payload 824 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.750+0000 7f4038f20640 1 -- 172.21.15.98:0/310373985 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4034125570 con 0x7f4034123b00 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 10 client.64323.objecter ms_handle_connect 0x7f4034123b00 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 10 client.64323.objecter resend_mon_ops 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4038f20640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034123b00 0x7f4034123f10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4024001eb0 tx=0x7f4024000f50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4024017030 con 0x7f4034123b00 2024-06-11T19:13:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 10 monclient: dump: 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.773 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9569 (secure 0 0 0) 0x7f4024019830 con 0x7f4034123b00 2024-06-11T19:13:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034123b00 msgr2=0x7f4034123f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034123b00 0x7f4034123f10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4024001eb0 tx=0x7f4024000f50 comp rx=0 tx=0).stop 2024-06-11T19:13:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 1 -- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034123b00 msgr2=0x7f4034123f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4029ffb640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4034123b00 0x7f4034123f10 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4024001eb0 tx=0x7f4024000f50 comp rx=0 tx=0).stop 2024-06-11T19:13:54.774 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:54.774 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:54.774 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:54.774 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:54.774 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 24, 2024-06-11T19:13:54.774 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 147508, 2024-06-11T19:13:54.774 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:54.774 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:54.774 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:54.774 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:54.774 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: 6 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:54.776 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:54.777 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:54.778 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:54.779 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.783 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:54.785 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:13:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:54.796 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 10 librados: watch_flush enter 2024-06-11T19:13:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 10 librados: watch_flush exit 2024-06-11T19:13:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 20 client.64323.objecter shutdown clearing up homeless session... 2024-06-11T19:13:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 10 client.64323.objecter successfully canceled tick 2024-06-11T19:13:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 1 -- 172.21.15.98:0/310373985 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f401803fdb0 msgr2=0x7f4018042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f401803fdb0 0x7f4018042270 secure :-1 s=READY pgs=4092 cs=0 l=1 rev1=1 crypto rx=0x7f4034125300 tx=0x7f402c0071c0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 10 monclient: shutdown 2024-06-11T19:13:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 1 -- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40340a2530 msgr2=0x7f40341375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40340a2530 0x7f40341375d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f40300090b0 tx=0x7f40300046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f4039f22640 1 -- 172.21.15.98:0/310373985 reap_dead start 2024-06-11T19:13:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 1 -- 172.21.15.98:0/310373985 shutdown_connections 2024-06-11T19:13:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 1 --2- 172.21.15.98:0/310373985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40340a2530 0x7f40341375d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.751+0000 7f403b985640 1 -- 172.21.15.98:0/310373985 >> 172.21.15.98:0/310373985 conn(0x7f40340ab850 msgr2=0x7f40341be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.752+0000 7f403b985640 1 -- 172.21.15.98:0/310373985 shutdown_connections 2024-06-11T19:13:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.752+0000 7f403b985640 1 -- 172.21.15.98:0/310373985 wait complete. 2024-06-11T19:13:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.752+0000 7f403b985640 1 librados: shutdown 2024-06-11T19:13:54.799 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-06-11T19:13:54.800 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon f 2024-06-11T19:13:54.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:13:54.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.900+0000 7ffb1b492640 10 monclient: get_monmap_and_config 2024-06-11T19:13:54.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.900+0000 7ffb1b492640 10 monclient: build_initial_monmap 2024-06-11T19:13:54.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.900+0000 7ffb1b492640 10 monclient: monmap: 2024-06-11T19:13:54.899 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:54.899 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.899 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:54.901088+0000 2024-06-11T19:13:54.900 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:54.901088+0000 2024-06-11T19:13:54.900 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:54.900 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.900 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:54.900 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:54.900 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:54.900 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:54.900 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:54.900 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:54.900 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:54.901 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:54.901 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:54.901 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:54.901 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:54.901 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.900+0000 7ffb1b492640 1 Processor -- start 2024-06-11T19:13:54.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.900+0000 7ffb1b492640 1 -- start start 2024-06-11T19:13:54.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.900+0000 7ffb1b492640 10 monclient: init 2024-06-11T19:13:54.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:54.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 10 monclient: _add_conns ranks=[10,3,5,9,0,7,6,8,1,2,4] 2024-06-11T19:13:54.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ffb1409d330 0x7ffb1409d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 10 monclient(hunting): picked mon.noname-f con 0x7ffb1409d330 addr v2:172.21.15.117:3300/0 2024-06-11T19:13:54.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb1409dc50 0x7ffb14136770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 10 monclient(hunting): picked mon.noname-g con 0x7ffb1409dc50 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:13:54.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb140a2850 0x7ffb14136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 10 monclient(hunting): picked mon.noname-h con 0x7ffb140a2850 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:54.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7ffb1409bb20 con 0x7ffb1409d330 2024-06-11T19:13:54.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ffb1409b560 con 0x7ffb1409dc50 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ffb1409b840 con 0x7ffb140a2850 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1b492640 10 monclient(hunting): authenticate will time out at 1044.708433s 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb18a2d640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb1409dc50 msgr2=0x7ffb14136770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb18a2d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb1409dc50 0x7ffb14136770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1922e640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ffb1409d330 0x7ffb1409d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb19a2f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb140a2850 0x7ffb14136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1922e640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ffb1409d330 0x7ffb1409d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:36200/0 (socket says 172.21.15.98:36200) 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb19a2f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb140a2850 0x7ffb14136cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:43306/0 (socket says 172.21.15.98:43306) 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1922e640 1 -- 172.21.15.98:0/112549914 learned_addr learned my addr 172.21.15.98:0/112549914 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:54.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1922e640 10 monclient(hunting): get_auth_request con 0x7ffb1409d330 auth_method 0 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1922e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1922e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1922e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb19a2f640 10 monclient(hunting): get_auth_request con 0x7ffb140a2850 auth_method 0 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb19a2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb19a2f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb19a2f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1922e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1922e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb1922e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb19a2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb19a2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:54.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.901+0000 7ffb19a2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.902+0000 7ffb1922e640 10 monclient(hunting): handle_auth_done global_id 31627 payload 274 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.902+0000 7ffb1922e640 1 -- 172.21.15.98:0/112549914 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb140a2850 msgr2=0x7ffb14136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.902+0000 7ffb1922e640 1 --2- 172.21.15.98:0/112549914 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb140a2850 0x7ffb14136cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.902+0000 7ffb1922e640 1 -- 172.21.15.98:0/112549914 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb1409dc50 msgr2=0x7ffb14136770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.902+0000 7ffb1922e640 1 --2- 172.21.15.98:0/112549914 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb1409dc50 0x7ffb14136770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.902+0000 7ffb1922e640 10 monclient: _finish_hunting 0 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.902+0000 7ffb1922e640 1 monclient: found mon.noname-f 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.902+0000 7ffb1922e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.902+0000 7ffb1922e640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.902+0000 7ffb1922e640 1 -- 172.21.15.98:0/112549914 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb140769a0 con 0x7ffb1409d330 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.902+0000 7ffb1922e640 1 --2- 172.21.15.98:0/112549914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1409d330 0x7ffb1409d710 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009870 tx=0x7ffb1002b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=311f349d2af24211 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 1 -- 172.21.15.98:0/112549914 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb10038070 con 0x7ffb1409d330 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 10 monclient: dump: 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.912 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 10 monclient: _finish_auth 0 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 1 -- 172.21.15.98:0/112549914 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb100319b0 con 0x7ffb1409d330 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 5 monclient: authenticate success, global_id 31627 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 1 -- 172.21.15.98:0/112549914 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb1002f050 con 0x7ffb1409d330 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 4 set_mon_vals no callback set 2024-06-11T19:13:54.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb0bfff640 10 monclient: dump: 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:54.914 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 10 monclient: shutdown 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 1 -- 172.21.15.98:0/112549914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1409d330 msgr2=0x7ffb1409d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 1 --2- 172.21.15.98:0/112549914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1409d330 0x7ffb1409d710 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009870 tx=0x7ffb1002b7e0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 1 -- 172.21.15.98:0/112549914 shutdown_connections 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 1 --2- 172.21.15.98:0/112549914 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb1409dc50 0x7ffb14136770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 1 --2- 172.21.15.98:0/112549914 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb140a2850 0x7ffb14136cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 1 --2- 172.21.15.98:0/112549914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1409d330 0x7ffb1409d710 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 1 -- 172.21.15.98:0/112549914 >> 172.21.15.98:0/112549914 conn(0x7ffb140a7850 msgr2=0x7ffb140a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 1 -- 172.21.15.98:0/112549914 shutdown_connections 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.903+0000 7ffb1b492640 1 -- 172.21.15.98:0/112549914 wait complete. 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.904+0000 7ffb1b492640 10 monclient: build_initial_monmap 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.904+0000 7ffb1b492640 10 monclient: monmap: 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.904+0000 7ffb1b492640 1 librados: starting msgr at 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.904+0000 7ffb1b492640 1 librados: starting objecter 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.904+0000 7ffb1b492640 1 Processor -- start 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.904+0000 7ffb1b492640 1 -- start start 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.904+0000 7ffb1b492640 1 librados: setting wanted keys 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.904+0000 7ffb1b492640 1 librados: calling monclient init 2024-06-11T19:13:54.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.904+0000 7ffb1b492640 10 monclient: init 2024-06-11T19:13:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 10 monclient: _add_conns ranks=[7,2,8,6,1,5,3,0,4] 2024-06-11T19:13:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffb1409d330 0x7ffb141cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 10 monclient(hunting): picked mon.noname-i con 0x7ffb1409d330 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb140a2850 0x7ffb14135860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 10 monclient(hunting): picked mon.noname-b con 0x7ffb140a2850 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:13:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb14135da0 0x7ffb14130730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 10 monclient(hunting): picked mon.noname-e con 0x7ffb14135da0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ffb141371f0 con 0x7ffb140a2850 2024-06-11T19:13:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ffb1409e270 con 0x7ffb1409d330 2024-06-11T19:13:54.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:54.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ffb14137370 con 0x7ffb14135da0 2024-06-11T19:13:54.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:54.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1b492640 10 monclient(hunting): authenticate will time out at 1044.712532s 2024-06-11T19:13:54.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb18a2d640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb140a2850 msgr2=0x7ffb14135860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-06-11T19:13:54.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb18a2d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb140a2850 0x7ffb14135860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:54.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb19a2f640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb14135da0 msgr2=0x7ffb14130730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:13:54.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb19a2f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb14135da0 0x7ffb14130730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:54.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1922e640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffb1409d330 msgr2=0x7ffb141cb840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:13:54.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:54.905+0000 7ffb1922e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffb1409d330 0x7ffb141cb840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:55.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:55.106+0000 7ffb19a2f640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb14135da0 msgr2=0x7ffb14130730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:13:55.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:55.106+0000 7ffb18a2d640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb140a2850 msgr2=0x7ffb14135860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-06-11T19:13:55.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:55.106+0000 7ffb1922e640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffb1409d330 msgr2=0x7ffb141cb840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:13:55.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:55.106+0000 7ffb19a2f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb14135da0 0x7ffb14130730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:13:55.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:55.106+0000 7ffb18a2d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb140a2850 0x7ffb14135860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:13:55.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:55.106+0000 7ffb1922e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffb1409d330 0x7ffb141cb840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:13:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:55.507+0000 7ffb19a2f640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb14135da0 msgr2=0x7ffb14130730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:13:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:55.507+0000 7ffb18a2d640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb140a2850 msgr2=0x7ffb14135860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-06-11T19:13:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:55.507+0000 7ffb19a2f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb14135da0 0x7ffb14130730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:13:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:55.507+0000 7ffb18a2d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb140a2850 0x7ffb14135860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:13:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:55.507+0000 7ffb1922e640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffb1409d330 msgr2=0x7ffb141cb840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:13:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:55.507+0000 7ffb1922e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffb1409d330 0x7ffb141cb840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:13:56.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:56.308+0000 7ffb19a2f640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb14135da0 msgr2=0x7ffb14130730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:13:56.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:56.308+0000 7ffb18a2d640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb140a2850 msgr2=0x7ffb14135860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-06-11T19:13:56.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:56.308+0000 7ffb18a2d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb140a2850 0x7ffb14135860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:13:56.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:56.308+0000 7ffb19a2f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb14135da0 0x7ffb14130730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:13:56.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:56.308+0000 7ffb1922e640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffb1409d330 msgr2=0x7ffb141cb840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:13:56.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:56.308+0000 7ffb1922e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffb1409d330 0x7ffb141cb840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:13:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 10 monclient(hunting): tick 2024-06-11T19:13:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:13:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 monclient(hunting): continuing hunt 2024-06-11T19:13:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:13:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb14135da0 msgr2=0x7ffb14130730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb14135da0 0x7ffb14130730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffb1409d330 msgr2=0x7ffb141cb840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffb1409d330 0x7ffb141cb840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb140a2850 msgr2=0x7ffb14135860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:13:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffb140a2850 0x7ffb14135860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 10 monclient: _add_conns ranks=[6,0,5,4,1,3] 2024-06-11T19:13:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffafc0037f0 0x7ffafc005c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 10 monclient(hunting): picked mon.noname-d con 0x7ffafc0037f0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc006200 0x7ffafc008690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 10 monclient(hunting): picked mon.noname-a con 0x7ffafc006200 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffafc008d80 0x7ffafc00b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 10 monclient(hunting): picked mon.noname-h con 0x7ffafc008d80 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb141371f0 con 0x7ffafc006200 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ffb1409e270 con 0x7ffafc008d80 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ffb14137370 con 0x7ffafc0037f0 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.905+0000 7ffb08ff9640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffafc0037f0 0x7ffafc005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb18a2d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc006200 0x7ffafc008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffafc0037f0 0x7ffafc005c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49474/0 (socket says 172.21.15.98:49474) 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 1 -- 172.21.15.98:0/3263708373 learned_addr learned my addr 172.21.15.98:0/3263708373 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 10 monclient(hunting): get_auth_request con 0x7ffafc0037f0 auth_method 0 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb18a2d640 10 monclient(hunting): get_auth_request con 0x7ffafc006200 auth_method 0 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb18a2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb18a2d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb18a2d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb18a2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb18a2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb18a2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb19a2f640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffafc008d80 0x7ffafc00b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb19a2f640 10 monclient(hunting): get_auth_request con 0x7ffafc008d80 auth_method 0 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb19a2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb19a2f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb19a2f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 10 monclient(hunting): handle_auth_done global_id 54780 payload 824 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 1 -- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffafc008d80 msgr2=0x7ffafc00b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffafc008d80 0x7ffafc00b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 1 -- 172.21.15.98:0/3263708373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc006200 msgr2=0x7ffafc008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffafc006200 0x7ffafc008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 10 monclient: _finish_hunting 0 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 1 monclient: found mon.noname-d 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 1 -- 172.21.15.98:0/3263708373 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc00ba40 con 0x7ffafc0037f0 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb19a2f640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffafc008d80 0x7ffafc00b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb19a2f640 1 -- 172.21.15.98:0/3263708373 reap_dead start 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb1922e640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffafc0037f0 0x7ffafc005c00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009870 tx=0x7ffb100046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb09ffb640 10 client.?.objecter ms_handle_connect 0x7ffafc0037f0 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb09ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb10038050 con 0x7ffafc0037f0 2024-06-11T19:13:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.906+0000 7ffb09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb09ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb09ffb640 10 monclient: dump: 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb09ffb640 10 monclient: _finish_auth 0 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb09ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb1b492640 5 monclient: authenticate success, global_id 54780 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb10044070 con 0x7ffafc0037f0 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb09ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb1003f450 con 0x7ffafc0037f0 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb1b492640 10 monclient: _renew_subs 2024-06-11T19:13:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb1b492640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb1b492640 1 -- 172.21.15.98:0/3263708373 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb14073b00 con 0x7ffafc0037f0 2024-06-11T19:13:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb09ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:13:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb09ffb640 10 monclient: dump: 2024-06-11T19:13:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb0b7fe640 4 set_mon_vals no callback set 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb1b492640 10 client.54780.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb1b492640 10 monclient: _renew_subs 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb1b492640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb1b492640 1 -- 172.21.15.98:0/3263708373 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb14133d70 con 0x7ffafc0037f0 2024-06-11T19:13:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.907+0000 7ffb1b492640 1 librados: init done 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb1b492640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ffb1003f5f0 con 0x7ffafc0037f0 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb1b492640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1409dc50 0x7ffb14135860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb1b492640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb1b492640 1 -- 172.21.15.98:0/3263708373 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffb14131fd0 con 0x7ffb1409dc50 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb09ffb640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffaf803fd50 0x7ffaf8042210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(188..188 src has 1..188) ==== 6627+0+0 (secure 0 0 0) 0x7ffb10072f60 con 0x7ffafc0037f0 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb09ffb640 10 client.54780.objecter ms_dispatch 0x7ffb14000b90 osd_map(188..188 src has 1..188) 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb09ffb640 3 client.54780.objecter handle_osd_map got epochs [188,188] > 0 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb09ffb640 3 client.54780.objecter handle_osd_map decoding full epoch 188 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb18a2d640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1409dc50 0x7ffb14135860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb18a2d640 10 monclient: get_auth_request con 0x7ffb1409dc50 auth_method 0 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb18a2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb18a2d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.908+0000 7ffb18a2d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb19a2f640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffaf803fd50 0x7ffaf8042210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb18a2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb18a2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb18a2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb19a2f640 10 monclient: get_auth_request con 0x7ffaf803fd50 auth_method 0 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb09ffb640 20 client.54780.objecter dump_active .. 0 homeless 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb18a2d640 10 monclient(hunting): handle_auth_done global_id 31645 payload 824 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb18a2d640 1 -- 172.21.15.98:0/3263708373 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffb1409d710 con 0x7ffb1409dc50 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb09ffb640 10 client.54780.objecter ms_handle_connect 0x7ffb1409dc50 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb18a2d640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1409dc50 0x7ffb14135860 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7ffb14074fe0 tx=0x7ffb04009a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb09ffb640 10 client.54780.objecter resend_mon_ops 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb09ffb640 10 client.54780.objecter ms_handle_connect 0x7ffaf803fd50 2024-06-11T19:13:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb19a2f640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffaf803fd50 0x7ffaf8042210 secure :-1 s=READY pgs=4094 cs=0 l=1 rev1=1 crypto rx=0x7ffafc003630 tx=0x7ffb0c00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb04016070 con 0x7ffb1409dc50 2024-06-11T19:13:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb09ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:13:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.909+0000 7ffb09ffb640 10 monclient: dump: 2024-06-11T19:13:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.910+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffb04012600 con 0x7ffb1409dc50 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.910+0000 7ffb09ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.910+0000 7ffb09ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.910+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1409dc50 msgr2=0x7ffb14135860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.910+0000 7ffb09ffb640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1409dc50 0x7ffb14135860 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7ffb14074fe0 tx=0x7ffb04009a70 comp rx=0 tx=0).stop 2024-06-11T19:13:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.910+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1409dc50 msgr2=0x7ffb14135860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.910+0000 7ffb09ffb640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1409dc50 0x7ffb14135860 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.920+0000 7ffb1b492640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.920+0000 7ffb1b492640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb141cb840 0x7ffb14130730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.920+0000 7ffb1b492640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.920+0000 7ffb1b492640 1 -- 172.21.15.98:0/3263708373 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffb14050830 con 0x7ffb141cb840 2024-06-11T19:13:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.922+0000 7ffb18a2d640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb141cb840 0x7ffb14130730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.922+0000 7ffb18a2d640 10 monclient: get_auth_request con 0x7ffb141cb840 auth_method 0 2024-06-11T19:13:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.922+0000 7ffb18a2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.922+0000 7ffb18a2d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.922+0000 7ffb18a2d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.922+0000 7ffb18a2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.922+0000 7ffb18a2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.922+0000 7ffb18a2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.923+0000 7ffb18a2d640 10 monclient(hunting): handle_auth_done global_id 31654 payload 824 2024-06-11T19:13:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.923+0000 7ffb18a2d640 1 -- 172.21.15.98:0/3263708373 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffb14136180 con 0x7ffb141cb840 2024-06-11T19:13:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.923+0000 7ffb09ffb640 10 client.54780.objecter ms_handle_connect 0x7ffb141cb840 2024-06-11T19:13:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.923+0000 7ffb09ffb640 10 client.54780.objecter resend_mon_ops 2024-06-11T19:13:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.923+0000 7ffb18a2d640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb141cb840 0x7ffb14130730 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7ffb14134260 tx=0x7ffb04005ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.928+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb04016040 con 0x7ffb141cb840 2024-06-11T19:13:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.928+0000 7ffb09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.928+0000 7ffb09ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:13:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.928+0000 7ffb09ffb640 10 monclient: dump: 2024-06-11T19:13:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:57.929 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9401 (secure 0 0 0) 0x7ffb0401d0d0 con 0x7ffb141cb840 2024-06-11T19:13:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb09ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb09ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb141cb840 msgr2=0x7ffb14130730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb09ffb640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb141cb840 0x7ffb14130730 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7ffb14134260 tx=0x7ffb04005ac0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb09ffb640 1 -- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb141cb840 msgr2=0x7ffb14130730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb09ffb640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb141cb840 0x7ffb14130730 secure :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7ffb14134260 tx=0x7ffb04005ac0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.931 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:57.931 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:57.931 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:57.931 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:13:57.931 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 24, 2024-06-11T19:13:57.931 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 288147, 2024-06-11T19:13:57.931 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:57.931 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:57.931 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:57.931 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:57.932 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:57.932 INFO:teuthology.orchestra.run.smithi098.stdout: 6 2024-06-11T19:13:57.932 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:57.932 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:13:57.932 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:57.932 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:57.932 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:57.932 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:57.932 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:57.932 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:57.932 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:57.932 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:57.933 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:57.934 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:57.934 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:57.934 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:57.934 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:57.934 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:57.934 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:57.934 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:57.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.934 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:57.934 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:57.934 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:57.934 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:57.935 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:57.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.937 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:57.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:57.938 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:57.938 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:57.938 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:57.938 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:57.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.938 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:57.938 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:57.938 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:57.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:13:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:57.954 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:57.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb1b492640 10 librados: watch_flush enter 2024-06-11T19:13:57.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb1b492640 10 librados: watch_flush exit 2024-06-11T19:13:57.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb1b492640 20 client.54780.objecter shutdown clearing up homeless session... 2024-06-11T19:13:57.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb1b492640 10 client.54780.objecter successfully canceled tick 2024-06-11T19:13:57.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb1b492640 1 -- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffaf803fd50 msgr2=0x7ffaf8042210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:57.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb1b492640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffaf803fd50 0x7ffaf8042210 secure :-1 s=READY pgs=4094 cs=0 l=1 rev1=1 crypto rx=0x7ffafc003630 tx=0x7ffb0c00b040 comp rx=0 tx=0).stop 2024-06-11T19:13:57.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb1b492640 10 monclient: shutdown 2024-06-11T19:13:57.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb1b492640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:57.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb1b492640 1 -- 172.21.15.98:0/3263708373 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffafc0037f0 msgr2=0x7ffafc005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:57.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.929+0000 7ffb1b492640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffafc0037f0 0x7ffafc005c00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009870 tx=0x7ffb100046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.930+0000 7ffb1b492640 1 -- 172.21.15.98:0/3263708373 shutdown_connections 2024-06-11T19:13:57.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.930+0000 7ffb1b492640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffaf803fd50 0x7ffaf8042210 unknown :-1 s=CLOSED pgs=4094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.930+0000 7ffb1b492640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffafc0037f0 0x7ffafc005c00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.930+0000 7ffb1b492640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1409dc50 0x7ffb14135860 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.930+0000 7ffb1b492640 1 --2- 172.21.15.98:0/3263708373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb141cb840 0x7ffb14130730 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:57.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.930+0000 7ffb1b492640 1 -- 172.21.15.98:0/3263708373 >> 172.21.15.98:0/3263708373 conn(0x7ffb140a7850 msgr2=0x7ffb140a5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:57.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.930+0000 7ffb1b492640 1 -- 172.21.15.98:0/3263708373 shutdown_connections 2024-06-11T19:13:57.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.930+0000 7ffb1b492640 1 -- 172.21.15.98:0/3263708373 wait complete. 2024-06-11T19:13:57.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:57.930+0000 7ffb1b492640 1 librados: shutdown 2024-06-11T19:13:57.957 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-06-11T19:13:57.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon h 2024-06-11T19:13:57.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:13:58.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.061+0000 7efd1c385640 10 monclient: get_monmap_and_config 2024-06-11T19:13:58.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.061+0000 7efd1c385640 10 monclient: build_initial_monmap 2024-06-11T19:13:58.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.061+0000 7efd1c385640 10 monclient: monmap: 2024-06-11T19:13:58.061 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:58.061 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:58.061 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:13:58.062679+0000 2024-06-11T19:13:58.061 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:13:58.062679+0000 2024-06-11T19:13:58.061 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:58.061 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:58.062 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:13:58.062 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:13:58.062 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:58.062 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:58.062 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:58.062 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:58.062 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:58.062 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:58.062 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:58.062 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:13:58.062 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:13:58.063 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:58.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 1 Processor -- start 2024-06-11T19:13:58.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 1 -- start start 2024-06-11T19:13:58.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 10 monclient: init 2024-06-11T19:13:58.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:58.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 10 monclient: _add_conns ranks=[5,7,8,1,10,4,2,0,9,6,3] 2024-06-11T19:13:58.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd140a4920 0x7efd140a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:58.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 10 monclient(hunting): picked mon.noname-h con 0x7efd140a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:58.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efd140a5240 0x7efd140a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:58.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 10 monclient(hunting): picked mon.noname-i con 0x7efd140a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:13:58.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efd140a2530 0x7efd140a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:58.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 10 monclient(hunting): picked mon.noname-e con 0x7efd140a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:13:58.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:58.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7efd140a7140 con 0x7efd140a4920 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7efd140a2b50 con 0x7efd140a5240 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7efd140a2e30 con 0x7efd140a2530 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1c385640 10 monclient(hunting): authenticate will time out at 1047.869929s 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.062+0000 7efd1a922640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efd140a2530 msgr2=0x7efd140a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a922640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efd140a2530 0x7efd140a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd19920640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efd140a5240 msgr2=0x7efd140a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd19920640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efd140a5240 0x7efd140a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd140a4920 0x7efd140a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd140a4920 0x7efd140a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:43318/0 (socket says 172.21.15.98:43318) 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 1 -- 172.21.15.98:0/1132758671 learned_addr learned my addr 172.21.15.98:0/1132758671 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 10 monclient(hunting): get_auth_request con 0x7efd140a4920 auth_method 0 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 10 monclient(hunting): handle_auth_done global_id 22568 payload 274 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 1 -- 172.21.15.98:0/1132758671 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efd140a2530 msgr2=0x7efd140a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 1 --2- 172.21.15.98:0/1132758671 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efd140a2530 0x7efd140a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 1 -- 172.21.15.98:0/1132758671 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efd140a5240 msgr2=0x7efd140a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 1 --2- 172.21.15.98:0/1132758671 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efd140a5240 0x7efd140a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 10 monclient: _finish_hunting 0 2024-06-11T19:13:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 1 monclient: found mon.noname-h 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.063+0000 7efd1a121640 1 -- 172.21.15.98:0/1132758671 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd140a3110 con 0x7efd140a4920 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1a121640 1 --2- 172.21.15.98:0/1132758671 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd140a4920 0x7efd140a4d00 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7efd100098a0 tx=0x7efd1002b7e0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=51fa11f7297bf115 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 1 -- 172.21.15.98:0/1132758671 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efd10038070 con 0x7efd140a4920 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 10 monclient: dump: 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 10 monclient: _finish_auth 0 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 10 monclient: _check_auth_tickets 2024-06-11T19:13:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 1 -- 172.21.15.98:0/1132758671 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd100319b0 con 0x7efd140a4920 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 1 -- 172.21.15.98:0/1132758671 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efd1002f050 con 0x7efd140a4920 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1c385640 5 monclient: authenticate success, global_id 22568 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1c385640 10 monclient: get_monmap_and_config success 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1c385640 4 set_mon_vals no callback set 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.064+0000 7efd1911f640 10 monclient: dump: 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 10 monclient: shutdown 2024-06-11T19:13:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 1 -- 172.21.15.98:0/1132758671 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd140a4920 msgr2=0x7efd140a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 1 --2- 172.21.15.98:0/1132758671 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd140a4920 0x7efd140a4d00 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7efd100098a0 tx=0x7efd1002b7e0 comp rx=0 tx=0).stop 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 1 -- 172.21.15.98:0/1132758671 shutdown_connections 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 1 --2- 172.21.15.98:0/1132758671 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efd140a5240 0x7efd140a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 1 --2- 172.21.15.98:0/1132758671 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efd140a2530 0x7efd140a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 1 --2- 172.21.15.98:0/1132758671 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd140a4920 0x7efd140a4d00 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 1 -- 172.21.15.98:0/1132758671 >> 172.21.15.98:0/1132758671 conn(0x7efd140ab850 msgr2=0x7efd140a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 1 -- 172.21.15.98:0/1132758671 shutdown_connections 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 1 -- 172.21.15.98:0/1132758671 wait complete. 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 10 monclient: build_initial_monmap 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.065+0000 7efd1c385640 10 monclient: monmap: 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:13:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 1 librados: starting msgr at 2024-06-11T19:13:58.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 1 librados: starting objecter 2024-06-11T19:13:58.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 1 Processor -- start 2024-06-11T19:13:58.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 1 -- start start 2024-06-11T19:13:58.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 1 librados: setting wanted keys 2024-06-11T19:13:58.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 1 librados: calling monclient init 2024-06-11T19:13:58.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 10 monclient: init 2024-06-11T19:13:58.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 10 monclient: _reopen_session rank -1 2024-06-11T19:13:58.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 10 monclient: _add_conns ranks=[6,0,5,8,1,4,7,3,2] 2024-06-11T19:13:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7efd140a2530 0x7efd14136c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 10 monclient(hunting): picked mon.noname-d con 0x7efd140a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:13:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd140a4920 0x7efd141371c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 10 monclient(hunting): picked mon.noname-a con 0x7efd140a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:13:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd140a5240 0x7efd141226f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 10 monclient(hunting): picked mon.noname-h con 0x7efd140a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:13:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd140a6da0 con 0x7efd140a4920 2024-06-11T19:13:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7efd14138e50 con 0x7efd140a5240 2024-06-11T19:13:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7efd140a6420 con 0x7efd140a2530 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 10 monclient(hunting): _renew_subs 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1c385640 10 monclient(hunting): authenticate will time out at 1047.873699s 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a121640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7efd140a2530 0x7efd14136c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a121640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7efd140a2530 0x7efd14136c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49482/0 (socket says 172.21.15.98:49482) 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a121640 1 -- 172.21.15.98:0/2438430338 learned_addr learned my addr 172.21.15.98:0/2438430338 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a121640 10 monclient(hunting): get_auth_request con 0x7efd140a2530 auth_method 0 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a121640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a121640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a121640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd19920640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd140a4920 0x7efd141371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd19920640 10 monclient(hunting): get_auth_request con 0x7efd140a4920 auth_method 0 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd19920640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd19920640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd19920640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a121640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a121640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a121640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd19920640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd19920640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd19920640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a922640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd140a5240 0x7efd141226f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a922640 10 monclient(hunting): get_auth_request con 0x7efd140a5240 auth_method 0 2024-06-11T19:13:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a922640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a922640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.066+0000 7efd1a922640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd19920640 10 monclient(hunting): handle_auth_done global_id 64350 payload 824 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd19920640 1 -- 172.21.15.98:0/2438430338 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7efd140a2530 msgr2=0x7efd14136c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd19920640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7efd140a2530 0x7efd14136c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd19920640 1 -- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd140a5240 msgr2=0x7efd141226f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd19920640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd140a5240 0x7efd141226f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd19920640 10 monclient: _finish_hunting 0 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd19920640 1 monclient: found mon.noname-a 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd19920640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd19920640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd19920640 1 -- 172.21.15.98:0/2438430338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd14126660 con 0x7efd140a4920 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd1a121640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7efd140a2530 0x7efd14136c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd1a922640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd140a5240 0x7efd141226f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd19920640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd140a4920 0x7efd141371c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efd04003930 tx=0x7efd0400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 10 client.?.objecter ms_handle_connect 0x7efd140a4920 2024-06-11T19:13:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efd040092b0 con 0x7efd140a4920 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 10 monclient: dump: 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 10 monclient: _finish_auth 0 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 10 monclient: _check_auth_tickets 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd1c385640 5 monclient: authenticate success, global_id 64350 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd04015070 con 0x7efd140a4920 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd1c385640 10 monclient: _renew_subs 2024-06-11T19:13:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd1c385640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:13:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd1c385640 1 -- 172.21.15.98:0/2438430338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd14126380 con 0x7efd140a4920 2024-06-11T19:13:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:13:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efd04003bc0 con 0x7efd140a4920 2024-06-11T19:13:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:13:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd0affd640 10 monclient: dump: 2024-06-11T19:13:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:58.090 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd1891e640 4 set_mon_vals no callback set 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd1c385640 10 client.64350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd1c385640 10 monclient: _renew_subs 2024-06-11T19:13:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd1c385640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd1c385640 1 -- 172.21.15.98:0/2438430338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd14125a90 con 0x7efd140a4920 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.067+0000 7efd1c385640 1 librados: init done 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1c385640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1c385640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14122c30 0x7efd14123010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1c385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1c385640 1 -- 172.21.15.98:0/2438430338 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7efd140a93f0 con 0x7efd14122c30 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7efd0400f070 con 0x7efd140a4920 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1a121640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14122c30 0x7efd14123010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1a121640 10 monclient: get_auth_request con 0x7efd14122c30 auth_method 0 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1a121640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1a121640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1a121640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd0affd640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efcf803fdb0 0x7efcf8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:58.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1a121640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:58.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1a121640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:58.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1a121640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:58.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6736+0+0 (secure 0 0 0) 0x7efd04051320 con 0x7efd140a4920 2024-06-11T19:13:58.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd0affd640 10 client.64350.objecter ms_dispatch 0x7efd14000b90 osd_map(189..189 src has 1..189) 2024-06-11T19:13:58.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd0affd640 3 client.64350.objecter handle_osd_map got epochs [189,189] > 0 2024-06-11T19:13:58.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd0affd640 3 client.64350.objecter handle_osd_map decoding full epoch 189 2024-06-11T19:13:58.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd0affd640 20 client.64350.objecter dump_active .. 0 homeless 2024-06-11T19:13:58.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1a121640 10 monclient(hunting): handle_auth_done global_id 22586 payload 824 2024-06-11T19:13:58.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1a922640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efcf803fdb0 0x7efcf8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:58.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1a121640 1 -- 172.21.15.98:0/2438430338 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd141ce8a0 con 0x7efd14122c30 2024-06-11T19:13:58.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.068+0000 7efd1a922640 10 monclient: get_auth_request con 0x7efcf803fdb0 auth_method 0 2024-06-11T19:13:58.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 10 client.64350.objecter ms_handle_connect 0x7efd14122c30 2024-06-11T19:13:58.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 10 client.64350.objecter resend_mon_ops 2024-06-11T19:13:58.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd1a121640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14122c30 0x7efd14123010 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7efd100097c0 tx=0x7efd100046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:58.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efd10044070 con 0x7efd14122c30 2024-06-11T19:13:58.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:58.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:13:58.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 10 monclient: dump: 2024-06-11T19:13:58.094 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:58.094 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:58.096 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd1a922640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efcf803fdb0 0x7efcf8042270 secure :-1 s=READY pgs=4096 cs=0 l=1 rev1=1 crypto rx=0x7efd14123e50 tx=0x7efd0c0094d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 10 client.64350.objecter ms_handle_connect 0x7efcf803fdb0 2024-06-11T19:13:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efd10004480 con 0x7efd14122c30 2024-06-11T19:13:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:13:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:13:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14122c30 msgr2=0x7efd14123010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14122c30 0x7efd14123010 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7efd100097c0 tx=0x7efd100046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14122c30 msgr2=0x7efd14123010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.069+0000 7efd0affd640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14122c30 0x7efd14123010 secure :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7efd100097c0 tx=0x7efd100046c0 comp rx=0 tx=0).stop 2024-06-11T19:13:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.075+0000 7efd1c385640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.075+0000 7efd1c385640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14123550 0x7efd1409e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.075+0000 7efd1c385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.075+0000 7efd1c385640 1 -- 172.21.15.98:0/2438430338 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7efd14050830 con 0x7efd14123550 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd1a121640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14123550 0x7efd1409e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd1a121640 10 monclient: get_auth_request con 0x7efd14123550 auth_method 0 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd1a121640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd1a121640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd1a121640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd1a121640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd1a121640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd1a121640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd1a121640 10 monclient(hunting): handle_auth_done global_id 22595 payload 824 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd1a121640 1 -- 172.21.15.98:0/2438430338 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efd140a8ec0 con 0x7efd14123550 2024-06-11T19:13:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd1a121640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14123550 0x7efd1409e9f0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7efd10043000 tx=0x7efd10031000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:13:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd0affd640 10 client.64350.objecter ms_handle_connect 0x7efd14123550 2024-06-11T19:13:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd0affd640 10 client.64350.objecter resend_mon_ops 2024-06-11T19:13:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efd1002f040 con 0x7efd14123550 2024-06-11T19:13:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:13:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd0affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:13:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd0affd640 10 monclient: dump: 2024-06-11T19:13:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:13:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:13:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:13:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:13:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9399 (secure 0 0 0) 0x7efd100498e0 con 0x7efd14123550 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd0affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd0affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14123550 msgr2=0x7efd1409e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.076+0000 7efd0affd640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14123550 0x7efd1409e9f0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7efd10043000 tx=0x7efd10031000 comp rx=0 tx=0).stop 2024-06-11T19:13:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd0affd640 1 -- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14123550 msgr2=0x7efd1409e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:13:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd0affd640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efd14123550 0x7efd1409e9f0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:13:58.102 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:13:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:13:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 24, 2024-06-11T19:13:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 288270, 2024-06-11T19:13:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: 6 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:13:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:13:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:13:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:13:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:13:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:13:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:13:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:13:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:13:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:13:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:13:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:13:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:13:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:13:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:13:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:13:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:13:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:13:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:13:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:13:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:13:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:13:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:13:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:13:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:13:58.125 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:13:58.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 10 librados: watch_flush enter 2024-06-11T19:13:58.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 10 librados: watch_flush exit 2024-06-11T19:13:58.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 20 client.64350.objecter shutdown clearing up homeless session... 2024-06-11T19:13:58.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 10 client.64350.objecter successfully canceled tick 2024-06-11T19:13:58.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 1 -- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efcf803fdb0 msgr2=0x7efcf8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:58.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efcf803fdb0 0x7efcf8042270 secure :-1 s=READY pgs=4096 cs=0 l=1 rev1=1 crypto rx=0x7efd14123e50 tx=0x7efd0c0094d0 comp rx=0 tx=0).stop 2024-06-11T19:13:58.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 10 monclient: shutdown 2024-06-11T19:13:58.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:13:58.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 1 -- 172.21.15.98:0/2438430338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd140a4920 msgr2=0x7efd141371c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:13:58.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 1 --2- 172.21.15.98:0/2438430338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd140a4920 0x7efd141371c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efd04003930 tx=0x7efd0400b900 comp rx=0 tx=0).stop 2024-06-11T19:13:58.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1a922640 1 -- 172.21.15.98:0/2438430338 reap_dead start 2024-06-11T19:13:58.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1a922640 1 -- 172.21.15.98:0/2438430338 reap_dead start 2024-06-11T19:13:58.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 1 -- 172.21.15.98:0/2438430338 shutdown_connections 2024-06-11T19:13:58.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 1 -- 172.21.15.98:0/2438430338 >> 172.21.15.98:0/2438430338 conn(0x7efd140ab850 msgr2=0x7efd14138150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:13:58.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 1 -- 172.21.15.98:0/2438430338 shutdown_connections 2024-06-11T19:13:58.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 1 -- 172.21.15.98:0/2438430338 wait complete. 2024-06-11T19:13:58.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:13:58.077+0000 7efd1c385640 1 librados: shutdown 2024-06-11T19:13:58.128 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-06-11T19:13:58.128 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-06-11T19:14:16.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-06-11T19:14:16.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-06-11T19:14:16.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-06-11T19:14:16.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: seed 36854 2024-06-11T19:14:16.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-06-11T19:14:16.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (291 ms) 2024-06-11T19:14:16.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-06-11T19:14:16.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10637 ms) 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (13 ms) 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10941 ms total) 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7024 ms) 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7991 ms) 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9995 ms) 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-06-11T19:14:16.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: my_snaps [3] 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: my_snaps [4,3] 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: my_snaps [5,4,3] 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: promoting some heads 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: promoting from clones for snap 6 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: promoting from clones for snap 5 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: promoting from clones for snap 4 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: promoting from clones for snap 3 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: waiting for scrubs... 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: done waiting 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (47018 ms) 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-06-11T19:14:16.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10003 ms) 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8004 ms) 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8015 ms) 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (7995 ms) 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10000 ms) 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9012 ms) 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10046 ms) 2024-06-11T19:14:16.878 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12961 ms) 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7982 ms) 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8016 ms) 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (26993 ms) 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8006 ms) 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8016 ms) 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSe api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7026 ms) 2024-06-11T19:14:16.879 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6997 ms) 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6991 ms) 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7025 ms) 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6979 ms) 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (6996 ms) 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7000 ms) 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7055 ms) 2024-06-11T19:14:16.880 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6956 ms) 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6999 ms) 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7015 ms) 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7003 ms) 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (21050 ms) 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6942 ms) 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-06-11T19:14:16.881 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6996 ms) 2024-06-11T19:14:16.882 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-06-11T19:14:16.882 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7015 ms) 2024-06-11T19:14:16.882 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (126046 ms total) 2024-06-11T19:14:16.882 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: 2024-06-11T19:14:16.882 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [----------] Global test environment tear-down 2024-06-11T19:14:16.882 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (222813 ms total) 2024-06-11T19:14:16.882 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio: [ PASSED ] 42 tests. 2024-06-11T19:14:32.145 INFO:tasks.workunit.client.0.smithi117.stdout:tRead 2024-06-11T19:14:32.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: hmm, no HitSet yet 2024-06-11T19:14:32.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: ok, hit_set contains 267:602f83fe:::foo:head 2024-06-11T19:14:32.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8992 ms) 2024-06-11T19:14:32.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-06-11T19:14:32.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg_num = 32 2024-06-11T19:14:32.145 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 0 ls 1718133272,0 2024-06-11T19:14:32.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 1 ls 1718133272,0 2024-06-11T19:14:32.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 2 ls 1718133272,0 2024-06-11T19:14:32.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 3 ls 1718133272,0 2024-06-11T19:14:32.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 4 ls 1718133272,0 2024-06-11T19:14:32.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 5 ls 1718133272,0 2024-06-11T19:14:32.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 6 ls 1718133272,0 2024-06-11T19:14:32.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 7 ls 1718133272,0 2024-06-11T19:14:32.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 8 ls 1718133272,0 2024-06-11T19:14:32.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 9 ls 1718133272,0 2024-06-11T19:14:32.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 10 ls 1718133272,0 2024-06-11T19:14:32.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 11 ls 1718133272,0 2024-06-11T19:14:32.146 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 12 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 13 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 14 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 15 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 16 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 17 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 18 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 19 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 20 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 21 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 22 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 23 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 24 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 25 ls 1718133272,0 2024-06-11T19:14:32.147 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 26 ls 1718133272,0 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 27 ls 1718133272,0 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 28 ls 1718133272,0 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 29 ls 1718133272,0 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 30 ls 1718133272,0 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg 31 ls 1718133272,0 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: pg_num = 32 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:6cac518f:::0:head 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:02547ec2:::1:head 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:f905c69b:::2:head 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:cfc208b3:::3:head 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:d83876eb:::4:head 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:b29083e3:::5:head 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:c4fdafeb:::6:head 2024-06-11T19:14:32.148 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:5c6b0b28:::7:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:bd63b0f1:::8:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:e960b815:::9:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:52ea6a34:::10:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:89d3ae78:::11:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:de5d7c5f:::12:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:566253c9:::13:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:62a1935d:::14:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:863748b0:::15:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:3958e169:::16:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:4d4dabf9:::17:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:8391935d:::18:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:28883081:::19:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:69259c59:::20:head 2024-06-11T19:14:32.149 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:4bdb80b7:::21:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:a11c5d71:::22:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:271af37b:::23:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:95b121be:::24:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:58d1031b:::25:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:0a050783:::26:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:c709704c:::27:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:cbe56eaf:::28:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:86b4b162:::29:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:70d89383:::30:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:dd450c7c:::31:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:6d5729b1:::32:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:c388f3fb:::33:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:56cfea31:::34:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:9dbc1bf7:::35:head 2024-06-11T19:14:32.150 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:40b74ccd:::36:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:4d5aaf42:::37:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:920f362c:::38:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:6cc53222:::39:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:9cad833f:::40:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:1ea84d41:::41:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:c4480ef6:::42:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:a694361e:::43:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:d1bd33e9:::44:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:ddc2cd5d:::45:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:2b782207:::46:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:7b187fca:::47:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:90ecdf6f:::48:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:a5ed95fe:::49:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:ea0eaa55:::50:head 2024-06-11T19:14:32.151 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:f33ef17b:::51:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:a0d1b2f6:::52:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:60c5229e:::53:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:edcbc575:::54:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:102cf253:::55:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:efb7fb0b:::56:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:50d0a326:::57:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:d4dc5daf:::58:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:3a130462:::59:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:ec87ed71:::60:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:d5bc9454:::61:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:3ddfe313:::62:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:7c2816b9:::63:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:47e00e4d:::64:head 2024-06-11T19:14:32.152 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:c6410c18:::65:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:b48ed237:::66:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:cd63ad31:::67:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:b179e92b:::68:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:0d9f741a:::69:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:6d3352ae:::70:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:c6d5c19e:::71:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:bc4729c3:::72:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:77e930b9:::73:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:0abeecfd:::74:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:b7c37e15:::75:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:b6378398:::76:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:02bd68de:::77:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:cc795d2d:::78:head 2024-06-11T19:14:32.153 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:630d4fea:::79:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:e0d29ef5:::80:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:fd6f13d2:::81:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:606461d5:::82:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:eadbdc43:::83:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:8761d0bb:::84:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:9ef0186f:::85:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:e0d41294:::86:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:961de695:::87:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:1423148f:::88:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:633a8fa2:::89:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:a8653809:::90:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:3dac8b33:::91:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:35aad435:::92:head 2024-06-11T19:14:32.154 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:f6dcc343:::93:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:dbbdad87:::94:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:1cb48ce0:::95:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:03cd461c:::96:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:17a4ea99:::97:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:9993c9a7:::98:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:6394211c:::99:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:94c7ae57:::100:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:6fdee5bb:::101:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:9a477fd1:::102:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:eb850916:::103:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:affc56b9:::104:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:b42dc814:::105:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:f319f8f0:::106:head 2024-06-11T19:14:32.155 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:9a40b9de:::107:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:8b524f28:::108:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:e3de589f:::109:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:90f90a5b:::110:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:a7b4f1d7:::111:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:af51766e:::112:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:b6f90bd1:::113:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:e0261208:::114:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:c9569ef7:::115:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:61bebe50:::116:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:fe93412b:::117:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:d3d38bee:::118:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:3100ba0c:::119:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:d0560ada:::120:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:f0ea8b35:::121:head 2024-06-11T19:14:32.156 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:766f231a:::122:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:a07a2582:::123:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:bd7c6b3a:::124:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:fb2ddaff:::125:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:4408e1fe:::126:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:ee1df7a7:::127:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:c3002909:::128:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:4f48ffa9:::129:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:edf38733:::130:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:c08425c0:::131:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:5f902d98:::132:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:41ea2c93:::133:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:813cee13:::134:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:0131818d:::135:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:26ba5a85:::136:head 2024-06-11T19:14:32.157 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:381b8a5a:::137:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:28797e47:::138:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:bfca7f22:::139:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:36807075:::140:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:80b03975:::141:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:5c15709b:::142:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:f39ea15e:::143:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:ea992956:::144:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:48887b1c:::145:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:9f24a9dd:::146:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:987f100b:::147:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:d2dd3581:::148:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:7fed1808:::149:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:c80b70e9:::150:head 2024-06-11T19:14:32.158 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:85ed90f9:::151:head 2024-06-11T19:14:32.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:36428b24:::152:head 2024-06-11T19:14:32.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:d044c34a:::153:head 2024-06-11T19:14:32.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:7c18bf58:::154:head 2024-06-11T19:14:32.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:d1c21232:::155:head 2024-06-11T19:14:32.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:a7a3c575:::156:head 2024-06-11T19:14:32.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:87da0633:::157:head 2024-06-11T19:14:32.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:d5ac3822:::158:head 2024-06-11T19:14:32.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:3f20522d:::159:head 2024-06-11T19:14:32.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:6ca26563:::160:head 2024-06-11T19:14:32.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:532ce135:::161:head 2024-06-11T19:14:32.159 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:c78863e6:::162:head 2024-06-11T19:15:13.872 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:7462ddf6:::o.RoundTripAppendPP (3011 ms) 2024-06-11T19:15:13.873 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-06-11T19:15:13.873 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3011 ms) 2024-06-11T19:15:13.873 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-06-11T19:15:13.873 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2977 ms) 2024-06-11T19:15:13.873 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-06-11T19:15:13.873 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3006 ms) 2024-06-11T19:15:13.873 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-06-11T19:15:13.873 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: setting pool EIO 2024-06-11T19:15:13.873 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: max_success 98, min_failed 99 2024-06-11T19:15:13.873 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4018 ms) 2024-06-11T19:15:13.874 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-06-11T19:15:13.874 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (2982 ms) 2024-06-11T19:15:13.874 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (135774 ms total) 2024-06-11T19:15:13.874 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: 2024-06-11T19:15:13.874 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-06-11T19:15:13.874 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-06-11T19:15:13.874 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3000 ms) 2024-06-11T19:15:13.874 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-06-11T19:15:13.874 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9009 ms) 2024-06-11T19:15:13.874 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-06-11T19:15:13.874 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15017 ms) 2024-06-11T19:15:13.874 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (2980 ms) 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30006 ms total) 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3012 ms) 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3012 ms total) 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14012 ms) 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (21064 ms) 2024-06-11T19:15:13.875 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6922 ms) 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7005 ms) 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3012 ms) 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6990 ms) 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7029 ms) 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6996 ms) 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6985 ms) 2024-06-11T19:15:13.876 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-06-11T19:15:13.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6989 ms) 2024-06-11T19:15:13.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-06-11T19:15:13.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7060 ms) 2024-06-11T19:15:13.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-06-11T19:15:13.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6956 ms) 2024-06-11T19:15:13.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-06-11T19:15:13.877 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (9995 ms) 2024-06-11T19:15:27.913 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2163:head 2024-06-11T19:15:27.913 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:5d165639:::164:head 2024-06-11T19:15:27.913 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:f43765fc:::165:head 2024-06-11T19:15:27.913 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:b4c720e9:::166:head 2024-06-11T19:15:27.913 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:e694b040:::167:head 2024-06-11T19:15:27.913 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:afa38db2:::168:head 2024-06-11T19:15:27.913 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:77ba9f53:::169:head 2024-06-11T19:15:27.913 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:87495034:::170:head 2024-06-11T19:15:27.913 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:7c96bf0e:::171:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:dbe346cc:::172:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:e943ec24:::173:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:f97a9c0c:::174:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:6f26e74d:::175:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:4f95e106:::176:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:0e6f2f8f:::177:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:05db05f1:::178:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:38a78d66:::179:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:d095610b:::180:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:a1a9d709:::181:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:1e5d39db:::182:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:f7df4fb9:::183:head 2024-06-11T19:15:27.914 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:03a7f161:::184:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:ba70721e:::185:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:28e5662d:::186:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:973d52de:::187:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:4303eb1c:::188:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:b990b48e:::189:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:29b8165b:::190:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:3547f197:::191:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:7e260936:::192:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:1abec7b1:::193:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:10fdda93:::194:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:15817eea:::195:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:770bab57:::196:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:ed9e13e7:::197:head 2024-06-11T19:15:27.915 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:71471a8f:::198:head 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: checking for 269:10fb1d02:::199:head 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8008 ms) 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133281,0 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: first is 1718133281 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133281,0 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133281,0 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133281,1718133283,1718133284,0 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133281,1718133283,1718133284,0 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133281,1718133283,1718133284,0 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133281,1718133283,1718133284,1718133286,1718133287,0 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133281,1718133283,1718133284,1718133286,1718133287,0 2024-06-11T19:15:27.916 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133281,1718133283,1718133284,1718133286,1718133287,0 2024-06-11T19:15:27.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133281,1718133283,1718133284,1718133286,1718133287,1718133289,1718133290,0 2024-06-11T19:15:27.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133281,1718133283,1718133284,1718133286,1718133287,1718133289,1718133290,0 2024-06-11T19:15:27.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133281,1718133283,1718133284,1718133286,1718133287,1718133289,1718133290,0 2024-06-11T19:15:27.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133284,1718133286,1718133287,1718133289,1718133290,1718133292,1718133293,0 2024-06-11T19:15:27.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: first now 1718133284, trimmed 2024-06-11T19:15:27.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21016 ms) 2024-06-11T19:15:27.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-06-11T19:15:27.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: foo0 2024-06-11T19:15:27.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-06-11T19:15:27.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14023 ms) 2024-06-11T19:15:27.917 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-06-11T19:15:28.130 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-06-11T19:15:28.130 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-06-11T19:15:28.130 INFO:tasks.ceph.mon.b:Restarting daemon 2024-06-11T19:15:28.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-06-11T19:15:28.134 INFO:tasks.ceph.mon.b:Started 2024-06-11T19:15:28.134 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-06-11T19:15:28.134 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-06-11T19:15:28.134 INFO:tasks.ceph.mon.g:Restarting daemon 2024-06-11T19:15:28.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-06-11T19:15:28.136 INFO:tasks.ceph.mon.g:Started 2024-06-11T19:15:28.136 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-06-11T19:15:28.137 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-06-11T19:15:28.137 INFO:tasks.ceph.mon.e:Restarting daemon 2024-06-11T19:15:28.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-06-11T19:15:28.139 INFO:tasks.ceph.mon.e:Started 2024-06-11T19:15:28.139 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-06-11T19:15:28.139 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-06-11T19:15:28.139 INFO:tasks.ceph.mon.i:Restarting daemon 2024-06-11T19:15:28.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-06-11T19:15:28.141 INFO:tasks.ceph.mon.i:Started 2024-06-11T19:15:28.141 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:15:28.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:15:28.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.251+0000 7f6b21054640 10 monclient: get_monmap_and_config 2024-06-11T19:15:28.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.251+0000 7f6b21054640 10 monclient: build_initial_monmap 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.251+0000 7f6b21054640 10 monclient: monmap: 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:28.252345+0000 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:28.252345+0000 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:28.252 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:28.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.251+0000 7f6b21054640 1 Processor -- start 2024-06-11T19:15:28.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.251+0000 7f6b21054640 1 -- start start 2024-06-11T19:15:28.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.251+0000 7f6b21054640 10 monclient: init 2024-06-11T19:15:28.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:28.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 10 monclient: _add_conns ranks=[1,5,8,3,0,2,10,6,9,7,4] 2024-06-11T19:15:28.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6b1c131c60 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:28.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6b1c0a1520 0x7f6b1c13a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:28.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 10 monclient(hunting): picked mon.noname-h con 0x7f6b1c0a1520 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:28.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b1c13a690 0x7f6b1c13ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:28.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 10 monclient(hunting): picked mon.noname-e con 0x7f6b1c13a690 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b1c0a1940 con 0x7f6b1c131c60 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f6b1c0a02a0 con 0x7f6b1c0a1520 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6b1c0a0580 con 0x7f6b1c13a690 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b21054640 10 monclient(hunting): authenticate will time out at 1138.059557s 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1b59e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b1c13a690 0x7f6b1c13ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1b59e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b1c13a690 0x7f6b1c13ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:33600/0 (socket says 172.21.15.98:33600) 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1b59e640 1 -- 172.21.15.98:0/1892569463 learned_addr learned my addr 172.21.15.98:0/1892569463 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1b59e640 10 monclient(hunting): get_auth_request con 0x7f6b1c13a690 auth_method 0 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:28.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1b59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1b59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 1 --2- 172.21.15.98:0/1892569463 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6b1c0a1520 0x7f6b1c13a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1b59e640 1 -- 172.21.15.98:0/1892569463 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b1c13a690 msgr2=0x7f6b1c13ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1b59e640 1 -- 172.21.15.98:0/1892569463 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b1c13a690 msgr2=0x7f6b1c13ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1b59e640 1 --2- 172.21.15.98:0/1892569463 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b1c13a690 0x7f6b1c13ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 10 monclient(hunting): get_auth_request con 0x7f6b1c0a1520 auth_method 0 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1b59e640 1 --2- 172.21.15.98:0/1892569463 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b1c13a690 0x7f6b1c13ca80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:28.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b19d9b640 1 -- 172.21.15.98:0/1892569463 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464910611 0 0) 0x7f6b0c001110 con 0x7f6b1c131c60 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b19d9b640 10 monclient(hunting): _negotiate 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b19d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b19d9b640 10 monclient(hunting): my global_id is 31789 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b19d9b640 1 -- 172.21.15.98:0/1892569463 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b04003520 con 0x7f6b1c131c60 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 10 monclient(hunting): handle_auth_done global_id 22748 payload 274 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 1 -- 172.21.15.98:0/1892569463 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b1c13a690 msgr2=0x7f6b1c13ca80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 1 --2- 172.21.15.98:0/1892569463 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b1c13a690 0x7f6b1c13ca80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 1 -- 172.21.15.98:0/1892569463 >> v1:172.21.15.117:6789/0 conn(0x7f6b1c131c60 legacy=0x7f6b1c132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 10 monclient: _finish_hunting 0 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 1 monclient: found mon.noname-h 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:28.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.252+0000 7f6b1a59c640 1 -- 172.21.15.98:0/1892569463 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1c0a0860 con 0x7f6b1c0a1520 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b1a59c640 1 --2- 172.21.15.98:0/1892569463 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6b1c0a1520 0x7f6b1c13a150 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000a960 tx=0x7f6b1002cc30 comp rx=0 tx=0).ready entity=mon.5 client_cookie=818a7f595135ab25 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 1 -- 172.21.15.98:0/1892569463 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6b1003a070 con 0x7f6b1c0a1520 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 10 monclient: dump: 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:28.266 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 10 monclient: _finish_auth 0 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:28.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 1 -- 172.21.15.98:0/1892569463 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b1002dc50 con 0x7f6b1c0a1520 2024-06-11T19:15:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 1 -- 172.21.15.98:0/1892569463 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6b1000b040 con 0x7f6b1c0a1520 2024-06-11T19:15:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:15:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b19d9b640 10 monclient: dump: 2024-06-11T19:15:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:28.268 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b21054640 5 monclient: authenticate success, global_id 22748 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b21054640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.253+0000 7f6b21054640 4 set_mon_vals no callback set 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.254+0000 7f6b21054640 10 monclient: shutdown 2024-06-11T19:15:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.254+0000 7f6b21054640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.254+0000 7f6b21054640 1 -- 172.21.15.98:0/1892569463 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6b1c0a1520 msgr2=0x7f6b1c13a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.254+0000 7f6b21054640 1 --2- 172.21.15.98:0/1892569463 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6b1c0a1520 0x7f6b1c13a150 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000a960 tx=0x7f6b1002cc30 comp rx=0 tx=0).stop 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.254+0000 7f6b21054640 1 -- 172.21.15.98:0/1892569463 shutdown_connections 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.254+0000 7f6b21054640 1 --2- 172.21.15.98:0/1892569463 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b1c13a690 0x7f6b1c13ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.254+0000 7f6b21054640 1 --2- 172.21.15.98:0/1892569463 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6b1c0a1520 0x7f6b1c13a150 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.254+0000 7f6b21054640 1 -- 172.21.15.98:0/1892569463 >> 172.21.15.98:0/1892569463 conn(0x7f6b1c0a7850 msgr2=0x7f6b1c0a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.254+0000 7f6b21054640 1 -- 172.21.15.98:0/1892569463 shutdown_connections 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.254+0000 7f6b21054640 1 -- 172.21.15.98:0/1892569463 wait complete. 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.255+0000 7f6b21054640 10 monclient: build_initial_monmap 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.255+0000 7f6b21054640 10 monclient: monmap: 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.255+0000 7f6b21054640 1 librados: starting msgr at 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.255+0000 7f6b21054640 1 librados: starting objecter 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.255+0000 7f6b21054640 1 Processor -- start 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.255+0000 7f6b21054640 1 -- start start 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.255+0000 7f6b21054640 1 librados: setting wanted keys 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.255+0000 7f6b21054640 1 librados: calling monclient init 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.255+0000 7f6b21054640 10 monclient: init 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 10 monclient: _add_conns ranks=[2,1,7,0,3,4,8,6,5] 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c0a1520 0x7f6b1c0704b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 10 monclient(hunting): picked mon.noname-b con 0x7f6b1c0a1520 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b1c131c60 0x7f6b1c0709f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 10 monclient(hunting): picked mon.noname-f con 0x7f6b1c131c60 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:15:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6b1c13a690 0x7f6b1c070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 10 monclient(hunting): picked mon.noname-i con 0x7f6b1c13a690 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:15:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c09f570 con 0x7f6b1c131c60 2024-06-11T19:15:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6b1c131930 con 0x7f6b1c0a1520 2024-06-11T19:15:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f6b1c0738a0 con 0x7f6b1c13a690 2024-06-11T19:15:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b21054640 10 monclient(hunting): authenticate will time out at 1138.063291s 2024-06-11T19:15:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1ad9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c0a1520 0x7f6b1c0704b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1ad9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c0a1520 0x7f6b1c0704b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:46222/0 (socket says 172.21.15.98:46222) 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1ad9d640 1 -- 172.21.15.98:0/1186920668 learned_addr learned my addr 172.21.15.98:0/1186920668 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1ad9d640 10 monclient(hunting): get_auth_request con 0x7f6b1c0a1520 auth_method 0 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1b59e640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6b1c13a690 0x7f6b1c070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b1c131c60 0x7f6b1c0709f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1ad9d640 1 -- 172.21.15.98:0/1186920668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c0a1520 msgr2=0x7f6b1c0704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1ad9d640 1 -- 172.21.15.98:0/1186920668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c0a1520 msgr2=0x7f6b1c0704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1b59e640 10 monclient(hunting): get_auth_request con 0x7f6b1c13a690 auth_method 0 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1ad9d640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c0a1520 0x7f6b1c0704b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1b59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1b59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 10 monclient(hunting): get_auth_request con 0x7f6b1c131c60 auth_method 0 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1ad9d640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c0a1520 0x7f6b1c0704b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1b59e640 1 -- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6b1c13a690 msgr2=0x7f6b1c070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1b59e640 1 -- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6b1c13a690 msgr2=0x7f6b1c070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1b59e640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6b1c13a690 0x7f6b1c070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1b59e640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6b1c13a690 0x7f6b1c070f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 10 monclient(hunting): handle_auth_done global_id 31798 payload 824 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 1 -- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6b1c13a690 msgr2=0x7f6b1c070f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6b1c13a690 0x7f6b1c070f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 1 -- 172.21.15.98:0/1186920668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c0a1520 msgr2=0x7f6b1c0704b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c0a1520 0x7f6b1c0704b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 10 monclient: _finish_hunting 0 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 1 monclient: found mon.noname-f 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 1 -- 172.21.15.98:0/1186920668 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1c13a030 con 0x7f6b1c131c60 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b1a59c640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b1c131c60 0x7f6b1c0709f0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f6b10034000 tx=0x7f6b100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b037fe640 10 client.?.objecter ms_handle_connect 0x7f6b1c131c60 2024-06-11T19:15:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.256+0000 7f6b037fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 1 -- 172.21.15.98:0/1186920668 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6b10043070 con 0x7f6b1c131c60 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 10 monclient: dump: 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:28.278 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 10 monclient: _finish_auth 0 2024-06-11T19:15:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 10 monclient: _check_auth_tickets 2024-06-11T19:15:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 1 -- 172.21.15.98:0/1186920668 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b10005e00 con 0x7f6b1c131c60 2024-06-11T19:15:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 1 -- 172.21.15.98:0/1186920668 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6b1003a040 con 0x7f6b1c131c60 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b037fe640 10 monclient: dump: 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:28.279 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b21054640 5 monclient: authenticate success, global_id 31798 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b21054640 10 monclient: _renew_subs 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b1959a640 4 set_mon_vals no callback set 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b21054640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b21054640 1 -- 172.21.15.98:0/1186920668 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b1c073e60 con 0x7f6b1c131c60 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b21054640 10 client.31798.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b21054640 10 monclient: _renew_subs 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b21054640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:15:28.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b21054640 1 -- 172.21.15.98:0/1186920668 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b1c071cc0 con 0x7f6b1c131c60 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.257+0000 7f6b21054640 1 librados: init done 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b017fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b017fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b017fa640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b017fa640 1 -- 172.21.15.98:0/1186920668 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6af4005080 con 0x7f6b1c131c60 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b037fe640 1 -- 172.21.15.98:0/1186920668 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f6b10005990 con 0x7f6b1c131c60 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b037fe640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6b0803fdb0 0x7f6b08042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b037fe640 1 -- 172.21.15.98:0/1186920668 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(279..279 src has 1..279) ==== 6684+0+0 (secure 0 0 0) 0x7f6b10072db0 con 0x7f6b1c131c60 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b037fe640 10 client.31798.objecter ms_dispatch 0x7f6b1c000b90 osd_map(279..279 src has 1..279) 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b037fe640 3 client.31798.objecter handle_osd_map got epochs [279,279] > 0 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b037fe640 3 client.31798.objecter handle_osd_map decoding full epoch 279 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b037fe640 20 client.31798.objecter dump_active .. 0 homeless 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b1ad9d640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6b0803fdb0 0x7f6b08042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:28.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.258+0000 7f6b1ad9d640 10 monclient: get_auth_request con 0x7f6b0803fdb0 auth_method 0 2024-06-11T19:15:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.259+0000 7f6b037fe640 10 client.31798.objecter ms_handle_connect 0x7f6b0803fdb0 2024-06-11T19:15:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.259+0000 7f6b1ad9d640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6b0803fdb0 0x7f6b08042270 secure :-1 s=READY pgs=4117 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c007fd0 tx=0x7f6b0c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.263+0000 7f6b037fe640 1 -- 172.21.15.98:0/1186920668 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6b1003d390 con 0x7f6b1c131c60 2024-06-11T19:15:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.263+0000 7f6b037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.263+0000 7f6b037fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:28.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.543+0000 7f6b017fa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:15:28.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.543+0000 7f6b017fa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:28.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.543+0000 7f6b017fa640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:15:28.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:28.543+0000 7f6b017fa640 1 -- 172.21.15.98:0/1186920668 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6af4005370 con 0x7f6b1c131c60 2024-06-11T19:15:31.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:31.256+0000 7f6b027fc640 10 monclient: tick 2024-06-11T19:15:31.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:31.256+0000 7f6b027fc640 10 monclient: _check_auth_tickets 2024-06-11T19:15:31.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:31.256+0000 7f6b027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:31.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:31.256+0000 7f6b027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:32.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:32.256+0000 7f6b027fc640 10 monclient: tick 2024-06-11T19:15:32.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:32.256+0000 7f6b027fc640 10 monclient: _check_auth_tickets 2024-06-11T19:15:32.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:32.256+0000 7f6b027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.256+0000 7f6b027fc640 10 monclient: tick 2024-06-11T19:15:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.256+0000 7f6b027fc640 10 monclient: _check_auth_tickets 2024-06-11T19:15:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.256+0000 7f6b027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:33.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.258+0000 7f6b03fff640 10 client.31798.objecter tick 2024-06-11T19:15:33.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.448+0000 7f6b037fe640 1 -- 172.21.15.98:0/1186920668 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3174 (secure 0 0 0) 0x7f6b10047ce0 con 0x7f6b1c131c60 2024-06-11T19:15:33.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.448+0000 7f6b037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:33.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.448+0000 7f6b037fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:33.448 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":26,"quorum":[0,1,4,5,6,8],"quorum_names":["a","f","c","h","d","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:15:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.451+0000 7f6b017fa640 10 librados: watch_flush enter 2024-06-11T19:15:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 10 librados: watch_flush exit 2024-06-11T19:15:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 20 client.31798.objecter shutdown clearing up homeless session... 2024-06-11T19:15:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 10 client.31798.objecter successfully canceled tick 2024-06-11T19:15:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 -- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6b0803fdb0 msgr2=0x7f6b08042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6b0803fdb0 0x7f6b08042270 secure :-1 s=READY pgs=4117 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c007fd0 tx=0x7f6b0c009040 comp rx=0 tx=0).stop 2024-06-11T19:15:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 10 monclient: shutdown 2024-06-11T19:15:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:33.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 -- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b1c131c60 msgr2=0x7f6b1c0709f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b1c131c60 0x7f6b1c0709f0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f6b10034000 tx=0x7f6b100046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 -- 172.21.15.98:0/1186920668 shutdown_connections 2024-06-11T19:15:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6b0803fdb0 0x7f6b08042270 unknown :-1 s=CLOSED pgs=4117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6b1c13a690 0x7f6b1c070f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b1c131c60 0x7f6b1c0709f0 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 --2- 172.21.15.98:0/1186920668 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6b1c0a1520 0x7f6b1c0704b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 -- 172.21.15.98:0/1186920668 >> 172.21.15.98:0/1186920668 conn(0x7f6b1c0a7850 msgr2=0x7f6b1c0a21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 -- 172.21.15.98:0/1186920668 shutdown_connections 2024-06-11T19:15:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 -- 172.21.15.98:0/1186920668 wait complete. 2024-06-11T19:15:33.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:33.452+0000 7f6b017fa640 1 librados: shutdown 2024-06-11T19:15:36.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:15:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.576+0000 7f3bf5966640 10 monclient: get_monmap_and_config 2024-06-11T19:15:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.576+0000 7f3bf5966640 10 monclient: build_initial_monmap 2024-06-11T19:15:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.576+0000 7f3bf5966640 10 monclient: monmap: 2024-06-11T19:15:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:36.577830+0000 2024-06-11T19:15:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:36.577830+0000 2024-06-11T19:15:36.576 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:36.577 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:36.578 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:36.578 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.577+0000 7f3bf5966640 1 Processor -- start 2024-06-11T19:15:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.577+0000 7f3bf5966640 1 -- start start 2024-06-11T19:15:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.577+0000 7f3bf5966640 10 monclient: init 2024-06-11T19:15:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.577+0000 7f3bf5966640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 10 monclient: _add_conns ranks=[5,7,1,8,0,4,2,6,9,10,3] 2024-06-11T19:15:36.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 10 monclient(hunting): picked mon.noname-h con 0x7f3bf00a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 10 monclient(hunting): picked mon.noname-i con 0x7f3bf00a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3bf00a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bf00a7140 con 0x7f3bf00a2530 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f3bf00a2b50 con 0x7f3bf00a4920 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f3bf00a2e30 con 0x7f3bf00a5240 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bf5966640 10 monclient(hunting): authenticate will time out at 1146.385128s 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3beeffd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3beeffd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:60400/0 (socket says 172.21.15.98:60400) 2024-06-11T19:15:36.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3beeffd640 1 -- 172.21.15.98:0/1243633765 learned_addr learned my addr 172.21.15.98:0/1243633765 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3beeffd640 10 monclient(hunting): get_auth_request con 0x7f3bf00a4920 auth_method 0 2024-06-11T19:15:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3beeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3beeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3beeffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bee7fc640 10 monclient(hunting): get_auth_request con 0x7f3bf00a5240 auth_method 0 2024-06-11T19:15:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bee7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3beeffd640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 msgr2=0x7f3bf00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:15:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3beeffd640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 msgr2=0x7f3bf00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:36.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bee7fc640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 msgr2=0x7f3bf00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:15:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bee7fc640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 msgr2=0x7f3bf00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:36.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.578+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:36.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:36.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3beeffd640 10 monclient(hunting): get_auth_request con 0x7f3bf00a4920 auth_method 0 2024-06-11T19:15:36.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3beeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:36.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3beeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:36.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3beeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3bee7fc640 10 monclient(hunting): get_auth_request con 0x7f3bf00a5240 auth_method 0 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3bee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3bee7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3bee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3beeffd640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 msgr2=0x7f3bf00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3beeffd640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 msgr2=0x7f3bf00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3bee7fc640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 msgr2=0x7f3bf00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3bee7fc640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 msgr2=0x7f3bf00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:36.779+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:15:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:37.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3beeffd640 10 monclient(hunting): get_auth_request con 0x7f3bf00a4920 auth_method 0 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3beeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3beeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3beeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3bee7fc640 10 monclient(hunting): get_auth_request con 0x7f3bf00a5240 auth_method 0 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3bee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3bee7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3bee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3beeffd640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 msgr2=0x7f3bf00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3beeffd640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 msgr2=0x7f3bf00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3bee7fc640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 msgr2=0x7f3bf00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:15:37.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3bee7fc640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 msgr2=0x7f3bf00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:37.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:37.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.180+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:15:37.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.981+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:37.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.981+0000 7f3beeffd640 10 monclient(hunting): get_auth_request con 0x7f3bf00a4920 auth_method 0 2024-06-11T19:15:37.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.981+0000 7f3beeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:37.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.981+0000 7f3beeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:37.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.981+0000 7f3beeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:37.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.981+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:37.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.981+0000 7f3bee7fc640 10 monclient(hunting): get_auth_request con 0x7f3bf00a5240 auth_method 0 2024-06-11T19:15:37.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.981+0000 7f3bee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:37.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.981+0000 7f3bee7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:37.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.981+0000 7f3bee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:37.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.982+0000 7f3beeffd640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 msgr2=0x7f3bf00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:15:37.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.982+0000 7f3beeffd640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 msgr2=0x7f3bf00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:37.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.982+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:37.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.982+0000 7f3bee7fc640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 msgr2=0x7f3bf00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:15:37.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.982+0000 7f3bee7fc640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 msgr2=0x7f3bf00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:37.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.982+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:37.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.982+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:15:37.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:37.982+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:15:38.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.495+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4154381228 0 0) 0x7f3bf00a7140 con 0x7f3bf00a2530 2024-06-11T19:15:38.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.495+0000 7f3bedffb640 10 monclient(hunting): _negotiate 2024-06-11T19:15:38.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.495+0000 7f3bedffb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:38.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.495+0000 7f3bedffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:38.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.495+0000 7f3bedffb640 10 monclient(hunting): my global_id is 31807 2024-06-11T19:15:38.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.495+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bd8003520 con 0x7f3bf00a2530 2024-06-11T19:15:38.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2707907783 0 0) 0x7f3bd8003520 con 0x7f3bf00a2530 2024-06-11T19:15:38.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 msgr2=0x7f3bf00a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:15:38.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3bf00a5240 0x7f3bf00a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:38.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 msgr2=0x7f3bf00a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:15:38.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf00a4920 0x7f3bf00a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:38.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient: _finish_hunting 0 2024-06-11T19:15:38.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 monclient: found mon.noname-f-legacy 2024-06-11T19:15:38.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:38.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.117:6789/0 2024-06-11T19:15:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf00a3110 con 0x7f3bf00a2530 2024-06-11T19:15:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient: _finish_auth 0 2024-06-11T19:15:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient: _check_auth_tickets 2024-06-11T19:15:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7f3be8001ce0 con 0x7f3bf00a2530 2024-06-11T19:15:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2024-06-11T19:15:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient: dump: 2024-06-11T19:15:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:38.517 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:38.518 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:38.518 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 monclient: mon.f has (v2) addrs [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] but i'm connected to v1:172.21.15.117:6789/0, reconnecting 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 >> v1:172.21.15.117:6789/0 conn(0x7f3bf00a2530 legacy=0x7f3bf00a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient: _add_conns ranks=[8,0,6,3,2,1,7,5,4] 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3bd80041b0 0x7f3bd8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient(hunting): picked mon.e con 0x7f3bd80041b0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd8004e60 0x7f3bd80099b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient(hunting): picked mon.a con 0x7f3bd8004e60 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3bd8009ef0 0x7f3bd800c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient(hunting): picked mon.d con 0x7f3bd8009ef0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf00a2b50 con 0x7f3bd8004e60 2024-06-11T19:15:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3bf00a2e30 con 0x7f3bd8009ef0 2024-06-11T19:15:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3bd800c960 con 0x7f3bd80041b0 2024-06-11T19:15:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bedffb640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:15:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3bd80041b0 0x7f3bd8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd8004e60 0x7f3bd80099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3beeffd640 10 monclient(hunting): get_auth_request con 0x7f3bd80041b0 auth_method 0 2024-06-11T19:15:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3beeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3beeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3beeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bee7fc640 10 monclient(hunting): get_auth_request con 0x7f3bd8004e60 auth_method 0 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bee7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bef7fe640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3bd8009ef0 0x7f3bd800c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3beeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3beeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3beeffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.503+0000 7f3bee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3bee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3bef7fe640 10 monclient(hunting): get_auth_request con 0x7f3bd8009ef0 auth_method 0 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3bef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:38.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3bef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3bef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3bef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3bef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3bef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:15:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 10 monclient(hunting): handle_auth_done global_id 31807 payload 293 2024-06-11T19:15:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3bd8009ef0 msgr2=0x7f3bd800c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3bd8009ef0 0x7f3bd800c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd8004e60 msgr2=0x7f3bd80099b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd8004e60 0x7f3bd80099b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 10 monclient: _finish_hunting 0 2024-06-11T19:15:38.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 1 monclient: found mon.e 2024-06-11T19:15:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:15:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 1 -- 172.21.15.98:0/1243633765 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf00a3110 con 0x7f3bd80041b0 2024-06-11T19:15:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 10 monclient: _finish_auth 0 2024-06-11T19:15:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 10 monclient: _check_auth_tickets 2024-06-11T19:15:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3beeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3bef7fe640 1 -- 172.21.15.98:0/1243633765 reap_dead start 2024-06-11T19:15:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3bf5966640 5 monclient: authenticate success, global_id 31807 2024-06-11T19:15:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd8004e60 0x7f3bd80099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.504+0000 7f3bf5966640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.505+0000 7f3beeffd640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3bd80041b0 0x7f3bd8004800 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3be0012a00 tx=0x7f3bd8005830 comp rx=0 tx=0).ready entity=mon.8 client_cookie=689294ee3c968a4e server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:38.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3be003a070 con 0x7f3bd80041b0 2024-06-11T19:15:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bedffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bedffb640 10 monclient: dump: 2024-06-11T19:15:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:38.524 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3be00347e0 con 0x7f3bd80041b0 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bf5966640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bedffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bf5966640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:38.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bedffb640 1 -- 172.21.15.98:0/1243633765 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3be000f040 con 0x7f3bd80041b0 2024-06-11T19:15:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bf5966640 4 set_mon_vals no callback set 2024-06-11T19:15:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bedffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.506+0000 7f3bedffb640 10 monclient: dump: 2024-06-11T19:15:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:38.527 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.507+0000 7f3bf5966640 10 monclient: shutdown 2024-06-11T19:15:38.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.507+0000 7f3bf5966640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.507+0000 7f3bf5966640 1 -- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3bd80041b0 msgr2=0x7f3bd8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.507+0000 7f3bf5966640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3bd80041b0 0x7f3bd8004800 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3be0012a00 tx=0x7f3bd8005830 comp rx=0 tx=0).stop 2024-06-11T19:15:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.507+0000 7f3bf5966640 1 -- 172.21.15.98:0/1243633765 shutdown_connections 2024-06-11T19:15:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.507+0000 7f3bf5966640 1 --2- 172.21.15.98:0/1243633765 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3bd80041b0 0x7f3bd8004800 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.507+0000 7f3bf5966640 1 -- 172.21.15.98:0/1243633765 >> 172.21.15.98:0/1243633765 conn(0x7f3bf00ab850 msgr2=0x7f3bf00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.507+0000 7f3bf5966640 1 -- 172.21.15.98:0/1243633765 shutdown_connections 2024-06-11T19:15:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.507+0000 7f3bf5966640 1 -- 172.21.15.98:0/1243633765 wait complete. 2024-06-11T19:15:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.508+0000 7f3bf5966640 10 monclient: build_initial_monmap 2024-06-11T19:15:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.508+0000 7f3bf5966640 10 monclient: monmap: 2024-06-11T19:15:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:38.530 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.508+0000 7f3bf5966640 1 librados: starting msgr at 2024-06-11T19:15:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.508+0000 7f3bf5966640 1 librados: starting objecter 2024-06-11T19:15:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.508+0000 7f3bf5966640 1 Processor -- start 2024-06-11T19:15:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 1 -- start start 2024-06-11T19:15:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 1 librados: setting wanted keys 2024-06-11T19:15:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 1 librados: calling monclient init 2024-06-11T19:15:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 10 monclient: init 2024-06-11T19:15:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 10 monclient: _add_conns ranks=[4,5,3,2,0,1,8,6,7] 2024-06-11T19:15:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3bd80041b0 0x7f3bf01cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd80041b0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:15:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf01ce010 0x7f3bf0124e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 10 monclient(hunting): picked mon.noname-h con 0x7f3bf01ce010 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bf01ce3f0 0x7f3bf0125360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 10 monclient(hunting): picked mon.noname-g con 0x7f3bf01ce3f0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f3bf00a6da0 con 0x7f3bf01ce3f0 2024-06-11T19:15:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3bf0138e50 con 0x7f3bd80041b0 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f3bf00a4d00 con 0x7f3bf01ce010 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bf5966640 10 monclient(hunting): authenticate will time out at 1148.316483s 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3bd80041b0 0x7f3bf01cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3bd80041b0 0x7f3bf01cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59104/0 (socket says 172.21.15.98:59104) 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 1 -- 172.21.15.98:0/872329295 learned_addr learned my addr 172.21.15.98:0/872329295 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 10 monclient(hunting): get_auth_request con 0x7f3bd80041b0 auth_method 0 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bef7fe640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bf01ce3f0 0x7f3bf0125360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf01ce010 0x7f3bf0124e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bef7fe640 10 monclient(hunting): get_auth_request con 0x7f3bf01ce3f0 auth_method 0 2024-06-11T19:15:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bee7fc640 10 monclient(hunting): get_auth_request con 0x7f3bf01ce010 auth_method 0 2024-06-11T19:15:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bee7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bef7fe640 1 -- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bf01ce3f0 msgr2=0x7f3bf0125360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:15:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bef7fe640 1 -- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bf01ce3f0 msgr2=0x7f3bf0125360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:15:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bef7fe640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bf01ce3f0 0x7f3bf0125360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 10 monclient(hunting): handle_auth_done global_id 23656 payload 824 2024-06-11T19:15:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3bef7fe640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bf01ce3f0 0x7f3bf0125360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 1 -- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf01ce010 msgr2=0x7f3bf0124e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf01ce010 0x7f3bf0124e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 1 -- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bf01ce3f0 msgr2=0x7f3bf0125360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:15:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bf01ce3f0 0x7f3bf0125360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 10 monclient: _finish_hunting 0 2024-06-11T19:15:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 1 monclient: found mon.noname-c 2024-06-11T19:15:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:15:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.509+0000 7f3beeffd640 1 -- 172.21.15.98:0/872329295 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf0127650 con 0x7f3bd80041b0 2024-06-11T19:15:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3beeffd640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3bd80041b0 0x7f3bf01cdad0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f3be0012a00 tx=0x7f3be00046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 10 client.?.objecter ms_handle_connect 0x7f3bd80041b0 2024-06-11T19:15:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 1 -- 172.21.15.98:0/872329295 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3be003a040 con 0x7f3bd80041b0 2024-06-11T19:15:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:15:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 10 monclient: dump: 2024-06-11T19:15:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 10 monclient: _finish_auth 0 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 1 -- 172.21.15.98:0/872329295 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3be0045070 con 0x7f3bd80041b0 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bf5966640 5 monclient: authenticate success, global_id 23656 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bf5966640 10 monclient: _renew_subs 2024-06-11T19:15:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bf5966640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:38.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bf5966640 1 -- 172.21.15.98:0/872329295 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bf0128b00 con 0x7f3bd80041b0 2024-06-11T19:15:38.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:38.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 1 -- 172.21.15.98:0/872329295 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3be0034dc0 con 0x7f3bd80041b0 2024-06-11T19:15:38.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:38.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:38.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bf5966640 10 client.23656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:38.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bd7fff640 10 monclient: dump: 2024-06-11T19:15:38.538 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:38.538 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:38.538 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:38.539 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bed7fa640 4 set_mon_vals no callback set 2024-06-11T19:15:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bf5966640 10 monclient: _renew_subs 2024-06-11T19:15:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bf5966640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bf5966640 1 -- 172.21.15.98:0/872329295 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bf0128820 con 0x7f3bd80041b0 2024-06-11T19:15:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.510+0000 7f3bf5966640 1 librados: init done 2024-06-11T19:15:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.511+0000 7f3bd7fff640 1 -- 172.21.15.98:0/872329295 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3be004a460 con 0x7f3bd80041b0 2024-06-11T19:15:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.511+0000 7f3bf5966640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.511+0000 7f3bf5966640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.511+0000 7f3bf5966640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.511+0000 7f3bf5966640 1 -- 172.21.15.98:0/872329295 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bf00a5000 con 0x7f3bd80041b0 2024-06-11T19:15:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.511+0000 7f3bd7fff640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3bdc03fde0 0x7f3bdc0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.511+0000 7f3bd7fff640 1 -- 172.21.15.98:0/872329295 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(279..279 src has 1..279) ==== 6684+0+0 (secure 0 0 0) 0x7f3be0032070 con 0x7f3bd80041b0 2024-06-11T19:15:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.511+0000 7f3bd7fff640 10 client.23656.objecter ms_dispatch 0x7f3bf0000b90 osd_map(279..279 src has 1..279) 2024-06-11T19:15:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.511+0000 7f3bd7fff640 3 client.23656.objecter handle_osd_map got epochs [279,279] > 0 2024-06-11T19:15:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.511+0000 7f3bd7fff640 3 client.23656.objecter handle_osd_map decoding full epoch 279 2024-06-11T19:15:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.511+0000 7f3bd7fff640 20 client.23656.objecter dump_active .. 0 homeless 2024-06-11T19:15:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.511+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3bdc03fde0 0x7f3bdc0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.512+0000 7f3bee7fc640 10 monclient: get_auth_request con 0x7f3bdc03fde0 auth_method 0 2024-06-11T19:15:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.512+0000 7f3bd7fff640 10 client.23656.objecter ms_handle_connect 0x7f3bdc03fde0 2024-06-11T19:15:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.512+0000 7f3bee7fc640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3bdc03fde0 0x7f3bdc0422a0 secure :-1 s=READY pgs=4118 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8004120 tx=0x7f3be4008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.516+0000 7f3bd7fff640 1 -- 172.21.15.98:0/872329295 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3be003f580 con 0x7f3bd80041b0 2024-06-11T19:15:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.516+0000 7f3bd7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.516+0000 7f3bd7fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.781+0000 7f3bf5966640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:15:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.781+0000 7f3bf5966640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.782+0000 7f3bf5966640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.782+0000 7f3bf5966640 1 -- 172.21.15.98:0/872329295 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3bf0126f50 con 0x7f3bd80041b0 2024-06-11T19:15:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.782+0000 7f3bd7fff640 1 -- 172.21.15.98:0/872329295 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3180 (secure 0 0 0) 0x7f3be003f860 con 0x7f3bd80041b0 2024-06-11T19:15:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.782+0000 7f3bd7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.782+0000 7f3bd7fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:38.782 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":28,"quorum":[0,1,3,4,5,6,8],"quorum_names":["a","f","g","c","h","d","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:15:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 10 librados: watch_flush enter 2024-06-11T19:15:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 10 librados: watch_flush exit 2024-06-11T19:15:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 20 client.23656.objecter shutdown clearing up homeless session... 2024-06-11T19:15:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 10 client.23656.objecter successfully canceled tick 2024-06-11T19:15:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 -- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3bdc03fde0 msgr2=0x7f3bdc0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3bdc03fde0 0x7f3bdc0422a0 secure :-1 s=READY pgs=4118 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8004120 tx=0x7f3be4008040 comp rx=0 tx=0).stop 2024-06-11T19:15:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 10 monclient: shutdown 2024-06-11T19:15:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 -- 172.21.15.98:0/872329295 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3bd80041b0 msgr2=0x7f3bf01cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3bd80041b0 0x7f3bf01cdad0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f3be0012a00 tx=0x7f3be00046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 -- 172.21.15.98:0/872329295 shutdown_connections 2024-06-11T19:15:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3bdc03fde0 0x7f3bdc0422a0 unknown :-1 s=CLOSED pgs=4118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bf01ce3f0 0x7f3bf0125360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3bf01ce010 0x7f3bf0124e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 --2- 172.21.15.98:0/872329295 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3bd80041b0 0x7f3bf01cdad0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 -- 172.21.15.98:0/872329295 >> 172.21.15.98:0/872329295 conn(0x7f3bf00ab850 msgr2=0x7f3bf009c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 -- 172.21.15.98:0/872329295 shutdown_connections 2024-06-11T19:15:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 -- 172.21.15.98:0/872329295 wait complete. 2024-06-11T19:15:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:38.784+0000 7f3bf5966640 1 librados: shutdown 2024-06-11T19:15:41.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:15:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.906+0000 7f3fd51f0640 10 monclient: get_monmap_and_config 2024-06-11T19:15:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.906+0000 7f3fd51f0640 10 monclient: build_initial_monmap 2024-06-11T19:15:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.906+0000 7f3fd51f0640 10 monclient: monmap: 2024-06-11T19:15:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:41.907153+0000 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:41.907153+0000 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:41.907 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.906+0000 7f3fd51f0640 1 Processor -- start 2024-06-11T19:15:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.906+0000 7f3fd51f0640 1 -- start start 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.906+0000 7f3fd51f0640 10 monclient: init 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 10 monclient: _add_conns ranks=[1,4,2,9,5,7,6,8,3,10,0] 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3fd00a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fd00a2530 0x7f3fd013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 10 monclient(hunting): picked mon.noname-c con 0x7f3fd00a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fd013e1a0 0x7f3fd0140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 10 monclient(hunting): picked mon.noname-b con 0x7f3fd013e1a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fd00a7140 con 0x7f3fd00a4920 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3fd00a2b50 con 0x7f3fd013e1a0 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3fd00a2e30 con 0x7f3fd00a2530 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fd51f0640 10 monclient(hunting): authenticate will time out at 1151.714296s 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fd00a2530 0x7f3fd013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fcf59e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fd013e1a0 0x7f3fd0140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fcf59e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fd013e1a0 0x7f3fd0140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55532/0 (socket says 172.21.15.98:55532) 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fd00a2530 0x7f3fd013dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59114/0 (socket says 172.21.15.98:59114) 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fcf59e640 1 -- 172.21.15.98:0/1390077252 learned_addr learned my addr 172.21.15.98:0/1390077252 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fcf59e640 10 monclient(hunting): get_auth_request con 0x7f3fd013e1a0 auth_method 0 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fcf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fcf59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fcf59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 10 monclient(hunting): get_auth_request con 0x7f3fd00a2530 auth_method 0 2024-06-11T19:15:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fcf59e640 1 -- 172.21.15.98:0/1390077252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fd013e1a0 msgr2=0x7f3fd0140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:15:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fcf59e640 1 -- 172.21.15.98:0/1390077252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fd013e1a0 msgr2=0x7f3fd0140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fcf59e640 1 --2- 172.21.15.98:0/1390077252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fd013e1a0 0x7f3fd0140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fcf59e640 1 --2- 172.21.15.98:0/1390077252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fd013e1a0 0x7f3fd0140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 10 monclient(hunting): handle_auth_done global_id 23665 payload 274 2024-06-11T19:15:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 1 -- 172.21.15.98:0/1390077252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fd013e1a0 msgr2=0x7f3fd0140590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:15:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 1 --2- 172.21.15.98:0/1390077252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fd013e1a0 0x7f3fd0140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 1 -- 172.21.15.98:0/1390077252 >> v1:172.21.15.117:6789/0 conn(0x7f3fd00a4920 legacy=0x7f3fd00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 10 monclient: _finish_hunting 0 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 1 monclient: found mon.noname-c 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.907+0000 7f3fce59c640 1 -- 172.21.15.98:0/1390077252 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd00a3110 con 0x7f3fd00a2530 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fce59c640 1 --2- 172.21.15.98:0/1390077252 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fd00a2530 0x7f3fd013dc60 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40097d0 tx=0x7f3fc402b770 comp rx=0 tx=0).ready entity=mon.4 client_cookie=beb76274f86725a3 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 1 -- 172.21.15.98:0/1390077252 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3fc4039070 con 0x7f3fd00a2530 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 10 monclient: dump: 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 10 monclient: _finish_auth 0 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 10 monclient: _check_auth_tickets 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 1 -- 172.21.15.98:0/1390077252 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc4030070 con 0x7f3fd00a2530 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fd51f0640 5 monclient: authenticate success, global_id 23665 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fd51f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 1 -- 172.21.15.98:0/1390077252 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3fc4031db0 con 0x7f3fd00a2530 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fd51f0640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fd51f0640 4 set_mon_vals no callback set 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.908+0000 7f3fcdd9b640 10 monclient: dump: 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:41.923 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.909+0000 7f3fd51f0640 10 monclient: shutdown 2024-06-11T19:15:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.909+0000 7f3fd51f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.909+0000 7f3fd51f0640 1 -- 172.21.15.98:0/1390077252 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fd00a2530 msgr2=0x7f3fd013dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.909+0000 7f3fd51f0640 1 --2- 172.21.15.98:0/1390077252 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fd00a2530 0x7f3fd013dc60 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40097d0 tx=0x7f3fc402b770 comp rx=0 tx=0).stop 2024-06-11T19:15:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.909+0000 7f3fd51f0640 1 -- 172.21.15.98:0/1390077252 shutdown_connections 2024-06-11T19:15:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.909+0000 7f3fd51f0640 1 --2- 172.21.15.98:0/1390077252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fd013e1a0 0x7f3fd0140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.909+0000 7f3fd51f0640 1 --2- 172.21.15.98:0/1390077252 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fd00a2530 0x7f3fd013dc60 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.909+0000 7f3fd51f0640 1 -- 172.21.15.98:0/1390077252 >> 172.21.15.98:0/1390077252 conn(0x7f3fd00ab850 msgr2=0x7f3fd00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.909+0000 7f3fd51f0640 1 -- 172.21.15.98:0/1390077252 shutdown_connections 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.909+0000 7f3fd51f0640 1 -- 172.21.15.98:0/1390077252 wait complete. 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.910+0000 7f3fd51f0640 10 monclient: build_initial_monmap 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.910+0000 7f3fd51f0640 10 monclient: monmap: 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.910+0000 7f3fd51f0640 1 librados: starting msgr at 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.910+0000 7f3fd51f0640 1 librados: starting objecter 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.910+0000 7f3fd51f0640 1 Processor -- start 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.910+0000 7f3fd51f0640 1 -- start start 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.910+0000 7f3fd51f0640 1 librados: setting wanted keys 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.910+0000 7f3fd51f0640 1 librados: calling monclient init 2024-06-11T19:15:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.910+0000 7f3fd51f0640 10 monclient: init 2024-06-11T19:15:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 10 monclient: _add_conns ranks=[6,1,3,8,0,2,7,5,4] 2024-06-11T19:15:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3fd00a2530 0x7f3fd01cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 10 monclient(hunting): picked mon.noname-d con 0x7f3fd00a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fd00a4920 0x7f3fd0135c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 10 monclient(hunting): picked mon.noname-f con 0x7f3fd00a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:15:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3fd013e1a0 0x7f3fd01361d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 10 monclient(hunting): picked mon.noname-g con 0x7f3fd013e1a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd0138e50 con 0x7f3fd00a4920 2024-06-11T19:15:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f3fd00a84b0 con 0x7f3fd013e1a0 2024-06-11T19:15:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3fd00a6da0 con 0x7f3fd00a2530 2024-06-11T19:15:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 10 monclient(hunting): authenticate will time out at 1151.718161s 2024-06-11T19:15:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3fd00a2530 0x7f3fd01cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3fd00a2530 0x7f3fd01cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:56744/0 (socket says 172.21.15.98:56744) 2024-06-11T19:15:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 1 -- 172.21.15.98:0/21190102 learned_addr learned my addr 172.21.15.98:0/21190102 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 10 monclient(hunting): get_auth_request con 0x7f3fd00a2530 auth_method 0 2024-06-11T19:15:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fcf59e640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3fd013e1a0 0x7f3fd01361d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fce59c640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fd00a4920 0x7f3fd0135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fcf59e640 10 monclient(hunting): get_auth_request con 0x7f3fd013e1a0 auth_method 0 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fcf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fcf59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fcf59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fce59c640 10 monclient(hunting): get_auth_request con 0x7f3fd00a4920 auth_method 0 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fce59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fce59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:41.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fcf59e640 1 -- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3fd013e1a0 msgr2=0x7f3fd01361d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fcf59e640 1 -- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3fd013e1a0 msgr2=0x7f3fd01361d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fcf59e640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3fd013e1a0 0x7f3fd01361d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 10 monclient(hunting): handle_auth_done global_id 54816 payload 824 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fcf59e640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3fd013e1a0 0x7f3fd01361d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 1 -- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3fd013e1a0 msgr2=0x7f3fd01361d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3fd013e1a0 0x7f3fd01361d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 1 -- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fd00a4920 msgr2=0x7f3fd0135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fd00a4920 0x7f3fd0135c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 10 monclient: _finish_hunting 0 2024-06-11T19:15:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 1 monclient: found mon.noname-d 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 1 -- 172.21.15.98:0/21190102 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd0122380 con 0x7f3fd00a2530 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fced9d640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3fd00a2530 0x7f3fd01cd720 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3fc800c450 tx=0x7f3fc800c910 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fb77fe640 10 client.?.objecter ms_handle_connect 0x7f3fd00a2530 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fb77fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fb77fe640 1 -- 172.21.15.98:0/21190102 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3fc8001180 con 0x7f3fd00a2530 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fb77fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fb77fe640 10 monclient: dump: 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:41.931 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fb77fe640 10 monclient: _finish_auth 0 2024-06-11T19:15:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fb77fe640 10 monclient: _check_auth_tickets 2024-06-11T19:15:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 5 monclient: authenticate success, global_id 54816 2024-06-11T19:15:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fb77fe640 1 -- 172.21.15.98:0/21190102 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc8012630 con 0x7f3fd00a2530 2024-06-11T19:15:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 10 monclient: _renew_subs 2024-06-11T19:15:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.911+0000 7f3fd51f0640 1 -- 172.21.15.98:0/21190102 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fd0122660 con 0x7f3fd00a2530 2024-06-11T19:15:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.912+0000 7f3fb77fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.912+0000 7f3fb77fe640 1 -- 172.21.15.98:0/21190102 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3fc8012c50 con 0x7f3fd00a2530 2024-06-11T19:15:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.912+0000 7f3fb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.912+0000 7f3fb77fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.912+0000 7f3fb77fe640 10 monclient: dump: 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:41.933 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.912+0000 7f3fd51f0640 10 client.54816.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.912+0000 7f3fd51f0640 10 monclient: _renew_subs 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.912+0000 7f3fd51f0640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.912+0000 7f3fcd59a640 4 set_mon_vals no callback set 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.912+0000 7f3fd51f0640 1 -- 172.21.15.98:0/21190102 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fd00a1360 con 0x7f3fd00a2530 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.912+0000 7f3fd51f0640 1 librados: init done 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.912+0000 7f3fb77fe640 1 -- 172.21.15.98:0/21190102 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3fc80127e0 con 0x7f3fd00a2530 2024-06-11T19:15:41.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fd51f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fd51f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fd51f0640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fd51f0640 1 -- 172.21.15.98:0/21190102 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fd00a4380 con 0x7f3fd00a2530 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fb77fe640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3fbc03fdb0 0x7f3fbc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fb77fe640 1 -- 172.21.15.98:0/21190102 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(283..283 src has 1..283) ==== 6549+0+0 (secure 0 0 0) 0x7f3fc8051690 con 0x7f3fd00a2530 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fb77fe640 10 client.54816.objecter ms_dispatch 0x7f3fd0000b90 osd_map(283..283 src has 1..283) 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fb77fe640 3 client.54816.objecter handle_osd_map got epochs [283,283] > 0 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fb77fe640 3 client.54816.objecter handle_osd_map decoding full epoch 283 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fb77fe640 20 client.54816.objecter dump_active .. 0 homeless 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fce59c640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3fbc03fdb0 0x7f3fbc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fce59c640 10 monclient: get_auth_request con 0x7f3fbc03fdb0 auth_method 0 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fb77fe640 10 client.54816.objecter ms_handle_connect 0x7f3fbc03fdb0 2024-06-11T19:15:41.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.913+0000 7f3fce59c640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3fbc03fdb0 0x7f3fbc042270 secure :-1 s=READY pgs=4120 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4036000 tx=0x7f3fc4008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:41.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.917+0000 7f3fb77fe640 1 -- 172.21.15.98:0/21190102 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3fc8003d80 con 0x7f3fd00a2530 2024-06-11T19:15:41.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.917+0000 7f3fb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:41.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:41.917+0000 7f3fb77fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:42.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.172+0000 7f3fd51f0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:15:42.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.172+0000 7f3fd51f0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:42.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.172+0000 7f3fd51f0640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:42.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.172+0000 7f3fd51f0640 1 -- 172.21.15.98:0/21190102 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3fd00aa980 con 0x7f3fd00a2530 2024-06-11T19:15:42.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.172+0000 7f3fb77fe640 1 -- 172.21.15.98:0/21190102 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f3fc8003970 con 0x7f3fd00a2530 2024-06-11T19:15:42.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.172+0000 7f3fb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:42.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.172+0000 7f3fb77fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:42.172 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":30,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:15:42.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 10 librados: watch_flush enter 2024-06-11T19:15:42.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 10 librados: watch_flush exit 2024-06-11T19:15:42.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 20 client.54816.objecter shutdown clearing up homeless session... 2024-06-11T19:15:42.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 10 client.54816.objecter successfully canceled tick 2024-06-11T19:15:42.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 -- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3fbc03fdb0 msgr2=0x7f3fbc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:42.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3fbc03fdb0 0x7f3fbc042270 secure :-1 s=READY pgs=4120 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4036000 tx=0x7f3fc4008030 comp rx=0 tx=0).stop 2024-06-11T19:15:42.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 10 monclient: shutdown 2024-06-11T19:15:42.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:42.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 -- 172.21.15.98:0/21190102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3fd00a2530 msgr2=0x7f3fd01cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3fd00a2530 0x7f3fd01cd720 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3fc800c450 tx=0x7f3fc800c910 comp rx=0 tx=0).stop 2024-06-11T19:15:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 -- 172.21.15.98:0/21190102 shutdown_connections 2024-06-11T19:15:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3fbc03fdb0 0x7f3fbc042270 unknown :-1 s=CLOSED pgs=4120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3fd013e1a0 0x7f3fd01361d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fd00a4920 0x7f3fd0135c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 --2- 172.21.15.98:0/21190102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3fd00a2530 0x7f3fd01cd720 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 -- 172.21.15.98:0/21190102 >> 172.21.15.98:0/21190102 conn(0x7f3fd00ab850 msgr2=0x7f3fd0137b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 -- 172.21.15.98:0/21190102 shutdown_connections 2024-06-11T19:15:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 -- 172.21.15.98:0/21190102 wait complete. 2024-06-11T19:15:42.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.174+0000 7f3fd51f0640 1 librados: shutdown 2024-06-11T19:15:42.184 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:15:42.184 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:15:42.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:15:42.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.284+0000 7f5d0e8c8640 10 monclient: get_monmap_and_config 2024-06-11T19:15:42.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.284+0000 7f5d0e8c8640 10 monclient: build_initial_monmap 2024-06-11T19:15:42.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.284+0000 7f5d0e8c8640 10 monclient: monmap: 2024-06-11T19:15:42.283 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:42.283 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:42.285178+0000 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:42.285178+0000 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:42.284 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.284+0000 7f5d0e8c8640 1 Processor -- start 2024-06-11T19:15:42.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.284+0000 7f5d0e8c8640 1 -- start start 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.284+0000 7f5d0e8c8640 10 monclient: init 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 10 monclient: _add_conns ranks=[2,0,8,4,5,6,10,1,3,9,7] 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d080a52a0 0x7f5d080a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 10 monclient(hunting): picked mon.noname-b con 0x7f5d080a52a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d0809d520 addr v1:172.21.15.98:6789/0 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d08073830 0x7f5d080a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 10 monclient(hunting): picked mon.noname-e con 0x7f5d08073830 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d0809d2e0 con 0x7f5d0809d520 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5d0809daa0 con 0x7f5d080a52a0 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5d080a5680 con 0x7f5d08073830 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:42.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0e8c8640 10 monclient(hunting): authenticate will time out at 1152.092487s 2024-06-11T19:15:42.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0ce65640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d08073830 0x7f5d080a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d07fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d080a52a0 0x7f5d080a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0ce65640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d08073830 0x7f5d080a3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:33814/0 (socket says 172.21.15.98:33814) 2024-06-11T19:15:42.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d07fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d080a52a0 0x7f5d080a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55540/0 (socket says 172.21.15.98:55540) 2024-06-11T19:15:42.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0ce65640 1 -- 172.21.15.98:0/3137388041 learned_addr learned my addr 172.21.15.98:0/3137388041 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:42.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0ce65640 10 monclient(hunting): get_auth_request con 0x7f5d08073830 auth_method 0 2024-06-11T19:15:42.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0ce65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0ce65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0ce65640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d07fff640 10 monclient(hunting): get_auth_request con 0x7f5d080a52a0 auth_method 0 2024-06-11T19:15:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d07fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0ce65640 1 -- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d08073830 msgr2=0x7f5d080a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:15:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0ce65640 1 -- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d08073830 msgr2=0x7f5d080a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0ce65640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d08073830 0x7f5d080a3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d07fff640 1 -- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d080a52a0 msgr2=0x7f5d080a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:15:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d07fff640 1 -- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d080a52a0 msgr2=0x7f5d080a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d07fff640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d080a52a0 0x7f5d080a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d0ce65640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d08073830 0x7f5d080a3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d07fff640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d080a52a0 0x7f5d080a1910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:42.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2565165056 0 0) 0x7f5d0809d2e0 con 0x7f5d0809d520 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d06ffd640 10 monclient(hunting): _negotiate 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d06ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d06ffd640 10 monclient(hunting): my global_id is 64494 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.285+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cf0003520 con 0x7f5d0809d520 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3672528427 0 0) 0x7f5cf0003520 con 0x7f5d0809d520 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d08073830 msgr2=0x7f5d080a3da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d08073830 0x7f5d080a3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d080a52a0 msgr2=0x7f5d080a1910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d080a52a0 0x7f5d080a1910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:42.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:15:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0809e9d0 con 0x7f5d0809d520 2024-06-11T19:15:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient: _finish_auth 0 2024-06-11T19:15:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:15:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7f5cfc001960 con 0x7f5d0809d520 2024-06-11T19:15:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0e8c8640 5 monclient: authenticate success, global_id 64494 2024-06-11T19:15:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0e8c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:15:42.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient: dump: 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:42.301 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:42.302 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:15:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 >> v1:172.21.15.98:6789/0 conn(0x7f5d0809d520 legacy=0x7f5d080a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient: _add_conns ranks=[3,6,0,7,4,1,8,2,5] 2024-06-11T19:15:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5cf0004230 0x7f5cf0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient(hunting): picked mon.g con 0x7f5cf0004230 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5cf0004e40 0x7f5cf00052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient(hunting): picked mon.d con 0x7f5cf0004e40 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 0x7f5cf000e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient(hunting): picked mon.a con 0x7f5cf000be40 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0809daa0 con 0x7f5cf000be40 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f5d080a5680 con 0x7f5cf0004230 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5cf000ebe0 con 0x7f5cf0004e40 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d06ffd640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0e8c8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 0x7f5cf000e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d077fe640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5cf0004e40 0x7f5cf00052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 10 monclient(hunting): get_auth_request con 0x7f5cf000be40 auth_method 0 2024-06-11T19:15:42.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d077fe640 10 monclient(hunting): get_auth_request con 0x7f5cf0004e40 auth_method 0 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d077fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d077fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:15:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 10 monclient(hunting): handle_auth_done global_id 64494 payload 293 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 1 -- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5cf0004e40 msgr2=0x7f5cf00052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5cf0004e40 0x7f5cf00052c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 1 -- 172.21.15.98:0/3137388041 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5cf0004230 msgr2=0x7f5cf0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5cf0004230 0x7f5cf0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 10 monclient: _finish_hunting 0 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 1 monclient: found mon.a 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 1 -- 172.21.15.98:0/3137388041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf0003520 con 0x7f5cf000be40 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 10 monclient: _finish_auth 0 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 10 monclient: _check_auth_tickets 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.286+0000 7f5d0ce65640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d0ce65640 1 -- 172.21.15.98:0/3137388041 reap_dead start 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d0ce65640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 0x7f5cf000e2d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5d08136780 tx=0x7f5cf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9527142c1cdd83f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d00032070 con 0x7f5cf000be40 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d06ffd640 10 monclient: dump: 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d00030070 con 0x7f5cf000be40 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d0e8c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d06ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d06ffd640 1 -- 172.21.15.98:0/3137388041 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d000098b0 con 0x7f5cf000be40 2024-06-11T19:15:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d0e8c8640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d0e8c8640 4 set_mon_vals no callback set 2024-06-11T19:15:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d06ffd640 10 monclient: dump: 2024-06-11T19:15:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d0e8c8640 10 monclient: shutdown 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d0e8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 msgr2=0x7f5cf000e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.287+0000 7f5d0e8c8640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 0x7f5cf000e2d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5d08136780 tx=0x7f5cf0005830 comp rx=0 tx=0).stop 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.288+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/3137388041 shutdown_connections 2024-06-11T19:15:42.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.288+0000 7f5d0e8c8640 1 --2- 172.21.15.98:0/3137388041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 0x7f5cf000e2d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.288+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/3137388041 >> 172.21.15.98:0/3137388041 conn(0x7f5d080a4780 msgr2=0x7f5d080a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.288+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/3137388041 shutdown_connections 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.288+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/3137388041 wait complete. 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.288+0000 7f5d0e8c8640 10 monclient: build_initial_monmap 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.288+0000 7f5d0e8c8640 10 monclient: monmap: 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:42.310 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.288+0000 7f5d0e8c8640 1 librados: starting msgr at 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.288+0000 7f5d0e8c8640 1 librados: starting objecter 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.288+0000 7f5d0e8c8640 1 Processor -- start 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 1 -- start start 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 1 librados: setting wanted keys 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 1 librados: calling monclient init 2024-06-11T19:15:42.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 10 monclient: init 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 10 monclient: _add_conns ranks=[0,8,5,3,6,4,7,1,2] 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 0x7f5d081cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 10 monclient(hunting): picked mon.noname-a con 0x7f5cf000be40 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d081cdce0 0x7f5d08122fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 10 monclient(hunting): picked mon.noname-e con 0x7f5d081cdce0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d081ce0c0 0x7f5d081234f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 10 monclient(hunting): picked mon.noname-h con 0x7f5d081ce0c0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d080a1a90 con 0x7f5cf000be40 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f5d080a1910 con 0x7f5d081ce0c0 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5d080a1c10 con 0x7f5d081cdce0 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d0e8c8640 10 monclient(hunting): authenticate will time out at 1152.096629s 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 0x7f5d081cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d077fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d081cdce0 0x7f5d08122fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 0x7f5d081cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54428/0 (socket says 172.21.15.98:54428) 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 1 -- 172.21.15.98:0/1129160954 learned_addr learned my addr 172.21.15.98:0/1129160954 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d077fe640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d081cdce0 0x7f5d08122fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:33828/0 (socket says 172.21.15.98:33828) 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 10 monclient(hunting): get_auth_request con 0x7f5cf000be40 auth_method 0 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d077fe640 10 monclient(hunting): get_auth_request con 0x7f5d081cdce0 auth_method 0 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d077fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d077fe640 1 -- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d081cdce0 msgr2=0x7f5d08122fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d077fe640 1 -- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d081cdce0 msgr2=0x7f5d08122fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d077fe640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d081cdce0 0x7f5d08122fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d077fe640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d081cdce0 0x7f5d08122fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 10 monclient(hunting): handle_auth_done global_id 64503 payload 824 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 1 -- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d081cdce0 msgr2=0x7f5d08122fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d081cdce0 0x7f5d08122fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 1 -- 172.21.15.98:0/1129160954 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d081ce0c0 msgr2=0x7f5d081234f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d081ce0c0 0x7f5d081234f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 10 monclient: _finish_hunting 0 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 1 monclient: found mon.noname-a 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.289+0000 7f5d07fff640 1 -- 172.21.15.98:0/1129160954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d08126bb0 con 0x7f5cf000be40 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d07fff640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 0x7f5d081cd7a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800d600 tx=0x7f5cf800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 10 client.?.objecter ms_handle_connect 0x7f5cf000be40 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5cf8009a70 con 0x7f5cf000be40 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 10 monclient: dump: 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.315 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 10 monclient: _finish_auth 0 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf8019070 con 0x7f5cf000be40 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d0e8c8640 5 monclient: authenticate success, global_id 64503 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d0e8c8640 10 monclient: _renew_subs 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d0e8c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/1129160954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d08126e90 con 0x7f5cf000be40 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5cf8009c70 con 0x7f5cf000be40 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d04ff9640 10 monclient: dump: 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.317 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d0e8c8640 10 client.64503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:42.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d0e8c8640 10 monclient: _renew_subs 2024-06-11T19:15:42.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d067fc640 4 set_mon_vals no callback set 2024-06-11T19:15:42.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d0e8c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:42.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/1129160954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d081277e0 con 0x7f5cf000be40 2024-06-11T19:15:42.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.290+0000 7f5d0e8c8640 1 librados: init done 2024-06-11T19:15:42.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d0e8c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:42.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d0e8c8640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08123a30 0x7f5d08123e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d0e8c8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/1129160954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d08050830 con 0x7f5d08123a30 2024-06-11T19:15:42.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d077fe640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08123a30 0x7f5d08123e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d077fe640 10 monclient: get_auth_request con 0x7f5d08123a30 auth_method 0 2024-06-11T19:15:42.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d077fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f5cf8009600 con 0x7f5cf000be40 2024-06-11T19:15:42.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:42.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:42.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:42.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d077fe640 10 monclient(hunting): handle_auth_done global_id 64512 payload 824 2024-06-11T19:15:42.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d077fe640 1 -- 172.21.15.98:0/1129160954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d08068a40 con 0x7f5d08123a30 2024-06-11T19:15:42.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5cf403fde0 0x7f5cf40422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6549+0+0 (secure 0 0 0) 0x7f5cf800f040 con 0x7f5cf000be40 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 10 client.64503.objecter ms_dispatch 0x7f5d08000b90 osd_map(283..283 src has 1..283) 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 3 client.64503.objecter handle_osd_map got epochs [283,283] > 0 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 3 client.64503.objecter handle_osd_map decoding full epoch 283 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d077fe640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08123a30 0x7f5d08123e10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc001930 tx=0x7f5cfc003cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 20 client.64503.objecter dump_active .. 0 homeless 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 10 client.64503.objecter ms_handle_connect 0x7f5d08123a30 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 10 client.64503.objecter resend_mon_ops 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d0ce65640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5cf403fde0 0x7f5cf40422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5cfc019020 con 0x7f5d08123a30 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:42.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d04ff9640 10 monclient: dump: 2024-06-11T19:15:42.322 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:42.322 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.322 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.322 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.322 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:42.322 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.322 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:42.322 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:42.322 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:42.322 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:42.322 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:42.323 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:42.323 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:42.323 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:42.323 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:42.323 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.291+0000 7f5d0ce65640 10 monclient: get_auth_request con 0x7f5cf403fde0 auth_method 0 2024-06-11T19:15:42.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.292+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5cfc011700 con 0x7f5d08123a30 2024-06-11T19:15:42.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.292+0000 7f5d04ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:42.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.292+0000 7f5d04ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:42.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.292+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08123a30 msgr2=0x7f5d08123e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:42.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.292+0000 7f5d04ff9640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08123a30 0x7f5d08123e10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc001930 tx=0x7f5cfc003cb0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.292+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08123a30 msgr2=0x7f5d08123e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:42.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.292+0000 7f5d04ff9640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08123a30 0x7f5d08123e10 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc001930 tx=0x7f5cfc003cb0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.292+0000 7f5d04ff9640 10 client.64503.objecter ms_handle_connect 0x7f5cf403fde0 2024-06-11T19:15:42.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.292+0000 7f5d0ce65640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5cf403fde0 0x7f5cf40422a0 secure :-1 s=READY pgs=4121 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000eb80 tx=0x7f5d0002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:42.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d0e8c8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:42.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d0e8c8640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0806f510 0x7f5d0806f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d0e8c8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/1129160954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d080685f0 con 0x7f5d0806f510 2024-06-11T19:15:42.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d077fe640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0806f510 0x7f5d0806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d077fe640 10 monclient: get_auth_request con 0x7f5d0806f510 auth_method 0 2024-06-11T19:15:42.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d077fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:42.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:42.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:42.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d077fe640 10 monclient(hunting): handle_auth_done global_id 64521 payload 824 2024-06-11T19:15:42.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d077fe640 1 -- 172.21.15.98:0/1129160954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5d080a1fa0 con 0x7f5d0806f510 2024-06-11T19:15:42.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d077fe640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0806f510 0x7f5d0806f920 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc000fb0 tx=0x7f5d0809eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:42.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d04ff9640 10 client.64503.objecter ms_handle_connect 0x7f5d0806f510 2024-06-11T19:15:42.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d04ff9640 10 client.64503.objecter resend_mon_ops 2024-06-11T19:15:42.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5cfc019040 con 0x7f5d0806f510 2024-06-11T19:15:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d04ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.298+0000 7f5d04ff9640 10 monclient: dump: 2024-06-11T19:15:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:42.328 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:42.329 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:42.329 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:42.329 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:42.329 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:42.329 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:42.329 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:42.329 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f5cfc011e40 con 0x7f5d0806f510 2024-06-11T19:15:42.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d04ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:42.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d04ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:42.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0806f510 msgr2=0x7f5d0806f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:42.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d04ff9640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0806f510 0x7f5d0806f920 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc000fb0 tx=0x7f5d0809eba0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d04ff9640 1 -- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0806f510 msgr2=0x7f5d0806f920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:42.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d04ff9640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0806f510 0x7f5d0806f920 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc000fb0 tx=0x7f5d0809eba0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.330 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:42.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:42.331 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:42.331 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:42.331 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:42.331 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 134126, 2024-06-11T19:15:42.331 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:42.331 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:42.331 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:42.331 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:42.331 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:42.331 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:42.331 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:42.332 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:42.332 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:42.332 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:42.332 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:42.332 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-06-11T19:15:42.332 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:42.332 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:42.332 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:42.332 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:42.332 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:42.332 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:42.332 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:42.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:42.333 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:42.333 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:42.333 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:42.333 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:42.333 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:42.333 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:42.333 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:42.333 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:42.333 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:42.333 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:42.334 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:42.334 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:42.334 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:42.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:42.334 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:42.334 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:42.334 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:42.334 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:42.334 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:42.334 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:42.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:42.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.335 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:42.335 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:42.335 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:42.335 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:42.335 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:42.335 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:42.335 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:42.335 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:42.335 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:42.336 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.337 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:42.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.338 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:42.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:42.338 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:42.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:42.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:42.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:42.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:42.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:42.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:42.339 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:42.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:42.339 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:42.340 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:42.340 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:42.340 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:42.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.340 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:42.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:42.342 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:42.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:42.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:42.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:42.343 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:42.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:42.343 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.344 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:42.344 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:42.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:42.344 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:42.344 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:42.344 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:42.344 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:42.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.344 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:42.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:42.345 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:42.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:42.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:42.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:42.345 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:42.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:42.346 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.346 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:42.346 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:42.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:42.346 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:42.346 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:42.346 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:42.347 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:42.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.347 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:42.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:42.347 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:42.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:42.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:42.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:42.347 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:42.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:42.348 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.348 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:42.348 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:42.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:42.348 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:42.348 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:42.349 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:42.349 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:42.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.349 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:42.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:42.349 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:42.349 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:42.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:42.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:42.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:42.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:42.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.350 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:42.350 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:42.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:42.351 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:42.351 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:42.351 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:42.351 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:42.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.351 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:42.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:42.351 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:42.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:42.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:42.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:42.352 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:42.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:42.352 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.352 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:42.352 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:42.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:42.353 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:42.353 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:42.353 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:42.353 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:42.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.353 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:42.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:42.353 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:42.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:42.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:42.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:42.354 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:42.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:42.354 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:42.354 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:42.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:42.355 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:42.355 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:42.355 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:42.355 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:42.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.355 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:42.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:42.355 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:42.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:42.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:42.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:42.356 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:42.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:42.356 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:42.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:42.357 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:42.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.357 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:42.357 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:42.357 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:42.357 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:42.357 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:42.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:42.357 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:42.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.357 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:42.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:42.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.358 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:42.358 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:42.358 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:42.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:42.358 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:42.358 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:42.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:42.358 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:42.358 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:42.359 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:15:42.359 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:42.359 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:42.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:42.359 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:42.359 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:42.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 10 librados: watch_flush enter 2024-06-11T19:15:42.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 10 librados: watch_flush exit 2024-06-11T19:15:42.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 20 client.64503.objecter shutdown clearing up homeless session... 2024-06-11T19:15:42.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 10 client.64503.objecter successfully canceled tick 2024-06-11T19:15:42.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/1129160954 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5cf403fde0 msgr2=0x7f5cf40422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:42.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5cf403fde0 0x7f5cf40422a0 secure :-1 s=READY pgs=4121 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000eb80 tx=0x7f5d0002e000 comp rx=0 tx=0).stop 2024-06-11T19:15:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 10 monclient: shutdown 2024-06-11T19:15:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 msgr2=0x7f5d081cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 0x7f5d081cd7a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800d600 tx=0x7f5cf800dac0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0ce65640 1 -- 172.21.15.98:0/1129160954 reap_dead start 2024-06-11T19:15:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/1129160954 shutdown_connections 2024-06-11T19:15:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 1 --2- 172.21.15.98:0/1129160954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5cf000be40 0x7f5d081cd7a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/1129160954 >> 172.21.15.98:0/1129160954 conn(0x7f5d080a4780 msgr2=0x7f5d080a3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/1129160954 shutdown_connections 2024-06-11T19:15:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 1 -- 172.21.15.98:0/1129160954 wait complete. 2024-06-11T19:15:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.299+0000 7f5d0e8c8640 1 librados: shutdown 2024-06-11T19:15:42.363 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:42.363 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:15:42.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:15:42.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.464+0000 7f616ed14640 10 monclient: get_monmap_and_config 2024-06-11T19:15:42.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.464+0000 7f616ed14640 10 monclient: build_initial_monmap 2024-06-11T19:15:42.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.464+0000 7f616ed14640 10 monclient: monmap: 2024-06-11T19:15:42.464 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:42.464 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.464 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:42.465418+0000 2024-06-11T19:15:42.464 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:42.465418+0000 2024-06-11T19:15:42.464 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:42.464 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.464 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:42.464 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:42.464 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:42.464 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:42.464 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:42.465 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:42.465 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:42.465 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:42.465 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:42.465 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:42.465 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:42.465 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 1 Processor -- start 2024-06-11T19:15:42.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 1 -- start start 2024-06-11T19:15:42.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 10 monclient: init 2024-06-11T19:15:42.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:42.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 10 monclient: _add_conns ranks=[0,5,7,9,4,6,2,10,3,8,1] 2024-06-11T19:15:42.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61680a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:15:42.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f61680a2530 0x7f616813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 10 monclient(hunting): picked mon.noname-h con 0x7f61680a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f616813e1a0 0x7f6168140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 10 monclient(hunting): picked mon.noname-i con 0x7f616813e1a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61680a7140 con 0x7f61680a4920 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f61680a2b50 con 0x7f61680a2530 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f61680a2e30 con 0x7f616813e1a0 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616ed14640 10 monclient(hunting): authenticate will time out at 1152.272663s 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616cab0640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f61680a4920 0x7f61680a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58280/0 (socket says 172.21.15.98:58280) 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f616cab0640 1 -- 172.21.15.98:0/3992838043 learned_addr learned my addr 172.21.15.98:0/3992838043 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.465+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3140925499 0 0) 0x7f61680a7140 con 0x7f61680a4920 2024-06-11T19:15:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f61680a2530 0x7f616813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient(hunting): _negotiate 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient(hunting): my global_id is 64530 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f616813e1a0 0x7f6168140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6158003520 con 0x7f61680a4920 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): get_auth_request con 0x7f61680a2530 auth_method 0 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient(hunting): get_auth_request con 0x7f616813e1a0 auth_method 0 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 1 -- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f616813e1a0 msgr2=0x7f6168140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 1 -- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f616813e1a0 msgr2=0x7f6168140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f616813e1a0 0x7f6168140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f616813e1a0 0x7f6168140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3651412273 0 0) 0x7f6158003520 con 0x7f61680a4920 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f616813e1a0 msgr2=0x7f6168140590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f616813e1a0 0x7f6168140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f61680a2530 msgr2=0x7f616813dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f61680a2530 0x7f616813dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient: _finish_hunting 0 2024-06-11T19:15:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61680a3110 con 0x7f61680a4920 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient: _finish_auth 0 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient: _check_auth_tickets 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7f615c001960 con 0x7f61680a4920 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616ed14640 5 monclient: authenticate success, global_id 64530 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616ed14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient: dump: 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 >> v1:172.21.15.98:6789/0 conn(0x7f61680a4920 legacy=0x7f61680a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient: _add_conns ranks=[2,5,0,3,7,4,8,1,6] 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6158004200 0x7f6158004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient(hunting): picked mon.b con 0x7f6158004200 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f61680a2530 0x7f6158004df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient(hunting): picked mon.h con 0x7f61680a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61580099b0 0x7f615800bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient(hunting): picked mon.a con 0x7f61580099b0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61680a2b50 con 0x7f61580099b0 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f61680a2e30 con 0x7f6158004200 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f615800c340 con 0x7f61680a2530 2024-06-11T19:15:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f61677fe640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616ed14640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616cab0640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6158004200 0x7f6158004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61580099b0 0x7f615800bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616cab0640 10 monclient(hunting): get_auth_request con 0x7f6158004200 auth_method 0 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616cab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616cab0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616cab0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient(hunting): get_auth_request con 0x7f61580099b0 auth_method 0 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f61680a2530 0x7f6158004df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): get_auth_request con 0x7f61680a2530 auth_method 0 2024-06-11T19:15:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616cab0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616cab0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616cab0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f6167fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient(hunting): handle_auth_done global_id 64530 payload 293 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 1 -- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f61680a2530 msgr2=0x7f6158004df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f61680a2530 0x7f6158004df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 1 -- 172.21.15.98:0/3992838043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6158004200 msgr2=0x7f6158004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6158004200 0x7f6158004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient: _finish_hunting 0 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 1 monclient: found mon.a 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 1 -- 172.21.15.98:0/3992838043 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6158003520 con 0x7f61580099b0 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient: _finish_auth 0 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 10 monclient: _check_auth_tickets 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616d2b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.466+0000 7f616cab0640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6158004200 0x7f6158004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f616d2b1640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61580099b0 0x7f615800bda0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6160009610 tx=0x7f6158005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c985069bd2d8daa5 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6160039070 con 0x7f61580099b0 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f61677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f61677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f61677fe640 10 monclient: dump: 2024-06-11T19:15:42.485 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6160032340 con 0x7f61580099b0 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f616ed14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f61677fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f616ed14640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f61677fe640 1 -- 172.21.15.98:0/3992838043 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6160009bd0 con 0x7f61580099b0 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f61677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f616ed14640 4 set_mon_vals no callback set 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f61677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.467+0000 7f61677fe640 10 monclient: dump: 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.468+0000 7f616ed14640 10 monclient: shutdown 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.468+0000 7f616ed14640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.468+0000 7f616ed14640 1 -- 172.21.15.98:0/3992838043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61580099b0 msgr2=0x7f615800bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.468+0000 7f616ed14640 1 --2- 172.21.15.98:0/3992838043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61580099b0 0x7f615800bda0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6160009610 tx=0x7f6158005830 comp rx=0 tx=0).stop 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.468+0000 7f616d2b1640 1 -- 172.21.15.98:0/3992838043 reap_dead start 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.468+0000 7f616ed14640 1 -- 172.21.15.98:0/3992838043 shutdown_connections 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.468+0000 7f616ed14640 1 -- 172.21.15.98:0/3992838043 >> 172.21.15.98:0/3992838043 conn(0x7f61680ab850 msgr2=0x7f61680a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.468+0000 7f616ed14640 1 -- 172.21.15.98:0/3992838043 shutdown_connections 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.468+0000 7f616ed14640 1 -- 172.21.15.98:0/3992838043 wait complete. 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.468+0000 7f616ed14640 10 monclient: build_initial_monmap 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.468+0000 7f616ed14640 10 monclient: monmap: 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 1 librados: starting msgr at 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 1 librados: starting objecter 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 1 Processor -- start 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 1 -- start start 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 1 librados: setting wanted keys 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 1 librados: calling monclient init 2024-06-11T19:15:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 10 monclient: init 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 10 monclient: _add_conns ranks=[8,0,6,5,1,4,7,2,3] 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61580099b0 0x7f6168137a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 10 monclient(hunting): picked mon.noname-e con 0x7f61580099b0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6168137f50 0x7f61681215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 10 monclient(hunting): picked mon.noname-a con 0x7f6168137f50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6168121b30 0x7f6168121fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 10 monclient(hunting): picked mon.noname-d con 0x7f6168121b30 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6168138e50 con 0x7f6168137f50 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f61680a84b0 con 0x7f6168121b30 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f61680a6da0 con 0x7f61580099b0 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616ed14640 10 monclient(hunting): authenticate will time out at 1152.276798s 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f6167fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6168137f50 0x7f61681215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616cab0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61580099b0 0x7f6168137a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f6167fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6168137f50 0x7f61681215f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54462/0 (socket says 172.21.15.98:54462) 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f6167fff640 1 -- 172.21.15.98:0/2068408843 learned_addr learned my addr 172.21.15.98:0/2068408843 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f6167fff640 10 monclient(hunting): get_auth_request con 0x7f6168137f50 auth_method 0 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f6167fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f6167fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f6167fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616cab0640 10 monclient(hunting): get_auth_request con 0x7f61580099b0 auth_method 0 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616cab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616cab0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616cab0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616d2b1640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6168121b30 0x7f6168121fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616d2b1640 10 monclient(hunting): get_auth_request con 0x7f6168121b30 auth_method 0 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616d2b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616d2b1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f616d2b1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f6167fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f6167fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.469+0000 7f6167fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616cab0640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61580099b0 msgr2=0x7f6168137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616d2b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616cab0640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61580099b0 msgr2=0x7f6168137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616d2b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61580099b0 0x7f6168137a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616d2b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61580099b0 0x7f6168137a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f6167fff640 10 monclient(hunting): handle_auth_done global_id 64539 payload 824 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f6167fff640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61580099b0 msgr2=0x7f6168137a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f6167fff640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61580099b0 0x7f6168137a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f6167fff640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6168121b30 msgr2=0x7f6168121fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f6167fff640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6168121b30 0x7f6168121fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f6167fff640 10 monclient: _finish_hunting 0 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f6167fff640 1 monclient: found mon.noname-a 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f6167fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f6167fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f6167fff640 1 -- 172.21.15.98:0/2068408843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6168124290 con 0x7f6168137f50 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f6167fff640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6168137f50 0x7f61681215f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6154011960 tx=0x7f6154011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 10 client.?.objecter ms_handle_connect 0x7f6168137f50 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6154019070 con 0x7f6168137f50 2024-06-11T19:15:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 10 monclient: dump: 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:42.495 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 10 monclient: _finish_auth 0 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 10 monclient: _check_auth_tickets 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6154004300 con 0x7f6168137f50 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616ed14640 5 monclient: authenticate success, global_id 64539 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616ed14640 10 monclient: _renew_subs 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616ed14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616ed14640 1 -- 172.21.15.98:0/2068408843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6168122720 con 0x7f6168137f50 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f61540044b0 con 0x7f6168137f50 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:42.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f61657fa640 10 monclient: dump: 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:42.497 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f6166ffd640 4 set_mon_vals no callback set 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616ed14640 10 client.64539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616ed14640 10 monclient: _renew_subs 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616ed14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616ed14640 1 -- 172.21.15.98:0/2068408843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61680a18c0 con 0x7f6168137f50 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.470+0000 7f616ed14640 1 librados: init done 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616ed14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616ed14640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616ed14640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616ed14640 1 -- 172.21.15.98:0/2068408843 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f61680aa490 con 0x7f61680a0ea0 2024-06-11T19:15:42.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f615401c070 con 0x7f6168137f50 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616cab0640 10 monclient: get_auth_request con 0x7f61680a0ea0 auth_method 0 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616cab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616cab0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616cab0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616cab0640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 msgr2=0x7f616809c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616cab0640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 msgr2=0x7f616809c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f61657fa640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f614c03fdb0 0x7f614c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6549+0+0 (secure 0 0 0) 0x7f615400f040 con 0x7f6168137f50 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f61657fa640 10 client.64539.objecter ms_dispatch 0x7f6168000b90 osd_map(283..283 src has 1..283) 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f61657fa640 3 client.64539.objecter handle_osd_map got epochs [283,283] > 0 2024-06-11T19:15:42.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f61657fa640 3 client.64539.objecter handle_osd_map decoding full epoch 283 2024-06-11T19:15:42.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f61657fa640 20 client.64539.objecter dump_active .. 0 homeless 2024-06-11T19:15:42.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.471+0000 7f616d2b1640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f614c03fdb0 0x7f614c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.472+0000 7f616d2b1640 10 monclient: get_auth_request con 0x7f614c03fdb0 auth_method 0 2024-06-11T19:15:42.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.472+0000 7f61657fa640 10 client.64539.objecter ms_handle_connect 0x7f614c03fdb0 2024-06-11T19:15:42.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.472+0000 7f616d2b1640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f614c03fdb0 0x7f614c042270 secure :-1 s=READY pgs=4122 cs=0 l=1 rev1=1 crypto rx=0x7f6160036000 tx=0x7f6160030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:42.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.672+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:42.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.672+0000 7f616cab0640 10 monclient: get_auth_request con 0x7f61680a0ea0 auth_method 0 2024-06-11T19:15:42.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.672+0000 7f616cab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:42.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.672+0000 7f616cab0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:42.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.672+0000 7f616cab0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:42.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.672+0000 7f616cab0640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 msgr2=0x7f616809c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:15:42.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.672+0000 7f616cab0640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 msgr2=0x7f616809c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:15:42.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.672+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:42.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:42.672+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:15:43.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.073+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:43.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.073+0000 7f616cab0640 10 monclient: get_auth_request con 0x7f61680a0ea0 auth_method 0 2024-06-11T19:15:43.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.073+0000 7f616cab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:43.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.073+0000 7f616cab0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:43.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.073+0000 7f616cab0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:43.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.073+0000 7f616cab0640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 msgr2=0x7f616809c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:15:43.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.073+0000 7f616cab0640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 msgr2=0x7f616809c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:15:43.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.073+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:43.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.073+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:15:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f616cab0640 10 monclient: get_auth_request con 0x7f61680a0ea0 auth_method 0 2024-06-11T19:15:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f616cab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f616cab0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f616cab0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f616cab0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:43.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f616cab0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f616cab0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f616cab0640 10 monclient(hunting): handle_auth_done global_id 84107 payload 824 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f616cab0640 1 -- 172.21.15.98:0/2068408843 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61680a42f0 con 0x7f61680a0ea0 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f615c001160 tx=0x7f615c007780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f61657fa640 10 client.64539.objecter ms_handle_connect 0x7f61680a0ea0 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.874+0000 7f61657fa640 10 client.64539.objecter resend_mon_ops 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.875+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f615c018440 con 0x7f61680a0ea0 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.875+0000 7f61657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.875+0000 7f61657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.875+0000 7f61657fa640 10 monclient: dump: 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:43.875 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:43.876 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:43.876 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:43.876 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:43.876 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:43.876 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:43.876 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:43.876 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:43.876 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:43.876 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:43.876 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:43.876 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:43.876 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:43.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.875+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f615c01c030 con 0x7f61680a0ea0 2024-06-11T19:15:43.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.875+0000 7f61657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:43.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.875+0000 7f61657fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:43.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.875+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 msgr2=0x7f616809c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:43.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.875+0000 7f61657fa640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f615c001160 tx=0x7f615c007780 comp rx=0 tx=0).stop 2024-06-11T19:15:43.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.875+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 msgr2=0x7f616809c9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:43.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.875+0000 7f61657fa640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61680a0ea0 0x7f616809c9e0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f615c001160 tx=0x7f615c007780 comp rx=0 tx=0).stop 2024-06-11T19:15:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.882+0000 7f616ed14640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.882+0000 7f616ed14640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f616809cf20 0x7f616809d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.882+0000 7f616ed14640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.882+0000 7f616ed14640 1 -- 172.21.15.98:0/2068408843 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6168050830 con 0x7f616809cf20 2024-06-11T19:15:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.882+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f616809cf20 0x7f616809d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.882+0000 7f616cab0640 10 monclient: get_auth_request con 0x7f616809cf20 auth_method 0 2024-06-11T19:15:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.882+0000 7f616cab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.882+0000 7f616cab0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.882+0000 7f616cab0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.882+0000 7f616cab0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.882+0000 7f616cab0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.882+0000 7f616cab0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f616cab0640 10 monclient(hunting): handle_auth_done global_id 84116 payload 824 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f616cab0640 1 -- 172.21.15.98:0/2068408843 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f61680a4d00 con 0x7f616809cf20 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f616cab0640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f616809cf20 0x7f616809d300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f615c01f000 tx=0x7f615c001cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 10 client.64539.objecter ms_handle_connect 0x7f616809cf20 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 10 client.64539.objecter resend_mon_ops 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f615c018d50 con 0x7f616809cf20 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 10 monclient: dump: 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:43.884 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f615c018d50 con 0x7f616809cf20 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f616809cf20 msgr2=0x7f616809d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f616809cf20 0x7f616809d300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f615c01f000 tx=0x7f615c001cd0 comp rx=0 tx=0).stop 2024-06-11T19:15:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f616809cf20 msgr2=0x7f616809d300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.883+0000 7f61657fa640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f616809cf20 0x7f616809d300 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f615c01f000 tx=0x7f615c001cd0 comp rx=0 tx=0).stop 2024-06-11T19:15:43.886 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:43.886 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:43.886 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:43.886 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:43.886 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:43.886 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 15657, 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:43.887 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:43.888 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:43.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:43.891 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:43.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:43.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:43.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:43.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:43.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:43.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:43.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:43.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:43.898 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:43.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:43.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.902 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:43.902 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:43.904 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:43.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:43.906 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:43.906 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:43.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:43.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:43.906 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:43.906 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:43.906 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:43.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:43.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:43.909 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.910 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:43.911 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:43.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 10 librados: watch_flush enter 2024-06-11T19:15:43.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 10 librados: watch_flush exit 2024-06-11T19:15:43.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 20 client.64539.objecter shutdown clearing up homeless session... 2024-06-11T19:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 10 client.64539.objecter successfully canceled tick 2024-06-11T19:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f614c03fdb0 msgr2=0x7f614c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f614c03fdb0 0x7f614c042270 secure :-1 s=READY pgs=4122 cs=0 l=1 rev1=1 crypto rx=0x7f6160036000 tx=0x7f6160030000 comp rx=0 tx=0).stop 2024-06-11T19:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 10 monclient: shutdown 2024-06-11T19:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 1 -- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6168137f50 msgr2=0x7f61681215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 1 --2- 172.21.15.98:0/2068408843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6168137f50 0x7f61681215f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6154011960 tx=0x7f6154011d60 comp rx=0 tx=0).stop 2024-06-11T19:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616d2b1640 1 -- 172.21.15.98:0/2068408843 reap_dead start 2024-06-11T19:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616d2b1640 1 -- 172.21.15.98:0/2068408843 reap_dead start 2024-06-11T19:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 1 -- 172.21.15.98:0/2068408843 shutdown_connections 2024-06-11T19:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 1 -- 172.21.15.98:0/2068408843 >> 172.21.15.98:0/2068408843 conn(0x7f61680ab850 msgr2=0x7f61680a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 1 -- 172.21.15.98:0/2068408843 shutdown_connections 2024-06-11T19:15:43.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 1 -- 172.21.15.98:0/2068408843 wait complete. 2024-06-11T19:15:43.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:43.884+0000 7f616ed14640 1 librados: shutdown 2024-06-11T19:15:43.915 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:43.915 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:15:43.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:15:44.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.022+0000 7f31e42b2640 10 monclient: get_monmap_and_config 2024-06-11T19:15:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.022+0000 7f31e42b2640 10 monclient: build_initial_monmap 2024-06-11T19:15:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.022+0000 7f31e42b2640 10 monclient: monmap: 2024-06-11T19:15:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:44.023674+0000 2024-06-11T19:15:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:44.023674+0000 2024-06-11T19:15:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:44.023 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 1 Processor -- start 2024-06-11T19:15:44.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 1 -- start start 2024-06-11T19:15:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 10 monclient: init 2024-06-11T19:15:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 10 monclient: _add_conns ranks=[1,4,2,9,3,10,8,5,6,7,0] 2024-06-11T19:15:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f31dc0a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a2530 0x7f31dc13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 10 monclient(hunting): picked mon.noname-c con 0x7f31dc0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:15:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc13e1a0 0x7f31dc140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 10 monclient(hunting): picked mon.noname-b con 0x7f31dc13e1a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31dc0a7140 con 0x7f31dc0a4920 2024-06-11T19:15:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f31dc0a2b50 con 0x7f31dc13e1a0 2024-06-11T19:15:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f31dc0a2e30 con 0x7f31dc0a2530 2024-06-11T19:15:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.023+0000 7f31e42b2640 10 monclient(hunting): authenticate will time out at 1153.830997s 2024-06-11T19:15:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e184d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a2530 0x7f31dc13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc13e1a0 0x7f31dc140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc13e1a0 0x7f31dc140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:59798/0 (socket says 172.21.15.98:59798) 2024-06-11T19:15:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e184d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a2530 0x7f31dc13dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:53624/0 (socket says 172.21.15.98:53624) 2024-06-11T19:15:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 1 -- 172.21.15.98:0/3934392902 learned_addr learned my addr 172.21.15.98:0/3934392902 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 10 monclient(hunting): get_auth_request con 0x7f31dc13e1a0 auth_method 0 2024-06-11T19:15:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e184d640 10 monclient(hunting): get_auth_request con 0x7f31dc0a2530 auth_method 0 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e184d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e184d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e184d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e184d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e184d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e184d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 10 monclient(hunting): handle_auth_done global_id 84125 payload 274 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 1 -- 172.21.15.98:0/3934392902 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a2530 msgr2=0x7f31dc13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 1 --2- 172.21.15.98:0/3934392902 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a2530 0x7f31dc13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 1 -- 172.21.15.98:0/3934392902 >> v1:172.21.15.117:6789/0 conn(0x7f31dc0a4920 legacy=0x7f31dc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 1 monclient: found mon.noname-b 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e284f640 1 -- 172.21.15.98:0/3934392902 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc0a3110 con 0x7f31dc13e1a0 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.024+0000 7f31e184d640 1 --2- 172.21.15.98:0/3934392902 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a2530 0x7f31dc13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e284f640 1 --2- 172.21.15.98:0/3934392902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc13e1a0 0x7f31dc140590 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009e80 tx=0x7f31d402bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ab9cd0a4da30832 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 1 -- 172.21.15.98:0/3934392902 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31d402cb80 con 0x7f31dc13e1a0 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 10 monclient: dump: 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.036 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.038 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e42b2640 5 monclient: authenticate success, global_id 84125 2024-06-11T19:15:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 1 -- 172.21.15.98:0/3934392902 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31d402cd40 con 0x7f31dc13e1a0 2024-06-11T19:15:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e42b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e42b2640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e42b2640 4 set_mon_vals no callback set 2024-06-11T19:15:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 1 -- 172.21.15.98:0/3934392902 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31d4034860 con 0x7f31dc13e1a0 2024-06-11T19:15:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e104c640 10 monclient: dump: 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.039 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e42b2640 10 monclient: shutdown 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e42b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e42b2640 1 -- 172.21.15.98:0/3934392902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc13e1a0 msgr2=0x7f31dc140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e42b2640 1 --2- 172.21.15.98:0/3934392902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc13e1a0 0x7f31dc140590 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009e80 tx=0x7f31d402bd70 comp rx=0 tx=0).stop 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.025+0000 7f31e42b2640 1 -- 172.21.15.98:0/3934392902 shutdown_connections 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 1 --2- 172.21.15.98:0/3934392902 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc13e1a0 0x7f31dc140590 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 1 --2- 172.21.15.98:0/3934392902 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a2530 0x7f31dc13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 1 -- 172.21.15.98:0/3934392902 >> 172.21.15.98:0/3934392902 conn(0x7f31dc0ab850 msgr2=0x7f31dc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 1 -- 172.21.15.98:0/3934392902 shutdown_connections 2024-06-11T19:15:44.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 1 -- 172.21.15.98:0/3934392902 wait complete. 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 10 monclient: build_initial_monmap 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 10 monclient: monmap: 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:44.042 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 1 librados: starting msgr at 2024-06-11T19:15:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 1 librados: starting objecter 2024-06-11T19:15:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 1 Processor -- start 2024-06-11T19:15:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 1 -- start start 2024-06-11T19:15:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 1 librados: setting wanted keys 2024-06-11T19:15:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 1 librados: calling monclient init 2024-06-11T19:15:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.026+0000 7f31e42b2640 10 monclient: init 2024-06-11T19:15:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 10 monclient: _add_conns ranks=[3,4,0,6,5,1,7,2,8] 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31dc0a2530 0x7f31dc137190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 10 monclient(hunting): picked mon.noname-g con 0x7f31dc0a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a4920 0x7f31dc09fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 10 monclient(hunting): picked mon.noname-c con 0x7f31dc0a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc13e1a0 0x7f31dc0a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 10 monclient(hunting): picked mon.noname-a con 0x7f31dc13e1a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc138e50 con 0x7f31dc13e1a0 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f31dc0a84b0 con 0x7f31dc0a2530 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f31dc0a6da0 con 0x7f31dc0a4920 2024-06-11T19:15:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e42b2640 10 monclient(hunting): authenticate will time out at 1153.834383s 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a4920 0x7f31dc09fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a4920 0x7f31dc09fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:53632/0 (socket says 172.21.15.98:53632) 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 1 -- 172.21.15.98:0/2342358830 learned_addr learned my addr 172.21.15.98:0/2342358830 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 10 monclient(hunting): get_auth_request con 0x7f31dc0a4920 auth_method 0 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e284f640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc13e1a0 0x7f31dc0a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e284f640 10 monclient(hunting): get_auth_request con 0x7f31dc13e1a0 auth_method 0 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e284f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e284f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e284f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e284f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e284f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e284f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e204e640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31dc0a2530 0x7f31dc137190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e204e640 10 monclient(hunting): get_auth_request con 0x7f31dc0a2530 auth_method 0 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e204e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e204e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e204e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 10 monclient(hunting): handle_auth_done global_id 23683 payload 824 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 1 -- 172.21.15.98:0/2342358830 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31dc0a2530 msgr2=0x7f31dc137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31dc0a2530 0x7f31dc137190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 1 -- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc13e1a0 msgr2=0x7f31dc0a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc13e1a0 0x7f31dc0a0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 1 monclient: found mon.noname-c 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 1 -- 172.21.15.98:0/2342358830 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc0aa750 con 0x7f31dc0a4920 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31e184d640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a4920 0x7f31dc09fe30 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00e9b0 tx=0x7f31cc00ee70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31d2ffd640 10 client.?.objecter ms_handle_connect 0x7f31dc0a4920 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.027+0000 7f31d2ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31cc017070 con 0x7f31dc0a4920 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:15:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 10 monclient: dump: 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.047 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc00ce70 con 0x7f31dc0a4920 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 5 monclient: authenticate success, global_id 23683 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 10 monclient: _renew_subs 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 1 -- 172.21.15.98:0/2342358830 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31dc1232c0 con 0x7f31dc0a4920 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31cc0138e0 con 0x7f31dc0a4920 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31d2ffd640 10 monclient: dump: 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.050 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.050 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e084b640 4 set_mon_vals no callback set 2024-06-11T19:15:44.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 10 client.23683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:44.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 10 monclient: _renew_subs 2024-06-11T19:15:44.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:44.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 1 -- 172.21.15.98:0/2342358830 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f31dc123880 con 0x7f31dc0a4920 2024-06-11T19:15:44.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 1 librados: init done 2024-06-11T19:15:44.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:44.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a08b0 0x7f31dc13bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.028+0000 7f31e42b2640 1 -- 172.21.15.98:0/2342358830 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f31dc050830 con 0x7f31dc0a08b0 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e204e640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a08b0 0x7f31dc13bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f31cc01a070 con 0x7f31dc0a4920 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e204e640 10 monclient: get_auth_request con 0x7f31dc0a08b0 auth_method 0 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e204e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e204e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e204e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e204e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e204e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e204e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e204e640 10 monclient(hunting): handle_auth_done global_id 23692 payload 824 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e204e640 1 -- 172.21.15.98:0/2342358830 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31dc1365e0 con 0x7f31dc0a08b0 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f31c003fdb0 0x7f31c0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(285..285 src has 1..285) ==== 7069+0+0 (secure 0 0 0) 0x7f31cc00f040 con 0x7f31dc0a4920 2024-06-11T19:15:44.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 10 client.23683.objecter ms_dispatch 0x7f31dc000b90 osd_map(285..285 src has 1..285) 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 3 client.23683.objecter handle_osd_map got epochs [285,285] > 0 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 3 client.23683.objecter handle_osd_map decoding full epoch 285 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e204e640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a08b0 0x7f31dc13bc30 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f31d800e4d0 tx=0x7f31d800e990 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 20 client.23683.objecter dump_active .. 0 homeless 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 10 client.23683.objecter ms_handle_connect 0x7f31dc0a08b0 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 10 client.23683.objecter resend_mon_ops 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e284f640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f31c003fdb0 0x7f31c0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31d80067d0 con 0x7f31dc0a08b0 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 10 monclient: dump: 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.052 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31e284f640 10 monclient: get_auth_request con 0x7f31c003fdb0 auth_method 0 2024-06-11T19:15:44.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f31d8007240 con 0x7f31dc0a08b0 2024-06-11T19:15:44.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:44.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:44.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a08b0 msgr2=0x7f31dc13bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a08b0 0x7f31dc13bc30 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f31d800e4d0 tx=0x7f31d800e990 comp rx=0 tx=0).stop 2024-06-11T19:15:44.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a08b0 msgr2=0x7f31dc13bc30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:44.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.029+0000 7f31d2ffd640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a08b0 0x7f31dc13bc30 secure :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f31d800e4d0 tx=0x7f31d800e990 comp rx=0 tx=0).stop 2024-06-11T19:15:44.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.030+0000 7f31e284f640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f31c003fdb0 0x7f31c0042270 secure :-1 s=READY pgs=4123 cs=0 l=1 rev1=1 crypto rx=0x7f31d4035000 tx=0x7f31d4030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.030+0000 7f31d2ffd640 10 client.23683.objecter ms_handle_connect 0x7f31c003fdb0 2024-06-11T19:15:44.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e42b2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:44.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e42b2640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc13c170 0x7f31dc13c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e42b2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e42b2640 1 -- 172.21.15.98:0/2342358830 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f31dc0a8820 con 0x7f31dc13c170 2024-06-11T19:15:44.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e204e640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc13c170 0x7f31dc13c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e204e640 10 monclient: get_auth_request con 0x7f31dc13c170 auth_method 0 2024-06-11T19:15:44.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e204e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e204e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e204e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e204e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e204e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e204e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e204e640 10 monclient(hunting): handle_auth_done global_id 23701 payload 824 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e204e640 1 -- 172.21.15.98:0/2342358830 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f31dc0a0c90 con 0x7f31dc13c170 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31d2ffd640 10 client.23683.objecter ms_handle_connect 0x7f31dc13c170 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31d2ffd640 10 client.23683.objecter resend_mon_ops 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31e204e640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc13c170 0x7f31dc13c580 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f31d800efd0 tx=0x7f31d8003760 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31d80288a0 con 0x7f31dc13c170 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.036+0000 7f31d2ffd640 10 monclient: dump: 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.059 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f31d8028430 con 0x7f31dc13c170 2024-06-11T19:15:44.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31d2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:44.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31d2ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:44.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc13c170 msgr2=0x7f31dc13c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31d2ffd640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc13c170 0x7f31dc13c580 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f31d800efd0 tx=0x7f31d8003760 comp rx=0 tx=0).stop 2024-06-11T19:15:44.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31d2ffd640 1 -- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc13c170 msgr2=0x7f31dc13c580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:44.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31d2ffd640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc13c170 0x7f31dc13c580 secure :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f31d800efd0 tx=0x7f31d8003760 comp rx=0 tx=0).stop 2024-06-11T19:15:44.059 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:44.059 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:44.060 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:44.060 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:44.060 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:44.060 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 394254, 2024-06-11T19:15:44.060 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:44.060 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:44.060 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:44.060 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:44.060 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:44.060 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:44.060 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:44.060 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.061 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:44.063 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.064 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:44.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.066 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:44.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:44.066 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:44.066 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:44.067 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.068 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:44.070 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:44.070 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.070 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.070 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:44.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.073 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:44.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:44.073 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:44.073 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:44.074 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.074 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.074 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:44.074 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:44.074 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.074 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.074 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.075 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:44.078 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.080 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.083 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:44.083 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 10 librados: watch_flush enter 2024-06-11T19:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 10 librados: watch_flush exit 2024-06-11T19:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 20 client.23683.objecter shutdown clearing up homeless session... 2024-06-11T19:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 10 client.23683.objecter successfully canceled tick 2024-06-11T19:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 1 -- 172.21.15.98:0/2342358830 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f31c003fdb0 msgr2=0x7f31c0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f31c003fdb0 0x7f31c0042270 secure :-1 s=READY pgs=4123 cs=0 l=1 rev1=1 crypto rx=0x7f31d4035000 tx=0x7f31d4030000 comp rx=0 tx=0).stop 2024-06-11T19:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 10 monclient: shutdown 2024-06-11T19:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 1 -- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a4920 msgr2=0x7f31dc09fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 1 --2- 172.21.15.98:0/2342358830 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a4920 0x7f31dc09fe30 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00e9b0 tx=0x7f31cc00ee70 comp rx=0 tx=0).stop 2024-06-11T19:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e284f640 1 -- 172.21.15.98:0/2342358830 reap_dead start 2024-06-11T19:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e284f640 1 -- 172.21.15.98:0/2342358830 reap_dead start 2024-06-11T19:15:44.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 1 -- 172.21.15.98:0/2342358830 shutdown_connections 2024-06-11T19:15:44.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 1 -- 172.21.15.98:0/2342358830 >> 172.21.15.98:0/2342358830 conn(0x7f31dc0ab850 msgr2=0x7f31dc0a9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:44.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 1 -- 172.21.15.98:0/2342358830 shutdown_connections 2024-06-11T19:15:44.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 1 -- 172.21.15.98:0/2342358830 wait complete. 2024-06-11T19:15:44.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.037+0000 7f31e42b2640 1 librados: shutdown 2024-06-11T19:15:44.086 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:44.086 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:15:44.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:15:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.186+0000 7f46fe4c0640 10 monclient: get_monmap_and_config 2024-06-11T19:15:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.186+0000 7f46fe4c0640 10 monclient: build_initial_monmap 2024-06-11T19:15:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.186+0000 7f46fe4c0640 10 monclient: monmap: 2024-06-11T19:15:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:44.187815+0000 2024-06-11T19:15:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:44.187815+0000 2024-06-11T19:15:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:44.187 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:44.187 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:44.187 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:44.187 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:44.187 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:44.187 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:44.187 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:44.187 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:44.187 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:44.187 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.187+0000 7f46fe4c0640 1 Processor -- start 2024-06-11T19:15:44.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.187+0000 7f46fe4c0640 1 -- start start 2024-06-11T19:15:44.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.187+0000 7f46fe4c0640 10 monclient: init 2024-06-11T19:15:44.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fe4c0640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fe4c0640 10 monclient: _add_conns ranks=[0,1,2,10,5,6,3,8,7,9,4] 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fe4c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46f80a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fe4c0640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f46f80a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fe4c0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46f813f380 0x7f46f8141770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fe4c0640 10 monclient(hunting): picked mon.noname-b con 0x7f46f813f380 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fe4c0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46f80a7140 con 0x7f46f80a4920 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fe4c0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46f80a2b50 con 0x7f46f80a2530 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fe4c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fe4c0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f46f80a2e30 con 0x7f46f813f380 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fe4c0640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fe4c0640 10 monclient(hunting): authenticate will time out at 1153.995126s 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46f7fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f46f80a4920 0x7f46f80a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35708/0 (socket says 172.21.15.98:35708) 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46f7fff640 1 -- 172.21.15.98:0/4128541722 learned_addr learned my addr 172.21.15.98:0/4128541722 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:15:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 1 --2- 172.21.15.98:0/4128541722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46f813f380 0x7f46f8141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 10 monclient(hunting): get_auth_request con 0x7f46f813f380 auth_method 0 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46f6ffd640 1 -- 172.21.15.98:0/4128541722 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3703014164 0 0) 0x7f46f80a7140 con 0x7f46f80a4920 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46f6ffd640 10 monclient(hunting): _negotiate 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46f6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46f6ffd640 10 monclient(hunting): my global_id is 64557 2024-06-11T19:15:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46f6ffd640 1 -- 172.21.15.98:0/4128541722 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46e0003520 con 0x7f46f80a4920 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 10 monclient(hunting): handle_auth_done global_id 84134 payload 274 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 1 -- 172.21.15.98:0/4128541722 >> v1:172.21.15.117:6789/0 conn(0x7f46f80a2530 legacy=0x7f46f813dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 1 -- 172.21.15.98:0/4128541722 >> v1:172.21.15.98:6789/0 conn(0x7f46f80a4920 legacy=0x7f46f80a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 1 monclient: found mon.noname-b 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.188+0000 7f46fca5d640 1 -- 172.21.15.98:0/4128541722 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f80a3110 con 0x7f46f813f380 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46fca5d640 1 --2- 172.21.15.98:0/4128541722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46f813f380 0x7f46f8141770 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009dc0 tx=0x7f46f002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=546bf411a51adec0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 1 -- 172.21.15.98:0/4128541722 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46f0039070 con 0x7f46f813f380 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 10 monclient: dump: 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.200 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46fe4c0640 5 monclient: authenticate success, global_id 84134 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 1 -- 172.21.15.98:0/4128541722 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46f002ce20 con 0x7f46f813f380 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46fe4c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46fe4c0640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 1 -- 172.21.15.98:0/4128541722 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46f00327b0 con 0x7f46f813f380 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46fe4c0640 4 set_mon_vals no callback set 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46f6ffd640 10 monclient: dump: 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.202 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.203 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46fe4c0640 10 monclient: shutdown 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46fe4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46fe4c0640 1 -- 172.21.15.98:0/4128541722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46f813f380 msgr2=0x7f46f8141770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.189+0000 7f46fe4c0640 1 --2- 172.21.15.98:0/4128541722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46f813f380 0x7f46f8141770 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009dc0 tx=0x7f46f002be00 comp rx=0 tx=0).stop 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.190+0000 7f46fe4c0640 1 -- 172.21.15.98:0/4128541722 shutdown_connections 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.190+0000 7f46fe4c0640 1 --2- 172.21.15.98:0/4128541722 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46f813f380 0x7f46f8141770 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.190+0000 7f46fe4c0640 1 -- 172.21.15.98:0/4128541722 >> 172.21.15.98:0/4128541722 conn(0x7f46f80ab850 msgr2=0x7f46f80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.190+0000 7f46fe4c0640 1 -- 172.21.15.98:0/4128541722 shutdown_connections 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.190+0000 7f46fe4c0640 1 -- 172.21.15.98:0/4128541722 wait complete. 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.190+0000 7f46fe4c0640 10 monclient: build_initial_monmap 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.190+0000 7f46fe4c0640 10 monclient: monmap: 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:44.204 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.190+0000 7f46fe4c0640 1 librados: starting msgr at 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.190+0000 7f46fe4c0640 1 librados: starting objecter 2024-06-11T19:15:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.190+0000 7f46fe4c0640 1 Processor -- start 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 1 -- start start 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 1 librados: setting wanted keys 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 1 librados: calling monclient init 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 10 monclient: init 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 10 monclient: _add_conns ranks=[6,8,0,1,3,5,4,7,2] 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f80a2530 0x7f46f8135790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 10 monclient(hunting): picked mon.noname-d con 0x7f46f80a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f46f80a4920 0x7f46f809d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 10 monclient(hunting): picked mon.noname-e con 0x7f46f80a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46f813f380 0x7f46f809dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 10 monclient(hunting): picked mon.noname-a con 0x7f46f813f380 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:15:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f46f8138e50 con 0x7f46f813f380 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f46f80a84b0 con 0x7f46f80a2530 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f46f80a6da0 con 0x7f46f80a4920 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fe4c0640 10 monclient(hunting): authenticate will time out at 1153.998606s 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f7fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f80a2530 0x7f46f8135790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f46f80a4920 0x7f46f809d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f7fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f80a2530 0x7f46f8135790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34584/0 (socket says 172.21.15.98:34584) 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f7fff640 1 -- 172.21.15.98:0/1055260757 learned_addr learned my addr 172.21.15.98:0/1055260757 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f7fff640 10 monclient(hunting): get_auth_request con 0x7f46f80a2530 auth_method 0 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f77fe640 10 monclient(hunting): get_auth_request con 0x7f46f80a4920 auth_method 0 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fca5d640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46f813f380 0x7f46f809dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fca5d640 10 monclient(hunting): get_auth_request con 0x7f46f813f380 auth_method 0 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fca5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fca5d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fca5d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fca5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fca5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.191+0000 7f46fca5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fca5d640 10 monclient(hunting): handle_auth_done global_id 64566 payload 824 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fca5d640 1 -- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f46f80a4920 msgr2=0x7f46f809d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fca5d640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f46f80a4920 0x7f46f809d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fca5d640 1 -- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f80a2530 msgr2=0x7f46f8135790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fca5d640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f80a2530 0x7f46f8135790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fca5d640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fca5d640 1 monclient: found mon.noname-a 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fca5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fca5d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:15:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fca5d640 1 -- 172.21.15.98:0/1055260757 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f80a11b0 con 0x7f46f813f380 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f7fff640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f80a2530 0x7f46f8135790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f77fe640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f46f80a4920 0x7f46f809d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fca5d640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46f813f380 0x7f46f809dcf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f46f0033000 tx=0x7f46f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 10 client.?.objecter ms_handle_connect 0x7f46f813f380 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46f0039050 con 0x7f46f813f380 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 10 monclient: dump: 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fe4c0640 5 monclient: authenticate success, global_id 64566 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46f0045070 con 0x7f46f813f380 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fe4c0640 10 monclient: _renew_subs 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fe4c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fe4c0640 1 -- 172.21.15.98:0/1055260757 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f809ffe0 con 0x7f46f813f380 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46f00046c0 con 0x7f46f813f380 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f67fc640 4 set_mon_vals no callback set 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46f4ff9640 10 monclient: dump: 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fe4c0640 10 client.64566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fe4c0640 10 monclient: _renew_subs 2024-06-11T19:15:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fe4c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fe4c0640 1 -- 172.21.15.98:0/1055260757 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f813f0a0 con 0x7f46f813f380 2024-06-11T19:15:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.192+0000 7f46fe4c0640 1 librados: init done 2024-06-11T19:15:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46fe4c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46fe4c0640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f8135cd0 0x7f46f802cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46fe4c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46fe4c0640 1 -- 172.21.15.98:0/1055260757 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f46f8050830 con 0x7f46f8135cd0 2024-06-11T19:15:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f7fff640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f8135cd0 0x7f46f802cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f7fff640 10 monclient: get_auth_request con 0x7f46f8135cd0 auth_method 0 2024-06-11T19:15:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f46f0004250 con 0x7f46f813f380 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f7fff640 10 monclient(hunting): handle_auth_done global_id 54843 payload 824 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f7fff640 1 -- 172.21.15.98:0/1055260757 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46f80a4d00 con 0x7f46f8135cd0 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f4ff9640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f46e403fce0 0x7f46e40421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(285..285 src has 1..285) ==== 7069+0+0 (secure 0 0 0) 0x7f46f0074560 con 0x7f46f813f380 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f4ff9640 10 client.64566.objecter ms_dispatch 0x7f46f8000b90 osd_map(285..285 src has 1..285) 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f4ff9640 3 client.64566.objecter handle_osd_map got epochs [285,285] > 0 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f4ff9640 3 client.64566.objecter handle_osd_map decoding full epoch 285 2024-06-11T19:15:44.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.193+0000 7f46f7fff640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f8135cd0 0x7f46f802cb40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f46e8000f60 tx=0x7f46e800ff70 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 20 client.64566.objecter dump_active .. 0 homeless 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 10 client.64566.objecter ms_handle_connect 0x7f46f8135cd0 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 10 client.64566.objecter resend_mon_ops 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46e800d800 con 0x7f46f8135cd0 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 10 monclient: dump: 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.216 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f77fe640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f46e403fce0 0x7f46e40421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f77fe640 10 monclient: get_auth_request con 0x7f46e403fce0 auth_method 0 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f46e8018070 con 0x7f46f8135cd0 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:44.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f8135cd0 msgr2=0x7f46f802cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f8135cd0 0x7f46f802cb40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f46e8000f60 tx=0x7f46e800ff70 comp rx=0 tx=0).stop 2024-06-11T19:15:44.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f8135cd0 msgr2=0x7f46f802cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:44.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f8135cd0 0x7f46f802cb40 secure :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f46e8000f60 tx=0x7f46e800ff70 comp rx=0 tx=0).stop 2024-06-11T19:15:44.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f4ff9640 10 client.64566.objecter ms_handle_connect 0x7f46e403fce0 2024-06-11T19:15:44.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.194+0000 7f46f77fe640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f46e403fce0 0x7f46e40421a0 secure :-1 s=READY pgs=4124 cs=0 l=1 rev1=1 crypto rx=0x7f46ec000f60 tx=0x7f46ec009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46fe4c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46fe4c0640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f802d080 0x7f46f802d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46fe4c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46fe4c0640 1 -- 172.21.15.98:0/1055260757 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f46f80a16d0 con 0x7f46f802d080 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46f7fff640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f802d080 0x7f46f802d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46f7fff640 10 monclient: get_auth_request con 0x7f46f802d080 auth_method 0 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46f7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46f7fff640 10 monclient(hunting): handle_auth_done global_id 54852 payload 824 2024-06-11T19:15:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.200+0000 7f46f7fff640 1 -- 172.21.15.98:0/1055260757 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f46f813b180 con 0x7f46f802d080 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f7fff640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f802d080 0x7f46f802d490 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f46e8017000 tx=0x7f46e80057f0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 10 client.64566.objecter ms_handle_connect 0x7f46f802d080 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 10 client.64566.objecter resend_mon_ops 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46e8007750 con 0x7f46f802d080 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 10 monclient: dump: 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f46e80072e0 con 0x7f46f802d080 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f802d080 msgr2=0x7f46f802d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f802d080 0x7f46f802d490 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f46e8017000 tx=0x7f46e80057f0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 1 -- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f802d080 msgr2=0x7f46f802d490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46f4ff9640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46f802d080 0x7f46f802d490 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f46e8017000 tx=0x7f46e80057f0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.223 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:44.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:44.223 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:44.223 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:44.223 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:44.223 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 256958, 2024-06-11T19:15:44.223 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:44.223 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:44.223 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:44.224 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:44.225 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:44.227 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.228 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:44.230 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.233 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:44.235 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:44.238 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:44.240 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:44.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:44.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.245 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.245 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.245 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:44.245 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.245 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.245 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:15:44.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.245 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.245 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.245 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:44.245 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.245 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.246 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:44.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.246 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.246 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.246 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:15:44.246 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.246 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.246 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:44.246 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 10 librados: watch_flush enter 2024-06-11T19:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 10 librados: watch_flush exit 2024-06-11T19:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 20 client.64566.objecter shutdown clearing up homeless session... 2024-06-11T19:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 10 client.64566.objecter successfully canceled tick 2024-06-11T19:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 1 -- 172.21.15.98:0/1055260757 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f46e403fce0 msgr2=0x7f46e40421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f46e403fce0 0x7f46e40421a0 secure :-1 s=READY pgs=4124 cs=0 l=1 rev1=1 crypto rx=0x7f46ec000f60 tx=0x7f46ec009040 comp rx=0 tx=0).stop 2024-06-11T19:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 10 monclient: shutdown 2024-06-11T19:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 1 -- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46f813f380 msgr2=0x7f46f809dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fca5d640 1 -- 172.21.15.98:0/1055260757 reap_dead start 2024-06-11T19:15:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46f813f380 0x7f46f809dcf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f46f0033000 tx=0x7f46f0004a40 comp rx=0 tx=0).stop 2024-06-11T19:15:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 1 -- 172.21.15.98:0/1055260757 shutdown_connections 2024-06-11T19:15:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 1 --2- 172.21.15.98:0/1055260757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46f813f380 0x7f46f809dcf0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 1 -- 172.21.15.98:0/1055260757 >> 172.21.15.98:0/1055260757 conn(0x7f46f80ab850 msgr2=0x7f46f80aa340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 1 -- 172.21.15.98:0/1055260757 shutdown_connections 2024-06-11T19:15:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.201+0000 7f46fe4c0640 1 -- 172.21.15.98:0/1055260757 wait complete. 2024-06-11T19:15:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.202+0000 7f46fe4c0640 1 librados: shutdown 2024-06-11T19:15:44.249 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:44.249 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:15:44.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:15:44.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.350+0000 7fda3e0ca640 10 monclient: get_monmap_and_config 2024-06-11T19:15:44.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.350+0000 7fda3e0ca640 10 monclient: build_initial_monmap 2024-06-11T19:15:44.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.350+0000 7fda3e0ca640 10 monclient: monmap: 2024-06-11T19:15:44.349 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:44.351196+0000 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:44.351196+0000 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:44.350 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:44.351 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:44.351 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:44.351 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 1 Processor -- start 2024-06-11T19:15:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 1 -- start start 2024-06-11T19:15:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 10 monclient: init 2024-06-11T19:15:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 10 monclient: _add_conns ranks=[10,6,7,2,0,1,4,9,8,5,3] 2024-06-11T19:15:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fda38136c90 0x7fda38137070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 10 monclient(hunting): picked mon.noname-f con 0x7fda38136c90 addr v2:172.21.15.117:3300/0 2024-06-11T19:15:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fda381367d0 0x7fda380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 10 monclient(hunting): picked mon.noname-d con 0x7fda381367d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fda380a3bd0 0x7fda380a4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 10 monclient(hunting): picked mon.noname-i con 0x7fda380a3bd0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:15:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fda38068180 con 0x7fda38136c90 2024-06-11T19:15:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fda380a0270 con 0x7fda381367d0 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fda3809fc20 con 0x7fda380a3bd0 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda3e0ca640 10 monclient(hunting): authenticate will time out at 1154.158675s 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda2ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fda381367d0 0x7fda380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda2ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fda381367d0 0x7fda380a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34608/0 (socket says 172.21.15.98:34608) 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda2ffff640 1 -- 172.21.15.98:0/90050292 learned_addr learned my addr 172.21.15.98:0/90050292 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda2ffff640 10 monclient(hunting): get_auth_request con 0x7fda381367d0 auth_method 0 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda2ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.351+0000 7fda2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda2ffff640 10 monclient(hunting): handle_auth_done global_id 54861 payload 274 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda377fe640 1 --2- 172.21.15.98:0/90050292 >> v2:172.21.15.117:3300/0 conn(0x7fda38136c90 0x7fda38137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda2ffff640 1 -- 172.21.15.98:0/90050292 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fda380a3bd0 msgr2=0x7fda380a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda37fff640 1 --2- 172.21.15.98:0/90050292 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fda380a3bd0 0x7fda380a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda2ffff640 1 --2- 172.21.15.98:0/90050292 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fda380a3bd0 0x7fda380a4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda2ffff640 1 -- 172.21.15.98:0/90050292 >> v2:172.21.15.117:3300/0 conn(0x7fda38136c90 msgr2=0x7fda38137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda2ffff640 1 --2- 172.21.15.98:0/90050292 >> v2:172.21.15.117:3300/0 conn(0x7fda38136c90 0x7fda38137070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda2ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda2ffff640 1 monclient: found mon.noname-d 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda2ffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:15:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda2ffff640 1 -- 172.21.15.98:0/90050292 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3809ff00 con 0x7fda381367d0 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda377fe640 10 monclient: get_auth_request con 0x7fda38136c90 auth_method 0 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda377fe640 1 --2- 172.21.15.98:0/90050292 >> v2:172.21.15.117:3300/0 conn(0x7fda38136c90 0x7fda38137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda37fff640 10 monclient: get_auth_request con 0x7fda380a3bd0 auth_method 0 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda37fff640 1 --2- 172.21.15.98:0/90050292 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fda380a3bd0 0x7fda380a4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda2ffff640 1 --2- 172.21.15.98:0/90050292 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fda381367d0 0x7fda380a3690 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fda200098a0 tx=0x7fda2002b840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=54bf06033f4ce50a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda36ffd640 1 -- 172.21.15.98:0/90050292 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda20031a10 con 0x7fda381367d0 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda36ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.352+0000 7fda36ffd640 10 monclient: dump: 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.364 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda36ffd640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda36ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda36ffd640 1 -- 172.21.15.98:0/90050292 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda20037070 con 0x7fda381367d0 2024-06-11T19:15:44.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 5 monclient: authenticate success, global_id 54861 2024-06-11T19:15:44.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:44.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda36ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:44.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:44.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda36ffd640 1 -- 172.21.15.98:0/90050292 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda20031c10 con 0x7fda381367d0 2024-06-11T19:15:44.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 4 set_mon_vals no callback set 2024-06-11T19:15:44.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda36ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:44.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda36ffd640 10 monclient: dump: 2024-06-11T19:15:44.366 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.367 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 10 monclient: shutdown 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 1 -- 172.21.15.98:0/90050292 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fda381367d0 msgr2=0x7fda380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 1 --2- 172.21.15.98:0/90050292 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fda381367d0 0x7fda380a3690 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fda200098a0 tx=0x7fda2002b840 comp rx=0 tx=0).stop 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 1 -- 172.21.15.98:0/90050292 shutdown_connections 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 1 --2- 172.21.15.98:0/90050292 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fda380a3bd0 0x7fda380a4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 1 --2- 172.21.15.98:0/90050292 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fda381367d0 0x7fda380a3690 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 1 --2- 172.21.15.98:0/90050292 >> v2:172.21.15.117:3300/0 conn(0x7fda38136c90 0x7fda38137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 1 -- 172.21.15.98:0/90050292 >> 172.21.15.98:0/90050292 conn(0x7fda380a4780 msgr2=0x7fda380a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 1 -- 172.21.15.98:0/90050292 shutdown_connections 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.353+0000 7fda3e0ca640 1 -- 172.21.15.98:0/90050292 wait complete. 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.354+0000 7fda3e0ca640 10 monclient: build_initial_monmap 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.354+0000 7fda3e0ca640 10 monclient: monmap: 2024-06-11T19:15:44.368 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:44.369 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.354+0000 7fda3e0ca640 1 librados: starting msgr at 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.354+0000 7fda3e0ca640 1 librados: starting objecter 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.354+0000 7fda3e0ca640 1 Processor -- start 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.354+0000 7fda3e0ca640 1 -- start start 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.354+0000 7fda3e0ca640 1 librados: setting wanted keys 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.354+0000 7fda3e0ca640 1 librados: calling monclient init 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.354+0000 7fda3e0ca640 10 monclient: init 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 monclient: _add_conns ranks=[3,1,8,5,6,7,2,0,4] 2024-06-11T19:15:44.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fda380a3bd0 0x7fda381cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 monclient(hunting): picked mon.noname-g con 0x7fda380a3bd0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda381367d0 0x7fda381cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 monclient(hunting): picked mon.noname-f con 0x7fda381367d0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:15:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38136c90 0x7fda38120a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 monclient(hunting): picked mon.noname-e con 0x7fda38136c90 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fda381375b0 con 0x7fda381367d0 2024-06-11T19:15:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fda38068560 con 0x7fda380a3bd0 2024-06-11T19:15:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fda381227a0 con 0x7fda38136c90 2024-06-11T19:15:44.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 monclient(hunting): authenticate will time out at 1154.162134s 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38136c90 0x7fda38120a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38136c90 0x7fda38120a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34786/0 (socket says 172.21.15.98:34786) 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 1 -- 172.21.15.98:0/2644550282 learned_addr learned my addr 172.21.15.98:0/2644550282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 10 monclient(hunting): get_auth_request con 0x7fda38136c90 auth_method 0 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda377fe640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fda380a3bd0 0x7fda381cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda2ffff640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda381367d0 0x7fda381cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda2ffff640 10 monclient(hunting): get_auth_request con 0x7fda381367d0 auth_method 0 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda2ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda377fe640 10 monclient(hunting): get_auth_request con 0x7fda380a3bd0 auth_method 0 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 10 monclient(hunting): handle_auth_done global_id 84122 payload 824 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 1 -- 172.21.15.98:0/2644550282 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fda380a3bd0 msgr2=0x7fda381cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fda380a3bd0 0x7fda381cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 1 -- 172.21.15.98:0/2644550282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda381367d0 msgr2=0x7fda381cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda381367d0 0x7fda381cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 1 monclient: found mon.noname-e 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 1 -- 172.21.15.98:0/2644550282 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda38124160 con 0x7fda38136c90 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda37fff640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38136c90 0x7fda38120a40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fda3000cef0 tx=0x7fda3000a710 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 10 client.?.objecter ms_handle_connect 0x7fda38136c90 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda3001a070 con 0x7fda38136c90 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 10 monclient: dump: 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.375 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 5 monclient: authenticate success, global_id 84122 2024-06-11T19:15:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda3000dab0 con 0x7fda38136c90 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 monclient: _renew_subs 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 1 -- 172.21.15.98:0/2644550282 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda38124a00 con 0x7fda38136c90 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda3000dc60 con 0x7fda38136c90 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda367fc640 4 set_mon_vals no callback set 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda3e0ca640 10 client.84122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.355+0000 7fda34ff9640 10 monclient: dump: 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.378 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda3e0ca640 10 monclient: _renew_subs 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda3e0ca640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda3e0ca640 1 -- 172.21.15.98:0/2644550282 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fda38124440 con 0x7fda38136c90 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda3e0ca640 1 librados: init done 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda3e0ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda3e0ca640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38120f80 0x7fda38121360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda3e0ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda3e0ca640 1 -- 172.21.15.98:0/2644550282 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fda38137730 con 0x7fda38120f80 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fda3000d640 con 0x7fda38136c90 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda377fe640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38120f80 0x7fda38121360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda377fe640 10 monclient: get_auth_request con 0x7fda38120f80 auth_method 0 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.356+0000 7fda377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fda1803fce0 0x7fda180421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda377fe640 10 monclient(hunting): handle_auth_done global_id 84131 payload 824 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda377fe640 1 -- 172.21.15.98:0/2644550282 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda3809f2a0 con 0x7fda38120f80 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(285..285 src has 1..285) ==== 7069+0+0 (secure 0 0 0) 0x7fda30052610 con 0x7fda38136c90 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 10 client.84122.objecter ms_dispatch 0x7fda38000b90 osd_map(285..285 src has 1..285) 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 3 client.84122.objecter handle_osd_map got epochs [285,285] > 0 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 3 client.84122.objecter handle_osd_map decoding full epoch 285 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 20 client.84122.objecter dump_active .. 0 homeless 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda377fe640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38120f80 0x7fda38121360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fda2800f760 tx=0x7fda2800fc20 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 10 client.84122.objecter ms_handle_connect 0x7fda38120f80 2024-06-11T19:15:44.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 10 client.84122.objecter resend_mon_ops 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda2ffff640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fda1803fce0 0x7fda180421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda2ffff640 10 monclient: get_auth_request con 0x7fda1803fce0 auth_method 0 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda2800de90 con 0x7fda38120f80 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 10 monclient: dump: 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.381 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fda2800fe70 con 0x7fda38120f80 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38120f80 msgr2=0x7fda38121360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38120f80 0x7fda38121360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fda2800f760 tx=0x7fda2800fc20 comp rx=0 tx=0).stop 2024-06-11T19:15:44.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38120f80 msgr2=0x7fda38121360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:44.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38120f80 0x7fda38121360 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda34ff9640 10 client.84122.objecter ms_handle_connect 0x7fda1803fce0 2024-06-11T19:15:44.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.357+0000 7fda2ffff640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fda1803fce0 0x7fda180421a0 secure :-1 s=READY pgs=4125 cs=0 l=1 rev1=1 crypto rx=0x7fda20036000 tx=0x7fda2003d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.363+0000 7fda3e0ca640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:44.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda3e0ca640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda381218a0 0x7fda38071540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda3e0ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda3e0ca640 1 -- 172.21.15.98:0/2644550282 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fda38050830 con 0x7fda381218a0 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda377fe640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda381218a0 0x7fda38071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda377fe640 10 monclient: get_auth_request con 0x7fda381218a0 auth_method 0 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda377fe640 10 monclient(hunting): handle_auth_done global_id 84140 payload 824 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda377fe640 1 -- 172.21.15.98:0/2644550282 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fda381371e0 con 0x7fda381218a0 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 10 client.84122.objecter ms_handle_connect 0x7fda381218a0 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 10 client.84122.objecter resend_mon_ops 2024-06-11T19:15:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda377fe640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda381218a0 0x7fda38071540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fda2801a000 tx=0x7fda28004730 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda28006040 con 0x7fda381218a0 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 10 monclient: dump: 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fda28020b00 con 0x7fda381218a0 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda381218a0 msgr2=0x7fda38071540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda381218a0 0x7fda38071540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fda2801a000 tx=0x7fda28004730 comp rx=0 tx=0).stop 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 1 -- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda381218a0 msgr2=0x7fda38071540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.364+0000 7fda34ff9640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda381218a0 0x7fda38071540 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fda2801a000 tx=0x7fda28004730 comp rx=0 tx=0).stop 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 16138, 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:44.388 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.389 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.390 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:44.391 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.392 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.393 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.394 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.396 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.403 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:44.404 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.407 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.408 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:44.409 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:44.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 10 librados: watch_flush enter 2024-06-11T19:15:44.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 10 librados: watch_flush exit 2024-06-11T19:15:44.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 20 client.84122.objecter shutdown clearing up homeless session... 2024-06-11T19:15:44.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 10 client.84122.objecter successfully canceled tick 2024-06-11T19:15:44.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 1 -- 172.21.15.98:0/2644550282 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fda1803fce0 msgr2=0x7fda180421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fda1803fce0 0x7fda180421a0 secure :-1 s=READY pgs=4125 cs=0 l=1 rev1=1 crypto rx=0x7fda20036000 tx=0x7fda2003d040 comp rx=0 tx=0).stop 2024-06-11T19:15:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 10 monclient: shutdown 2024-06-11T19:15:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 1 -- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38136c90 msgr2=0x7fda38120a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 1 --2- 172.21.15.98:0/2644550282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda38136c90 0x7fda38120a40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fda3000cef0 tx=0x7fda3000a710 comp rx=0 tx=0).stop 2024-06-11T19:15:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda37fff640 1 -- 172.21.15.98:0/2644550282 reap_dead start 2024-06-11T19:15:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda37fff640 1 -- 172.21.15.98:0/2644550282 reap_dead start 2024-06-11T19:15:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 1 -- 172.21.15.98:0/2644550282 shutdown_connections 2024-06-11T19:15:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 1 -- 172.21.15.98:0/2644550282 >> 172.21.15.98:0/2644550282 conn(0x7fda380a4780 msgr2=0x7fda381bc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 1 -- 172.21.15.98:0/2644550282 shutdown_connections 2024-06-11T19:15:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 1 -- 172.21.15.98:0/2644550282 wait complete. 2024-06-11T19:15:44.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.365+0000 7fda3e0ca640 1 librados: shutdown 2024-06-11T19:15:44.413 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:44.413 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:15:44.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:15:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.513+0000 7fae1dc2e640 10 monclient: get_monmap_and_config 2024-06-11T19:15:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.513+0000 7fae1dc2e640 10 monclient: build_initial_monmap 2024-06-11T19:15:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.513+0000 7fae1dc2e640 10 monclient: monmap: 2024-06-11T19:15:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:44.514117+0000 2024-06-11T19:15:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:44.514117+0000 2024-06-11T19:15:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:44.513 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.513+0000 7fae1dc2e640 1 Processor -- start 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.513+0000 7fae1dc2e640 1 -- start start 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.513+0000 7fae1dc2e640 10 monclient: init 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae1dc2e640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae1dc2e640 10 monclient: _add_conns ranks=[0,1,5,6,9,2,7,8,3,10,4] 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae1dc2e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae180a0cd0 addr v1:172.21.15.98:6789/0 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae1dc2e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fae18138c80 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae1dc2e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fae18139060 0x7fae18141380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae1dc2e640 10 monclient(hunting): picked mon.noname-h con 0x7fae18139060 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae1dc2e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae180a17c0 con 0x7fae180a0cd0 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae1dc2e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae1809efd0 con 0x7fae18138c80 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae1dc2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae1dc2e640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fae180739b0 con 0x7fae18139060 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae1dc2e640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae1dc2e640 10 monclient(hunting): authenticate will time out at 1154.321378s 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae177fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fae180a0cd0 0x7fae180a10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35714/0 (socket says 172.21.15.98:35714) 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae177fe640 1 -- 172.21.15.98:0/2618380230 learned_addr learned my addr 172.21.15.98:0/2618380230 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 641437757 0 0) 0x7fae180a17c0 con 0x7fae180a0cd0 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient(hunting): _negotiate 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient(hunting): my global_id is 64575 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae04003520 con 0x7fae180a0cd0 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae17fff640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fae18139060 0x7fae18141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae17fff640 10 monclient(hunting): get_auth_request con 0x7fae18139060 auth_method 0 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae17fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae17fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3243424155 0 0) 0x7fae04003520 con 0x7fae180a0cd0 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fae18139060 msgr2=0x7fae18141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fae18139060 0x7fae18141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 >> v1:172.21.15.117:6789/0 conn(0x7fae18138c80 legacy=0x7fae1813dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae1809f2b0 con 0x7fae180a0cd0 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7fae08001960 con 0x7fae180a0cd0 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient: dump: 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 >> v1:172.21.15.98:6789/0 conn(0x7fae180a0cd0 legacy=0x7fae180a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 10 monclient: _add_conns ranks=[2,3,8,6,0,7,5,4,1] 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.514+0000 7fae167fc640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae04004230 0x7fae04004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 10 monclient(hunting): picked mon.b con 0x7fae04004230 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae04004d90 0x7fae04005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 10 monclient(hunting): picked mon.g con 0x7fae04004d90 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fae04009c50 0x7fae0400c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 10 monclient(hunting): picked mon.e con 0x7fae04009c50 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fae180739b0 con 0x7fae04004230 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fae0400c9c0 con 0x7fae04004d90 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fae0400cb70 con 0x7fae04009c50 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae167fc640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae04004230 0x7fae04004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 10 monclient(hunting): get_auth_request con 0x7fae04004230 auth_method 0 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae17fff640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fae04009c50 0x7fae0400c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae17fff640 10 monclient(hunting): get_auth_request con 0x7fae04009c50 auth_method 0 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae17fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae17fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae16ffd640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae04004d90 0x7fae04005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae16ffd640 10 monclient(hunting): get_auth_request con 0x7fae04004d90 auth_method 0 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae16ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae16ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:44.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae17fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 10 monclient(hunting): handle_auth_done global_id 64575 payload 293 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 1 -- 172.21.15.98:0/2618380230 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fae04009c50 msgr2=0x7fae0400c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fae04009c50 0x7fae0400c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 1 -- 172.21.15.98:0/2618380230 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae04004d90 msgr2=0x7fae04005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae04004d90 0x7fae04005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 1 monclient: found mon.b 2024-06-11T19:15:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 1 -- 172.21.15.98:0/2618380230 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae04003520 con 0x7fae04004230 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae17fff640 1 -- 172.21.15.98:0/2618380230 reap_dead start 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae1dc2e640 5 monclient: authenticate success, global_id 64575 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.515+0000 7fae1dc2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae177fe640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae04004230 0x7fae04004800 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fae08007ed0 tx=0x7fae04005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b657d77428a7ad68 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fae08007af0 con 0x7fae04004230 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae167fc640 10 monclient: dump: 2024-06-11T19:15:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae1dc2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae08030070 con 0x7fae04004230 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae167fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae167fc640 1 -- 172.21.15.98:0/2618380230 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fae08005d40 con 0x7fae04004230 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae1dc2e640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae1dc2e640 4 set_mon_vals no callback set 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae167fc640 10 monclient: dump: 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.539 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae1dc2e640 10 monclient: shutdown 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae1dc2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae1dc2e640 1 -- 172.21.15.98:0/2618380230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae04004230 msgr2=0x7fae04004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae1dc2e640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae04004230 0x7fae04004800 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fae08007ed0 tx=0x7fae04005830 comp rx=0 tx=0).stop 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae1dc2e640 1 -- 172.21.15.98:0/2618380230 shutdown_connections 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae1dc2e640 1 --2- 172.21.15.98:0/2618380230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fae04004230 0x7fae04004800 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae1dc2e640 1 -- 172.21.15.98:0/2618380230 >> 172.21.15.98:0/2618380230 conn(0x7fae180a7850 msgr2=0x7fae180a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.516+0000 7fae1dc2e640 1 -- 172.21.15.98:0/2618380230 shutdown_connections 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.517+0000 7fae1dc2e640 1 -- 172.21.15.98:0/2618380230 wait complete. 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.517+0000 7fae1dc2e640 10 monclient: build_initial_monmap 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.517+0000 7fae1dc2e640 10 monclient: monmap: 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:44.541 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:44.542 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.517+0000 7fae1dc2e640 1 librados: starting msgr at 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.517+0000 7fae1dc2e640 1 librados: starting objecter 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.517+0000 7fae1dc2e640 1 Processor -- start 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 1 -- start start 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 1 librados: setting wanted keys 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 1 librados: calling monclient init 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 10 monclient: init 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 10 monclient: _add_conns ranks=[1,4,3,2,8,7,0,5,6] 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04004230 0x7fae181cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 10 monclient(hunting): picked mon.noname-f con 0x7fae04004230 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fae181ce090 0x7fae1812df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 10 monclient(hunting): picked mon.noname-c con 0x7fae181ce090 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae181ce4b0 0x7fae1812e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 10 monclient(hunting): picked mon.noname-g con 0x7fae181ce4b0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fae180a10b0 con 0x7fae04004230 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fae1809ecf0 con 0x7fae181ce4b0 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fae180a1230 con 0x7fae181ce090 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae1dc2e640 10 monclient(hunting): authenticate will time out at 1154.325515s 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fae181ce090 0x7fae1812df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fae181ce090 0x7fae1812df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:53660/0 (socket says 172.21.15.98:53660) 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 1 -- 172.21.15.98:0/4260984170 learned_addr learned my addr 172.21.15.98:0/4260984170 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 10 monclient(hunting): get_auth_request con 0x7fae181ce090 auth_method 0 2024-06-11T19:15:44.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae17fff640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae181ce4b0 0x7fae1812e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 10 monclient(hunting): handle_auth_done global_id 23710 payload 824 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae177fe640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04004230 0x7fae181cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 1 -- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae181ce4b0 msgr2=0x7fae1812e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae181ce4b0 0x7fae1812e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 1 -- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04004230 msgr2=0x7fae181cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04004230 0x7fae181cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 1 monclient: found mon.noname-c 2024-06-11T19:15:44.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:15:44.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae16ffd640 1 -- 172.21.15.98:0/4260984170 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae18131590 con 0x7fae181ce090 2024-06-11T19:15:44.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae17fff640 10 monclient: get_auth_request con 0x7fae181ce4b0 auth_method 0 2024-06-11T19:15:44.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae177fe640 10 monclient: get_auth_request con 0x7fae04004230 auth_method 0 2024-06-11T19:15:44.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae17fff640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae181ce4b0 0x7fae1812e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:15:44.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.518+0000 7fae177fe640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04004230 0x7fae181cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:15:44.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae16ffd640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fae181ce090 0x7fae1812df00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fae0000f5f0 tx=0x7fae0000fab0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 10 client.?.objecter ms_handle_connect 0x7fae181ce090 2024-06-11T19:15:44.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:44.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fae000052e0 con 0x7fae181ce090 2024-06-11T19:15:44.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:15:44.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 10 monclient: dump: 2024-06-11T19:15:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.602 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae1dc2e640 5 monclient: authenticate success, global_id 23710 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae00018070 con 0x7fae181ce090 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fae000054a0 con 0x7fae181ce090 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae1dc2e640 10 monclient: _renew_subs 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae1dc2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:44.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae1dc2e640 1 -- 172.21.15.98:0/4260984170 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae18131e30 con 0x7fae181ce090 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae15ffb640 4 set_mon_vals no callback set 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae0ffff640 10 monclient: dump: 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae1dc2e640 10 client.23710.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae1dc2e640 10 monclient: _renew_subs 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae1dc2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae1dc2e640 1 -- 172.21.15.98:0/4260984170 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fae18136ac0 con 0x7fae181ce090 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.519+0000 7fae1dc2e640 1 librados: init done 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae1dc2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae1dc2e640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae181317d0 0x7fae1812e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae1dc2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae1dc2e640 1 -- 172.21.15.98:0/4260984170 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fae18050830 con 0x7fae181317d0 2024-06-11T19:15:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fae00003640 con 0x7fae181ce090 2024-06-11T19:15:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae177fe640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae181317d0 0x7fae1812e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae177fe640 10 monclient: get_auth_request con 0x7fae181317d0 auth_method 0 2024-06-11T19:15:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae177fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae177fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae0ffff640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fadf803fdb0 0x7fadf8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(285..285 src has 1..285) ==== 7069+0+0 (secure 0 0 0) 0x7fae00010040 con 0x7fae181ce090 2024-06-11T19:15:44.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae0ffff640 10 client.23710.objecter ms_dispatch 0x7fae18000b90 osd_map(285..285 src has 1..285) 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae0ffff640 3 client.23710.objecter handle_osd_map got epochs [285,285] > 0 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae0ffff640 3 client.23710.objecter handle_osd_map decoding full epoch 285 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae0ffff640 20 client.23710.objecter dump_active .. 0 homeless 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae177fe640 10 monclient(hunting): handle_auth_done global_id 31861 payload 824 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae17fff640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fadf803fdb0 0x7fadf8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae177fe640 1 -- 172.21.15.98:0/4260984170 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae180a1560 con 0x7fae181317d0 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae17fff640 10 monclient: get_auth_request con 0x7fadf803fdb0 auth_method 0 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae0ffff640 10 client.23710.objecter ms_handle_connect 0x7fae181317d0 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae177fe640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae181317d0 0x7fae1812e980 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7fae08007ea0 tx=0x7fae1809f480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.520+0000 7fae0ffff640 10 client.23710.objecter resend_mon_ops 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fae08001920 con 0x7fae181317d0 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae0ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:15:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae0ffff640 10 monclient: dump: 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae0ffff640 10 client.23710.objecter ms_handle_connect 0x7fadf803fdb0 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae17fff640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fadf803fdb0 0x7fadf8042270 secure :-1 s=READY pgs=4126 cs=0 l=1 rev1=1 crypto rx=0x7fae040039b0 tx=0x7fae1000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fae08030040 con 0x7fae181317d0 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae0ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae0ffff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae181317d0 msgr2=0x7fae1812e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae0ffff640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae181317d0 0x7fae1812e980 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7fae08007ea0 tx=0x7fae1809f480 comp rx=0 tx=0).stop 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae181317d0 msgr2=0x7fae1812e980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.521+0000 7fae0ffff640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae181317d0 0x7fae1812e980 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.527+0000 7fae1dc2e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.527+0000 7fae1dc2e640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1812eec0 0x7fae1809c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.527+0000 7fae1dc2e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.527+0000 7fae1dc2e640 1 -- 172.21.15.98:0/4260984170 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fae1812f2d0 con 0x7fae1812eec0 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae177fe640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1812eec0 0x7fae1809c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae177fe640 10 monclient: get_auth_request con 0x7fae1812eec0 auth_method 0 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae177fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae177fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae177fe640 10 monclient(hunting): handle_auth_done global_id 31870 payload 824 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae177fe640 1 -- 172.21.15.98:0/4260984170 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fae181306a0 con 0x7fae1812eec0 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae177fe640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1812eec0 0x7fae1809c340 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7fae08001620 tx=0x7fae08009030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae0ffff640 10 client.23710.objecter ms_handle_connect 0x7fae1812eec0 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae0ffff640 10 client.23710.objecter resend_mon_ops 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fae08046040 con 0x7fae1812eec0 2024-06-11T19:15:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae0ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.528+0000 7fae0ffff640 10 monclient: dump: 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.612 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.612 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9615 (secure 0 0 0) 0x7fae08035470 con 0x7fae1812eec0 2024-06-11T19:15:44.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae0ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:44.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae0ffff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:44.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1812eec0 msgr2=0x7fae1809c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae0ffff640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1812eec0 0x7fae1809c340 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7fae08001620 tx=0x7fae08009030 comp rx=0 tx=0).stop 2024-06-11T19:15:44.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae0ffff640 1 -- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1812eec0 msgr2=0x7fae1809c340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:44.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae0ffff640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1812eec0 0x7fae1809c340 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.612 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 394747, 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:44.613 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.614 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.615 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:44.616 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:44.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:44.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:44.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.629 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:44.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.632 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.633 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.634 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:44.635 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:44.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 10 librados: watch_flush enter 2024-06-11T19:15:44.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 10 librados: watch_flush exit 2024-06-11T19:15:44.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 20 client.23710.objecter shutdown clearing up homeless session... 2024-06-11T19:15:44.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 10 client.23710.objecter successfully canceled tick 2024-06-11T19:15:44.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 1 -- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fadf803fdb0 msgr2=0x7fadf8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fadf803fdb0 0x7fadf8042270 secure :-1 s=READY pgs=4126 cs=0 l=1 rev1=1 crypto rx=0x7fae040039b0 tx=0x7fae1000b040 comp rx=0 tx=0).stop 2024-06-11T19:15:44.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 10 monclient: shutdown 2024-06-11T19:15:44.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:44.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 1 -- 172.21.15.98:0/4260984170 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fae181ce090 msgr2=0x7fae1812df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 1 --2- 172.21.15.98:0/4260984170 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fae181ce090 0x7fae1812df00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fae0000f5f0 tx=0x7fae0000fab0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae17fff640 1 -- 172.21.15.98:0/4260984170 reap_dead start 2024-06-11T19:15:44.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae17fff640 1 -- 172.21.15.98:0/4260984170 reap_dead start 2024-06-11T19:15:44.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 1 -- 172.21.15.98:0/4260984170 shutdown_connections 2024-06-11T19:15:44.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 1 -- 172.21.15.98:0/4260984170 >> 172.21.15.98:0/4260984170 conn(0x7fae180a7850 msgr2=0x7fae181be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:44.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 1 -- 172.21.15.98:0/4260984170 shutdown_connections 2024-06-11T19:15:44.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 1 -- 172.21.15.98:0/4260984170 wait complete. 2024-06-11T19:15:44.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.529+0000 7fae1dc2e640 1 librados: shutdown 2024-06-11T19:15:44.639 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:44.639 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:15:44.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:15:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.739+0000 7fcbf6aff640 10 monclient: get_monmap_and_config 2024-06-11T19:15:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.739+0000 7fcbf6aff640 10 monclient: build_initial_monmap 2024-06-11T19:15:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.739+0000 7fcbf6aff640 10 monclient: monmap: 2024-06-11T19:15:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:44.740609+0000 2024-06-11T19:15:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:44.740609+0000 2024-06-11T19:15:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:44.740 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 1 Processor -- start 2024-06-11T19:15:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 1 -- start start 2024-06-11T19:15:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 10 monclient: init 2024-06-11T19:15:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 10 monclient: _add_conns ranks=[1,4,2,5,3,9,7,8,6,0,10] 2024-06-11T19:15:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fcbf00a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbf00a2530 0x7fcbf013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 10 monclient(hunting): picked mon.noname-c con 0x7fcbf00a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:15:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbf013e1a0 0x7fcbf0140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 10 monclient(hunting): picked mon.noname-b con 0x7fcbf013e1a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf00a7140 con 0x7fcbf00a4920 2024-06-11T19:15:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcbf00a2b50 con 0x7fcbf013e1a0 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fcbf00a2e30 con 0x7fcbf00a2530 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.740+0000 7fcbf6aff640 10 monclient(hunting): authenticate will time out at 1154.547975s 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbf00a2530 0x7fcbf013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbf509c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbf013e1a0 0x7fcbf0140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbf00a2530 0x7fcbf013dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:53664/0 (socket says 172.21.15.98:53664) 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 1 -- 172.21.15.98:0/2075566699 learned_addr learned my addr 172.21.15.98:0/2075566699 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 10 monclient(hunting): get_auth_request con 0x7fcbf00a2530 auth_method 0 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbf489b640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fcbf00a4920 0x7fcbf00a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:49726/0 (socket says 172.21.15.98:49726) 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbf509c640 10 monclient(hunting): get_auth_request con 0x7fcbf013e1a0 auth_method 0 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbf509c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbf509c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbf509c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbf509c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbf509c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbf509c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 10 monclient(hunting): handle_auth_done global_id 23719 payload 274 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 1 -- 172.21.15.98:0/2075566699 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbf013e1a0 msgr2=0x7fcbf0140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 1 --2- 172.21.15.98:0/2075566699 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbf013e1a0 0x7fcbf0140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 1 -- 172.21.15.98:0/2075566699 >> v1:172.21.15.117:6789/0 conn(0x7fcbf00a4920 legacy=0x7fcbf00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe77fe640 1 -- 172.21.15.98:0/2075566699 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2705399850 0 0) 0x7fcbf00a7140 con 0x7fcbf00a4920 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 1 monclient: found mon.noname-c 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe7fff640 1 -- 172.21.15.98:0/2075566699 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf00a3110 con 0x7fcbf00a2530 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbf509c640 1 --2- 172.21.15.98:0/2075566699 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbf013e1a0 0x7fcbf0140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.741+0000 7fcbe77fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe7fff640 1 --2- 172.21.15.98:0/2075566699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbf00a2530 0x7fcbf013dc60 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00098a0 tx=0x7fcbe002b840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=70fe9e1ff485d166 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 1 -- 172.21.15.98:0/2075566699 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbe0031a10 con 0x7fcbf00a2530 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 10 monclient: dump: 2024-06-11T19:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 5 monclient: authenticate success, global_id 23719 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 1 -- 172.21.15.98:0/2075566699 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbe0037070 con 0x7fcbf00a2530 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 4 set_mon_vals no callback set 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 1 -- 172.21.15.98:0/2075566699 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbe0031bd0 con 0x7fcbf00a2530 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbe77fe640 10 monclient: dump: 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 10 monclient: shutdown 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 1 -- 172.21.15.98:0/2075566699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbf00a2530 msgr2=0x7fcbf013dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 1 --2- 172.21.15.98:0/2075566699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbf00a2530 0x7fcbf013dc60 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00098a0 tx=0x7fcbe002b840 comp rx=0 tx=0).stop 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 1 -- 172.21.15.98:0/2075566699 shutdown_connections 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 1 --2- 172.21.15.98:0/2075566699 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbf013e1a0 0x7fcbf0140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 1 --2- 172.21.15.98:0/2075566699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbf00a2530 0x7fcbf013dc60 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 1 -- 172.21.15.98:0/2075566699 >> 172.21.15.98:0/2075566699 conn(0x7fcbf00ab850 msgr2=0x7fcbf00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 1 -- 172.21.15.98:0/2075566699 shutdown_connections 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.742+0000 7fcbf6aff640 1 -- 172.21.15.98:0/2075566699 wait complete. 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.743+0000 7fcbf6aff640 10 monclient: build_initial_monmap 2024-06-11T19:15:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.743+0000 7fcbf6aff640 10 monclient: monmap: 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.743+0000 7fcbf6aff640 1 librados: starting msgr at 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.743+0000 7fcbf6aff640 1 librados: starting objecter 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.743+0000 7fcbf6aff640 1 Processor -- start 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.743+0000 7fcbf6aff640 1 -- start start 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.743+0000 7fcbf6aff640 1 librados: setting wanted keys 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.743+0000 7fcbf6aff640 1 librados: calling monclient init 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.743+0000 7fcbf6aff640 10 monclient: init 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 10 monclient: _add_conns ranks=[1,8,5,2,0,4,3,7,6] 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbf00a2530 0x7fcbf01215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 10 monclient(hunting): picked mon.noname-f con 0x7fcbf00a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbf00a4920 0x7fcbf0121b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 10 monclient(hunting): picked mon.noname-e con 0x7fcbf00a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcbf013e1a0 0x7fcbf0122070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 10 monclient(hunting): picked mon.noname-h con 0x7fcbf013e1a0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf0138e50 con 0x7fcbf00a2530 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fcbf00a84b0 con 0x7fcbf013e1a0 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcbf00a6da0 con 0x7fcbf00a4920 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf6aff640 10 monclient(hunting): authenticate will time out at 1154.551368s 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbf00a4920 0x7fcbf0121b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbf00a4920 0x7fcbf0121b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34832/0 (socket says 172.21.15.98:34832) 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 1 -- 172.21.15.98:0/146803243 learned_addr learned my addr 172.21.15.98:0/146803243 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 10 monclient(hunting): get_auth_request con 0x7fcbf00a4920 auth_method 0 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf489b640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbf00a2530 0x7fcbf01215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf509c640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcbf013e1a0 0x7fcbf0122070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf509c640 10 monclient(hunting): get_auth_request con 0x7fcbf013e1a0 auth_method 0 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf509c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf509c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf509c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf489b640 10 monclient(hunting): get_auth_request con 0x7fcbf00a2530 auth_method 0 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf489b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf489b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbf489b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 10 monclient(hunting): handle_auth_done global_id 84149 payload 824 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 1 -- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcbf013e1a0 msgr2=0x7fcbf0122070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcbf013e1a0 0x7fcbf0122070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 1 -- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbf00a2530 msgr2=0x7fcbf01215f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbf00a2530 0x7fcbf01215f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 1 monclient: found mon.noname-e 2024-06-11T19:15:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 1 -- 172.21.15.98:0/146803243 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf0137b00 con 0x7fcbf00a4920 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe7fff640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbf00a4920 0x7fcbf0121b30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00090b0 tx=0x7fcbe00046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe57fa640 10 client.?.objecter ms_handle_connect 0x7fcbf00a4920 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.744+0000 7fcbe57fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbe0041070 con 0x7fcbf00a4920 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 10 monclient: dump: 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbf6aff640 5 monclient: authenticate success, global_id 84149 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbe00315a0 con 0x7fcbf00a4920 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbf6aff640 10 monclient: _renew_subs 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbe0031bc0 con 0x7fcbf00a4920 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbf6aff640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbf6aff640 1 -- 172.21.15.98:0/146803243 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf00a4260 con 0x7fcbf00a4920 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe6ffd640 4 set_mon_vals no callback set 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbe57fa640 10 monclient: dump: 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.765 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbf6aff640 10 client.84149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbf6aff640 10 monclient: _renew_subs 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbf6aff640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbf6aff640 1 -- 172.21.15.98:0/146803243 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbf00aa6d0 con 0x7fcbf00a4920 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.745+0000 7fcbf6aff640 1 librados: init done 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fcbe004c060 con 0x7fcbf00a4920 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf6aff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf6aff640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf0124510 0x7fcbf009c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf6aff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf6aff640 1 -- 172.21.15.98:0/146803243 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fcbf013de40 con 0x7fcbf0124510 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf489b640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf0124510 0x7fcbf009c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbe57fa640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcbdc03fdb0 0x7fcbdc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf489b640 10 monclient: get_auth_request con 0x7fcbf0124510 auth_method 0 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf489b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf489b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf489b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(286..286 src has 1..286) ==== 7069+0+0 (secure 0 0 0) 0x7fcbe002f050 con 0x7fcbf00a4920 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbe57fa640 10 client.84149.objecter ms_dispatch 0x7fcbf0000b90 osd_map(286..286 src has 1..286) 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbe57fa640 3 client.84149.objecter handle_osd_map got epochs [286,286] > 0 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbe57fa640 3 client.84149.objecter handle_osd_map decoding full epoch 286 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf489b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf489b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf489b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbe57fa640 20 client.84149.objecter dump_active .. 0 homeless 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf509c640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcbdc03fdb0 0x7fcbdc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf509c640 10 monclient: get_auth_request con 0x7fcbdc03fdb0 auth_method 0 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf489b640 10 monclient(hunting): handle_auth_done global_id 84126 payload 824 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf489b640 1 -- 172.21.15.98:0/146803243 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcbf00a70c0 con 0x7fcbf0124510 2024-06-11T19:15:44.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbe57fa640 10 client.84149.objecter ms_handle_connect 0x7fcbf0124510 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbe57fa640 10 client.84149.objecter resend_mon_ops 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf489b640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf0124510 0x7fcbf009c9e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcbec000c00 tx=0x7fcbec0019b0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbe57fa640 10 client.84149.objecter ms_handle_connect 0x7fcbdc03fdb0 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.746+0000 7fcbf509c640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcbdc03fdb0 0x7fcbdc042270 secure :-1 s=READY pgs=4127 cs=0 l=1 rev1=1 crypto rx=0x7fcbf01252a0 tx=0x7fcbe800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.747+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbec010980 con 0x7fcbf0124510 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.747+0000 7fcbe57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.747+0000 7fcbe57fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.747+0000 7fcbe57fa640 10 monclient: dump: 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.769 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.747+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcbec010510 con 0x7fcbf0124510 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.747+0000 7fcbe57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:44.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.747+0000 7fcbe57fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:44.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.747+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf0124510 msgr2=0x7fcbf009c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.747+0000 7fcbe57fa640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf0124510 0x7fcbf009c9e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcbec000c00 tx=0x7fcbec0019b0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.747+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf0124510 msgr2=0x7fcbf009c9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:44.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.747+0000 7fcbe57fa640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf0124510 0x7fcbf009c9e0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcbec000c00 tx=0x7fcbec0019b0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.753+0000 7fcbf6aff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.753+0000 7fcbf6aff640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf01248f0 0x7fcbf009cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.753+0000 7fcbf6aff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.753+0000 7fcbf6aff640 1 -- 172.21.15.98:0/146803243 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fcbf0050830 con 0x7fcbf01248f0 2024-06-11T19:15:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbf489b640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf01248f0 0x7fcbf009cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbf489b640 10 monclient: get_auth_request con 0x7fcbf01248f0 auth_method 0 2024-06-11T19:15:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbf489b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbf489b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbf489b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbf489b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbf489b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbf489b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbf489b640 10 monclient(hunting): handle_auth_done global_id 84135 payload 824 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbf489b640 1 -- 172.21.15.98:0/146803243 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcbf00a4d00 con 0x7fcbf01248f0 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbf489b640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf01248f0 0x7fcbf009cf20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcbec015000 tx=0x7fcbec000ed0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbe57fa640 10 client.84149.objecter ms_handle_connect 0x7fcbf01248f0 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.754+0000 7fcbe57fa640 10 client.84149.objecter resend_mon_ops 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbec01cd70 con 0x7fcbf01248f0 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbe57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbe57fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbe57fa640 10 monclient: dump: 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.775 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fcbec014430 con 0x7fcbf01248f0 2024-06-11T19:15:44.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbe57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:44.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbe57fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:44.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf01248f0 msgr2=0x7fcbf009cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbe57fa640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf01248f0 0x7fcbf009cf20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcbec015000 tx=0x7fcbec000ed0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbe57fa640 1 -- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf01248f0 msgr2=0x7fcbf009cf20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:44.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbe57fa640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbf01248f0 0x7fcbf009cf20 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcbec015000 tx=0x7fcbec000ed0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.776 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:44.776 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:44.776 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:44.776 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 16529, 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.777 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.778 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:44.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.781 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:44.783 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:44.786 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:44.788 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.791 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:44.793 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.796 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.797 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.798 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:44.799 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:44.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 10 librados: watch_flush enter 2024-06-11T19:15:44.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 10 librados: watch_flush exit 2024-06-11T19:15:44.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 20 client.84149.objecter shutdown clearing up homeless session... 2024-06-11T19:15:44.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 10 client.84149.objecter successfully canceled tick 2024-06-11T19:15:44.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 1 -- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcbdc03fdb0 msgr2=0x7fcbdc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcbdc03fdb0 0x7fcbdc042270 secure :-1 s=READY pgs=4127 cs=0 l=1 rev1=1 crypto rx=0x7fcbf01252a0 tx=0x7fcbe800b040 comp rx=0 tx=0).stop 2024-06-11T19:15:44.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 10 monclient: shutdown 2024-06-11T19:15:44.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:44.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 1 -- 172.21.15.98:0/146803243 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbf00a4920 msgr2=0x7fcbf0121b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 1 --2- 172.21.15.98:0/146803243 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbf00a4920 0x7fcbf0121b30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00090b0 tx=0x7fcbe00046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf509c640 1 -- 172.21.15.98:0/146803243 reap_dead start 2024-06-11T19:15:44.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf509c640 1 -- 172.21.15.98:0/146803243 reap_dead start 2024-06-11T19:15:44.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 1 -- 172.21.15.98:0/146803243 shutdown_connections 2024-06-11T19:15:44.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 1 -- 172.21.15.98:0/146803243 >> 172.21.15.98:0/146803243 conn(0x7fcbf00ab850 msgr2=0x7fcbf00a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:44.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 1 -- 172.21.15.98:0/146803243 shutdown_connections 2024-06-11T19:15:44.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 1 -- 172.21.15.98:0/146803243 wait complete. 2024-06-11T19:15:44.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.755+0000 7fcbf6aff640 1 librados: shutdown 2024-06-11T19:15:44.801 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:44.802 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:15:44.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:15:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.905+0000 7f380598c640 10 monclient: get_monmap_and_config 2024-06-11T19:15:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.905+0000 7f380598c640 10 monclient: build_initial_monmap 2024-06-11T19:15:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.905+0000 7f380598c640 10 monclient: monmap: 2024-06-11T19:15:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:44.906767+0000 2024-06-11T19:15:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:44.906767+0000 2024-06-11T19:15:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:44.905 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:44.906 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:44.906 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:44.906 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:44.906 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:44.906 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:44.906 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:44.906 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:44.906 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:44.906 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:44.906 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 1 Processor -- start 2024-06-11T19:15:44.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 1 -- start start 2024-06-11T19:15:44.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 10 monclient: init 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 10 monclient: _add_conns ranks=[6,9,1,3,7,8,5,2,0,4,10] 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a4920 0x7f38000a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 10 monclient(hunting): picked mon.noname-d con 0x7f38000a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f38000a5240 0x7f38000a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 10 monclient(hunting): picked mon.noname-a con 0x7f38000a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f38000a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38000a7140 con 0x7f38000a2530 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f38000a2b50 con 0x7f38000a5240 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f38000a2e30 con 0x7f38000a4920 2024-06-11T19:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.906+0000 7f380598c640 10 monclient(hunting): authenticate will time out at 1154.713953s 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37fe7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f38000a5240 0x7f38000a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a4920 0x7f38000a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a4920 0x7f38000a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34616/0 (socket says 172.21.15.98:34616) 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37fe7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f38000a5240 0x7f38000a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46304/0 (socket says 172.21.15.98:46304) 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 1 -- 172.21.15.98:0/795310154 learned_addr learned my addr 172.21.15.98:0/795310154 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 10 monclient(hunting): get_auth_request con 0x7f38000a4920 auth_method 0 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37fe7fc640 10 monclient(hunting): get_auth_request con 0x7f38000a5240 auth_method 0 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37fe7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 10 monclient(hunting): handle_auth_done global_id 54870 payload 274 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 1 -- 172.21.15.98:0/795310154 >> v2:172.21.15.98:3300/0 conn(0x7f38000a5240 msgr2=0x7f38000a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 1 --2- 172.21.15.98:0/795310154 >> v2:172.21.15.98:3300/0 conn(0x7f38000a5240 0x7f38000a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 1 -- 172.21.15.98:0/795310154 >> v1:172.21.15.117:6789/0 conn(0x7f38000a2530 legacy=0x7f38000a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 1 monclient: found mon.noname-d 2024-06-11T19:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37feffd640 1 -- 172.21.15.98:0/795310154 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38000a3110 con 0x7f38000a4920 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.907+0000 7f37fe7fc640 1 --2- 172.21.15.98:0/795310154 >> v2:172.21.15.98:3300/0 conn(0x7f38000a5240 0x7f38000a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37feffd640 1 --2- 172.21.15.98:0/795310154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a4920 0x7f38000a4d00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009e50 tx=0x7f37f802bdb0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=fdeb20f237d77431 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 1 -- 172.21.15.98:0/795310154 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f37f802cbc0 con 0x7f38000a4920 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 10 monclient: dump: 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 1 -- 172.21.15.98:0/795310154 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37f802cd80 con 0x7f38000a4920 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 5 monclient: authenticate success, global_id 54870 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 4 set_mon_vals no callback set 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 1 -- 172.21.15.98:0/795310154 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f37f8034860 con 0x7f38000a4920 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f37fdffb640 10 monclient: dump: 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 10 monclient: shutdown 2024-06-11T19:15:44.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 1 -- 172.21.15.98:0/795310154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a4920 msgr2=0x7f38000a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 1 --2- 172.21.15.98:0/795310154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a4920 0x7f38000a4d00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009e50 tx=0x7f37f802bdb0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 1 -- 172.21.15.98:0/795310154 shutdown_connections 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 1 --2- 172.21.15.98:0/795310154 >> v2:172.21.15.98:3300/0 conn(0x7f38000a5240 0x7f38000a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 1 --2- 172.21.15.98:0/795310154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a4920 0x7f38000a4d00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 1 -- 172.21.15.98:0/795310154 >> 172.21.15.98:0/795310154 conn(0x7f38000ab850 msgr2=0x7f38000a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 1 -- 172.21.15.98:0/795310154 shutdown_connections 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.908+0000 7f380598c640 1 -- 172.21.15.98:0/795310154 wait complete. 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.909+0000 7f380598c640 10 monclient: build_initial_monmap 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.909+0000 7f380598c640 10 monclient: monmap: 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.909+0000 7f380598c640 1 librados: starting msgr at 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.909+0000 7f380598c640 1 librados: starting objecter 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.909+0000 7f380598c640 1 Processor -- start 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.909+0000 7f380598c640 1 -- start start 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.909+0000 7f380598c640 1 librados: setting wanted keys 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.909+0000 7f380598c640 1 librados: calling monclient init 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.909+0000 7f380598c640 10 monclient: init 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 10 monclient: _add_conns ranks=[6,5,3,2,0,7,1,4,8] 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a2530 0x7f38001cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 10 monclient(hunting): picked mon.noname-d con 0x7f38000a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f38000a4920 0x7f38001ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 10 monclient(hunting): picked mon.noname-h con 0x7f38000a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38000a5240 0x7f3800122fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 10 monclient(hunting): picked mon.noname-g con 0x7f38000a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f38000a6da0 con 0x7f38000a5240 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f3800138e50 con 0x7f38000a4920 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f38000a65f0 con 0x7f38000a2530 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 10 monclient(hunting): authenticate will time out at 1154.717325s 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a2530 0x7f38001cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a2530 0x7f38001cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34630/0 (socket says 172.21.15.98:34630) 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 1 -- 172.21.15.98:0/768162621 learned_addr learned my addr 172.21.15.98:0/768162621 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 10 monclient(hunting): get_auth_request con 0x7f38000a2530 auth_method 0 2024-06-11T19:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37ff7fe640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38000a5240 0x7f3800122fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 10 monclient(hunting): handle_auth_done global_id 54879 payload 824 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 1 -- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f38000a4920 msgr2=0x7f38001ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f38000a4920 0x7f38001ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 1 -- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38000a5240 msgr2=0x7f3800122fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38000a5240 0x7f3800122fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 10 monclient: _finish_hunting 0 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 1 monclient: found mon.noname-d 2024-06-11T19:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 1 -- 172.21.15.98:0/768162621 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38000a8ed0 con 0x7f38000a2530 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37ff7fe640 10 monclient: get_auth_request con 0x7f38000a5240 auth_method 0 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37ff7fe640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38000a5240 0x7f3800122fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37feffd640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a2530 0x7f38001cdcf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009f70 tx=0x7f37f8004a40 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 10 client.?.objecter ms_handle_connect 0x7f38000a2530 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f37f8043020 con 0x7f38000a2530 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 10 monclient: dump: 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 10 monclient: _finish_auth 0 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 5 monclient: authenticate success, global_id 54879 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37f80046c0 con 0x7f38000a2530 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 10 monclient: _renew_subs 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f380598c640 1 -- 172.21.15.98:0/768162621 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3800124bc0 con 0x7f38000a2530 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f37f803e8c0 con 0x7f38000a2530 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f37e7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f380598c640 10 client.54879.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f37e7fff640 10 monclient: dump: 2024-06-11T19:15:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f380598c640 10 monclient: _renew_subs 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f380598c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f380598c640 1 -- 172.21.15.98:0/768162621 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f3800126c40 con 0x7f38000a2530 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f380598c640 1 librados: init done 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.910+0000 7f37fd7fa640 4 set_mon_vals no callback set 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f380598c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f380598c640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123520 0x7f3800123900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f380598c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f380598c640 1 -- 172.21.15.98:0/768162621 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f3800050830 con 0x7f3800123520 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f37f8046020 con 0x7f38000a2530 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.911+0000 7f37fe7fc640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123520 0x7f3800123900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37fe7fc640 10 monclient: get_auth_request con 0x7f3800123520 auth_method 0 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37fe7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f37ec03fdb0 0x7f37ec042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(287..287 src has 1..287) ==== 6682+0+0 (secure 0 0 0) 0x7f37f803a040 con 0x7f38000a2530 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 10 client.54879.objecter ms_dispatch 0x7f3800000b90 osd_map(287..287 src has 1..287) 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 3 client.54879.objecter handle_osd_map got epochs [287,287] > 0 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 3 client.54879.objecter handle_osd_map decoding full epoch 287 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 20 client.54879.objecter dump_active .. 0 homeless 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37ff7fe640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f37ec03fdb0 0x7f37ec042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37ff7fe640 10 monclient: get_auth_request con 0x7f37ec03fdb0 auth_method 0 2024-06-11T19:15:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37fe7fc640 10 monclient(hunting): handle_auth_done global_id 22802 payload 824 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37fe7fc640 1 -- 172.21.15.98:0/768162621 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38000a4d00 con 0x7f3800123520 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 10 client.54879.objecter ms_handle_connect 0x7f3800123520 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37fe7fc640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123520 0x7f3800123900 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f38000a5620 tx=0x7f37f400ec00 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 10 client.54879.objecter resend_mon_ops 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 10 client.54879.objecter ms_handle_connect 0x7f37ec03fdb0 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37ff7fe640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f37ec03fdb0 0x7f37ec042270 secure :-1 s=READY pgs=4128 cs=0 l=1 rev1=1 crypto rx=0x7f3800124740 tx=0x7f37f0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f37f40044a0 con 0x7f3800123520 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 10 monclient: dump: 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f37f4004030 con 0x7f3800123520 2024-06-11T19:15:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123520 msgr2=0x7f3800123900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123520 0x7f3800123900 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f38000a5620 tx=0x7f37f400ec00 comp rx=0 tx=0).stop 2024-06-11T19:15:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123520 msgr2=0x7f3800123900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.912+0000 7f37e7fff640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123520 0x7f3800123900 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f380598c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f380598c640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123e40 0x7f380009e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f380598c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f380598c640 1 -- 172.21.15.98:0/768162621 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f3800127950 con 0x7f3800123e40 2024-06-11T19:15:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f37fe7fc640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123e40 0x7f380009e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f37fe7fc640 10 monclient: get_auth_request con 0x7f3800123e40 auth_method 0 2024-06-11T19:15:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f37fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f37fe7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f37fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f37fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f37fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f37fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f37fe7fc640 10 monclient(hunting): handle_auth_done global_id 22811 payload 824 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.919+0000 7f37fe7fc640 1 -- 172.21.15.98:0/768162621 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3800125460 con 0x7f3800123e40 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37fe7fc640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123e40 0x7f380009e9f0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f38000a5620 tx=0x7f37f40046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 10 client.54879.objecter ms_handle_connect 0x7f3800123e40 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 10 client.54879.objecter resend_mon_ops 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f37f4005e00 con 0x7f3800123e40 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 10 monclient: dump: 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f37f4003bf0 con 0x7f3800123e40 2024-06-11T19:15:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123e40 msgr2=0x7f380009e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123e40 0x7f380009e9f0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f38000a5620 tx=0x7f37f40046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 1 -- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123e40 msgr2=0x7f380009e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37e7fff640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3800123e40 0x7f380009e9f0 secure :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f38000a5620 tx=0x7f37f40046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:44.941 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:44.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:44.941 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:44.941 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:44.941 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 395115, 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.942 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.943 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.944 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:44.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:44.946 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:44.947 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:44.951 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:44.953 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.954 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:44.957 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:44.961 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.962 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:44.963 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:44.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:44.963 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:44.963 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:44.963 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:15:44.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:44.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:44.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:44.963 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:44.963 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:44.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f380598c640 10 librados: watch_flush enter 2024-06-11T19:15:44.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f380598c640 10 librados: watch_flush exit 2024-06-11T19:15:44.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f380598c640 20 client.54879.objecter shutdown clearing up homeless session... 2024-06-11T19:15:44.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f380598c640 10 client.54879.objecter successfully canceled tick 2024-06-11T19:15:44.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f380598c640 1 -- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f37ec03fdb0 msgr2=0x7f37ec042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f380598c640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f37ec03fdb0 0x7f37ec042270 secure :-1 s=READY pgs=4128 cs=0 l=1 rev1=1 crypto rx=0x7f3800124740 tx=0x7f37f0009040 comp rx=0 tx=0).stop 2024-06-11T19:15:44.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f380598c640 10 monclient: shutdown 2024-06-11T19:15:44.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f380598c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:44.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f380598c640 1 -- 172.21.15.98:0/768162621 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a2530 msgr2=0x7f38001cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:44.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f380598c640 1 --2- 172.21.15.98:0/768162621 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f38000a2530 0x7f38001cdcf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009f70 tx=0x7f37f8004a40 comp rx=0 tx=0).stop 2024-06-11T19:15:44.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37ff7fe640 1 -- 172.21.15.98:0/768162621 reap_dead start 2024-06-11T19:15:44.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f37ff7fe640 1 -- 172.21.15.98:0/768162621 reap_dead start 2024-06-11T19:15:44.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f380598c640 1 -- 172.21.15.98:0/768162621 shutdown_connections 2024-06-11T19:15:44.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.920+0000 7f380598c640 1 -- 172.21.15.98:0/768162621 >> 172.21.15.98:0/768162621 conn(0x7f38000ab850 msgr2=0x7f38001be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:44.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.921+0000 7f380598c640 1 -- 172.21.15.98:0/768162621 shutdown_connections 2024-06-11T19:15:44.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.921+0000 7f380598c640 1 -- 172.21.15.98:0/768162621 wait complete. 2024-06-11T19:15:44.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:44.921+0000 7f380598c640 1 librados: shutdown 2024-06-11T19:15:44.966 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:44.966 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:15:44.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:15:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.067+0000 7f8245e93640 10 monclient: get_monmap_and_config 2024-06-11T19:15:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.067+0000 7f8245e93640 10 monclient: build_initial_monmap 2024-06-11T19:15:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.067+0000 7f8245e93640 10 monclient: monmap: 2024-06-11T19:15:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:45.068777+0000 2024-06-11T19:15:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:45.068777+0000 2024-06-11T19:15:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:45.068 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:45.068 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:45.068 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:45.068 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:45.068 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:45.068 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:45.068 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:45.068 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:45.068 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.068+0000 7f8245e93640 1 Processor -- start 2024-06-11T19:15:45.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.068+0000 7f8245e93640 1 -- start start 2024-06-11T19:15:45.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.068+0000 7f8245e93640 10 monclient: init 2024-06-11T19:15:45.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.068+0000 7f8245e93640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:45.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.068+0000 7f8245e93640 10 monclient: _add_conns ranks=[3,4,1,5,6,7,10,9,8,0,2] 2024-06-11T19:15:45.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f8245e93640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82400a4920 0x7f82400a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f8245e93640 10 monclient(hunting): picked mon.noname-g con 0x7f82400a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:45.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f8245e93640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82400a5240 0x7f82400a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f8245e93640 10 monclient(hunting): picked mon.noname-c con 0x7f82400a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:15:45.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f8245e93640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f82400a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:45.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f8245e93640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82400a7140 con 0x7f82400a2530 2024-06-11T19:15:45.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f8245e93640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:45.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f8245e93640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f82400a2b50 con 0x7f82400a4920 2024-06-11T19:15:45.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f8245e93640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:45.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f8245e93640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f82400a2e30 con 0x7f82400a5240 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f8245e93640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f8245e93640 10 monclient(hunting): authenticate will time out at 1154.876073s 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82400a5240 0x7f82400a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82400a5240 0x7f82400a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:53678/0 (socket says 172.21.15.98:53678) 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 1 -- 172.21.15.98:0/2493352109 learned_addr learned my addr 172.21.15.98:0/2493352109 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 10 monclient(hunting): get_auth_request con 0x7f82400a5240 auth_method 0 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:45.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 10 monclient(hunting): handle_auth_done global_id 23728 payload 274 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 1 -- 172.21.15.98:0/2493352109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82400a4920 msgr2=0x7f82400a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823f7fe640 1 --2- 172.21.15.98:0/2493352109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82400a4920 0x7f82400a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 1 --2- 172.21.15.98:0/2493352109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82400a4920 0x7f82400a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 1 -- 172.21.15.98:0/2493352109 >> v1:172.21.15.117:6789/0 conn(0x7f82400a2530 legacy=0x7f82400a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 10 monclient: _finish_hunting 0 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 1 monclient: found mon.noname-c 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823effd640 1 -- 172.21.15.98:0/2493352109 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82400a3110 con 0x7f82400a5240 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823f7fe640 10 monclient: get_auth_request con 0x7f82400a4920 auth_method 0 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.069+0000 7f823f7fe640 1 --2- 172.21.15.98:0/2493352109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82400a4920 0x7f82400a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823effd640 1 --2- 172.21.15.98:0/2493352109 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82400a5240 0x7f82400a59a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f82340098a0 tx=0x7f823402b7a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=e0fbf3d8769fbd59 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 1 -- 172.21.15.98:0/2493352109 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8234038070 con 0x7f82400a5240 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:45.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 10 monclient: dump: 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:45.081 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:45.082 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:45.082 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 10 monclient: _finish_auth 0 2024-06-11T19:15:45.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:15:45.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:45.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 1 -- 172.21.15.98:0/2493352109 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8234031970 con 0x7f82400a5240 2024-06-11T19:15:45.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 5 monclient: authenticate success, global_id 23728 2024-06-11T19:15:45.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:45.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:45.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:45.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 1 -- 172.21.15.98:0/2493352109 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8234031ba0 con 0x7f82400a5240 2024-06-11T19:15:45.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 4 set_mon_vals no callback set 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f823e7fc640 10 monclient: dump: 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:45.083 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 10 monclient: shutdown 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 1 -- 172.21.15.98:0/2493352109 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82400a5240 msgr2=0x7f82400a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 1 --2- 172.21.15.98:0/2493352109 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82400a5240 0x7f82400a59a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f82340098a0 tx=0x7f823402b7a0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 1 -- 172.21.15.98:0/2493352109 shutdown_connections 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 1 --2- 172.21.15.98:0/2493352109 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82400a5240 0x7f82400a59a0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 1 --2- 172.21.15.98:0/2493352109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82400a4920 0x7f82400a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 1 -- 172.21.15.98:0/2493352109 >> 172.21.15.98:0/2493352109 conn(0x7f82400ab850 msgr2=0x7f82400a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 1 -- 172.21.15.98:0/2493352109 shutdown_connections 2024-06-11T19:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.070+0000 7f8245e93640 1 -- 172.21.15.98:0/2493352109 wait complete. 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.071+0000 7f8245e93640 10 monclient: build_initial_monmap 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.071+0000 7f8245e93640 10 monclient: monmap: 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.071+0000 7f8245e93640 1 librados: starting msgr at 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.071+0000 7f8245e93640 1 librados: starting objecter 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.071+0000 7f8245e93640 1 Processor -- start 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 1 -- start start 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 1 librados: setting wanted keys 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 1 librados: calling monclient init 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 10 monclient: init 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 10 monclient: _add_conns ranks=[3,0,8,5,1,2,7,4,6] 2024-06-11T19:15:45.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82400a2530 0x7f8240137620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 10 monclient(hunting): picked mon.noname-g con 0x7f82400a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82400a4920 0x7f8240137b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 10 monclient(hunting): picked mon.noname-a con 0x7f82400a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f82400a5240 0x7f82401230e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 10 monclient(hunting): picked mon.noname-e con 0x7f82400a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f82400a6da0 con 0x7f82400a4920 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f8240138e50 con 0x7f82400a2530 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f82400a65f0 con 0x7f82400a5240 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f8245e93640 10 monclient(hunting): authenticate will time out at 1154.879446s 2024-06-11T19:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82400a4920 0x7f8240137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82400a4920 0x7f8240137b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46316/0 (socket says 172.21.15.98:46316) 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 1 -- 172.21.15.98:0/3006036108 learned_addr learned my addr 172.21.15.98:0/3006036108 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 10 monclient(hunting): get_auth_request con 0x7f82400a4920 auth_method 0 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823ffff640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f82400a5240 0x7f82401230e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823ffff640 10 monclient(hunting): get_auth_request con 0x7f82400a5240 auth_method 0 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823f7fe640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82400a2530 0x7f8240137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823f7fe640 10 monclient(hunting): get_auth_request con 0x7f82400a2530 auth_method 0 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 10 monclient(hunting): handle_auth_done global_id 64593 payload 824 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 1 -- 172.21.15.98:0/3006036108 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f82400a5240 msgr2=0x7f82401230e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f82400a5240 0x7f82401230e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 1 -- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82400a2530 msgr2=0x7f8240137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82400a2530 0x7f8240137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 10 monclient: _finish_hunting 0 2024-06-11T19:15:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 1 monclient: found mon.noname-a 2024-06-11T19:15:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:15:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.072+0000 7f823effd640 1 -- 172.21.15.98:0/3006036108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82400a8da0 con 0x7f82400a4920 2024-06-11T19:15:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f823effd640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82400a4920 0x7f8240137b60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f82340090b0 tx=0x7f82340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 10 client.?.objecter ms_handle_connect 0x7f82400a4920 2024-06-11T19:15:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8234038050 con 0x7f82400a4920 2024-06-11T19:15:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:15:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 10 monclient: dump: 2024-06-11T19:15:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 10 monclient: _finish_auth 0 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 10 monclient: _check_auth_tickets 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8245e93640 5 monclient: authenticate success, global_id 64593 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8234039040 con 0x7f82400a4920 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8245e93640 10 monclient: _renew_subs 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8245e93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8245e93640 1 -- 172.21.15.98:0/3006036108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82400a9280 con 0x7f82400a4920 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f823403d400 con 0x7f82400a4920 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8245e93640 10 client.64593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8227fff640 10 monclient: dump: 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:45.094 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f823dffb640 4 set_mon_vals no callback set 2024-06-11T19:15:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8245e93640 10 monclient: _renew_subs 2024-06-11T19:15:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8245e93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8245e93640 1 -- 172.21.15.98:0/3006036108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8240126d20 con 0x7f82400a4920 2024-06-11T19:15:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.073+0000 7f8245e93640 1 librados: init done 2024-06-11T19:15:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8234046030 con 0x7f82400a4920 2024-06-11T19:15:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8245e93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8245e93640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8240063180 0x7f8240063560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8245e93640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8245e93640 1 -- 172.21.15.98:0/3006036108 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f82401c0c70 con 0x7f8240063180 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8227fff640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f822c03fde0 0x7f822c0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823f7fe640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8240063180 0x7f8240063560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823f7fe640 10 monclient: get_auth_request con 0x7f8240063180 auth_method 0 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(287..287 src has 1..287) ==== 6682+0+0 (secure 0 0 0) 0x7f8234044020 con 0x7f82400a4920 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8227fff640 10 client.64593.objecter ms_dispatch 0x7f8240000b90 osd_map(287..287 src has 1..287) 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8227fff640 3 client.64593.objecter handle_osd_map got epochs [287,287] > 0 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8227fff640 3 client.64593.objecter handle_osd_map decoding full epoch 287 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823ffff640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f822c03fde0 0x7f822c0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823ffff640 10 monclient: get_auth_request con 0x7f822c03fde0 auth_method 0 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8227fff640 20 client.64593.objecter dump_active .. 0 homeless 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823f7fe640 10 monclient(hunting): handle_auth_done global_id 84112 payload 824 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823f7fe640 1 -- 172.21.15.98:0/3006036108 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f82400a4d00 con 0x7f8240063180 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8227fff640 10 client.64593.objecter ms_handle_connect 0x7f8240063180 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823f7fe640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8240063180 0x7f8240063560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8230007fa0 tx=0x7f823000c850 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8227fff640 10 client.64593.objecter resend_mon_ops 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f8227fff640 10 client.64593.objecter ms_handle_connect 0x7f822c03fde0 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.074+0000 7f823ffff640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f822c03fde0 0x7f822c0422a0 secure :-1 s=READY pgs=4129 cs=0 l=1 rev1=1 crypto rx=0x7f82400a2950 tx=0x7f8238007db0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.075+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f823000a8d0 con 0x7f8240063180 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.075+0000 7f8227fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.075+0000 7f8227fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.075+0000 7f8227fff640 10 monclient: dump: 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.075+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f823000a460 con 0x7f8240063180 2024-06-11T19:15:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.075+0000 7f8227fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.075+0000 7f8227fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.075+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8240063180 msgr2=0x7f8240063560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.075+0000 7f8227fff640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8240063180 0x7f8240063560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8230007fa0 tx=0x7f823000c850 comp rx=0 tx=0).stop 2024-06-11T19:15:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.075+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8240063180 msgr2=0x7f8240063560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.075+0000 7f8227fff640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8240063180 0x7f8240063560 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8230007fa0 tx=0x7f823000c850 comp rx=0 tx=0).stop 2024-06-11T19:15:45.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.081+0000 7f8245e93640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:45.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.081+0000 7f8245e93640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f824002cb40 0x7f824002cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.081+0000 7f8245e93640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:45.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.081+0000 7f8245e93640 1 -- 172.21.15.98:0/3006036108 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8240050830 con 0x7f824002cb40 2024-06-11T19:15:45.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f823f7fe640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f824002cb40 0x7f824002cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f823f7fe640 10 monclient: get_auth_request con 0x7f824002cb40 auth_method 0 2024-06-11T19:15:45.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f823f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:45.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f823f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:45.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f823f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:45.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:45.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:45.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f823f7fe640 10 monclient(hunting): handle_auth_done global_id 84121 payload 824 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f823f7fe640 1 -- 172.21.15.98:0/3006036108 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f82400a0100 con 0x7f824002cb40 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f823f7fe640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f824002cb40 0x7f824002cf50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8230000f80 tx=0x7f8230004a40 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f8227fff640 10 client.64593.objecter ms_handle_connect 0x7f824002cb40 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f8227fff640 10 client.64593.objecter resend_mon_ops 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8230004620 con 0x7f824002cb40 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f8227fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f8227fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.082+0000 7f8227fff640 10 monclient: dump: 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f8230004620 con 0x7f824002cb40 2024-06-11T19:15:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8227fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8227fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f824002cb40 msgr2=0x7f824002cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8227fff640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f824002cb40 0x7f824002cf50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8230000f80 tx=0x7f8230004a40 comp rx=0 tx=0).stop 2024-06-11T19:15:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8227fff640 1 -- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f824002cb40 msgr2=0x7f824002cf50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8227fff640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f824002cb40 0x7f824002cf50 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8230000f80 tx=0x7f8230004a40 comp rx=0 tx=0).stop 2024-06-11T19:15:45.103 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:45.103 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:45.103 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:45.103 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:45.103 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:45.103 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 16852, 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:45.104 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:45.105 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:45.107 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.108 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:45.109 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:45.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:45.111 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:45.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:45.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:45.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:45.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.117 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:45.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:45.119 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:45.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:45.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:45.122 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:45.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:45.124 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:45.124 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:45.124 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:45.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:45.124 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:45.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:45.124 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:45.124 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:45.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 10 librados: watch_flush enter 2024-06-11T19:15:45.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 10 librados: watch_flush exit 2024-06-11T19:15:45.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 20 client.64593.objecter shutdown clearing up homeless session... 2024-06-11T19:15:45.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 10 client.64593.objecter successfully canceled tick 2024-06-11T19:15:45.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 1 -- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f822c03fde0 msgr2=0x7f822c0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f822c03fde0 0x7f822c0422a0 secure :-1 s=READY pgs=4129 cs=0 l=1 rev1=1 crypto rx=0x7f82400a2950 tx=0x7f8238007db0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 10 monclient: shutdown 2024-06-11T19:15:45.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:45.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 1 -- 172.21.15.98:0/3006036108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82400a4920 msgr2=0x7f8240137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 1 --2- 172.21.15.98:0/3006036108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82400a4920 0x7f8240137b60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f82340090b0 tx=0x7f82340046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f823ffff640 1 -- 172.21.15.98:0/3006036108 reap_dead start 2024-06-11T19:15:45.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f823ffff640 1 -- 172.21.15.98:0/3006036108 reap_dead start 2024-06-11T19:15:45.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 1 -- 172.21.15.98:0/3006036108 shutdown_connections 2024-06-11T19:15:45.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 1 -- 172.21.15.98:0/3006036108 >> 172.21.15.98:0/3006036108 conn(0x7f82400ab850 msgr2=0x7f82400a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:45.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 1 -- 172.21.15.98:0/3006036108 shutdown_connections 2024-06-11T19:15:45.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 1 -- 172.21.15.98:0/3006036108 wait complete. 2024-06-11T19:15:45.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.083+0000 7f8245e93640 1 librados: shutdown 2024-06-11T19:15:45.127 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:45.127 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-06-11T19:15:45.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-06-11T19:15:45.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.225+0000 7f3ac8574640 10 monclient: get_monmap_and_config 2024-06-11T19:15:45.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.225+0000 7f3ac8574640 10 monclient: build_initial_monmap 2024-06-11T19:15:45.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.225+0000 7f3ac8574640 10 monclient: monmap: 2024-06-11T19:15:45.224 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:45.224 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:45.226208+0000 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:45.226208+0000 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:45.225 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.225+0000 7f3ac8574640 1 Processor -- start 2024-06-11T19:15:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.225+0000 7f3ac8574640 1 -- start start 2024-06-11T19:15:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.225+0000 7f3ac8574640 10 monclient: init 2024-06-11T19:15:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 10 monclient: _add_conns ranks=[2,8,0,1,7,10,5,3,9,6,4] 2024-06-11T19:15:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 0x7f3ac00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 10 monclient(hunting): picked mon.noname-b con 0x7f3ac00a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ac00a5240 0x7f3ac00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 10 monclient(hunting): picked mon.noname-e con 0x7f3ac00a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3ac00a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ac00a7140 con 0x7f3ac00a2530 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3ac00a2b50 con 0x7f3ac00a4920 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3ac00a2e30 con 0x7f3ac00a5240 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac8574640 10 monclient(hunting): authenticate will time out at 1155.033505s 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 0x7f3ac00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6b11640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3ac00a2530 0x7f3ac00a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35724/0 (socket says 172.21.15.98:35724) 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6b11640 1 -- 172.21.15.98:0/33525186 learned_addr learned my addr 172.21.15.98:0/33525186 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 10 monclient(hunting): get_auth_request con 0x7f3ac00a4920 auth_method 0 2024-06-11T19:15:45.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac5b0f640 1 --2- 172.21.15.98:0/33525186 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ac00a5240 0x7f3ac00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac5b0f640 10 monclient(hunting): get_auth_request con 0x7f3ac00a5240 auth_method 0 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac5b0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac5b0f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac5b0f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac5b0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac5b0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac5b0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:45.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 10 monclient(hunting): handle_auth_done global_id 84152 payload 274 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 1 -- 172.21.15.98:0/33525186 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ac00a5240 msgr2=0x7f3ac00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 1 --2- 172.21.15.98:0/33525186 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ac00a5240 0x7f3ac00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 1 -- 172.21.15.98:0/33525186 >> v1:172.21.15.98:6789/0 conn(0x7f3ac00a2530 legacy=0x7f3ac00a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac530e640 1 -- 172.21.15.98:0/33525186 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 447149780 0 0) 0x7f3ac00a7140 con 0x7f3ac00a2530 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 10 monclient: _finish_hunting 0 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 1 monclient: found mon.noname-b 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac6310640 1 -- 172.21.15.98:0/33525186 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac00a3110 con 0x7f3ac00a4920 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac5b0f640 1 --2- 172.21.15.98:0/33525186 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ac00a5240 0x7f3ac00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.226+0000 7f3ac530e640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac6310640 1 --2- 172.21.15.98:0/33525186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 0x7f3ac00a4d00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0097d0 tx=0x7f3abc02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e27743427e03e53 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 1 -- 172.21.15.98:0/33525186 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3abc039070 con 0x7f3ac00a4920 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:15:45.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 10 monclient: dump: 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:45.237 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 10 monclient: _finish_auth 0 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 10 monclient: _check_auth_tickets 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 1 -- 172.21.15.98:0/33525186 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3abc031960 con 0x7f3ac00a4920 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac8574640 5 monclient: authenticate success, global_id 84152 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac8574640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac8574640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac8574640 4 set_mon_vals no callback set 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 1 -- 172.21.15.98:0/33525186 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3abc032380 con 0x7f3ac00a4920 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:45.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.227+0000 7f3ac530e640 10 monclient: dump: 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:45.239 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.228+0000 7f3ac8574640 10 monclient: shutdown 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.228+0000 7f3ac8574640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.228+0000 7f3ac8574640 1 -- 172.21.15.98:0/33525186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 msgr2=0x7f3ac00a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.228+0000 7f3ac8574640 1 --2- 172.21.15.98:0/33525186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 0x7f3ac00a4d00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0097d0 tx=0x7f3abc02b790 comp rx=0 tx=0).stop 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.228+0000 7f3ac8574640 1 -- 172.21.15.98:0/33525186 shutdown_connections 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.228+0000 7f3ac8574640 1 --2- 172.21.15.98:0/33525186 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ac00a5240 0x7f3ac00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.228+0000 7f3ac8574640 1 --2- 172.21.15.98:0/33525186 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 0x7f3ac00a4d00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.228+0000 7f3ac8574640 1 -- 172.21.15.98:0/33525186 >> 172.21.15.98:0/33525186 conn(0x7f3ac00ab850 msgr2=0x7f3ac00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.228+0000 7f3ac8574640 1 -- 172.21.15.98:0/33525186 shutdown_connections 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.228+0000 7f3ac8574640 1 -- 172.21.15.98:0/33525186 wait complete. 2024-06-11T19:15:45.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.228+0000 7f3ac8574640 10 monclient: build_initial_monmap 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 10 monclient: monmap: 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:45.241 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 1 librados: starting msgr at 2024-06-11T19:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 1 librados: starting objecter 2024-06-11T19:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 1 Processor -- start 2024-06-11T19:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 1 -- start start 2024-06-11T19:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 1 librados: setting wanted keys 2024-06-11T19:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 1 librados: calling monclient init 2024-06-11T19:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 10 monclient: init 2024-06-11T19:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 10 monclient: _add_conns ranks=[1,2,6,5,3,0,7,8,4] 2024-06-11T19:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ac00a2530 0x7f3ac01cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 10 monclient(hunting): picked mon.noname-f con 0x7f3ac00a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 0x7f3ac01cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 10 monclient(hunting): picked mon.noname-b con 0x7f3ac00a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3ac00a5240 0x7f3ac0124da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 10 monclient(hunting): picked mon.noname-d con 0x7f3ac00a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac00a6da0 con 0x7f3ac00a2530 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3ac0138e50 con 0x7f3ac00a4920 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3ac00a65f0 con 0x7f3ac00a5240 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.229+0000 7f3ac8574640 10 monclient(hunting): authenticate will time out at 1155.036945s 2024-06-11T19:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 0x7f3ac01cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac6b11640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3ac00a5240 0x7f3ac0124da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 0x7f3ac01cdf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:59856/0 (socket says 172.21.15.98:59856) 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 1 -- 172.21.15.98:0/1724375793 learned_addr learned my addr 172.21.15.98:0/1724375793 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 10 monclient(hunting): get_auth_request con 0x7f3ac00a4920 auth_method 0 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac6b11640 10 monclient(hunting): get_auth_request con 0x7f3ac00a5240 auth_method 0 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac6b11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac6b11640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac6b11640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac6b11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac6b11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac6b11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 10 monclient(hunting): handle_auth_done global_id 84161 payload 824 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 1 -- 172.21.15.98:0/1724375793 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3ac00a5240 msgr2=0x7f3ac0124da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3ac00a5240 0x7f3ac0124da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac6310640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ac00a2530 0x7f3ac01cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 1 -- 172.21.15.98:0/1724375793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ac00a2530 msgr2=0x7f3ac01cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ac00a2530 0x7f3ac01cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 10 monclient: _finish_hunting 0 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 1 monclient: found mon.noname-b 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 1 -- 172.21.15.98:0/1724375793 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac0128750 con 0x7f3ac00a4920 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac5b0f640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 0x7f3ac01cdf80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400e8d0 tx=0x7f3ab400ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 10 client.?.objecter ms_handle_connect 0x7f3ac00a4920 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 1 -- 172.21.15.98:0/1724375793 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ab4017070 con 0x7f3ac00a4920 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 10 monclient: dump: 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:45.246 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 10 monclient: _finish_auth 0 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 1 -- 172.21.15.98:0/1724375793 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ab400cdb0 con 0x7f3ac00a4920 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac8574640 5 monclient: authenticate success, global_id 84161 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac8574640 10 monclient: _renew_subs 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac8574640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac8574640 1 -- 172.21.15.98:0/1724375793 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac0128a30 con 0x7f3ac00a4920 2024-06-11T19:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 1 -- 172.21.15.98:0/1724375793 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ab40138e0 con 0x7f3ac00a4920 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ab2ffd640 10 monclient: dump: 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac8574640 10 client.84161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac8574640 10 monclient: _renew_subs 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac8574640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac4b0d640 4 set_mon_vals no callback set 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac8574640 1 -- 172.21.15.98:0/1724375793 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac0127580 con 0x7f3ac00a4920 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.230+0000 7f3ac8574640 1 librados: init done 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.231+0000 7f3ab2ffd640 1 -- 172.21.15.98:0/1724375793 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3ab401a070 con 0x7f3ac00a4920 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.231+0000 7f3ac8574640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.231+0000 7f3ac8574640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.231+0000 7f3ac8574640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.231+0000 7f3ac8574640 1 -- 172.21.15.98:0/1724375793 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac00a4d00 con 0x7f3ac00a4920 2024-06-11T19:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.231+0000 7f3ab2ffd640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3aa403fdb0 0x7f3aa4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.231+0000 7f3ab2ffd640 1 -- 172.21.15.98:0/1724375793 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(287..287 src has 1..287) ==== 6682+0+0 (secure 0 0 0) 0x7f3ab400f040 con 0x7f3ac00a4920 2024-06-11T19:15:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.231+0000 7f3ab2ffd640 10 client.84161.objecter ms_dispatch 0x7f3ac0000b90 osd_map(287..287 src has 1..287) 2024-06-11T19:15:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.231+0000 7f3ab2ffd640 3 client.84161.objecter handle_osd_map got epochs [287,287] > 0 2024-06-11T19:15:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.231+0000 7f3ab2ffd640 3 client.84161.objecter handle_osd_map decoding full epoch 287 2024-06-11T19:15:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.231+0000 7f3ab2ffd640 20 client.84161.objecter dump_active .. 0 homeless 2024-06-11T19:15:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.231+0000 7f3ac6310640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3aa403fdb0 0x7f3aa4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.232+0000 7f3ac6310640 10 monclient: get_auth_request con 0x7f3aa403fdb0 auth_method 0 2024-06-11T19:15:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.232+0000 7f3ab2ffd640 10 client.84161.objecter ms_handle_connect 0x7f3aa403fdb0 2024-06-11T19:15:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.232+0000 7f3ac6310640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3aa403fdb0 0x7f3aa4042270 secure :-1 s=READY pgs=4130 cs=0 l=1 rev1=1 crypto rx=0x7f3abc036000 tx=0x7f3abc008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.236+0000 7f3ab2ffd640 1 -- 172.21.15.98:0/1724375793 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3ab4021070 con 0x7f3ac00a4920 2024-06-11T19:15:45.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.236+0000 7f3ab2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:45.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.236+0000 7f3ab2ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.490+0000 7f3ac8574640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-06-11T19:15:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.490+0000 7f3ac8574640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-06-11T19:15:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.490+0000 7f3ac8574640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.490+0000 7f3ac8574640 1 -- 172.21.15.98:0/1724375793 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f3ac0128c70 con 0x7f3ac00a4920 2024-06-11T19:15:45.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.493+0000 7f3ab2ffd640 1 -- 172.21.15.98:0/1724375793 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f3ab4013520 con 0x7f3ac00a4920 2024-06-11T19:15:45.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.493+0000 7f3ab2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-06-11T19:15:45.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.493+0000 7f3ab2ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:45.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=2520807208,config=692619927,health=1995790572,kv=2000936593,logm=19716690}) 2024-06-11T19:15:45.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=10,kv=10,logm=64} crc {auth=4172230797,config=692619927,health=2657241731,kv=2363734839,logm=1613559006}) 2024-06-11T19:15:45.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=2520807208,config=692619927,health=1995790572,kv=2000936593,logm=19716690}) 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=2042966434,config=692619927,health=3445046098,kv=126839869,logm=685156379}) 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=2520807208,config=692619927,health=1995790572,kv=2000936593,logm=19716690}) 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=1300499772,config=692619927,health=1995790572,kv=2000936593,logm=508400181}) 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=2520807208,config=692619927,health=1995790572,kv=2000936593,logm=19716690}) 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=15,config=2,health=11,kv=10,logm=62} crc {auth=1419513379,config=692619927,health=4180235331,kv=2000936593,logm=2225082292}) 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=2520807208,config=692619927,health=1995790572,kv=2000936593,logm=19716690}) 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=15,config=2,health=12,kv=10,logm=61} crc {auth=3081493357,config=692619927,health=1995790572,kv=2000936593,logm=614312406}) 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=2520807208,config=692619927,health=1995790572,kv=2000936593,logm=19716690}) 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=15,config=2,health=10,kv=9,logm=64} crc {auth=708848241,config=2435484489,health=3007005210,kv=4103510944,logm=1168748671}) 2024-06-11T19:15:45.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=2520807208,config=692619927,health=1995790572,kv=2000936593,logm=19716690}) 2024-06-11T19:15:45.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=13,config=2,health=12,kv=10,logm=63} crc {auth=576491958,config=692619927,health=1995790572,kv=2000936593,logm=912846282}) 2024-06-11T19:15:45.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=2520807208,config=692619927,health=1995790572,kv=2000936593,logm=19716690}) 2024-06-11T19:15:45.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.493+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=15,config=2,health=11,kv=9,logm=63} crc {auth=1419513379,config=692619927,health=1266736088,kv=2031577660,logm=2130599296}) 2024-06-11T19:15:45.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 10 librados: watch_flush enter 2024-06-11T19:15:45.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 10 librados: watch_flush exit 2024-06-11T19:15:45.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 20 client.84161.objecter shutdown clearing up homeless session... 2024-06-11T19:15:45.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 10 client.84161.objecter successfully canceled tick 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 -- 172.21.15.98:0/1724375793 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3aa403fdb0 msgr2=0x7f3aa4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3aa403fdb0 0x7f3aa4042270 secure :-1 s=READY pgs=4130 cs=0 l=1 rev1=1 crypto rx=0x7f3abc036000 tx=0x7f3abc008030 comp rx=0 tx=0).stop 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 10 monclient: shutdown 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 -- 172.21.15.98:0/1724375793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 msgr2=0x7f3ac01cdf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 0x7f3ac01cdf80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400e8d0 tx=0x7f3ab400ed90 comp rx=0 tx=0).stop 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 -- 172.21.15.98:0/1724375793 shutdown_connections 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3aa403fdb0 0x7f3aa4042270 unknown :-1 s=CLOSED pgs=4130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3ac00a5240 0x7f3ac0124da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ac00a4920 0x7f3ac01cdf80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 --2- 172.21.15.98:0/1724375793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ac00a2530 0x7f3ac01cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 -- 172.21.15.98:0/1724375793 >> 172.21.15.98:0/1724375793 conn(0x7f3ac00ab850 msgr2=0x7f3ac009c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 -- 172.21.15.98:0/1724375793 shutdown_connections 2024-06-11T19:15:45.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 -- 172.21.15.98:0/1724375793 wait complete. 2024-06-11T19:15:45.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:45.495+0000 7f3ac8574640 1 librados: shutdown 2024-06-11T19:15:45.510 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-06-11T19:15:45.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.495+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.495+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826178345}) 2024-06-11T19:15:45.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.495+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1009423180}) 2024-06-11T19:15:45.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.495+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.495+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826178345}) 2024-06-11T19:15:45.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.495+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1618112766}) 2024-06-11T19:15:45.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.495+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.495+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826178345}) 2024-06-11T19:15:45.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.495+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=422338445}) 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826178345}) 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3709815215}) 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826178345}) 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2825906676}) 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826178345}) 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2376027571}) 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826178345}) 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1513086764}) 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826178345}) 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.496+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=516872727}) 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404976797}) 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571330181}) 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404976797}) 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1999422083}) 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404976797}) 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2375958280}) 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404976797}) 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=638467398}) 2024-06-11T19:15:45.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404976797}) 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2144768252}) 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404976797}) 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1606463889}) 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404976797}) 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=287271372}) 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404976797}) 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.498+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=373332536}) 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394409467}) 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3356054169}) 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394409467}) 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2949453923}) 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394409467}) 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1445478738}) 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394409467}) 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2125245633}) 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394409467}) 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4270679853}) 2024-06-11T19:15:45.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394409467}) 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3172298281}) 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394409467}) 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1211024945}) 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394409467}) 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.500+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1966004322}) 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2469008077}) 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2735695125}) 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2469008077}) 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526761059}) 2024-06-11T19:15:45.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2469008077}) 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1012981805}) 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2469008077}) 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1872991246}) 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2469008077}) 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3052169472}) 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2469008077}) 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=717686079}) 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2469008077}) 2024-06-11T19:15:45.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3673015872}) 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2469008077}) 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.501+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1170038409}) 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.502+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.502+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805923898}) 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.502+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1671511425}) 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.502+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.502+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805923898}) 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2759876436}) 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805923898}) 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3008913120}) 2024-06-11T19:15:45.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805923898}) 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1344373390}) 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805923898}) 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3452441741}) 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805923898}) 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=861469135}) 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805923898}) 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1665870274}) 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805923898}) 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.503+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2855393505}) 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093828300}) 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528770595}) 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093828300}) 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2984778575}) 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093828300}) 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2759903883}) 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093828300}) 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3986764842}) 2024-06-11T19:15:45.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093828300}) 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2147169351}) 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093828300}) 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3631196362}) 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093828300}) 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1160419561}) 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093828300}) 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.504+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2817922585}) 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.505+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.505+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2033142363}) 2024-06-11T19:15:45.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.505+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3972530016}) 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2033142363}) 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1445281525}) 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2033142363}) 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2845298245}) 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2033142363}) 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1514850970}) 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2033142363}) 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=647518545}) 2024-06-11T19:15:45.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2033142363}) 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1737134927}) 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2033142363}) 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1425588046}) 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2033142363}) 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.506+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3017688253}) 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094180223}) 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3843115293}) 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094180223}) 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4176343888}) 2024-06-11T19:15:45.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094180223}) 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=450249902}) 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094180223}) 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3291142092}) 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094180223}) 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1903932675}) 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094180223}) 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1821775484}) 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094180223}) 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1210522733}) 2024-06-11T19:15:45.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094180223}) 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.507+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=844780822}) 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.508+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.508+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930817695}) 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.508+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420881432}) 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.508+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.508+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930817695}) 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.508+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2027878023}) 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.508+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930817695}) 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4231304553}) 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930817695}) 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1554458625}) 2024-06-11T19:15:45.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930817695}) 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3645485399}) 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930817695}) 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3577869839}) 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930817695}) 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2606232928}) 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930817695}) 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.509+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1545955304}) 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777171401}) 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007774520}) 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777171401}) 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2018465026}) 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777171401}) 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1868266110}) 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777171401}) 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=594306416}) 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777171401}) 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1087961429}) 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777171401}) 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4167612799}) 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777171401}) 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=472094955}) 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777171401}) 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.510+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2217628445}) 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1895322665}) 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=622778016}) 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1895322665}) 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4171893211}) 2024-06-11T19:15:45.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1895322665}) 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2829110008}) 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1895322665}) 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3609001549}) 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1895322665}) 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2624754550}) 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1895322665}) 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4210059110}) 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1895322665}) 2024-06-11T19:15:45.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4093319323}) 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1895322665}) 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.512+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=946208811}) 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.513+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.513+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603758161}) 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.513+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240495564}) 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603758161}) 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=878210587}) 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603758161}) 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3597459115}) 2024-06-11T19:15:45.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603758161}) 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=496051281}) 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603758161}) 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=543515545}) 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603758161}) 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3932515633}) 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603758161}) 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3893369991}) 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603758161}) 2024-06-11T19:15:45.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.514+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=902353051}) 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074881664}) 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=530943759}) 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074881664}) 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4017708618}) 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074881664}) 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1685232001}) 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074881664}) 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=241763261}) 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074881664}) 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1222718798}) 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074881664}) 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2490705531}) 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074881664}) 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4265662637}) 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074881664}) 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.515+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=639691631}) 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937063689}) 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=222036992}) 2024-06-11T19:15:45.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937063689}) 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1316490676}) 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937063689}) 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3774488652}) 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937063689}) 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=122793963}) 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937063689}) 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2211638102}) 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937063689}) 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1497014655}) 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937063689}) 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2431123208}) 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937063689}) 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.517+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1145147387}) 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590347645}) 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782928593}) 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590347645}) 2024-06-11T19:15:45.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4052457830}) 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590347645}) 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1427513992}) 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590347645}) 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=386743961}) 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590347645}) 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3741930937}) 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590347645}) 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3071624200}) 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590347645}) 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=523894784}) 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590347645}) 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.518+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2730241234}) 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1948580176}) 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574385723}) 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1948580176}) 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987672309}) 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1948580176}) 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=197886896}) 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1948580176}) 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2833063039}) 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1948580176}) 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2764944880}) 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1948580176}) 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3526853112}) 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1948580176}) 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=631494785}) 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1948580176}) 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.520+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1928556233}) 2024-06-11T19:15:45.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=187934498}) 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837040795}) 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=187934498}) 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2108336348}) 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=187934498}) 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3565674983}) 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=187934498}) 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=541979884}) 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=187934498}) 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3335990844}) 2024-06-11T19:15:45.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=187934498}) 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3236492650}) 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=187934498}) 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3933896114}) 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=187934498}) 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.521+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2554673551}) 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79648744}) 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743490184}) 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79648744}) 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3909819271}) 2024-06-11T19:15:45.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79648744}) 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3435826840}) 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79648744}) 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1833863910}) 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79648744}) 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2259827454}) 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79648744}) 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1658873944}) 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79648744}) 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=679264075}) 2024-06-11T19:15:45.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79648744}) 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.523+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1155880896}) 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676999324}) 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303630956}) 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676999324}) 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=48518825}) 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676999324}) 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2159058381}) 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676999324}) 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3250853236}) 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676999324}) 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2253060213}) 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676999324}) 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1404886904}) 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676999324}) 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2309847263}) 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676999324}) 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.524+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=193168688}) 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1090147570}) 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2559419062}) 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1090147570}) 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=469690377}) 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1090147570}) 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3931496626}) 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1090147570}) 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=532403407}) 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1090147570}) 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3493636904}) 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1090147570}) 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3302470761}) 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1090147570}) 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1038558666}) 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1090147570}) 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.526+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2882898650}) 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.527+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.527+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409470436}) 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.527+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784621403}) 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.527+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.527+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409470436}) 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.527+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=391496969}) 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.527+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409470436}) 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=825420024}) 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409470436}) 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3207177778}) 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409470436}) 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3812183087}) 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409470436}) 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=901079450}) 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409470436}) 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2804852966}) 2024-06-11T19:15:45.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409470436}) 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.528+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3057024656}) 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181914935}) 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3766600171}) 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181914935}) 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=652601832}) 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181914935}) 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=133017619}) 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181914935}) 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3247268200}) 2024-06-11T19:15:45.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181914935}) 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3413095809}) 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181914935}) 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4116280775}) 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181914935}) 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3275228875}) 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181914935}) 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.529+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3581179958}) 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.530+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.530+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922667585}) 2024-06-11T19:15:45.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.530+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3830429023}) 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.530+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.530+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922667585}) 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=771473422}) 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922667585}) 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4157956652}) 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922667585}) 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1230530319}) 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922667585}) 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1607273297}) 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922667585}) 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2225976233}) 2024-06-11T19:15:45.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922667585}) 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3395773851}) 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922667585}) 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.531+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3122822049}) 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936552038}) 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=185778790}) 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936552038}) 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2922195862}) 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936552038}) 2024-06-11T19:15:45.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2723537106}) 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936552038}) 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3305132616}) 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936552038}) 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1743751138}) 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936552038}) 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3028102431}) 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936552038}) 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3630363311}) 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936552038}) 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.532+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1842685828}) 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.533+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.533+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698965879}) 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.533+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291048583}) 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.533+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.533+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698965879}) 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.533+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079632518}) 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.533+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.533+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698965879}) 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.533+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3300016781}) 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.533+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.533+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698965879}) 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.533+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=776829161}) 2024-06-11T19:15:45.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.534+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.534+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698965879}) 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.534+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1759184657}) 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.534+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.534+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698965879}) 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.534+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=390041475}) 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.534+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.534+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698965879}) 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.534+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1885530349}) 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.534+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.534+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698965879}) 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.534+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4035858011}) 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179113621}) 2024-06-11T19:15:45.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=871401553}) 2024-06-11T19:15:45.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179113621}) 2024-06-11T19:15:45.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2801687821}) 2024-06-11T19:15:45.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179113621}) 2024-06-11T19:15:45.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3864727691}) 2024-06-11T19:15:45.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179113621}) 2024-06-11T19:15:45.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=802836661}) 2024-06-11T19:15:45.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179113621}) 2024-06-11T19:15:45.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1303540959}) 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179113621}) 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=250582625}) 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179113621}) 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4278746581}) 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179113621}) 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.535+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1444125283}) 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.536+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.536+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784243446}) 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.536+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4153856341}) 2024-06-11T19:15:45.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.536+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784243446}) 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3555197659}) 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784243446}) 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3700930086}) 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784243446}) 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=899971470}) 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784243446}) 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4101717893}) 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784243446}) 2024-06-11T19:15:45.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3795799218}) 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784243446}) 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3302223130}) 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784243446}) 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.537+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1255636011}) 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906002639}) 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1924272631}) 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906002639}) 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1575357706}) 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906002639}) 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2803558974}) 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906002639}) 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2270518145}) 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906002639}) 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=178787492}) 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906002639}) 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=351179048}) 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906002639}) 2024-06-11T19:15:45.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=646808491}) 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906002639}) 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.538+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3116737197}) 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3876779613}) 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355405890}) 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3876779613}) 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2199608982}) 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3876779613}) 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3708684216}) 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3876779613}) 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=526984404}) 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3876779613}) 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3277101462}) 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.539+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.540+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3876779613}) 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.540+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3858686578}) 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.540+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.540+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3876779613}) 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.540+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=99133461}) 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.540+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.540+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3876779613}) 2024-06-11T19:15:45.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.540+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2147371002}) 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105570456}) 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329620043}) 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105570456}) 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1647183953}) 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105570456}) 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1197634528}) 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105570456}) 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=375929198}) 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105570456}) 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2579607458}) 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105570456}) 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1615126836}) 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105570456}) 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=373746294}) 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105570456}) 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.541+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3421863692}) 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859176000}) 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316548427}) 2024-06-11T19:15:45.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859176000}) 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126031529}) 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859176000}) 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2722359922}) 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859176000}) 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3770961216}) 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859176000}) 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=65463458}) 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859176000}) 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2010728408}) 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859176000}) 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1482744686}) 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859176000}) 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.542+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3963130804}) 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540221456}) 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846213657}) 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540221456}) 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3918892871}) 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540221456}) 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2029123805}) 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540221456}) 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1510244061}) 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540221456}) 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1825769327}) 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540221456}) 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=863352688}) 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540221456}) 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3976356327}) 2024-06-11T19:15:45.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540221456}) 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.544+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3689510249}) 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=577437045}) 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=465474200}) 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=577437045}) 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2432295878}) 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=577437045}) 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3955488289}) 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=577437045}) 2024-06-11T19:15:45.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1801436070}) 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=577437045}) 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1358602328}) 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=577437045}) 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=807954931}) 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=577437045}) 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1262754546}) 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.545+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.546+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=577437045}) 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.546+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=369019673}) 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720937757}) 2024-06-11T19:15:45.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009129567}) 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720937757}) 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1115541250}) 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720937757}) 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1822686021}) 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720937757}) 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2427996788}) 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720937757}) 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3318520736}) 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720937757}) 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3764941769}) 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720937757}) 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1357014212}) 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720937757}) 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.547+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4236019580}) 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615184154}) 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074800647}) 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615184154}) 2024-06-11T19:15:45.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2390017996}) 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615184154}) 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2546707661}) 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615184154}) 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1703828884}) 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615184154}) 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2391595429}) 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615184154}) 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3524154158}) 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615184154}) 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2364392840}) 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615184154}) 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.548+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3845227054}) 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3051541135}) 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1401909868}) 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3051541135}) 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4194960961}) 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3051541135}) 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=201206726}) 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3051541135}) 2024-06-11T19:15:45.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2380443211}) 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.549+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.550+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3051541135}) 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.550+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2820764930}) 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.550+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.550+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3051541135}) 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.550+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3139491229}) 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.550+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.550+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3051541135}) 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.550+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2328396989}) 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.550+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.550+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3051541135}) 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.550+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1464556043}) 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972150849}) 2024-06-11T19:15:45.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1246235520}) 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972150849}) 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4277215536}) 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972150849}) 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1476631425}) 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972150849}) 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1149192934}) 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972150849}) 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3861829933}) 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972150849}) 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3640644159}) 2024-06-11T19:15:45.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972150849}) 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=967164039}) 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972150849}) 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.551+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1093330374}) 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651144369}) 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2103479547}) 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651144369}) 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3146732385}) 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651144369}) 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3606024344}) 2024-06-11T19:15:45.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651144369}) 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=887449617}) 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651144369}) 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3918123237}) 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651144369}) 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1564882480}) 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651144369}) 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4006216442}) 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651144369}) 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.552+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=489852294}) 2024-06-11T19:15:45.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.553+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.553+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3514485379}) 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.553+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739059622}) 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.553+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.553+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3514485379}) 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.553+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2808405478}) 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.553+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3514485379}) 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2785978421}) 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3514485379}) 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4171929917}) 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3514485379}) 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3887276041}) 2024-06-11T19:15:45.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3514485379}) 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1230315550}) 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3514485379}) 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1197017628}) 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3514485379}) 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.554+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1742316974}) 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2875884076}) 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299496180}) 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2875884076}) 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4108208057}) 2024-06-11T19:15:45.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2875884076}) 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2908799606}) 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2875884076}) 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=856870799}) 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2875884076}) 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=359594728}) 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2875884076}) 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3342350111}) 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2875884076}) 2024-06-11T19:15:45.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3272775378}) 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2875884076}) 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.555+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2482004073}) 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.556+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.556+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123116275}) 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.556+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141947481}) 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.556+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.556+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123116275}) 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.556+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2023386373}) 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.556+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.556+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123116275}) 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.556+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1180578489}) 2024-06-11T19:15:45.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.556+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.556+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123116275}) 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.556+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3438472707}) 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.557+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.557+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123116275}) 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.557+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=961100207}) 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.557+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.557+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123116275}) 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.557+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=906786417}) 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.557+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.557+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123116275}) 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.557+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3087126878}) 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.557+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.557+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123116275}) 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.557+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=52175083}) 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3017137534}) 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2377215528}) 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3017137534}) 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2160180097}) 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3017137534}) 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1773106704}) 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3017137534}) 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3641282056}) 2024-06-11T19:15:45.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3017137534}) 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=578678689}) 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3017137534}) 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1815176298}) 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3017137534}) 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1632674066}) 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3017137534}) 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.558+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=134825426}) 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842940076}) 2024-06-11T19:15:45.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789872546}) 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842940076}) 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4147033877}) 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842940076}) 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=837329082}) 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842940076}) 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1399710649}) 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842940076}) 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3494105971}) 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842940076}) 2024-06-11T19:15:45.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1822193445}) 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842940076}) 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2098474120}) 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842940076}) 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.559+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=799940895}) 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2937719352,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3622589405}) 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=2} crc {logm=2999023305,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2476767968,mon_config_key=1555703263}) 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2937719352,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3622589405}) 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=81,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=3105897752,mdsmap=2840952586,mgr=3166249245,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2392081758}) 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2937719352,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3622589405}) 2024-06-11T19:15:45.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=1336570773,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=486783348}) 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2937719352,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3622589405}) 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=80,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=1} crc {logm=3294122822,mdsmap=1401145748,mgr=2241844087,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2729711379,mon_config_key=3250354011}) 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2937719352,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3622589405}) 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=950898658,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=315206617}) 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2937719352,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3622589405}) 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=1756028808,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3622589405}) 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2937719352,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3622589405}) 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=2} crc {logm=559987505,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=490215970,mon_config_key=1555703263}) 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2937719352,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3622589405}) 2024-06-11T19:15:45.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.561+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=82,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=3698218299,mdsmap=2840952586,mgr=1965732024,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2392081758}) 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=53} crc {mgrstat=342112659,mon_config_key=486334433,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=3053678937,osdmap=2613985867}) 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=4,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=24,osdmap=52} crc {mgrstat=342112659,mon_config_key=2314039359,monmap=506305176,osd_metadata=371781185,osd_pg_creating=2625009735,osd_snap=1115563319,osdmap=4263092647}) 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=53} crc {mgrstat=342112659,mon_config_key=486334433,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=3053678937,osdmap=2613985867}) 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=4,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=52} crc {mgrstat=342112659,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=2987474761,osdmap=4272701117}) 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=53} crc {mgrstat=342112659,mon_config_key=486334433,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=3053678937,osdmap=2613985867}) 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mgrstat=4,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=51} crc {mgrstat=342112659,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=1759306115,osdmap=2798731769}) 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=53} crc {mgrstat=342112659,mon_config_key=486334433,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=3053678937,osdmap=2613985867}) 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {mgrstat=4,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=52} crc {mgrstat=342112659,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=3170975072,osdmap=2253581847}) 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=53} crc {mgrstat=342112659,mon_config_key=486334433,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=3053678937,osdmap=2613985867}) 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {mgrstat=4,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=24,osdmap=52} crc {mgrstat=342112660,mon_config_key=2314039359,monmap=506305176,osd_metadata=1424309990,osd_pg_creating=2625009735,osd_snap=3149892819,osdmap=3653083014}) 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=53} crc {mgrstat=342112659,mon_config_key=486334433,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=3053678937,osdmap=2613985867}) 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {mgrstat=4,mon_config_key=9,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=24,osdmap=54} crc {mgrstat=342112659,mon_config_key=1498955399,monmap=506305176,osd_metadata=371781185,osd_pg_creating=2625009735,osd_snap=467760934,osdmap=3055700683}) 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=53} crc {mgrstat=342112659,mon_config_key=486334433,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=3053678937,osdmap=2613985867}) 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mgrstat=3,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=53} crc {mgrstat=2122954184,mon_config_key=3837142432,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=3462785741,osdmap=3869352061}) 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=53} crc {mgrstat=342112659,mon_config_key=486334433,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=3053678937,osdmap=2613985867}) 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.563+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mgrstat=4,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=51} crc {mgrstat=342112659,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2625009735,osd_snap=1684536126,osdmap=676976579}) 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2058637916}) 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=620470677}) 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2058637916}) 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=780311942}) 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2058637916}) 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=604588925}) 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2058637916}) 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3054599742}) 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2058637916}) 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1551200485}) 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2058637916}) 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2813345275}) 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2058637916}) 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1753912106}) 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2058637916}) 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.565+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=783210331}) 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3106948130}) 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=178016786}) 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3106948130}) 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3262261376}) 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3106948130}) 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1422960601}) 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3106948130}) 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1512896946}) 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3106948130}) 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=875012713}) 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3106948130}) 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=932957724}) 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3106948130}) 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3675563460}) 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3106948130}) 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.567+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=330786902}) 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.569+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1640384790}) 2024-06-11T19:15:45.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2292074565}) 2024-06-11T19:15:45.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1640384790}) 2024-06-11T19:15:45.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2480056448}) 2024-06-11T19:15:45.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1640384790}) 2024-06-11T19:15:45.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3673022104}) 2024-06-11T19:15:45.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1640384790}) 2024-06-11T19:15:45.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2715070811}) 2024-06-11T19:15:45.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1640384790}) 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3791543982}) 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1640384790}) 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=4260586170}) 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1640384790}) 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=363089089}) 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1640384790}) 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.570+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2378747645}) 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.578+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=596615011}) 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=69} crc {osdmap=1406154979}) 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=596615011}) 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=70} crc {osdmap=2761279973}) 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=596615011}) 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=70} crc {osdmap=10889193}) 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=596615011}) 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=67} crc {osdmap=1383368061}) 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=596615011}) 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=70} crc {osdmap=1937091617}) 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=596615011}) 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=67} crc {osdmap=4235955553}) 2024-06-11T19:15:45.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=596615011}) 2024-06-11T19:15:45.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=71} crc {osdmap=4043170057}) 2024-06-11T19:15:45.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:15:45.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=596615011}) 2024-06-11T19:15:45.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:15:45.579+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=69} crc {osdmap=851046571}) 2024-06-11T19:15:46.511 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:15:46.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:15:46.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.612+0000 7f936da2d640 10 monclient: get_monmap_and_config 2024-06-11T19:15:46.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.612+0000 7f936da2d640 10 monclient: build_initial_monmap 2024-06-11T19:15:46.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.613+0000 7f936da2d640 10 monclient: monmap: 2024-06-11T19:15:46.612 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:46.612 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:46.612 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:46.614005+0000 2024-06-11T19:15:46.612 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:46.614005+0000 2024-06-11T19:15:46.612 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:46.612 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:46.612 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:46.613 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:46.613 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:46.613 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:46.613 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:46.613 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:46.613 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:46.613 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:46.613 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:46.613 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:46.613 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:46.613 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:46.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.613+0000 7f936da2d640 1 Processor -- start 2024-06-11T19:15:46.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.613+0000 7f936da2d640 1 -- start start 2024-06-11T19:15:46.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.613+0000 7f936da2d640 10 monclient: init 2024-06-11T19:15:46.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:46.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 10 monclient: _add_conns ranks=[3,5,9,1,2,7,0,4,6,10,8] 2024-06-11T19:15:46.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93680a4920 0x7f93680a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:46.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 10 monclient(hunting): picked mon.noname-g con 0x7f93680a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:46.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f93680a5240 0x7f93680a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 10 monclient(hunting): picked mon.noname-h con 0x7f93680a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93680a2530 0x7f93680a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 10 monclient(hunting): picked mon.noname-a con 0x7f93680a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f93680a7140 con 0x7f93680a2530 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f93680a2b50 con 0x7f93680a4920 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f93680a2e30 con 0x7f93680a5240 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f936da2d640 10 monclient(hunting): authenticate will time out at 1156.421122s 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93680a2530 0x7f93680a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93680a2530 0x7f93680a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46320/0 (socket says 172.21.15.98:46320) 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 1 -- 172.21.15.98:0/3206709317 learned_addr learned my addr 172.21.15.98:0/3206709317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:46.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 10 monclient(hunting): get_auth_request con 0x7f93680a2530 auth_method 0 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f9366ffd640 1 --2- 172.21.15.98:0/3206709317 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93680a4920 0x7f93680a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93667fc640 1 --2- 172.21.15.98:0/3206709317 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f93680a5240 0x7f93680a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f9366ffd640 10 monclient(hunting): get_auth_request con 0x7f93680a4920 auth_method 0 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f9366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f9366ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f9366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93667fc640 10 monclient(hunting): get_auth_request con 0x7f93680a5240 auth_method 0 2024-06-11T19:15:46.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93667fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93667fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f9366ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f9366ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f9366ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 10 monclient(hunting): handle_auth_done global_id 64611 payload 274 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 1 -- 172.21.15.98:0/3206709317 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f93680a5240 msgr2=0x7f93680a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 1 --2- 172.21.15.98:0/3206709317 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f93680a5240 0x7f93680a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 1 -- 172.21.15.98:0/3206709317 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93680a4920 msgr2=0x7f93680a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 1 --2- 172.21.15.98:0/3206709317 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93680a4920 0x7f93680a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 10 monclient: _finish_hunting 0 2024-06-11T19:15:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 1 monclient: found mon.noname-a 2024-06-11T19:15:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:15:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.614+0000 7f93677fe640 1 -- 172.21.15.98:0/3206709317 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93680a3110 con 0x7f93680a2530 2024-06-11T19:15:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f93677fe640 1 --2- 172.21.15.98:0/3206709317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93680a2530 0x7f93680a5ee0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9360009e50 tx=0x7f936002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5c2208560ce3d6 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 1 -- 172.21.15.98:0/3206709317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f936002cbc0 con 0x7f93680a2530 2024-06-11T19:15:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:15:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 10 monclient: dump: 2024-06-11T19:15:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 10 monclient: _finish_auth 0 2024-06-11T19:15:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 1 -- 172.21.15.98:0/3206709317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f936002cd80 con 0x7f93680a2530 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f936da2d640 5 monclient: authenticate success, global_id 64611 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f936da2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f936da2d640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 1 -- 172.21.15.98:0/3206709317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9360034860 con 0x7f93680a2530 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f936da2d640 4 set_mon_vals no callback set 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.615+0000 7f9365ffb640 10 monclient: dump: 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 10 monclient: shutdown 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 1 -- 172.21.15.98:0/3206709317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93680a2530 msgr2=0x7f93680a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 1 --2- 172.21.15.98:0/3206709317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93680a2530 0x7f93680a5ee0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9360009e50 tx=0x7f936002bdb0 comp rx=0 tx=0).stop 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 1 -- 172.21.15.98:0/3206709317 shutdown_connections 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 1 --2- 172.21.15.98:0/3206709317 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f93680a5240 0x7f93680a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 1 --2- 172.21.15.98:0/3206709317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93680a2530 0x7f93680a5ee0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 1 --2- 172.21.15.98:0/3206709317 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93680a4920 0x7f93680a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 1 -- 172.21.15.98:0/3206709317 >> 172.21.15.98:0/3206709317 conn(0x7f93680ab850 msgr2=0x7f93680a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 1 -- 172.21.15.98:0/3206709317 shutdown_connections 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 1 -- 172.21.15.98:0/3206709317 wait complete. 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 10 monclient: build_initial_monmap 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 10 monclient: monmap: 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 1 librados: starting msgr at 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 1 librados: starting objecter 2024-06-11T19:15:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.616+0000 7f936da2d640 1 Processor -- start 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 1 -- start start 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 1 librados: setting wanted keys 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 1 librados: calling monclient init 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 10 monclient: init 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 10 monclient: _add_conns ranks=[8,5,2,6,3,1,4,0,7] 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93680a2530 0x7f93681cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 10 monclient(hunting): picked mon.noname-e con 0x7f93680a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f93680a4920 0x7f93681ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 10 monclient(hunting): picked mon.noname-h con 0x7f93680a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93680a5240 0x7f9368124ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 10 monclient(hunting): picked mon.noname-b con 0x7f93680a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f93680a6da0 con 0x7f93680a5240 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f9368138e50 con 0x7f93680a4920 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f93680a6420 con 0x7f93680a2530 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f936da2d640 10 monclient(hunting): authenticate will time out at 1156.424627s 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f93677fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93680a5240 0x7f9368124ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f9366ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93680a2530 0x7f93681cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f9366ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93680a2530 0x7f93681cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34848/0 (socket says 172.21.15.98:34848) 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f9366ffd640 1 -- 172.21.15.98:0/3454758230 learned_addr learned my addr 172.21.15.98:0/3454758230 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f9366ffd640 10 monclient(hunting): get_auth_request con 0x7f93680a2530 auth_method 0 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f9366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f9366ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f9366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f93677fe640 10 monclient(hunting): get_auth_request con 0x7f93680a5240 auth_method 0 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f93677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f93677fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f93677fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f9366ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f9366ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f9366ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f93677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f93677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f93677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f93677fe640 10 monclient(hunting): handle_auth_done global_id 84170 payload 824 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.617+0000 7f93667fc640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f93680a4920 0x7f93681ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f93677fe640 1 -- 172.21.15.98:0/3454758230 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93680a2530 msgr2=0x7f93681cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f93677fe640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93680a2530 0x7f93681cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f93677fe640 1 -- 172.21.15.98:0/3454758230 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f93680a4920 msgr2=0x7f93681ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f93677fe640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f93680a4920 0x7f93681ce080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f93677fe640 10 monclient: _finish_hunting 0 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f93677fe640 1 monclient: found mon.noname-b 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f93677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f93677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f93677fe640 1 -- 172.21.15.98:0/3454758230 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9368129860 con 0x7f93680a5240 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f9366ffd640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93680a2530 0x7f93681cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f93677fe640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93680a5240 0x7f9368124ec0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9360032000 tx=0x7f9360004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 10 client.?.objecter ms_handle_connect 0x7f93680a5240 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 1 -- 172.21.15.98:0/3454758230 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9360042020 con 0x7f93680a5240 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 10 monclient: dump: 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 10 monclient: _finish_auth 0 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 1 -- 172.21.15.98:0/3454758230 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93600046c0 con 0x7f93680a5240 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f936da2d640 5 monclient: authenticate success, global_id 84170 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f936da2d640 10 monclient: _renew_subs 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f936da2d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f936da2d640 1 -- 172.21.15.98:0/3454758230 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9368128ba0 con 0x7f93680a5240 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 1 -- 172.21.15.98:0/3454758230 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f936003e8c0 con 0x7f93680a5240 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f934ffff640 10 monclient: dump: 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f936da2d640 10 client.84170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f93657fa640 4 set_mon_vals no callback set 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f936da2d640 10 monclient: _renew_subs 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f936da2d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f936da2d640 1 -- 172.21.15.98:0/3454758230 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9368128e80 con 0x7f93680a5240 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.618+0000 7f936da2d640 1 librados: init done 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f934ffff640 1 -- 172.21.15.98:0/3454758230 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f9360045020 con 0x7f93680a5240 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f936da2d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f936da2d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f936da2d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f936da2d640 1 -- 172.21.15.98:0/3454758230 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9368126be0 con 0x7f93680a5240 2024-06-11T19:15:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f934ffff640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f935403fdb0 0x7f9354042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f934ffff640 1 -- 172.21.15.98:0/3454758230 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(288..288 src has 1..288) ==== 6355+0+0 (secure 0 0 0) 0x7f936003a040 con 0x7f93680a5240 2024-06-11T19:15:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f934ffff640 10 client.84170.objecter ms_dispatch 0x7f9368000b90 osd_map(288..288 src has 1..288) 2024-06-11T19:15:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f934ffff640 3 client.84170.objecter handle_osd_map got epochs [288,288] > 0 2024-06-11T19:15:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f934ffff640 3 client.84170.objecter handle_osd_map decoding full epoch 288 2024-06-11T19:15:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f934ffff640 20 client.84170.objecter dump_active .. 0 homeless 2024-06-11T19:15:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f9366ffd640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f935403fdb0 0x7f9354042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.619+0000 7f9366ffd640 10 monclient: get_auth_request con 0x7f935403fdb0 auth_method 0 2024-06-11T19:15:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.620+0000 7f934ffff640 10 client.84170.objecter ms_handle_connect 0x7f935403fdb0 2024-06-11T19:15:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.620+0000 7f9366ffd640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f935403fdb0 0x7f9354042270 secure :-1 s=READY pgs=4131 cs=0 l=1 rev1=1 crypto rx=0x7f93580112c0 tx=0x7f9358009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.624+0000 7f934ffff640 1 -- 172.21.15.98:0/3454758230 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f936004c070 con 0x7f93680a5240 2024-06-11T19:15:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.624+0000 7f934ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.624+0000 7f934ffff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:46.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.877+0000 7f936da2d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:15:46.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.877+0000 7f936da2d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:46.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.877+0000 7f936da2d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:46.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.877+0000 7f936da2d640 1 -- 172.21.15.98:0/3454758230 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f93681387f0 con 0x7f93680a5240 2024-06-11T19:15:46.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.880+0000 7f934ffff640 1 -- 172.21.15.98:0/3454758230 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f9368129860 con 0x7f93680a5240 2024-06-11T19:15:46.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.880+0000 7f934ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:46.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.880+0000 7f934ffff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:46.880 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":30,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:15:46.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 10 librados: watch_flush enter 2024-06-11T19:15:46.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 10 librados: watch_flush exit 2024-06-11T19:15:46.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 20 client.84170.objecter shutdown clearing up homeless session... 2024-06-11T19:15:46.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 10 client.84170.objecter successfully canceled tick 2024-06-11T19:15:46.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 -- 172.21.15.98:0/3454758230 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f935403fdb0 msgr2=0x7f9354042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f935403fdb0 0x7f9354042270 secure :-1 s=READY pgs=4131 cs=0 l=1 rev1=1 crypto rx=0x7f93580112c0 tx=0x7f9358009190 comp rx=0 tx=0).stop 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 10 monclient: shutdown 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 -- 172.21.15.98:0/3454758230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93680a5240 msgr2=0x7f9368124ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93680a5240 0x7f9368124ec0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9360032000 tx=0x7f9360004a40 comp rx=0 tx=0).stop 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 -- 172.21.15.98:0/3454758230 shutdown_connections 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f935403fdb0 0x7f9354042270 unknown :-1 s=CLOSED pgs=4131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f93680a5240 0x7f9368124ec0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f93680a4920 0x7f93681ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 --2- 172.21.15.98:0/3454758230 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f93680a2530 0x7f93681cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 -- 172.21.15.98:0/3454758230 >> 172.21.15.98:0/3454758230 conn(0x7f93680ab850 msgr2=0x7f936809c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 -- 172.21.15.98:0/3454758230 shutdown_connections 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 -- 172.21.15.98:0/3454758230 wait complete. 2024-06-11T19:15:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.883+0000 7f936da2d640 1 librados: shutdown 2024-06-11T19:15:46.895 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:15:46.895 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-06-11T19:15:46.895 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:15:46.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:15:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.994+0000 7febf910b640 10 monclient: get_monmap_and_config 2024-06-11T19:15:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.994+0000 7febf910b640 10 monclient: build_initial_monmap 2024-06-11T19:15:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.994+0000 7febf910b640 10 monclient: monmap: 2024-06-11T19:15:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:46.995864+0000 2024-06-11T19:15:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:46.995864+0000 2024-06-11T19:15:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:46.995 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:46.995 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 1 Processor -- start 2024-06-11T19:15:47.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 1 -- start start 2024-06-11T19:15:47.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 10 monclient: init 2024-06-11T19:15:47.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:47.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 10 monclient: _add_conns ranks=[10,1,6,7,8,4,0,9,3,2,5] 2024-06-11T19:15:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7febf40a4920 0x7febf40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 10 monclient(hunting): picked mon.noname-f con 0x7febf40a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:15:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7febf40a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7febf40a2530 0x7febf40a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 10 monclient(hunting): picked mon.noname-d con 0x7febf40a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febf40a7140 con 0x7febf40a5240 2024-06-11T19:15:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7febf40a2b50 con 0x7febf40a4920 2024-06-11T19:15:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7febf40a2e30 con 0x7febf40a2530 2024-06-11T19:15:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.995+0000 7febf910b640 10 monclient(hunting): authenticate will time out at 1156.802954s 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7febf40a2530 0x7febf40a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7febf40a2530 0x7febf40a60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34634/0 (socket says 172.21.15.98:34634) 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 1 -- 172.21.15.98:0/2994310631 learned_addr learned my addr 172.21.15.98:0/2994310631 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 10 monclient(hunting): get_auth_request con 0x7febf40a2530 auth_method 0 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf2d9d640 1 --2- 172.21.15.98:0/2994310631 >> v2:172.21.15.117:3300/0 conn(0x7febf40a4920 0x7febf40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf2d9d640 10 monclient(hunting): get_auth_request con 0x7febf40a4920 auth_method 0 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf2d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 10 monclient(hunting): handle_auth_done global_id 54906 payload 274 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 1 -- 172.21.15.98:0/2994310631 >> v2:172.21.15.117:3300/0 conn(0x7febf40a4920 msgr2=0x7febf40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 1 --2- 172.21.15.98:0/2994310631 >> v2:172.21.15.117:3300/0 conn(0x7febf40a4920 0x7febf40a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 1 -- 172.21.15.98:0/2994310631 >> v1:172.21.15.117:6789/0 conn(0x7febf40a5240 legacy=0x7febf40a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 10 monclient: _finish_hunting 0 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 1 monclient: found mon.noname-d 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.996+0000 7febf359e640 1 -- 172.21.15.98:0/2994310631 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf40a3110 con 0x7febf40a2530 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf359e640 1 --2- 172.21.15.98:0/2994310631 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7febf40a2530 0x7febf40a60b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7febe4009870 tx=0x7febe402b860 comp rx=0 tx=0).ready entity=mon.6 client_cookie=a670df785efdb89a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 1 -- 172.21.15.98:0/2994310631 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7febe4031a30 con 0x7febf40a2530 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 10 monclient: dump: 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.010 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.010 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.010 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.010 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 10 monclient: _finish_auth 0 2024-06-11T19:15:47.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:15:47.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:47.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 1 -- 172.21.15.98:0/2994310631 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febe4037070 con 0x7febf40a2530 2024-06-11T19:15:47.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 5 monclient: authenticate success, global_id 54906 2024-06-11T19:15:47.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:47.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:47.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 1 -- 172.21.15.98:0/2994310631 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7febe4031c30 con 0x7febf40a2530 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 4 set_mon_vals no callback set 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf1d9b640 10 monclient: dump: 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 10 monclient: shutdown 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 1 -- 172.21.15.98:0/2994310631 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7febf40a2530 msgr2=0x7febf40a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 1 --2- 172.21.15.98:0/2994310631 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7febf40a2530 0x7febf40a60b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7febe4009870 tx=0x7febe402b860 comp rx=0 tx=0).stop 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 1 -- 172.21.15.98:0/2994310631 shutdown_connections 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 1 --2- 172.21.15.98:0/2994310631 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7febf40a2530 0x7febf40a60b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 1 --2- 172.21.15.98:0/2994310631 >> v2:172.21.15.117:3300/0 conn(0x7febf40a4920 0x7febf40a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 1 -- 172.21.15.98:0/2994310631 >> 172.21.15.98:0/2994310631 conn(0x7febf40ab850 msgr2=0x7febf40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 1 -- 172.21.15.98:0/2994310631 shutdown_connections 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.997+0000 7febf910b640 1 -- 172.21.15.98:0/2994310631 wait complete. 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.998+0000 7febf910b640 10 monclient: build_initial_monmap 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.998+0000 7febf910b640 10 monclient: monmap: 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:47.013 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.998+0000 7febf910b640 1 librados: starting msgr at 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.998+0000 7febf910b640 1 librados: starting objecter 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 1 Processor -- start 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 1 -- start start 2024-06-11T19:15:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 1 librados: setting wanted keys 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 1 librados: calling monclient init 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 10 monclient: init 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 10 monclient: _add_conns ranks=[5,2,6,1,8,0,3,4,7] 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7febf40a2530 0x7febf4137680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 10 monclient(hunting): picked mon.noname-h con 0x7febf40a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febf40a4920 0x7febf4137bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 10 monclient(hunting): picked mon.noname-b con 0x7febf40a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7febf40a5240 0x7febf41230a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 10 monclient(hunting): picked mon.noname-d con 0x7febf40a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7febf40a6da0 con 0x7febf40a4920 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7febf4138e50 con 0x7febf40a2530 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7febf40a65f0 con 0x7febf40a5240 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf910b640 10 monclient(hunting): authenticate will time out at 1156.806617s 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febf40a4920 0x7febf4137bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf359e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7febf40a5240 0x7febf41230a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febf40a4920 0x7febf4137bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:59864/0 (socket says 172.21.15.98:59864) 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 1 -- 172.21.15.98:0/1716979746 learned_addr learned my addr 172.21.15.98:0/1716979746 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 10 monclient(hunting): get_auth_request con 0x7febf40a4920 auth_method 0 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf359e640 10 monclient(hunting): get_auth_request con 0x7febf40a5240 auth_method 0 2024-06-11T19:15:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf359e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf359e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf2d9d640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7febf40a2530 0x7febf4137680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf2d9d640 10 monclient(hunting): get_auth_request con 0x7febf40a2530 auth_method 0 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf2d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 10 monclient(hunting): handle_auth_done global_id 84197 payload 824 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 1 -- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7febf40a5240 msgr2=0x7febf41230a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7febf40a5240 0x7febf41230a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 1 -- 172.21.15.98:0/1716979746 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7febf40a2530 msgr2=0x7febf4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7febf40a2530 0x7febf4137680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:46.999+0000 7febf259c640 10 monclient: _finish_hunting 0 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf259c640 1 monclient: found mon.noname-b 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf259c640 1 -- 172.21.15.98:0/1716979746 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf4126aa0 con 0x7febf40a4920 2024-06-11T19:15:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf359e640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7febf40a5240 0x7febf41230a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf259c640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febf40a4920 0x7febf4137bc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7febdc00c3d0 tx=0x7febdc00c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 10 client.?.objecter ms_handle_connect 0x7febf40a4920 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7febdc017070 con 0x7febf40a4920 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 10 monclient: dump: 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 10 monclient: _finish_auth 0 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:47.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febdc0035f0 con 0x7febf40a4920 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf910b640 5 monclient: authenticate success, global_id 84197 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf910b640 10 monclient: _renew_subs 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf910b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf910b640 1 -- 172.21.15.98:0/1716979746 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf40a6310 con 0x7febf40a4920 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7febdc0010e0 con 0x7febf40a4920 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf159a640 4 set_mon_vals no callback set 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf910b640 10 client.84197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febeb7fe640 10 monclient: dump: 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf910b640 10 monclient: _renew_subs 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf910b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf910b640 1 -- 172.21.15.98:0/1716979746 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7febf4124d70 con 0x7febf40a4920 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.000+0000 7febf910b640 1 librados: init done 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf910b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf910b640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4123da0 0x7febf402cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf910b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf910b640 1 -- 172.21.15.98:0/1716979746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7febf4050830 con 0x7febf4123da0 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7febdc0050d0 con 0x7febf40a4920 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf2d9d640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4123da0 0x7febf402cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf2d9d640 10 monclient: get_auth_request con 0x7febf4123da0 auth_method 0 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf2d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7febd403fdb0 0x7febd4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf2d9d640 10 monclient(hunting): handle_auth_done global_id 64629 payload 824 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(289..289 src has 1..289) ==== 6248+0+0 (secure 0 0 0) 0x7febdc050ca0 con 0x7febf40a4920 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 10 client.84197.objecter ms_dispatch 0x7febf4000b90 osd_map(289..289 src has 1..289) 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 3 client.84197.objecter handle_osd_map got epochs [289,289] > 0 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 3 client.84197.objecter handle_osd_map decoding full epoch 289 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf2d9d640 1 -- 172.21.15.98:0/1716979746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7febf41267c0 con 0x7febf4123da0 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 20 client.84197.objecter dump_active .. 0 homeless 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf359e640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7febd403fdb0 0x7febd4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf359e640 10 monclient: get_auth_request con 0x7febd403fdb0 auth_method 0 2024-06-11T19:15:47.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 10 client.84197.objecter ms_handle_connect 0x7febf4123da0 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 10 client.84197.objecter resend_mon_ops 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febf2d9d640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4123da0 0x7febf402cb40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7febec004680 tx=0x7febec007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7febec015890 con 0x7febf4123da0 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.001+0000 7febeb7fe640 10 monclient: dump: 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.025 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.002+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7febec01a070 con 0x7febf4123da0 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.002+0000 7febeb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.002+0000 7febeb7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.002+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4123da0 msgr2=0x7febf402cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.002+0000 7febeb7fe640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4123da0 0x7febf402cb40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7febec004680 tx=0x7febec007700 comp rx=0 tx=0).stop 2024-06-11T19:15:47.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.002+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4123da0 msgr2=0x7febf402cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:47.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.002+0000 7febeb7fe640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4123da0 0x7febf402cb40 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7febec004680 tx=0x7febec007700 comp rx=0 tx=0).stop 2024-06-11T19:15:47.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.002+0000 7febf359e640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7febd403fdb0 0x7febd4042270 secure :-1 s=READY pgs=4133 cs=0 l=1 rev1=1 crypto rx=0x7febe4036000 tx=0x7febe403d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.002+0000 7febeb7fe640 10 client.84197.objecter ms_handle_connect 0x7febd403fdb0 2024-06-11T19:15:47.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf910b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf910b640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4124180 0x7febf402d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf910b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf910b640 1 -- 172.21.15.98:0/1716979746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7febf40a4d00 con 0x7febf4124180 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf2d9d640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4124180 0x7febf402d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf2d9d640 10 monclient: get_auth_request con 0x7febf4124180 auth_method 0 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf2d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf2d9d640 10 monclient(hunting): handle_auth_done global_id 64638 payload 824 2024-06-11T19:15:47.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf2d9d640 1 -- 172.21.15.98:0/1716979746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7febf40a0100 con 0x7febf4124180 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febeb7fe640 10 client.84197.objecter ms_handle_connect 0x7febf4124180 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.008+0000 7febf2d9d640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4124180 0x7febf402d080 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7febec004680 tx=0x7febec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febeb7fe640 10 client.84197.objecter resend_mon_ops 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7febec015bf0 con 0x7febf4124180 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febeb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febeb7fe640 10 monclient: dump: 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7febec015780 con 0x7febf4124180 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febeb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febeb7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:47.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4124180 msgr2=0x7febf402d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febeb7fe640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4124180 0x7febf402d080 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7febec004680 tx=0x7febec004a40 comp rx=0 tx=0).stop 2024-06-11T19:15:47.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febeb7fe640 1 -- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4124180 msgr2=0x7febf402d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:47.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febeb7fe640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7febf4124180 0x7febf402d080 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7febec004680 tx=0x7febec004a40 comp rx=0 tx=0).stop 2024-06-11T19:15:47.032 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:47.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:47.032 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:47.032 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:47.032 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:47.032 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 138837, 2024-06-11T19:15:47.032 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:47.033 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.034 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.035 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:47.036 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.039 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:47.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.042 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.044 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.046 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.047 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.049 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.051 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.052 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.053 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:47.054 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:47.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 10 librados: watch_flush enter 2024-06-11T19:15:47.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 10 librados: watch_flush exit 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 20 client.84197.objecter shutdown clearing up homeless session... 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 10 client.84197.objecter successfully canceled tick 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 1 -- 172.21.15.98:0/1716979746 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7febd403fdb0 msgr2=0x7febd4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7febd403fdb0 0x7febd4042270 secure :-1 s=READY pgs=4133 cs=0 l=1 rev1=1 crypto rx=0x7febe4036000 tx=0x7febe403d040 comp rx=0 tx=0).stop 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 10 monclient: shutdown 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 1 -- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febf40a4920 msgr2=0x7febf4137bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febf40a4920 0x7febf4137bc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7febdc00c3d0 tx=0x7febdc00c890 comp rx=0 tx=0).stop 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf359e640 1 -- 172.21.15.98:0/1716979746 reap_dead start 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 1 -- 172.21.15.98:0/1716979746 shutdown_connections 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 1 --2- 172.21.15.98:0/1716979746 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7febf40a4920 0x7febf4137bc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 1 -- 172.21.15.98:0/1716979746 >> 172.21.15.98:0/1716979746 conn(0x7febf40ab850 msgr2=0x7febf41beed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 1 -- 172.21.15.98:0/1716979746 shutdown_connections 2024-06-11T19:15:47.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 1 -- 172.21.15.98:0/1716979746 wait complete. 2024-06-11T19:15:47.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.009+0000 7febf910b640 1 librados: shutdown 2024-06-11T19:15:47.058 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:47.058 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:15:47.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:15:47.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.158+0000 7f56a91a8640 10 monclient: get_monmap_and_config 2024-06-11T19:15:47.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.158+0000 7f56a91a8640 10 monclient: build_initial_monmap 2024-06-11T19:15:47.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.158+0000 7f56a91a8640 10 monclient: monmap: 2024-06-11T19:15:47.157 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:47.157 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.157 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:47.159099+0000 2024-06-11T19:15:47.157 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:47.159099+0000 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:47.158 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.158+0000 7f56a91a8640 1 Processor -- start 2024-06-11T19:15:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.158+0000 7f56a91a8640 1 -- start start 2024-06-11T19:15:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.158+0000 7f56a91a8640 10 monclient: init 2024-06-11T19:15:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 10 monclient: _add_conns ranks=[0,2,10,4,5,3,9,1,7,6,8] 2024-06-11T19:15:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56a40a6260 addr v1:172.21.15.98:6789/0 2024-06-11T19:15:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a406c890 0x7f56a413da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 10 monclient(hunting): picked mon.noname-b con 0x7f56a406c890 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f56a40a2d00 0x7f56a413ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 10 monclient(hunting): picked mon.noname-f con 0x7f56a40a2d00 addr v2:172.21.15.117:3300/0 2024-06-11T19:15:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a40a2460 con 0x7f56a40a6260 2024-06-11T19:15:47.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f56a40a05b0 con 0x7f56a40a2d00 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f56a40a6640 con 0x7f56a406c890 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a91a8640 10 monclient(hunting): authenticate will time out at 1156.966318s 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a2d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f56a40a6260 0x7f56a40a1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35732/0 (socket says 172.21.15.98:35732) 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a2d9d640 1 -- 172.21.15.98:0/1578387560 learned_addr learned my addr 172.21.15.98:0/1578387560 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 1 --2- 172.21.15.98:0/1578387560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a406c890 0x7f56a413da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 10 monclient(hunting): get_auth_request con 0x7f56a406c890 auth_method 0 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 10 monclient(hunting): handle_auth_done global_id 84206 payload 274 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 1 -- 172.21.15.98:0/1578387560 >> v2:172.21.15.117:3300/0 conn(0x7f56a40a2d00 msgr2=0x7f56a413ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a359e640 1 --2- 172.21.15.98:0/1578387560 >> v2:172.21.15.117:3300/0 conn(0x7f56a40a2d00 0x7f56a413ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a259c640 1 -- 172.21.15.98:0/1578387560 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 169879687 0 0) 0x7f56a40a2460 con 0x7f56a40a6260 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 1 --2- 172.21.15.98:0/1578387560 >> v2:172.21.15.117:3300/0 conn(0x7f56a40a2d00 0x7f56a413ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 1 -- 172.21.15.98:0/1578387560 >> v1:172.21.15.98:6789/0 conn(0x7f56a40a6260 legacy=0x7f56a40a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 10 monclient: _finish_hunting 0 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 1 monclient: found mon.noname-b 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f569a59c640 1 -- 172.21.15.98:0/1578387560 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a406c650 con 0x7f56a406c890 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a259c640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:15:47.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a359e640 10 monclient: get_auth_request con 0x7f56a40a2d00 auth_method 0 2024-06-11T19:15:47.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.159+0000 7f56a359e640 1 --2- 172.21.15.98:0/1578387560 >> v2:172.21.15.117:3300/0 conn(0x7f56a40a2d00 0x7f56a413ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:15:47.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f569a59c640 1 --2- 172.21.15.98:0/1578387560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a406c890 0x7f56a413da80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f568c0097a0 tx=0x7f568c02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24ebb486c983ab8c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 1 -- 172.21.15.98:0/1578387560 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f568c038070 con 0x7f56a406c890 2024-06-11T19:15:47.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:15:47.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 10 monclient: dump: 2024-06-11T19:15:47.171 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.171 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.173 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.173 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.173 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.173 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.173 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.173 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.173 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.173 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.173 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 10 monclient: _finish_auth 0 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 10 monclient: _check_auth_tickets 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 1 -- 172.21.15.98:0/1578387560 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f568c031970 con 0x7f56a406c890 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a91a8640 5 monclient: authenticate success, global_id 84206 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a91a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:47.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a91a8640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 1 -- 172.21.15.98:0/1578387560 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f568c032390 con 0x7f56a406c890 2024-06-11T19:15:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a91a8640 4 set_mon_vals no callback set 2024-06-11T19:15:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.160+0000 7f56a259c640 10 monclient: dump: 2024-06-11T19:15:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.175 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.161+0000 7f56a91a8640 10 monclient: shutdown 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.161+0000 7f56a91a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.161+0000 7f56a91a8640 1 -- 172.21.15.98:0/1578387560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a406c890 msgr2=0x7f56a413da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.161+0000 7f56a91a8640 1 --2- 172.21.15.98:0/1578387560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a406c890 0x7f56a413da80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f568c0097a0 tx=0x7f568c02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.161+0000 7f56a91a8640 1 -- 172.21.15.98:0/1578387560 shutdown_connections 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.161+0000 7f56a91a8640 1 --2- 172.21.15.98:0/1578387560 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a406c890 0x7f56a413da80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.161+0000 7f56a91a8640 1 --2- 172.21.15.98:0/1578387560 >> v2:172.21.15.117:3300/0 conn(0x7f56a40a2d00 0x7f56a413ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.161+0000 7f56a91a8640 1 -- 172.21.15.98:0/1578387560 >> 172.21.15.98:0/1578387560 conn(0x7f56a40a95b0 msgr2=0x7f56a40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.161+0000 7f56a91a8640 1 -- 172.21.15.98:0/1578387560 shutdown_connections 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.161+0000 7f56a91a8640 1 -- 172.21.15.98:0/1578387560 wait complete. 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.161+0000 7f56a91a8640 10 monclient: build_initial_monmap 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.161+0000 7f56a91a8640 10 monclient: monmap: 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:47.177 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 1 librados: starting msgr at 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 1 librados: starting objecter 2024-06-11T19:15:47.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 1 Processor -- start 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 1 -- start start 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 1 librados: setting wanted keys 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 1 librados: calling monclient init 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 10 monclient: init 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 10 monclient: _add_conns ranks=[3,6,5,7,8,4,2,0,1] 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56a406c890 0x7f56a409ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 10 monclient(hunting): picked mon.noname-g con 0x7f56a406c890 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f56a40a2d00 0x7f56a4138d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 10 monclient(hunting): picked mon.noname-d con 0x7f56a40a2d00 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f56a40a6260 0x7f56a41cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 10 monclient(hunting): picked mon.noname-h con 0x7f56a40a6260 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:47.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f56a4140510 con 0x7f56a406c890 2024-06-11T19:15:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f56a406cd10 con 0x7f56a40a6260 2024-06-11T19:15:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f56a4140690 con 0x7f56a40a2d00 2024-06-11T19:15:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f56a91a8640 10 monclient(hunting): authenticate will time out at 1156.969819s 2024-06-11T19:15:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f569a59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f56a40a2d00 0x7f56a4138d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f569a59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f56a40a2d00 0x7f56a4138d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34654/0 (socket says 172.21.15.98:34654) 2024-06-11T19:15:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f569a59c640 1 -- 172.21.15.98:0/3340660725 learned_addr learned my addr 172.21.15.98:0/3340660725 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f569a59c640 10 monclient(hunting): get_auth_request con 0x7f56a40a2d00 auth_method 0 2024-06-11T19:15:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f569a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f569a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.162+0000 7f569a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a359e640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f56a40a6260 0x7f56a41cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a2d9d640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56a406c890 0x7f56a409ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a359e640 10 monclient(hunting): get_auth_request con 0x7f56a40a6260 auth_method 0 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a359e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a359e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 10 monclient(hunting): handle_auth_done global_id 54924 payload 824 2024-06-11T19:15:47.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 1 -- 172.21.15.98:0/3340660725 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f56a40a6260 msgr2=0x7f56a41cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f56a40a6260 0x7f56a41cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 1 -- 172.21.15.98:0/3340660725 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56a406c890 msgr2=0x7f56a409ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56a406c890 0x7f56a409ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 10 monclient: _finish_hunting 0 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 1 monclient: found mon.noname-d 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 1 -- 172.21.15.98:0/3340660725 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a413d620 con 0x7f56a40a2d00 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a2d9d640 10 monclient: get_auth_request con 0x7f56a406c890 auth_method 0 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a2d9d640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56a406c890 0x7f56a409ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569a59c640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f56a40a2d00 0x7f56a4138d30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f568c0097a0 tx=0x7f568c0046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 10 client.?.objecter ms_handle_connect 0x7f56a40a2d00 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:47.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f568c038070 con 0x7f56a40a2d00 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 10 monclient: dump: 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.183 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 10 monclient: _finish_auth 0 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a91a8640 5 monclient: authenticate success, global_id 54924 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f568c042070 con 0x7f56a40a2d00 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:47.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f568c03d440 con 0x7f56a40a2d00 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a91a8640 10 monclient: _renew_subs 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a91a8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a91a8640 1 -- 172.21.15.98:0/3340660725 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56a413ea70 con 0x7f56a40a2d00 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a1d9b640 4 set_mon_vals no callback set 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f569bfff640 10 monclient: dump: 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.185 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a91a8640 10 client.54924.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a91a8640 10 monclient: _renew_subs 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a91a8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a91a8640 1 -- 172.21.15.98:0/3340660725 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f56a413f9d0 con 0x7f56a40a2d00 2024-06-11T19:15:47.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.163+0000 7f56a91a8640 1 librados: init done 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a91a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a91a8640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a4139270 0x7f56a41cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a91a8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a91a8640 1 -- 172.21.15.98:0/3340660725 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f56a4050830 con 0x7f56a4139270 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f568c003610 con 0x7f56a40a2d00 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a2d9d640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a4139270 0x7f56a41cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a2d9d640 10 monclient: get_auth_request con 0x7f56a4139270 auth_method 0 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a2d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f569bfff640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f568403fde0 0x7f56840422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(289..289 src has 1..289) ==== 6248+0+0 (secure 0 0 0) 0x7f568c0733d0 con 0x7f56a40a2d00 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f569bfff640 10 client.54924.objecter ms_dispatch 0x7f56a4000b90 osd_map(289..289 src has 1..289) 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f569bfff640 3 client.54924.objecter handle_osd_map got epochs [289,289] > 0 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f569bfff640 3 client.54924.objecter handle_osd_map decoding full epoch 289 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a2d9d640 10 monclient(hunting): handle_auth_done global_id 84215 payload 824 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a2d9d640 1 -- 172.21.15.98:0/3340660725 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f56a40a3aa0 con 0x7f56a4139270 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f569bfff640 20 client.54924.objecter dump_active .. 0 homeless 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.164+0000 7f56a359e640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f568403fde0 0x7f56840422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f56a359e640 10 monclient: get_auth_request con 0x7f568403fde0 auth_method 0 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 10 client.54924.objecter ms_handle_connect 0x7f56a4139270 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 10 client.54924.objecter resend_mon_ops 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f56a2d9d640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a4139270 0x7f56a41cdcc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5694000c00 tx=0x7f569400d8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f56940047e0 con 0x7f56a4139270 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 10 monclient: dump: 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 10 client.54924.objecter ms_handle_connect 0x7f568403fde0 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f56a359e640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f568403fde0 0x7f56840422a0 secure :-1 s=READY pgs=4134 cs=0 l=1 rev1=1 crypto rx=0x7f569c007fb0 tx=0x7f569c007540 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f56940072e0 con 0x7f56a4139270 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a4139270 msgr2=0x7f56a41cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a4139270 0x7f56a41cdcc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5694000c00 tx=0x7f569400d8a0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a4139270 msgr2=0x7f56a41cdcc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.165+0000 7f569bfff640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a4139270 0x7f56a41cdcc0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5694000c00 tx=0x7f569400d8a0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.171+0000 7f56a91a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.171+0000 7f56a91a8640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a41ce200 0x7f56a4070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.171+0000 7f56a91a8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.171+0000 7f56a91a8640 1 -- 172.21.15.98:0/3340660725 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f56a41ce610 con 0x7f56a41ce200 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.171+0000 7f56a2d9d640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a41ce200 0x7f56a4070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f56a2d9d640 10 monclient: get_auth_request con 0x7f56a41ce200 auth_method 0 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f56a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f56a2d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f56a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f56a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f56a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f56a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f56a2d9d640 10 monclient(hunting): handle_auth_done global_id 84224 payload 824 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f56a2d9d640 1 -- 172.21.15.98:0/3340660725 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f56a40a8250 con 0x7f56a41ce200 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f56a2d9d640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a41ce200 0x7f56a4070ad0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5694000c00 tx=0x7f5694000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 10 client.54924.objecter ms_handle_connect 0x7f56a41ce200 2024-06-11T19:15:47.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 10 client.54924.objecter resend_mon_ops 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f569401acf0 con 0x7f56a41ce200 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 10 monclient: dump: 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.194 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f569401a880 con 0x7f56a41ce200 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a41ce200 msgr2=0x7f56a4070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a41ce200 0x7f56a4070ad0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5694000c00 tx=0x7f5694000ed0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 1 -- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a41ce200 msgr2=0x7f56a4070ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:47.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f569bfff640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f56a41ce200 0x7f56a4070ad0 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5694000c00 tx=0x7f5694000ed0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.196 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:47.196 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:47.196 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:47.196 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:47.196 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:47.196 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 18946, 2024-06-11T19:15:47.196 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:47.196 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:47.196 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:47.196 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:47.196 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.197 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.198 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:47.199 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.200 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.201 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.204 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.205 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:47.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.207 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:47.208 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:47.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.215 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:47.217 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.172+0000 7f56a91a8640 10 librados: watch_flush enter 2024-06-11T19:15:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 10 librados: watch_flush exit 2024-06-11T19:15:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 20 client.54924.objecter shutdown clearing up homeless session... 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 10 client.54924.objecter successfully canceled tick 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 1 -- 172.21.15.98:0/3340660725 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f568403fde0 msgr2=0x7f56840422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f568403fde0 0x7f56840422a0 secure :-1 s=READY pgs=4134 cs=0 l=1 rev1=1 crypto rx=0x7f569c007fb0 tx=0x7f569c007540 comp rx=0 tx=0).stop 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 10 monclient: shutdown 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 1 -- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f56a40a2d00 msgr2=0x7f56a4138d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 1 --2- 172.21.15.98:0/3340660725 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f56a40a2d00 0x7f56a4138d30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f568c0097a0 tx=0x7f568c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a359e640 1 -- 172.21.15.98:0/3340660725 reap_dead start 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a359e640 1 -- 172.21.15.98:0/3340660725 reap_dead start 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 1 -- 172.21.15.98:0/3340660725 shutdown_connections 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 1 -- 172.21.15.98:0/3340660725 >> 172.21.15.98:0/3340660725 conn(0x7f56a40a95b0 msgr2=0x7f56a413be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 1 -- 172.21.15.98:0/3340660725 shutdown_connections 2024-06-11T19:15:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 1 -- 172.21.15.98:0/3340660725 wait complete. 2024-06-11T19:15:47.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.173+0000 7f56a91a8640 1 librados: shutdown 2024-06-11T19:15:47.221 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:47.221 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:15:47.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:15:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.321+0000 7ff34465c640 10 monclient: get_monmap_and_config 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.321+0000 7ff34465c640 10 monclient: build_initial_monmap 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.321+0000 7ff34465c640 10 monclient: monmap: 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:47.322078+0000 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:47.322078+0000 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:47.321 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:47.322 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.321+0000 7ff34465c640 1 Processor -- start 2024-06-11T19:15:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.321+0000 7ff34465c640 1 -- start start 2024-06-11T19:15:47.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.321+0000 7ff34465c640 10 monclient: init 2024-06-11T19:15:47.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:47.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 10 monclient: _add_conns ranks=[1,6,3,9,5,10,2,4,7,0,8] 2024-06-11T19:15:47.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff33c0a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:47.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff33c0a2530 0x7ff33c13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 10 monclient(hunting): picked mon.noname-d con 0x7ff33c0a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:47.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff33c13e1a0 0x7ff33c140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 10 monclient(hunting): picked mon.noname-g con 0x7ff33c13e1a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:47.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff33c0a7140 con 0x7ff33c0a4920 2024-06-11T19:15:47.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ff33c0a2b50 con 0x7ff33c13e1a0 2024-06-11T19:15:47.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff33c0a2e30 con 0x7ff33c0a2530 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff34465c640 10 monclient(hunting): authenticate will time out at 1157.129460s 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff33c0a2530 0x7ff33c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff33c0a2530 0x7ff33c13dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34664/0 (socket says 172.21.15.98:34664) 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 1 -- 172.21.15.98:0/1781616496 learned_addr learned my addr 172.21.15.98:0/1781616496 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 10 monclient(hunting): get_auth_request con 0x7ff33c0a2530 auth_method 0 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff342bf9640 1 --2- 172.21.15.98:0/1781616496 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff33c13e1a0 0x7ff33c140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff342bf9640 10 monclient(hunting): get_auth_request con 0x7ff33c13e1a0 auth_method 0 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff342bf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff342bf9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff342bf9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff342bf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff342bf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff342bf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 10 monclient(hunting): handle_auth_done global_id 54933 payload 274 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 1 -- 172.21.15.98:0/1781616496 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff33c13e1a0 msgr2=0x7ff33c140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 1 --2- 172.21.15.98:0/1781616496 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff33c13e1a0 0x7ff33c140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 1 -- 172.21.15.98:0/1781616496 >> v1:172.21.15.117:6789/0 conn(0x7ff33c0a4920 legacy=0x7ff33c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 10 monclient: _finish_hunting 0 2024-06-11T19:15:47.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 1 monclient: found mon.noname-d 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.322+0000 7ff341bf7640 1 -- 172.21.15.98:0/1781616496 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33c0a3110 con 0x7ff33c0a2530 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff341bf7640 1 --2- 172.21.15.98:0/1781616496 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff33c0a2530 0x7ff33c13dc60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0098a0 tx=0x7ff32c02b840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=f8571dafdd6375bf server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 1 -- 172.21.15.98:0/1781616496 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff32c031a10 con 0x7ff33c0a2530 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 10 monclient: dump: 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.335 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 10 monclient: _finish_auth 0 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 10 monclient: _check_auth_tickets 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:47.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff34465c640 5 monclient: authenticate success, global_id 54933 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff34465c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 1 -- 172.21.15.98:0/1781616496 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff32c037070 con 0x7ff33c0a2530 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff34465c640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff34465c640 4 set_mon_vals no callback set 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 1 -- 172.21.15.98:0/1781616496 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff32c031c10 con 0x7ff33c0a2530 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.323+0000 7ff3413f6640 10 monclient: dump: 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.324+0000 7ff34465c640 10 monclient: shutdown 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.324+0000 7ff34465c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.324+0000 7ff34465c640 1 -- 172.21.15.98:0/1781616496 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff33c0a2530 msgr2=0x7ff33c13dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.324+0000 7ff34465c640 1 --2- 172.21.15.98:0/1781616496 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff33c0a2530 0x7ff33c13dc60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0098a0 tx=0x7ff32c02b840 comp rx=0 tx=0).stop 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.324+0000 7ff34465c640 1 -- 172.21.15.98:0/1781616496 shutdown_connections 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.324+0000 7ff34465c640 1 --2- 172.21.15.98:0/1781616496 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff33c13e1a0 0x7ff33c140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.324+0000 7ff34465c640 1 --2- 172.21.15.98:0/1781616496 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff33c0a2530 0x7ff33c13dc60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.324+0000 7ff34465c640 1 -- 172.21.15.98:0/1781616496 >> 172.21.15.98:0/1781616496 conn(0x7ff33c0ab850 msgr2=0x7ff33c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.324+0000 7ff34465c640 1 -- 172.21.15.98:0/1781616496 shutdown_connections 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.324+0000 7ff34465c640 1 -- 172.21.15.98:0/1781616496 wait complete. 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.324+0000 7ff34465c640 10 monclient: build_initial_monmap 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.324+0000 7ff34465c640 10 monclient: monmap: 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 1 librados: starting msgr at 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 1 librados: starting objecter 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 1 Processor -- start 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 1 -- start start 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 1 librados: setting wanted keys 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 1 librados: calling monclient init 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 10 monclient: init 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 10 monclient: _add_conns ranks=[3,4,5,8,1,0,2,6,7] 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff33c0a2530 0x7ff33c135bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 10 monclient(hunting): picked mon.noname-g con 0x7ff33c0a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c0a4920 0x7ff33c136110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 10 monclient(hunting): picked mon.noname-c con 0x7ff33c0a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff33c13e1a0 0x7ff33c09ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 10 monclient(hunting): picked mon.noname-h con 0x7ff33c13e1a0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ff33c138e50 con 0x7ff33c0a2530 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff33c0a84b0 con 0x7ff33c0a4920 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ff33c0a6da0 con 0x7ff33c13e1a0 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff34465c640 10 monclient(hunting): authenticate will time out at 1157.132815s 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff341bf7640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c0a4920 0x7ff33c136110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff341bf7640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c0a4920 0x7ff33c136110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:53684/0 (socket says 172.21.15.98:53684) 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff341bf7640 1 -- 172.21.15.98:0/2526316053 learned_addr learned my addr 172.21.15.98:0/2526316053 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff341bf7640 10 monclient(hunting): get_auth_request con 0x7ff33c0a4920 auth_method 0 2024-06-11T19:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff341bf7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff341bf7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff341bf7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff3423f8640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff33c0a2530 0x7ff33c135bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff3423f8640 10 monclient(hunting): get_auth_request con 0x7ff33c0a2530 auth_method 0 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff3423f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff3423f8640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.325+0000 7ff3423f8640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff342bf9640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff33c13e1a0 0x7ff33c09ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff342bf9640 10 monclient(hunting): get_auth_request con 0x7ff33c13e1a0 auth_method 0 2024-06-11T19:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff342bf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff342bf9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff342bf9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff3423f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff3423f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff3423f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff342bf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff342bf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff342bf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 10 monclient(hunting): handle_auth_done global_id 23746 payload 824 2024-06-11T19:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 1 -- 172.21.15.98:0/2526316053 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff33c13e1a0 msgr2=0x7ff33c09ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff33c13e1a0 0x7ff33c09ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 1 -- 172.21.15.98:0/2526316053 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff33c0a2530 msgr2=0x7ff33c135bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff33c0a2530 0x7ff33c135bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 10 monclient: _finish_hunting 0 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 1 monclient: found mon.noname-c 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 1 -- 172.21.15.98:0/2526316053 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33c0a4260 con 0x7ff33c0a4920 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff341bf7640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c0a4920 0x7ff33c136110 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0090b0 tx=0x7ff32c0046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 10 client.?.objecter ms_handle_connect 0x7ff33c0a4920 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff32c041070 con 0x7ff33c0a4920 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 10 monclient: dump: 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 10 monclient: _finish_auth 0 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff32c0315a0 con 0x7ff33c0a4920 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff34465c640 5 monclient: authenticate success, global_id 23746 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff34465c640 10 monclient: _renew_subs 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff32c031bc0 con 0x7ff33c0a4920 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff34465c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff34465c640 1 -- 172.21.15.98:0/2526316053 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff33c0aa770 con 0x7ff33c0a4920 2024-06-11T19:15:47.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff340bf5640 4 set_mon_vals no callback set 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff34465c640 10 client.23746.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff332ffd640 10 monclient: dump: 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff34465c640 10 monclient: _renew_subs 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff34465c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff34465c640 1 -- 172.21.15.98:0/2526316053 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ff33c0a0860 con 0x7ff33c0a4920 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.326+0000 7ff34465c640 1 librados: init done 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff34465c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff34465c640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c1cd950 0x7ff33c122200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff32c04c060 con 0x7ff33c0a4920 2024-06-11T19:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff34465c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff34465c640 1 -- 172.21.15.98:0/2526316053 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff33c050830 con 0x7ff33c1cd950 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff3423f8640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c1cd950 0x7ff33c122200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff3423f8640 10 monclient: get_auth_request con 0x7ff33c1cd950 auth_method 0 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff3423f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff3423f8640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff3423f8640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff3423f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff3423f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff332ffd640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff32003fdb0 0x7ff320042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff3423f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(289..289 src has 1..289) ==== 6248+0+0 (secure 0 0 0) 0x7ff32c02f050 con 0x7ff33c0a4920 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff332ffd640 10 client.23746.objecter ms_dispatch 0x7ff33c000b90 osd_map(289..289 src has 1..289) 2024-06-11T19:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff332ffd640 3 client.23746.objecter handle_osd_map got epochs [289,289] > 0 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff332ffd640 3 client.23746.objecter handle_osd_map decoding full epoch 289 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff3423f8640 10 monclient(hunting): handle_auth_done global_id 23755 payload 824 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff332ffd640 20 client.23746.objecter dump_active .. 0 homeless 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff3423f8640 1 -- 172.21.15.98:0/2526316053 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff33c1ce5f0 con 0x7ff33c1cd950 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff342bf9640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff32003fdb0 0x7ff320042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff342bf9640 10 monclient: get_auth_request con 0x7ff32003fdb0 auth_method 0 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff332ffd640 10 client.23746.objecter ms_handle_connect 0x7ff33c1cd950 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff332ffd640 10 client.23746.objecter resend_mon_ops 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.327+0000 7ff3423f8640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c1cd950 0x7ff33c122200 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ff33800f570 tx=0x7ff33800fa30 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff3380052e0 con 0x7ff33c1cd950 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff332ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff332ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff332ffd640 10 monclient: dump: 2024-06-11T19:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.353 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff332ffd640 10 client.23746.objecter ms_handle_connect 0x7ff32003fdb0 2024-06-11T19:15:47.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff342bf9640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff32003fdb0 0x7ff320042270 secure :-1 s=READY pgs=4135 cs=0 l=1 rev1=1 crypto rx=0x7ff33c121a60 tx=0x7ff33400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff338018070 con 0x7ff33c1cd950 2024-06-11T19:15:47.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff332ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:47.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff332ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:47.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c1cd950 msgr2=0x7ff33c122200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff332ffd640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c1cd950 0x7ff33c122200 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ff33800f570 tx=0x7ff33800fa30 comp rx=0 tx=0).stop 2024-06-11T19:15:47.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c1cd950 msgr2=0x7ff33c122200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:47.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.328+0000 7ff332ffd640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c1cd950 0x7ff33c122200 secure :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ff33800f570 tx=0x7ff33800fa30 comp rx=0 tx=0).stop 2024-06-11T19:15:47.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff34465c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:47.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff34465c640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c09f170 0x7ff33c09f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff34465c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff34465c640 1 -- 172.21.15.98:0/2526316053 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff33c09fac0 con 0x7ff33c09f170 2024-06-11T19:15:47.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff3423f8640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c09f170 0x7ff33c09f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff3423f8640 10 monclient: get_auth_request con 0x7ff33c09f170 auth_method 0 2024-06-11T19:15:47.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff3423f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff3423f8640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff3423f8640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff3423f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff3423f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff3423f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.334+0000 7ff3423f8640 10 monclient(hunting): handle_auth_done global_id 23764 payload 824 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff3423f8640 1 -- 172.21.15.98:0/2526316053 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff33c0a4d00 con 0x7ff33c09f170 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff3423f8640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c09f170 0x7ff33c09f580 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ff33800f570 tx=0x7ff338000ed0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 10 client.23746.objecter ms_handle_connect 0x7ff33c09f170 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 10 client.23746.objecter resend_mon_ops 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff338003d40 con 0x7ff33c09f170 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:47.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 10 monclient: dump: 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.357 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.358 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.358 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7ff3380038d0 con 0x7ff33c09f170 2024-06-11T19:15:47.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:47.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:47.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c09f170 msgr2=0x7ff33c09f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c09f170 0x7ff33c09f580 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ff33800f570 tx=0x7ff338000ed0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 1 -- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c09f170 msgr2=0x7ff33c09f580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:47.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff332ffd640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c09f170 0x7ff33c09f580 secure :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ff33800f570 tx=0x7ff338000ed0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.358 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 397553, 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:15:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:47.382 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 10 librados: watch_flush enter 2024-06-11T19:15:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 10 librados: watch_flush exit 2024-06-11T19:15:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 20 client.23746.objecter shutdown clearing up homeless session... 2024-06-11T19:15:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 10 client.23746.objecter successfully canceled tick 2024-06-11T19:15:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 1 -- 172.21.15.98:0/2526316053 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff32003fdb0 msgr2=0x7ff320042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff32003fdb0 0x7ff320042270 secure :-1 s=READY pgs=4135 cs=0 l=1 rev1=1 crypto rx=0x7ff33c121a60 tx=0x7ff33400b040 comp rx=0 tx=0).stop 2024-06-11T19:15:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 10 monclient: shutdown 2024-06-11T19:15:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 1 -- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c0a4920 msgr2=0x7ff33c136110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 1 --2- 172.21.15.98:0/2526316053 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff33c0a4920 0x7ff33c136110 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0090b0 tx=0x7ff32c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff342bf9640 1 -- 172.21.15.98:0/2526316053 reap_dead start 2024-06-11T19:15:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff342bf9640 1 -- 172.21.15.98:0/2526316053 reap_dead start 2024-06-11T19:15:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 1 -- 172.21.15.98:0/2526316053 shutdown_connections 2024-06-11T19:15:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 1 -- 172.21.15.98:0/2526316053 >> 172.21.15.98:0/2526316053 conn(0x7ff33c0ab850 msgr2=0x7ff33c0a9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 1 -- 172.21.15.98:0/2526316053 shutdown_connections 2024-06-11T19:15:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.335+0000 7ff34465c640 1 -- 172.21.15.98:0/2526316053 wait complete. 2024-06-11T19:15:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.336+0000 7ff34465c640 1 librados: shutdown 2024-06-11T19:15:47.385 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:47.386 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:15:47.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:15:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.485+0000 7fe171320640 10 monclient: get_monmap_and_config 2024-06-11T19:15:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.485+0000 7fe171320640 10 monclient: build_initial_monmap 2024-06-11T19:15:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.486+0000 7fe171320640 10 monclient: monmap: 2024-06-11T19:15:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:47.487024+0000 2024-06-11T19:15:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:47.487024+0000 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:47.487 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.486+0000 7fe171320640 1 Processor -- start 2024-06-11T19:15:47.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.486+0000 7fe171320640 1 -- start start 2024-06-11T19:15:47.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.486+0000 7fe171320640 10 monclient: init 2024-06-11T19:15:47.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:47.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 10 monclient: _add_conns ranks=[7,3,2,0,5,8,1,6,9,10,4] 2024-06-11T19:15:47.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe16c0a4920 0x7fe16c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 10 monclient(hunting): picked mon.noname-i con 0x7fe16c0a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:15:47.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe16c0a5240 0x7fe16c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 10 monclient(hunting): picked mon.noname-g con 0x7fe16c0a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:47.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe16c0a2530 0x7fe16c0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 10 monclient(hunting): picked mon.noname-b con 0x7fe16c0a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe16c0a7140 con 0x7fe16c0a2530 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fe16c0a2b50 con 0x7fe16c0a5240 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fe16c0a2e30 con 0x7fe16c0a4920 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe171320640 10 monclient(hunting): authenticate will time out at 1157.294121s 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe16c0a2530 0x7fe16c0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe16c0a2530 0x7fe16c0a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:59894/0 (socket says 172.21.15.98:59894) 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 1 -- 172.21.15.98:0/867260975 learned_addr learned my addr 172.21.15.98:0/867260975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 10 monclient(hunting): get_auth_request con 0x7fe16c0a2530 auth_method 0 2024-06-11T19:15:47.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16a7fc640 1 --2- 172.21.15.98:0/867260975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe16c0a5240 0x7fe16c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16affd640 1 --2- 172.21.15.98:0/867260975 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe16c0a4920 0x7fe16c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16a7fc640 10 monclient(hunting): get_auth_request con 0x7fe16c0a5240 auth_method 0 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16affd640 10 monclient(hunting): get_auth_request con 0x7fe16c0a4920 auth_method 0 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_done global_id 84233 payload 274 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 1 -- 172.21.15.98:0/867260975 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe16c0a4920 msgr2=0x7fe16c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 1 --2- 172.21.15.98:0/867260975 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe16c0a4920 0x7fe16c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 1 -- 172.21.15.98:0/867260975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe16c0a5240 msgr2=0x7fe16c0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 1 --2- 172.21.15.98:0/867260975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe16c0a5240 0x7fe16c0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 1 monclient: found mon.noname-b 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.487+0000 7fe16b7fe640 1 -- 172.21.15.98:0/867260975 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe16c0a3110 con 0x7fe16c0a2530 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe16b7fe640 1 --2- 172.21.15.98:0/867260975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe16c0a2530 0x7fe16c0a5ee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe164009e80 tx=0x7fe16402bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c77c20a2ae97f395 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 1 -- 172.21.15.98:0/867260975 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe16402cb80 con 0x7fe16c0a2530 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 10 monclient: dump: 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 10 monclient: _finish_auth 0 2024-06-11T19:15:47.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 1 -- 172.21.15.98:0/867260975 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16402cd40 con 0x7fe16c0a2530 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe171320640 5 monclient: authenticate success, global_id 84233 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe171320640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 1 -- 172.21.15.98:0/867260975 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe164034860 con 0x7fe16c0a2530 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe171320640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe171320640 4 set_mon_vals no callback set 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.488+0000 7fe169ffb640 10 monclient: dump: 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.504 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 10 monclient: shutdown 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 1 -- 172.21.15.98:0/867260975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe16c0a2530 msgr2=0x7fe16c0a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 1 --2- 172.21.15.98:0/867260975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe16c0a2530 0x7fe16c0a5ee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe164009e80 tx=0x7fe16402bd70 comp rx=0 tx=0).stop 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 1 -- 172.21.15.98:0/867260975 shutdown_connections 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 1 --2- 172.21.15.98:0/867260975 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe16c0a2530 0x7fe16c0a5ee0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 1 --2- 172.21.15.98:0/867260975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe16c0a5240 0x7fe16c0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 1 --2- 172.21.15.98:0/867260975 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe16c0a4920 0x7fe16c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 1 -- 172.21.15.98:0/867260975 >> 172.21.15.98:0/867260975 conn(0x7fe16c0ab850 msgr2=0x7fe16c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 1 -- 172.21.15.98:0/867260975 shutdown_connections 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 1 -- 172.21.15.98:0/867260975 wait complete. 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 10 monclient: build_initial_monmap 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.489+0000 7fe171320640 10 monclient: monmap: 2024-06-11T19:15:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 1 librados: starting msgr at 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 1 librados: starting objecter 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 1 Processor -- start 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 1 -- start start 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 1 librados: setting wanted keys 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 1 librados: calling monclient init 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 10 monclient: init 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 10 monclient: _add_conns ranks=[4,0,3,1,8,5,7,6,2] 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe16c0a2530 0x7fe16c1cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 10 monclient(hunting): picked mon.noname-c con 0x7fe16c0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe16c0a4920 0x7fe16c1ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 10 monclient(hunting): picked mon.noname-a con 0x7fe16c0a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe16c0a5240 0x7fe16c121e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 10 monclient(hunting): picked mon.noname-g con 0x7fe16c0a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe16c0a6da0 con 0x7fe16c0a4920 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fe16c138e50 con 0x7fe16c0a5240 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe16c0a6420 con 0x7fe16c0a2530 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe171320640 10 monclient(hunting): authenticate will time out at 1157.297857s 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16affd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe16c0a2530 0x7fe16c1cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16a7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe16c0a4920 0x7fe16c1ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16affd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe16c0a2530 0x7fe16c1cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:53716/0 (socket says 172.21.15.98:53716) 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16a7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe16c0a4920 0x7fe16c1ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46344/0 (socket says 172.21.15.98:46344) 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16affd640 1 -- 172.21.15.98:0/1854522097 learned_addr learned my addr 172.21.15.98:0/1854522097 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16affd640 10 monclient(hunting): get_auth_request con 0x7fe16c0a2530 auth_method 0 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16a7fc640 10 monclient(hunting): get_auth_request con 0x7fe16c0a4920 auth_method 0 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.490+0000 7fe16a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16b7fe640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe16c0a5240 0x7fe16c121e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16b7fe640 10 monclient(hunting): get_auth_request con 0x7fe16c0a5240 auth_method 0 2024-06-11T19:15:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 10 monclient(hunting): handle_auth_done global_id 64656 payload 824 2024-06-11T19:15:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 1 -- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe16c0a2530 msgr2=0x7fe16c1cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe16c0a2530 0x7fe16c1cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 1 -- 172.21.15.98:0/1854522097 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe16c0a5240 msgr2=0x7fe16c121e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe16c0a5240 0x7fe16c121e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 1 monclient: found mon.noname-a 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 1 -- 172.21.15.98:0/1854522097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe16c1257c0 con 0x7fe16c0a4920 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16affd640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe16c0a2530 0x7fe16c1cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe16a7fc640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe16c0a4920 0x7fe16c1ce010 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe16000e840 tx=0x7fe16000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 10 client.?.objecter ms_handle_connect 0x7fe16c0a4920 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe160017070 con 0x7fe16c0a4920 2024-06-11T19:15:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 10 monclient: dump: 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 10 monclient: _finish_auth 0 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 10 monclient: _check_auth_tickets 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16000cba0 con 0x7fe16c0a4920 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe171320640 5 monclient: authenticate success, global_id 64656 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe171320640 10 monclient: _renew_subs 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe171320640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe171320640 1 -- 172.21.15.98:0/1854522097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe16c1245f0 con 0x7fe16c0a4920 2024-06-11T19:15:47.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe16000cd50 con 0x7fe16c0a4920 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1537fe640 10 monclient: dump: 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.514 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe171320640 10 client.64656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe171320640 10 monclient: _renew_subs 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe1697fa640 4 set_mon_vals no callback set 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe171320640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe171320640 1 -- 172.21.15.98:0/1854522097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe16c138760 con 0x7fe16c0a4920 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.491+0000 7fe171320640 1 librados: init done 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe171320640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe171320640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122b90 0x7fe16c02cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe171320640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe171320640 1 -- 172.21.15.98:0/1854522097 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe16c050830 con 0x7fe16c122b90 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fe16001a070 con 0x7fe16c0a4920 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe16affd640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122b90 0x7fe16c02cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe16affd640 10 monclient: get_auth_request con 0x7fe16c122b90 auth_method 0 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe16affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe16affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe16affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe1537fe640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe15803fdb0 0x7fe158042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe16affd640 10 monclient(hunting): handle_auth_done global_id 54942 payload 824 2024-06-11T19:15:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6248+0+0 (secure 0 0 0) 0x7fe16000f040 con 0x7fe16c0a4920 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe1537fe640 10 client.64656.objecter ms_dispatch 0x7fe16c000b90 osd_map(289..289 src has 1..289) 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe1537fe640 3 client.64656.objecter handle_osd_map got epochs [289,289] > 0 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe1537fe640 3 client.64656.objecter handle_osd_map decoding full epoch 289 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.492+0000 7fe16affd640 1 -- 172.21.15.98:0/1854522097 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe16c123a90 con 0x7fe16c122b90 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 20 client.64656.objecter dump_active .. 0 homeless 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe16b7fe640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe15803fdb0 0x7fe158042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 10 client.64656.objecter ms_handle_connect 0x7fe16c122b90 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 10 client.64656.objecter resend_mon_ops 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe16affd640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122b90 0x7fe16c02cb40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe15c011db0 tx=0x7fe15c00f610 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe16b7fe640 10 monclient: get_auth_request con 0x7fe15803fdb0 auth_method 0 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe15c0044a0 con 0x7fe16c122b90 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 10 monclient: dump: 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.519 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe15c004030 con 0x7fe16c122b90 2024-06-11T19:15:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122b90 msgr2=0x7fe16c02cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122b90 0x7fe16c02cb40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe15c011db0 tx=0x7fe15c00f610 comp rx=0 tx=0).stop 2024-06-11T19:15:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122b90 msgr2=0x7fe16c02cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122b90 0x7fe16c02cb40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe1537fe640 10 client.64656.objecter ms_handle_connect 0x7fe15803fdb0 2024-06-11T19:15:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.493+0000 7fe16b7fe640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe15803fdb0 0x7fe158042270 secure :-1 s=READY pgs=4136 cs=0 l=1 rev1=1 crypto rx=0x7fe164035000 tx=0x7fe164030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.499+0000 7fe171320640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:47.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.499+0000 7fe171320640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122f70 0x7fe16c02d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe171320640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe171320640 1 -- 172.21.15.98:0/1854522097 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe16c0a6720 con 0x7fe16c122f70 2024-06-11T19:15:47.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe16affd640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122f70 0x7fe16c02d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe16affd640 10 monclient: get_auth_request con 0x7fe16c122f70 auth_method 0 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe16affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe16affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe16affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe16affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe16affd640 10 monclient(hunting): handle_auth_done global_id 54951 payload 824 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe16affd640 1 -- 172.21.15.98:0/1854522097 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe16c125ce0 con 0x7fe16c122f70 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 10 client.64656.objecter ms_handle_connect 0x7fe16c122f70 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe16affd640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122f70 0x7fe16c02d080 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe15c01b000 tx=0x7fe15c0046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 10 client.64656.objecter resend_mon_ops 2024-06-11T19:15:47.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe15c014d30 con 0x7fe16c122f70 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 10 monclient: dump: 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.523 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.524 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.524 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.524 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.524 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.524 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fe15c0273a0 con 0x7fe16c122f70 2024-06-11T19:15:47.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:47.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:47.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122f70 msgr2=0x7fe16c02d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122f70 0x7fe16c02d080 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe15c01b000 tx=0x7fe15c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 1 -- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122f70 msgr2=0x7fe16c02d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:47.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.500+0000 7fe1537fe640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe16c122f70 0x7fe16c02d080 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe15c01b000 tx=0x7fe15c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.525 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:47.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:47.525 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:47.525 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:47.525 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:47.525 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 260257, 2024-06-11T19:15:47.525 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:47.525 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:47.525 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:47.525 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:47.525 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:47.525 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.526 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.527 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:47.529 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:47.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:47.532 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:47.534 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:47.534 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.534 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.534 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.534 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:47.534 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:47.534 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.535 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:47.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.538 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.542 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.543 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.545 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:15:47.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:47.547 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:47.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 10 librados: watch_flush enter 2024-06-11T19:15:47.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 10 librados: watch_flush exit 2024-06-11T19:15:47.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 20 client.64656.objecter shutdown clearing up homeless session... 2024-06-11T19:15:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 10 client.64656.objecter successfully canceled tick 2024-06-11T19:15:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 1 -- 172.21.15.98:0/1854522097 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe15803fdb0 msgr2=0x7fe158042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe15803fdb0 0x7fe158042270 secure :-1 s=READY pgs=4136 cs=0 l=1 rev1=1 crypto rx=0x7fe164035000 tx=0x7fe164030000 comp rx=0 tx=0).stop 2024-06-11T19:15:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 10 monclient: shutdown 2024-06-11T19:15:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 1 -- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe16c0a4920 msgr2=0x7fe16c1ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 1 --2- 172.21.15.98:0/1854522097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe16c0a4920 0x7fe16c1ce010 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe16000e840 tx=0x7fe16000ed00 comp rx=0 tx=0).stop 2024-06-11T19:15:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe16b7fe640 1 -- 172.21.15.98:0/1854522097 reap_dead start 2024-06-11T19:15:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe16b7fe640 1 -- 172.21.15.98:0/1854522097 reap_dead start 2024-06-11T19:15:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 1 -- 172.21.15.98:0/1854522097 shutdown_connections 2024-06-11T19:15:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 1 -- 172.21.15.98:0/1854522097 >> 172.21.15.98:0/1854522097 conn(0x7fe16c0ab850 msgr2=0x7fe16c0a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:47.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 1 -- 172.21.15.98:0/1854522097 shutdown_connections 2024-06-11T19:15:47.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 1 -- 172.21.15.98:0/1854522097 wait complete. 2024-06-11T19:15:47.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.501+0000 7fe171320640 1 librados: shutdown 2024-06-11T19:15:47.551 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:47.551 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:15:47.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:15:47.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.651+0000 7f893210b640 10 monclient: get_monmap_and_config 2024-06-11T19:15:47.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.651+0000 7f893210b640 10 monclient: build_initial_monmap 2024-06-11T19:15:47.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.651+0000 7f893210b640 10 monclient: monmap: 2024-06-11T19:15:47.650 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:47.652246+0000 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:47.652246+0000 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:47.651 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:47.652 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:47.652 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:47.652 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 1 Processor -- start 2024-06-11T19:15:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 1 -- start start 2024-06-11T19:15:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 10 monclient: init 2024-06-11T19:15:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 10 monclient: _add_conns ranks=[3,9,8,10,6,2,1,7,4,0,5] 2024-06-11T19:15:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f892c0a4920 0x7f892c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 10 monclient(hunting): picked mon.noname-g con 0x7f892c0a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f892c0a5240 0x7f892c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 10 monclient(hunting): picked mon.noname-a con 0x7f892c0a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c0a2530 0x7f892c0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 10 monclient(hunting): picked mon.noname-e con 0x7f892c0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f892c0a7140 con 0x7f892c0a5240 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f892c0a2b50 con 0x7f892c0a4920 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f892c0a2e30 con 0x7f892c0a2530 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f893210b640 10 monclient(hunting): authenticate will time out at 1157.459698s 2024-06-11T19:15:47.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f8923fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f892c0a5240 0x7f892c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f892bfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c0a2530 0x7f892c0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f892bfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c0a2530 0x7f892c0a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34854/0 (socket says 172.21.15.98:34854) 2024-06-11T19:15:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f8923fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f892c0a5240 0x7f892c0a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46354/0 (socket says 172.21.15.98:46354) 2024-06-11T19:15:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f892bfff640 1 -- 172.21.15.98:0/602847222 learned_addr learned my addr 172.21.15.98:0/602847222 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f892bfff640 10 monclient(hunting): get_auth_request con 0x7f892c0a2530 auth_method 0 2024-06-11T19:15:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f892bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f892bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f892bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f8923fff640 10 monclient(hunting): get_auth_request con 0x7f892c0a5240 auth_method 0 2024-06-11T19:15:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f8923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f8923fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f8923fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f892bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f892bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f892bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.652+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f892b7fe640 1 --2- 172.21.15.98:0/602847222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f892c0a4920 0x7f892c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f892b7fe640 10 monclient(hunting): get_auth_request con 0x7f892c0a4920 auth_method 0 2024-06-11T19:15:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f892b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f892b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f892b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f8923fff640 10 monclient(hunting): handle_auth_done global_id 64665 payload 274 2024-06-11T19:15:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f8923fff640 1 -- 172.21.15.98:0/602847222 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c0a2530 msgr2=0x7f892c0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f8923fff640 1 --2- 172.21.15.98:0/602847222 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c0a2530 0x7f892c0a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f8923fff640 1 -- 172.21.15.98:0/602847222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f892c0a4920 msgr2=0x7f892c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f8923fff640 1 --2- 172.21.15.98:0/602847222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f892c0a4920 0x7f892c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f8923fff640 10 monclient: _finish_hunting 0 2024-06-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f8923fff640 1 monclient: found mon.noname-a 2024-06-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f8923fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f8923fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:15:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f8923fff640 1 -- 172.21.15.98:0/602847222 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892c0a3110 con 0x7f892c0a5240 2024-06-11T19:15:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f892bfff640 1 --2- 172.21.15.98:0/602847222 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c0a2530 0x7f892c0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f8923fff640 1 --2- 172.21.15.98:0/602847222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f892c0a5240 0x7f892c0a59a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f89140097a0 tx=0x7f891402b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13127a8d780e2a9e server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.653+0000 7f892affd640 1 -- 172.21.15.98:0/602847222 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8914037070 con 0x7f892c0a5240 2024-06-11T19:15:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f892affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f892affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:15:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f892affd640 10 monclient: dump: 2024-06-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.669 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f892affd640 10 monclient: _finish_auth 0 2024-06-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f892affd640 10 monclient: _check_auth_tickets 2024-06-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f892affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f892affd640 1 -- 172.21.15.98:0/602847222 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8914030980 con 0x7f892c0a5240 2024-06-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 5 monclient: authenticate success, global_id 64665 2024-06-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f892affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f892affd640 1 -- 172.21.15.98:0/602847222 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8914030bb0 con 0x7f892c0a5240 2024-06-11T19:15:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 4 set_mon_vals no callback set 2024-06-11T19:15:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f892affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f892affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f892affd640 10 monclient: dump: 2024-06-11T19:15:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.674 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 10 monclient: shutdown 2024-06-11T19:15:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 1 -- 172.21.15.98:0/602847222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f892c0a5240 msgr2=0x7f892c0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 1 --2- 172.21.15.98:0/602847222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f892c0a5240 0x7f892c0a59a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f89140097a0 tx=0x7f891402b6f0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 1 -- 172.21.15.98:0/602847222 shutdown_connections 2024-06-11T19:15:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 1 --2- 172.21.15.98:0/602847222 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c0a2530 0x7f892c0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 1 --2- 172.21.15.98:0/602847222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f892c0a5240 0x7f892c0a59a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 1 --2- 172.21.15.98:0/602847222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f892c0a4920 0x7f892c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 1 -- 172.21.15.98:0/602847222 >> 172.21.15.98:0/602847222 conn(0x7f892c0ab850 msgr2=0x7f892c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 1 -- 172.21.15.98:0/602847222 shutdown_connections 2024-06-11T19:15:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.654+0000 7f893210b640 1 -- 172.21.15.98:0/602847222 wait complete. 2024-06-11T19:15:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.655+0000 7f893210b640 10 monclient: build_initial_monmap 2024-06-11T19:15:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.655+0000 7f893210b640 10 monclient: monmap: 2024-06-11T19:15:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:47.678 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.655+0000 7f893210b640 1 librados: starting msgr at 2024-06-11T19:15:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.655+0000 7f893210b640 1 librados: starting objecter 2024-06-11T19:15:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.655+0000 7f893210b640 1 Processor -- start 2024-06-11T19:15:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.655+0000 7f893210b640 1 -- start start 2024-06-11T19:15:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.655+0000 7f893210b640 1 librados: setting wanted keys 2024-06-11T19:15:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.655+0000 7f893210b640 1 librados: calling monclient init 2024-06-11T19:15:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.655+0000 7f893210b640 10 monclient: init 2024-06-11T19:15:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient: _add_conns ranks=[4,2,3,8,6,7,1,5,0] 2024-06-11T19:15:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f892c0a2530 0x7f892c1cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient(hunting): picked mon.noname-c con 0x7f892c0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:15:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f892c0a4920 0x7f892c1ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient(hunting): picked mon.noname-b con 0x7f892c0a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f892c0a5240 0x7f892c123090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient(hunting): picked mon.noname-g con 0x7f892c0a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f892c0a6da0 con 0x7f892c0a4920 2024-06-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f892c138e50 con 0x7f892c0a5240 2024-06-11T19:15:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f892c0a6420 con 0x7f892c0a2530 2024-06-11T19:15:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient(hunting): authenticate will time out at 1157.463156s 2024-06-11T19:15:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892b7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f892c0a2530 0x7f892c1cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f892c0a4920 0x7f892c1ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892b7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f892c0a2530 0x7f892c1cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:53720/0 (socket says 172.21.15.98:53720) 2024-06-11T19:15:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892b7fe640 1 -- 172.21.15.98:0/2318446739 learned_addr learned my addr 172.21.15.98:0/2318446739 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:47.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 10 monclient(hunting): get_auth_request con 0x7f892c0a4920 auth_method 0 2024-06-11T19:15:47.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892b7fe640 10 monclient(hunting): get_auth_request con 0x7f892c0a2530 auth_method 0 2024-06-11T19:15:47.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 10 monclient(hunting): handle_auth_done global_id 84242 payload 824 2024-06-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892bfff640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f892c0a5240 0x7f892c123090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 1 -- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f892c0a2530 msgr2=0x7f892c1cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f892c0a2530 0x7f892c1cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 1 -- 172.21.15.98:0/2318446739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f892c0a5240 msgr2=0x7f892c123090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f892c0a5240 0x7f892c123090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 10 monclient: _finish_hunting 0 2024-06-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 1 monclient: found mon.noname-b 2024-06-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 1 -- 172.21.15.98:0/2318446739 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892c1269c0 con 0x7f892c0a4920 2024-06-11T19:15:47.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892b7fe640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f892c0a2530 0x7f892c1cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892bfff640 10 monclient: get_auth_request con 0x7f892c0a5240 auth_method 0 2024-06-11T19:15:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892bfff640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f892c0a5240 0x7f892c123090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:15:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8923fff640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f892c0a4920 0x7f892c1ce230 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8914032000 tx=0x7f89140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 10 client.?.objecter ms_handle_connect 0x7f892c0a4920 2024-06-11T19:15:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8914037050 con 0x7f892c0a4920 2024-06-11T19:15:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 10 monclient: dump: 2024-06-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.690 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.690 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.690 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.690 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 10 monclient: _finish_auth 0 2024-06-11T19:15:47.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:15:47.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:47.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8914038040 con 0x7f892c0a4920 2024-06-11T19:15:47.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 5 monclient: authenticate success, global_id 84242 2024-06-11T19:15:47.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient: _renew_subs 2024-06-11T19:15:47.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:47.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 1 -- 172.21.15.98:0/2318446739 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f892c126ca0 con 0x7f892c0a4920 2024-06-11T19:15:47.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:47.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8914004390 con 0x7f892c0a4920 2024-06-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f8928ff9640 10 monclient: dump: 2024-06-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.693 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.693 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.693 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.693 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.693 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.693 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.693 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.694 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.694 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.694 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.694 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.694 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.694 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 client.84242.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:47.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient: _renew_subs 2024-06-11T19:15:47.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:47.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f892a7fc640 4 set_mon_vals no callback set 2024-06-11T19:15:47.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 1 -- 172.21.15.98:0/2318446739 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f892c0a92d0 con 0x7f892c0a4920 2024-06-11T19:15:47.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.656+0000 7f893210b640 1 librados: init done 2024-06-11T19:15:47.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8914045030 con 0x7f892c0a4920 2024-06-11T19:15:47.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f893210b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:47.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f893210b640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c1235d0 0x7f892c1239b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f893210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f893210b640 1 -- 172.21.15.98:0/2318446739 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f892c1257f0 con 0x7f892c1235d0 2024-06-11T19:15:47.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f8928ff9640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f890c03fdb0 0x7f890c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(289..289 src has 1..289) ==== 6248+0+0 (secure 0 0 0) 0x7f8914043020 con 0x7f892c0a4920 2024-06-11T19:15:47.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f8928ff9640 10 client.84242.objecter ms_dispatch 0x7f892c000b90 osd_map(289..289 src has 1..289) 2024-06-11T19:15:47.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f8928ff9640 3 client.84242.objecter handle_osd_map got epochs [289,289] > 0 2024-06-11T19:15:47.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f8928ff9640 3 client.84242.objecter handle_osd_map decoding full epoch 289 2024-06-11T19:15:47.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f892b7fe640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c1235d0 0x7f892c1239b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f892b7fe640 10 monclient: get_auth_request con 0x7f892c1235d0 auth_method 0 2024-06-11T19:15:47.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f892b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f892b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.657+0000 7f892b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f892b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f892b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f892b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 20 client.84242.objecter dump_active .. 0 homeless 2024-06-11T19:15:47.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f892b7fe640 10 monclient(hunting): handle_auth_done global_id 84203 payload 824 2024-06-11T19:15:47.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f892bfff640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f890c03fdb0 0x7f890c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f892b7fe640 1 -- 172.21.15.98:0/2318446739 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f892c0a6750 con 0x7f892c1235d0 2024-06-11T19:15:47.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f892bfff640 10 monclient: get_auth_request con 0x7f890c03fdb0 auth_method 0 2024-06-11T19:15:47.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 10 client.84242.objecter ms_handle_connect 0x7f892c1235d0 2024-06-11T19:15:47.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 10 client.84242.objecter resend_mon_ops 2024-06-11T19:15:47.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f892b7fe640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c1235d0 0x7f892c1239b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f891c010c60 tx=0x7f891c00e5a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f891c018070 con 0x7f892c1235d0 2024-06-11T19:15:47.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:47.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 10 monclient: dump: 2024-06-11T19:15:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 10 client.84242.objecter ms_handle_connect 0x7f890c03fdb0 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f892bfff640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f890c03fdb0 0x7f890c042270 secure :-1 s=READY pgs=4137 cs=0 l=1 rev1=1 crypto rx=0x7f892c1247f0 tx=0x7f892400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f891c014430 con 0x7f892c1235d0 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c1235d0 msgr2=0x7f892c1239b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c1235d0 0x7f892c1239b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f891c010c60 tx=0x7f891c00e5a0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c1235d0 msgr2=0x7f892c1239b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:47.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.658+0000 7f8928ff9640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c1235d0 0x7f892c1239b0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f891c010c60 tx=0x7f891c00e5a0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f893210b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:47.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f893210b640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c123ef0 0x7f892c09fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f893210b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f893210b640 1 -- 172.21.15.98:0/2318446739 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f892c050830 con 0x7f892c123ef0 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f892b7fe640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c123ef0 0x7f892c09fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f892b7fe640 10 monclient: get_auth_request con 0x7f892c123ef0 auth_method 0 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f892b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f892b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f892b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f892b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f892b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f892b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f892b7fe640 10 monclient(hunting): handle_auth_done global_id 84212 payload 824 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f892b7fe640 1 -- 172.21.15.98:0/2318446739 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f892c138640 con 0x7f892c123ef0 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 10 client.84242.objecter ms_handle_connect 0x7f892c123ef0 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f892b7fe640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c123ef0 0x7f892c09fa00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f891c000c00 tx=0x7f891c0046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 10 client.84242.objecter resend_mon_ops 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f891c010ea0 con 0x7f892c123ef0 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 10 monclient: dump: 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.705 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f891c003e70 con 0x7f892c123ef0 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c123ef0 msgr2=0x7f892c09fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c123ef0 0x7f892c09fa00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f891c000c00 tx=0x7f891c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 1 -- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c123ef0 msgr2=0x7f892c09fa00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:47.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.665+0000 7f8928ff9640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f892c123ef0 0x7f892c09fa00 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f891c000c00 tx=0x7f891c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.707 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 19439, 2024-06-11T19:15:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:47.707 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.708 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.709 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:47.711 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:47.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:47.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.717 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:47.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.722 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:47.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.727 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:47.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:47.729 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 10 librados: watch_flush enter 2024-06-11T19:15:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 10 librados: watch_flush exit 2024-06-11T19:15:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 20 client.84242.objecter shutdown clearing up homeless session... 2024-06-11T19:15:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 10 client.84242.objecter successfully canceled tick 2024-06-11T19:15:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 1 -- 172.21.15.98:0/2318446739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f890c03fdb0 msgr2=0x7f890c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f890c03fdb0 0x7f890c042270 secure :-1 s=READY pgs=4137 cs=0 l=1 rev1=1 crypto rx=0x7f892c1247f0 tx=0x7f892400b040 comp rx=0 tx=0).stop 2024-06-11T19:15:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 10 monclient: shutdown 2024-06-11T19:15:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 1 -- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f892c0a4920 msgr2=0x7f892c1ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f892c0a4920 0x7f892c1ce230 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8914032000 tx=0x7f89140046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f892bfff640 1 -- 172.21.15.98:0/2318446739 reap_dead start 2024-06-11T19:15:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 1 -- 172.21.15.98:0/2318446739 shutdown_connections 2024-06-11T19:15:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 1 --2- 172.21.15.98:0/2318446739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f892c0a4920 0x7f892c1ce230 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 1 -- 172.21.15.98:0/2318446739 >> 172.21.15.98:0/2318446739 conn(0x7f892c0ab850 msgr2=0x7f892c1be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 1 -- 172.21.15.98:0/2318446739 shutdown_connections 2024-06-11T19:15:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 1 -- 172.21.15.98:0/2318446739 wait complete. 2024-06-11T19:15:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.666+0000 7f893210b640 1 librados: shutdown 2024-06-11T19:15:47.732 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:47.732 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:15:47.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:15:47.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.832+0000 7f5b9e442640 10 monclient: get_monmap_and_config 2024-06-11T19:15:47.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.832+0000 7f5b9e442640 10 monclient: build_initial_monmap 2024-06-11T19:15:47.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.832+0000 7f5b9e442640 10 monclient: monmap: 2024-06-11T19:15:47.832 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:47.832 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.832 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:47.833808+0000 2024-06-11T19:15:47.832 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:47.833808+0000 2024-06-11T19:15:47.832 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:47.832 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.832 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:47.833 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:47.833 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:47.833 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:47.833 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:47.833 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:47.833 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:47.833 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:47.833 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:47.833 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:47.833 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:47.833 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.833+0000 7f5b9e442640 1 Processor -- start 2024-06-11T19:15:47.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.833+0000 7f5b9e442640 1 -- start start 2024-06-11T19:15:47.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.833+0000 7f5b9e442640 10 monclient: init 2024-06-11T19:15:47.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.833+0000 7f5b9e442640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:47.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 10 monclient: _add_conns ranks=[8,9,10,6,5,1,3,0,7,2,4] 2024-06-11T19:15:47.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5b980a4920 0x7f5b980a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 10 monclient(hunting): picked mon.noname-e con 0x7f5b980a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5b980a5240 0x7f5b980a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 10 monclient(hunting): picked mon.noname-a con 0x7f5b980a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:15:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5b980a2530 0x7f5b980a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 10 monclient(hunting): picked mon.noname-f con 0x7f5b980a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:15:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5b980a7140 con 0x7f5b980a5240 2024-06-11T19:15:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f5b980a2b50 con 0x7f5b980a2530 2024-06-11T19:15:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5b980a2e30 con 0x7f5b980a4920 2024-06-11T19:15:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:47.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9e442640 10 monclient(hunting): authenticate will time out at 1157.641081s 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b97fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5b980a4920 0x7f5b980a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5b980a5240 0x7f5b980a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5b980a5240 0x7f5b980a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46356/0 (socket says 172.21.15.98:46356) 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b97fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5b980a4920 0x7f5b980a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34880/0 (socket says 172.21.15.98:34880) 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 1 -- 172.21.15.98:0/1432774538 learned_addr learned my addr 172.21.15.98:0/1432774538 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 10 monclient(hunting): get_auth_request con 0x7f5b980a5240 auth_method 0 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b97fff640 10 monclient(hunting): get_auth_request con 0x7f5b980a4920 auth_method 0 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b97fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9c9df640 1 --2- 172.21.15.98:0/1432774538 >> v2:172.21.15.117:3300/0 conn(0x7f5b980a2530 0x7f5b980a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9c9df640 10 monclient(hunting): get_auth_request con 0x7f5b980a2530 auth_method 0 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9c9df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9c9df640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b9c9df640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 10 monclient(hunting): handle_auth_done global_id 64674 payload 274 2024-06-11T19:15:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 1 -- 172.21.15.98:0/1432774538 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5b980a4920 msgr2=0x7f5b980a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 1 --2- 172.21.15.98:0/1432774538 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5b980a4920 0x7f5b980a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 1 -- 172.21.15.98:0/1432774538 >> v2:172.21.15.117:3300/0 conn(0x7f5b980a2530 msgr2=0x7f5b980a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 1 --2- 172.21.15.98:0/1432774538 >> v2:172.21.15.117:3300/0 conn(0x7f5b980a2530 0x7f5b980a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 10 monclient: _finish_hunting 0 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 1 monclient: found mon.noname-a 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b977fe640 1 -- 172.21.15.98:0/1432774538 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b980a3110 con 0x7f5b980a5240 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.834+0000 7f5b97fff640 1 --2- 172.21.15.98:0/1432774538 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5b980a4920 0x7f5b980a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b977fe640 1 --2- 172.21.15.98:0/1432774538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b980a5240 0x7f5b980a59a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5b980abfb0 tx=0x7f5b8c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ae30b00547f418 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 1 -- 172.21.15.98:0/1432774538 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5b8c038070 con 0x7f5b980a5240 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 10 monclient: dump: 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.848 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 10 monclient: _finish_auth 0 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 1 -- 172.21.15.98:0/1432774538 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b8c02f050 con 0x7f5b980a5240 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b9e442640 5 monclient: authenticate success, global_id 64674 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b9e442640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 1 -- 172.21.15.98:0/1432774538 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5b8c031e30 con 0x7f5b980a5240 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b9e442640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b9e442640 4 set_mon_vals no callback set 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b96ffd640 10 monclient: dump: 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.850 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b9e442640 10 monclient: shutdown 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b9e442640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b9e442640 1 -- 172.21.15.98:0/1432774538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b980a5240 msgr2=0x7f5b980a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.835+0000 7f5b9e442640 1 --2- 172.21.15.98:0/1432774538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b980a5240 0x7f5b980a59a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5b980abfb0 tx=0x7f5b8c02b7f0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.836+0000 7f5b9e442640 1 -- 172.21.15.98:0/1432774538 shutdown_connections 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.836+0000 7f5b9e442640 1 --2- 172.21.15.98:0/1432774538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b980a5240 0x7f5b980a59a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.836+0000 7f5b9e442640 1 --2- 172.21.15.98:0/1432774538 >> v2:172.21.15.117:3300/0 conn(0x7f5b980a2530 0x7f5b980a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.836+0000 7f5b9e442640 1 --2- 172.21.15.98:0/1432774538 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5b980a4920 0x7f5b980a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.836+0000 7f5b9e442640 1 -- 172.21.15.98:0/1432774538 >> 172.21.15.98:0/1432774538 conn(0x7f5b980ab850 msgr2=0x7f5b980a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.836+0000 7f5b9e442640 1 -- 172.21.15.98:0/1432774538 shutdown_connections 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.836+0000 7f5b9e442640 1 -- 172.21.15.98:0/1432774538 wait complete. 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.836+0000 7f5b9e442640 10 monclient: build_initial_monmap 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.836+0000 7f5b9e442640 10 monclient: monmap: 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.836+0000 7f5b9e442640 1 librados: starting msgr at 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.836+0000 7f5b9e442640 1 librados: starting objecter 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.836+0000 7f5b9e442640 1 Processor -- start 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 1 -- start start 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 1 librados: setting wanted keys 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 1 librados: calling monclient init 2024-06-11T19:15:47.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 10 monclient: init 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 10 monclient: _add_conns ranks=[6,7,1,8,0,2,3,4,5] 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5b980a2530 0x7f5b981cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 10 monclient(hunting): picked mon.noname-d con 0x7f5b980a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5b980a4920 0x7f5b981ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 10 monclient(hunting): picked mon.noname-i con 0x7f5b980a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b980a5240 0x7f5b98123020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 10 monclient(hunting): picked mon.noname-f con 0x7f5b980a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5b980a6da0 con 0x7f5b980a5240 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5b98138e50 con 0x7f5b980a2530 2024-06-11T19:15:47.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f5b980a6420 con 0x7f5b980a4920 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9e442640 10 monclient(hunting): authenticate will time out at 1157.644594s 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5b980a2530 0x7f5b981cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5b980a2530 0x7f5b981cdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34694/0 (socket says 172.21.15.98:34694) 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 1 -- 172.21.15.98:0/1803575673 learned_addr learned my addr 172.21.15.98:0/1803575673 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 10 monclient(hunting): get_auth_request con 0x7f5b980a2530 auth_method 0 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9c9df640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b980a5240 0x7f5b98123020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9c9df640 10 monclient(hunting): get_auth_request con 0x7f5b980a5240 auth_method 0 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9c9df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9c9df640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9c9df640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b977fe640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5b980a4920 0x7f5b981ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b977fe640 10 monclient(hunting): get_auth_request con 0x7f5b980a4920 auth_method 0 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b977fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b977fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 10 monclient(hunting): handle_auth_done global_id 54960 payload 824 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 1 -- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5b980a4920 msgr2=0x7f5b981ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5b980a4920 0x7f5b981ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 1 -- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b980a5240 msgr2=0x7f5b98123020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b980a5240 0x7f5b98123020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 10 monclient: _finish_hunting 0 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 1 monclient: found mon.noname-d 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b97fff640 1 -- 172.21.15.98:0/1803575673 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b98126bf0 con 0x7f5b980a2530 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.837+0000 7f5b9c9df640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b980a5240 0x7f5b98123020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b97fff640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5b980a2530 0x7f5b981cdbb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5b8800ebe0 tx=0x7f5b8800c520 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 10 client.?.objecter ms_handle_connect 0x7f5b980a2530 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5b88017070 con 0x7f5b980a2530 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 10 monclient: dump: 2024-06-11T19:15:47.857 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.858 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 10 monclient: _finish_auth 0 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b8800cc30 con 0x7f5b980a2530 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b9e442640 5 monclient: authenticate success, global_id 54960 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5b8800cde0 con 0x7f5b980a2530 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b9e442640 10 monclient: _renew_subs 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b9e442640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b9e442640 1 -- 172.21.15.98:0/1803575673 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b98126ed0 con 0x7f5b980a2530 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b967fc640 4 set_mon_vals no callback set 2024-06-11T19:15:47.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b94ff9640 10 monclient: dump: 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.860 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b9e442640 10 client.54960.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b9e442640 10 monclient: _renew_subs 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b9e442640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b9e442640 1 -- 172.21.15.98:0/1803575673 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b98127900 con 0x7f5b980a2530 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.838+0000 7f5b9e442640 1 librados: init done 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f5b8801a070 con 0x7f5b980a2530 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b9e442640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b9e442640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98123d20 0x7f5b9802cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b9e442640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b9e442640 1 -- 172.21.15.98:0/1803575673 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5b98050830 con 0x7f5b98123d20 2024-06-11T19:15:47.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b94ff9640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5b8403fdb0 0x7f5b84042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(289..289 src has 1..289) ==== 6248+0+0 (secure 0 0 0) 0x7f5b8800f040 con 0x7f5b980a2530 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b94ff9640 10 client.54960.objecter ms_dispatch 0x7f5b98000b90 osd_map(289..289 src has 1..289) 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b94ff9640 3 client.54960.objecter handle_osd_map got epochs [289,289] > 0 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b94ff9640 3 client.54960.objecter handle_osd_map decoding full epoch 289 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b977fe640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98123d20 0x7f5b9802cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b977fe640 10 monclient: get_auth_request con 0x7f5b98123d20 auth_method 0 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b977fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b977fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b94ff9640 20 client.54960.objecter dump_active .. 0 homeless 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b9c9df640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5b8403fdb0 0x7f5b84042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b9c9df640 10 monclient: get_auth_request con 0x7f5b8403fdb0 auth_method 0 2024-06-11T19:15:47.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b977fe640 10 monclient(hunting): handle_auth_done global_id 31933 payload 824 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b977fe640 1 -- 172.21.15.98:0/1803575673 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b980a4f90 con 0x7f5b98123d20 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b94ff9640 10 client.54960.objecter ms_handle_connect 0x7f5b8403fdb0 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.839+0000 7f5b9c9df640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5b8403fdb0 0x7f5b84042270 secure :-1 s=READY pgs=4138 cs=0 l=1 rev1=1 crypto rx=0x7f5b98124700 tx=0x7f5b9000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 10 client.54960.objecter ms_handle_connect 0x7f5b98123d20 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 10 client.54960.objecter resend_mon_ops 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b977fe640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98123d20 0x7f5b9802cb40 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f5b980a8420 tx=0x7f5b8c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5b8c043070 con 0x7f5b98123d20 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 10 monclient: dump: 2024-06-11T19:15:47.863 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.864 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.865 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5b8c003ba0 con 0x7f5b98123d20 2024-06-11T19:15:47.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:47.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:47.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98123d20 msgr2=0x7f5b9802cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98123d20 0x7f5b9802cb40 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f5b980a8420 tx=0x7f5b8c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98123d20 msgr2=0x7f5b9802cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:47.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.840+0000 7f5b94ff9640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98123d20 0x7f5b9802cb40 secure :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f5b980a8420 tx=0x7f5b8c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.846+0000 7f5b9e442640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:47.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.846+0000 7f5b9e442640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98124100 0x7f5b9802d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:47.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.846+0000 7f5b9e442640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:47.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.846+0000 7f5b9e442640 1 -- 172.21.15.98:0/1803575673 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5b980a6750 con 0x7f5b98124100 2024-06-11T19:15:47.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b977fe640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98124100 0x7f5b9802d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:47.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b977fe640 10 monclient: get_auth_request con 0x7f5b98124100 auth_method 0 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b977fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b977fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b977fe640 10 monclient(hunting): handle_auth_done global_id 31942 payload 824 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b977fe640 1 -- 172.21.15.98:0/1803575673 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5b980a0100 con 0x7f5b98124100 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b977fe640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98124100 0x7f5b9802d080 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c000c00 tx=0x7f5b8c031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b94ff9640 10 client.54960.objecter ms_handle_connect 0x7f5b98124100 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b94ff9640 10 client.54960.objecter resend_mon_ops 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5b8c038040 con 0x7f5b98124100 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:47.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b94ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.847+0000 7f5b94ff9640 10 monclient: dump: 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:47.869 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:47.870 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:47.870 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:47.870 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:47.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9615 (secure 0 0 0) 0x7f5b8c02f040 con 0x7f5b98124100 2024-06-11T19:15:47.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b94ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:47.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b94ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:47.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98124100 msgr2=0x7f5b9802d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b94ff9640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98124100 0x7f5b9802d080 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c000c00 tx=0x7f5b8c031000 comp rx=0 tx=0).stop 2024-06-11T19:15:47.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b94ff9640 1 -- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98124100 msgr2=0x7f5b9802d080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:47.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b94ff9640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b98124100 0x7f5b9802d080 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 398065, 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:47.871 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.872 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.873 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:47.874 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:47.876 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:47.876 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:47.876 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:47.876 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:47.876 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:47.876 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:47.876 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:47.876 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.876 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:47.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.876 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:47.877 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.880 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.882 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:47.883 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:47.885 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.886 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:47.888 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.888 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.888 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:47.888 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:47.888 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.888 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.888 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:47.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:47.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:47.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:47.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:47.895 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:47.895 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:47.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 10 librados: watch_flush enter 2024-06-11T19:15:47.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 10 librados: watch_flush exit 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 20 client.54960.objecter shutdown clearing up homeless session... 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 10 client.54960.objecter successfully canceled tick 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 1 -- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5b8403fdb0 msgr2=0x7f5b84042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5b8403fdb0 0x7f5b84042270 secure :-1 s=READY pgs=4138 cs=0 l=1 rev1=1 crypto rx=0x7f5b98124700 tx=0x7f5b9000b040 comp rx=0 tx=0).stop 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 10 monclient: shutdown 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 1 -- 172.21.15.98:0/1803575673 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5b980a2530 msgr2=0x7f5b981cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 1 --2- 172.21.15.98:0/1803575673 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5b980a2530 0x7f5b981cdbb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5b8800ebe0 tx=0x7f5b8800c520 comp rx=0 tx=0).stop 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9c9df640 1 -- 172.21.15.98:0/1803575673 reap_dead start 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9c9df640 1 -- 172.21.15.98:0/1803575673 reap_dead start 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 1 -- 172.21.15.98:0/1803575673 shutdown_connections 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 1 -- 172.21.15.98:0/1803575673 >> 172.21.15.98:0/1803575673 conn(0x7f5b980ab850 msgr2=0x7f5b981bf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:47.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 1 -- 172.21.15.98:0/1803575673 shutdown_connections 2024-06-11T19:15:47.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 1 -- 172.21.15.98:0/1803575673 wait complete. 2024-06-11T19:15:47.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.848+0000 7f5b9e442640 1 librados: shutdown 2024-06-11T19:15:47.898 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:47.898 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:15:47.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.998+0000 7fd3efbe2640 10 monclient: get_monmap_and_config 2024-06-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.998+0000 7fd3efbe2640 10 monclient: build_initial_monmap 2024-06-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.998+0000 7fd3efbe2640 10 monclient: monmap: 2024-06-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:47.999606+0000 2024-06-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:47.999606+0000 2024-06-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:47.999 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 1 Processor -- start 2024-06-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 1 -- start start 2024-06-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 10 monclient: init 2024-06-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 10 monclient: _add_conns ranks=[9,10,8,7,0,6,1,2,3,4,5] 2024-06-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3e80a4920 0x7fd3e80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 10 monclient(hunting): picked mon.noname-a con 0x7fd3e80a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd3e80a5240 0x7fd3e80a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 10 monclient(hunting): picked mon.noname-f con 0x7fd3e80a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3e80a2530 0x7fd3e80a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 10 monclient(hunting): picked mon.noname-e con 0x7fd3e80a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd3e80a7140 con 0x7fd3e80a4920 2024-06-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fd3e80a2b50 con 0x7fd3e80a5240 2024-06-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd3e80a2e30 con 0x7fd3e80a2530 2024-06-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3efbe2640 10 monclient(hunting): authenticate will time out at 1157.806840s 2024-06-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3ed97e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3e80a4920 0x7fd3e80a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3ee17f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3e80a2530 0x7fd3e80a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3ed97e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3e80a4920 0x7fd3e80a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46362/0 (socket says 172.21.15.98:46362) 2024-06-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3ee17f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3e80a2530 0x7fd3e80a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34884/0 (socket says 172.21.15.98:34884) 2024-06-11T19:15:48.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3ed97e640 1 -- 172.21.15.98:0/1233311848 learned_addr learned my addr 172.21.15.98:0/1233311848 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:47.999+0000 7fd3ed97e640 10 monclient(hunting): get_auth_request con 0x7fd3e80a4920 auth_method 0 2024-06-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ed97e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ed97e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ed97e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 10 monclient(hunting): get_auth_request con 0x7fd3e80a2530 auth_method 0 2024-06-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 10 monclient(hunting): handle_auth_done global_id 84230 payload 274 2024-06-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 1 -- 172.21.15.98:0/1233311848 >> v2:172.21.15.117:3300/0 conn(0x7fd3e80a5240 msgr2=0x7fd3e80a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 1 --2- 172.21.15.98:0/1233311848 >> v2:172.21.15.117:3300/0 conn(0x7fd3e80a5240 0x7fd3e80a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 1 -- 172.21.15.98:0/1233311848 >> v2:172.21.15.98:3300/0 conn(0x7fd3e80a4920 msgr2=0x7fd3e80a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 1 --2- 172.21.15.98:0/1233311848 >> v2:172.21.15.98:3300/0 conn(0x7fd3e80a4920 0x7fd3e80a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 10 monclient: _finish_hunting 0 2024-06-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 1 monclient: found mon.noname-e 2024-06-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ee17f640 1 -- 172.21.15.98:0/1233311848 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e80a3110 con 0x7fd3e80a2530 2024-06-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.000+0000 7fd3ed97e640 1 --2- 172.21.15.98:0/1233311848 >> v2:172.21.15.98:3300/0 conn(0x7fd3e80a4920 0x7fd3e80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ee17f640 1 --2- 172.21.15.98:0/1233311848 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3e80a2530 0x7fd3e80a5ee0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8073fc0 tx=0x7fd3e002bd00 comp rx=0 tx=0).ready entity=mon.8 client_cookie=4c1019cdb9dcf285 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 1 -- 172.21.15.98:0/1233311848 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3e0039070 con 0x7fd3e80a2530 2024-06-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 10 monclient: dump: 2024-06-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.015 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 10 monclient: _finish_auth 0 2024-06-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 10 monclient: _check_auth_tickets 2024-06-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 1 -- 172.21.15.98:0/1233311848 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3e002cd20 con 0x7fd3e80a2530 2024-06-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 5 monclient: authenticate success, global_id 84230 2024-06-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 1 -- 172.21.15.98:0/1233311848 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3e00327b0 con 0x7fd3e80a2530 2024-06-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 4 set_mon_vals no callback set 2024-06-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:48.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3ec97c640 10 monclient: dump: 2024-06-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.018 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 10 monclient: shutdown 2024-06-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 1 -- 172.21.15.98:0/1233311848 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3e80a2530 msgr2=0x7fd3e80a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 1 --2- 172.21.15.98:0/1233311848 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3e80a2530 0x7fd3e80a5ee0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8073fc0 tx=0x7fd3e002bd00 comp rx=0 tx=0).stop 2024-06-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 1 -- 172.21.15.98:0/1233311848 shutdown_connections 2024-06-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 1 --2- 172.21.15.98:0/1233311848 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3e80a2530 0x7fd3e80a5ee0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 1 --2- 172.21.15.98:0/1233311848 >> v2:172.21.15.117:3300/0 conn(0x7fd3e80a5240 0x7fd3e80a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 1 --2- 172.21.15.98:0/1233311848 >> v2:172.21.15.98:3300/0 conn(0x7fd3e80a4920 0x7fd3e80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 1 -- 172.21.15.98:0/1233311848 >> 172.21.15.98:0/1233311848 conn(0x7fd3e80ab850 msgr2=0x7fd3e80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:48.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 1 -- 172.21.15.98:0/1233311848 shutdown_connections 2024-06-11T19:15:48.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.001+0000 7fd3efbe2640 1 -- 172.21.15.98:0/1233311848 wait complete. 2024-06-11T19:15:48.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.002+0000 7fd3efbe2640 10 monclient: build_initial_monmap 2024-06-11T19:15:48.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.002+0000 7fd3efbe2640 10 monclient: monmap: 2024-06-11T19:15:48.020 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:48.020 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:48.020 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:48.020 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:48.020 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:48.020 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:48.021 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:48.021 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:48.021 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:48.021 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:48.021 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:48.021 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:48.021 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:48.021 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:48.021 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:48.021 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.002+0000 7fd3efbe2640 1 librados: starting msgr at 2024-06-11T19:15:48.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.002+0000 7fd3efbe2640 1 librados: starting objecter 2024-06-11T19:15:48.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.002+0000 7fd3efbe2640 1 Processor -- start 2024-06-11T19:15:48.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.002+0000 7fd3efbe2640 1 -- start start 2024-06-11T19:15:48.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.002+0000 7fd3efbe2640 1 librados: setting wanted keys 2024-06-11T19:15:48.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.002+0000 7fd3efbe2640 1 librados: calling monclient init 2024-06-11T19:15:48.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.002+0000 7fd3efbe2640 10 monclient: init 2024-06-11T19:15:48.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:48.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 10 monclient: _add_conns ranks=[2,8,6,4,7,0,1,3,5] 2024-06-11T19:15:48.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3e80a2530 0x7fd3e81cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 10 monclient(hunting): picked mon.noname-b con 0x7fd3e80a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:48.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3e80a4920 0x7fd3e81ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 10 monclient(hunting): picked mon.noname-e con 0x7fd3e80a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:48.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3e80a5240 0x7fd3e8123030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 10 monclient(hunting): picked mon.noname-d con 0x7fd3e80a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:48.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd3e80a6da0 con 0x7fd3e80a2530 2024-06-11T19:15:48.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd3e8138e50 con 0x7fd3e80a5240 2024-06-11T19:15:48.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd3e80a6420 con 0x7fd3e80a4920 2024-06-11T19:15:48.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:48.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3efbe2640 10 monclient(hunting): authenticate will time out at 1157.810331s 2024-06-11T19:15:48.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed97e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3e80a2530 0x7fd3e81cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3e80a4920 0x7fd3e81ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed97e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3e80a2530 0x7fd3e81cdbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:59902/0 (socket says 172.21.15.98:59902) 2024-06-11T19:15:48.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed97e640 1 -- 172.21.15.98:0/571697832 learned_addr learned my addr 172.21.15.98:0/571697832 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed97e640 10 monclient(hunting): get_auth_request con 0x7fd3e80a2530 auth_method 0 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed97e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed97e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed97e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 10 monclient(hunting): get_auth_request con 0x7fd3e80a4920 auth_method 0 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ee17f640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3e80a5240 0x7fd3e8123030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ee17f640 10 monclient(hunting): get_auth_request con 0x7fd3e80a5240 auth_method 0 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ee17f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ee17f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ee17f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ee17f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ee17f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ee17f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 10 monclient(hunting): handle_auth_done global_id 84239 payload 824 2024-06-11T19:15:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 1 -- 172.21.15.98:0/571697832 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3e80a5240 msgr2=0x7fd3e8123030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd3e80a5240 0x7fd3e8123030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 1 -- 172.21.15.98:0/571697832 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3e80a2530 msgr2=0x7fd3e81cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3e80a2530 0x7fd3e81cdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 10 monclient: _finish_hunting 0 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 1 monclient: found mon.noname-e 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 1 -- 172.21.15.98:0/571697832 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e8126bf0 con 0x7fd3e80a4920 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed97e640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3e80a2530 0x7fd3e81cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3ed17d640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3e80a4920 0x7fd3e81ce100 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800b410 tx=0x7fd3d800b8d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3de7fc640 10 client.?.objecter ms_handle_connect 0x7fd3e80a4920 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.003+0000 7fd3de7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3d8015020 con 0x7fd3e80a4920 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 10 monclient: dump: 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.028 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 10 monclient: _finish_auth 0 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3d8011aa0 con 0x7fd3e80a4920 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3efbe2640 5 monclient: authenticate success, global_id 84239 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3efbe2640 10 monclient: _renew_subs 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3d8011c50 con 0x7fd3e80a4920 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3efbe2640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3efbe2640 1 -- 172.21.15.98:0/571697832 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e80a4f90 con 0x7fd3e80a4920 2024-06-11T19:15:48.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3dffff640 4 set_mon_vals no callback set 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3de7fc640 10 monclient: dump: 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.030 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3efbe2640 10 client.84239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3efbe2640 10 monclient: _renew_subs 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3efbe2640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3efbe2640 1 -- 172.21.15.98:0/571697832 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e81256d0 con 0x7fd3e80a4920 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.004+0000 7fd3efbe2640 1 librados: init done 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3efbe2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3efbe2640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123570 0x7fd3e8123950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3efbe2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3efbe2640 1 -- 172.21.15.98:0/571697832 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd3e8050830 con 0x7fd3e8123570 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd3d8018020 con 0x7fd3e80a4920 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ed97e640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123570 0x7fd3e8123950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ed97e640 10 monclient: get_auth_request con 0x7fd3e8123570 auth_method 0 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ed97e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ed97e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ed97e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3cc03fdb0 0x7fd3cc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(290..290 src has 1..290) ==== 6577+0+0 (secure 0 0 0) 0x7fd3d8051750 con 0x7fd3e80a4920 2024-06-11T19:15:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 10 client.84239.objecter ms_dispatch 0x7fd3e8000b90 osd_map(290..290 src has 1..290) 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 3 client.84239.objecter handle_osd_map got epochs [290,290] > 0 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 3 client.84239.objecter handle_osd_map decoding full epoch 290 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 20 client.84239.objecter dump_active .. 0 homeless 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_done global_id 84207 payload 824 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ee17f640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3cc03fdb0 0x7fd3cc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ed97e640 1 -- 172.21.15.98:0/571697832 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3e80a66c0 con 0x7fd3e8123570 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ee17f640 10 monclient: get_auth_request con 0x7fd3cc03fdb0 auth_method 0 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 10 client.84239.objecter ms_handle_connect 0x7fd3e8123570 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 10 client.84239.objecter resend_mon_ops 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ed97e640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123570 0x7fd3e8123950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009f20 tx=0x7fd3e400f610 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3e40044a0 con 0x7fd3e8123570 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 10 monclient: dump: 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.034 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.035 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3de7fc640 10 client.84239.objecter ms_handle_connect 0x7fd3cc03fdb0 2024-06-11T19:15:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.005+0000 7fd3ee17f640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3cc03fdb0 0x7fd3cc042270 secure :-1 s=READY pgs=4139 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0036000 tx=0x7fd3e0030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.006+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd3e4004030 con 0x7fd3e8123570 2024-06-11T19:15:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.006+0000 7fd3de7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.006+0000 7fd3de7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.006+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123570 msgr2=0x7fd3e8123950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.006+0000 7fd3de7fc640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123570 0x7fd3e8123950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009f20 tx=0x7fd3e400f610 comp rx=0 tx=0).stop 2024-06-11T19:15:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.006+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123570 msgr2=0x7fd3e8123950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:48.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.006+0000 7fd3de7fc640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123570 0x7fd3e8123950 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009f20 tx=0x7fd3e400f610 comp rx=0 tx=0).stop 2024-06-11T19:15:48.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.012+0000 7fd3efbe2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:48.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.012+0000 7fd3efbe2640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123e90 0x7fd3e809e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.012+0000 7fd3efbe2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.012+0000 7fd3efbe2640 1 -- 172.21.15.98:0/571697832 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd3e8126630 con 0x7fd3e8123e90 2024-06-11T19:15:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.012+0000 7fd3ed97e640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123e90 0x7fd3e809e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.012+0000 7fd3ed97e640 10 monclient: get_auth_request con 0x7fd3e8123e90 auth_method 0 2024-06-11T19:15:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.012+0000 7fd3ed97e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.012+0000 7fd3ed97e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.012+0000 7fd3ed97e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3ed97e640 10 monclient(hunting): handle_auth_done global_id 84216 payload 824 2024-06-11T19:15:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3ed97e640 1 -- 172.21.15.98:0/571697832 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd3e8127900 con 0x7fd3e8123e90 2024-06-11T19:15:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 10 client.84239.objecter ms_handle_connect 0x7fd3e8123e90 2024-06-11T19:15:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 10 client.84239.objecter resend_mon_ops 2024-06-11T19:15:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3ed97e640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123e90 0x7fd3e809e9f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009f20 tx=0x7fd3e40046c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3e4003c90 con 0x7fd3e8123e90 2024-06-11T19:15:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:15:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 10 monclient: dump: 2024-06-11T19:15:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fd3e4003820 con 0x7fd3e8123e90 2024-06-11T19:15:48.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123e90 msgr2=0x7fd3e809e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123e90 0x7fd3e809e9f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009f20 tx=0x7fd3e40046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 1 -- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123e90 msgr2=0x7fd3e809e9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:48.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.013+0000 7fd3de7fc640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3e8123e90 0x7fd3e809e9f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.041 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:48.041 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 19787, 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.042 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:15:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:48.044 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:48.045 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:48.047 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.048 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:48.049 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.050 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.051 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:48.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:48.057 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:48.058 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.059 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:48.060 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.062 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:48.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:48.062 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.062 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.062 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:48.062 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:48.063 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.063 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.063 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:48.063 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:48.063 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.063 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.063 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.063 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.063 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.064 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:48.064 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:48.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.064 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:48.064 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:48.064 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:48.064 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.064 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.064 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:48.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.064 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:48.065 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:48.065 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:48.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.065 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.065 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:48.065 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 10 librados: watch_flush enter 2024-06-11T19:15:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 10 librados: watch_flush exit 2024-06-11T19:15:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 20 client.84239.objecter shutdown clearing up homeless session... 2024-06-11T19:15:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 10 client.84239.objecter successfully canceled tick 2024-06-11T19:15:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 1 -- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3cc03fdb0 msgr2=0x7fd3cc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3cc03fdb0 0x7fd3cc042270 secure :-1 s=READY pgs=4139 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0036000 tx=0x7fd3e0030000 comp rx=0 tx=0).stop 2024-06-11T19:15:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 10 monclient: shutdown 2024-06-11T19:15:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 1 -- 172.21.15.98:0/571697832 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3e80a4920 msgr2=0x7fd3e81ce100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 1 --2- 172.21.15.98:0/571697832 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd3e80a4920 0x7fd3e81ce100 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800b410 tx=0x7fd3d800b8d0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3ee17f640 1 -- 172.21.15.98:0/571697832 reap_dead start 2024-06-11T19:15:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3ee17f640 1 -- 172.21.15.98:0/571697832 reap_dead start 2024-06-11T19:15:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 1 -- 172.21.15.98:0/571697832 shutdown_connections 2024-06-11T19:15:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 1 -- 172.21.15.98:0/571697832 >> 172.21.15.98:0/571697832 conn(0x7fd3e80ab850 msgr2=0x7fd3e81be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 1 -- 172.21.15.98:0/571697832 shutdown_connections 2024-06-11T19:15:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 1 -- 172.21.15.98:0/571697832 wait complete. 2024-06-11T19:15:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.014+0000 7fd3efbe2640 1 librados: shutdown 2024-06-11T19:15:48.069 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:48.069 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:15:48.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:15:48.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.168+0000 7f12c15e3640 10 monclient: get_monmap_and_config 2024-06-11T19:15:48.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.168+0000 7f12c15e3640 10 monclient: build_initial_monmap 2024-06-11T19:15:48.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.168+0000 7f12c15e3640 10 monclient: monmap: 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:48.169876+0000 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:48.169876+0000 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:48.169 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:48.170 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:48.170 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:48.170 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:48.170 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:48.170 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.169+0000 7f12c15e3640 1 Processor -- start 2024-06-11T19:15:48.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.169+0000 7f12c15e3640 1 -- start start 2024-06-11T19:15:48.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.169+0000 7f12c15e3640 10 monclient: init 2024-06-11T19:15:48.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.169+0000 7f12c15e3640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:48.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.169+0000 7f12c15e3640 10 monclient: _add_conns ranks=[1,0,10,2,6,8,5,4,9,3,7] 2024-06-11T19:15:48.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.169+0000 7f12c15e3640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f12bc0a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:48.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12c15e3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12bc0a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:15:48.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12c15e3640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f12bc13f380 0x7f12bc141770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12c15e3640 10 monclient(hunting): picked mon.noname-f con 0x7f12bc13f380 addr v2:172.21.15.117:3300/0 2024-06-11T19:15:48.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12c15e3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12bc0a7140 con 0x7f12bc0a2530 2024-06-11T19:15:48.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12c15e3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12bc0a2b50 con 0x7f12bc0a4920 2024-06-11T19:15:48.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12c15e3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12c15e3640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f12bc0a2e30 con 0x7f12bc13f380 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12c15e3640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12c15e3640 10 monclient(hunting): authenticate will time out at 1157.977101s 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12b27fc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f12bc0a2530 0x7f12bc13dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35748/0 (socket says 172.21.15.98:35748) 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12b27fc640 1 -- 172.21.15.98:0/2229969636 learned_addr learned my addr 172.21.15.98:0/2229969636 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 1 --2- 172.21.15.98:0/2229969636 >> v2:172.21.15.117:3300/0 conn(0x7f12bc13f380 0x7f12bc141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 10 monclient(hunting): get_auth_request con 0x7f12bc13f380 auth_method 0 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12ba7fc640 1 -- 172.21.15.98:0/2229969636 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2600404831 0 0) 0x7f12bc0a7140 con 0x7f12bc0a2530 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12ba7fc640 10 monclient(hunting): _negotiate 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12ba7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12ba7fc640 10 monclient(hunting): my global_id is 64692 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12ba7fc640 1 -- 172.21.15.98:0/2229969636 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12a8003520 con 0x7f12bc0a2530 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 10 monclient(hunting): handle_auth_done global_id 31951 payload 274 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 1 -- 172.21.15.98:0/2229969636 >> v1:172.21.15.117:6789/0 conn(0x7f12bc0a4920 legacy=0x7f12bc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 1 -- 172.21.15.98:0/2229969636 >> v1:172.21.15.98:6789/0 conn(0x7f12bc0a2530 legacy=0x7f12bc13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 1 monclient: found mon.noname-f 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:15:48.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.170+0000 7f12bb7fe640 1 -- 172.21.15.98:0/2229969636 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12bc0a3110 con 0x7f12bc13f380 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12bb7fe640 1 --2- 172.21.15.98:0/2229969636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12bc13f380 0x7f12bc141770 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009d90 tx=0x7f12b402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1f068d3752651f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 1 -- 172.21.15.98:0/2229969636 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12b4039070 con 0x7f12bc13f380 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 10 monclient: dump: 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.183 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 10 monclient: _finish_auth 0 2024-06-11T19:15:48.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12c15e3640 5 monclient: authenticate success, global_id 31951 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12c15e3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 1 -- 172.21.15.98:0/2229969636 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12b402ce40 con 0x7f12bc13f380 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 1 -- 172.21.15.98:0/2229969636 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12b40327b0 con 0x7f12bc13f380 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12c15e3640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12c15e3640 4 set_mon_vals no callback set 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.171+0000 7f12ba7fc640 10 monclient: dump: 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.185 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.186 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 10 monclient: shutdown 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 1 -- 172.21.15.98:0/2229969636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12bc13f380 msgr2=0x7f12bc141770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 1 --2- 172.21.15.98:0/2229969636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12bc13f380 0x7f12bc141770 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009d90 tx=0x7f12b402be20 comp rx=0 tx=0).stop 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 1 -- 172.21.15.98:0/2229969636 shutdown_connections 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 1 --2- 172.21.15.98:0/2229969636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12bc13f380 0x7f12bc141770 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 1 -- 172.21.15.98:0/2229969636 >> 172.21.15.98:0/2229969636 conn(0x7f12bc0ab850 msgr2=0x7f12bc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 1 -- 172.21.15.98:0/2229969636 shutdown_connections 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 1 -- 172.21.15.98:0/2229969636 wait complete. 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 10 monclient: build_initial_monmap 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 10 monclient: monmap: 2024-06-11T19:15:48.187 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:48.188 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 1 librados: starting msgr at 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.172+0000 7f12c15e3640 1 librados: starting objecter 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 1 Processor -- start 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 1 -- start start 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 1 librados: setting wanted keys 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 1 librados: calling monclient init 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 10 monclient: init 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 10 monclient: _add_conns ranks=[5,3,0,2,7,6,4,1,8] 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc0a2530 0x7f12bc1cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 10 monclient(hunting): picked mon.noname-h con 0x7f12bc0a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:48.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12bc0a4920 0x7f12bc137d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 10 monclient(hunting): picked mon.noname-g con 0x7f12bc0a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc13f380 0x7f12bc09fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 10 monclient(hunting): picked mon.noname-a con 0x7f12bc13f380 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f12bc138e50 con 0x7f12bc13f380 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f12bc0a84b0 con 0x7f12bc0a4920 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f12bc0a6da0 con 0x7f12bc0a2530 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12c15e3640 10 monclient(hunting): authenticate will time out at 1157.980744s 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12bb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc13f380 0x7f12bc09fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12bb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc13f380 0x7f12bc09fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46374/0 (socket says 172.21.15.98:46374) 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12bb7fe640 1 -- 172.21.15.98:0/4273295075 learned_addr learned my addr 172.21.15.98:0/4273295075 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12bb7fe640 10 monclient(hunting): get_auth_request con 0x7f12bc13f380 auth_method 0 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12bb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12b27fc640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12bc0a4920 0x7f12bc137d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b27fc640 10 monclient(hunting): get_auth_request con 0x7f12bc0a4920 auth_method 0 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.173+0000 7f12baffd640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc0a2530 0x7f12bc1cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12baffd640 10 monclient(hunting): get_auth_request con 0x7f12bc0a2530 auth_method 0 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12baffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12baffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12bb7fe640 10 monclient(hunting): handle_auth_done global_id 64701 payload 824 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12bb7fe640 1 -- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc0a2530 msgr2=0x7f12bc1cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12bb7fe640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc0a2530 0x7f12bc1cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12bb7fe640 1 -- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12bc0a4920 msgr2=0x7f12bc137d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12bb7fe640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12bc0a4920 0x7f12bc137d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12bb7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12bb7fe640 1 monclient: found mon.noname-a 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:48.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12bb7fe640 1 -- 172.21.15.98:0/4273295075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12bc13eb70 con 0x7f12bc13f380 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b27fc640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12bc0a4920 0x7f12bc137d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12bb7fe640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc13f380 0x7f12bc09fc80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009eb0 tx=0x7f12b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 10 client.?.objecter ms_handle_connect 0x7f12bc13f380 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12b4043030 con 0x7f12bc13f380 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 10 monclient: dump: 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.193 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 10 monclient: _finish_auth 0 2024-06-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 10 monclient: _check_auth_tickets 2024-06-11T19:15:48.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12b40044b0 con 0x7f12bc13f380 2024-06-11T19:15:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12c15e3640 5 monclient: authenticate success, global_id 64701 2024-06-11T19:15:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12c15e3640 10 monclient: _renew_subs 2024-06-11T19:15:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12c15e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12c15e3640 1 -- 172.21.15.98:0/4273295075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12bc0a44f0 con 0x7f12bc13f380 2024-06-11T19:15:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12b4039040 con 0x7f12bc13f380 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b3fff640 10 monclient: dump: 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.200 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12c15e3640 10 client.64701.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12c15e3640 10 monclient: _renew_subs 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12c15e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12c15e3640 1 -- 172.21.15.98:0/4273295075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12bc1233e0 con 0x7f12bc13f380 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12b9ffb640 4 set_mon_vals no callback set 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.174+0000 7f12c15e3640 1 librados: init done 2024-06-11T19:15:48.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12c15e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12c15e3640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc138260 0x7f12bc02cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12c15e3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12c15e3640 1 -- 172.21.15.98:0/4273295075 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f12bc050830 con 0x7f12bc138260 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f12b4046070 con 0x7f12bc13f380 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12baffd640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc138260 0x7f12bc02cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12baffd640 10 monclient: get_auth_request con 0x7f12bc138260 auth_method 0 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12baffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12baffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12b3fff640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f129c03fdb0 0x7f129c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(290..290 src has 1..290) ==== 6577+0+0 (secure 0 0 0) 0x7f12b4044070 con 0x7f12bc13f380 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12b3fff640 10 client.64701.objecter ms_dispatch 0x7f12bc000b90 osd_map(290..290 src has 1..290) 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12b3fff640 3 client.64701.objecter handle_osd_map got epochs [290,290] > 0 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12b3fff640 3 client.64701.objecter handle_osd_map decoding full epoch 290 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12baffd640 10 monclient(hunting): handle_auth_done global_id 22865 payload 824 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12b3fff640 20 client.64701.objecter dump_active .. 0 homeless 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12baffd640 1 -- 172.21.15.98:0/4273295075 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12bc13f0c0 con 0x7f12bc138260 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12b27fc640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f129c03fdb0 0x7f129c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.175+0000 7f12b27fc640 10 monclient: get_auth_request con 0x7f129c03fdb0 auth_method 0 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 10 client.64701.objecter ms_handle_connect 0x7f12bc138260 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 10 client.64701.objecter resend_mon_ops 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12baffd640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc138260 0x7f12bc02cb40 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00e420 tx=0x7f12ac00e8e0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12ac0052e0 con 0x7f12bc138260 2024-06-11T19:15:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:15:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 10 monclient: dump: 2024-06-11T19:15:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 10 client.64701.objecter ms_handle_connect 0x7f129c03fdb0 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b27fc640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f129c03fdb0 0x7f129c042270 secure :-1 s=READY pgs=4140 cs=0 l=1 rev1=1 crypto rx=0x7f12a4007fd0 tx=0x7f12a4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f12ac018070 con 0x7f12bc138260 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc138260 msgr2=0x7f12bc02cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc138260 0x7f12bc02cb40 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00e420 tx=0x7f12ac00e8e0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc138260 msgr2=0x7f12bc02cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:48.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.176+0000 7f12b3fff640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc138260 0x7f12bc02cb40 secure :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00e420 tx=0x7f12ac00e8e0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.182+0000 7f12c15e3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.182+0000 7f12c15e3640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc02d080 0x7f12bc02d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.182+0000 7f12c15e3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.182+0000 7f12c15e3640 1 -- 172.21.15.98:0/4273295075 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f12bc0a4d00 con 0x7f12bc02d080 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12baffd640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc02d080 0x7f12bc02d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12baffd640 10 monclient: get_auth_request con 0x7f12bc02d080 auth_method 0 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12baffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12baffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12baffd640 10 monclient(hunting): handle_auth_done global_id 22874 payload 824 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12baffd640 1 -- 172.21.15.98:0/4273295075 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f12bc13cf10 con 0x7f12bc02d080 2024-06-11T19:15:48.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12baffd640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc02d080 0x7f12bc02d490 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00e420 tx=0x7f12ac001120 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12b3fff640 10 client.64701.objecter ms_handle_connect 0x7f12bc02d080 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12b3fff640 10 client.64701.objecter resend_mon_ops 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12ac003c70 con 0x7f12bc02d080 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12b3fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.183+0000 7f12b3fff640 10 monclient: dump: 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f12ac003800 con 0x7f12bc02d080 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12b3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12b3fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc02d080 msgr2=0x7f12bc02d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12b3fff640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc02d080 0x7f12bc02d490 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00e420 tx=0x7f12ac001120 comp rx=0 tx=0).stop 2024-06-11T19:15:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12b3fff640 1 -- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc02d080 msgr2=0x7f12bc02d490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12b3fff640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12bc02d080 0x7f12bc02d490 secure :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00e420 tx=0x7f12ac001120 comp rx=0 tx=0).stop 2024-06-11T19:15:48.211 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 398378, 2024-06-11T19:15:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:48.211 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:48.212 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:48.213 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:48.214 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:48.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:48.218 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:48.223 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:48.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.232 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:48.233 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 10 librados: watch_flush enter 2024-06-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 10 librados: watch_flush exit 2024-06-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 20 client.64701.objecter shutdown clearing up homeless session... 2024-06-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 10 client.64701.objecter successfully canceled tick 2024-06-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 1 -- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f129c03fdb0 msgr2=0x7f129c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f129c03fdb0 0x7f129c042270 secure :-1 s=READY pgs=4140 cs=0 l=1 rev1=1 crypto rx=0x7f12a4007fd0 tx=0x7f12a4009040 comp rx=0 tx=0).stop 2024-06-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 10 monclient: shutdown 2024-06-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 1 -- 172.21.15.98:0/4273295075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc13f380 msgr2=0x7f12bc09fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 1 --2- 172.21.15.98:0/4273295075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc13f380 0x7f12bc09fc80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009eb0 tx=0x7f12b4004a40 comp rx=0 tx=0).stop 2024-06-11T19:15:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12bb7fe640 1 -- 172.21.15.98:0/4273295075 reap_dead start 2024-06-11T19:15:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12bb7fe640 1 -- 172.21.15.98:0/4273295075 reap_dead start 2024-06-11T19:15:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 1 -- 172.21.15.98:0/4273295075 shutdown_connections 2024-06-11T19:15:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 1 -- 172.21.15.98:0/4273295075 >> 172.21.15.98:0/4273295075 conn(0x7f12bc0ab850 msgr2=0x7f12bc0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 1 -- 172.21.15.98:0/4273295075 shutdown_connections 2024-06-11T19:15:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 1 -- 172.21.15.98:0/4273295075 wait complete. 2024-06-11T19:15:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.184+0000 7f12c15e3640 1 librados: shutdown 2024-06-11T19:15:48.236 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:48.236 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:15:48.237 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:15:48.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.337+0000 7f616723e640 10 monclient: get_monmap_and_config 2024-06-11T19:15:48.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.337+0000 7f616723e640 10 monclient: build_initial_monmap 2024-06-11T19:15:48.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.337+0000 7f616723e640 10 monclient: monmap: 2024-06-11T19:15:48.337 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:48.337 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.337 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:48.338472+0000 2024-06-11T19:15:48.337 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:48.338472+0000 2024-06-11T19:15:48.337 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:48.337 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.337 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:48.337 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:48.337 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:48.337 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:48.338 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:48.338 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:48.338 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:48.338 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:48.338 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:48.338 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:48.338 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:48.338 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 1 Processor -- start 2024-06-11T19:15:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 1 -- start start 2024-06-11T19:15:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 10 monclient: init 2024-06-11T19:15:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 10 monclient: _add_conns ranks=[10,8,4,7,5,1,6,2,9,0,3] 2024-06-11T19:15:48.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f61600a4920 0x7f61600a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 10 monclient(hunting): picked mon.noname-f con 0x7f61600a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61600a5240 0x7f61600a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 10 monclient(hunting): picked mon.noname-e con 0x7f61600a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f61600a2530 0x7f61600a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 10 monclient(hunting): picked mon.noname-c con 0x7f61600a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f61600a7140 con 0x7f61600a4920 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f61600a2b50 con 0x7f61600a2530 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f61600a2e30 con 0x7f61600a5240 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f616723e640 10 monclient(hunting): authenticate will time out at 1158.145709s 2024-06-11T19:15:48.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f615ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61600a5240 0x7f61600a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f61657db640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f61600a2530 0x7f61600a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f615ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61600a5240 0x7f61600a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34900/0 (socket says 172.21.15.98:34900) 2024-06-11T19:15:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f615ffff640 1 -- 172.21.15.98:0/277167998 learned_addr learned my addr 172.21.15.98:0/277167998 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f61657db640 10 monclient(hunting): get_auth_request con 0x7f61600a2530 auth_method 0 2024-06-11T19:15:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f61657db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f61657db640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f61657db640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f615ffff640 10 monclient(hunting): get_auth_request con 0x7f61600a5240 auth_method 0 2024-06-11T19:15:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f615ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f615ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f615ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f615ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f615ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f615ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f61657db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f61657db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.338+0000 7f61657db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f6164fda640 1 --2- 172.21.15.98:0/277167998 >> v2:172.21.15.117:3300/0 conn(0x7f61600a4920 0x7f61600a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f6164fda640 10 monclient(hunting): get_auth_request con 0x7f61600a4920 auth_method 0 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f6164fda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f6164fda640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f6164fda640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f61657db640 10 monclient(hunting): handle_auth_done global_id 23791 payload 274 2024-06-11T19:15:48.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f61657db640 1 -- 172.21.15.98:0/277167998 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61600a5240 msgr2=0x7f61600a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f61657db640 1 --2- 172.21.15.98:0/277167998 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61600a5240 0x7f61600a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f61657db640 1 -- 172.21.15.98:0/277167998 >> v2:172.21.15.117:3300/0 conn(0x7f61600a4920 msgr2=0x7f61600a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f61657db640 1 --2- 172.21.15.98:0/277167998 >> v2:172.21.15.117:3300/0 conn(0x7f61600a4920 0x7f61600a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f61657db640 10 monclient: _finish_hunting 0 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f61657db640 1 monclient: found mon.noname-c 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f61657db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f61657db640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f61657db640 1 -- 172.21.15.98:0/277167998 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61600a3110 con 0x7f61600a2530 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f615ffff640 1 --2- 172.21.15.98:0/277167998 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61600a5240 0x7f61600a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f61657db640 1 --2- 172.21.15.98:0/277167998 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f61600a2530 0x7f61600a5ee0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f6158009d90 tx=0x7f615802bce0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=a71ba7f774f4eca5 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f615f7fe640 1 -- 172.21.15.98:0/277167998 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6158039070 con 0x7f61600a2530 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.339+0000 7f615f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f615f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:15:48.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f615f7fe640 10 monclient: dump: 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.353 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f615f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f615f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f615f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f615f7fe640 1 -- 172.21.15.98:0/277167998 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f615802cd00 con 0x7f61600a2530 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 5 monclient: authenticate success, global_id 23791 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f615f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f615f7fe640 1 -- 172.21.15.98:0/277167998 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f61580327b0 con 0x7f61600a2530 2024-06-11T19:15:48.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f615f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 4 set_mon_vals no callback set 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f615f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f615f7fe640 10 monclient: dump: 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 10 monclient: shutdown 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 1 -- 172.21.15.98:0/277167998 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f61600a2530 msgr2=0x7f61600a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 1 --2- 172.21.15.98:0/277167998 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f61600a2530 0x7f61600a5ee0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f6158009d90 tx=0x7f615802bce0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 1 -- 172.21.15.98:0/277167998 shutdown_connections 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 1 --2- 172.21.15.98:0/277167998 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f61600a2530 0x7f61600a5ee0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 1 --2- 172.21.15.98:0/277167998 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f61600a5240 0x7f61600a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 1 --2- 172.21.15.98:0/277167998 >> v2:172.21.15.117:3300/0 conn(0x7f61600a4920 0x7f61600a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 1 -- 172.21.15.98:0/277167998 >> 172.21.15.98:0/277167998 conn(0x7f61600ab850 msgr2=0x7f61600a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 1 -- 172.21.15.98:0/277167998 shutdown_connections 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.340+0000 7f616723e640 1 -- 172.21.15.98:0/277167998 wait complete. 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.341+0000 7f616723e640 10 monclient: build_initial_monmap 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.341+0000 7f616723e640 10 monclient: monmap: 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.341+0000 7f616723e640 1 librados: starting msgr at 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.341+0000 7f616723e640 1 librados: starting objecter 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.341+0000 7f616723e640 1 Processor -- start 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.341+0000 7f616723e640 1 -- start start 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.341+0000 7f616723e640 1 librados: setting wanted keys 2024-06-11T19:15:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.341+0000 7f616723e640 1 librados: calling monclient init 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.341+0000 7f616723e640 10 monclient: init 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient: _add_conns ranks=[2,5,6,1,0,8,7,3,4] 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61600a2530 0x7f61601cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient(hunting): picked mon.noname-b con 0x7f61600a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f61600a4920 0x7f61601ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient(hunting): picked mon.noname-h con 0x7f61600a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f61600a5240 0x7f6160125070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient(hunting): picked mon.noname-d con 0x7f61600a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f61600a6da0 con 0x7f61600a2530 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f6160138e50 con 0x7f61600a4920 2024-06-11T19:15:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f61600a6420 con 0x7f61600a5240 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient(hunting): authenticate will time out at 1158.149093s 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f6164fda640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61600a2530 0x7f61601cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f61600a5240 0x7f6160125070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f6164fda640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61600a2530 0x7f61601cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:59918/0 (socket says 172.21.15.98:59918) 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f61600a5240 0x7f6160125070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34714/0 (socket says 172.21.15.98:34714) 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f6164fda640 1 -- 172.21.15.98:0/3453310771 learned_addr learned my addr 172.21.15.98:0/3453310771 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f6164fda640 10 monclient(hunting): get_auth_request con 0x7f61600a2530 auth_method 0 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f6164fda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f6164fda640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f6164fda640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 10 monclient(hunting): get_auth_request con 0x7f61600a5240 auth_method 0 2024-06-11T19:15:48.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615ffff640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f61600a4920 0x7f61601ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f6164fda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f6164fda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f6164fda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615ffff640 10 monclient(hunting): get_auth_request con 0x7f61600a4920 auth_method 0 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 10 monclient(hunting): handle_auth_done global_id 54978 payload 824 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 1 -- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f61600a4920 msgr2=0x7f61601ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f61600a4920 0x7f61601ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 1 -- 172.21.15.98:0/3453310771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61600a2530 msgr2=0x7f61601cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61600a2530 0x7f61601cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 10 monclient: _finish_hunting 0 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 1 monclient: found mon.noname-d 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 1 -- 172.21.15.98:0/3453310771 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61601387f0 con 0x7f61600a5240 2024-06-11T19:15:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f6164fda640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f61600a2530 0x7f61601cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f61657db640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f61600a5240 0x7f6160125070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6158033000 tx=0x7f6158004a40 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 10 client.?.objecter ms_handle_connect 0x7f61600a5240 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6158039050 con 0x7f61600a5240 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 10 monclient: dump: 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.363 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 10 monclient: _finish_auth 0 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 5 monclient: authenticate success, global_id 54978 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6158045070 con 0x7f61600a5240 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f61580046c0 con 0x7f61600a5240 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient: _renew_subs 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 1 -- 172.21.15.98:0/3453310771 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6160128a70 con 0x7f61600a5240 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615effd640 4 set_mon_vals no callback set 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 client.54978.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f615d7fa640 10 monclient: dump: 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient: _renew_subs 2024-06-11T19:15:48.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.342+0000 7f616723e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:48.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.343+0000 7f616723e640 1 -- 172.21.15.98:0/3453310771 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f61601278a0 con 0x7f61600a5240 2024-06-11T19:15:48.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.343+0000 7f616723e640 1 librados: init done 2024-06-11T19:15:48.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.343+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f6158004250 con 0x7f61600a5240 2024-06-11T19:15:48.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.343+0000 7f616723e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:48.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.343+0000 7f616723e640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f61601255b0 0x7f6160125990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.343+0000 7f616723e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.343+0000 7f616723e640 1 -- 172.21.15.98:0/3453310771 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f6160127250 con 0x7f61601255b0 2024-06-11T19:15:48.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f614403fdb0 0x7f6144042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(290..290 src has 1..290) ==== 6577+0+0 (secure 0 0 0) 0x7f6158074370 con 0x7f61600a5240 2024-06-11T19:15:48.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 10 client.54978.objecter ms_dispatch 0x7f6160000b90 osd_map(290..290 src has 1..290) 2024-06-11T19:15:48.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 3 client.54978.objecter handle_osd_map got epochs [290,290] > 0 2024-06-11T19:15:48.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 3 client.54978.objecter handle_osd_map decoding full epoch 290 2024-06-11T19:15:48.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f6164fda640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f61601255b0 0x7f6160125990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 20 client.54978.objecter dump_active .. 0 homeless 2024-06-11T19:15:48.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f6164fda640 10 monclient: get_auth_request con 0x7f61601255b0 auth_method 0 2024-06-11T19:15:48.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f6164fda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f6164fda640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f6164fda640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f6164fda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f6164fda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f6164fda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615ffff640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f614403fdb0 0x7f6144042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615ffff640 10 monclient: get_auth_request con 0x7f614403fdb0 auth_method 0 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f6164fda640 10 monclient(hunting): handle_auth_done global_id 84148 payload 824 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f6164fda640 1 -- 172.21.15.98:0/3453310771 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61600a8fa0 con 0x7f61601255b0 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 10 client.54978.objecter ms_handle_connect 0x7f61601255b0 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 10 client.54978.objecter resend_mon_ops 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f6164fda640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f61601255b0 0x7f6160125990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6154010c80 tx=0x7f615400e5a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 10 client.54978.objecter ms_handle_connect 0x7f614403fdb0 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615ffff640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f614403fdb0 0x7f6144042270 secure :-1 s=READY pgs=4141 cs=0 l=1 rev1=1 crypto rx=0x7f614c009630 tx=0x7f614c009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6154018070 con 0x7f61601255b0 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.344+0000 7f615d7fa640 10 monclient: dump: 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.369 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.370 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.345+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f615401d450 con 0x7f61601255b0 2024-06-11T19:15:48.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.345+0000 7f615d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:48.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.345+0000 7f615d7fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:48.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.345+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f61601255b0 msgr2=0x7f6160125990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.345+0000 7f615d7fa640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f61601255b0 0x7f6160125990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6154010c80 tx=0x7f615400e5a0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.345+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f61601255b0 msgr2=0x7f6160125990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:48.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.345+0000 7f615d7fa640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f61601255b0 0x7f6160125990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.351+0000 7f616723e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:15:48.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.351+0000 7f616723e640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6160125ed0 0x7f61601215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.351+0000 7f616723e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.351+0000 7f616723e640 1 -- 172.21.15.98:0/3453310771 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f6160050830 con 0x7f6160125ed0 2024-06-11T19:15:48.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.351+0000 7f6164fda640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6160125ed0 0x7f61601215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.351+0000 7f6164fda640 10 monclient: get_auth_request con 0x7f6160125ed0 auth_method 0 2024-06-11T19:15:48.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.351+0000 7f6164fda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.351+0000 7f6164fda640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.351+0000 7f6164fda640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f6164fda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f6164fda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f6164fda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f6164fda640 10 monclient(hunting): handle_auth_done global_id 84157 payload 824 2024-06-11T19:15:48.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f6164fda640 1 -- 172.21.15.98:0/3453310771 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f61600a6750 con 0x7f6160125ed0 2024-06-11T19:15:48.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f6164fda640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6160125ed0 0x7f61601215f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6154010c80 tx=0x7f61540046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 10 client.54978.objecter ms_handle_connect 0x7f6160125ed0 2024-06-11T19:15:48.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 10 client.54978.objecter resend_mon_ops 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f615401f2e0 con 0x7f6160125ed0 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 10 monclient: dump: 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.375 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f615401f2e0 con 0x7f6160125ed0 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6160125ed0 msgr2=0x7f61601215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6160125ed0 0x7f61601215f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6154010c80 tx=0x7f61540046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 1 -- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6160125ed0 msgr2=0x7f61601215f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:48.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.352+0000 7f615d7fa640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6160125ed0 0x7f61601215f0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6154010c80 tx=0x7f61540046c0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.377 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:15:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 30, 2024-06-11T19:15:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 20122, 2024-06-11T19:15:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:15:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:15:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:15:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:15:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:15:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:15:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:15:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:15:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:15:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:15:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:15:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:15:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:15:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:15:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:15:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:15:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:15:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:15:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:15:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:15:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:15:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:15:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:15:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:15:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:15:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:15:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:15:48.401 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 10 librados: watch_flush enter 2024-06-11T19:15:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 10 librados: watch_flush exit 2024-06-11T19:15:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 20 client.54978.objecter shutdown clearing up homeless session... 2024-06-11T19:15:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 10 client.54978.objecter successfully canceled tick 2024-06-11T19:15:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 1 -- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f614403fdb0 msgr2=0x7f6144042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f614403fdb0 0x7f6144042270 secure :-1 s=READY pgs=4141 cs=0 l=1 rev1=1 crypto rx=0x7f614c009630 tx=0x7f614c009240 comp rx=0 tx=0).stop 2024-06-11T19:15:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 10 monclient: shutdown 2024-06-11T19:15:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 1 -- 172.21.15.98:0/3453310771 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f61600a5240 msgr2=0x7f6160125070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 1 --2- 172.21.15.98:0/3453310771 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f61600a5240 0x7f6160125070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6158033000 tx=0x7f6158004a40 comp rx=0 tx=0).stop 2024-06-11T19:15:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f61657db640 1 -- 172.21.15.98:0/3453310771 reap_dead start 2024-06-11T19:15:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f61657db640 1 -- 172.21.15.98:0/3453310771 reap_dead start 2024-06-11T19:15:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 1 -- 172.21.15.98:0/3453310771 shutdown_connections 2024-06-11T19:15:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 1 -- 172.21.15.98:0/3453310771 >> 172.21.15.98:0/3453310771 conn(0x7f61600ab850 msgr2=0x7f616009c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 1 -- 172.21.15.98:0/3453310771 shutdown_connections 2024-06-11T19:15:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 1 -- 172.21.15.98:0/3453310771 wait complete. 2024-06-11T19:15:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.353+0000 7f616723e640 1 librados: shutdown 2024-06-11T19:15:48.405 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:15:48.405 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['e', 'g'] 2024-06-11T19:15:48.405 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-06-11T19:15:48.405 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-06-11T19:15:48.405 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e store 2024-06-11T19:15:48.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e sync_force --yes-i-really-mean-it 2024-06-11T19:15:48.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.507+0000 7f43f43f5640 10 monclient: get_monmap_and_config 2024-06-11T19:15:48.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.507+0000 7f43f43f5640 10 monclient: build_initial_monmap 2024-06-11T19:15:48.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.507+0000 7f43f43f5640 10 monclient: monmap: 2024-06-11T19:15:48.507 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:48.507 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.507 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:48.508212+0000 2024-06-11T19:15:48.507 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:48.508212+0000 2024-06-11T19:15:48.507 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:48.507 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.507 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:48.507 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:48.507 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:48.508 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:48.508 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:48.508 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:48.508 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:48.508 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:48.508 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:48.508 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:48.508 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:48.508 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.507+0000 7f43f43f5640 1 Processor -- start 2024-06-11T19:15:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.507+0000 7f43f43f5640 1 -- start start 2024-06-11T19:15:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.507+0000 7f43f43f5640 10 monclient: init 2024-06-11T19:15:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 10 monclient: _add_conns ranks=[0,8,9,10,7,4,3,1,5,6,2] 2024-06-11T19:15:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43ec0a0cd0 addr v1:172.21.15.98:6789/0 2024-06-11T19:15:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec138c80 0x7f43ec139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 10 monclient(hunting): picked mon.noname-e con 0x7f43ec138c80 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43ec13dc50 0x7f43ec140040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 10 monclient(hunting): picked mon.noname-a con 0x7f43ec13dc50 addr v2:172.21.15.98:3300/0 2024-06-11T19:15:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43ec0a17c0 con 0x7f43ec0a0cd0 2024-06-11T19:15:48.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f43ec0739b0 con 0x7f43ec13dc50 2024-06-11T19:15:48.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f43ec09ecf0 con 0x7f43ec138c80 2024-06-11T19:15:48.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:48.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f43f5640 10 monclient(hunting): authenticate will time out at 1158.315471s 2024-06-11T19:15:48.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec138c80 0x7f43ec139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f2191640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f43ec0a0cd0 0x7f43ec0a10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35750/0 (socket says 172.21.15.98:35750) 2024-06-11T19:15:48.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f2191640 1 -- 172.21.15.98:0/78170310 learned_addr learned my addr 172.21.15.98:0/78170310 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:15:48.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 10 monclient(hunting): get_auth_request con 0x7f43ec138c80 auth_method 0 2024-06-11T19:15:48.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f2992640 1 --2- 172.21.15.98:0/78170310 >> v2:172.21.15.98:3300/0 conn(0x7f43ec13dc50 0x7f43ec140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f2992640 10 monclient(hunting): get_auth_request con 0x7f43ec13dc50 auth_method 0 2024-06-11T19:15:48.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f2992640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f2992640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f2992640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f2992640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f2992640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f2992640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 10 monclient(hunting): handle_auth_done global_id 84257 payload 274 2024-06-11T19:15:48.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f118f640 1 -- 172.21.15.98:0/78170310 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1149654737 0 0) 0x7f43ec0a17c0 con 0x7f43ec0a0cd0 2024-06-11T19:15:48.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 1 -- 172.21.15.98:0/78170310 >> v2:172.21.15.98:3300/0 conn(0x7f43ec13dc50 msgr2=0x7f43ec140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:48.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 1 --2- 172.21.15.98:0/78170310 >> v2:172.21.15.98:3300/0 conn(0x7f43ec13dc50 0x7f43ec140040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 1 -- 172.21.15.98:0/78170310 >> v1:172.21.15.98:6789/0 conn(0x7f43ec0a0cd0 legacy=0x7f43ec0a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 10 monclient: _finish_hunting 0 2024-06-11T19:15:48.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 1 monclient: found mon.noname-e 2024-06-11T19:15:48.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:48.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:15:48.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f1990640 1 -- 172.21.15.98:0/78170310 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ec09efd0 con 0x7f43ec138c80 2024-06-11T19:15:48.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f118f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:15:48.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.508+0000 7f43f2992640 1 --2- 172.21.15.98:0/78170310 >> v2:172.21.15.98:3300/0 conn(0x7f43ec13dc50 0x7f43ec140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:48.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f1990640 1 --2- 172.21.15.98:0/78170310 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec138c80 0x7f43ec139060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009790 tx=0x7f43e002b790 comp rx=0 tx=0).ready entity=mon.8 client_cookie=2964bd1883e2e057 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 1 -- 172.21.15.98:0/78170310 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43e0039070 con 0x7f43ec138c80 2024-06-11T19:15:48.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:15:48.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 10 monclient: dump: 2024-06-11T19:15:48.523 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.523 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.523 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 10 monclient: _finish_auth 0 2024-06-11T19:15:48.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 10 monclient: _check_auth_tickets 2024-06-11T19:15:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 1 -- 172.21.15.98:0/78170310 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e0031960 con 0x7f43ec138c80 2024-06-11T19:15:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f43f5640 5 monclient: authenticate success, global_id 84257 2024-06-11T19:15:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f43f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f43f5640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 1 -- 172.21.15.98:0/78170310 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43e0032380 con 0x7f43ec138c80 2024-06-11T19:15:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f43f5640 4 set_mon_vals no callback set 2024-06-11T19:15:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.509+0000 7f43f118f640 10 monclient: dump: 2024-06-11T19:15:48.525 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.526 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.510+0000 7f43f43f5640 10 monclient: shutdown 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.510+0000 7f43f43f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.510+0000 7f43f43f5640 1 -- 172.21.15.98:0/78170310 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec138c80 msgr2=0x7f43ec139060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.510+0000 7f43f43f5640 1 --2- 172.21.15.98:0/78170310 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec138c80 0x7f43ec139060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009790 tx=0x7f43e002b790 comp rx=0 tx=0).stop 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.510+0000 7f43f43f5640 1 -- 172.21.15.98:0/78170310 shutdown_connections 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.510+0000 7f43f43f5640 1 --2- 172.21.15.98:0/78170310 >> v2:172.21.15.98:3300/0 conn(0x7f43ec13dc50 0x7f43ec140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.510+0000 7f43f43f5640 1 --2- 172.21.15.98:0/78170310 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec138c80 0x7f43ec139060 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.510+0000 7f43f43f5640 1 -- 172.21.15.98:0/78170310 >> 172.21.15.98:0/78170310 conn(0x7f43ec0a7850 msgr2=0x7f43ec0a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.510+0000 7f43f43f5640 1 -- 172.21.15.98:0/78170310 shutdown_connections 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.510+0000 7f43f43f5640 1 -- 172.21.15.98:0/78170310 wait complete. 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.510+0000 7f43f43f5640 10 monclient: build_initial_monmap 2024-06-11T19:15:48.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.510+0000 7f43f43f5640 10 monclient: monmap: 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:48.528 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 1 librados: starting msgr at 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 1 librados: starting objecter 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 1 Processor -- start 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 1 -- start start 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 1 librados: setting wanted keys 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 1 librados: calling monclient init 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 10 monclient: init 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 10 monclient: _add_conns ranks=[8,5,0,2,7,1,3,6,4] 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec0a0cd0 0x7f43ec1cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 10 monclient(hunting): picked mon.noname-e con 0x7f43ec0a0cd0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:48.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f43ec138c80 0x7f43ec1358a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 10 monclient(hunting): picked mon.noname-h con 0x7f43ec138c80 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43ec13dc50 0x7f43ec135de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 10 monclient(hunting): picked mon.noname-a con 0x7f43ec13dc50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec140580 con 0x7f43ec13dc50 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f43ec076a00 con 0x7f43ec138c80 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f43ec140700 con 0x7f43ec0a0cd0 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f43f5640 10 monclient(hunting): authenticate will time out at 1158.318786s 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2191640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec0a0cd0 0x7f43ec1cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2992640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43ec13dc50 0x7f43ec135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2191640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec0a0cd0 0x7f43ec1cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34920/0 (socket says 172.21.15.98:34920) 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2191640 1 -- 172.21.15.98:0/1637295187 learned_addr learned my addr 172.21.15.98:0/1637295187 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2191640 10 monclient(hunting): get_auth_request con 0x7f43ec0a0cd0 auth_method 0 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2191640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2191640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2992640 10 monclient(hunting): get_auth_request con 0x7f43ec13dc50 auth_method 0 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2992640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2992640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2992640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2191640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2992640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2992640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.511+0000 7f43f2992640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f2992640 10 monclient(hunting): handle_auth_done global_id 64728 payload 824 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f1990640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f43ec138c80 0x7f43ec1358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f2992640 1 -- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec0a0cd0 msgr2=0x7f43ec1cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f2992640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec0a0cd0 0x7f43ec1cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f2992640 1 -- 172.21.15.98:0/1637295187 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f43ec138c80 msgr2=0x7f43ec1358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f2992640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f43ec138c80 0x7f43ec1358a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f2992640 10 monclient: _finish_hunting 0 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f2992640 1 monclient: found mon.noname-a 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f2992640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f2992640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:15:48.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f2992640 1 -- 172.21.15.98:0/1637295187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ec0a45e0 con 0x7f43ec13dc50 2024-06-11T19:15:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f1990640 10 monclient: get_auth_request con 0x7f43ec138c80 auth_method 0 2024-06-11T19:15:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f2191640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec0a0cd0 0x7f43ec1cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f1990640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f43ec138c80 0x7f43ec1358a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:15:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f2992640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43ec13dc50 0x7f43ec135de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f43e400ef90 tx=0x7f43e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 10 client.?.objecter ms_handle_connect 0x7f43ec13dc50 2024-06-11T19:15:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43e4018020 con 0x7f43ec13dc50 2024-06-11T19:15:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:15:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 10 monclient: dump: 2024-06-11T19:15:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 10 monclient: _finish_auth 0 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 10 monclient: _check_auth_tickets 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e400cd30 con 0x7f43ec13dc50 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f43f5640 5 monclient: authenticate success, global_id 64728 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f43f5640 10 monclient: _renew_subs 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43e4014d60 con 0x7f43ec13dc50 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f43f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f43f5640 1 -- 172.21.15.98:0/1637295187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43ec140050 con 0x7f43ec13dc50 2024-06-11T19:15:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f098e640 4 set_mon_vals no callback set 2024-06-11T19:15:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43deffd640 10 monclient: dump: 2024-06-11T19:15:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.537 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f43f5640 10 client.64728.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f43f5640 10 monclient: _renew_subs 2024-06-11T19:15:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f43f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:15:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f43f5640 1 -- 172.21.15.98:0/1637295187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43ec12d320 con 0x7f43ec13dc50 2024-06-11T19:15:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.512+0000 7f43f43f5640 1 librados: init done 2024-06-11T19:15:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f43f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f43f5640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13abc0 0x7f43ec13afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f43f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f43f5640 1 -- 172.21.15.98:0/1637295187 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f43ec050830 con 0x7f43ec13abc0 2024-06-11T19:15:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f43e4005d10 con 0x7f43ec13dc50 2024-06-11T19:15:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f2191640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13abc0 0x7f43ec13afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f2191640 10 monclient: get_auth_request con 0x7f43ec13abc0 auth_method 0 2024-06-11T19:15:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f2191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f2191640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f2191640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f2191640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f2191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f2191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f43d003fde0 0x7f43d00422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f2191640 10 monclient(hunting): handle_auth_done global_id 84275 payload 824 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f2191640 1 -- 172.21.15.98:0/1637295187 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43ec12d670 con 0x7f43ec13abc0 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(290..290 src has 1..290) ==== 6577+0+0 (secure 0 0 0) 0x7f43e4052690 con 0x7f43ec13dc50 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 10 client.64728.objecter ms_dispatch 0x7f43ec000b90 osd_map(290..290 src has 1..290) 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 3 client.64728.objecter handle_osd_map got epochs [290,290] > 0 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 3 client.64728.objecter handle_osd_map decoding full epoch 290 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f2191640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13abc0 0x7f43ec13afa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f43e8000f60 tx=0x7f43e8001e70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 20 client.64728.objecter dump_active .. 0 homeless 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 10 client.64728.objecter ms_handle_connect 0x7f43ec13abc0 2024-06-11T19:15:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 10 client.64728.objecter resend_mon_ops 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43e800fe90 con 0x7f43ec13abc0 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43f1990640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f43d003fde0 0x7f43d00422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.513+0000 7f43deffd640 10 monclient: dump: 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.514+0000 7f43f1990640 10 monclient: get_auth_request con 0x7f43d003fde0 auth_method 0 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.514+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f43e8018070 con 0x7f43ec13abc0 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.514+0000 7f43deffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.514+0000 7f43deffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.514+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13abc0 msgr2=0x7f43ec13afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.514+0000 7f43deffd640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13abc0 0x7f43ec13afa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f43e8000f60 tx=0x7f43e8001e70 comp rx=0 tx=0).stop 2024-06-11T19:15:48.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.514+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13abc0 msgr2=0x7f43ec13afa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:48.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.514+0000 7f43deffd640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13abc0 0x7f43ec13afa0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f43e8000f60 tx=0x7f43e8001e70 comp rx=0 tx=0).stop 2024-06-11T19:15:48.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.514+0000 7f43deffd640 10 client.64728.objecter ms_handle_connect 0x7f43d003fde0 2024-06-11T19:15:48.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.514+0000 7f43f1990640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f43d003fde0 0x7f43d00422a0 secure :-1 s=READY pgs=4142 cs=0 l=1 rev1=1 crypto rx=0x7f43e0036000 tx=0x7f43e0008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f43f5640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:15:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f43f5640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13b4e0 0x7f43ec09b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f43f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f43f5640 1 -- 172.21.15.98:0/1637295187 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f43ec12cdd0 con 0x7f43ec13b4e0 2024-06-11T19:15:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f2191640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13b4e0 0x7f43ec09b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f2191640 10 monclient: get_auth_request con 0x7f43ec13b4e0 auth_method 0 2024-06-11T19:15:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f2191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f2191640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f2191640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f2191640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f2191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f2191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f2191640 10 monclient(hunting): handle_auth_done global_id 84284 payload 824 2024-06-11T19:15:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.521+0000 7f43f2191640 1 -- 172.21.15.98:0/1637295187 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f43ec12cfa0 con 0x7f43ec13b4e0 2024-06-11T19:15:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43f2191640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13b4e0 0x7f43ec09b330 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f43e8000f60 tx=0x7f43e8000ed0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 10 client.64728.objecter ms_handle_connect 0x7f43ec13b4e0 2024-06-11T19:15:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 10 client.64728.objecter resend_mon_ops 2024-06-11T19:15:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43e802d080 con 0x7f43ec13b4e0 2024-06-11T19:15:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:15:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 10 monclient: dump: 2024-06-11T19:15:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.547 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f43e8013790 con 0x7f43ec13b4e0 2024-06-11T19:15:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:15:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13b4e0 msgr2=0x7f43ec09b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13b4e0 0x7f43ec09b330 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f43e8000f60 tx=0x7f43e8000ed0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 1 -- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13b4e0 msgr2=0x7f43ec09b330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:15:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43deffd640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43ec13b4e0 0x7f43ec09b330 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f43e8000f60 tx=0x7f43e8000ed0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.547 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:15:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-06-11T19:15:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-06-11T19:15:48.548 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:15:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43f43f5640 10 librados: watch_flush enter 2024-06-11T19:15:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43f43f5640 10 librados: watch_flush exit 2024-06-11T19:15:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43f43f5640 20 client.64728.objecter shutdown clearing up homeless session... 2024-06-11T19:15:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.522+0000 7f43f43f5640 10 client.64728.objecter successfully canceled tick 2024-06-11T19:15:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f43f5640 1 -- 172.21.15.98:0/1637295187 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f43d003fde0 msgr2=0x7f43d00422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f43f5640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f43d003fde0 0x7f43d00422a0 secure :-1 s=READY pgs=4142 cs=0 l=1 rev1=1 crypto rx=0x7f43e0036000 tx=0x7f43e0008030 comp rx=0 tx=0).stop 2024-06-11T19:15:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f43f5640 10 monclient: shutdown 2024-06-11T19:15:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f43f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f43f5640 1 -- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43ec13dc50 msgr2=0x7f43ec135de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f43f5640 1 --2- 172.21.15.98:0/1637295187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43ec13dc50 0x7f43ec135de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f43e400ef90 tx=0x7f43e400c490 comp rx=0 tx=0).stop 2024-06-11T19:15:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f2992640 1 -- 172.21.15.98:0/1637295187 reap_dead start 2024-06-11T19:15:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f2992640 1 -- 172.21.15.98:0/1637295187 reap_dead start 2024-06-11T19:15:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f43f5640 1 -- 172.21.15.98:0/1637295187 shutdown_connections 2024-06-11T19:15:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f43f5640 1 -- 172.21.15.98:0/1637295187 >> 172.21.15.98:0/1637295187 conn(0x7f43ec0a7850 msgr2=0x7f43ec0a21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f43f5640 1 -- 172.21.15.98:0/1637295187 shutdown_connections 2024-06-11T19:15:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f43f5640 1 -- 172.21.15.98:0/1637295187 wait complete. 2024-06-11T19:15:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.523+0000 7f43f43f5640 1 librados: shutdown 2024-06-11T19:15:48.551 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-06-11T19:15:48.552 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-06-11T19:15:48.552 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:15:48.562 INFO:tasks.ceph.mon.e:Stopped 2024-06-11T19:15:48.562 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-06-11T19:15:48.562 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-06-11T19:15:48.562 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-06-11T19:15:48.562 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:15:48.575 INFO:tasks.ceph.mon.g:Stopped 2024-06-11T19:15:48.575 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 7 2024-06-11T19:15:48.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:15:48.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.700+0000 7fa24a59c640 10 monclient: get_monmap_and_config 2024-06-11T19:15:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.700+0000 7fa24a59c640 10 monclient: build_initial_monmap 2024-06-11T19:15:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.700+0000 7fa24a59c640 10 monclient: monmap: 2024-06-11T19:15:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:48.701486+0000 2024-06-11T19:15:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:48.701486+0000 2024-06-11T19:15:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:48.701 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:48.701 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:48.701 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:48.701 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:48.701 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:48.701 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:48.701 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:48.701 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:48.701 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 1 Processor -- start 2024-06-11T19:15:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 1 -- start start 2024-06-11T19:15:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 10 monclient: init 2024-06-11T19:15:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 10 monclient: _add_conns ranks=[9,10,8,0,4,2,5,3,6,1,7] 2024-06-11T19:15:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa2440a4920 0x7fa2440a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 10 monclient(hunting): picked mon.noname-a con 0x7fa2440a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:15:48.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa2440a5240 0x7fa2440a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 10 monclient(hunting): picked mon.noname-f con 0x7fa2440a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:15:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2440a2530 0x7fa2440a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 10 monclient(hunting): picked mon.noname-e con 0x7fa2440a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa2440a7140 con 0x7fa2440a4920 2024-06-11T19:15:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fa2440a2b50 con 0x7fa2440a5240 2024-06-11T19:15:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa2440a2e30 con 0x7fa2440a2530 2024-06-11T19:15:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa24a59c640 10 monclient(hunting): authenticate will time out at 1158.508742s 2024-06-11T19:15:48.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa248b39640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2440a2530 msgr2=0x7fa2440a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa248b39640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2440a2530 0x7fa2440a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa243fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa2440a4920 0x7fa2440a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa243fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa2440a4920 0x7fa2440a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46408/0 (socket says 172.21.15.98:46408) 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa243fff640 1 -- 172.21.15.98:0/298521043 learned_addr learned my addr 172.21.15.98:0/298521043 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa243fff640 10 monclient(hunting): get_auth_request con 0x7fa2440a4920 auth_method 0 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa243fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa243fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa243fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.701+0000 7fa243fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa2437fe640 1 --2- 172.21.15.98:0/298521043 >> v2:172.21.15.117:3300/0 conn(0x7fa2440a5240 0x7fa2440a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa2437fe640 10 monclient(hunting): get_auth_request con 0x7fa2440a5240 auth_method 0 2024-06-11T19:15:48.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa2437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa2437fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa2437fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 10 monclient(hunting): handle_auth_done global_id 64737 payload 274 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 1 -- 172.21.15.98:0/298521043 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2440a2530 msgr2=0x7fa2440a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 1 --2- 172.21.15.98:0/298521043 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2440a2530 0x7fa2440a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 1 -- 172.21.15.98:0/298521043 >> v2:172.21.15.117:3300/0 conn(0x7fa2440a5240 msgr2=0x7fa2440a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 1 --2- 172.21.15.98:0/298521043 >> v2:172.21.15.117:3300/0 conn(0x7fa2440a5240 0x7fa2440a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 10 monclient: _finish_hunting 0 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 1 monclient: found mon.noname-a 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 1 -- 172.21.15.98:0/298521043 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2440a3110 con 0x7fa2440a4920 2024-06-11T19:15:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.702+0000 7fa243fff640 1 --2- 172.21.15.98:0/298521043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2440a4920 0x7fa2440a4d00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa244073fc0 tx=0x7fa23402b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13bfc64457e4b336 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 1 -- 172.21.15.98:0/298521043 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2340319d0 con 0x7fa2440a4920 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 10 monclient: dump: 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.715 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 10 monclient: _finish_auth 0 2024-06-11T19:15:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:15:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 1 -- 172.21.15.98:0/298521043 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa234037070 con 0x7fa2440a4920 2024-06-11T19:15:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 5 monclient: authenticate success, global_id 64737 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 1 -- 172.21.15.98:0/298521043 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa234031bd0 con 0x7fa2440a4920 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 4 set_mon_vals no callback set 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa242ffd640 10 monclient: dump: 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 10 monclient: shutdown 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 1 -- 172.21.15.98:0/298521043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2440a4920 msgr2=0x7fa2440a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 1 --2- 172.21.15.98:0/298521043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2440a4920 0x7fa2440a4d00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa244073fc0 tx=0x7fa23402b800 comp rx=0 tx=0).stop 2024-06-11T19:15:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 1 -- 172.21.15.98:0/298521043 shutdown_connections 2024-06-11T19:15:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 1 --2- 172.21.15.98:0/298521043 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2440a2530 0x7fa2440a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 1 --2- 172.21.15.98:0/298521043 >> v2:172.21.15.117:3300/0 conn(0x7fa2440a5240 0x7fa2440a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 1 --2- 172.21.15.98:0/298521043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2440a4920 0x7fa2440a4d00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 1 -- 172.21.15.98:0/298521043 >> 172.21.15.98:0/298521043 conn(0x7fa2440ab850 msgr2=0x7fa2440a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 1 -- 172.21.15.98:0/298521043 shutdown_connections 2024-06-11T19:15:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.703+0000 7fa24a59c640 1 -- 172.21.15.98:0/298521043 wait complete. 2024-06-11T19:15:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.704+0000 7fa24a59c640 10 monclient: build_initial_monmap 2024-06-11T19:15:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.704+0000 7fa24a59c640 10 monclient: monmap: 2024-06-11T19:15:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.704+0000 7fa24a59c640 1 librados: starting msgr at 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.704+0000 7fa24a59c640 1 librados: starting objecter 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.704+0000 7fa24a59c640 1 Processor -- start 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.704+0000 7fa24a59c640 1 -- start start 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.704+0000 7fa24a59c640 1 librados: setting wanted keys 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.704+0000 7fa24a59c640 1 librados: calling monclient init 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.704+0000 7fa24a59c640 10 monclient: init 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 10 monclient: _add_conns ranks=[8,3,6,4,0,7,5,2,1] 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2440a2530 0x7fa2441cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 10 monclient(hunting): picked mon.noname-e con 0x7fa2440a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2440a4920 0x7fa2441ce200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 10 monclient(hunting): picked mon.noname-g con 0x7fa2440a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2440a5240 0x7fa2441230a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 10 monclient(hunting): picked mon.noname-d con 0x7fa2440a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fa2440a6da0 con 0x7fa2440a4920 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa244138e50 con 0x7fa2440a5240 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa243fff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2440a2530 msgr2=0x7fa2441cdcc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa2440a6420 con 0x7fa2440a2530 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa24a59c640 10 monclient(hunting): authenticate will time out at 1158.512226s 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa243fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2440a2530 0x7fa2441cdcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa2437fe640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2440a4920 msgr2=0x7fa2441ce200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa2437fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2440a4920 0x7fa2441ce200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2440a5240 0x7fa2441230a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2440a5240 0x7fa2441230a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34728/0 (socket says 172.21.15.98:34728) 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 1 -- 172.21.15.98:0/2362201437 learned_addr learned my addr 172.21.15.98:0/2362201437 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 10 monclient(hunting): get_auth_request con 0x7fa2440a5240 auth_method 0 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 10 monclient(hunting): handle_auth_done global_id 54987 payload 824 2024-06-11T19:15:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 1 -- 172.21.15.98:0/2362201437 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2440a2530 msgr2=0x7fa2441cdcc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 1 --2- 172.21.15.98:0/2362201437 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2440a2530 0x7fa2441cdcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 1 -- 172.21.15.98:0/2362201437 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2440a4920 msgr2=0x7fa2441ce200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 1 --2- 172.21.15.98:0/2362201437 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2440a4920 0x7fa2441ce200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 10 monclient: _finish_hunting 0 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 1 monclient: found mon.noname-d 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 1 -- 172.21.15.98:0/2362201437 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2441269e0 con 0x7fa2440a5240 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa248b39640 1 --2- 172.21.15.98:0/2362201437 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2440a5240 0x7fa2441230a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa23c00be60 tx=0x7fa23c00bf50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa240ff9640 10 client.?.objecter ms_handle_connect 0x7fa2440a5240 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa240ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa240ff9640 1 -- 172.21.15.98:0/2362201437 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa23c00cdc0 con 0x7fa2440a5240 2024-06-11T19:15:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa240ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa240ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.705+0000 7fa240ff9640 10 monclient: dump: 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa240ff9640 10 monclient: _finish_auth 0 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa240ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa240ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa240ff9640 1 -- 172.21.15.98:0/2362201437 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa23c010030 con 0x7fa2440a5240 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 5 monclient: authenticate success, global_id 54987 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 10 monclient: _renew_subs 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 1 -- 172.21.15.98:0/2362201437 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa244126cc0 con 0x7fa2440a5240 2024-06-11T19:15:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa240ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa240ff9640 1 -- 172.21.15.98:0/2362201437 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa23c015850 con 0x7fa2440a5240 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa240ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa240ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa240ff9640 10 monclient: dump: 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 10 client.54987.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 10 monclient: _renew_subs 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 1 -- 172.21.15.98:0/2362201437 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fa244124c40 con 0x7fa2440a5240 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa2427fc640 4 set_mon_vals no callback set 2024-06-11T19:15:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 1 librados: init done 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa240ff9640 1 -- 172.21.15.98:0/2362201437 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fa23c008070 con 0x7fa2440a5240 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.706+0000 7fa24a59c640 1 -- 172.21.15.98:0/2362201437 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2440a66c0 con 0x7fa2440a5240 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.707+0000 7fa240ff9640 1 --2- 172.21.15.98:0/2362201437 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa23003fdb0 0x7fa230042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.707+0000 7fa240ff9640 1 -- 172.21.15.98:0/2362201437 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(290..290 src has 1..290) ==== 6577+0+0 (secure 0 0 0) 0x7fa23c051910 con 0x7fa2440a5240 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.707+0000 7fa240ff9640 10 client.54987.objecter ms_dispatch 0x7fa244000b90 osd_map(290..290 src has 1..290) 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.707+0000 7fa240ff9640 3 client.54987.objecter handle_osd_map got epochs [290,290] > 0 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.707+0000 7fa240ff9640 3 client.54987.objecter handle_osd_map decoding full epoch 290 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.707+0000 7fa240ff9640 20 client.54987.objecter dump_active .. 0 homeless 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.707+0000 7fa243fff640 1 --2- 172.21.15.98:0/2362201437 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa23003fdb0 0x7fa230042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.707+0000 7fa243fff640 10 monclient: get_auth_request con 0x7fa23003fdb0 auth_method 0 2024-06-11T19:15:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.707+0000 7fa240ff9640 10 client.54987.objecter ms_handle_connect 0x7fa23003fdb0 2024-06-11T19:15:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.707+0000 7fa243fff640 1 --2- 172.21.15.98:0/2362201437 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa23003fdb0 0x7fa230042270 secure :-1 s=READY pgs=4143 cs=0 l=1 rev1=1 crypto rx=0x7fa234036000 tx=0x7fa234008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.711+0000 7fa240ff9640 1 -- 172.21.15.98:0/2362201437 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa23c01d300 con 0x7fa2440a5240 2024-06-11T19:15:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.711+0000 7fa240ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.711+0000 7fa240ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:48.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.962+0000 7fa24a59c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:15:48.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.962+0000 7fa24a59c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:48.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.962+0000 7fa24a59c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:15:48.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.962+0000 7fa24a59c640 1 -- 172.21.15.98:0/2362201437 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa24406c300 con 0x7fa2440a5240 2024-06-11T19:15:48.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.963+0000 7fa240ff9640 1 -- 172.21.15.98:0/2362201437 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fa23c015490 con 0x7fa2440a5240 2024-06-11T19:15:48.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.963+0000 7fa240ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:48.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.963+0000 7fa240ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:48.963 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":30,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:15:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 10 librados: watch_flush enter 2024-06-11T19:15:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 10 librados: watch_flush exit 2024-06-11T19:15:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 20 client.54987.objecter shutdown clearing up homeless session... 2024-06-11T19:15:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 10 client.54987.objecter successfully canceled tick 2024-06-11T19:15:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 -- 172.21.15.98:0/2362201437 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa23003fdb0 msgr2=0x7fa230042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 --2- 172.21.15.98:0/2362201437 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa23003fdb0 0x7fa230042270 secure :-1 s=READY pgs=4143 cs=0 l=1 rev1=1 crypto rx=0x7fa234036000 tx=0x7fa234008030 comp rx=0 tx=0).stop 2024-06-11T19:15:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 10 monclient: shutdown 2024-06-11T19:15:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:48.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 -- 172.21.15.98:0/2362201437 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2440a5240 msgr2=0x7fa2441230a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:48.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 --2- 172.21.15.98:0/2362201437 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2440a5240 0x7fa2441230a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa23c00be60 tx=0x7fa23c00bf50 comp rx=0 tx=0).stop 2024-06-11T19:15:48.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 -- 172.21.15.98:0/2362201437 shutdown_connections 2024-06-11T19:15:48.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 --2- 172.21.15.98:0/2362201437 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa23003fdb0 0x7fa230042270 unknown :-1 s=CLOSED pgs=4143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 --2- 172.21.15.98:0/2362201437 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2440a5240 0x7fa2441230a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 --2- 172.21.15.98:0/2362201437 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2440a4920 0x7fa2441ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 --2- 172.21.15.98:0/2362201437 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2440a2530 0x7fa2441cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:48.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 -- 172.21.15.98:0/2362201437 >> 172.21.15.98:0/2362201437 conn(0x7fa2440ab850 msgr2=0x7fa2441be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:48.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 -- 172.21.15.98:0/2362201437 shutdown_connections 2024-06-11T19:15:48.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 -- 172.21.15.98:0/2362201437 wait complete. 2024-06-11T19:15:48.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:48.965+0000 7fa24a59c640 1 librados: shutdown 2024-06-11T19:15:51.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:15:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.086+0000 7f84e87cd640 10 monclient: get_monmap_and_config 2024-06-11T19:15:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.086+0000 7f84e87cd640 10 monclient: build_initial_monmap 2024-06-11T19:15:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.086+0000 7f84e87cd640 10 monclient: monmap: 2024-06-11T19:15:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:52.087298+0000 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:52.087298+0000 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:52.098 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.086+0000 7f84e87cd640 1 Processor -- start 2024-06-11T19:15:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.086+0000 7f84e87cd640 1 -- start start 2024-06-11T19:15:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.086+0000 7f84e87cd640 10 monclient: init 2024-06-11T19:15:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 10 monclient: _add_conns ranks=[1,7,3,4,5,0,2,8,10,6,9] 2024-06-11T19:15:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f84e00a52a0 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84e009d520 0x7f84e013a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 10 monclient(hunting): picked mon.noname-i con 0x7f84e009d520 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:15:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84e0073830 0x7f84e013cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:52.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 10 monclient(hunting): picked mon.noname-g con 0x7f84e0073830 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:15:52.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e009d2e0 con 0x7f84e00a52a0 2024-06-11T19:15:52.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:52.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f84e009daa0 con 0x7f84e0073830 2024-06-11T19:15:52.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:52.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f84e00a5680 con 0x7f84e009d520 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e87cd640 10 monclient(hunting): authenticate will time out at 1161.894773s 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e6d6a640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84e0073830 msgr2=0x7f84e013cf30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.087+0000 7f84e6d6a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84e0073830 0x7f84e013cf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84e009d520 0x7f84e013a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e6569640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f84e00a52a0 0x7f84e00a1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:49792/0 (socket says 172.21.15.98:49792) 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e6569640 1 -- 172.21.15.98:0/3347386729 learned_addr learned my addr 172.21.15.98:0/3347386729 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 10 monclient(hunting): get_auth_request con 0x7f84e009d520 auth_method 0 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5567640 1 -- 172.21.15.98:0/3347386729 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2078084497 0 0) 0x7f84e009d2e0 con 0x7f84e00a52a0 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5567640 10 monclient(hunting): _negotiate 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5567640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5567640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5567640 10 monclient(hunting): my global_id is 31978 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5567640 1 -- 172.21.15.98:0/3347386729 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84cc003520 con 0x7f84e00a52a0 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 10 monclient(hunting): handle_auth_done global_id 84166 payload 274 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 1 -- 172.21.15.98:0/3347386729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84e0073830 msgr2=0x7f84e013cf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 1 --2- 172.21.15.98:0/3347386729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84e0073830 0x7f84e013cf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 1 -- 172.21.15.98:0/3347386729 >> v1:172.21.15.117:6789/0 conn(0x7f84e00a52a0 legacy=0x7f84e00a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 10 monclient: _finish_hunting 0 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 1 monclient: found mon.noname-i 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:15:52.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.088+0000 7f84e5d68640 1 -- 172.21.15.98:0/3347386729 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e009e9d0 con 0x7f84e009d520 2024-06-11T19:15:52.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5d68640 1 --2- 172.21.15.98:0/3347386729 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84e009d520 0x7f84e013a9e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f84dc009800 tx=0x7f84dc02b790 comp rx=0 tx=0).ready entity=mon.7 client_cookie=fec470752a5879ee server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:52.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 1 -- 172.21.15.98:0/3347386729 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84dc039070 con 0x7f84e009d520 2024-06-11T19:15:52.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:52.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:15:52.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 10 monclient: dump: 2024-06-11T19:15:52.102 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:52.102 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:52.102 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:52.102 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:52.102 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 10 monclient: _finish_auth 0 2024-06-11T19:15:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 10 monclient: _check_auth_tickets 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 1 -- 172.21.15.98:0/3347386729 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84dc031960 con 0x7f84e009d520 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e87cd640 5 monclient: authenticate success, global_id 84166 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e87cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 1 -- 172.21.15.98:0/3347386729 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84dc032380 con 0x7f84e009d520 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e87cd640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e87cd640 4 set_mon_vals no callback set 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e5567640 10 monclient: dump: 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e87cd640 10 monclient: shutdown 2024-06-11T19:15:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e87cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:52.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e87cd640 1 -- 172.21.15.98:0/3347386729 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84e009d520 msgr2=0x7f84e013a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:52.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e87cd640 1 --2- 172.21.15.98:0/3347386729 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84e009d520 0x7f84e013a9e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f84dc009800 tx=0x7f84dc02b790 comp rx=0 tx=0).stop 2024-06-11T19:15:52.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.089+0000 7f84e87cd640 1 -- 172.21.15.98:0/3347386729 shutdown_connections 2024-06-11T19:15:52.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.090+0000 7f84e87cd640 1 --2- 172.21.15.98:0/3347386729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84e0073830 0x7f84e013cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:52.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.090+0000 7f84e87cd640 1 --2- 172.21.15.98:0/3347386729 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84e009d520 0x7f84e013a9e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:52.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.090+0000 7f84e87cd640 1 -- 172.21.15.98:0/3347386729 >> 172.21.15.98:0/3347386729 conn(0x7f84e00a4780 msgr2=0x7f84e00a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:52.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.090+0000 7f84e87cd640 1 -- 172.21.15.98:0/3347386729 shutdown_connections 2024-06-11T19:15:52.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.090+0000 7f84e87cd640 1 -- 172.21.15.98:0/3347386729 wait complete. 2024-06-11T19:15:52.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.090+0000 7f84e87cd640 10 monclient: build_initial_monmap 2024-06-11T19:15:52.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.090+0000 7f84e87cd640 10 monclient: monmap: 2024-06-11T19:15:52.106 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:52.107 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.090+0000 7f84e87cd640 1 librados: starting msgr at 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.090+0000 7f84e87cd640 1 librados: starting objecter 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.090+0000 7f84e87cd640 1 Processor -- start 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 1 -- start start 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 1 librados: setting wanted keys 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 1 librados: calling monclient init 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 10 monclient: init 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 10 monclient: _add_conns ranks=[2,8,5,7,6,3,1,4,0] 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e0073830 0x7f84e0071790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 10 monclient(hunting): picked mon.noname-b con 0x7f84e0073830 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84e009d520 0x7f84e0071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 10 monclient(hunting): picked mon.noname-e con 0x7f84e009d520 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f84e00a52a0 0x7f84e01ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 10 monclient(hunting): picked mon.noname-h con 0x7f84e00a52a0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f84e00a21a0 con 0x7f84e0073830 2024-06-11T19:15:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f84e00a2020 con 0x7f84e00a52a0 2024-06-11T19:15:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f84e013d470 con 0x7f84e009d520 2024-06-11T19:15:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e87cd640 10 monclient(hunting): authenticate will time out at 1161.898557s 2024-06-11T19:15:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e5d68640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84e009d520 msgr2=0x7f84e0071cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:15:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e5d68640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84e009d520 0x7f84e0071cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e0073830 0x7f84e0071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e0073830 0x7f84e0071790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:59928/0 (socket says 172.21.15.98:59928) 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 1 -- 172.21.15.98:0/3983586651 learned_addr learned my addr 172.21.15.98:0/3983586651 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 10 monclient(hunting): get_auth_request con 0x7f84e0073830 auth_method 0 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6d6a640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f84e00a52a0 0x7f84e01ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 10 monclient(hunting): handle_auth_done global_id 84269 payload 824 2024-06-11T19:15:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 1 -- 172.21.15.98:0/3983586651 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84e009d520 msgr2=0x7f84e0071cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84e009d520 0x7f84e0071cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 1 -- 172.21.15.98:0/3983586651 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f84e00a52a0 msgr2=0x7f84e01ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f84e00a52a0 0x7f84e01ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 10 monclient: _finish_hunting 0 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 1 monclient: found mon.noname-b 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6569640 1 -- 172.21.15.98:0/3983586651 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e0138300 con 0x7f84e0073830 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6d6a640 10 monclient: get_auth_request con 0x7f84e00a52a0 auth_method 0 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.091+0000 7f84e6d6a640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f84e00a52a0 0x7f84e01ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84e6569640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e0073830 0x7f84e0071790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f84d000c410 tx=0x7f84d000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 10 client.?.objecter ms_handle_connect 0x7f84e0073830 2024-06-11T19:15:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 1 -- 172.21.15.98:0/3983586651 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84d0017070 con 0x7f84e0073830 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 10 monclient: dump: 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:52.112 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 10 monclient: _finish_auth 0 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 1 -- 172.21.15.98:0/3983586651 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84d00070d0 con 0x7f84e0073830 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84e87cd640 5 monclient: authenticate success, global_id 84269 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 1 -- 172.21.15.98:0/3983586651 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84d0011360 con 0x7f84e0073830 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:52.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:15:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84d6ffd640 10 monclient: dump: 2024-06-11T19:15:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:52.115 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84e4d66640 4 set_mon_vals no callback set 2024-06-11T19:15:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84e87cd640 10 monclient: _renew_subs 2024-06-11T19:15:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84e87cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84e87cd640 1 -- 172.21.15.98:0/3983586651 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e01385e0 con 0x7f84e0073830 2024-06-11T19:15:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84e87cd640 10 client.84269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84e87cd640 10 monclient: _renew_subs 2024-06-11T19:15:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84e87cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84e87cd640 1 -- 172.21.15.98:0/3983586651 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e00730b0 con 0x7f84e0073830 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.092+0000 7f84e87cd640 1 librados: init done 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84d6ffd640 1 -- 172.21.15.98:0/3983586651 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f84d0005ae0 con 0x7f84e0073830 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84e87cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84e87cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84e87cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84e87cd640 1 -- 172.21.15.98:0/3983586651 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84e00a1c60 con 0x7f84e0073830 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84d6ffd640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f84c403fdb0 0x7f84c4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84d6ffd640 1 -- 172.21.15.98:0/3983586651 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(290..290 src has 1..290) ==== 6577+0+0 (secure 0 0 0) 0x7f84d0050e40 con 0x7f84e0073830 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84d6ffd640 10 client.84269.objecter ms_dispatch 0x7f84e0000b90 osd_map(290..290 src has 1..290) 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84d6ffd640 3 client.84269.objecter handle_osd_map got epochs [290,290] > 0 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84d6ffd640 3 client.84269.objecter handle_osd_map decoding full epoch 290 2024-06-11T19:15:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84d6ffd640 20 client.84269.objecter dump_active .. 0 homeless 2024-06-11T19:15:52.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84e5d68640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f84c403fdb0 0x7f84c4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:52.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.093+0000 7f84e5d68640 10 monclient: get_auth_request con 0x7f84c403fdb0 auth_method 0 2024-06-11T19:15:52.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.094+0000 7f84d6ffd640 10 client.84269.objecter ms_handle_connect 0x7f84c403fdb0 2024-06-11T19:15:52.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.094+0000 7f84e5d68640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f84c403fdb0 0x7f84c4042270 secure :-1 s=READY pgs=4144 cs=0 l=1 rev1=1 crypto rx=0x7f84dc036000 tx=0x7f84dc008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:52.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.098+0000 7f84d6ffd640 1 -- 172.21.15.98:0/3983586651 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f84e00a1c60 con 0x7f84e0073830 2024-06-11T19:15:52.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.098+0000 7f84d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:52.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.098+0000 7f84d6ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:52.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.347+0000 7f84e87cd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:15:52.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.347+0000 7f84e87cd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:52.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.347+0000 7f84e87cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:15:52.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.347+0000 7f84e87cd640 1 -- 172.21.15.98:0/3983586651 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f84e00a12d0 con 0x7f84e0073830 2024-06-11T19:15:52.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.347+0000 7f84d6ffd640 1 -- 172.21.15.98:0/3983586651 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f84d0007540 con 0x7f84e0073830 2024-06-11T19:15:52.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.347+0000 7f84d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:52.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.347+0000 7f84d6ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:52.347 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":30,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:15:52.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 10 librados: watch_flush enter 2024-06-11T19:15:52.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 10 librados: watch_flush exit 2024-06-11T19:15:52.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 20 client.84269.objecter shutdown clearing up homeless session... 2024-06-11T19:15:52.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 10 client.84269.objecter successfully canceled tick 2024-06-11T19:15:52.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 -- 172.21.15.98:0/3983586651 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f84c403fdb0 msgr2=0x7f84c4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f84c403fdb0 0x7f84c4042270 secure :-1 s=READY pgs=4144 cs=0 l=1 rev1=1 crypto rx=0x7f84dc036000 tx=0x7f84dc008030 comp rx=0 tx=0).stop 2024-06-11T19:15:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 10 monclient: shutdown 2024-06-11T19:15:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 -- 172.21.15.98:0/3983586651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e0073830 msgr2=0x7f84e0071790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e0073830 0x7f84e0071790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f84d000c410 tx=0x7f84d000c8d0 comp rx=0 tx=0).stop 2024-06-11T19:15:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 -- 172.21.15.98:0/3983586651 shutdown_connections 2024-06-11T19:15:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f84c403fdb0 0x7f84c4042270 unknown :-1 s=CLOSED pgs=4144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f84e00a52a0 0x7f84e01ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84e009d520 0x7f84e0071cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 --2- 172.21.15.98:0/3983586651 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e0073830 0x7f84e0071790 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 -- 172.21.15.98:0/3983586651 >> 172.21.15.98:0/3983586651 conn(0x7f84e00a4780 msgr2=0x7f84e0133820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:52.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 -- 172.21.15.98:0/3983586651 shutdown_connections 2024-06-11T19:15:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 -- 172.21.15.98:0/3983586651 wait complete. 2024-06-11T19:15:52.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:52.349+0000 7f84e87cd640 1 librados: shutdown 2024-06-11T19:15:55.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:15:55.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.471+0000 7ffa1c16e640 10 monclient: get_monmap_and_config 2024-06-11T19:15:55.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.471+0000 7ffa1c16e640 10 monclient: build_initial_monmap 2024-06-11T19:15:55.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.471+0000 7ffa1c16e640 10 monclient: monmap: 2024-06-11T19:15:55.471 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:55.471 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:55.471 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:55.472573+0000 2024-06-11T19:15:55.471 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:55.472573+0000 2024-06-11T19:15:55.471 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:55.471 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:55.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 1 Processor -- start 2024-06-11T19:15:55.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 1 -- start start 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 10 monclient: init 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 10 monclient: _add_conns ranks=[1,5,9,0,4,3,2,10,8,7,6] 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ffa140a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffa140a2530 0x7ffa1413dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 10 monclient(hunting): picked mon.noname-h con 0x7ffa140a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffa1413e1a0 0x7ffa14140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 10 monclient(hunting): picked mon.noname-a con 0x7ffa1413e1a0 addr v2:172.21.15.98:3300/0 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa140a7140 con 0x7ffa140a4920 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ffa140a2b50 con 0x7ffa1413e1a0 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:55.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ffa140a2e30 con 0x7ffa140a2530 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.472+0000 7ffa1c16e640 10 monclient(hunting): authenticate will time out at 1165.279894s 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffa1413e1a0 0x7ffa14140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffa1413e1a0 0x7ffa14140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55576/0 (socket says 172.21.15.98:55576) 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 1 -- 172.21.15.98:0/554961007 learned_addr learned my addr 172.21.15.98:0/554961007 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 10 monclient(hunting): get_auth_request con 0x7ffa1413e1a0 auth_method 0 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa19709640 1 --2- 172.21.15.98:0/554961007 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffa140a2530 0x7ffa1413dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:55.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa19709640 10 monclient(hunting): get_auth_request con 0x7ffa140a2530 auth_method 0 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa19709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa19709640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa19709640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa19709640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa19709640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa19709640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 10 monclient(hunting): handle_auth_done global_id 64746 payload 274 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 1 -- 172.21.15.98:0/554961007 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffa140a2530 msgr2=0x7ffa1413dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 1 --2- 172.21.15.98:0/554961007 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffa140a2530 0x7ffa1413dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 1 -- 172.21.15.98:0/554961007 >> v1:172.21.15.117:6789/0 conn(0x7ffa140a4920 legacy=0x7ffa140a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 10 monclient: _finish_hunting 0 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 1 monclient: found mon.noname-a 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:55.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.473+0000 7ffa1a70b640 1 -- 172.21.15.98:0/554961007 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa140a3110 con 0x7ffa1413e1a0 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1a70b640 1 --2- 172.21.15.98:0/554961007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1413e1a0 0x7ffa14140590 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009e50 tx=0x7ffa0c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182727f304c33fab server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 1 -- 172.21.15.98:0/554961007 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffa0c02cc50 con 0x7ffa1413e1a0 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 10 monclient: dump: 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:55.484 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 10 monclient: _finish_auth 0 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 10 monclient: _check_auth_tickets 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 1 -- 172.21.15.98:0/554961007 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa0c02ce10 con 0x7ffa1413e1a0 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 5 monclient: authenticate success, global_id 64746 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 10 monclient: get_monmap_and_config success 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 1 -- 172.21.15.98:0/554961007 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffa0c034860 con 0x7ffa1413e1a0 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 4 set_mon_vals no callback set 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa18f08640 10 monclient: dump: 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:55.486 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 10 monclient: shutdown 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 1 -- 172.21.15.98:0/554961007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1413e1a0 msgr2=0x7ffa14140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 1 --2- 172.21.15.98:0/554961007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1413e1a0 0x7ffa14140590 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009e50 tx=0x7ffa0c02be40 comp rx=0 tx=0).stop 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 1 -- 172.21.15.98:0/554961007 shutdown_connections 2024-06-11T19:15:55.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 1 --2- 172.21.15.98:0/554961007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1413e1a0 0x7ffa14140590 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 1 --2- 172.21.15.98:0/554961007 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffa140a2530 0x7ffa1413dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 1 -- 172.21.15.98:0/554961007 >> 172.21.15.98:0/554961007 conn(0x7ffa140ab850 msgr2=0x7ffa140a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 1 -- 172.21.15.98:0/554961007 shutdown_connections 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.474+0000 7ffa1c16e640 1 -- 172.21.15.98:0/554961007 wait complete. 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.475+0000 7ffa1c16e640 10 monclient: build_initial_monmap 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.475+0000 7ffa1c16e640 10 monclient: monmap: 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:15:55.488 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.475+0000 7ffa1c16e640 1 librados: starting msgr at 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.475+0000 7ffa1c16e640 1 librados: starting objecter 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.475+0000 7ffa1c16e640 1 Processor -- start 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.475+0000 7ffa1c16e640 1 -- start start 2024-06-11T19:15:55.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.475+0000 7ffa1c16e640 1 librados: setting wanted keys 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.475+0000 7ffa1c16e640 1 librados: calling monclient init 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.475+0000 7ffa1c16e640 10 monclient: init 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 10 monclient: _add_conns ranks=[4,7,0,3,1,5,2,8,6] 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa140a2530 0x7ffa141cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 10 monclient(hunting): picked mon.noname-c con 0x7ffa140a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffa140a4920 0x7ffa14136c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 10 monclient(hunting): picked mon.noname-i con 0x7ffa140a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1413e1a0 0x7ffa141371b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 10 monclient(hunting): picked mon.noname-a con 0x7ffa1413e1a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa14138e50 con 0x7ffa1413e1a0 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:55.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ffa140a84b0 con 0x7ffa140a2530 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ffa140a6da0 con 0x7ffa140a4920 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1c16e640 10 monclient(hunting): authenticate will time out at 1165.283319s 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1a70b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1413e1a0 0x7ffa141371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa140a2530 0x7ffa141cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1a70b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1413e1a0 0x7ffa141371b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55580/0 (socket says 172.21.15.98:55580) 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1a70b640 1 -- 172.21.15.98:0/2123226631 learned_addr learned my addr 172.21.15.98:0/2123226631 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1a70b640 10 monclient(hunting): get_auth_request con 0x7ffa1413e1a0 auth_method 0 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1a70b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1a70b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1a70b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 10 monclient(hunting): get_auth_request con 0x7ffa140a2530 auth_method 0 2024-06-11T19:15:55.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1a70b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1a70b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1a70b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19709640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffa140a4920 0x7ffa14136c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19709640 10 monclient(hunting): get_auth_request con 0x7ffa140a4920 auth_method 0 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19709640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19709640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:55.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 10 monclient(hunting): handle_auth_done global_id 23800 payload 824 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 1 -- 172.21.15.98:0/2123226631 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffa140a4920 msgr2=0x7ffa14136c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffa140a4920 0x7ffa14136c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 1 -- 172.21.15.98:0/2123226631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1413e1a0 msgr2=0x7ffa141371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1413e1a0 0x7ffa141371b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 10 monclient: _finish_hunting 0 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 1 monclient: found mon.noname-c 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 1 -- 172.21.15.98:0/2123226631 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa140a4220 con 0x7ffa140a2530 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa1a70b640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1413e1a0 0x7ffa141371b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa19f0a640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa140a2530 0x7ffa141cd790 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000c3f0 tx=0x7ffa1000c8b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa0a7fc640 10 client.?.objecter ms_handle_connect 0x7ffa140a2530 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.476+0000 7ffa0a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:15:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 1 -- 172.21.15.98:0/2123226631 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffa10017070 con 0x7ffa140a2530 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 10 monclient: dump: 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 1 -- 172.21.15.98:0/2123226631 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa100035f0 con 0x7ffa140a2530 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa1c16e640 5 monclient: authenticate success, global_id 23800 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa1c16e640 10 monclient: _renew_subs 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa1c16e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:55.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa1c16e640 1 -- 172.21.15.98:0/2123226631 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa14123340 con 0x7ffa140a2530 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 1 -- 172.21.15.98:0/2123226631 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffa10003c10 con 0x7ffa140a2530 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa1c16e640 10 client.23800.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0bfff640 4 set_mon_vals no callback set 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 10 monclient: dump: 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa1c16e640 10 monclient: _renew_subs 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa1c16e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa1c16e640 1 -- 172.21.15.98:0/2123226631 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa1413dd60 con 0x7ffa140a2530 2024-06-11T19:15:55.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa1c16e640 1 librados: init done 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.477+0000 7ffa0a7fc640 1 -- 172.21.15.98:0/2123226631 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ffa100037a0 con 0x7ffa140a2530 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.478+0000 7ffa1c16e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.478+0000 7ffa1c16e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.478+0000 7ffa1c16e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.478+0000 7ffa1c16e640 1 -- 172.21.15.98:0/2123226631 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa14122d80 con 0x7ffa140a2530 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.478+0000 7ffa0a7fc640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff9f803fdb0 0x7ff9f8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.478+0000 7ffa0a7fc640 1 -- 172.21.15.98:0/2123226631 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(290..290 src has 1..290) ==== 6577+0+0 (secure 0 0 0) 0x7ffa10050d80 con 0x7ffa140a2530 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.478+0000 7ffa0a7fc640 10 client.23800.objecter ms_dispatch 0x7ffa14000b90 osd_map(290..290 src has 1..290) 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.478+0000 7ffa0a7fc640 3 client.23800.objecter handle_osd_map got epochs [290,290] > 0 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.478+0000 7ffa0a7fc640 3 client.23800.objecter handle_osd_map decoding full epoch 290 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.478+0000 7ffa0a7fc640 20 client.23800.objecter dump_active .. 0 homeless 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.478+0000 7ffa19709640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff9f803fdb0 0x7ff9f8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:55.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.478+0000 7ffa19709640 10 monclient: get_auth_request con 0x7ff9f803fdb0 auth_method 0 2024-06-11T19:15:55.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.479+0000 7ffa0a7fc640 10 client.23800.objecter ms_handle_connect 0x7ff9f803fdb0 2024-06-11T19:15:55.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.479+0000 7ffa19709640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff9f803fdb0 0x7ff9f8042270 secure :-1 s=READY pgs=4145 cs=0 l=1 rev1=1 crypto rx=0x7ffa140a1410 tx=0x7ffa04008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:15:55.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.482+0000 7ffa0a7fc640 1 -- 172.21.15.98:0/2123226631 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ffa10054030 con 0x7ffa140a2530 2024-06-11T19:15:55.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.482+0000 7ffa0a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:15:55.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.482+0000 7ffa0a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:15:55.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.737+0000 7ffa1c16e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:15:55.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.737+0000 7ffa1c16e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:55.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.737+0000 7ffa1c16e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:15:55.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.737+0000 7ffa1c16e640 1 -- 172.21.15.98:0/2123226631 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffa141cdcd0 con 0x7ffa140a2530 2024-06-11T19:15:55.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.738+0000 7ffa0a7fc640 1 -- 172.21.15.98:0/2123226631 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7ffa140a4220 con 0x7ffa140a2530 2024-06-11T19:15:55.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.738+0000 7ffa0a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:15:55.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.738+0000 7ffa0a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:15:55.738 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":30,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 10 librados: watch_flush enter 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 10 librados: watch_flush exit 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 20 client.23800.objecter shutdown clearing up homeless session... 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 10 client.23800.objecter successfully canceled tick 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 -- 172.21.15.98:0/2123226631 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff9f803fdb0 msgr2=0x7ff9f8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff9f803fdb0 0x7ff9f8042270 secure :-1 s=READY pgs=4145 cs=0 l=1 rev1=1 crypto rx=0x7ffa140a1410 tx=0x7ffa04008040 comp rx=0 tx=0).stop 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 10 monclient: shutdown 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 -- 172.21.15.98:0/2123226631 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa140a2530 msgr2=0x7ffa141cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa140a2530 0x7ffa141cd790 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000c3f0 tx=0x7ffa1000c8b0 comp rx=0 tx=0).stop 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 -- 172.21.15.98:0/2123226631 shutdown_connections 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff9f803fdb0 0x7ff9f8042270 unknown :-1 s=CLOSED pgs=4145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:55.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa1413e1a0 0x7ffa141371b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ffa140a4920 0x7ffa14136c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 --2- 172.21.15.98:0/2123226631 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa140a2530 0x7ffa141cd790 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:15:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 -- 172.21.15.98:0/2123226631 >> 172.21.15.98:0/2123226631 conn(0x7ffa140ab850 msgr2=0x7ffa14137960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:15:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 -- 172.21.15.98:0/2123226631 shutdown_connections 2024-06-11T19:15:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 -- 172.21.15.98:0/2123226631 wait complete. 2024-06-11T19:15:55.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:55.740+0000 7ffa1c16e640 1 librados: shutdown 2024-06-11T19:15:58.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:15:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.860+0000 7f5d5bc26640 10 monclient: get_monmap_and_config 2024-06-11T19:15:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.860+0000 7f5d5bc26640 10 monclient: build_initial_monmap 2024-06-11T19:15:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.860+0000 7f5d5bc26640 10 monclient: monmap: 2024-06-11T19:15:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:15:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:15:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:15:58.861774+0000 2024-06-11T19:15:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:15:58.861774+0000 2024-06-11T19:15:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:15:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:15:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:15:58.861 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:15:58.861 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:15:58.861 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:15:58.861 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:15:58.861 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:15:58.861 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:15:58.861 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:15:58.861 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:15:58.861 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:15:58.861 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:15:58.861 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.861+0000 7f5d5bc26640 1 Processor -- start 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.861+0000 7f5d5bc26640 1 -- start start 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.861+0000 7f5d5bc26640 10 monclient: init 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 10 monclient: _reopen_session rank -1 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 10 monclient: _add_conns ranks=[6,10,7,8,3,0,4,9,1,2,5] 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 10 monclient(hunting): picked mon.noname-d con 0x7f5d540a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 10 monclient(hunting): picked mon.noname-f con 0x7f5d540a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 10 monclient(hunting): picked mon.noname-i con 0x7f5d540a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f5d540a7140 con 0x7f5d540a5240 2024-06-11T19:15:58.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5d540a2b50 con 0x7f5d540a4920 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 10 monclient(hunting): start opening mon connection 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f5d540a2e30 con 0x7f5d540a2530 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 10 monclient(hunting): _renew_subs 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5bc26640 10 monclient(hunting): authenticate will time out at 1168.669203s 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d599c2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d599c2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34932/0 (socket says 172.21.15.98:34932) 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 learned_addr learned my addr 172.21.15.98:0/1923936502 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d599c2640 10 monclient(hunting): get_auth_request con 0x7f5d540a4920 auth_method 0 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d599c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d599c2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d599c2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 msgr2=0x7f5d540a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:15:58.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 msgr2=0x7f5d540a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5a1c3640 10 monclient(hunting): get_auth_request con 0x7f5d540a2530 auth_method 0 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5a1c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5a1c3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5a1c3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d591c1640 10 monclient(hunting): get_auth_request con 0x7f5d540a5240 auth_method 0 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d591c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d591c1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d591c1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5a1c3640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 msgr2=0x7f5d540a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:15:58.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5a1c3640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 msgr2=0x7f5d540a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:58.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 msgr2=0x7f5d540a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:15:58.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 msgr2=0x7f5d540a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:58.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:58.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:58.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:58.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:58.862+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:15:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d599c2640 10 monclient(hunting): get_auth_request con 0x7f5d540a4920 auth_method 0 2024-06-11T19:15:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d599c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d599c2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d599c2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 msgr2=0x7f5d540a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:15:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 msgr2=0x7f5d540a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d5a1c3640 10 monclient(hunting): get_auth_request con 0x7f5d540a2530 auth_method 0 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d5a1c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d5a1c3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d5a1c3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d591c1640 10 monclient(hunting): get_auth_request con 0x7f5d540a5240 auth_method 0 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d591c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d591c1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d591c1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d5a1c3640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 msgr2=0x7f5d540a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:15:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d5a1c3640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 msgr2=0x7f5d540a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:59.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:59.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 msgr2=0x7f5d540a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:15:59.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 msgr2=0x7f5d540a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:59.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:59.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:15:59.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.063+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:15:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d599c2640 10 monclient(hunting): get_auth_request con 0x7f5d540a4920 auth_method 0 2024-06-11T19:15:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d599c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d599c2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d599c2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 msgr2=0x7f5d540a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:15:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 msgr2=0x7f5d540a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:15:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d591c1640 10 monclient(hunting): get_auth_request con 0x7f5d540a5240 auth_method 0 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d591c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d591c1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d591c1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d5a1c3640 10 monclient(hunting): get_auth_request con 0x7f5d540a2530 auth_method 0 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d5a1c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d5a1c3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d5a1c3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 msgr2=0x7f5d540a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 msgr2=0x7f5d540a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d5a1c3640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 msgr2=0x7f5d540a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:15:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d5a1c3640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 msgr2=0x7f5d540a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:15:59.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:15:59.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:15:59.464+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:16:00.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:00.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d599c2640 10 monclient(hunting): get_auth_request con 0x7f5d540a4920 auth_method 0 2024-06-11T19:16:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d599c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d599c2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d599c2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:16:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 msgr2=0x7f5d540a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:16:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 msgr2=0x7f5d540a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:16:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:16:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:16:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d5a1c3640 10 monclient(hunting): get_auth_request con 0x7f5d540a2530 auth_method 0 2024-06-11T19:16:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d5a1c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d5a1c3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:00.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d5a1c3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d591c1640 10 monclient(hunting): get_auth_request con 0x7f5d540a5240 auth_method 0 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d591c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d591c1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d591c1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d5a1c3640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 msgr2=0x7f5d540a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d5a1c3640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 msgr2=0x7f5d540a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 msgr2=0x7f5d540a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 msgr2=0x7f5d540a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:16:00.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:00.265+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:16:01.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 10 monclient(hunting): tick 2024-06-11T19:16:01.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:16:01.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 1 monclient(hunting): continuing hunt 2024-06-11T19:16:01.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:16:01.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 msgr2=0x7f5d540a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d540a2530 0x7f5d540a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 msgr2=0x7f5d540a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d540a4920 0x7f5d540a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 1 -- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 msgr2=0x7f5d540a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 1 --2- 172.21.15.98:0/1923936502 >> v2:172.21.15.117:3300/0 conn(0x7f5d540a5240 0x7f5d540a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 10 monclient: _add_conns ranks=[4,3,0,8,9,5,1,2] 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 0x7f5d40005c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5d40003810 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d40006200 0x7f5d40008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 10 monclient(hunting): picked mon.noname-g con 0x7f5d40006200 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d40008d60 addr v1:172.21.15.98:6789/0 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 1 -- 172.21.15.98:0/1923936502 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d540a3110 con 0x7f5d40008d60 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:01.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 1 -- 172.21.15.98:0/1923936502 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f5d540a7140 con 0x7f5d40006200 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 1 -- 172.21.15.98:0/1923936502 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5d540a2b50 con 0x7f5d40003810 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d4b7fe640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 0x7f5d40005c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d40006200 msgr2=0x7f5d40008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d599c2640 10 monclient(hunting): get_auth_request con 0x7f5d40003810 auth_method 0 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d599c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d599c2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d599c2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d40006200 0x7f5d40008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 msgr2=0x7f5d40005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 msgr2=0x7f5d40005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.862+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 0x7f5d40005c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:16:01.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:01.863+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 0x7f5d40005c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:02.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.063+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d40006200 msgr2=0x7f5d40008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:02.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.063+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d40006200 0x7f5d40008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:16:02.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.064+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 0x7f5d40005c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:02.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.064+0000 7f5d599c2640 10 monclient(hunting): get_auth_request con 0x7f5d40003810 auth_method 0 2024-06-11T19:16:02.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.064+0000 7f5d599c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:02.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.064+0000 7f5d599c2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:02.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.064+0000 7f5d599c2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:16:02.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.064+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 msgr2=0x7f5d40005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:16:02.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.064+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 msgr2=0x7f5d40005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:16:02.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.064+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 0x7f5d40005c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:16:02.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.064+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 0x7f5d40005c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:16:02.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.464+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d40006200 msgr2=0x7f5d40008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:02.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.464+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d40006200 0x7f5d40008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:16:02.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.465+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 0x7f5d40005c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:02.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.465+0000 7f5d599c2640 10 monclient(hunting): get_auth_request con 0x7f5d40003810 auth_method 0 2024-06-11T19:16:02.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.465+0000 7f5d599c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:02.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.465+0000 7f5d599c2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:02.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.465+0000 7f5d599c2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:16:02.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.465+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 msgr2=0x7f5d40005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:16:02.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.465+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 msgr2=0x7f5d40005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:16:02.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.465+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 0x7f5d40005c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:16:02.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.465+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 0x7f5d40005c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:16:02.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.919+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1554302536 0 0) 0x7f5d540a3110 con 0x7f5d40008d60 2024-06-11T19:16:02.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.919+0000 7f5d589c0640 10 monclient(hunting): _negotiate 2024-06-11T19:16:02.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.919+0000 7f5d589c0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:02.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.919+0000 7f5d589c0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:02.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.919+0000 7f5d589c0640 10 monclient(hunting): my global_id is 64764 2024-06-11T19:16:02.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.920+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d38003520 con 0x7f5d40008d60 2024-06-11T19:16:02.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3608071103 0 0) 0x7f5d38003520 con 0x7f5d40008d60 2024-06-11T19:16:02.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 msgr2=0x7f5d40005c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:16:02.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d40003810 0x7f5d40005c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:02.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d40006200 msgr2=0x7f5d40008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:16:02.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d40006200 0x7f5d40008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient: _finish_hunting 0 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d5a1c3640 1 -- 172.21.15.98:0/1923936502 reap_dead start 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4000dc20 con 0x7f5d40008d60 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient: _finish_auth 0 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient: _check_auth_tickets 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7f5d4c009c70 con 0x7f5d40008d60 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d5bc26640 5 monclient: authenticate success, global_id 64764 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d5bc26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:02.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient: dump: 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:02.939 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 >> v1:172.21.15.98:6789/0 conn(0x7f5d40008d60 legacy=0x7f5d4000b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient: _add_conns ranks=[3,2,8,4,7,5,6,0,1] 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d38004200 0x7f5d38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient(hunting): picked mon.g con 0x7f5d38004200 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d38004e10 0x7f5d38005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient(hunting): picked mon.b con 0x7f5d38004e10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d38009cd0 0x7f5d3800c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient(hunting): picked mon.e con 0x7f5d38009cd0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5d540a7140 con 0x7f5d38004e10 2024-06-11T19:16:02.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f5d540a2b50 con 0x7f5d38004200 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5d3800ca70 con 0x7f5d38009cd0 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d589c0640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d5bc26640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d5a1c3640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d38009cd0 msgr2=0x7f5d3800c160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d38009cd0 0x7f5d3800c160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d599c2640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d38004200 msgr2=0x7f5d38004800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d599c2640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d38004200 0x7f5d38004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d38004e10 0x7f5d38005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 10 monclient(hunting): get_auth_request con 0x7f5d38004e10 auth_method 0 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:02.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 10 monclient(hunting): handle_auth_done global_id 64764 payload 293 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d38009cd0 msgr2=0x7f5d3800c160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d38009cd0 0x7f5d3800c160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d38004200 msgr2=0x7f5d38004800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.930+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d38004200 0x7f5d38004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.931+0000 7f5d591c1640 10 monclient: _finish_hunting 0 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.931+0000 7f5d591c1640 1 monclient: found mon.b 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.931+0000 7f5d591c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.931+0000 7f5d591c1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:16:02.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.931+0000 7f5d591c1640 1 -- 172.21.15.98:0/1923936502 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d38003520 con 0x7f5d38004e10 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.931+0000 7f5d591c1640 10 monclient: _finish_auth 0 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.931+0000 7f5d591c1640 10 monclient: _check_auth_tickets 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.931+0000 7f5d591c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.931+0000 7f5d591c1640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d38004e10 0x7f5d38005290 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5d54073fc0 tx=0x7f5d38005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab8188955c58b8d4 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d4403a070 con 0x7f5d38004e10 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d589c0640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d589c0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d589c0640 10 monclient: dump: 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:02.943 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d440347e0 con 0x7f5d38004e10 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d5bc26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d589c0640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d589c0640 1 -- 172.21.15.98:0/1923936502 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d44003610 con 0x7f5d38004e10 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d5bc26640 10 monclient: get_monmap_and_config success 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d5bc26640 4 set_mon_vals no callback set 2024-06-11T19:16:02.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d589c0640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d589c0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d589c0640 10 monclient: dump: 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:02.945 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d5bc26640 10 monclient: shutdown 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d5bc26640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d5bc26640 1 -- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d38004e10 msgr2=0x7f5d38005290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d5bc26640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d38004e10 0x7f5d38005290 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5d54073fc0 tx=0x7f5d38005830 comp rx=0 tx=0).stop 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d5bc26640 1 -- 172.21.15.98:0/1923936502 shutdown_connections 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d5bc26640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5d38004e10 0x7f5d38005290 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d5bc26640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d38004200 0x7f5d38004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d5bc26640 1 --2- 172.21.15.98:0/1923936502 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d38009cd0 0x7f5d3800c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.932+0000 7f5d5bc26640 1 -- 172.21.15.98:0/1923936502 >> 172.21.15.98:0/1923936502 conn(0x7f5d540ab850 msgr2=0x7f5d540a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.933+0000 7f5d5bc26640 1 -- 172.21.15.98:0/1923936502 shutdown_connections 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.933+0000 7f5d5bc26640 1 -- 172.21.15.98:0/1923936502 wait complete. 2024-06-11T19:16:02.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.933+0000 7f5d5bc26640 10 monclient: build_initial_monmap 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.933+0000 7f5d5bc26640 10 monclient: monmap: 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:02.947 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 1 librados: starting msgr at 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 1 librados: starting objecter 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 1 Processor -- start 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 1 -- start start 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 1 librados: setting wanted keys 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 1 librados: calling monclient init 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 10 monclient: init 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 10 monclient: _add_conns ranks=[8,4,3,2,1,5,7,0,6] 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d38004200 0x7f5d541cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 10 monclient(hunting): picked mon.noname-e con 0x7f5d38004200 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d38004e10 0x7f5d541cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 10 monclient(hunting): picked mon.noname-c con 0x7f5d38004e10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d38009cd0 0x7f5d54124050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:02.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 10 monclient(hunting): picked mon.noname-g con 0x7f5d38009cd0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d599c2640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d38004200 msgr2=0x7f5d541cdaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f5d540a6da0 con 0x7f5d38009cd0 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5d54138e50 con 0x7f5d38004e10 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5d540a4d00 con 0x7f5d38004200 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d599c2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d38004200 0x7f5d541cdaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5bc26640 10 monclient(hunting): authenticate will time out at 1172.741859s 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d591c1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d38004e10 0x7f5d541cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d591c1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d38004e10 0x7f5d541cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46812/0 (socket says 172.21.15.98:46812) 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d591c1640 1 -- 172.21.15.98:0/2072932780 learned_addr learned my addr 172.21.15.98:0/2072932780 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d591c1640 10 monclient(hunting): get_auth_request con 0x7f5d38004e10 auth_method 0 2024-06-11T19:16:02.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d591c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d591c1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d591c1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.934+0000 7f5d5a1c3640 1 -- 172.21.15.98:0/2072932780 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d38009cd0 msgr2=0x7f5d54124050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d5a1c3640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d38009cd0 0x7f5d54124050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 10 monclient(hunting): handle_auth_done global_id 23809 payload 824 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 1 -- 172.21.15.98:0/2072932780 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d38004200 msgr2=0x7f5d541cdaa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d38004200 0x7f5d541cdaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 1 -- 172.21.15.98:0/2072932780 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d38009cd0 msgr2=0x7f5d54124050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d38009cd0 0x7f5d54124050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 10 monclient: _finish_hunting 0 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 1 monclient: found mon.noname-c 2024-06-11T19:16:02.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 1 -- 172.21.15.98:0/2072932780 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d54127970 con 0x7f5d38004e10 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d591c1640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d38004e10 0x7f5d541cdfe0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f5d44038040 tx=0x7f5d440046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 10 client.?.objecter ms_handle_connect 0x7f5d38004e10 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 1 -- 172.21.15.98:0/2072932780 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d4403a040 con 0x7f5d38004e10 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 10 monclient: dump: 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:02.951 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 1 -- 172.21.15.98:0/2072932780 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d44045070 con 0x7f5d38004e10 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d5bc26640 5 monclient: authenticate success, global_id 23809 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:02.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 1 -- 172.21.15.98:0/2072932780 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d4402ce60 con 0x7f5d38004e10 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d5bc26640 10 monclient: _renew_subs 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d5bc26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d5bc26640 1 -- 172.21.15.98:0/2072932780 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d541260f0 con 0x7f5d38004e10 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4bfff640 4 set_mon_vals no callback set 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d4a7fc640 10 monclient: dump: 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d5bc26640 10 client.23809.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d5bc26640 10 monclient: _renew_subs 2024-06-11T19:16:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d5bc26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d5bc26640 1 -- 172.21.15.98:0/2072932780 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d540aaa10 con 0x7f5d38004e10 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.935+0000 7f5d5bc26640 1 librados: init done 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.936+0000 7f5d5bc26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.936+0000 7f5d4a7fc640 1 -- 172.21.15.98:0/2072932780 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f5d44034b90 con 0x7f5d38004e10 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.936+0000 7f5d5bc26640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.936+0000 7f5d5bc26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.936+0000 7f5d5bc26640 1 -- 172.21.15.98:0/2072932780 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d540a4280 con 0x7f5d38004e10 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.937+0000 7f5d4a7fc640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d40040aa0 0x7f5d40042f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.937+0000 7f5d4a7fc640 1 -- 172.21.15.98:0/2072932780 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(291..291 src has 1..291) ==== 6686+0+0 (secure 0 0 0) 0x7f5d44032070 con 0x7f5d38004e10 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.937+0000 7f5d4a7fc640 10 client.23809.objecter ms_dispatch 0x7f5d54000b90 osd_map(291..291 src has 1..291) 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.937+0000 7f5d4a7fc640 3 client.23809.objecter handle_osd_map got epochs [291,291] > 0 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.937+0000 7f5d4a7fc640 3 client.23809.objecter handle_osd_map decoding full epoch 291 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.937+0000 7f5d4a7fc640 20 client.23809.objecter dump_active .. 0 homeless 2024-06-11T19:16:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.937+0000 7f5d599c2640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d40040aa0 0x7f5d40042f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.937+0000 7f5d599c2640 10 monclient: get_auth_request con 0x7f5d40040aa0 auth_method 0 2024-06-11T19:16:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.937+0000 7f5d4a7fc640 10 client.23809.objecter ms_handle_connect 0x7f5d40040aa0 2024-06-11T19:16:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.937+0000 7f5d599c2640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d40040aa0 0x7f5d40042f60 secure :-1 s=READY pgs=4146 cs=0 l=1 rev1=1 crypto rx=0x7f5d50009100 tx=0x7f5d50008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.941+0000 7f5d4a7fc640 1 -- 172.21.15.98:0/2072932780 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5d540a4280 con 0x7f5d38004e10 2024-06-11T19:16:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.941+0000 7f5d4a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:02.941+0000 7f5d4a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:16:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.190+0000 7f5d5bc26640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:16:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.190+0000 7f5d5bc26640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:16:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.190+0000 7f5d5bc26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:16:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.190+0000 7f5d5bc26640 1 -- 172.21.15.98:0/2072932780 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5d540aa6e0 con 0x7f5d38004e10 2024-06-11T19:16:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.190+0000 7f5d4a7fc640 1 -- 172.21.15.98:0/2072932780 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3180 (secure 0 0 0) 0x7f5d4403e3f0 con 0x7f5d38004e10 2024-06-11T19:16:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.190+0000 7f5d4a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:16:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.190+0000 7f5d4a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:16:03.190 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":34,"quorum":[0,1,2,4,5,6,7],"quorum_names":["a","f","b","c","h","d","i"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:16:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 10 librados: watch_flush enter 2024-06-11T19:16:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 10 librados: watch_flush exit 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 20 client.23809.objecter shutdown clearing up homeless session... 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 10 client.23809.objecter successfully canceled tick 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 1 -- 172.21.15.98:0/2072932780 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d40040aa0 msgr2=0x7f5d40042f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d40040aa0 0x7f5d40042f60 secure :-1 s=READY pgs=4146 cs=0 l=1 rev1=1 crypto rx=0x7f5d50009100 tx=0x7f5d50008040 comp rx=0 tx=0).stop 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 10 monclient: shutdown 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 1 -- 172.21.15.98:0/2072932780 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d38004e10 msgr2=0x7f5d541cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d38004e10 0x7f5d541cdfe0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f5d44038040 tx=0x7f5d440046c0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 1 -- 172.21.15.98:0/2072932780 shutdown_connections 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d40040aa0 0x7f5d40042f60 unknown :-1 s=CLOSED pgs=4146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d38009cd0 0x7f5d54124050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d38004e10 0x7f5d541cdfe0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 1 --2- 172.21.15.98:0/2072932780 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d38004200 0x7f5d541cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 1 -- 172.21.15.98:0/2072932780 >> 172.21.15.98:0/2072932780 conn(0x7f5d540ab850 msgr2=0x7f5d541380d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.192+0000 7f5d5bc26640 1 -- 172.21.15.98:0/2072932780 shutdown_connections 2024-06-11T19:16:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.193+0000 7f5d5bc26640 1 -- 172.21.15.98:0/2072932780 wait complete. 2024-06-11T19:16:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.193+0000 7f5d5bc26640 1 librados: shutdown 2024-06-11T19:16:03.203 INFO:tasks.mon_thrash.ceph_manager:quorum is size 7 2024-06-11T19:16:03.203 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon a 2024-06-11T19:16:03.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:16:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.301+0000 7fe15f05c640 10 monclient: get_monmap_and_config 2024-06-11T19:16:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.301+0000 7fe15f05c640 10 monclient: build_initial_monmap 2024-06-11T19:16:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.301+0000 7fe15f05c640 10 monclient: monmap: 2024-06-11T19:16:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:16:03.302931+0000 2024-06-11T19:16:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:16:03.302931+0000 2024-06-11T19:16:03.301 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:16:03.302 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.302+0000 7fe15f05c640 1 Processor -- start 2024-06-11T19:16:03.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.302+0000 7fe15f05c640 1 -- start start 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.302+0000 7fe15f05c640 10 monclient: init 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 10 monclient: _add_conns ranks=[0,3,2,8,7,5,6,1,4,10,9] 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe1580a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1580a2530 0x7fe15813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 10 monclient(hunting): picked mon.noname-g con 0x7fe1580a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe15813e1a0 0x7fe158140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 10 monclient(hunting): picked mon.noname-b con 0x7fe15813e1a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1580a7140 con 0x7fe1580a4920 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe1580a2b50 con 0x7fe15813e1a0 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fe1580a2e30 con 0x7fe1580a2530 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:03.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15f05c640 10 monclient(hunting): authenticate will time out at 1173.110210s 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe14ffff640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1580a2530 msgr2=0x7fe15813dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe14ffff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1580a2530 0x7fe15813dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe15813e1a0 0x7fe158140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15cdf8640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe1580a4920 0x7fe1580a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35550/0 (socket says 172.21.15.98:35550) 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15cdf8640 1 -- 172.21.15.98:0/553232277 learned_addr learned my addr 172.21.15.98:0/553232277 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 10 monclient(hunting): get_auth_request con 0x7fe15813e1a0 auth_method 0 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 10 monclient(hunting): handle_auth_done global_id 84278 payload 274 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe14f7fe640 1 -- 172.21.15.98:0/553232277 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2223961971 0 0) 0x7fe1580a7140 con 0x7fe1580a4920 2024-06-11T19:16:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 1 -- 172.21.15.98:0/553232277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1580a2530 msgr2=0x7fe15813dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 1 --2- 172.21.15.98:0/553232277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1580a2530 0x7fe15813dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 1 -- 172.21.15.98:0/553232277 >> v1:172.21.15.98:6789/0 conn(0x7fe1580a4920 legacy=0x7fe1580a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 10 monclient: _finish_hunting 0 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 1 monclient: found mon.noname-b 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe15d5f9640 1 -- 172.21.15.98:0/553232277 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1580a3110 con 0x7fe15813e1a0 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.303+0000 7fe14f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe15d5f9640 1 --2- 172.21.15.98:0/553232277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe15813e1a0 0x7fe158140590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe150009dc0 tx=0x7fe15002bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a69d612db23b17bf server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 1 -- 172.21.15.98:0/553232277 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe150039070 con 0x7fe15813e1a0 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 10 monclient: dump: 2024-06-11T19:16:03.314 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe15f05c640 5 monclient: authenticate success, global_id 84278 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 1 -- 172.21.15.98:0/553232277 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe15002cd90 con 0x7fe15813e1a0 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe15f05c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe15f05c640 10 monclient: get_monmap_and_config success 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe15f05c640 4 set_mon_vals no callback set 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 1 -- 172.21.15.98:0/553232277 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe1500327b0 con 0x7fe15813e1a0 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.304+0000 7fe14f7fe640 10 monclient: dump: 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.317 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 10 monclient: shutdown 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 1 -- 172.21.15.98:0/553232277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe15813e1a0 msgr2=0x7fe158140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 1 --2- 172.21.15.98:0/553232277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe15813e1a0 0x7fe158140590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe150009dc0 tx=0x7fe15002bd70 comp rx=0 tx=0).stop 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 1 -- 172.21.15.98:0/553232277 shutdown_connections 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 1 --2- 172.21.15.98:0/553232277 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe15813e1a0 0x7fe158140590 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 1 --2- 172.21.15.98:0/553232277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1580a2530 0x7fe15813dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 1 -- 172.21.15.98:0/553232277 >> 172.21.15.98:0/553232277 conn(0x7fe1580ab850 msgr2=0x7fe1580a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 1 -- 172.21.15.98:0/553232277 shutdown_connections 2024-06-11T19:16:03.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 1 -- 172.21.15.98:0/553232277 wait complete. 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 10 monclient: build_initial_monmap 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 10 monclient: monmap: 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:03.319 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 1 librados: starting msgr at 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.305+0000 7fe15f05c640 1 librados: starting objecter 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 1 Processor -- start 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 1 -- start start 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 1 librados: setting wanted keys 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 1 librados: calling monclient init 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 10 monclient: init 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:03.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 10 monclient: _add_conns ranks=[0,8,5,1,3,7,2,4,6] 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a2530 0x7fe158136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 10 monclient(hunting): picked mon.noname-a con 0x7fe1580a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1580a4920 0x7fe158136e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 10 monclient(hunting): picked mon.noname-e con 0x7fe1580a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe15813e1a0 0x7fe15809faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 10 monclient(hunting): picked mon.noname-h con 0x7fe15813e1a0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe158138e50 con 0x7fe1580a2530 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fe1580a84b0 con 0x7fe15813e1a0 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe1580a6da0 con 0x7fe1580a4920 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:03.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15f05c640 10 monclient(hunting): authenticate will time out at 1173.113756s 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe14ffff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1580a4920 msgr2=0x7fe158136e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe14ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1580a4920 0x7fe158136e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15cdf8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a2530 0x7fe158136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15cdf8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a2530 0x7fe158136950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32856/0 (socket says 172.21.15.98:32856) 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15cdf8640 1 -- 172.21.15.98:0/2337337713 learned_addr learned my addr 172.21.15.98:0/2337337713 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15cdf8640 10 monclient(hunting): get_auth_request con 0x7fe1580a2530 auth_method 0 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15cdf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15cdf8640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15cdf8640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15cdf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15cdf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.306+0000 7fe15cdf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15d5f9640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe15813e1a0 0x7fe15809faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15d5f9640 10 monclient(hunting): get_auth_request con 0x7fe15813e1a0 auth_method 0 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15d5f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15d5f9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15d5f9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15cdf8640 10 monclient(hunting): handle_auth_done global_id 64782 payload 824 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15cdf8640 1 -- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1580a4920 msgr2=0x7fe158136e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15cdf8640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1580a4920 0x7fe158136e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15cdf8640 1 -- 172.21.15.98:0/2337337713 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe15813e1a0 msgr2=0x7fe15809faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15cdf8640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe15813e1a0 0x7fe15809faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15cdf8640 10 monclient: _finish_hunting 0 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15cdf8640 1 monclient: found mon.noname-a 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15cdf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15cdf8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15cdf8640 1 -- 172.21.15.98:0/2337337713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe158123400 con 0x7fe1580a2530 2024-06-11T19:16:03.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15cdf8640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a2530 0x7fe158136950 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe15400c480 tx=0x7fe15400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 10 client.?.objecter ms_handle_connect 0x7fe1580a2530 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe154018070 con 0x7fe1580a2530 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 10 monclient: dump: 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.324 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 10 monclient: _finish_auth 0 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15f05c640 5 monclient: authenticate success, global_id 64782 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15f05c640 10 monclient: _renew_subs 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1540047e0 con 0x7fe1580a2530 2024-06-11T19:16:03.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15f05c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15f05c640 1 -- 172.21.15.98:0/2337337713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe158072e10 con 0x7fe1580a2530 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe154004e40 con 0x7fe1580a2530 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14d7fa640 10 monclient: dump: 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.326 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe14effd640 4 set_mon_vals no callback set 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15f05c640 10 client.64782.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15f05c640 10 monclient: _renew_subs 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15f05c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15f05c640 1 -- 172.21.15.98:0/2337337713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1580a1730 con 0x7fe1580a2530 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.307+0000 7fe15f05c640 1 librados: init done 2024-06-11T19:16:03.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe15f05c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe15f05c640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1581cd7c0 0x7fe1580a0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe15f05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe15f05c640 1 -- 172.21.15.98:0/2337337713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1581373d0 con 0x7fe1581cd7c0 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fe1540049d0 con 0x7fe1580a2530 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14ffff640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1581cd7c0 0x7fe1580a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14ffff640 10 monclient: get_auth_request con 0x7fe1581cd7c0 auth_method 0 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14d7fa640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe14403fde0 0x7fe1440422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(291..291 src has 1..291) ==== 6686+0+0 (secure 0 0 0) 0x7fe154050e80 con 0x7fe1580a2530 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14d7fa640 10 client.64782.objecter ms_dispatch 0x7fe158000b90 osd_map(291..291 src has 1..291) 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14d7fa640 3 client.64782.objecter handle_osd_map got epochs [291,291] > 0 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14d7fa640 3 client.64782.objecter handle_osd_map decoding full epoch 291 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14ffff640 10 monclient(hunting): handle_auth_done global_id 64791 payload 824 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14ffff640 1 -- 172.21.15.98:0/2337337713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe158122cd0 con 0x7fe1581cd7c0 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14d7fa640 20 client.64782.objecter dump_active .. 0 homeless 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe15d5f9640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe14403fde0 0x7fe1440422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe15d5f9640 10 monclient: get_auth_request con 0x7fe14403fde0 auth_method 0 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14d7fa640 10 client.64782.objecter ms_handle_connect 0x7fe1581cd7c0 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14d7fa640 10 client.64782.objecter resend_mon_ops 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.308+0000 7fe14ffff640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1581cd7c0 0x7fe1580a0030 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe1580a1c10 tx=0x7fe14800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe148016070 con 0x7fe1581cd7c0 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe14d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe14d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe14d7fa640 10 monclient: dump: 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.330 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe148003bc0 con 0x7fe1581cd7c0 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe14d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe14d7fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1581cd7c0 msgr2=0x7fe1580a0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe14d7fa640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1581cd7c0 0x7fe1580a0030 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe1580a1c10 tx=0x7fe14800b8e0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1581cd7c0 msgr2=0x7fe1580a0030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe14d7fa640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1581cd7c0 0x7fe1580a0030 secure :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe1580a1c10 tx=0x7fe14800b8e0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe14d7fa640 10 client.64782.objecter ms_handle_connect 0x7fe14403fde0 2024-06-11T19:16:03.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.309+0000 7fe15d5f9640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe14403fde0 0x7fe1440422a0 secure :-1 s=READY pgs=4147 cs=0 l=1 rev1=1 crypto rx=0x7fe150036000 tx=0x7fe150030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.315+0000 7fe15f05c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:16:03.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.315+0000 7fe15f05c640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a0570 0x7fe15813bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.315+0000 7fe15f05c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.315+0000 7fe15f05c640 1 -- 172.21.15.98:0/2337337713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1580aa590 con 0x7fe1580a0570 2024-06-11T19:16:03.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.315+0000 7fe14ffff640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a0570 0x7fe15813bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.315+0000 7fe14ffff640 10 monclient: get_auth_request con 0x7fe1580a0570 auth_method 0 2024-06-11T19:16:03.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.315+0000 7fe14ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.315+0000 7fe14ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.315+0000 7fe14ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.315+0000 7fe14ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.315+0000 7fe14ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.315+0000 7fe14ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14ffff640 10 monclient(hunting): handle_auth_done global_id 64800 payload 824 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14ffff640 1 -- 172.21.15.98:0/2337337713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe1580a0980 con 0x7fe1580a0570 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14ffff640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a0570 0x7fe15813bde0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe1580a1250 tx=0x7fe1480059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 10 client.64782.objecter ms_handle_connect 0x7fe1580a0570 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 10 client.64782.objecter resend_mon_ops 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe148016030 con 0x7fe1580a0570 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 10 monclient: dump: 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.334 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9251 (secure 0 0 0) 0x7fe148018830 con 0x7fe1580a0570 2024-06-11T19:16:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:16:03.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:16:03.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a0570 msgr2=0x7fe15813bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a0570 0x7fe15813bde0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe1580a1250 tx=0x7fe1480059d0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 1 -- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a0570 msgr2=0x7fe15813bde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:03.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe14d7fa640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a0570 0x7fe15813bde0 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe1580a1250 tx=0x7fe1480059d0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.336 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:16:03.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:16:03.336 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:16:03.336 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:16:03.336 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 34, 2024-06-11T19:16:03.336 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 155144, 2024-06-11T19:16:03.336 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.337 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.338 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:16:03.339 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:16:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:16:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:16:03.343 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.343 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.343 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:16:03.343 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:16:03.343 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.343 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.343 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.344 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:16:03.345 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.347 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.350 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:16:03.351 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.352 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.357 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:16:03.357 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:16:03.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe15f05c640 10 librados: watch_flush enter 2024-06-11T19:16:03.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe15f05c640 10 librados: watch_flush exit 2024-06-11T19:16:03.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe15f05c640 20 client.64782.objecter shutdown clearing up homeless session... 2024-06-11T19:16:03.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe15f05c640 10 client.64782.objecter successfully canceled tick 2024-06-11T19:16:03.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe15f05c640 1 -- 172.21.15.98:0/2337337713 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe14403fde0 msgr2=0x7fe1440422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe15f05c640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe14403fde0 0x7fe1440422a0 secure :-1 s=READY pgs=4147 cs=0 l=1 rev1=1 crypto rx=0x7fe150036000 tx=0x7fe150030000 comp rx=0 tx=0).stop 2024-06-11T19:16:03.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe15f05c640 10 monclient: shutdown 2024-06-11T19:16:03.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe15f05c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:03.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe15f05c640 1 -- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a2530 msgr2=0x7fe158136950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe15f05c640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a2530 0x7fe158136950 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe15400c480 tx=0x7fe15400c940 comp rx=0 tx=0).stop 2024-06-11T19:16:03.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.316+0000 7fe15d5f9640 1 -- 172.21.15.98:0/2337337713 reap_dead start 2024-06-11T19:16:03.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.317+0000 7fe15f05c640 1 -- 172.21.15.98:0/2337337713 shutdown_connections 2024-06-11T19:16:03.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.317+0000 7fe15f05c640 1 --2- 172.21.15.98:0/2337337713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1580a2530 0x7fe158136950 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.317+0000 7fe15f05c640 1 -- 172.21.15.98:0/2337337713 >> 172.21.15.98:0/2337337713 conn(0x7fe1580ab850 msgr2=0x7fe1580726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:03.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.317+0000 7fe15f05c640 1 -- 172.21.15.98:0/2337337713 shutdown_connections 2024-06-11T19:16:03.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.317+0000 7fe15f05c640 1 -- 172.21.15.98:0/2337337713 wait complete. 2024-06-11T19:16:03.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.317+0000 7fe15f05c640 1 librados: shutdown 2024-06-11T19:16:03.361 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:16:03.362 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon b 2024-06-11T19:16:03.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:16:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.461+0000 7f3402741640 10 monclient: get_monmap_and_config 2024-06-11T19:16:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.461+0000 7f3402741640 10 monclient: build_initial_monmap 2024-06-11T19:16:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.461+0000 7f3402741640 10 monclient: monmap: 2024-06-11T19:16:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:16:03.462856+0000 2024-06-11T19:16:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:16:03.462856+0000 2024-06-11T19:16:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:16:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:16:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:16:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:16:03.462 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.462+0000 7f3402741640 1 Processor -- start 2024-06-11T19:16:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.462+0000 7f3402741640 1 -- start start 2024-06-11T19:16:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.462+0000 7f3402741640 10 monclient: init 2024-06-11T19:16:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 10 monclient: _add_conns ranks=[1,5,3,10,9,7,6,0,4,2,8] 2024-06-11T19:16:03.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f33fc0a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f33fc0a2530 0x7f33fc13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 10 monclient(hunting): picked mon.noname-h con 0x7f33fc0a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc13e1a0 0x7f33fc140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 10 monclient(hunting): picked mon.noname-g con 0x7f33fc13e1a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33fc0a7140 con 0x7f33fc0a4920 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f33fc0a2b50 con 0x7f33fc13e1a0 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f33fc0a2e30 con 0x7f33fc0a2530 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3402741640 10 monclient(hunting): authenticate will time out at 1173.270259s 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3400cde640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc13e1a0 msgr2=0x7f33fc140590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f3400cde640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc13e1a0 0x7f33fc140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f33fc0a2530 0x7f33fc13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f33fc0a2530 0x7f33fc13dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:34120/0 (socket says 172.21.15.98:34120) 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 1 -- 172.21.15.98:0/881193833 learned_addr learned my addr 172.21.15.98:0/881193833 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fbfff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f33fc0a4920 0x7f33fc0a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:34872/0 (socket says 172.21.15.98:34872) 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 10 monclient(hunting): get_auth_request con 0x7f33fc0a2530 auth_method 0 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 10 monclient(hunting): handle_auth_done global_id 22910 payload 274 2024-06-11T19:16:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 1 -- 172.21.15.98:0/881193833 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc13e1a0 msgr2=0x7f33fc140590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 1 --2- 172.21.15.98:0/881193833 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc13e1a0 0x7f33fc140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 1 -- 172.21.15.98:0/881193833 >> v1:172.21.15.117:6789/0 conn(0x7f33fc0a4920 legacy=0x7f33fc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 1 monclient: found mon.noname-h 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33fb7fe640 1 -- 172.21.15.98:0/881193833 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33fc0a3110 con 0x7f33fc0a2530 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33faffd640 1 -- 172.21.15.98:0/881193833 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388625163 0 0) 0x7f33fc0a7140 con 0x7f33fc0a4920 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.463+0000 7f33faffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33fb7fe640 1 --2- 172.21.15.98:0/881193833 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f33fc0a2530 0x7f33fc13dc60 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f33f00098a0 tx=0x7f33f002b860 comp rx=0 tx=0).ready entity=mon.5 client_cookie=73d4ad575032b4b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 1 -- 172.21.15.98:0/881193833 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f33f0031a30 con 0x7f33fc0a2530 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:16:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 10 monclient: dump: 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 10 monclient: _finish_auth 0 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 10 monclient: _check_auth_tickets 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f3402741640 5 monclient: authenticate success, global_id 22910 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f3402741640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 1 -- 172.21.15.98:0/881193833 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f0037070 con 0x7f33fc0a2530 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f3402741640 10 monclient: get_monmap_and_config success 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f3402741640 4 set_mon_vals no callback set 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 1 -- 172.21.15.98:0/881193833 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f33f0031bf0 con 0x7f33fc0a2530 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.464+0000 7f33faffd640 10 monclient: dump: 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.465+0000 7f3402741640 10 monclient: shutdown 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.465+0000 7f3402741640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.465+0000 7f3402741640 1 -- 172.21.15.98:0/881193833 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f33fc0a2530 msgr2=0x7f33fc13dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.465+0000 7f3402741640 1 --2- 172.21.15.98:0/881193833 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f33fc0a2530 0x7f33fc13dc60 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f33f00098a0 tx=0x7f33f002b860 comp rx=0 tx=0).stop 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.465+0000 7f3402741640 1 -- 172.21.15.98:0/881193833 shutdown_connections 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.465+0000 7f3402741640 1 --2- 172.21.15.98:0/881193833 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc13e1a0 0x7f33fc140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.465+0000 7f3402741640 1 --2- 172.21.15.98:0/881193833 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f33fc0a2530 0x7f33fc13dc60 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.465+0000 7f3402741640 1 -- 172.21.15.98:0/881193833 >> 172.21.15.98:0/881193833 conn(0x7f33fc0ab850 msgr2=0x7f33fc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.465+0000 7f3402741640 1 -- 172.21.15.98:0/881193833 shutdown_connections 2024-06-11T19:16:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.465+0000 7f3402741640 1 -- 172.21.15.98:0/881193833 wait complete. 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.465+0000 7f3402741640 10 monclient: build_initial_monmap 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.465+0000 7f3402741640 10 monclient: monmap: 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:03.480 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 1 librados: starting msgr at 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 1 librados: starting objecter 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 1 Processor -- start 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 1 -- start start 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 1 librados: setting wanted keys 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 1 librados: calling monclient init 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 10 monclient: init 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 10 monclient: _add_conns ranks=[3,6,0,4,2,1,8,5,7] 2024-06-11T19:16:03.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc0a2530 0x7f33fc136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 10 monclient(hunting): picked mon.noname-g con 0x7f33fc0a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f33fc0a4920 0x7f33fc1370f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 10 monclient(hunting): picked mon.noname-d con 0x7f33fc0a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc13e1a0 0x7f33fc09fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 10 monclient(hunting): picked mon.noname-a con 0x7f33fc13e1a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc138e50 con 0x7f33fc13e1a0 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f33fc0a84b0 con 0x7f33fc0a2530 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f33fc0a6da0 con 0x7f33fc0a4920 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:03.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f3402741640 10 monclient(hunting): authenticate will time out at 1173.273898s 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f33fbfff640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc0a2530 msgr2=0x7f33fc136bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.466+0000 7f33fbfff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc0a2530 0x7f33fc136bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f33fc0a4920 0x7f33fc1370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3400cde640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc13e1a0 0x7f33fc09fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f33fc0a4920 0x7f33fc1370f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:50436/0 (socket says 172.21.15.98:50436) 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 1 -- 172.21.15.98:0/2234546589 learned_addr learned my addr 172.21.15.98:0/2234546589 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 10 monclient(hunting): get_auth_request con 0x7f33fc0a4920 auth_method 0 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3400cde640 10 monclient(hunting): get_auth_request con 0x7f33fc13e1a0 auth_method 0 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3400cde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3400cde640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3400cde640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3400cde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3400cde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3400cde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 10 monclient(hunting): handle_auth_done global_id 54996 payload 824 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 1 -- 172.21.15.98:0/2234546589 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc0a2530 msgr2=0x7f33fc136bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc0a2530 0x7f33fc136bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 1 -- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc13e1a0 msgr2=0x7f33fc09fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc13e1a0 0x7f33fc09fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 1 monclient: found mon.noname-d 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 1 -- 172.21.15.98:0/2234546589 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33fc123560 con 0x7f33fc0a4920 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3400cde640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33fc13e1a0 0x7f33fc09fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fb7fe640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f33fc0a4920 0x7f33fc1370f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f33f00099c0 tx=0x7f33f00046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 10 client.?.objecter ms_handle_connect 0x7f33fc0a4920 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f33f0041070 con 0x7f33fc0a4920 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 10 monclient: dump: 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 10 monclient: _finish_auth 0 2024-06-11T19:16:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f0003c50 con 0x7f33fc0a4920 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f33f0031ea0 con 0x7f33fc0a4920 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3402741640 5 monclient: authenticate success, global_id 54996 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3402741640 10 monclient: _renew_subs 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3402741640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3402741640 1 -- 172.21.15.98:0/2234546589 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33fc0a4060 con 0x7f33fc0a4920 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33fa7fc640 4 set_mon_vals no callback set 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f33f8ff9640 10 monclient: dump: 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.488 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3402741640 10 client.54996.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3402741640 10 monclient: _renew_subs 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3402741640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3402741640 1 -- 172.21.15.98:0/2234546589 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f33fc13dea0 con 0x7f33fc0a4920 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.467+0000 7f3402741640 1 librados: init done 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f3402741640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f3402741640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc1cd990 0x7f33fc122da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f3402741640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f3402741640 1 -- 172.21.15.98:0/2234546589 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f33fc050830 con 0x7f33fc1cd990 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f33f004c060 con 0x7f33fc0a4920 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33fbfff640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc1cd990 0x7f33fc122da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33fbfff640 10 monclient: get_auth_request con 0x7f33fc1cd990 auth_method 0 2024-06-11T19:16:03.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33fbfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33f8ff9640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f33e803fdb0 0x7f33e8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33fbfff640 10 monclient(hunting): handle_auth_done global_id 84287 payload 824 2024-06-11T19:16:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(291..291 src has 1..291) ==== 6686+0+0 (secure 0 0 0) 0x7f33f00730b0 con 0x7f33fc0a4920 2024-06-11T19:16:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33f8ff9640 10 client.54996.objecter ms_dispatch 0x7f33fc000b90 osd_map(291..291 src has 1..291) 2024-06-11T19:16:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33f8ff9640 3 client.54996.objecter handle_osd_map got epochs [291,291] > 0 2024-06-11T19:16:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33f8ff9640 3 client.54996.objecter handle_osd_map decoding full epoch 291 2024-06-11T19:16:03.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.468+0000 7f33fbfff640 1 -- 172.21.15.98:0/2234546589 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f33fc0a1f10 con 0x7f33fc1cd990 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 20 client.54996.objecter dump_active .. 0 homeless 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f3400cde640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f33e803fdb0 0x7f33e8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f3400cde640 10 monclient: get_auth_request con 0x7f33e803fdb0 auth_method 0 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 10 client.54996.objecter ms_handle_connect 0x7f33fc1cd990 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 10 client.54996.objecter resend_mon_ops 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33fbfff640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc1cd990 0x7f33fc122da0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00c450 tx=0x7f33ec00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f33ec017070 con 0x7f33fc1cd990 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 10 monclient: dump: 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.491 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f33ec0073a0 con 0x7f33fc1cd990 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:03.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:16:03.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc1cd990 msgr2=0x7f33fc122da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc1cd990 0x7f33fc122da0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00c450 tx=0x7f33ec00c910 comp rx=0 tx=0).stop 2024-06-11T19:16:03.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc1cd990 msgr2=0x7f33fc122da0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:03.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc1cd990 0x7f33fc122da0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f33f8ff9640 10 client.54996.objecter ms_handle_connect 0x7f33e803fdb0 2024-06-11T19:16:03.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.469+0000 7f3400cde640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f33e803fdb0 0x7f33e8042270 secure :-1 s=READY pgs=4148 cs=0 l=1 rev1=1 crypto rx=0x7f33f4003cb0 tx=0x7f33f40095f0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.475+0000 7f3402741640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:16:03.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.475+0000 7f3402741640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0a0120 0x7f33fc0a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.475+0000 7f3402741640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.475+0000 7f3402741640 1 -- 172.21.15.98:0/2234546589 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f33fc0a0a70 con 0x7f33fc0a0120 2024-06-11T19:16:03.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33fbfff640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0a0120 0x7f33fc0a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33fbfff640 10 monclient: get_auth_request con 0x7f33fc0a0120 auth_method 0 2024-06-11T19:16:03.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33fbfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33fbfff640 10 monclient(hunting): handle_auth_done global_id 84296 payload 824 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33fbfff640 1 -- 172.21.15.98:0/2234546589 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f33fc0a4d00 con 0x7f33fc0a0120 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 10 client.54996.objecter ms_handle_connect 0x7f33fc0a0120 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33fbfff640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0a0120 0x7f33fc0a0530 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00cfd0 tx=0x7f33ec000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 10 client.54996.objecter resend_mon_ops 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f33ec017070 con 0x7f33fc0a0120 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 10 monclient: dump: 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.496 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9250 (secure 0 0 0) 0x7f33ec018040 con 0x7f33fc0a0120 2024-06-11T19:16:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:16:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:16:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0a0120 msgr2=0x7f33fc0a0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0a0120 0x7f33fc0a0530 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00cfd0 tx=0x7f33ec000bc0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 1 -- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0a0120 msgr2=0x7f33fc0a0530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f33f8ff9640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f33fc0a0120 0x7f33fc0a0530 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00cfd0 tx=0x7f33ec000bc0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.498 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:16:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:16:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:16:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:16:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 34, 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 35250, 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:16:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:16:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:16:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:16:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.503 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:16:03.505 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.508 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.511 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:16:03.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.514 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:16:03.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:16:03.517 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:16:03.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.518 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.519 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.520 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:16:03.520 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:16:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.476+0000 7f3402741640 10 librados: watch_flush enter 2024-06-11T19:16:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 10 librados: watch_flush exit 2024-06-11T19:16:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 20 client.54996.objecter shutdown clearing up homeless session... 2024-06-11T19:16:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 10 client.54996.objecter successfully canceled tick 2024-06-11T19:16:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 1 -- 172.21.15.98:0/2234546589 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f33e803fdb0 msgr2=0x7f33e8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f33e803fdb0 0x7f33e8042270 secure :-1 s=READY pgs=4148 cs=0 l=1 rev1=1 crypto rx=0x7f33f4003cb0 tx=0x7f33f40095f0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 10 monclient: shutdown 2024-06-11T19:16:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 1 -- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f33fc0a4920 msgr2=0x7f33fc1370f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f33fc0a4920 0x7f33fc1370f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f33f00099c0 tx=0x7f33f00046c0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3400cde640 1 -- 172.21.15.98:0/2234546589 reap_dead start 2024-06-11T19:16:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 1 -- 172.21.15.98:0/2234546589 shutdown_connections 2024-06-11T19:16:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 1 --2- 172.21.15.98:0/2234546589 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f33fc0a4920 0x7f33fc1370f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 1 -- 172.21.15.98:0/2234546589 >> 172.21.15.98:0/2234546589 conn(0x7f33fc0ab850 msgr2=0x7f33fc137960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 1 -- 172.21.15.98:0/2234546589 shutdown_connections 2024-06-11T19:16:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 1 -- 172.21.15.98:0/2234546589 wait complete. 2024-06-11T19:16:03.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.477+0000 7f3402741640 1 librados: shutdown 2024-06-11T19:16:03.523 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:16:03.523 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon c 2024-06-11T19:16:03.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:16:03.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.623+0000 7f706f32a640 10 monclient: get_monmap_and_config 2024-06-11T19:16:03.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.623+0000 7f706f32a640 10 monclient: build_initial_monmap 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.623+0000 7f706f32a640 10 monclient: monmap: 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:16:03.624614+0000 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:16:03.624614+0000 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:03.623 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:03.624 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:03.624 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:03.624 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:16:03.624 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:16:03.624 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 1 Processor -- start 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 1 -- start start 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 10 monclient: init 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 10 monclient: _add_conns ranks=[0,2,4,6,8,7,5,10,3,1,9] 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70680a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680a2530 0x7f706813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 10 monclient(hunting): picked mon.noname-b con 0x7f70680a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f706813e1a0 0x7f7068140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 10 monclient(hunting): picked mon.noname-c con 0x7f706813e1a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70680a7140 con 0x7f70680a4920 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f70680a2b50 con 0x7f70680a2530 2024-06-11T19:16:03.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f70680a2e30 con 0x7f706813e1a0 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706f32a640 10 monclient(hunting): authenticate will time out at 1173.431854s 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706c8c5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680a2530 0x7f706813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706d8c7640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f706813e1a0 0x7f7068140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706c8c5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680a2530 0x7f706813dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45312/0 (socket says 172.21.15.98:45312) 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706c8c5640 1 -- 172.21.15.98:0/3956413728 learned_addr learned my addr 172.21.15.98:0/3956413728 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706c8c5640 10 monclient(hunting): get_auth_request con 0x7f70680a2530 auth_method 0 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706c8c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706c8c5640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706c8c5640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.624+0000 7f706d0c6640 1 --1- 172.21.15.98:0/3956413728 >> v1:172.21.15.98:6789/0 conn(0x7f70680a4920 0x7f70680a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35558/0 (socket says 172.21.15.98:35558) 2024-06-11T19:16:03.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706d8c7640 10 monclient(hunting): get_auth_request con 0x7f706813e1a0 auth_method 0 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706d8c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706d8c7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706d8c7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706c8c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706c8c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706c8c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706d8c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706d8c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706d8c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706c8c5640 10 monclient(hunting): handle_auth_done global_id 84305 payload 274 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706c8c5640 1 -- 172.21.15.98:0/3956413728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f706813e1a0 msgr2=0x7f7068140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706c8c5640 1 --2- 172.21.15.98:0/3956413728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f706813e1a0 0x7f7068140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706c8c5640 1 -- 172.21.15.98:0/3956413728 >> v1:172.21.15.98:6789/0 conn(0x7f70680a4920 legacy=0x7f70680a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706c8c5640 10 monclient: _finish_hunting 0 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706c8c5640 1 monclient: found mon.noname-b 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706c8c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706c8c5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706c8c5640 1 -- 172.21.15.98:0/3956413728 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70680a3110 con 0x7f70680a2530 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f705ffff640 1 -- 172.21.15.98:0/3956413728 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4202397895 0 0) 0x7f70680a7140 con 0x7f70680a4920 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f705ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.625+0000 7f706d8c7640 1 --2- 172.21.15.98:0/3956413728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f706813e1a0 0x7f7068140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706c8c5640 1 --2- 172.21.15.98:0/3956413728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680a2530 0x7f706813dc60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f70580098a0 tx=0x7f705802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c6006ecf74df853 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 1 -- 172.21.15.98:0/3956413728 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7058031a10 con 0x7f70680a2530 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 10 monclient: dump: 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.637 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.638 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 10 monclient: _finish_auth 0 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 5 monclient: authenticate success, global_id 84305 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 1 -- 172.21.15.98:0/3956413728 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7058037070 con 0x7f70680a2530 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 10 monclient: get_monmap_and_config success 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 4 set_mon_vals no callback set 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 1 -- 172.21.15.98:0/3956413728 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7058031bd0 con 0x7f70680a2530 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f705ffff640 10 monclient: dump: 2024-06-11T19:16:03.639 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 10 monclient: shutdown 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 1 -- 172.21.15.98:0/3956413728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680a2530 msgr2=0x7f706813dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 1 --2- 172.21.15.98:0/3956413728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680a2530 0x7f706813dc60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f70580098a0 tx=0x7f705802b840 comp rx=0 tx=0).stop 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 1 -- 172.21.15.98:0/3956413728 shutdown_connections 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 1 --2- 172.21.15.98:0/3956413728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f706813e1a0 0x7f7068140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 1 --2- 172.21.15.98:0/3956413728 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f70680a2530 0x7f706813dc60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 1 -- 172.21.15.98:0/3956413728 >> 172.21.15.98:0/3956413728 conn(0x7f70680ab850 msgr2=0x7f70680a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 1 -- 172.21.15.98:0/3956413728 shutdown_connections 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.626+0000 7f706f32a640 1 -- 172.21.15.98:0/3956413728 wait complete. 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.627+0000 7f706f32a640 10 monclient: build_initial_monmap 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.627+0000 7f706f32a640 10 monclient: monmap: 2024-06-11T19:16:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:03.642 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.627+0000 7f706f32a640 1 librados: starting msgr at 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.627+0000 7f706f32a640 1 librados: starting objecter 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.627+0000 7f706f32a640 1 Processor -- start 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.627+0000 7f706f32a640 1 -- start start 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.627+0000 7f706f32a640 1 librados: setting wanted keys 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.627+0000 7f706f32a640 1 librados: calling monclient init 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.627+0000 7f706f32a640 10 monclient: init 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 10 monclient: _add_conns ranks=[7,3,0,6,2,4,8,1,5] 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f70680a2530 0x7f7068137fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 10 monclient(hunting): picked mon.noname-i con 0x7f70680a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:16:03.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70680a4920 0x7f7068138510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 10 monclient(hunting): picked mon.noname-g con 0x7f70680a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f706813e1a0 0x7f70681215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 10 monclient(hunting): picked mon.noname-a con 0x7f706813e1a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7068138e50 con 0x7f706813e1a0 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f70680a84b0 con 0x7f70680a4920 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f70680a6da0 con 0x7f70680a2530 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 10 monclient(hunting): authenticate will time out at 1173.435154s 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706c8c5640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70680a4920 msgr2=0x7f7068138510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f706813e1a0 0x7f70681215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706c8c5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70680a4920 0x7f7068138510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f706813e1a0 0x7f70681215f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32904/0 (socket says 172.21.15.98:32904) 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 1 -- 172.21.15.98:0/2895264786 learned_addr learned my addr 172.21.15.98:0/2895264786 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 10 monclient(hunting): get_auth_request con 0x7f706813e1a0 auth_method 0 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d0c6640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f70680a2530 0x7f7068137fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d0c6640 10 monclient(hunting): get_auth_request con 0x7f70680a2530 auth_method 0 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d0c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d0c6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d0c6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d0c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d0c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d0c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 10 monclient(hunting): handle_auth_done global_id 64827 payload 824 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 1 -- 172.21.15.98:0/2895264786 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f70680a2530 msgr2=0x7f7068137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f70680a2530 0x7f7068137fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 1 -- 172.21.15.98:0/2895264786 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70680a4920 msgr2=0x7f7068138510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70680a4920 0x7f7068138510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 10 monclient: _finish_hunting 0 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 1 monclient: found mon.noname-a 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:03.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 1 -- 172.21.15.98:0/2895264786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7068124980 con 0x7f706813e1a0 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 10 client.?.objecter ms_handle_connect 0x7f706813e1a0 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706d8c7640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f706813e1a0 0x7f70681215f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7060009e60 tx=0x7f706000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7060018030 con 0x7f706813e1a0 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 10 monclient: dump: 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.647 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 10 monclient: _finish_auth 0 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 10 monclient: _check_auth_tickets 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f706000cd90 con 0x7f706813e1a0 2024-06-11T19:16:03.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 5 monclient: authenticate success, global_id 64827 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 10 monclient: _renew_subs 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f706f32a640 1 -- 172.21.15.98:0/2895264786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70680aa6d0 con 0x7f706813e1a0 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7060014d60 con 0x7f706813e1a0 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705dffb640 10 monclient: dump: 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.649 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.628+0000 7f705f7fe640 4 set_mon_vals no callback set 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.629+0000 7f706f32a640 10 client.64827.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.629+0000 7f706f32a640 10 monclient: _renew_subs 2024-06-11T19:16:03.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.629+0000 7f706f32a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.629+0000 7f706f32a640 1 -- 172.21.15.98:0/2895264786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f706813de80 con 0x7f706813e1a0 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.629+0000 7f706f32a640 1 librados: init done 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.629+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f7060005d10 con 0x7f706813e1a0 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.629+0000 7f706f32a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.629+0000 7f706f32a640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f70681242e0 0x7f7068121b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.629+0000 7f706f32a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.629+0000 7f706f32a640 1 -- 172.21.15.98:0/2895264786 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7068050830 con 0x7f70681242e0 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706d0c6640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f70681242e0 0x7f7068121b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706d0c6640 10 monclient: get_auth_request con 0x7f70681242e0 auth_method 0 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706d0c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706d0c6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706d0c6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f704c03fdb0 0x7f704c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706d0c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706d0c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706d0c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(291..291 src has 1..291) ==== 6686+0+0 (secure 0 0 0) 0x7f7060019070 con 0x7f706813e1a0 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 10 client.64827.objecter ms_dispatch 0x7f7068000b90 osd_map(291..291 src has 1..291) 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 3 client.64827.objecter handle_osd_map got epochs [291,291] > 0 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 3 client.64827.objecter handle_osd_map decoding full epoch 291 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706d0c6640 10 monclient(hunting): handle_auth_done global_id 23827 payload 824 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706d0c6640 1 -- 172.21.15.98:0/2895264786 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70681ce560 con 0x7f70681242e0 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 20 client.64827.objecter dump_active .. 0 homeless 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706c8c5640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f704c03fdb0 0x7f704c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706c8c5640 10 monclient: get_auth_request con 0x7f704c03fdb0 auth_method 0 2024-06-11T19:16:03.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 10 client.64827.objecter ms_handle_connect 0x7f70681242e0 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706d0c6640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f70681242e0 0x7f7068121b30 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f7064000c00 tx=0x7f70640019b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 10 client.64827.objecter resend_mon_ops 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70640119f0 con 0x7f70681242e0 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 10 monclient: dump: 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7064018070 con 0x7f70681242e0 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f70681242e0 msgr2=0x7f7068121b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f70681242e0 0x7f7068121b30 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f7064000c00 tx=0x7f70640019b0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f706c8c5640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f704c03fdb0 0x7f704c042270 secure :-1 s=READY pgs=4149 cs=0 l=1 rev1=1 crypto rx=0x7f7058036000 tx=0x7f7058008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f70681242e0 msgr2=0x7f7068121b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:03.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f70681242e0 0x7f7068121b30 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.630+0000 7f705dffb640 10 client.64827.objecter ms_handle_connect 0x7f704c03fdb0 2024-06-11T19:16:03.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706f32a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:16:03.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706f32a640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7068122070 0x7f706809c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706f32a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706f32a640 1 -- 172.21.15.98:0/2895264786 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7068122450 con 0x7f7068122070 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706d0c6640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7068122070 0x7f706809c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706d0c6640 10 monclient: get_auth_request con 0x7f7068122070 auth_method 0 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706d0c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706d0c6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706d0c6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706d0c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706d0c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706d0c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706d0c6640 10 monclient(hunting): handle_auth_done global_id 23836 payload 824 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706d0c6640 1 -- 172.21.15.98:0/2895264786 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f70680a4d00 con 0x7f7068122070 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f705dffb640 10 client.64827.objecter ms_handle_connect 0x7f7068122070 2024-06-11T19:16:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f706d0c6640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7068122070 0x7f706809c9e0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f7064000fb0 tx=0x7f7064000ed0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f705dffb640 10 client.64827.objecter resend_mon_ops 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7064013860 con 0x7f7068122070 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f705dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f705dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.637+0000 7f705dffb640 10 monclient: dump: 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9421 (secure 0 0 0) 0x7f70640133f0 con 0x7f7068122070 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f705dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f705dffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7068122070 msgr2=0x7f706809c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f705dffb640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7068122070 0x7f706809c9e0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f7064000fb0 tx=0x7f7064000ed0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f705dffb640 1 -- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7068122070 msgr2=0x7f706809c9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f705dffb640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7068122070 0x7f706809c9e0 secure :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f7064000fb0 tx=0x7f7064000ed0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.660 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:16:03.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:16:03.660 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:16:03.660 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:16:03.660 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 34, 2024-06-11T19:16:03.660 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 413855, 2024-06-11T19:16:03.660 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:16:03.660 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:16:03.660 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:16:03.660 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.661 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:16:03.663 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.664 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:16:03.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.666 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:16:03.671 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:16:03.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:16:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.676 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:16:03.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.679 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:03.680 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:16:03.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.681 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.681 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:16:03.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.681 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:03.681 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:03.681 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:16:03.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.681 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.681 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:16:03.681 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:16:03.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 10 librados: watch_flush enter 2024-06-11T19:16:03.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 10 librados: watch_flush exit 2024-06-11T19:16:03.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 20 client.64827.objecter shutdown clearing up homeless session... 2024-06-11T19:16:03.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 10 client.64827.objecter successfully canceled tick 2024-06-11T19:16:03.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 1 -- 172.21.15.98:0/2895264786 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f704c03fdb0 msgr2=0x7f704c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f704c03fdb0 0x7f704c042270 secure :-1 s=READY pgs=4149 cs=0 l=1 rev1=1 crypto rx=0x7f7058036000 tx=0x7f7058008030 comp rx=0 tx=0).stop 2024-06-11T19:16:03.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 10 monclient: shutdown 2024-06-11T19:16:03.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:03.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 1 -- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f706813e1a0 msgr2=0x7f70681215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 1 --2- 172.21.15.98:0/2895264786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f706813e1a0 0x7f70681215f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7060009e60 tx=0x7f706000c490 comp rx=0 tx=0).stop 2024-06-11T19:16:03.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706d8c7640 1 -- 172.21.15.98:0/2895264786 reap_dead start 2024-06-11T19:16:03.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706d8c7640 1 -- 172.21.15.98:0/2895264786 reap_dead start 2024-06-11T19:16:03.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 1 -- 172.21.15.98:0/2895264786 shutdown_connections 2024-06-11T19:16:03.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 1 -- 172.21.15.98:0/2895264786 >> 172.21.15.98:0/2895264786 conn(0x7f70680ab850 msgr2=0x7f70680a8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:03.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 1 -- 172.21.15.98:0/2895264786 shutdown_connections 2024-06-11T19:16:03.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 1 -- 172.21.15.98:0/2895264786 wait complete. 2024-06-11T19:16:03.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.638+0000 7f706f32a640 1 librados: shutdown 2024-06-11T19:16:03.684 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:16:03.685 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon d 2024-06-11T19:16:03.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:16:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.785+0000 7f583e27a640 10 monclient: get_monmap_and_config 2024-06-11T19:16:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.785+0000 7f583e27a640 10 monclient: build_initial_monmap 2024-06-11T19:16:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.785+0000 7f583e27a640 10 monclient: monmap: 2024-06-11T19:16:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:16:03.786213+0000 2024-06-11T19:16:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:16:03.786213+0000 2024-06-11T19:16:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:16:03.785 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.785+0000 7f583e27a640 1 Processor -- start 2024-06-11T19:16:03.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.785+0000 7f583e27a640 1 -- start start 2024-06-11T19:16:03.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 10 monclient: init 2024-06-11T19:16:03.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:03.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 10 monclient: _add_conns ranks=[5,2,3,4,0,1,6,7,10,8,9] 2024-06-11T19:16:03.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58380a0cf0 0x7f5838138df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 10 monclient(hunting): picked mon.noname-h con 0x7f58380a0cf0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58380a3a70 0x7f58380a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 10 monclient(hunting): picked mon.noname-b con 0x7f58380a3a70 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58380a2250 0x7f58380a26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 10 monclient(hunting): picked mon.noname-g con 0x7f58380a2250 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5838139850 con 0x7f58380a3a70 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f5838073920 con 0x7f58380a2250 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f583809ecf0 con 0x7f58380a0cf0 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583e27a640 10 monclient(hunting): authenticate will time out at 1173.593475s 2024-06-11T19:16:03.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58380a3a70 0x7f58380a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58380a3a70 0x7f58380a10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45316/0 (socket says 172.21.15.98:45316) 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 1 -- 172.21.15.98:0/1270852788 learned_addr learned my addr 172.21.15.98:0/1270852788 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 10 monclient(hunting): get_auth_request con 0x7f58380a3a70 auth_method 0 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583c817640 1 -- 172.21.15.98:0/1270852788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58380a2250 msgr2=0x7f58380a26d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f583c817640 1 --2- 172.21.15.98:0/1270852788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58380a2250 0x7f58380a26d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 10 monclient(hunting): handle_auth_done global_id 84314 payload 274 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 1 -- 172.21.15.98:0/1270852788 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58380a0cf0 msgr2=0x7f5838138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 1 --2- 172.21.15.98:0/1270852788 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58380a0cf0 0x7f5838138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 1 -- 172.21.15.98:0/1270852788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58380a2250 msgr2=0x7f58380a26d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 1 --2- 172.21.15.98:0/1270852788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58380a2250 0x7f58380a26d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 10 monclient: _finish_hunting 0 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 1 monclient: found mon.noname-b 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.786+0000 7f58377fe640 1 -- 172.21.15.98:0/1270852788 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58380769d0 con 0x7f58380a3a70 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f58377fe640 1 --2- 172.21.15.98:0/1270852788 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58380a3a70 0x7f58380a10d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5820009880 tx=0x7f582002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4fcb904a3bd8fa3 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 1 -- 172.21.15.98:0/1270852788 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5820038070 con 0x7f58380a3a70 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 10 monclient: dump: 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.798 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 10 monclient: _finish_auth 0 2024-06-11T19:16:03.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 1 -- 172.21.15.98:0/1270852788 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58200044a0 con 0x7f58380a3a70 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f583e27a640 5 monclient: authenticate success, global_id 84314 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f583e27a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 1 -- 172.21.15.98:0/1270852788 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5820003a80 con 0x7f58380a3a70 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f5836ffd640 10 monclient: dump: 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.800 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f583e27a640 10 monclient: get_monmap_and_config success 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.787+0000 7f583e27a640 4 set_mon_vals no callback set 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 10 monclient: shutdown 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 1 -- 172.21.15.98:0/1270852788 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58380a3a70 msgr2=0x7f58380a10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 1 --2- 172.21.15.98:0/1270852788 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58380a3a70 0x7f58380a10d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5820009880 tx=0x7f582002b7a0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 1 -- 172.21.15.98:0/1270852788 shutdown_connections 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 1 --2- 172.21.15.98:0/1270852788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58380a2250 0x7f58380a26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 1 --2- 172.21.15.98:0/1270852788 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f58380a3a70 0x7f58380a10d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 1 --2- 172.21.15.98:0/1270852788 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58380a0cf0 0x7f5838138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 1 -- 172.21.15.98:0/1270852788 >> 172.21.15.98:0/1270852788 conn(0x7f58380a7850 msgr2=0x7f58380a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 1 -- 172.21.15.98:0/1270852788 shutdown_connections 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 1 -- 172.21.15.98:0/1270852788 wait complete. 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 10 monclient: build_initial_monmap 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.788+0000 7f583e27a640 10 monclient: monmap: 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:16:03.802 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.789+0000 7f583e27a640 1 librados: starting msgr at 2024-06-11T19:16:03.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.789+0000 7f583e27a640 1 librados: starting objecter 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.789+0000 7f583e27a640 1 Processor -- start 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.789+0000 7f583e27a640 1 -- start start 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.789+0000 7f583e27a640 1 librados: setting wanted keys 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.789+0000 7f583e27a640 1 librados: calling monclient init 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.789+0000 7f583e27a640 10 monclient: init 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.789+0000 7f583e27a640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.789+0000 7f583e27a640 10 monclient: _add_conns ranks=[7,5,3,8,0,6,1,2,4] 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.789+0000 7f583e27a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58380a0cf0 0x7f58381cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.789+0000 7f583e27a640 10 monclient(hunting): picked mon.noname-i con 0x7f58380a0cf0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.789+0000 7f583e27a640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58380a2250 0x7f58381376e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583e27a640 10 monclient(hunting): picked mon.noname-h con 0x7f58380a2250 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583e27a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58380a3a70 0x7f5838137c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583e27a640 10 monclient(hunting): picked mon.noname-g con 0x7f58380a3a70 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583e27a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583e27a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f58380a2d90 con 0x7f58380a3a70 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583e27a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583e27a640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f58380a2c10 con 0x7f58380a2250 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583e27a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583e27a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f58380a2f10 con 0x7f58380a0cf0 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583e27a640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583e27a640 10 monclient(hunting): authenticate will time out at 1173.597054s 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583c817640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58380a3a70 msgr2=0x7f5838137c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f583c817640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58380a3a70 0x7f5838137c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58380a0cf0 0x7f58381cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58380a0cf0 0x7f58381cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:58846/0 (socket says 172.21.15.98:58846) 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 1 -- 172.21.15.98:0/2254769789 learned_addr learned my addr 172.21.15.98:0/2254769789 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 10 monclient(hunting): get_auth_request con 0x7f58380a0cf0 auth_method 0 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f58377fe640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58380a2250 0x7f58381376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f58377fe640 10 monclient(hunting): get_auth_request con 0x7f58380a2250 auth_method 0 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f58377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f58377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f58377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f58377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f58377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f58377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 10 monclient(hunting): handle_auth_done global_id 84193 payload 824 2024-06-11T19:16:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 1 -- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58380a2250 msgr2=0x7f58381376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58380a2250 0x7f58381376e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 1 -- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58380a3a70 msgr2=0x7f5838137c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58380a3a70 0x7f5838137c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 10 monclient: _finish_hunting 0 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 1 monclient: found mon.noname-i 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 1 -- 172.21.15.98:0/2254769789 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5838132a00 con 0x7f58380a0cf0 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f58377fe640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f58380a2250 0x7f58381376e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5837fff640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58380a0cf0 0x7f58381cdaa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5838133ab0 tx=0x7f582800b960 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5834ff9640 10 client.?.objecter ms_handle_connect 0x7f58380a0cf0 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.790+0000 7f5834ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5828014070 con 0x7f58380a0cf0 2024-06-11T19:16:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 10 monclient: dump: 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 10 monclient: _finish_auth 0 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 5 monclient: authenticate success, global_id 84193 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5828010a80 con 0x7f58380a0cf0 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5828010c30 con 0x7f58380a0cf0 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 10 monclient: _renew_subs 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:16:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 1 -- 172.21.15.98:0/2254769789 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5838133eb0 con 0x7f58380a0cf0 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f58367fc640 4 set_mon_vals no callback set 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f5834ff9640 10 monclient: dump: 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 10 client.84193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 10 monclient: _renew_subs 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 1 -- 172.21.15.98:0/2254769789 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f58380a55c0 con 0x7f58380a0cf0 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 1 librados: init done 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58381301d0 0x7f58381305b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.791+0000 7f583e27a640 1 -- 172.21.15.98:0/2254769789 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5838050830 con 0x7f58381301d0 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f58377fe640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58381301d0 0x7f58381305b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f58377fe640 10 monclient: get_auth_request con 0x7f58381301d0 auth_method 0 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f58377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f58377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f58377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f58377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f58377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f58377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 <== mon.7 v2:172.21.15.117:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f5828017070 con 0x7f58380a0cf0 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f58377fe640 10 monclient(hunting): handle_auth_done global_id 55005 payload 824 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f58377fe640 1 -- 172.21.15.98:0/2254769789 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f58380753a0 con 0x7f58381301d0 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f58377fe640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58381301d0 0x7f58381305b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5820035000 tx=0x7f58200046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f581803fdb0 0x7f5818042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 10 client.84193.objecter ms_handle_connect 0x7f58381301d0 2024-06-11T19:16:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 10 client.84193.objecter resend_mon_ops 2024-06-11T19:16:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 <== mon.7 v2:172.21.15.117:3303/0 5 ==== osd_map(291..291 src has 1..291) ==== 6686+0+0 (secure 0 0 0) 0x7f582800d040 con 0x7f58380a0cf0 2024-06-11T19:16:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 10 client.84193.objecter ms_dispatch 0x7f5838000b90 osd_map(291..291 src has 1..291) 2024-06-11T19:16:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 3 client.84193.objecter handle_osd_map got epochs [291,291] > 0 2024-06-11T19:16:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 3 client.84193.objecter handle_osd_map decoding full epoch 291 2024-06-11T19:16:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 20 client.84193.objecter dump_active .. 0 homeless 2024-06-11T19:16:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5820038040 con 0x7f58381301d0 2024-06-11T19:16:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:16:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 10 monclient: dump: 2024-06-11T19:16:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.815 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f583c817640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f581803fdb0 0x7f5818042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f583c817640 10 monclient: get_auth_request con 0x7f581803fdb0 auth_method 0 2024-06-11T19:16:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5820039040 con 0x7f58381301d0 2024-06-11T19:16:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:16:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58381301d0 msgr2=0x7f58381305b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58381301d0 0x7f58381305b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5820035000 tx=0x7f58200046c0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58381301d0 msgr2=0x7f58381305b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.792+0000 7f5834ff9640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58381301d0 0x7f58381305b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.793+0000 7f583c817640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f581803fdb0 0x7f5818042270 secure :-1 s=READY pgs=4150 cs=0 l=1 rev1=1 crypto rx=0x7f5830003c60 tx=0x7f58300091f0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.793+0000 7f5834ff9640 10 client.84193.objecter ms_handle_connect 0x7f581803fdb0 2024-06-11T19:16:03.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f583e27a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:16:03.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f583e27a640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5838130af0 0x7f583812c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f583e27a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f583e27a640 1 -- 172.21.15.98:0/2254769789 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f58380a26d0 con 0x7f5838130af0 2024-06-11T19:16:03.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f58377fe640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5838130af0 0x7f583812c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f58377fe640 10 monclient: get_auth_request con 0x7f5838130af0 auth_method 0 2024-06-11T19:16:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f58377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f58377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f58377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f58377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f58377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f58377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f58377fe640 10 monclient(hunting): handle_auth_done global_id 55014 payload 824 2024-06-11T19:16:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f58377fe640 1 -- 172.21.15.98:0/2254769789 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f58381ce1c0 con 0x7f5838130af0 2024-06-11T19:16:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f58377fe640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5838130af0 0x7f583812c4f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5820009880 tx=0x7f5820031000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f5834ff9640 10 client.84193.objecter ms_handle_connect 0x7f5838130af0 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f5834ff9640 10 client.84193.objecter resend_mon_ops 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5820038030 con 0x7f5838130af0 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f5834ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f5834ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.799+0000 7f5834ff9640 10 monclient: dump: 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9421 (secure 0 0 0) 0x7f582004a850 con 0x7f5838130af0 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f5834ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f5834ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5838130af0 msgr2=0x7f583812c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f5834ff9640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5838130af0 0x7f583812c4f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5820009880 tx=0x7f5820031000 comp rx=0 tx=0).stop 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f5834ff9640 1 -- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5838130af0 msgr2=0x7f583812c4f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f5834ff9640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5838130af0 0x7f583812c4f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.821 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:16:03.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:16:03.821 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:16:03.821 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:16:03.821 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 34, 2024-06-11T19:16:03.821 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 276555, 2024-06-11T19:16:03.821 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:16:03.821 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:16:03.821 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:16:03.821 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.822 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.823 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:16:03.824 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.825 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:16:03.826 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:16:03.829 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.831 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:16:03.834 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:16:03.836 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:16:03.837 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:16:03.838 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:16:03.839 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.839 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:16:03.839 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:16:03.839 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.839 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:16:03.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.842 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:16:03.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.843 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:03.843 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:03.843 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:16:03.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.843 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.843 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:16:03.843 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:16:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 10 librados: watch_flush enter 2024-06-11T19:16:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 10 librados: watch_flush exit 2024-06-11T19:16:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 20 client.84193.objecter shutdown clearing up homeless session... 2024-06-11T19:16:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 10 client.84193.objecter successfully canceled tick 2024-06-11T19:16:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 1 -- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f581803fdb0 msgr2=0x7f5818042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f581803fdb0 0x7f5818042270 secure :-1 s=READY pgs=4150 cs=0 l=1 rev1=1 crypto rx=0x7f5830003c60 tx=0x7f58300091f0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 10 monclient: shutdown 2024-06-11T19:16:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 1 -- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58380a0cf0 msgr2=0x7f58381cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 1 --2- 172.21.15.98:0/2254769789 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58380a0cf0 0x7f58381cdaa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5838133ab0 tx=0x7f582800b960 comp rx=0 tx=0).stop 2024-06-11T19:16:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583c817640 1 -- 172.21.15.98:0/2254769789 reap_dead start 2024-06-11T19:16:03.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583c817640 1 -- 172.21.15.98:0/2254769789 reap_dead start 2024-06-11T19:16:03.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 1 -- 172.21.15.98:0/2254769789 shutdown_connections 2024-06-11T19:16:03.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 1 -- 172.21.15.98:0/2254769789 >> 172.21.15.98:0/2254769789 conn(0x7f58380a7850 msgr2=0x7f58380a5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:03.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 1 -- 172.21.15.98:0/2254769789 shutdown_connections 2024-06-11T19:16:03.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 1 -- 172.21.15.98:0/2254769789 wait complete. 2024-06-11T19:16:03.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.800+0000 7f583e27a640 1 librados: shutdown 2024-06-11T19:16:03.846 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:16:03.846 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon f 2024-06-11T19:16:03.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:16:03.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.952+0000 7fd74f195640 10 monclient: get_monmap_and_config 2024-06-11T19:16:03.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.952+0000 7fd74f195640 10 monclient: build_initial_monmap 2024-06-11T19:16:03.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.952+0000 7fd74f195640 10 monclient: monmap: 2024-06-11T19:16:03.951 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:03.951 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.951 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:16:03.953361+0000 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:16:03.953361+0000 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:03.952 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:16:03.953 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:16:03.953 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 1 Processor -- start 2024-06-11T19:16:03.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 1 -- start start 2024-06-11T19:16:03.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 10 monclient: init 2024-06-11T19:16:03.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:03.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 10 monclient: _add_conns ranks=[0,2,9,6,3,4,1,5,10,7,8] 2024-06-11T19:16:03.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7480a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:16:03.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7480a2530 0x7fd74813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 10 monclient(hunting): picked mon.noname-b con 0x7fd7480a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:16:03.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd74813e1a0 0x7fd748140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 10 monclient(hunting): picked mon.noname-a con 0x7fd74813e1a0 addr v2:172.21.15.98:3300/0 2024-06-11T19:16:03.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7480a7140 con 0x7fd7480a4920 2024-06-11T19:16:03.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd7480a2b50 con 0x7fd74813e1a0 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd7480a2e30 con 0x7fd7480a2530 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74f195640 10 monclient(hunting): authenticate will time out at 1173.760602s 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd747fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7480a2530 0x7fd74813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74cf31640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd7480a4920 0x7fd7480a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35568/0 (socket says 172.21.15.98:35568) 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74d732640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd74813e1a0 0x7fd748140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74cf31640 1 -- 172.21.15.98:0/4159622418 learned_addr learned my addr 172.21.15.98:0/4159622418 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd747fff640 10 monclient(hunting): get_auth_request con 0x7fd7480a2530 auth_method 0 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd747fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd747fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74d732640 10 monclient(hunting): get_auth_request con 0x7fd74813e1a0 auth_method 0 2024-06-11T19:16:03.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74d732640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74d732640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74d732640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74d732640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74d732640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd74d732640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.953+0000 7fd747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74d732640 10 monclient(hunting): handle_auth_done global_id 64836 payload 274 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74d732640 1 -- 172.21.15.98:0/4159622418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7480a2530 msgr2=0x7fd74813dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74d732640 1 --2- 172.21.15.98:0/4159622418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7480a2530 0x7fd74813dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74d732640 1 -- 172.21.15.98:0/4159622418 >> v1:172.21.15.98:6789/0 conn(0x7fd7480a4920 legacy=0x7fd7480a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74d732640 10 monclient: _finish_hunting 0 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74d732640 1 monclient: found mon.noname-a 2024-06-11T19:16:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74d732640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74d732640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74d732640 1 -- 172.21.15.98:0/4159622418 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7480a3110 con 0x7fd74813e1a0 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 1 -- 172.21.15.98:0/4159622418 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2281089971 0 0) 0x7fd7480a7140 con 0x7fd7480a4920 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd747fff640 1 --2- 172.21.15.98:0/4159622418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7480a2530 0x7fd74813dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74d732640 1 --2- 172.21.15.98:0/4159622418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd74813e1a0 0x7fd748140590 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd740009d90 tx=0x7fd74002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f43483d776c45f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 1 -- 172.21.15.98:0/4159622418 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd740039070 con 0x7fd74813e1a0 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 10 monclient: dump: 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 10 monclient: _finish_auth 0 2024-06-11T19:16:03.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 10 monclient: _check_auth_tickets 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74f195640 5 monclient: authenticate success, global_id 64836 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 1 -- 172.21.15.98:0/4159622418 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd74002cda0 con 0x7fd74813e1a0 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74f195640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74f195640 10 monclient: get_monmap_and_config success 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 1 -- 172.21.15.98:0/4159622418 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7400327b0 con 0x7fd74813e1a0 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd74f195640 4 set_mon_vals no callback set 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.954+0000 7fd7477fe640 10 monclient: dump: 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.968 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.969 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.955+0000 7fd74f195640 10 monclient: shutdown 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.955+0000 7fd74f195640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.955+0000 7fd74f195640 1 -- 172.21.15.98:0/4159622418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd74813e1a0 msgr2=0x7fd748140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.955+0000 7fd74f195640 1 --2- 172.21.15.98:0/4159622418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd74813e1a0 0x7fd748140590 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd740009d90 tx=0x7fd74002bd80 comp rx=0 tx=0).stop 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.955+0000 7fd74f195640 1 -- 172.21.15.98:0/4159622418 shutdown_connections 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.955+0000 7fd74f195640 1 --2- 172.21.15.98:0/4159622418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd74813e1a0 0x7fd748140590 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.955+0000 7fd74f195640 1 --2- 172.21.15.98:0/4159622418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd7480a2530 0x7fd74813dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.955+0000 7fd74f195640 1 -- 172.21.15.98:0/4159622418 >> 172.21.15.98:0/4159622418 conn(0x7fd7480ab850 msgr2=0x7fd7480a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.955+0000 7fd74f195640 1 -- 172.21.15.98:0/4159622418 shutdown_connections 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.955+0000 7fd74f195640 1 -- 172.21.15.98:0/4159622418 wait complete. 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.956+0000 7fd74f195640 10 monclient: build_initial_monmap 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.956+0000 7fd74f195640 10 monclient: monmap: 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:16:03.970 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.956+0000 7fd74f195640 1 librados: starting msgr at 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.956+0000 7fd74f195640 1 librados: starting objecter 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.956+0000 7fd74f195640 1 Processor -- start 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.956+0000 7fd74f195640 1 -- start start 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.956+0000 7fd74f195640 1 librados: setting wanted keys 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.956+0000 7fd74f195640 1 librados: calling monclient init 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.956+0000 7fd74f195640 10 monclient: init 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 10 monclient: _add_conns ranks=[5,3,8,1,6,2,0,4,7] 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd7480a2530 0x7fd7481cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 10 monclient(hunting): picked mon.noname-h con 0x7fd7480a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7480a4920 0x7fd748136890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 10 monclient(hunting): picked mon.noname-g con 0x7fd7480a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd74813e1a0 0x7fd748136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 10 monclient(hunting): picked mon.noname-e con 0x7fd74813e1a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:16:03.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd748138e50 con 0x7fd7480a4920 2024-06-11T19:16:03.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd7480a84b0 con 0x7fd7480a2530 2024-06-11T19:16:03.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd7480a6da0 con 0x7fd74813e1a0 2024-06-11T19:16:03.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:03.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74f195640 10 monclient(hunting): authenticate will time out at 1173.764131s 2024-06-11T19:16:03.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74d732640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd74813e1a0 msgr2=0x7fd748136dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:16:03.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74d732640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd74813e1a0 0x7fd748136dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:03.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd747fff640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7480a4920 msgr2=0x7fd748136890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:03.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd747fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7480a4920 0x7fd748136890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:03.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd7480a2530 0x7fd7481cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd7480a2530 0x7fd7481cd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:34152/0 (socket says 172.21.15.98:34152) 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 1 -- 172.21.15.98:0/157487708 learned_addr learned my addr 172.21.15.98:0/157487708 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 10 monclient(hunting): get_auth_request con 0x7fd7480a2530 auth_method 0 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 10 monclient(hunting): handle_auth_done global_id 22928 payload 824 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 1 -- 172.21.15.98:0/157487708 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd74813e1a0 msgr2=0x7fd748136dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd74813e1a0 0x7fd748136dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 1 -- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7480a4920 msgr2=0x7fd748136890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:16:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7480a4920 0x7fd748136890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 10 monclient: _finish_hunting 0 2024-06-11T19:16:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 1 monclient: found mon.noname-h 2024-06-11T19:16:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:16:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 1 -- 172.21.15.98:0/157487708 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7481cd950 con 0x7fd7480a2530 2024-06-11T19:16:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd74cf31640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd7480a2530 0x7fd7481cd410 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00c450 tx=0x7fd73c00c910 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.957+0000 7fd7457fa640 10 client.?.objecter ms_handle_connect 0x7fd7480a2530 2024-06-11T19:16:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:16:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd73c018070 con 0x7fd7480a2530 2024-06-11T19:16:03.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:16:03.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 10 monclient: dump: 2024-06-11T19:16:03.977 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.977 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.977 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.977 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 10 monclient: _finish_auth 0 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 10 monclient: _check_auth_tickets 2024-06-11T19:16:03.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd74f195640 5 monclient: authenticate success, global_id 22928 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd73c0047e0 con 0x7fd7480a2530 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd74f195640 10 monclient: _renew_subs 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd74f195640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd74f195640 1 -- 172.21.15.98:0/157487708 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd748121ab0 con 0x7fd7480a2530 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd73c004e40 con 0x7fd7480a2530 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd74f195640 10 client.22928.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd746ffd640 4 set_mon_vals no callback set 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd7457fa640 10 monclient: dump: 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.979 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd74f195640 10 monclient: _renew_subs 2024-06-11T19:16:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd74f195640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd74f195640 1 -- 172.21.15.98:0/157487708 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7480a4340 con 0x7fd7480a2530 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.958+0000 7fd74f195640 1 librados: init done 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd74f195640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd74f195640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd74809f720 0x7fd74809fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd74f195640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd74f195640 1 -- 172.21.15.98:0/157487708 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd748050830 con 0x7fd74809f720 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 <== mon.5 v2:172.21.15.117:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd73c0049d0 con 0x7fd7480a2530 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd747fff640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd74809f720 0x7fd74809fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd747fff640 10 monclient: get_auth_request con 0x7fd74809f720 auth_method 0 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd747fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd747fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd7457fa640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd72c03fdb0 0x7fd72c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 <== mon.5 v2:172.21.15.117:3302/0 5 ==== osd_map(292..292 src has 1..292) ==== 7103+0+0 (secure 0 0 0) 0x7fd73c050f90 con 0x7fd7480a2530 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd7457fa640 10 client.22928.objecter ms_dispatch 0x7fd748000b90 osd_map(292..292 src has 1..292) 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd7457fa640 3 client.22928.objecter handle_osd_map got epochs [292,292] > 0 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd7457fa640 3 client.22928.objecter handle_osd_map decoding full epoch 292 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd7457fa640 20 client.22928.objecter dump_active .. 0 homeless 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd74d732640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd72c03fdb0 0x7fd72c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd747fff640 10 monclient(hunting): handle_auth_done global_id 31996 payload 824 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd747fff640 1 -- 172.21.15.98:0/157487708 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7480a1450 con 0x7fd74809f720 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd74d732640 10 monclient: get_auth_request con 0x7fd72c03fdb0 auth_method 0 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd7457fa640 10 client.22928.objecter ms_handle_connect 0x7fd74809f720 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd7457fa640 10 client.22928.objecter resend_mon_ops 2024-06-11T19:16:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.959+0000 7fd747fff640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd74809f720 0x7fd74809fb00 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fd7481ce530 tx=0x7fd73400ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd73400ee50 con 0x7fd74809f720 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd7457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd7457fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd7457fa640 10 monclient: dump: 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.983 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd7457fa640 10 client.22928.objecter ms_handle_connect 0x7fd72c03fdb0 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd74d732640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd72c03fdb0 0x7fd72c042270 secure :-1 s=READY pgs=4151 cs=0 l=1 rev1=1 crypto rx=0x7fd740036000 tx=0x7fd740030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd734004030 con 0x7fd74809f720 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd7457fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd7457fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd74809f720 msgr2=0x7fd74809fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd7457fa640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd74809f720 0x7fd74809fb00 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fd7481ce530 tx=0x7fd73400ec00 comp rx=0 tx=0).stop 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd74809f720 msgr2=0x7fd74809fb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:03.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.960+0000 7fd7457fa640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd74809f720 0x7fd74809fb00 secure :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fd7481ce530 tx=0x7fd73400ec00 comp rx=0 tx=0).stop 2024-06-11T19:16:03.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.966+0000 7fd74f195640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:16:03.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.966+0000 7fd74f195640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7480a0040 0x7fd74813bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:03.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.966+0000 7fd74f195640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:03.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.966+0000 7fd74f195640 1 -- 172.21.15.98:0/157487708 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd748069740 con 0x7fd7480a0040 2024-06-11T19:16:03.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd747fff640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7480a0040 0x7fd74813bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd747fff640 10 monclient: get_auth_request con 0x7fd7480a0040 auth_method 0 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd747fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd747fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd747fff640 10 monclient(hunting): handle_auth_done global_id 32005 payload 824 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd747fff640 1 -- 172.21.15.98:0/157487708 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd7480a4d00 con 0x7fd7480a0040 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd7457fa640 10 client.22928.objecter ms_handle_connect 0x7fd7480a0040 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd7457fa640 10 client.22928.objecter resend_mon_ops 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd747fff640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7480a0040 0x7fd74813bce0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7fd74813e170 tx=0x7fd734004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd734005e00 con 0x7fd7480a0040 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd7457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:03.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd7457fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.967+0000 7fd7457fa640 10 monclient: dump: 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:03.988 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:03.989 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:03.989 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:03.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9593 (secure 0 0 0) 0x7fd734003bf0 con 0x7fd7480a0040 2024-06-11T19:16:03.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd7457fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:16:03.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd7457fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:16:03.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7480a0040 msgr2=0x7fd74813bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:03.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd7457fa640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7480a0040 0x7fd74813bce0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7fd74813e170 tx=0x7fd734004730 comp rx=0 tx=0).stop 2024-06-11T19:16:03.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd7457fa640 1 -- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7480a0040 msgr2=0x7fd74813bce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:03.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd7457fa640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7480a0040 0x7fd74813bce0 secure :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7fd74813e170 tx=0x7fd734004730 comp rx=0 tx=0).stop 2024-06-11T19:16:03.989 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:16:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 34, 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 414185, 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:16:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:16:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:16:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:16:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:16:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:16:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:16:04.011 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:16:04.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 10 librados: watch_flush enter 2024-06-11T19:16:04.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 10 librados: watch_flush exit 2024-06-11T19:16:04.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 20 client.22928.objecter shutdown clearing up homeless session... 2024-06-11T19:16:04.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 10 client.22928.objecter successfully canceled tick 2024-06-11T19:16:04.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 1 -- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd72c03fdb0 msgr2=0x7fd72c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd72c03fdb0 0x7fd72c042270 secure :-1 s=READY pgs=4151 cs=0 l=1 rev1=1 crypto rx=0x7fd740036000 tx=0x7fd740030000 comp rx=0 tx=0).stop 2024-06-11T19:16:04.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 10 monclient: shutdown 2024-06-11T19:16:04.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:04.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 1 -- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd7480a2530 msgr2=0x7fd7481cd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 1 --2- 172.21.15.98:0/157487708 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd7480a2530 0x7fd7481cd410 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00c450 tx=0x7fd73c00c910 comp rx=0 tx=0).stop 2024-06-11T19:16:04.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74d732640 1 -- 172.21.15.98:0/157487708 reap_dead start 2024-06-11T19:16:04.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74d732640 1 -- 172.21.15.98:0/157487708 reap_dead start 2024-06-11T19:16:04.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 1 -- 172.21.15.98:0/157487708 shutdown_connections 2024-06-11T19:16:04.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 1 -- 172.21.15.98:0/157487708 >> 172.21.15.98:0/157487708 conn(0x7fd7480ab850 msgr2=0x7fd7480a8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:04.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 1 -- 172.21.15.98:0/157487708 shutdown_connections 2024-06-11T19:16:04.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 1 -- 172.21.15.98:0/157487708 wait complete. 2024-06-11T19:16:04.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:03.968+0000 7fd74f195640 1 librados: shutdown 2024-06-11T19:16:04.014 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:16:04.015 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon h 2024-06-11T19:16:04.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:16:04.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.115+0000 7fdcfb728640 10 monclient: get_monmap_and_config 2024-06-11T19:16:04.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.115+0000 7fdcfb728640 10 monclient: build_initial_monmap 2024-06-11T19:16:04.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.115+0000 7fdcfb728640 10 monclient: monmap: 2024-06-11T19:16:04.114 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:16:04.116216+0000 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:16:04.116216+0000 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:04.115 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:16:04.116 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:16:04.116 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.115+0000 7fdcfb728640 1 Processor -- start 2024-06-11T19:16:04.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 1 -- start start 2024-06-11T19:16:04.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 10 monclient: init 2024-06-11T19:16:04.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 10 monclient: _add_conns ranks=[9,4,3,10,6,0,1,7,8,5,2] 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdcf40a0cd0 0x7fdcf40a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf40a0cd0 addr v2:172.21.15.98:3300/0 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdcf4138c80 0x7fdcf4139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf4138c80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcf40a2250 0x7fdcf40a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 10 monclient(hunting): picked mon.noname-g con 0x7fdcf40a2250 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdcf40739b0 con 0x7fdcf40a0cd0 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf409ecf0 con 0x7fdcf40a2250 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fdcf409efd0 con 0x7fdcf4138c80 2024-06-11T19:16:04.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcfb728640 10 monclient(hunting): authenticate will time out at 1173.923539s 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf94c4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdcf40a0cd0 0x7fdcf40a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf8cc3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdcf4138c80 0x7fdcf4139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf94c4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdcf40a0cd0 0x7fdcf40a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32922/0 (socket says 172.21.15.98:32922) 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf94c4640 1 -- 172.21.15.98:0/3346630324 learned_addr learned my addr 172.21.15.98:0/3346630324 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf94c4640 10 monclient(hunting): get_auth_request con 0x7fdcf40a0cd0 auth_method 0 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf94c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf8cc3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdcf4138c80 0x7fdcf4139060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:33528/0 (socket says 172.21.15.98:33528) 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf94c4640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf94c4640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf9cc5640 1 -- 172.21.15.98:0/3346630324 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcf40a2250 msgr2=0x7fdcf40a26e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf8cc3640 10 monclient(hunting): get_auth_request con 0x7fdcf4138c80 auth_method 0 2024-06-11T19:16:04.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf8cc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf8cc3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf8cc3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf9cc5640 1 --2- 172.21.15.98:0/3346630324 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcf40a2250 0x7fdcf40a26e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf8cc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf8cc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.116+0000 7fdcf8cc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_done global_id 64854 payload 274 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcf94c4640 1 -- 172.21.15.98:0/3346630324 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdcf4138c80 msgr2=0x7fdcf4139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcf94c4640 1 --2- 172.21.15.98:0/3346630324 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdcf4138c80 0x7fdcf4139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcf94c4640 1 -- 172.21.15.98:0/3346630324 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcf40a2250 msgr2=0x7fdcf40a26e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:16:04.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcf94c4640 1 --2- 172.21.15.98:0/3346630324 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcf40a2250 0x7fdcf40a26e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcf94c4640 10 monclient: _finish_hunting 0 2024-06-11T19:16:04.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcf94c4640 1 monclient: found mon.noname-a 2024-06-11T19:16:04.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcf94c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:04.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcf94c4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:16:04.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcf94c4640 1 -- 172.21.15.98:0/3346630324 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf409f2b0 con 0x7fdcf40a0cd0 2024-06-11T19:16:04.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcf94c4640 1 --2- 172.21.15.98:0/3346630324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf40a0cd0 0x7fdcf40a10b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00098a0 tx=0x7fdcf002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7466d4f744b5f5b0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:04.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 1 -- 172.21.15.98:0/3346630324 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdcf0038070 con 0x7fdcf40a0cd0 2024-06-11T19:16:04.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:04.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:16:04.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 10 monclient: dump: 2024-06-11T19:16:04.129 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:04.129 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.130 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.130 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.130 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:04.130 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.130 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:04.130 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:04.130 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:04.130 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:04.130 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:04.130 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:04.130 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:04.130 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 10 monclient: _finish_auth 0 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 10 monclient: _check_auth_tickets 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcfb728640 5 monclient: authenticate success, global_id 64854 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcfb728640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 1 -- 172.21.15.98:0/3346630324 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf00319b0 con 0x7fdcf40a0cd0 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 1 -- 172.21.15.98:0/3346630324 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdcf002f050 con 0x7fdcf40a0cd0 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcfb728640 10 monclient: get_monmap_and_config success 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcfb728640 4 set_mon_vals no callback set 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.117+0000 7fdcebfff640 10 monclient: dump: 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:04.132 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.118+0000 7fdcfb728640 10 monclient: shutdown 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.118+0000 7fdcfb728640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.118+0000 7fdcfb728640 1 -- 172.21.15.98:0/3346630324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf40a0cd0 msgr2=0x7fdcf40a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.118+0000 7fdcfb728640 1 --2- 172.21.15.98:0/3346630324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf40a0cd0 0x7fdcf40a10b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00098a0 tx=0x7fdcf002b7e0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.118+0000 7fdcfb728640 1 -- 172.21.15.98:0/3346630324 shutdown_connections 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.118+0000 7fdcfb728640 1 --2- 172.21.15.98:0/3346630324 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdcf4138c80 0x7fdcf4139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.118+0000 7fdcfb728640 1 --2- 172.21.15.98:0/3346630324 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcf40a2250 0x7fdcf40a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.118+0000 7fdcfb728640 1 --2- 172.21.15.98:0/3346630324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf40a0cd0 0x7fdcf40a10b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.118+0000 7fdcfb728640 1 -- 172.21.15.98:0/3346630324 >> 172.21.15.98:0/3346630324 conn(0x7fdcf40a7850 msgr2=0x7fdcf40a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:04.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.118+0000 7fdcfb728640 1 -- 172.21.15.98:0/3346630324 shutdown_connections 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.118+0000 7fdcfb728640 1 -- 172.21.15.98:0/3346630324 wait complete. 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient: build_initial_monmap 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient: monmap: 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:04.134 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 1 librados: starting msgr at 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 1 librados: starting objecter 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 1 Processor -- start 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 1 -- start start 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 1 librados: setting wanted keys 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 1 librados: calling monclient init 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient: init 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:04.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient: _add_conns ranks=[0,4,3,6,1,2,7,8,5] 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf40a0cd0 0x7fdcf41cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf40a0cd0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdcf40a2250 0x7fdcf41ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf40a2250 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcf4138c80 0x7fdcf412f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient(hunting): picked mon.noname-g con 0x7fdcf4138c80 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf40a1770 con 0x7fdcf40a0cd0 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf40a15f0 con 0x7fdcf4138c80 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fdcf40a2c20 con 0x7fdcf40a2250 2024-06-11T19:16:04.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.119+0000 7fdcfb728640 10 monclient(hunting): authenticate will time out at 1173.926997s 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf94c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf40a0cd0 0x7fdcf41cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdcf40a2250 0x7fdcf41ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf94c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf40a0cd0 0x7fdcf41cdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32930/0 (socket says 172.21.15.98:32930) 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf94c4640 1 -- 172.21.15.98:0/4223826489 learned_addr learned my addr 172.21.15.98:0/4223826489 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 10 monclient(hunting): get_auth_request con 0x7fdcf40a2250 auth_method 0 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf94c4640 10 monclient(hunting): get_auth_request con 0x7fdcf40a0cd0 auth_method 0 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf94c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf94c4640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf94c4640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf9cc5640 1 -- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcf4138c80 msgr2=0x7fdcf412f030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf9cc5640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcf4138c80 0x7fdcf412f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 10 monclient(hunting): handle_auth_done global_id 23854 payload 824 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 1 -- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcf4138c80 msgr2=0x7fdcf412f030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcf4138c80 0x7fdcf412f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 1 -- 172.21.15.98:0/4223826489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf40a0cd0 msgr2=0x7fdcf41cdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf40a0cd0 0x7fdcf41cdd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 10 monclient: _finish_hunting 0 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 1 monclient: found mon.noname-c 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 1 -- 172.21.15.98:0/4223826489 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf4132cd0 con 0x7fdcf40a2250 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf94c4640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcf40a0cd0 0x7fdcf41cdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcf8cc3640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdcf40a2250 0x7fdcf41ce2c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fdce400e8a0 tx=0x7fdce400ed60 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 10 client.?.objecter ms_handle_connect 0x7fdcf40a2250 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdce4017070 con 0x7fdcf40a2250 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 10 monclient: dump: 2024-06-11T19:16:04.139 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:04.140 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 10 monclient: _finish_auth 0 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce400cdb0 con 0x7fdcf40a2250 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcfb728640 5 monclient: authenticate success, global_id 23854 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcfb728640 10 monclient: _renew_subs 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcfb728640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcfb728640 1 -- 172.21.15.98:0/4223826489 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf4131160 con 0x7fdcf40a2250 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdce40138e0 con 0x7fdcf40a2250 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:16:04.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdce9ffb640 10 monclient: dump: 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:04.142 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcfb728640 10 client.23854.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdceb7fe640 4 set_mon_vals no callback set 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcfb728640 10 monclient: _renew_subs 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcfb728640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcfb728640 1 -- 172.21.15.98:0/4223826489 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf40a28c0 con 0x7fdcf40a2250 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.120+0000 7fdcfb728640 1 librados: init done 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.121+0000 7fdcfb728640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.121+0000 7fdcfb728640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412f570 0x7fdcf412f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.121+0000 7fdcfb728640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.121+0000 7fdcfb728640 1 -- 172.21.15.98:0/4223826489 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fdcf4050830 con 0x7fdcf412f570 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.121+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fdce401a070 con 0x7fdcf40a2250 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.121+0000 7fdcf94c4640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412f570 0x7fdcf412f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.121+0000 7fdcf94c4640 10 monclient: get_auth_request con 0x7fdcf412f570 auth_method 0 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.121+0000 7fdcf94c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.121+0000 7fdcf94c4640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.121+0000 7fdcf94c4640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.121+0000 7fdce9ffb640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdcd803fdb0 0x7fdcd8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(292..292 src has 1..292) ==== 7103+0+0 (secure 0 0 0) 0x7fdce400f040 con 0x7fdcf40a2250 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 10 client.23854.objecter ms_dispatch 0x7fdcf4000b90 osd_map(292..292 src has 1..292) 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 3 client.23854.objecter handle_osd_map got epochs [292,292] > 0 2024-06-11T19:16:04.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 3 client.23854.objecter handle_osd_map decoding full epoch 292 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_done global_id 22937 payload 824 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdcf94c4640 1 -- 172.21.15.98:0/4223826489 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdcf4138040 con 0x7fdcf412f570 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 20 client.23854.objecter dump_active .. 0 homeless 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdcf9cc5640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdcd803fdb0 0x7fdcd8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdcf9cc5640 10 monclient: get_auth_request con 0x7fdcd803fdb0 auth_method 0 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 10 client.23854.objecter ms_handle_connect 0x7fdcf412f570 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 10 client.23854.objecter resend_mon_ops 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdcf94c4640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412f570 0x7fdcf412f950 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00097c0 tx=0x7fdcf00046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdcf0044070 con 0x7fdcf412f570 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 10 monclient: dump: 2024-06-11T19:16:04.145 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:04.146 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.146 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.146 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.146 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:04.146 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.146 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:04.146 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:04.146 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:04.146 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 10 client.23854.objecter ms_handle_connect 0x7fdcd803fdb0 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdcf9cc5640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdcd803fdb0 0x7fdcd8042270 secure :-1 s=READY pgs=4152 cs=0 l=1 rev1=1 crypto rx=0x7fdcec003c60 tx=0x7fdcec0091f0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdcf0003610 con 0x7fdcf412f570 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412f570 msgr2=0x7fdcf412f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412f570 0x7fdcf412f950 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00097c0 tx=0x7fdcf00046c0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412f570 msgr2=0x7fdcf412f950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:04.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.122+0000 7fdce9ffb640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412f570 0x7fdcf412f950 secure :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00097c0 tx=0x7fdcf00046c0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcfb728640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:16:04.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcfb728640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412fe90 0x7fdcf41bf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcfb728640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcfb728640 1 -- 172.21.15.98:0/4223826489 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fdcf40a46d0 con 0x7fdcf412fe90 2024-06-11T19:16:04.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcf94c4640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412fe90 0x7fdcf41bf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcf94c4640 10 monclient: get_auth_request con 0x7fdcf412fe90 auth_method 0 2024-06-11T19:16:04.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcf94c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcf94c4640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcf94c4640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcf94c4640 10 monclient(hunting): handle_auth_done global_id 22946 payload 824 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcf94c4640 1 -- 172.21.15.98:0/4223826489 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdcf40a2f80 con 0x7fdcf412fe90 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdce9ffb640 10 client.23854.objecter ms_handle_connect 0x7fdcf412fe90 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdce9ffb640 10 client.23854.objecter resend_mon_ops 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdcf94c4640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412fe90 0x7fdcf41bf610 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fdcf002b820 tx=0x7fdcf0031000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdcf002f040 con 0x7fdcf412fe90 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdce9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdce9ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.129+0000 7fdce9ffb640 10 monclient: dump: 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:04.151 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:04.152 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:04.153 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9421 (secure 0 0 0) 0x7fdcf0049a30 con 0x7fdcf412fe90 2024-06-11T19:16:04.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdce9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:16:04.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdce9ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:16:04.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412fe90 msgr2=0x7fdcf41bf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdce9ffb640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412fe90 0x7fdcf41bf610 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fdcf002b820 tx=0x7fdcf0031000 comp rx=0 tx=0).stop 2024-06-11T19:16:04.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdce9ffb640 1 -- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412fe90 msgr2=0x7fdcf41bf610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:04.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdce9ffb640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdcf412fe90 0x7fdcf41bf610 secure :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fdcf002b820 tx=0x7fdcf0031000 comp rx=0 tx=0).stop 2024-06-11T19:16:04.153 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:16:04.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 34, 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 414324, 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:16:04.154 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:04.155 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:04.156 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:16:04.157 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:04.158 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:16:04.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:16:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:16:04.162 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:16:04.164 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:16:04.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.170 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:16:04.172 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:04.174 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:04.175 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:04.176 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:16:04.176 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.176 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.176 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:16:04.176 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:16:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 10 librados: watch_flush enter 2024-06-11T19:16:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 10 librados: watch_flush exit 2024-06-11T19:16:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 20 client.23854.objecter shutdown clearing up homeless session... 2024-06-11T19:16:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 10 client.23854.objecter successfully canceled tick 2024-06-11T19:16:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 1 -- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdcd803fdb0 msgr2=0x7fdcd8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdcd803fdb0 0x7fdcd8042270 secure :-1 s=READY pgs=4152 cs=0 l=1 rev1=1 crypto rx=0x7fdcec003c60 tx=0x7fdcec0091f0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 10 monclient: shutdown 2024-06-11T19:16:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 1 -- 172.21.15.98:0/4223826489 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdcf40a2250 msgr2=0x7fdcf41ce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 1 --2- 172.21.15.98:0/4223826489 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdcf40a2250 0x7fdcf41ce2c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fdce400e8a0 tx=0x7fdce400ed60 comp rx=0 tx=0).stop 2024-06-11T19:16:04.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcf9cc5640 1 -- 172.21.15.98:0/4223826489 reap_dead start 2024-06-11T19:16:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcf9cc5640 1 -- 172.21.15.98:0/4223826489 reap_dead start 2024-06-11T19:16:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 1 -- 172.21.15.98:0/4223826489 shutdown_connections 2024-06-11T19:16:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 1 -- 172.21.15.98:0/4223826489 >> 172.21.15.98:0/4223826489 conn(0x7fdcf40a7850 msgr2=0x7fdcf409d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 1 -- 172.21.15.98:0/4223826489 shutdown_connections 2024-06-11T19:16:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 1 -- 172.21.15.98:0/4223826489 wait complete. 2024-06-11T19:16:04.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.130+0000 7fdcfb728640 1 librados: shutdown 2024-06-11T19:16:04.179 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:16:04.179 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon i 2024-06-11T19:16:04.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:16:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.280+0000 7fad3a95f640 10 monclient: get_monmap_and_config 2024-06-11T19:16:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.280+0000 7fad3a95f640 10 monclient: build_initial_monmap 2024-06-11T19:16:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.280+0000 7fad3a95f640 10 monclient: monmap: 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:16:04.281209+0000 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:16:04.281209+0000 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:16:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:16:04.281 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 1 Processor -- start 2024-06-11T19:16:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 1 -- start start 2024-06-11T19:16:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 10 monclient: init 2024-06-11T19:16:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 10 monclient: _add_conns ranks=[7,9,4,8,5,2,1,6,0,10,3] 2024-06-11T19:16:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad340a4920 0x7fad340a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 10 monclient(hunting): picked mon.noname-i con 0x7fad340a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:16:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad340a5240 0x7fad340a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 10 monclient(hunting): picked mon.noname-a con 0x7fad340a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad340a2530 0x7fad340a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 10 monclient(hunting): picked mon.noname-c con 0x7fad340a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fad340a7140 con 0x7fad340a5240 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fad340a2b50 con 0x7fad340a2530 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fad340a2e30 con 0x7fad340a4920 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad3a95f640 10 monclient(hunting): authenticate will time out at 1174.088568s 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad340a5240 0x7fad340a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad38efc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad340a2530 0x7fad340a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad38efc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad340a2530 0x7fad340a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:33544/0 (socket says 172.21.15.98:33544) 2024-06-11T19:16:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad340a5240 0x7fad340a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32934/0 (socket says 172.21.15.98:32934) 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad38efc640 1 -- 172.21.15.98:0/3047735095 learned_addr learned my addr 172.21.15.98:0/3047735095 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad38efc640 10 monclient(hunting): get_auth_request con 0x7fad340a2530 auth_method 0 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad38efc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad38efc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad38efc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 10 monclient(hunting): get_auth_request con 0x7fad340a5240 auth_method 0 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad33fff640 1 --2- 172.21.15.98:0/3047735095 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad340a4920 0x7fad340a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad33fff640 10 monclient(hunting): get_auth_request con 0x7fad340a4920 auth_method 0 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad33fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad33fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad38efc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad38efc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad38efc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 10 monclient(hunting): handle_auth_done global_id 64872 payload 274 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 1 -- 172.21.15.98:0/3047735095 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad340a4920 msgr2=0x7fad340a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 1 --2- 172.21.15.98:0/3047735095 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad340a4920 0x7fad340a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.281+0000 7fad337fe640 1 -- 172.21.15.98:0/3047735095 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad340a2530 msgr2=0x7fad340a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad337fe640 1 --2- 172.21.15.98:0/3047735095 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad340a2530 0x7fad340a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad337fe640 10 monclient: _finish_hunting 0 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad337fe640 1 monclient: found mon.noname-a 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad337fe640 1 -- 172.21.15.98:0/3047735095 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad340a3110 con 0x7fad340a5240 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad38efc640 1 --2- 172.21.15.98:0/3047735095 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad340a2530 0x7fad340a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad337fe640 1 --2- 172.21.15.98:0/3047735095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad340a5240 0x7fad340a59a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fad280097a0 tx=0x7fad2802b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1073a4b6eeb7946d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 1 -- 172.21.15.98:0/3047735095 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad28037070 con 0x7fad340a5240 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 10 monclient: dump: 2024-06-11T19:16:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:04.296 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 10 monclient: _finish_auth 0 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad3a95f640 5 monclient: authenticate success, global_id 64872 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 1 -- 172.21.15.98:0/3047735095 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad28030980 con 0x7fad340a5240 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad3a95f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad3a95f640 10 monclient: get_monmap_and_config success 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 1 -- 172.21.15.98:0/3047735095 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad28030bb0 con 0x7fad340a5240 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad3a95f640 4 set_mon_vals no callback set 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.282+0000 7fad32ffd640 10 monclient: dump: 2024-06-11T19:16:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.283+0000 7fad3a95f640 10 monclient: shutdown 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.283+0000 7fad3a95f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.283+0000 7fad3a95f640 1 -- 172.21.15.98:0/3047735095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad340a5240 msgr2=0x7fad340a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.283+0000 7fad3a95f640 1 --2- 172.21.15.98:0/3047735095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad340a5240 0x7fad340a59a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fad280097a0 tx=0x7fad2802b6f0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.283+0000 7fad3a95f640 1 -- 172.21.15.98:0/3047735095 shutdown_connections 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.283+0000 7fad3a95f640 1 --2- 172.21.15.98:0/3047735095 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad340a2530 0x7fad340a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.283+0000 7fad3a95f640 1 --2- 172.21.15.98:0/3047735095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad340a5240 0x7fad340a59a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.283+0000 7fad3a95f640 1 --2- 172.21.15.98:0/3047735095 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad340a4920 0x7fad340a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.283+0000 7fad3a95f640 1 -- 172.21.15.98:0/3047735095 >> 172.21.15.98:0/3047735095 conn(0x7fad340ab850 msgr2=0x7fad340a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.283+0000 7fad3a95f640 1 -- 172.21.15.98:0/3047735095 shutdown_connections 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.283+0000 7fad3a95f640 1 -- 172.21.15.98:0/3047735095 wait complete. 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.284+0000 7fad3a95f640 10 monclient: build_initial_monmap 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.284+0000 7fad3a95f640 10 monclient: monmap: 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:16:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.284+0000 7fad3a95f640 1 librados: starting msgr at 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.284+0000 7fad3a95f640 1 librados: starting objecter 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.284+0000 7fad3a95f640 1 Processor -- start 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.284+0000 7fad3a95f640 1 -- start start 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.284+0000 7fad3a95f640 1 librados: setting wanted keys 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.284+0000 7fad3a95f640 1 librados: calling monclient init 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.284+0000 7fad3a95f640 10 monclient: init 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 10 monclient: _reopen_session rank -1 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 10 monclient: _add_conns ranks=[5,1,6,7,4,0,8,3,2] 2024-06-11T19:16:04.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fad340a2530 0x7fad34137670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 10 monclient(hunting): picked mon.noname-h con 0x7fad340a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad340a4920 0x7fad34137bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 10 monclient(hunting): picked mon.noname-f con 0x7fad340a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fad340a5240 0x7fad34123300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 10 monclient(hunting): picked mon.noname-d con 0x7fad340a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fad340a6da0 con 0x7fad340a4920 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fad34138e50 con 0x7fad340a2530 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fad340a6420 con 0x7fad340a5240 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 10 monclient(hunting): _renew_subs 2024-06-11T19:16:04.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 10 monclient(hunting): authenticate will time out at 1174.092161s 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fad340a5240 0x7fad34123300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fad340a5240 0x7fad34123300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:50452/0 (socket says 172.21.15.98:50452) 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 1 -- 172.21.15.98:0/190063845 learned_addr learned my addr 172.21.15.98:0/190063845 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 10 monclient(hunting): get_auth_request con 0x7fad340a5240 auth_method 0 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad33fff640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fad340a2530 0x7fad34137670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad33fff640 10 monclient(hunting): get_auth_request con 0x7fad340a2530 auth_method 0 2024-06-11T19:16:04.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad33fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad33fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad337fe640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad340a4920 0x7fad34137bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad337fe640 10 monclient(hunting): get_auth_request con 0x7fad340a4920 auth_method 0 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad337fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad337fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 10 monclient(hunting): handle_auth_done global_id 55023 payload 824 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 1 -- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fad340a2530 msgr2=0x7fad34137670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fad340a2530 0x7fad34137670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 1 -- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad340a4920 msgr2=0x7fad34137bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad340a4920 0x7fad34137bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 10 monclient: _finish_hunting 0 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 1 monclient: found mon.noname-d 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 1 -- 172.21.15.98:0/190063845 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad340a9180 con 0x7fad340a5240 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad38efc640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fad340a5240 0x7fad34123300 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00efc0 tx=0x7fad2c00c490 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 10 client.?.objecter ms_handle_connect 0x7fad340a5240 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad2c00cbe0 con 0x7fad340a5240 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 10 monclient: dump: 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:04.305 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 10 monclient: _finish_auth 0 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad2c00cd80 con 0x7fad340a5240 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 5 monclient: authenticate success, global_id 55023 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 10 monclient: _renew_subs 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad3a95f640 1 -- 172.21.15.98:0/190063845 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad34075480 con 0x7fad340a5240 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad2c014d60 con 0x7fad340a5240 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad30ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.286+0000 7fad30ff9640 10 monclient: dump: 2024-06-11T19:16:04.307 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:04.308 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.286+0000 7fad3a95f640 10 client.55023.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.286+0000 7fad3a95f640 10 monclient: _renew_subs 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.286+0000 7fad3a95f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.286+0000 7fad3a95f640 1 -- 172.21.15.98:0/190063845 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fad34127050 con 0x7fad340a5240 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.285+0000 7fad327fc640 4 set_mon_vals no callback set 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.286+0000 7fad3a95f640 1 librados: init done 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.286+0000 7fad3a95f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.286+0000 7fad3a95f640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34123840 0x7fad34123c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.286+0000 7fad3a95f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.286+0000 7fad3a95f640 1 -- 172.21.15.98:0/190063845 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fad341c0c40 con 0x7fad34123840 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.286+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fad2c01b070 con 0x7fad340a5240 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fad2003fdb0 0x7fad20042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad33fff640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34123840 0x7fad34123c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad33fff640 10 monclient: get_auth_request con 0x7fad34123840 auth_method 0 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(292..292 src has 1..292) ==== 7103+0+0 (secure 0 0 0) 0x7fad2c019020 con 0x7fad340a5240 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad33fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad33fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 10 client.55023.objecter ms_dispatch 0x7fad34000b90 osd_map(292..292 src has 1..292) 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 3 client.55023.objecter handle_osd_map got epochs [292,292] > 0 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 3 client.55023.objecter handle_osd_map decoding full epoch 292 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 20 client.55023.objecter dump_active .. 0 homeless 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad337fe640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fad2003fdb0 0x7fad20042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad337fe640 10 monclient: get_auth_request con 0x7fad2003fdb0 auth_method 0 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad33fff640 10 monclient(hunting): handle_auth_done global_id 84211 payload 824 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad33fff640 1 -- 172.21.15.98:0/190063845 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fad340a93b0 con 0x7fad34123840 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 10 client.55023.objecter ms_handle_connect 0x7fad34123840 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 10 client.55023.objecter resend_mon_ops 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad33fff640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34123840 0x7fad34123c20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fad24011860 tx=0x7fad24011d20 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 10 client.55023.objecter ms_handle_connect 0x7fad2003fdb0 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad24019070 con 0x7fad34123840 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad337fe640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fad2003fdb0 0x7fad20042270 secure :-1 s=READY pgs=4153 cs=0 l=1 rev1=1 crypto rx=0x7fad28034000 tx=0x7fad28008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.287+0000 7fad30ff9640 10 monclient: dump: 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.311 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.288+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fad24004240 con 0x7fad34123840 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.288+0000 7fad30ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:16:04.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.288+0000 7fad30ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:16:04.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.288+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34123840 msgr2=0x7fad34123c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.288+0000 7fad30ff9640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34123840 0x7fad34123c20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fad24011860 tx=0x7fad24011d20 comp rx=0 tx=0).stop 2024-06-11T19:16:04.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.288+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34123840 msgr2=0x7fad34123c20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:04.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.288+0000 7fad30ff9640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34123840 0x7fad34123c20 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fad24011860 tx=0x7fad24011d20 comp rx=0 tx=0).stop 2024-06-11T19:16:04.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.294+0000 7fad3a95f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:16:04.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.294+0000 7fad3a95f640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34124160 0x7fad3409e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.294+0000 7fad3a95f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.294+0000 7fad3a95f640 1 -- 172.21.15.98:0/190063845 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fad34050830 con 0x7fad34124160 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.294+0000 7fad33fff640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34124160 0x7fad3409e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.294+0000 7fad33fff640 10 monclient: get_auth_request con 0x7fad34124160 auth_method 0 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.294+0000 7fad33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.294+0000 7fad33fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.294+0000 7fad33fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad33fff640 10 monclient(hunting): handle_auth_done global_id 84220 payload 824 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad33fff640 1 -- 172.21.15.98:0/190063845 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fad34125610 con 0x7fad34124160 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad33fff640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34124160 0x7fad3409e9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fad2401b000 tx=0x7fad240046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:16:04.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 10 client.55023.objecter ms_handle_connect 0x7fad34124160 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 10 client.55023.objecter resend_mon_ops 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad24019040 con 0x7fad34124160 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 10 monclient: dump: 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:16:04.316 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:16:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:16:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:16:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:16:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:16:04.317 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:16:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9250 (secure 0 0 0) 0x7fad2401e450 con 0x7fad34124160 2024-06-11T19:16:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:16:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:16:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34124160 msgr2=0x7fad3409e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34124160 0x7fad3409e9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fad2401b000 tx=0x7fad240046c0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 1 -- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34124160 msgr2=0x7fad3409e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:16:04.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.295+0000 7fad30ff9640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fad34124160 0x7fad3409e9f0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fad2401b000 tx=0x7fad240046c0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.318 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:16:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:16:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:16:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:16:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 34, 2024-06-11T19:16:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 36065, 2024-06-11T19:16:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:16:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:16:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:16:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:16:04.318 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:04.319 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:04.320 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:16:04.321 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:16:04.322 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:04.323 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:16:04.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.326 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.328 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:16:04.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.331 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:16:04.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.338 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:16:04.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:16:04.340 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:16:04.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:16:04.340 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:16:04.340 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:16:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 10 librados: watch_flush enter 2024-06-11T19:16:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 10 librados: watch_flush exit 2024-06-11T19:16:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 20 client.55023.objecter shutdown clearing up homeless session... 2024-06-11T19:16:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 10 client.55023.objecter successfully canceled tick 2024-06-11T19:16:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 1 -- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fad2003fdb0 msgr2=0x7fad20042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fad2003fdb0 0x7fad20042270 secure :-1 s=READY pgs=4153 cs=0 l=1 rev1=1 crypto rx=0x7fad28034000 tx=0x7fad28008030 comp rx=0 tx=0).stop 2024-06-11T19:16:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 10 monclient: shutdown 2024-06-11T19:16:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:16:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 1 -- 172.21.15.98:0/190063845 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fad340a5240 msgr2=0x7fad34123300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:16:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad38efc640 1 -- 172.21.15.98:0/190063845 reap_dead start 2024-06-11T19:16:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fad340a5240 0x7fad34123300 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00efc0 tx=0x7fad2c00c490 comp rx=0 tx=0).stop 2024-06-11T19:16:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 1 -- 172.21.15.98:0/190063845 shutdown_connections 2024-06-11T19:16:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 1 --2- 172.21.15.98:0/190063845 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fad340a5240 0x7fad34123300 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:16:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 1 -- 172.21.15.98:0/190063845 >> 172.21.15.98:0/190063845 conn(0x7fad340ab850 msgr2=0x7fad341be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:16:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 1 -- 172.21.15.98:0/190063845 shutdown_connections 2024-06-11T19:16:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 1 -- 172.21.15.98:0/190063845 wait complete. 2024-06-11T19:16:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:16:04.296+0000 7fad3a95f640 1 librados: shutdown 2024-06-11T19:16:04.343 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-06-11T19:16:04.343 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-06-11T19:16:21.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ 2024-06-11T19:16:21.907 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3032 ms) 2024-06-11T19:16:21.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-06-11T19:16:21.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6981 ms) 2024-06-11T19:16:21.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-06-11T19:16:21.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (16701 ms) 2024-06-11T19:16:21.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-06-11T19:16:21.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6297 ms) 2024-06-11T19:16:21.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-06-11T19:16:21.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (21030 ms) 2024-06-11T19:16:21.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-06-11T19:16:21.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6971 ms) 2024-06-11T19:16:21.908 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-06-11T19:16:21.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7017 ms) 2024-06-11T19:16:21.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (179046 ms total) 2024-06-11T19:16:21.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: 2024-06-11T19:16:21.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-06-11T19:16:21.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (347838 ms total) 2024-06-11T19:16:21.909 INFO:tasks.workunit.client.0.smithi117.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-06-11T19:17:34.344 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-06-11T19:17:34.345 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-06-11T19:17:34.345 INFO:tasks.ceph.mon.e:Restarting daemon 2024-06-11T19:17:34.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-06-11T19:17:34.350 INFO:tasks.ceph.mon.e:Started 2024-06-11T19:17:34.350 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-06-11T19:17:34.350 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-06-11T19:17:34.350 INFO:tasks.ceph.mon.g:Restarting daemon 2024-06-11T19:17:34.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-06-11T19:17:34.353 INFO:tasks.ceph.mon.g:Started 2024-06-11T19:17:34.353 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:17:34.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:17:34.458 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-06-11T19:17:34.459+0000 7f7792255b00 -1 mon.e@-1(???).log v0 log_external_backlog rewinding external_log_to from 293 -> 0 (sync_force? mon rebuild?) 2024-06-11T19:17:34.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.465+0000 7fb67c23e640 10 monclient: get_monmap_and_config 2024-06-11T19:17:34.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.465+0000 7fb67c23e640 10 monclient: build_initial_monmap 2024-06-11T19:17:34.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.465+0000 7fb67c23e640 10 monclient: monmap: 2024-06-11T19:17:34.465 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:34.465 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:34.465 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:34.466828+0000 2024-06-11T19:17:34.465 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:34.466828+0000 2024-06-11T19:17:34.465 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:34.465 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:34.466 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:34.466 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:34.466 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:34.466 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:34.466 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:34.466 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:34.466 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:34.466 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:34.466 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:34.466 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:34.466 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:34.466 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:34.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.466+0000 7fb67c23e640 1 Processor -- start 2024-06-11T19:17:34.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.466+0000 7fb67c23e640 1 -- start start 2024-06-11T19:17:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.466+0000 7fb67c23e640 10 monclient: init 2024-06-11T19:17:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 10 monclient: _add_conns ranks=[9,3,5,1,10,0,6,4,8,2,7] 2024-06-11T19:17:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb67409d330 0x7fb67409d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 10 monclient(hunting): picked mon.noname-a con 0x7fb67409d330 addr v2:172.21.15.98:3300/0 2024-06-11T19:17:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409dc50 0x7fb674136770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 10 monclient(hunting): picked mon.noname-g con 0x7fb67409dc50 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb6740a2850 0x7fb674136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 10 monclient(hunting): picked mon.noname-h con 0x7fb6740a2850 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb67409bb20 con 0x7fb67409d330 2024-06-11T19:17:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fb67409b560 con 0x7fb67409dc50 2024-06-11T19:17:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fb67409b840 con 0x7fb6740a2850 2024-06-11T19:17:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67c23e640 10 monclient(hunting): authenticate will time out at 1264.274140s 2024-06-11T19:17:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb67409d330 0x7fb67409d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb67409d330 0x7fb67409d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46974/0 (socket says 172.21.15.98:46974) 2024-06-11T19:17:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb6797d9640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409dc50 0x7fb674136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 1 -- 172.21.15.98:0/2949728527 learned_addr learned my addr 172.21.15.98:0/2949728527 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67a7db640 1 --2- 172.21.15.98:0/2949728527 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb6740a2850 0x7fb674136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 10 monclient(hunting): get_auth_request con 0x7fb67409d330 auth_method 0 2024-06-11T19:17:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb6797d9640 10 monclient(hunting): get_auth_request con 0x7fb67409dc50 auth_method 0 2024-06-11T19:17:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb6797d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb6797d9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb6797d9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67a7db640 10 monclient(hunting): get_auth_request con 0x7fb6740a2850 auth_method 0 2024-06-11T19:17:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67a7db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67a7db640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67a7db640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb6797d9640 1 -- 172.21.15.98:0/2949728527 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409dc50 msgr2=0x7fb674136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb6797d9640 1 -- 172.21.15.98:0/2949728527 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409dc50 msgr2=0x7fb674136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb6797d9640 1 --2- 172.21.15.98:0/2949728527 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409dc50 0x7fb674136770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb6797d9640 1 --2- 172.21.15.98:0/2949728527 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409dc50 0x7fb674136770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67a7db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67a7db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67a7db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 10 monclient(hunting): handle_auth_done global_id 64890 payload 274 2024-06-11T19:17:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 1 -- 172.21.15.98:0/2949728527 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb6740a2850 msgr2=0x7fb674136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 1 --2- 172.21.15.98:0/2949728527 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb6740a2850 0x7fb674136cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 1 -- 172.21.15.98:0/2949728527 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409dc50 msgr2=0x7fb674136770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 1 --2- 172.21.15.98:0/2949728527 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409dc50 0x7fb674136770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 10 monclient: _finish_hunting 0 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 1 monclient: found mon.noname-a 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb679fda640 1 -- 172.21.15.98:0/2949728527 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6740769a0 con 0x7fb67409d330 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.467+0000 7fb67a7db640 1 --2- 172.21.15.98:0/2949728527 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb6740a2850 0x7fb674136cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb679fda640 1 --2- 172.21.15.98:0/2949728527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb67409d330 0x7fb67409d710 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb670009870 tx=0x7fb67002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4533819d5c381bf6 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 1 -- 172.21.15.98:0/2949728527 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb670038070 con 0x7fb67409d330 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:34.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 10 monclient: dump: 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 10 monclient: _finish_auth 0 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 10 monclient: _check_auth_tickets 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 1 -- 172.21.15.98:0/2949728527 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb670031990 con 0x7fb67409d330 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb67c23e640 5 monclient: authenticate success, global_id 64890 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb67c23e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb67c23e640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 1 -- 172.21.15.98:0/2949728527 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb6700323b0 con 0x7fb67409d330 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb67c23e640 4 set_mon_vals no callback set 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.468+0000 7fb678fd8640 10 monclient: dump: 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 10 monclient: shutdown 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 1 -- 172.21.15.98:0/2949728527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb67409d330 msgr2=0x7fb67409d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 1 --2- 172.21.15.98:0/2949728527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb67409d330 0x7fb67409d710 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb670009870 tx=0x7fb67002b7c0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 1 -- 172.21.15.98:0/2949728527 shutdown_connections 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 1 --2- 172.21.15.98:0/2949728527 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb6740a2850 0x7fb674136cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 1 --2- 172.21.15.98:0/2949728527 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409dc50 0x7fb674136770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 1 --2- 172.21.15.98:0/2949728527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb67409d330 0x7fb67409d710 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 1 -- 172.21.15.98:0/2949728527 >> 172.21.15.98:0/2949728527 conn(0x7fb6740a7850 msgr2=0x7fb6740a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 1 -- 172.21.15.98:0/2949728527 shutdown_connections 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 1 -- 172.21.15.98:0/2949728527 wait complete. 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 10 monclient: build_initial_monmap 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.469+0000 7fb67c23e640 10 monclient: monmap: 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:34.485 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 1 librados: starting msgr at 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 1 librados: starting objecter 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 1 Processor -- start 2024-06-11T19:17:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 1 -- start start 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 1 librados: setting wanted keys 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 1 librados: calling monclient init 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 10 monclient: init 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 10 monclient: _add_conns ranks=[3,6,1,7,4,8,0,2,5] 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409d330 0x7fb6741cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 10 monclient(hunting): picked mon.noname-g con 0x7fb67409d330 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb6740a2850 0x7fb67409a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 10 monclient(hunting): picked mon.noname-d con 0x7fb6740a2850 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67409ab10 0x7fb674134b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 10 monclient(hunting): picked mon.noname-f con 0x7fb67409ab10 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb6741371f0 con 0x7fb67409ab10 2024-06-11T19:17:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fb67409e270 con 0x7fb67409d330 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb674137370 con 0x7fb6740a2850 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb67c23e640 10 monclient(hunting): authenticate will time out at 1264.277835s 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb6797d9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb6740a2850 0x7fb67409a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb6797d9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb6740a2850 0x7fb67409a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:43630/0 (socket says 172.21.15.98:43630) 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb6797d9640 1 -- 172.21.15.98:0/3693794767 learned_addr learned my addr 172.21.15.98:0/3693794767 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb6797d9640 10 monclient(hunting): get_auth_request con 0x7fb6740a2850 auth_method 0 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb6797d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb6797d9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.470+0000 7fb6797d9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb679fda640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409d330 0x7fb6741cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67a7db640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67409ab10 0x7fb674134b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67a7db640 10 monclient(hunting): get_auth_request con 0x7fb67409ab10 auth_method 0 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67a7db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67a7db640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67a7db640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb679fda640 10 monclient(hunting): get_auth_request con 0x7fb67409d330 auth_method 0 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb679fda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb679fda640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb679fda640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67a7db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67a7db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67a7db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb679fda640 1 -- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409d330 msgr2=0x7fb6741cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 10 monclient(hunting): handle_auth_done global_id 55041 payload 824 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb679fda640 1 -- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409d330 msgr2=0x7fb6741cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb679fda640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409d330 0x7fb6741cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb679fda640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409d330 0x7fb6741cda00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 1 -- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409d330 msgr2=0x7fb6741cda00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409d330 0x7fb6741cda00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 1 -- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67409ab10 msgr2=0x7fb674134b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67409ab10 0x7fb674134b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 10 monclient: _finish_hunting 0 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 1 monclient: found mon.noname-d 2024-06-11T19:17:34.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 1 -- 172.21.15.98:0/3693794767 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67408e010 con 0x7fb6740a2850 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb6797d9640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb6740a2850 0x7fb67409a5d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb66400d5c0 tx=0x7fb66400d9c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 10 client.?.objecter ms_handle_connect 0x7fb6740a2850 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 1 -- 172.21.15.98:0/3693794767 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb664005ba0 con 0x7fb6740a2850 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 10 monclient: dump: 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:34.491 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:17:34.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 1 -- 172.21.15.98:0/3693794767 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb664018070 con 0x7fb6740a2850 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67c23e640 5 monclient: authenticate success, global_id 55041 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67c23e640 10 monclient: _renew_subs 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67c23e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67c23e640 1 -- 172.21.15.98:0/3693794767 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67409f9a0 con 0x7fb6740a2850 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 1 -- 172.21.15.98:0/3693794767 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb664005de0 con 0x7fb6740a2850 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66a7fc640 10 monclient: dump: 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:34.493 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:34.494 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67c23e640 10 client.55041.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb66bfff640 4 set_mon_vals no callback set 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67c23e640 10 monclient: _renew_subs 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67c23e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67c23e640 1 -- 172.21.15.98:0/3693794767 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6740a0590 con 0x7fb6740a2850 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.471+0000 7fb67c23e640 1 librados: init done 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.472+0000 7fb66a7fc640 1 -- 172.21.15.98:0/3693794767 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fb664006050 con 0x7fb6740a2850 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.472+0000 7fb67c23e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.472+0000 7fb67c23e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.472+0000 7fb67c23e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.472+0000 7fb67c23e640 1 -- 172.21.15.98:0/3693794767 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb67409d710 con 0x7fb6740a2850 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.473+0000 7fb66a7fc640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb65803fdb0 0x7fb658042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.473+0000 7fb66a7fc640 1 -- 172.21.15.98:0/3693794767 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(343..343 src has 1..343) ==== 6230+0+0 (secure 0 0 0) 0x7fb66400f040 con 0x7fb6740a2850 2024-06-11T19:17:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.473+0000 7fb66a7fc640 10 client.55041.objecter ms_dispatch 0x7fb674000b90 osd_map(343..343 src has 1..343) 2024-06-11T19:17:34.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.473+0000 7fb66a7fc640 3 client.55041.objecter handle_osd_map got epochs [343,343] > 0 2024-06-11T19:17:34.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.473+0000 7fb66a7fc640 3 client.55041.objecter handle_osd_map decoding full epoch 343 2024-06-11T19:17:34.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.473+0000 7fb66a7fc640 20 client.55041.objecter dump_active .. 0 homeless 2024-06-11T19:17:34.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.473+0000 7fb679fda640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb65803fdb0 0x7fb658042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:34.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.473+0000 7fb679fda640 10 monclient: get_auth_request con 0x7fb65803fdb0 auth_method 0 2024-06-11T19:17:34.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.473+0000 7fb66a7fc640 10 client.55041.objecter ms_handle_connect 0x7fb65803fdb0 2024-06-11T19:17:34.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.473+0000 7fb679fda640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb65803fdb0 0x7fb658042270 secure :-1 s=READY pgs=4157 cs=0 l=1 rev1=1 crypto rx=0x7fb670035000 tx=0x7fb670030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:34.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.477+0000 7fb66a7fc640 1 -- 172.21.15.98:0/3693794767 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb664013800 con 0x7fb6740a2850 2024-06-11T19:17:34.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.477+0000 7fb66a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:34.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.477+0000 7fb66a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:34.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.744+0000 7fb67c23e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:17:34.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.744+0000 7fb67c23e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:17:34.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.744+0000 7fb67c23e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:34.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.744+0000 7fb67c23e640 1 -- 172.21.15.98:0/3693794767 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb67409eed0 con 0x7fb6740a2850 2024-06-11T19:17:34.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.744+0000 7fb66a7fc640 1 -- 172.21.15.98:0/3693794767 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3181 (secure 0 0 0) 0x7fb66400f300 con 0x7fb6740a2850 2024-06-11T19:17:34.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.744+0000 7fb66a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:17:34.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.744+0000 7fb66a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:34.744 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":34,"quorum":[0,1,2,4,5,6,7],"quorum_names":["a","f","b","c","h","d","i"],"quorum_leader_name":"f","quorum_age":91,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:17:34.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 10 librados: watch_flush enter 2024-06-11T19:17:34.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 10 librados: watch_flush exit 2024-06-11T19:17:34.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 20 client.55041.objecter shutdown clearing up homeless session... 2024-06-11T19:17:34.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 10 client.55041.objecter successfully canceled tick 2024-06-11T19:17:34.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 -- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb65803fdb0 msgr2=0x7fb658042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:34.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb65803fdb0 0x7fb658042270 secure :-1 s=READY pgs=4157 cs=0 l=1 rev1=1 crypto rx=0x7fb670035000 tx=0x7fb670030000 comp rx=0 tx=0).stop 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 10 monclient: shutdown 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 -- 172.21.15.98:0/3693794767 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb6740a2850 msgr2=0x7fb67409a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb6740a2850 0x7fb67409a5d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb66400d5c0 tx=0x7fb66400d9c0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 -- 172.21.15.98:0/3693794767 shutdown_connections 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb65803fdb0 0x7fb658042270 unknown :-1 s=CLOSED pgs=4157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67409ab10 0x7fb674134b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb6740a2850 0x7fb67409a5d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 --2- 172.21.15.98:0/3693794767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb67409d330 0x7fb6741cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 -- 172.21.15.98:0/3693794767 >> 172.21.15.98:0/3693794767 conn(0x7fb6740a7850 msgr2=0x7fb6740a5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 -- 172.21.15.98:0/3693794767 shutdown_connections 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 -- 172.21.15.98:0/3693794767 wait complete. 2024-06-11T19:17:34.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:34.748+0000 7fb67c23e640 1 librados: shutdown 2024-06-11T19:17:37.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:17:37.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.870+0000 7f88306ca640 10 monclient: get_monmap_and_config 2024-06-11T19:17:37.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.870+0000 7f88306ca640 10 monclient: build_initial_monmap 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.870+0000 7f88306ca640 10 monclient: monmap: 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:37.871510+0000 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:37.871510+0000 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:37.871 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:37.871 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:37.871 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:37.871 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:37.871 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:37.871 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 1 Processor -- start 2024-06-11T19:17:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 1 -- start start 2024-06-11T19:17:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 10 monclient: init 2024-06-11T19:17:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 10 monclient: _add_conns ranks=[10,4,3,7,1,6,8,2,5,9,0] 2024-06-11T19:17:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 10 monclient(hunting): picked mon.noname-f con 0x7f88280a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:17:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 10 monclient(hunting): picked mon.noname-c con 0x7f88280a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 10 monclient(hunting): picked mon.noname-g con 0x7f88280a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f88280a7140 con 0x7f88280a4920 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f88280a2b50 con 0x7f88280a2530 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f88280a2e30 con 0x7f88280a5240 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f88306ca640 10 monclient(hunting): authenticate will time out at 1267.678877s 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.871+0000 7f882dc65640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882dc65640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:52808/0 (socket says 172.21.15.98:52808) 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882dc65640 1 -- 172.21.15.98:0/2511063414 learned_addr learned my addr 172.21.15.98:0/2511063414 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882dc65640 10 monclient(hunting): get_auth_request con 0x7f88280a5240 auth_method 0 2024-06-11T19:17:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882dc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882dc65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882dc65640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882dc65640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 msgr2=0x7f88280a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882dc65640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 msgr2=0x7f88280a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882ec67640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882ec67640 10 monclient(hunting): get_auth_request con 0x7f88280a2530 auth_method 0 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882ec67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882ec67640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882ec67640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882e466640 10 monclient(hunting): get_auth_request con 0x7f88280a4920 auth_method 0 2024-06-11T19:17:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882e466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882e466640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882e466640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882ec67640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 msgr2=0x7f88280a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:17:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882ec67640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 msgr2=0x7f88280a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882ec67640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882ec67640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:17:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882e466640 1 -- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 msgr2=0x7f88280a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:17:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882e466640 1 -- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 msgr2=0x7f88280a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:37.872+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:17:38.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:38.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882dc65640 10 monclient(hunting): get_auth_request con 0x7f88280a5240 auth_method 0 2024-06-11T19:17:38.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882dc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:38.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882dc65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:38.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882dc65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882dc65640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 msgr2=0x7f88280a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882dc65640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 msgr2=0x7f88280a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882ec67640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882e466640 10 monclient(hunting): get_auth_request con 0x7f88280a4920 auth_method 0 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882e466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882e466640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882e466640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882ec67640 10 monclient(hunting): get_auth_request con 0x7f88280a2530 auth_method 0 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882ec67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882ec67640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:38.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882ec67640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:17:38.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882ec67640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 msgr2=0x7f88280a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:17:38.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882e466640 1 -- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 msgr2=0x7f88280a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:17:38.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882ec67640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 msgr2=0x7f88280a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:38.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882e466640 1 -- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 msgr2=0x7f88280a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:38.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:38.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882ec67640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:38.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882ec67640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:17:38.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.073+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:17:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882dc65640 10 monclient(hunting): get_auth_request con 0x7f88280a5240 auth_method 0 2024-06-11T19:17:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882dc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882dc65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882dc65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:17:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882dc65640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 msgr2=0x7f88280a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:17:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882dc65640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 msgr2=0x7f88280a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882ec67640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882ec67640 10 monclient(hunting): get_auth_request con 0x7f88280a2530 auth_method 0 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882ec67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882ec67640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882ec67640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882e466640 10 monclient(hunting): get_auth_request con 0x7f88280a4920 auth_method 0 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882e466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882e466640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882e466640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882ec67640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 msgr2=0x7f88280a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:17:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882ec67640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 msgr2=0x7f88280a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882ec67640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882ec67640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:17:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882e466640 1 -- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 msgr2=0x7f88280a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:17:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882e466640 1 -- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 msgr2=0x7f88280a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:38.474+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:17:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882dc65640 10 monclient(hunting): get_auth_request con 0x7f88280a5240 auth_method 0 2024-06-11T19:17:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882dc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882dc65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882dc65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:17:39.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882dc65640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 msgr2=0x7f88280a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882dc65640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 msgr2=0x7f88280a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882ec67640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882ec67640 10 monclient(hunting): get_auth_request con 0x7f88280a2530 auth_method 0 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882ec67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882ec67640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882ec67640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882e466640 10 monclient(hunting): get_auth_request con 0x7f88280a4920 auth_method 0 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882e466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882e466640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.275+0000 7f882e466640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.276+0000 7f882ec67640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 msgr2=0x7f88280a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:17:39.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.276+0000 7f882ec67640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 msgr2=0x7f88280a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.276+0000 7f882ec67640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.276+0000 7f882ec67640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:17:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.276+0000 7f882e466640 1 -- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 msgr2=0x7f88280a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:17:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.276+0000 7f882e466640 1 -- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 msgr2=0x7f88280a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.276+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:39.276+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:17:40.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.871+0000 7f881ffff640 10 monclient(hunting): tick 2024-06-11T19:17:40.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.871+0000 7f881ffff640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:17:40.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.871+0000 7f881ffff640 1 monclient(hunting): continuing hunt 2024-06-11T19:17:40.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.871+0000 7f881ffff640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 msgr2=0x7f88280a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88280a5240 0x7f88280a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 msgr2=0x7f88280a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88280a2530 0x7f88280a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 1 -- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 msgr2=0x7f88280a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 1 --2- 172.21.15.98:0/2511063414 >> v2:172.21.15.117:3300/0 conn(0x7f88280a4920 0x7f88280a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 10 monclient: _add_conns ranks=[5,8,1,7,9,0,6,2] 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88140037d0 0x7f8814005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 10 monclient(hunting): picked mon.noname-h con 0x7f88140037d0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8814006200 0x7f8814008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 10 monclient(hunting): picked mon.noname-e con 0x7f8814006200 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f8814008d60 addr v1:172.21.15.117:6789/0 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 1 -- 172.21.15.98:0/2511063414 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88280a3110 con 0x7f8814008d60 2024-06-11T19:17:40.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 1 -- 172.21.15.98:0/2511063414 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f88280a7140 con 0x7f88140037d0 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 1 -- 172.21.15.98:0/2511063414 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f88280a2b50 con 0x7f8814006200 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f881ffff640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8814006200 0x7f8814008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882dc65640 10 monclient(hunting): get_auth_request con 0x7f8814006200 auth_method 0 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882dc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882dc65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882dc65640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88140037d0 0x7f8814005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 10 monclient(hunting): get_auth_request con 0x7f88140037d0 auth_method 0 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:40.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882dc65640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8814006200 msgr2=0x7f8814008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882dc65640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8814006200 msgr2=0x7f8814008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8814006200 0x7f8814008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882dc65640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8814006200 0x7f8814008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 10 monclient(hunting): handle_auth_done global_id 22973 payload 274 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882d464640 1 -- 172.21.15.98:0/2511063414 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3973039185 0 0) 0x7f88280a3110 con 0x7f8814008d60 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8814006200 msgr2=0x7f8814008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8814006200 0x7f8814008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 1 -- 172.21.15.98:0/2511063414 >> v1:172.21.15.117:6789/0 conn(0x7f8814008d60 legacy=0x7f881400b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:40.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 10 monclient: _finish_hunting 0 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 1 monclient: found mon.noname-h 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882ec67640 1 -- 172.21.15.98:0/2511063414 reap_dead start 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882e466640 1 -- 172.21.15.98:0/2511063414 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881400dc20 con 0x7f88140037d0 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.872+0000 7f882d464640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.873+0000 7f882e466640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88140037d0 0x7f8814005c00 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f8824006fd0 tx=0x7f882402c510 comp rx=0 tx=0).ready entity=mon.5 client_cookie=a7485473de223caa server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.873+0000 7f882d464640 1 -- 172.21.15.98:0/2511063414 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f882403a070 con 0x7f88140037d0 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.873+0000 7f882d464640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f882d464640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f882d464640 10 monclient: dump: 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:40.879 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:40.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f882d464640 10 monclient: _finish_auth 0 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f882d464640 10 monclient: _check_auth_tickets 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f882d464640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f882d464640 1 -- 172.21.15.98:0/2511063414 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88240347e0 con 0x7f88140037d0 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f88306ca640 5 monclient: authenticate success, global_id 22973 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f88306ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f882d464640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f882d464640 1 -- 172.21.15.98:0/2511063414 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8824034e00 con 0x7f88140037d0 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f882d464640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f882d464640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f882d464640 10 monclient: dump: 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:40.881 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:40.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f88306ca640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f88306ca640 4 set_mon_vals no callback set 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f88306ca640 10 monclient: shutdown 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f88306ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f88306ca640 1 -- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88140037d0 msgr2=0x7f8814005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f88306ca640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88140037d0 0x7f8814005c00 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f8824006fd0 tx=0x7f882402c510 comp rx=0 tx=0).stop 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f88306ca640 1 -- 172.21.15.98:0/2511063414 shutdown_connections 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.874+0000 7f88306ca640 1 --2- 172.21.15.98:0/2511063414 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f88140037d0 0x7f8814005c00 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.875+0000 7f88306ca640 1 -- 172.21.15.98:0/2511063414 >> 172.21.15.98:0/2511063414 conn(0x7f88280ab850 msgr2=0x7f88280a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.875+0000 7f88306ca640 1 -- 172.21.15.98:0/2511063414 shutdown_connections 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.875+0000 7f88306ca640 1 -- 172.21.15.98:0/2511063414 wait complete. 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.875+0000 7f88306ca640 10 monclient: build_initial_monmap 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.875+0000 7f88306ca640 10 monclient: monmap: 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:40.883 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 1 librados: starting msgr at 2024-06-11T19:17:40.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 1 librados: starting objecter 2024-06-11T19:17:40.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 1 Processor -- start 2024-06-11T19:17:40.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 1 -- start start 2024-06-11T19:17:40.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 1 librados: setting wanted keys 2024-06-11T19:17:40.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 1 librados: calling monclient init 2024-06-11T19:17:40.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 10 monclient: init 2024-06-11T19:17:40.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:40.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 10 monclient: _add_conns ranks=[1,2,0,3,8,5,7,4,6] 2024-06-11T19:17:40.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88140037d0 0x7f88281cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:40.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 10 monclient(hunting): picked mon.noname-f con 0x7f88140037d0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:17:40.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88281ce010 0x7f88281240a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:40.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 10 monclient(hunting): picked mon.noname-b con 0x7f88281ce010 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88281ce3f0 0x7f88281245e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 10 monclient(hunting): picked mon.noname-a con 0x7f88281ce3f0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f88280a6da0 con 0x7f88281ce3f0 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8828138e50 con 0x7f88140037d0 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f88280a4d00 con 0x7f88281ce010 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.876+0000 7f88306ca640 10 monclient(hunting): authenticate will time out at 1270.683878s 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882ec67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88281ce3f0 0x7f88281245e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882ec67640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88281ce3f0 0x7f88281245e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47002/0 (socket says 172.21.15.98:47002) 2024-06-11T19:17:40.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882ec67640 1 -- 172.21.15.98:0/926804314 learned_addr learned my addr 172.21.15.98:0/926804314 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882ec67640 10 monclient(hunting): get_auth_request con 0x7f88281ce3f0 auth_method 0 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882dc65640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88281ce010 0x7f88281240a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882ec67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882ec67640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882ec67640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88140037d0 0x7f88281cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882dc65640 10 monclient(hunting): get_auth_request con 0x7f88281ce010 auth_method 0 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882dc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882dc65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882dc65640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 10 monclient(hunting): get_auth_request con 0x7f88140037d0 auth_method 0 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:40.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882ec67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882ec67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882ec67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882dc65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882dc65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882dc65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 10 monclient(hunting): handle_auth_done global_id 32050 payload 824 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 1 -- 172.21.15.98:0/926804314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88281ce010 msgr2=0x7f88281240a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88281ce010 0x7f88281240a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 1 -- 172.21.15.98:0/926804314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88281ce3f0 msgr2=0x7f88281245e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:40.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88281ce3f0 0x7f88281245e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 10 monclient: _finish_hunting 0 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 1 monclient: found mon.noname-f 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 1 -- 172.21.15.98:0/926804314 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8828127cc0 con 0x7f88140037d0 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882dc65640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88281ce010 0x7f88281240a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882ec67640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88281ce3f0 0x7f88281245e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f882e466640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88140037d0 0x7f88281cdad0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f88140037a0 tx=0x7f88240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f881effd640 10 client.?.objecter ms_handle_connect 0x7f88140037d0 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f881effd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.877+0000 7f881effd640 1 -- 172.21.15.98:0/926804314 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8824044020 con 0x7f88140037d0 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f881effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f881effd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:17:40.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f881effd640 10 monclient: dump: 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:40.894 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f881effd640 10 monclient: _finish_auth 0 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f881effd640 10 monclient: _check_auth_tickets 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f881effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f881effd640 1 -- 172.21.15.98:0/926804314 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8824034e80 con 0x7f88140037d0 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f88306ca640 5 monclient: authenticate success, global_id 32050 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f88306ca640 10 monclient: _renew_subs 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f88306ca640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f88306ca640 1 -- 172.21.15.98:0/926804314 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88281267a0 con 0x7f88140037d0 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f881effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f881effd640 1 -- 172.21.15.98:0/926804314 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f882403a040 con 0x7f88140037d0 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f881effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:40.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f881effd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f881effd640 10 monclient: dump: 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:40.896 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f882cc63640 4 set_mon_vals no callback set 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f88306ca640 10 client.32050.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f88306ca640 10 monclient: _renew_subs 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f88306ca640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f88306ca640 1 -- 172.21.15.98:0/926804314 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88280a43a0 con 0x7f88140037d0 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.878+0000 7f88306ca640 1 librados: init done 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f881effd640 1 -- 172.21.15.98:0/926804314 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8824047020 con 0x7f88140037d0 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f88306ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f88306ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f88306ca640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:17:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f88306ca640 1 -- 172.21.15.98:0/926804314 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88280aac90 con 0x7f88140037d0 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f881effd640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8814040d20 0x7f88140431e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f881effd640 1 -- 172.21.15.98:0/926804314 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(343..343 src has 1..343) ==== 6230+0+0 (secure 0 0 0) 0x7f8824032070 con 0x7f88140037d0 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f881effd640 10 client.32050.objecter ms_dispatch 0x7f8828000b90 osd_map(343..343 src has 1..343) 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f881effd640 3 client.32050.objecter handle_osd_map got epochs [343,343] > 0 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f881effd640 3 client.32050.objecter handle_osd_map decoding full epoch 343 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f881effd640 20 client.32050.objecter dump_active .. 0 homeless 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f882dc65640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8814040d20 0x7f88140431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.879+0000 7f882dc65640 10 monclient: get_auth_request con 0x7f8814040d20 auth_method 0 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.880+0000 7f881effd640 10 client.32050.objecter ms_handle_connect 0x7f8814040d20 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.880+0000 7f882dc65640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8814040d20 0x7f88140431e0 secure :-1 s=READY pgs=4159 cs=0 l=1 rev1=1 crypto rx=0x7f88280a5620 tx=0x7f8818008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.884+0000 7f881effd640 1 -- 172.21.15.98:0/926804314 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f882404e030 con 0x7f88140037d0 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.884+0000 7f881effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:40.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:40.884+0000 7f881effd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.137+0000 7f88306ca640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:17:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.137+0000 7f88306ca640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:17:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.137+0000 7f88306ca640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:17:41.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.137+0000 7f88306ca640 1 -- 172.21.15.98:0/926804314 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f88280a4fa0 con 0x7f88140037d0 2024-06-11T19:17:41.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.145+0000 7f881effd640 1 -- 172.21.15.98:0/926804314 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7f882403ecc0 con 0x7f88140037d0 2024-06-11T19:17:41.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.145+0000 7f881effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:17:41.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.145+0000 7f881effd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:41.145 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":36,"quorum":[0,1,2,3,4,5,6,7],"quorum_names":["a","f","b","g","c","h","d","i"],"quorum_leader_name":"f","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:17:41.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 10 librados: watch_flush enter 2024-06-11T19:17:41.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 10 librados: watch_flush exit 2024-06-11T19:17:41.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 20 client.32050.objecter shutdown clearing up homeless session... 2024-06-11T19:17:41.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 10 client.32050.objecter successfully canceled tick 2024-06-11T19:17:41.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 -- 172.21.15.98:0/926804314 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8814040d20 msgr2=0x7f88140431e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:41.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8814040d20 0x7f88140431e0 secure :-1 s=READY pgs=4159 cs=0 l=1 rev1=1 crypto rx=0x7f88280a5620 tx=0x7f8818008040 comp rx=0 tx=0).stop 2024-06-11T19:17:41.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 10 monclient: shutdown 2024-06-11T19:17:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 -- 172.21.15.98:0/926804314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88140037d0 msgr2=0x7f88281cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88140037d0 0x7f88281cdad0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f88140037a0 tx=0x7f88240046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 -- 172.21.15.98:0/926804314 shutdown_connections 2024-06-11T19:17:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8814040d20 0x7f88140431e0 unknown :-1 s=CLOSED pgs=4159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88281ce3f0 0x7f88281245e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88281ce010 0x7f88281240a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 --2- 172.21.15.98:0/926804314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88140037d0 0x7f88281cdad0 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 -- 172.21.15.98:0/926804314 >> 172.21.15.98:0/926804314 conn(0x7f88280ab850 msgr2=0x7f88281380d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 -- 172.21.15.98:0/926804314 shutdown_connections 2024-06-11T19:17:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 -- 172.21.15.98:0/926804314 wait complete. 2024-06-11T19:17:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:41.147+0000 7f88306ca640 1 librados: shutdown 2024-06-11T19:17:44.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:17:44.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.267+0000 7f7dee306640 10 monclient: get_monmap_and_config 2024-06-11T19:17:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.267+0000 7f7dee306640 10 monclient: build_initial_monmap 2024-06-11T19:17:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.267+0000 7f7dee306640 10 monclient: monmap: 2024-06-11T19:17:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:44.268761+0000 2024-06-11T19:17:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:44.268761+0000 2024-06-11T19:17:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:44.268 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:44.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.268+0000 7f7dee306640 1 Processor -- start 2024-06-11T19:17:44.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.268+0000 7f7dee306640 1 -- start start 2024-06-11T19:17:44.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.268+0000 7f7dee306640 10 monclient: init 2024-06-11T19:17:44.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.268+0000 7f7dee306640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:44.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.268+0000 7f7dee306640 10 monclient: _add_conns ranks=[10,8,5,9,0,2,4,3,7,1,6] 2024-06-11T19:17:44.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.268+0000 7f7dee306640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7de80a4920 0x7f7de80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:44.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 10 monclient(hunting): picked mon.noname-f con 0x7f7de80a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:17:44.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a5240 0x7f7de80a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:44.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 10 monclient(hunting): picked mon.noname-e con 0x7f7de80a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:17:44.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de80a2530 0x7f7de80a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:44.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 10 monclient(hunting): picked mon.noname-h con 0x7f7de80a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:44.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f7de80a7140 con 0x7f7de80a4920 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f7de80a2b50 con 0x7f7de80a2530 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f7de80a2e30 con 0x7f7de80a5240 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dee306640 10 monclient(hunting): authenticate will time out at 1274.076079s 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de77fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a5240 0x7f7de80a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de77fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a5240 0x7f7de80a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:38992/0 (socket says 172.21.15.98:38992) 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de77fe640 1 -- 172.21.15.98:0/631989306 learned_addr learned my addr 172.21.15.98:0/631989306 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de77fe640 10 monclient(hunting): get_auth_request con 0x7f7de80a5240 auth_method 0 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de77fe640 1 -- 172.21.15.98:0/631989306 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a5240 msgr2=0x7f7de80a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de77fe640 1 -- 172.21.15.98:0/631989306 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a5240 msgr2=0x7f7de80a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de77fe640 1 --2- 172.21.15.98:0/631989306 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a5240 0x7f7de80a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de77fe640 1 --2- 172.21.15.98:0/631989306 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a5240 0x7f7de80a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de7fff640 1 --2- 172.21.15.98:0/631989306 >> v2:172.21.15.117:3300/0 conn(0x7f7de80a4920 0x7f7de80a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 1 --2- 172.21.15.98:0/631989306 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de80a2530 0x7f7de80a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 10 monclient(hunting): get_auth_request con 0x7f7de80a2530 auth_method 0 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de7fff640 10 monclient(hunting): get_auth_request con 0x7f7de80a4920 auth_method 0 2024-06-11T19:17:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 10 monclient(hunting): handle_auth_done global_id 22982 payload 274 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 1 -- 172.21.15.98:0/631989306 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a5240 msgr2=0x7f7de80a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 1 --2- 172.21.15.98:0/631989306 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a5240 0x7f7de80a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 1 -- 172.21.15.98:0/631989306 >> v2:172.21.15.117:3300/0 conn(0x7f7de80a4920 msgr2=0x7f7de80a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 1 --2- 172.21.15.98:0/631989306 >> v2:172.21.15.117:3300/0 conn(0x7f7de80a4920 0x7f7de80a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 10 monclient: _finish_hunting 0 2024-06-11T19:17:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 1 monclient: found mon.noname-h 2024-06-11T19:17:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:17:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7dec8a3640 1 -- 172.21.15.98:0/631989306 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de80a3110 con 0x7f7de80a2530 2024-06-11T19:17:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.269+0000 7f7de7fff640 1 --2- 172.21.15.98:0/631989306 >> v2:172.21.15.117:3300/0 conn(0x7f7de80a4920 0x7f7de80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7dec8a3640 1 --2- 172.21.15.98:0/631989306 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de80a2530 0x7f7de80a5ee0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009d90 tx=0x7f7de002bce0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=b551b70aba4c515c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 1 -- 172.21.15.98:0/631989306 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7de0039070 con 0x7f7de80a2530 2024-06-11T19:17:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:17:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 10 monclient: dump: 2024-06-11T19:17:44.282 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 10 monclient: _finish_auth 0 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 1 -- 172.21.15.98:0/631989306 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7de002cd00 con 0x7f7de80a2530 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7dee306640 5 monclient: authenticate success, global_id 22982 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7dee306640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7dee306640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 1 -- 172.21.15.98:0/631989306 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7de00327b0 con 0x7f7de80a2530 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7dee306640 4 set_mon_vals no callback set 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.270+0000 7f7de6ffd640 10 monclient: dump: 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 10 monclient: shutdown 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 1 -- 172.21.15.98:0/631989306 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de80a2530 msgr2=0x7f7de80a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 1 --2- 172.21.15.98:0/631989306 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de80a2530 0x7f7de80a5ee0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009d90 tx=0x7f7de002bce0 comp rx=0 tx=0).stop 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 1 -- 172.21.15.98:0/631989306 shutdown_connections 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 1 --2- 172.21.15.98:0/631989306 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de80a2530 0x7f7de80a5ee0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 1 --2- 172.21.15.98:0/631989306 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a5240 0x7f7de80a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 1 --2- 172.21.15.98:0/631989306 >> v2:172.21.15.117:3300/0 conn(0x7f7de80a4920 0x7f7de80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 1 -- 172.21.15.98:0/631989306 >> 172.21.15.98:0/631989306 conn(0x7f7de80ab850 msgr2=0x7f7de80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:44.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 1 -- 172.21.15.98:0/631989306 shutdown_connections 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 1 -- 172.21.15.98:0/631989306 wait complete. 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 10 monclient: build_initial_monmap 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.271+0000 7f7dee306640 10 monclient: monmap: 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:44.287 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 1 librados: starting msgr at 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 1 librados: starting objecter 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 1 Processor -- start 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 1 -- start start 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 1 librados: setting wanted keys 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 1 librados: calling monclient init 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 10 monclient: init 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 10 monclient: _add_conns ranks=[4,5,0,3,7,8,6,2,1] 2024-06-11T19:17:44.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7de80a2530 0x7f7de81cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 10 monclient(hunting): picked mon.noname-c con 0x7f7de80a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de80a4920 0x7f7de81cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 10 monclient(hunting): picked mon.noname-h con 0x7f7de80a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80a5240 0x7f7de8122d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 10 monclient(hunting): picked mon.noname-a con 0x7f7de80a5240 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7de80a6da0 con 0x7f7de80a5240 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7de8138e50 con 0x7f7de80a2530 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f7de80a6420 con 0x7f7de80a4920 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:44.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.272+0000 7f7dee306640 10 monclient(hunting): authenticate will time out at 1274.079977s 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7de80a2530 0x7f7de81cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dec8a3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80a5240 0x7f7de8122d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7de80a2530 0x7f7de81cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:50884/0 (socket says 172.21.15.98:50884) 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 1 -- 172.21.15.98:0/3972556160 learned_addr learned my addr 172.21.15.98:0/3972556160 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 10 monclient(hunting): get_auth_request con 0x7f7de80a2530 auth_method 0 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dec8a3640 10 monclient(hunting): get_auth_request con 0x7f7de80a5240 auth_method 0 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dec8a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dec8a3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dec8a3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:44.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de77fe640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de80a4920 0x7f7de81cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de77fe640 10 monclient(hunting): get_auth_request con 0x7f7de80a4920 auth_method 0 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dec8a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dec8a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dec8a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 10 monclient(hunting): handle_auth_done global_id 23881 payload 824 2024-06-11T19:17:44.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 1 -- 172.21.15.98:0/3972556160 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de80a4920 msgr2=0x7f7de81cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:44.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de80a4920 0x7f7de81cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:44.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 1 -- 172.21.15.98:0/3972556160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80a5240 msgr2=0x7f7de8122d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:44.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80a5240 0x7f7de8122d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:44.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:44.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 1 monclient: found mon.noname-c 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 1 -- 172.21.15.98:0/3972556160 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de80aabd0 con 0x7f7de80a2530 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dec8a3640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80a5240 0x7f7de8122d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de7fff640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7de80a2530 0x7f7de81cdaa0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800ec10 tx=0x7f7dd800c520 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 10 client.?.objecter ms_handle_connect 0x7f7de80a2530 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 1 -- 172.21.15.98:0/3972556160 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7dd8017070 con 0x7f7de80a2530 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 10 monclient: dump: 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:44.294 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 10 monclient: _finish_auth 0 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dee306640 5 monclient: authenticate success, global_id 23881 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 1 -- 172.21.15.98:0/3972556160 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dd800cc30 con 0x7f7de80a2530 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 1 -- 172.21.15.98:0/3972556160 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7dd800cde0 con 0x7f7de80a2530 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dee306640 10 monclient: _renew_subs 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dee306640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dee306640 1 -- 172.21.15.98:0/3972556160 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de8126cb0 con 0x7f7de80a2530 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de67fc640 4 set_mon_vals no callback set 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7de4ff9640 10 monclient: dump: 2024-06-11T19:17:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dee306640 10 client.23881.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dee306640 10 monclient: _renew_subs 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dee306640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dee306640 1 -- 172.21.15.98:0/3972556160 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de81269d0 con 0x7f7de80a2530 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.273+0000 7f7dee306640 1 librados: init done 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.274+0000 7f7dee306640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.274+0000 7f7dee306640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.274+0000 7f7dee306640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.274+0000 7f7de4ff9640 1 -- 172.21.15.98:0/3972556160 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f7dd801a070 con 0x7f7de80a2530 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.274+0000 7f7dee306640 1 -- 172.21.15.98:0/3972556160 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7de80a6750 con 0x7f7de80a2530 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.275+0000 7f7de4ff9640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7dd403fdb0 0x7f7dd4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.275+0000 7f7de4ff9640 1 -- 172.21.15.98:0/3972556160 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(343..343 src has 1..343) ==== 6230+0+0 (secure 0 0 0) 0x7f7dd800f040 con 0x7f7de80a2530 2024-06-11T19:17:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.275+0000 7f7de4ff9640 10 client.23881.objecter ms_dispatch 0x7f7de8000b90 osd_map(343..343 src has 1..343) 2024-06-11T19:17:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.275+0000 7f7de4ff9640 3 client.23881.objecter handle_osd_map got epochs [343,343] > 0 2024-06-11T19:17:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.275+0000 7f7de4ff9640 3 client.23881.objecter handle_osd_map decoding full epoch 343 2024-06-11T19:17:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.275+0000 7f7de4ff9640 20 client.23881.objecter dump_active .. 0 homeless 2024-06-11T19:17:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.275+0000 7f7de77fe640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7dd403fdb0 0x7f7dd4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.275+0000 7f7de77fe640 10 monclient: get_auth_request con 0x7f7dd403fdb0 auth_method 0 2024-06-11T19:17:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.275+0000 7f7de4ff9640 10 client.23881.objecter ms_handle_connect 0x7f7dd403fdb0 2024-06-11T19:17:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.275+0000 7f7de77fe640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7dd403fdb0 0x7f7dd4042270 secure :-1 s=READY pgs=4160 cs=0 l=1 rev1=1 crypto rx=0x7f7de8124450 tx=0x7f7ddc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.279+0000 7f7de4ff9640 1 -- 172.21.15.98:0/3972556160 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7dd8021070 con 0x7f7de80a2530 2024-06-11T19:17:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.279+0000 7f7de4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.279+0000 7f7de4ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.534+0000 7f7dee306640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:17:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.534+0000 7f7dee306640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:17:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.534+0000 7f7dee306640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:44.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.534+0000 7f7dee306640 1 -- 172.21.15.98:0/3972556160 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7de8125860 con 0x7f7de80a2530 2024-06-11T19:17:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.534+0000 7f7de4ff9640 1 -- 172.21.15.98:0/3972556160 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7f7de80aabd0 con 0x7f7de80a2530 2024-06-11T19:17:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.534+0000 7f7de4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:17:44.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.534+0000 7f7de4ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:44.534 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":36,"quorum":[0,1,2,3,4,5,6,7],"quorum_names":["a","f","b","g","c","h","d","i"],"quorum_leader_name":"f","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:17:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 10 librados: watch_flush enter 2024-06-11T19:17:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 10 librados: watch_flush exit 2024-06-11T19:17:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 20 client.23881.objecter shutdown clearing up homeless session... 2024-06-11T19:17:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 10 client.23881.objecter successfully canceled tick 2024-06-11T19:17:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 -- 172.21.15.98:0/3972556160 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7dd403fdb0 msgr2=0x7f7dd4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:44.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7dd403fdb0 0x7f7dd4042270 secure :-1 s=READY pgs=4160 cs=0 l=1 rev1=1 crypto rx=0x7f7de8124450 tx=0x7f7ddc008040 comp rx=0 tx=0).stop 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 10 monclient: shutdown 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 -- 172.21.15.98:0/3972556160 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7de80a2530 msgr2=0x7f7de81cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7de80a2530 0x7f7de81cdaa0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800ec10 tx=0x7f7dd800c520 comp rx=0 tx=0).stop 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 -- 172.21.15.98:0/3972556160 shutdown_connections 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7dd403fdb0 0x7f7dd4042270 unknown :-1 s=CLOSED pgs=4160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de80a5240 0x7f7de8122d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de80a4920 0x7f7de81cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 --2- 172.21.15.98:0/3972556160 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7de80a2530 0x7f7de81cdaa0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 -- 172.21.15.98:0/3972556160 >> 172.21.15.98:0/3972556160 conn(0x7f7de80ab850 msgr2=0x7f7de81bff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 -- 172.21.15.98:0/3972556160 shutdown_connections 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 -- 172.21.15.98:0/3972556160 wait complete. 2024-06-11T19:17:44.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:44.536+0000 7f7dee306640 1 librados: shutdown 2024-06-11T19:17:47.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:17:47.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.654+0000 7fc7f5748640 10 monclient: get_monmap_and_config 2024-06-11T19:17:47.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.654+0000 7fc7f5748640 10 monclient: build_initial_monmap 2024-06-11T19:17:47.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.654+0000 7fc7f5748640 10 monclient: monmap: 2024-06-11T19:17:47.653 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:47.653 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:47.653 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:47.655078+0000 2024-06-11T19:17:47.653 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:47.655078+0000 2024-06-11T19:17:47.653 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:47.654 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.654+0000 7fc7f5748640 1 Processor -- start 2024-06-11T19:17:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.654+0000 7fc7f5748640 1 -- start start 2024-06-11T19:17:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.654+0000 7fc7f5748640 10 monclient: init 2024-06-11T19:17:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 10 monclient: _add_conns ranks=[4,1,5,3,10,2,0,7,8,9,6] 2024-06-11T19:17:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f01347f0 0x7fc7f009e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 10 monclient(hunting): picked mon.noname-c con 0x7fc7f01347f0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc7f0134bd0 addr v1:172.21.15.117:6789/0 2024-06-11T19:17:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc7f0135010 0x7fc7f00a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 10 monclient(hunting): picked mon.noname-h con 0x7fc7f0135010 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:47.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7f009d410 con 0x7fc7f0134bd0 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc7f00687f0 con 0x7fc7f01347f0 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc7f009d6f0 con 0x7fc7f0135010 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7f5748640 10 monclient(hunting): authenticate will time out at 1277.462220s 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f01347f0 0x7fc7f009e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f01347f0 0x7fc7f009e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:50890/0 (socket says 172.21.15.98:50890) 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 1 -- 172.21.15.98:0/3482999690 learned_addr learned my addr 172.21.15.98:0/3482999690 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 10 monclient(hunting): get_auth_request con 0x7fc7f01347f0 auth_method 0 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:47.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7ef7fe640 1 --2- 172.21.15.98:0/3482999690 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc7f0135010 0x7fc7f00a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7ef7fe640 10 monclient(hunting): get_auth_request con 0x7fc7f0135010 auth_method 0 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 10 monclient(hunting): handle_auth_done global_id 23890 payload 274 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 1 -- 172.21.15.98:0/3482999690 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc7f0135010 msgr2=0x7fc7f00a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 1 --2- 172.21.15.98:0/3482999690 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc7f0135010 0x7fc7f00a3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 1 -- 172.21.15.98:0/3482999690 >> v1:172.21.15.117:6789/0 conn(0x7fc7f0134bd0 legacy=0x7fc7f00a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 10 monclient: _finish_hunting 0 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 1 monclient: found mon.noname-c 2024-06-11T19:17:47.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.655+0000 7fc7eeffd640 1 -- 172.21.15.98:0/3482999690 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f009d9d0 con 0x7fc7f01347f0 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7eeffd640 1 --2- 172.21.15.98:0/3482999690 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f01347f0 0x7fc7f009e6b0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00098a0 tx=0x7fc7e002b8d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=73f49ea4b1e609e2 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 1 -- 172.21.15.98:0/3482999690 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7e0035020 con 0x7fc7f01347f0 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 10 monclient: dump: 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:47.666 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 10 monclient: _finish_auth 0 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 10 monclient: _check_auth_tickets 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:47.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7f5748640 5 monclient: authenticate success, global_id 23890 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7f5748640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 1 -- 172.21.15.98:0/3482999690 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7e0031aa0 con 0x7fc7f01347f0 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7f5748640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 1 -- 172.21.15.98:0/3482999690 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7e0031cd0 con 0x7fc7f01347f0 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7f5748640 4 set_mon_vals no callback set 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.656+0000 7fc7edffb640 10 monclient: dump: 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:47.668 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 10 monclient: shutdown 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 1 -- 172.21.15.98:0/3482999690 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f01347f0 msgr2=0x7fc7f009e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:47.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 1 --2- 172.21.15.98:0/3482999690 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f01347f0 0x7fc7f009e6b0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00098a0 tx=0x7fc7e002b8d0 comp rx=0 tx=0).stop 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 1 -- 172.21.15.98:0/3482999690 shutdown_connections 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 1 --2- 172.21.15.98:0/3482999690 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc7f0135010 0x7fc7f00a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 1 --2- 172.21.15.98:0/3482999690 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f01347f0 0x7fc7f009e6b0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 1 -- 172.21.15.98:0/3482999690 >> 172.21.15.98:0/3482999690 conn(0x7fc7f00a4780 msgr2=0x7fc7f00a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 1 -- 172.21.15.98:0/3482999690 shutdown_connections 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 1 -- 172.21.15.98:0/3482999690 wait complete. 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 10 monclient: build_initial_monmap 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 10 monclient: monmap: 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:47.670 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 1 librados: starting msgr at 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.657+0000 7fc7f5748640 1 librados: starting objecter 2024-06-11T19:17:47.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 1 Processor -- start 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 1 -- start start 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 1 librados: setting wanted keys 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 1 librados: calling monclient init 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 10 monclient: init 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 10 monclient: _add_conns ranks=[1,4,2,6,5,3,8,7,0] 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f01347f0 0x7fc7f01ce020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 10 monclient(hunting): picked mon.noname-f con 0x7fc7f01347f0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f0134bd0 0x7fc7f01ce560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 10 monclient(hunting): picked mon.noname-c con 0x7fc7f0134bd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7f0135010 0x7fc7f011ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 10 monclient(hunting): picked mon.noname-b con 0x7fc7f0135010 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:47.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f0068c40 con 0x7fc7f01347f0 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc7f0068ac0 con 0x7fc7f0135010 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc7f00a42e0 con 0x7fc7f0134bd0 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7f5748640 10 monclient(hunting): authenticate will time out at 1277.465758s 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ee7fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f0134bd0 0x7fc7f01ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ef7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7f0135010 0x7fc7f011ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ee7fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f0134bd0 0x7fc7f01ce560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:50904/0 (socket says 172.21.15.98:50904) 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ee7fc640 1 -- 172.21.15.98:0/4032900629 learned_addr learned my addr 172.21.15.98:0/4032900629 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ee7fc640 10 monclient(hunting): get_auth_request con 0x7fc7f0134bd0 auth_method 0 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ee7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:47.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ef7fe640 10 monclient(hunting): get_auth_request con 0x7fc7f0135010 auth_method 0 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.658+0000 7fc7eeffd640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f01347f0 0x7fc7f01ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7eeffd640 10 monclient(hunting): get_auth_request con 0x7fc7f01347f0 auth_method 0 2024-06-11T19:17:47.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7ee7fc640 10 monclient(hunting): handle_auth_done global_id 23899 payload 824 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7ee7fc640 1 -- 172.21.15.98:0/4032900629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7f0135010 msgr2=0x7fc7f011ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7ee7fc640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7f0135010 0x7fc7f011ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7ee7fc640 1 -- 172.21.15.98:0/4032900629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f01347f0 msgr2=0x7fc7f01ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7ee7fc640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f01347f0 0x7fc7f01ce020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7ee7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7ee7fc640 1 monclient: found mon.noname-c 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7ee7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7ee7fc640 1 -- 172.21.15.98:0/4032900629 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f00a3960 con 0x7fc7f0134bd0 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7eeffd640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f01347f0 0x7fc7f01ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:17:47.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7ee7fc640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f0134bd0 0x7fc7f01ce560 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400c460 tx=0x7fc7e400c920 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 10 client.?.objecter ms_handle_connect 0x7fc7f0134bd0 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 1 -- 172.21.15.98:0/4032900629 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7e4016020 con 0x7fc7f0134bd0 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 10 monclient: dump: 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:47.676 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 10 monclient: _finish_auth 0 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 10 monclient: _check_auth_tickets 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 1 -- 172.21.15.98:0/4032900629 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7e4012630 con 0x7fc7f0134bd0 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7f5748640 5 monclient: authenticate success, global_id 23899 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7f5748640 10 monclient: _renew_subs 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7f5748640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7f5748640 1 -- 172.21.15.98:0/4032900629 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f00a3510 con 0x7fc7f0134bd0 2024-06-11T19:17:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 1 -- 172.21.15.98:0/4032900629 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7e4012c90 con 0x7fc7f0134bd0 2024-06-11T19:17:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:17:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7d77fe640 10 monclient: dump: 2024-06-11T19:17:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:47.678 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7f5748640 10 client.23899.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7f5748640 10 monclient: _renew_subs 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7f5748640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7f5748640 1 -- 172.21.15.98:0/4032900629 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f0122620 con 0x7fc7f0134bd0 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7f5748640 1 librados: init done 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.659+0000 7fc7ed7fa640 4 set_mon_vals no callback set 2024-06-11T19:17:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.660+0000 7fc7f5748640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.660+0000 7fc7f5748640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.660+0000 7fc7f5748640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.660+0000 7fc7d77fe640 1 -- 172.21.15.98:0/4032900629 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fc7e4019070 con 0x7fc7f0134bd0 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.660+0000 7fc7f5748640 1 -- 172.21.15.98:0/4032900629 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7f009e6b0 con 0x7fc7f0134bd0 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.660+0000 7fc7d77fe640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc7dc03fdb0 0x7fc7dc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.660+0000 7fc7d77fe640 1 -- 172.21.15.98:0/4032900629 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(343..343 src has 1..343) ==== 6230+0+0 (secure 0 0 0) 0x7fc7e4050b90 con 0x7fc7f0134bd0 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.660+0000 7fc7d77fe640 10 client.23899.objecter ms_dispatch 0x7fc7f0000b90 osd_map(343..343 src has 1..343) 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.660+0000 7fc7d77fe640 3 client.23899.objecter handle_osd_map got epochs [343,343] > 0 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.660+0000 7fc7d77fe640 3 client.23899.objecter handle_osd_map decoding full epoch 343 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.660+0000 7fc7d77fe640 20 client.23899.objecter dump_active .. 0 homeless 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.661+0000 7fc7eeffd640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc7dc03fdb0 0x7fc7dc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.661+0000 7fc7eeffd640 10 monclient: get_auth_request con 0x7fc7dc03fdb0 auth_method 0 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.661+0000 7fc7d77fe640 10 client.23899.objecter ms_handle_connect 0x7fc7dc03fdb0 2024-06-11T19:17:47.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.661+0000 7fc7eeffd640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc7dc03fdb0 0x7fc7dc042270 secure :-1 s=READY pgs=4161 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0036000 tx=0x7fc7e0008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.665+0000 7fc7d77fe640 1 -- 172.21.15.98:0/4032900629 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc7e401a3b0 con 0x7fc7f0134bd0 2024-06-11T19:17:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.665+0000 7fc7d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.665+0000 7fc7d77fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:47.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.924+0000 7fc7f5748640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:17:47.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.924+0000 7fc7f5748640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:17:47.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.924+0000 7fc7f5748640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:47.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.924+0000 7fc7f5748640 1 -- 172.21.15.98:0/4032900629 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc7f01235b0 con 0x7fc7f0134bd0 2024-06-11T19:17:47.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.924+0000 7fc7d77fe640 1 -- 172.21.15.98:0/4032900629 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fc7e4012820 con 0x7fc7f0134bd0 2024-06-11T19:17:47.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.924+0000 7fc7d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:17:47.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.924+0000 7fc7d77fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:47.924 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":38,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:17:47.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 10 librados: watch_flush enter 2024-06-11T19:17:47.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 10 librados: watch_flush exit 2024-06-11T19:17:47.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 20 client.23899.objecter shutdown clearing up homeless session... 2024-06-11T19:17:47.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 10 client.23899.objecter successfully canceled tick 2024-06-11T19:17:47.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 1 -- 172.21.15.98:0/4032900629 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc7dc03fdb0 msgr2=0x7fc7dc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc7dc03fdb0 0x7fc7dc042270 secure :-1 s=READY pgs=4161 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0036000 tx=0x7fc7e0008030 comp rx=0 tx=0).stop 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 10 monclient: shutdown 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 1 -- 172.21.15.98:0/4032900629 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f0134bd0 msgr2=0x7fc7f01ce560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f0134bd0 0x7fc7f01ce560 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400c460 tx=0x7fc7e400c920 comp rx=0 tx=0).stop 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 1 -- 172.21.15.98:0/4032900629 shutdown_connections 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc7dc03fdb0 0x7fc7dc042270 unknown :-1 s=CLOSED pgs=4161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc7f0134bd0 0x7fc7f01ce560 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.926+0000 7fc7f5748640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc7f0135010 0x7fc7f011ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.927+0000 7fc7f5748640 1 --2- 172.21.15.98:0/4032900629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f01347f0 0x7fc7f01ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.927+0000 7fc7f5748640 1 -- 172.21.15.98:0/4032900629 >> 172.21.15.98:0/4032900629 conn(0x7fc7f00a4780 msgr2=0x7fc7f01bf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.927+0000 7fc7f5748640 1 -- 172.21.15.98:0/4032900629 shutdown_connections 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.927+0000 7fc7f5748640 1 -- 172.21.15.98:0/4032900629 wait complete. 2024-06-11T19:17:47.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:47.927+0000 7fc7f5748640 1 librados: shutdown 2024-06-11T19:17:47.936 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:17:47.937 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:17:47.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:17:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.039+0000 7f82ad77d640 10 monclient: get_monmap_and_config 2024-06-11T19:17:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.039+0000 7f82ad77d640 10 monclient: build_initial_monmap 2024-06-11T19:17:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.039+0000 7f82ad77d640 10 monclient: monmap: 2024-06-11T19:17:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:48.038 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:48.040433+0000 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:48.040433+0000 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:48.039 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:48.040 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 1 Processor -- start 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 1 -- start start 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 10 monclient: init 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 10 monclient: _add_conns ranks=[10,2,1,6,3,0,5,7,8,4,9] 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f82a80a6260 0x7f82a80a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 10 monclient(hunting): picked mon.noname-f con 0x7f82a80a6260 addr v2:172.21.15.117:3300/0 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82a806c890 0x7f82a80a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 10 monclient(hunting): picked mon.noname-b con 0x7f82a806c890 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f82a80a2d00 addr v1:172.21.15.117:6789/0 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82a806c650 con 0x7f82a80a2d00 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f82a80a05b0 con 0x7f82a80a6260 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f82a80a6640 con 0x7f82a806c890 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82ad77d640 10 monclient(hunting): authenticate will time out at 1277.847713s 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f829ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82a806c890 0x7f82a80a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f829ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82a806c890 0x7f82a80a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45146/0 (socket says 172.21.15.98:45146) 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f829ffff640 1 -- 172.21.15.98:0/29245367 learned_addr learned my addr 172.21.15.98:0/29245367 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f829ffff640 10 monclient(hunting): get_auth_request con 0x7f82a806c890 auth_method 0 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f829ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f829ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f829ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f829ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f829ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f829ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82a6ffd640 1 --2- 172.21.15.98:0/29245367 >> v2:172.21.15.117:3300/0 conn(0x7f82a80a6260 0x7f82a80a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.040+0000 7f82a6ffd640 10 monclient(hunting): get_auth_request con 0x7f82a80a6260 auth_method 0 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f82a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f82a6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f82a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f829ffff640 10 monclient(hunting): handle_auth_done global_id 84359 payload 274 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f829ffff640 1 -- 172.21.15.98:0/29245367 >> v2:172.21.15.117:3300/0 conn(0x7f82a80a6260 msgr2=0x7f82a80a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f829ffff640 1 --2- 172.21.15.98:0/29245367 >> v2:172.21.15.117:3300/0 conn(0x7f82a80a6260 0x7f82a80a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f829ffff640 1 -- 172.21.15.98:0/29245367 >> v1:172.21.15.117:6789/0 conn(0x7f82a80a2d00 legacy=0x7f82a80a4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f829ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f829ffff640 1 monclient: found mon.noname-b 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f829ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f829ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f829ffff640 1 -- 172.21.15.98:0/29245367 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a8073b60 con 0x7f82a806c890 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f829ffff640 1 --2- 172.21.15.98:0/29245367 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82a806c890 0x7f82a80a48b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f82a806c400 tx=0x7f829002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3357f6885a95d4ec server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f82a67fc640 1 -- 172.21.15.98:0/29245367 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8290038070 con 0x7f82a806c890 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f82a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f82a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f82a67fc640 10 monclient: dump: 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.052 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f82a67fc640 10 monclient: _finish_auth 0 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f82a67fc640 10 monclient: _check_auth_tickets 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.041+0000 7f82a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82a67fc640 1 -- 172.21.15.98:0/29245367 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8290031970 con 0x7f82a806c890 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82a67fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82a67fc640 1 -- 172.21.15.98:0/29245367 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8290032390 con 0x7f82a806c890 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82a67fc640 10 monclient: dump: 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.053 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 5 monclient: authenticate success, global_id 84359 2024-06-11T19:17:48.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 4 set_mon_vals no callback set 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 10 monclient: shutdown 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 1 -- 172.21.15.98:0/29245367 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82a806c890 msgr2=0x7f82a80a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 1 --2- 172.21.15.98:0/29245367 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82a806c890 0x7f82a80a48b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f82a806c400 tx=0x7f829002b7a0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 1 -- 172.21.15.98:0/29245367 shutdown_connections 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 1 --2- 172.21.15.98:0/29245367 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f82a806c890 0x7f82a80a48b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 1 --2- 172.21.15.98:0/29245367 >> v2:172.21.15.117:3300/0 conn(0x7f82a80a6260 0x7f82a80a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 1 -- 172.21.15.98:0/29245367 >> 172.21.15.98:0/29245367 conn(0x7f82a80a95b0 msgr2=0x7f82a80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 1 -- 172.21.15.98:0/29245367 shutdown_connections 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.042+0000 7f82ad77d640 1 -- 172.21.15.98:0/29245367 wait complete. 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.043+0000 7f82ad77d640 10 monclient: build_initial_monmap 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.043+0000 7f82ad77d640 10 monclient: monmap: 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:48.055 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.043+0000 7f82ad77d640 1 librados: starting msgr at 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.043+0000 7f82ad77d640 1 librados: starting objecter 2024-06-11T19:17:48.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.043+0000 7f82ad77d640 1 Processor -- start 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.043+0000 7f82ad77d640 1 -- start start 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.043+0000 7f82ad77d640 1 librados: setting wanted keys 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.043+0000 7f82ad77d640 1 librados: calling monclient init 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.043+0000 7f82ad77d640 10 monclient: init 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient: _add_conns ranks=[0,4,3,6,1,2,5,8,7] 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a806c890 0x7f82a81cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient(hunting): picked mon.noname-a con 0x7f82a806c890 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82a80a2d00 0x7f82a81ce400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient(hunting): picked mon.noname-c con 0x7f82a80a2d00 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82a80a6260 0x7f82a809d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient(hunting): picked mon.noname-g con 0x7f82a80a6260 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f82a80a2410 con 0x7f82a806c890 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f82a80a2290 con 0x7f82a80a6260 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f82a80a5500 con 0x7f82a80a2d00 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient(hunting): authenticate will time out at 1277.851154s 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82a80a2d00 0x7f82a81ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a806c890 0x7f82a81cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82a80a2d00 0x7f82a81ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:50908/0 (socket says 172.21.15.98:50908) 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 1 -- 172.21.15.98:0/3051821360 learned_addr learned my addr 172.21.15.98:0/3051821360 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 10 monclient(hunting): get_auth_request con 0x7f82a80a2d00 auth_method 0 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a6ffd640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a806c890 0x7f82a81cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56584/0 (socket says 172.21.15.98:56584) 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a6ffd640 10 monclient(hunting): get_auth_request con 0x7f82a806c890 auth_method 0 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a77fe640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82a80a6260 0x7f82a809d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a77fe640 10 monclient(hunting): get_auth_request con 0x7f82a80a6260 auth_method 0 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 10 monclient(hunting): handle_auth_done global_id 23908 payload 824 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 1 -- 172.21.15.98:0/3051821360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82a80a6260 msgr2=0x7f82a809d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82a80a6260 0x7f82a809d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 1 -- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a806c890 msgr2=0x7f82a81cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a806c890 0x7f82a81cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 1 monclient: found mon.noname-c 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 1 -- 172.21.15.98:0/3051821360 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a80a8860 con 0x7f82a80a2d00 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829ffff640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82a80a2d00 0x7f82a81ce400 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f82900097c0 tx=0x7f82900046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 10 client.?.objecter ms_handle_connect 0x7f82a80a2d00 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8290042020 con 0x7f82a80a2d00 2024-06-11T19:17:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 10 monclient: dump: 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8290004470 con 0x7f82a80a2d00 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 5 monclient: authenticate success, global_id 23908 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient: _renew_subs 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 1 -- 172.21.15.98:0/3051821360 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a80a4ec0 con 0x7f82a80a2d00 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8290038040 con 0x7f82a80a2d00 2024-06-11T19:17:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:17:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f829f7fe640 10 monclient: dump: 2024-06-11T19:17:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82a5ffb640 4 set_mon_vals no callback set 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 client.23908.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient: _renew_subs 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.044+0000 7f82ad77d640 1 -- 172.21.15.98:0/3051821360 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a81a9d20 con 0x7f82a80a2d00 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.045+0000 7f82ad77d640 1 librados: init done 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.045+0000 7f82ad77d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.045+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8290045020 con 0x7f82a80a2d00 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.045+0000 7f82ad77d640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809d7a0 0x7f82a809db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.045+0000 7f82ad77d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.045+0000 7f82ad77d640 1 -- 172.21.15.98:0/3051821360 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f82a8050830 con 0x7f82a809d7a0 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.045+0000 7f82a6ffd640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809d7a0 0x7f82a809db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.045+0000 7f82a6ffd640 10 monclient: get_auth_request con 0x7f82a809d7a0 auth_method 0 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.045+0000 7f82a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.045+0000 7f82a6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.045+0000 7f82a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f82a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f82a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f82a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f828803fde0 0x7f82880422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(343..343 src has 1..343) ==== 6230+0+0 (secure 0 0 0) 0x7f8290072f60 con 0x7f82a80a2d00 2024-06-11T19:17:48.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 10 client.23908.objecter ms_dispatch 0x7f82a8000b90 osd_map(343..343 src has 1..343) 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 3 client.23908.objecter handle_osd_map got epochs [343,343] > 0 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 3 client.23908.objecter handle_osd_map decoding full epoch 343 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f82a6ffd640 10 monclient(hunting): handle_auth_done global_id 64926 payload 824 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 20 client.23908.objecter dump_active .. 0 homeless 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f82a6ffd640 1 -- 172.21.15.98:0/3051821360 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f82a80a1d50 con 0x7f82a809d7a0 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f82a77fe640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f828803fde0 0x7f82880422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f82a77fe640 10 monclient: get_auth_request con 0x7f828803fde0 auth_method 0 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 10 client.23908.objecter ms_handle_connect 0x7f82a809d7a0 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 10 client.23908.objecter resend_mon_ops 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f82a6ffd640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809d7a0 0x7f82a809db80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8298010820 tx=0x7f8298010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f829800eea0 con 0x7f82a809d7a0 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 10 monclient: dump: 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.067 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.068 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f829800ea30 con 0x7f82a809d7a0 2024-06-11T19:17:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809d7a0 msgr2=0x7f82a809db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809d7a0 0x7f82a809db80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8298010820 tx=0x7f8298010ce0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f82a77fe640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f828803fde0 0x7f82880422a0 secure :-1 s=READY pgs=4162 cs=0 l=1 rev1=1 crypto rx=0x7f82a0005520 tx=0x7f82a0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809d7a0 msgr2=0x7f82a809db80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809d7a0 0x7f82a809db80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.046+0000 7f829f7fe640 10 client.23908.objecter ms_handle_connect 0x7f828803fde0 2024-06-11T19:17:48.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82ad77d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:48.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82ad77d640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809e0c0 0x7f82a8070d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82ad77d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82ad77d640 1 -- 172.21.15.98:0/3051821360 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f82a809f380 con 0x7f82a809e0c0 2024-06-11T19:17:48.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82a6ffd640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809e0c0 0x7f82a8070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82a6ffd640 10 monclient: get_auth_request con 0x7f82a809e0c0 auth_method 0 2024-06-11T19:17:48.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82a6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82a6ffd640 10 monclient(hunting): handle_auth_done global_id 64935 payload 824 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82a6ffd640 1 -- 172.21.15.98:0/3051821360 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f82a806e510 con 0x7f82a809e0c0 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 10 client.23908.objecter ms_handle_connect 0x7f82a809e0c0 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 10 client.23908.objecter resend_mon_ops 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f82a6ffd640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809e0c0 0x7f82a8070d10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8298000c00 tx=0x7f82980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8298003bf0 con 0x7f82a809e0c0 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:48.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 10 monclient: dump: 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.072 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.073 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f8298003bf0 con 0x7f82a809e0c0 2024-06-11T19:17:48.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:48.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:48.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809e0c0 msgr2=0x7f82a8070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809e0c0 0x7f82a8070d10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8298000c00 tx=0x7f82980046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 1 -- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809e0c0 msgr2=0x7f82a8070d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:48.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.053+0000 7f829f7fe640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82a809e0c0 0x7f82a8070d10 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8298000c00 tx=0x7f82980046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.073 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:48.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:48.074 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:48.074 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:48.074 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:48.074 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 259881, 2024-06-11T19:17:48.074 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:48.074 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:48.074 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:48.074 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:48.074 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:48.074 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:48.074 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:48.074 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.075 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.076 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:48.077 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.078 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:48.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:48.082 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:48.086 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:48.086 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.086 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.086 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.086 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:48.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:48.086 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:48.087 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:48.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:48.091 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.092 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.093 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.094 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:48.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.095 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.095 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:48.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.095 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.095 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.095 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:48.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.095 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.095 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:48.095 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 10 librados: watch_flush enter 2024-06-11T19:17:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 10 librados: watch_flush exit 2024-06-11T19:17:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 20 client.23908.objecter shutdown clearing up homeless session... 2024-06-11T19:17:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 10 client.23908.objecter successfully canceled tick 2024-06-11T19:17:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 1 -- 172.21.15.98:0/3051821360 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f828803fde0 msgr2=0x7f82880422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f828803fde0 0x7f82880422a0 secure :-1 s=READY pgs=4162 cs=0 l=1 rev1=1 crypto rx=0x7f82a0005520 tx=0x7f82a0009040 comp rx=0 tx=0).stop 2024-06-11T19:17:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 10 monclient: shutdown 2024-06-11T19:17:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 1 -- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82a80a2d00 msgr2=0x7f82a81ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82a80a2d00 0x7f82a81ce400 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f82900097c0 tx=0x7f82900046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82a77fe640 1 -- 172.21.15.98:0/3051821360 reap_dead start 2024-06-11T19:17:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 1 -- 172.21.15.98:0/3051821360 shutdown_connections 2024-06-11T19:17:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 1 --2- 172.21.15.98:0/3051821360 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f82a80a2d00 0x7f82a81ce400 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 1 -- 172.21.15.98:0/3051821360 >> 172.21.15.98:0/3051821360 conn(0x7f82a80a95b0 msgr2=0x7f82a81bed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 1 -- 172.21.15.98:0/3051821360 shutdown_connections 2024-06-11T19:17:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 1 -- 172.21.15.98:0/3051821360 wait complete. 2024-06-11T19:17:48.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.054+0000 7f82ad77d640 1 librados: shutdown 2024-06-11T19:17:48.098 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:48.098 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:17:48.099 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:17:48.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.198+0000 7f2b9647a640 10 monclient: get_monmap_and_config 2024-06-11T19:17:48.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.198+0000 7f2b9647a640 10 monclient: build_initial_monmap 2024-06-11T19:17:48.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.198+0000 7f2b9647a640 10 monclient: monmap: 2024-06-11T19:17:48.198 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:48.198 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.198 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:48.199916+0000 2024-06-11T19:17:48.198 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:48.199916+0000 2024-06-11T19:17:48.198 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:48.198 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.198 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:48.199 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:48.199 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.199+0000 7f2b9647a640 1 Processor -- start 2024-06-11T19:17:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.199+0000 7f2b9647a640 1 -- start start 2024-06-11T19:17:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.199+0000 7f2b9647a640 10 monclient: init 2024-06-11T19:17:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 10 monclient: _add_conns ranks=[10,3,5,2,8,7,9,6,4,0,1] 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2b900a4920 0x7f2b900a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 10 monclient(hunting): picked mon.noname-f con 0x7f2b900a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900a5240 0x7f2b900a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 10 monclient(hunting): picked mon.noname-g con 0x7f2b900a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2b900a2530 0x7f2b900a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 10 monclient(hunting): picked mon.noname-h con 0x7f2b900a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f2b900a7140 con 0x7f2b900a4920 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f2b900a2b50 con 0x7f2b900a5240 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f2b900a2e30 con 0x7f2b900a2530 2024-06-11T19:17:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b9647a640 10 monclient(hunting): authenticate will time out at 1278.007293s 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8ffff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2b900a4920 0x7f2b900a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b94a17640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2b900a2530 0x7f2b900a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8f7fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900a5240 0x7f2b900a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b94a17640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2b900a2530 0x7f2b900a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:59130/0 (socket says 172.21.15.98:59130) 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8f7fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900a5240 0x7f2b900a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:35686/0 (socket says 172.21.15.98:35686) 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b94a17640 1 -- 172.21.15.98:0/4289723976 learned_addr learned my addr 172.21.15.98:0/4289723976 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b94a17640 10 monclient(hunting): get_auth_request con 0x7f2b900a2530 auth_method 0 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b94a17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b94a17640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b94a17640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8f7fe640 10 monclient(hunting): get_auth_request con 0x7f2b900a5240 auth_method 0 2024-06-11T19:17:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8ffff640 10 monclient(hunting): get_auth_request con 0x7f2b900a4920 auth_method 0 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b94a17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b94a17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b94a17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.200+0000 7f2b8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b94a17640 10 monclient(hunting): handle_auth_done global_id 23009 payload 274 2024-06-11T19:17:48.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b94a17640 1 -- 172.21.15.98:0/4289723976 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900a5240 msgr2=0x7f2b900a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:48.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b94a17640 1 --2- 172.21.15.98:0/4289723976 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900a5240 0x7f2b900a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b94a17640 1 -- 172.21.15.98:0/4289723976 >> v2:172.21.15.117:3300/0 conn(0x7f2b900a4920 msgr2=0x7f2b900a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:48.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b94a17640 1 --2- 172.21.15.98:0/4289723976 >> v2:172.21.15.117:3300/0 conn(0x7f2b900a4920 0x7f2b900a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b94a17640 10 monclient: _finish_hunting 0 2024-06-11T19:17:48.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b94a17640 1 monclient: found mon.noname-h 2024-06-11T19:17:48.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b94a17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:48.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b94a17640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:17:48.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b94a17640 1 -- 172.21.15.98:0/4289723976 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b900a3110 con 0x7f2b900a2530 2024-06-11T19:17:48.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b8ffff640 1 --2- 172.21.15.98:0/4289723976 >> v2:172.21.15.117:3300/0 conn(0x7f2b900a4920 0x7f2b900a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:48.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b8f7fe640 1 --2- 172.21.15.98:0/4289723976 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900a5240 0x7f2b900a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.201+0000 7f2b94a17640 1 --2- 172.21.15.98:0/4289723976 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2b900a2530 0x7f2b900a5ee0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009e50 tx=0x7f2b8802bdb0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=7fa954d48d5d4fc4 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 1 -- 172.21.15.98:0/4289723976 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b8802cbc0 con 0x7f2b900a2530 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 10 monclient: dump: 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.214 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 10 monclient: _finish_auth 0 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 10 monclient: _check_auth_tickets 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 1 -- 172.21.15.98:0/4289723976 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b8802cd80 con 0x7f2b900a2530 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 5 monclient: authenticate success, global_id 23009 2024-06-11T19:17:48.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 1 -- 172.21.15.98:0/4289723976 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b88034860 con 0x7f2b900a2530 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 4 set_mon_vals no callback set 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b8effd640 10 monclient: dump: 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.216 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 10 monclient: shutdown 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 1 -- 172.21.15.98:0/4289723976 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2b900a2530 msgr2=0x7f2b900a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 1 --2- 172.21.15.98:0/4289723976 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2b900a2530 0x7f2b900a5ee0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009e50 tx=0x7f2b8802bdb0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 1 -- 172.21.15.98:0/4289723976 shutdown_connections 2024-06-11T19:17:48.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 1 --2- 172.21.15.98:0/4289723976 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900a5240 0x7f2b900a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 1 --2- 172.21.15.98:0/4289723976 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2b900a2530 0x7f2b900a5ee0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 1 --2- 172.21.15.98:0/4289723976 >> v2:172.21.15.117:3300/0 conn(0x7f2b900a4920 0x7f2b900a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 1 -- 172.21.15.98:0/4289723976 >> 172.21.15.98:0/4289723976 conn(0x7f2b900ab850 msgr2=0x7f2b900a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 1 -- 172.21.15.98:0/4289723976 shutdown_connections 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.202+0000 7f2b9647a640 1 -- 172.21.15.98:0/4289723976 wait complete. 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.203+0000 7f2b9647a640 10 monclient: build_initial_monmap 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.203+0000 7f2b9647a640 10 monclient: monmap: 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:48.218 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.203+0000 7f2b9647a640 1 librados: starting msgr at 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.203+0000 7f2b9647a640 1 librados: starting objecter 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.203+0000 7f2b9647a640 1 Processor -- start 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.203+0000 7f2b9647a640 1 -- start start 2024-06-11T19:17:48.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.203+0000 7f2b9647a640 1 librados: setting wanted keys 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.203+0000 7f2b9647a640 1 librados: calling monclient init 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.203+0000 7f2b9647a640 10 monclient: init 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 10 monclient: _add_conns ranks=[8,1,4,5,7,0,6,3,2] 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b900a2530 0x7f2b901cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 10 monclient(hunting): picked mon.noname-e con 0x7f2b900a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b900a4920 0x7f2b901cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 10 monclient(hunting): picked mon.noname-f con 0x7f2b900a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2b900a5240 0x7f2b90124f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 10 monclient(hunting): picked mon.noname-c con 0x7f2b900a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2b900a6da0 con 0x7f2b900a4920 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2b90138e50 con 0x7f2b900a5240 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2b900a6420 con 0x7f2b900a2530 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b9647a640 10 monclient(hunting): authenticate will time out at 1278.011379s 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b900a2530 0x7f2b901cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b94a17640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2b900a5240 0x7f2b90124f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b900a2530 0x7f2b901cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39008/0 (socket says 172.21.15.98:39008) 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 1 -- 172.21.15.98:0/1554781883 learned_addr learned my addr 172.21.15.98:0/1554781883 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b94a17640 10 monclient(hunting): get_auth_request con 0x7f2b900a5240 auth_method 0 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b94a17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b94a17640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b94a17640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 10 monclient(hunting): get_auth_request con 0x7f2b900a2530 auth_method 0 2024-06-11T19:17:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b94a17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b94a17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b94a17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8f7fe640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b900a4920 0x7f2b901cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8f7fe640 10 monclient(hunting): get_auth_request con 0x7f2b900a4920 auth_method 0 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 10 monclient(hunting): handle_auth_done global_id 94112 payload 824 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 1 -- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2b900a5240 msgr2=0x7f2b90124f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2b900a5240 0x7f2b90124f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 1 -- 172.21.15.98:0/1554781883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b900a4920 msgr2=0x7f2b901cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b900a4920 0x7f2b901cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 1 monclient: found mon.noname-e 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 1 -- 172.21.15.98:0/1554781883 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b90128c70 con 0x7f2b900a2530 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b94a17640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2b900a5240 0x7f2b90124f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8f7fe640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b900a4920 0x7f2b901cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8ffff640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b900a2530 0x7f2b901cda70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000ec10 tx=0x7f2b8000c520 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8cff9640 10 client.?.objecter ms_handle_connect 0x7f2b900a2530 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.204+0000 7f2b8cff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b80017070 con 0x7f2b900a2530 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 10 monclient: dump: 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.224 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 10 monclient: _finish_auth 0 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 10 monclient: _check_auth_tickets 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b9647a640 5 monclient: authenticate success, global_id 94112 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b8000cc30 con 0x7f2b900a2530 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b8000cde0 con 0x7f2b900a2530 2024-06-11T19:17:48.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b9647a640 10 monclient: _renew_subs 2024-06-11T19:17:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b9647a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:17:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b9647a640 1 -- 172.21.15.98:0/1554781883 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b901277c0 con 0x7f2b900a2530 2024-06-11T19:17:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8e7fc640 4 set_mon_vals no callback set 2024-06-11T19:17:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:17:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b9647a640 10 client.94112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b8cff9640 10 monclient: dump: 2024-06-11T19:17:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.226 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b9647a640 10 monclient: _renew_subs 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b9647a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:17:48.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b9647a640 1 -- 172.21.15.98:0/1554781883 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b900a8fc0 con 0x7f2b900a2530 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.205+0000 7f2b9647a640 1 librados: init done 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f2b8001a070 con 0x7f2b900a2530 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b9647a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b9647a640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90128880 0x7f2b90125c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b9647a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b9647a640 1 -- 172.21.15.98:0/1554781883 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2b901283a0 con 0x7f2b90128880 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8f7fe640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90128880 0x7f2b90125c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8f7fe640 10 monclient: get_auth_request con 0x7f2b90128880 auth_method 0 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2b7c03fdb0 0x7f2b7c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(343..343 src has 1..343) ==== 6230+0+0 (secure 0 0 0) 0x7f2b8000f040 con 0x7f2b900a2530 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 10 client.94112.objecter ms_dispatch 0x7f2b90000b90 osd_map(343..343 src has 1..343) 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 3 client.94112.objecter handle_osd_map got epochs [343,343] > 0 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 3 client.94112.objecter handle_osd_map decoding full epoch 343 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8f7fe640 10 monclient(hunting): handle_auth_done global_id 84368 payload 824 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8f7fe640 1 -- 172.21.15.98:0/1554781883 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b900a91f0 con 0x7f2b90128880 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 20 client.94112.objecter dump_active .. 0 homeless 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b94a17640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2b7c03fdb0 0x7f2b7c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b94a17640 10 monclient: get_auth_request con 0x7f2b7c03fdb0 auth_method 0 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 10 client.94112.objecter ms_handle_connect 0x7f2b90128880 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 10 client.94112.objecter resend_mon_ops 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8f7fe640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90128880 0x7f2b90125c90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b900a5620 tx=0x7f2b84011990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b840044a0 con 0x7f2b90128880 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.206+0000 7f2b8cff9640 10 monclient: dump: 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.230 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.207+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2b84004030 con 0x7f2b90128880 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.207+0000 7f2b8cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.207+0000 7f2b8cff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.207+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90128880 msgr2=0x7f2b90125c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.207+0000 7f2b8cff9640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90128880 0x7f2b90125c90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b900a5620 tx=0x7f2b84011990 comp rx=0 tx=0).stop 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.207+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90128880 msgr2=0x7f2b90125c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:48.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.207+0000 7f2b94a17640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2b7c03fdb0 0x7f2b7c042270 secure :-1 s=READY pgs=4163 cs=0 l=1 rev1=1 crypto rx=0x7f2b88035000 tx=0x7f2b88030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.207+0000 7f2b8cff9640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90128880 0x7f2b90125c90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.207+0000 7f2b8cff9640 10 client.94112.objecter ms_handle_connect 0x7f2b7c03fdb0 2024-06-11T19:17:48.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b9647a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:48.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b9647a640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b9002cb40 0x7f2b9002cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b9647a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b9647a640 1 -- 172.21.15.98:0/1554781883 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2b90050830 con 0x7f2b9002cb40 2024-06-11T19:17:48.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8f7fe640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b9002cb40 0x7f2b9002cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8f7fe640 10 monclient: get_auth_request con 0x7f2b9002cb40 auth_method 0 2024-06-11T19:17:48.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8f7fe640 10 monclient(hunting): handle_auth_done global_id 84377 payload 824 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8f7fe640 1 -- 172.21.15.98:0/1554781883 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2b901387f0 con 0x7f2b9002cb40 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8cff9640 10 client.94112.objecter ms_handle_connect 0x7f2b9002cb40 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8cff9640 10 client.94112.objecter resend_mon_ops 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.213+0000 7f2b8f7fe640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b9002cb40 0x7f2b9002cf50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2b900a5620 tx=0x7f2b840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b840064b0 con 0x7f2b9002cb40 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b8cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b8cff9640 10 monclient: dump: 2024-06-11T19:17:48.235 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f2b84003610 con 0x7f2b9002cb40 2024-06-11T19:17:48.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b8cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:48.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b8cff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:48.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b9002cb40 msgr2=0x7f2b9002cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b8cff9640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b9002cb40 0x7f2b9002cf50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2b900a5620 tx=0x7f2b840046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b8cff9640 1 -- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b9002cb40 msgr2=0x7f2b9002cf50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:48.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b8cff9640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b9002cb40 0x7f2b9002cf50 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2b900a5620 tx=0x7f2b840046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.237 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:48.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:48.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:48.237 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:48.237 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:48.238 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 139988, 2024-06-11T19:17:48.238 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:48.238 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:48.238 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:48.238 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:48.238 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:48.238 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:48.238 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:48.238 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:48.238 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:48.238 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:48.238 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.239 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.240 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:48.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.242 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:48.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.251 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.254 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:48.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:48.257 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.257 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.257 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:48.257 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.259 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:48.260 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.260 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.260 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:48.260 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:48.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 10 librados: watch_flush enter 2024-06-11T19:17:48.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 10 librados: watch_flush exit 2024-06-11T19:17:48.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 20 client.94112.objecter shutdown clearing up homeless session... 2024-06-11T19:17:48.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 10 client.94112.objecter successfully canceled tick 2024-06-11T19:17:48.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 1 -- 172.21.15.98:0/1554781883 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2b7c03fdb0 msgr2=0x7f2b7c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2b7c03fdb0 0x7f2b7c042270 secure :-1 s=READY pgs=4163 cs=0 l=1 rev1=1 crypto rx=0x7f2b88035000 tx=0x7f2b88030000 comp rx=0 tx=0).stop 2024-06-11T19:17:48.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 10 monclient: shutdown 2024-06-11T19:17:48.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:48.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 1 -- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b900a2530 msgr2=0x7f2b901cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 1 --2- 172.21.15.98:0/1554781883 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b900a2530 0x7f2b901cda70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000ec10 tx=0x7f2b8000c520 comp rx=0 tx=0).stop 2024-06-11T19:17:48.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b94a17640 1 -- 172.21.15.98:0/1554781883 reap_dead start 2024-06-11T19:17:48.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b94a17640 1 -- 172.21.15.98:0/1554781883 reap_dead start 2024-06-11T19:17:48.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 1 -- 172.21.15.98:0/1554781883 shutdown_connections 2024-06-11T19:17:48.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 1 -- 172.21.15.98:0/1554781883 >> 172.21.15.98:0/1554781883 conn(0x7f2b900ab850 msgr2=0x7f2b9009c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:48.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 1 -- 172.21.15.98:0/1554781883 shutdown_connections 2024-06-11T19:17:48.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 1 -- 172.21.15.98:0/1554781883 wait complete. 2024-06-11T19:17:48.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.214+0000 7f2b9647a640 1 librados: shutdown 2024-06-11T19:17:48.263 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:48.263 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:17:48.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:17:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.364+0000 7f25b1592640 10 monclient: get_monmap_and_config 2024-06-11T19:17:48.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.364+0000 7f25b1592640 10 monclient: build_initial_monmap 2024-06-11T19:17:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.364+0000 7f25b1592640 10 monclient: monmap: 2024-06-11T19:17:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:48.365793+0000 2024-06-11T19:17:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:48.365793+0000 2024-06-11T19:17:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:48.365 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:48.366 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:48.366 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 1 Processor -- start 2024-06-11T19:17:48.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 1 -- start start 2024-06-11T19:17:48.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 10 monclient: init 2024-06-11T19:17:48.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:48.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 10 monclient: _add_conns ranks=[3,9,7,10,8,6,1,2,0,4,5] 2024-06-11T19:17:48.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25ac0a4920 0x7f25ac0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 10 monclient(hunting): picked mon.noname-g con 0x7f25ac0a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f25ac0a5240 0x7f25ac0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 10 monclient(hunting): picked mon.noname-a con 0x7f25ac0a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f25ac0a2530 0x7f25ac0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 10 monclient(hunting): picked mon.noname-i con 0x7f25ac0a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f25ac0a7140 con 0x7f25ac0a5240 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f25ac0a2b50 con 0x7f25ac0a4920 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f25ac0a2e30 con 0x7f25ac0a2530 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.365+0000 7f25b1592640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:48.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25b1592640 10 monclient(hunting): authenticate will time out at 1278.173009s 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f25ac0a5240 0x7f25ac0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f25ac0a5240 0x7f25ac0a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56608/0 (socket says 172.21.15.98:56608) 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 1 -- 172.21.15.98:0/4278910779 learned_addr learned my addr 172.21.15.98:0/4278910779 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 10 monclient(hunting): get_auth_request con 0x7f25ac0a5240 auth_method 0 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25ab7fe640 1 --2- 172.21.15.98:0/4278910779 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f25ac0a2530 0x7f25ac0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25ab7fe640 10 monclient(hunting): get_auth_request con 0x7f25ac0a2530 auth_method 0 2024-06-11T19:17:48.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25ab7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25aaffd640 1 --2- 172.21.15.98:0/4278910779 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25ac0a4920 0x7f25ac0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25aaffd640 10 monclient(hunting): get_auth_request con 0x7f25ac0a4920 auth_method 0 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25aaffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 10 monclient(hunting): handle_auth_done global_id 64944 payload 274 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 1 -- 172.21.15.98:0/4278910779 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f25ac0a2530 msgr2=0x7f25ac0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 1 --2- 172.21.15.98:0/4278910779 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f25ac0a2530 0x7f25ac0a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 1 -- 172.21.15.98:0/4278910779 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25ac0a4920 msgr2=0x7f25ac0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 1 --2- 172.21.15.98:0/4278910779 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25ac0a4920 0x7f25ac0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 1 monclient: found mon.noname-a 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.366+0000 7f25a3fff640 1 -- 172.21.15.98:0/4278910779 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25ac0a3110 con 0x7f25ac0a5240 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25a3fff640 1 --2- 172.21.15.98:0/4278910779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25ac0a5240 0x7f25ac0a59a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2594009870 tx=0x7f259402b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d8ef3b81224549 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 1 -- 172.21.15.98:0/4278910779 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2594038070 con 0x7f25ac0a5240 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 10 monclient: dump: 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 10 monclient: _finish_auth 0 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 1 -- 172.21.15.98:0/4278910779 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2594031990 con 0x7f25ac0a5240 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 5 monclient: authenticate success, global_id 64944 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 1 -- 172.21.15.98:0/4278910779 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25940323b0 con 0x7f25ac0a5240 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 4 set_mon_vals no callback set 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25aa7fc640 10 monclient: dump: 2024-06-11T19:17:48.385 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.386 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 10 monclient: shutdown 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 1 -- 172.21.15.98:0/4278910779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25ac0a5240 msgr2=0x7f25ac0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 1 --2- 172.21.15.98:0/4278910779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25ac0a5240 0x7f25ac0a59a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2594009870 tx=0x7f259402b7c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 1 -- 172.21.15.98:0/4278910779 shutdown_connections 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 1 --2- 172.21.15.98:0/4278910779 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f25ac0a2530 0x7f25ac0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 1 --2- 172.21.15.98:0/4278910779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25ac0a5240 0x7f25ac0a59a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 1 --2- 172.21.15.98:0/4278910779 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25ac0a4920 0x7f25ac0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 1 -- 172.21.15.98:0/4278910779 >> 172.21.15.98:0/4278910779 conn(0x7f25ac0ab850 msgr2=0x7f25ac0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 1 -- 172.21.15.98:0/4278910779 shutdown_connections 2024-06-11T19:17:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.367+0000 7f25b1592640 1 -- 172.21.15.98:0/4278910779 wait complete. 2024-06-11T19:17:48.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.368+0000 7f25b1592640 10 monclient: build_initial_monmap 2024-06-11T19:17:48.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.368+0000 7f25b1592640 10 monclient: monmap: 2024-06-11T19:17:48.388 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:48.388 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:48.388 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:48.388 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:48.388 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:48.388 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:48.388 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:48.388 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:48.388 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:48.389 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:48.389 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:48.389 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:48.389 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:48.389 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:48.389 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:48.389 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.368+0000 7f25b1592640 1 librados: starting msgr at 2024-06-11T19:17:48.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.368+0000 7f25b1592640 1 librados: starting objecter 2024-06-11T19:17:48.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.368+0000 7f25b1592640 1 Processor -- start 2024-06-11T19:17:48.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.368+0000 7f25b1592640 1 -- start start 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.368+0000 7f25b1592640 1 librados: setting wanted keys 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.368+0000 7f25b1592640 1 librados: calling monclient init 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.368+0000 7f25b1592640 10 monclient: init 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 10 monclient: _add_conns ranks=[8,6,5,4,1,3,7,2,0] 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25ac0a2530 0x7f25ac137640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 10 monclient(hunting): picked mon.noname-e con 0x7f25ac0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25ac0a4920 0x7f25ac137b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 10 monclient(hunting): picked mon.noname-d con 0x7f25ac0a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f25ac0a5240 0x7f25ac1230e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 10 monclient(hunting): picked mon.noname-h con 0x7f25ac0a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f25ac0a6da0 con 0x7f25ac0a5240 2024-06-11T19:17:48.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f25ac138e50 con 0x7f25ac0a4920 2024-06-11T19:17:48.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f25ac0a6420 con 0x7f25ac0a2530 2024-06-11T19:17:48.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:48.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25b1592640 10 monclient(hunting): authenticate will time out at 1278.176314s 2024-06-11T19:17:48.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25ac0a2530 0x7f25ac137640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25ac0a2530 0x7f25ac137640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39012/0 (socket says 172.21.15.98:39012) 2024-06-11T19:17:48.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 1 -- 172.21.15.98:0/1286009950 learned_addr learned my addr 172.21.15.98:0/1286009950 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:48.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 10 monclient(hunting): get_auth_request con 0x7f25ac0a2530 auth_method 0 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25a3fff640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25ac0a4920 0x7f25ac137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25a3fff640 10 monclient(hunting): get_auth_request con 0x7f25ac0a4920 auth_method 0 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25a3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25ab7fe640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f25ac0a5240 0x7f25ac1230e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25ab7fe640 10 monclient(hunting): get_auth_request con 0x7f25ac0a5240 auth_method 0 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25ab7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 10 monclient(hunting): handle_auth_done global_id 94121 payload 824 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 1 -- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25ac0a4920 msgr2=0x7f25ac137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25ac0a4920 0x7f25ac137b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 1 -- 172.21.15.98:0/1286009950 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f25ac0a5240 msgr2=0x7f25ac1230e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f25ac0a5240 0x7f25ac1230e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 10 monclient: _finish_hunting 0 2024-06-11T19:17:48.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 1 monclient: found mon.noname-e 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 1 -- 172.21.15.98:0/1286009950 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25ac126d40 con 0x7f25ac0a2530 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25a3fff640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25ac0a4920 0x7f25ac137b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25aaffd640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25ac0a2530 0x7f25ac137640 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f259c00d800 tx=0x7f259c00dcc0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25a37fe640 10 client.?.objecter ms_handle_connect 0x7f25ac0a2530 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.369+0000 7f25a37fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f259c00bb30 con 0x7f25ac0a2530 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 10 monclient: dump: 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.394 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.395 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 10 monclient: _finish_auth 0 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 10 monclient: _check_auth_tickets 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f259c019070 con 0x7f25ac0a2530 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f259c00bd10 con 0x7f25ac0a2530 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a9ffb640 4 set_mon_vals no callback set 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 10 monclient: dump: 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.396 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.397 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 5 monclient: authenticate success, global_id 94121 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 10 monclient: _renew_subs 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 1 -- 172.21.15.98:0/1286009950 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25ac0a9130 con 0x7f25ac0a2530 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 10 client.94121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 10 monclient: _renew_subs 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 1 -- 172.21.15.98:0/1286009950 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f25ac124cc0 con 0x7f25ac0a2530 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 1 librados: init done 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123620 0x7f25ac123a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25b1592640 1 -- 172.21.15.98:0/1286009950 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f25ac050830 con 0x7f25ac123620 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.370+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f259c00b6c0 con 0x7f25ac0a2530 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a3fff640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123620 0x7f25ac123a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a3fff640 10 monclient: get_auth_request con 0x7f25ac123620 auth_method 0 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f258c03fdb0 0x7f258c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(343..343 src has 1..343) ==== 6230+0+0 (secure 0 0 0) 0x7f259c00f040 con 0x7f25ac0a2530 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 10 client.94121.objecter ms_dispatch 0x7f25ac000b90 osd_map(343..343 src has 1..343) 2024-06-11T19:17:48.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 3 client.94121.objecter handle_osd_map got epochs [343,343] > 0 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 3 client.94121.objecter handle_osd_map decoding full epoch 343 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a3fff640 10 monclient(hunting): handle_auth_done global_id 23926 payload 824 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a3fff640 1 -- 172.21.15.98:0/1286009950 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25ac0a7360 con 0x7f25ac123620 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 20 client.94121.objecter dump_active .. 0 homeless 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25ab7fe640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f258c03fdb0 0x7f258c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25ab7fe640 10 monclient: get_auth_request con 0x7f258c03fdb0 auth_method 0 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a3fff640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123620 0x7f25ac123a00 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0a8420 tx=0x7f25940046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 10 client.94121.objecter ms_handle_connect 0x7f25ac123620 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 10 client.94121.objecter resend_mon_ops 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2594038040 con 0x7f25ac123620 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 10 monclient: dump: 2024-06-11T19:17:48.400 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.401 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 10 client.94121.objecter ms_handle_connect 0x7f258c03fdb0 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25ab7fe640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f258c03fdb0 0x7f258c042270 secure :-1 s=READY pgs=4164 cs=0 l=1 rev1=1 crypto rx=0x7f25ac124840 tx=0x7f25a400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2594031e60 con 0x7f25ac123620 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123620 msgr2=0x7f25ac123a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123620 0x7f25ac123a00 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0a8420 tx=0x7f25940046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123620 msgr2=0x7f25ac123a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.371+0000 7f25a37fe640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123620 0x7f25ac123a00 secure :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0a8420 tx=0x7f25940046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25b1592640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25b1592640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123f40 0x7f25ac09e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25b1592640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25b1592640 1 -- 172.21.15.98:0/1286009950 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f25ac069740 con 0x7f25ac123f40 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a3fff640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123f40 0x7f25ac09e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a3fff640 10 monclient: get_auth_request con 0x7f25ac123f40 auth_method 0 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a3fff640 10 monclient(hunting): handle_auth_done global_id 23935 payload 824 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a3fff640 1 -- 172.21.15.98:0/1286009950 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f25ac1255a0 con 0x7f25ac123f40 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a37fe640 10 client.94121.objecter ms_handle_connect 0x7f25ac123f40 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a37fe640 10 client.94121.objecter resend_mon_ops 2024-06-11T19:17:48.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a3fff640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123f40 0x7f25ac09e9f0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f25940097c0 tx=0x7f2594008030 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2594038050 con 0x7f25ac123f40 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.378+0000 7f25a37fe640 10 monclient: dump: 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.404 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f259404f070 con 0x7f25ac123f40 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25a37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25a37fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123f40 msgr2=0x7f25ac09e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25a37fe640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123f40 0x7f25ac09e9f0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f25940097c0 tx=0x7f2594008030 comp rx=0 tx=0).stop 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25a37fe640 1 -- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123f40 msgr2=0x7f25ac09e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:48.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25a37fe640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25ac123f40 0x7f25ac09e9f0 secure :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f25940097c0 tx=0x7f2594008030 comp rx=0 tx=0).stop 2024-06-11T19:17:48.406 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 518596, 2024-06-11T19:17:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:17:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:17:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:17:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:48.430 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 10 librados: watch_flush enter 2024-06-11T19:17:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 10 librados: watch_flush exit 2024-06-11T19:17:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 20 client.94121.objecter shutdown clearing up homeless session... 2024-06-11T19:17:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 10 client.94121.objecter successfully canceled tick 2024-06-11T19:17:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 1 -- 172.21.15.98:0/1286009950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f258c03fdb0 msgr2=0x7f258c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f258c03fdb0 0x7f258c042270 secure :-1 s=READY pgs=4164 cs=0 l=1 rev1=1 crypto rx=0x7f25ac124840 tx=0x7f25a400b040 comp rx=0 tx=0).stop 2024-06-11T19:17:48.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 10 monclient: shutdown 2024-06-11T19:17:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 1 -- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25ac0a2530 msgr2=0x7f25ac137640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25ab7fe640 1 -- 172.21.15.98:0/1286009950 reap_dead start 2024-06-11T19:17:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25ac0a2530 0x7f25ac137640 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f259c00d800 tx=0x7f259c00dcc0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 1 -- 172.21.15.98:0/1286009950 shutdown_connections 2024-06-11T19:17:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 1 --2- 172.21.15.98:0/1286009950 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25ac0a2530 0x7f25ac137640 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 1 -- 172.21.15.98:0/1286009950 >> 172.21.15.98:0/1286009950 conn(0x7f25ac0ab850 msgr2=0x7f25ac1be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 1 -- 172.21.15.98:0/1286009950 shutdown_connections 2024-06-11T19:17:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 1 -- 172.21.15.98:0/1286009950 wait complete. 2024-06-11T19:17:48.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.379+0000 7f25b1592640 1 librados: shutdown 2024-06-11T19:17:48.434 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:48.434 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:17:48.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:17:48.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.534+0000 7fab58820640 10 monclient: get_monmap_and_config 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.534+0000 7fab58820640 10 monclient: build_initial_monmap 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.534+0000 7fab58820640 10 monclient: monmap: 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:48.535584+0000 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:48.535584+0000 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:48.535 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:48.535 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 1 Processor -- start 2024-06-11T19:17:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 1 -- start start 2024-06-11T19:17:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 10 monclient: init 2024-06-11T19:17:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 10 monclient: _add_conns ranks=[1,8,9,4,6,10,7,0,2,5,3] 2024-06-11T19:17:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fab540a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:17:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab540a2530 0x7fab5413dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 10 monclient(hunting): picked mon.noname-e con 0x7fab540a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:17:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab5413e1a0 0x7fab54140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 10 monclient(hunting): picked mon.noname-a con 0x7fab5413e1a0 addr v2:172.21.15.98:3300/0 2024-06-11T19:17:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab540a7140 con 0x7fab540a4920 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fab540a2b50 con 0x7fab5413e1a0 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fab540a2e30 con 0x7fab540a2530 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab58820640 10 monclient(hunting): authenticate will time out at 1278.342708s 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab49d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab540a2530 0x7fab5413dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab52d9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab5413e1a0 0x7fab54140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab49d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab540a2530 0x7fab5413dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39018/0 (socket says 172.21.15.98:39018) 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab49d9b640 1 -- 172.21.15.98:0/1389970186 learned_addr learned my addr 172.21.15.98:0/1389970186 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab49d9b640 10 monclient(hunting): get_auth_request con 0x7fab540a2530 auth_method 0 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab49d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab52d9d640 10 monclient(hunting): get_auth_request con 0x7fab5413e1a0 auth_method 0 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab52d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab52d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab52d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.535+0000 7fab52d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab49d9b640 10 monclient(hunting): handle_auth_done global_id 94130 payload 274 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab49d9b640 1 -- 172.21.15.98:0/1389970186 >> v2:172.21.15.98:3300/0 conn(0x7fab5413e1a0 msgr2=0x7fab54140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab49d9b640 1 --2- 172.21.15.98:0/1389970186 >> v2:172.21.15.98:3300/0 conn(0x7fab5413e1a0 0x7fab54140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab49d9b640 1 -- 172.21.15.98:0/1389970186 >> v1:172.21.15.117:6789/0 conn(0x7fab540a4920 legacy=0x7fab540a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab49d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab49d9b640 1 monclient: found mon.noname-e 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab49d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab49d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab49d9b640 1 -- 172.21.15.98:0/1389970186 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab540a3110 con 0x7fab540a2530 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab49d9b640 1 --2- 172.21.15.98:0/1389970186 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab540a2530 0x7fab5413dc60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009870 tx=0x7fab3c02b7c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=3b5a0cb2bc8804ec server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab51d9b640 1 -- 172.21.15.98:0/1389970186 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab3c038070 con 0x7fab540a2530 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab51d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab51d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab51d9b640 10 monclient: dump: 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab51d9b640 10 monclient: _finish_auth 0 2024-06-11T19:17:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab51d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:17:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab51d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab51d9b640 1 -- 172.21.15.98:0/1389970186 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab3c031990 con 0x7fab540a2530 2024-06-11T19:17:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab58820640 5 monclient: authenticate success, global_id 94130 2024-06-11T19:17:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.536+0000 7fab58820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab51d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab58820640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab51d9b640 1 -- 172.21.15.98:0/1389970186 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab3c0323b0 con 0x7fab540a2530 2024-06-11T19:17:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab51d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab58820640 4 set_mon_vals no callback set 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab51d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab51d9b640 10 monclient: dump: 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab58820640 10 monclient: shutdown 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab58820640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab58820640 1 -- 172.21.15.98:0/1389970186 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab540a2530 msgr2=0x7fab5413dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab58820640 1 --2- 172.21.15.98:0/1389970186 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab540a2530 0x7fab5413dc60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009870 tx=0x7fab3c02b7c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab58820640 1 -- 172.21.15.98:0/1389970186 shutdown_connections 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab58820640 1 --2- 172.21.15.98:0/1389970186 >> v2:172.21.15.98:3300/0 conn(0x7fab5413e1a0 0x7fab54140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab58820640 1 --2- 172.21.15.98:0/1389970186 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab540a2530 0x7fab5413dc60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab58820640 1 -- 172.21.15.98:0/1389970186 >> 172.21.15.98:0/1389970186 conn(0x7fab540ab850 msgr2=0x7fab540a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab58820640 1 -- 172.21.15.98:0/1389970186 shutdown_connections 2024-06-11T19:17:48.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.537+0000 7fab58820640 1 -- 172.21.15.98:0/1389970186 wait complete. 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.538+0000 7fab58820640 10 monclient: build_initial_monmap 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.538+0000 7fab58820640 10 monclient: monmap: 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:48.553 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.538+0000 7fab58820640 1 librados: starting msgr at 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.538+0000 7fab58820640 1 librados: starting objecter 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.538+0000 7fab58820640 1 Processor -- start 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.538+0000 7fab58820640 1 -- start start 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.538+0000 7fab58820640 1 librados: setting wanted keys 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.538+0000 7fab58820640 1 librados: calling monclient init 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.538+0000 7fab58820640 10 monclient: init 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 10 monclient: _add_conns ranks=[0,3,6,8,5,1,2,4,7] 2024-06-11T19:17:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab540a2530 0x7fab541cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 10 monclient(hunting): picked mon.noname-a con 0x7fab540a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab540a4920 0x7fab54135a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 10 monclient(hunting): picked mon.noname-g con 0x7fab540a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5413e1a0 0x7fab5409ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 10 monclient(hunting): picked mon.noname-d con 0x7fab5413e1a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab54138e50 con 0x7fab540a2530 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fab540a84b0 con 0x7fab540a4920 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fab540a6da0 con 0x7fab5413e1a0 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 10 monclient(hunting): authenticate will time out at 1278.346206s 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab5259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab540a2530 0x7fab541cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab5259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab540a2530 0x7fab541cd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56618/0 (socket says 172.21.15.98:56618) 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab5259c640 1 -- 172.21.15.98:0/2028906061 learned_addr learned my addr 172.21.15.98:0/2028906061 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab5259c640 10 monclient(hunting): get_auth_request con 0x7fab540a2530 auth_method 0 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab5259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab5259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5413e1a0 0x7fab5409ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 10 monclient(hunting): get_auth_request con 0x7fab5413e1a0 auth_method 0 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab49d9b640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab540a4920 0x7fab54135a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab49d9b640 10 monclient(hunting): get_auth_request con 0x7fab540a4920 auth_method 0 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab49d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 10 monclient(hunting): handle_auth_done global_id 55059 payload 824 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 1 -- 172.21.15.98:0/2028906061 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab540a4920 msgr2=0x7fab54135a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab540a4920 0x7fab54135a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 1 -- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab540a2530 msgr2=0x7fab541cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab540a2530 0x7fab541cd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 1 monclient: found mon.noname-d 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 1 -- 172.21.15.98:0/2028906061 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab54121e70 con 0x7fab5413e1a0 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab49d9b640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab540a4920 0x7fab54135a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab5259c640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab540a2530 0x7fab541cd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab52d9d640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5413e1a0 0x7fab5409ec10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00ec00 tx=0x7fab4c00c490 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 10 client.?.objecter ms_handle_connect 0x7fab5413e1a0 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab4c018020 con 0x7fab5413e1a0 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 10 monclient: dump: 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.559 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 5 monclient: authenticate success, global_id 55059 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab4c014480 con 0x7fab5413e1a0 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab4c014aa0 con 0x7fab5413e1a0 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 10 monclient: _renew_subs 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab58820640 1 -- 172.21.15.98:0/2028906061 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5413da40 con 0x7fab5413e1a0 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.539+0000 7fab5159a640 4 set_mon_vals no callback set 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab4b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab58820640 10 client.55059.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab4b7fe640 10 monclient: dump: 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.561 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab58820640 10 monclient: _renew_subs 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab58820640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab58820640 1 -- 172.21.15.98:0/2028906061 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fab540a0680 con 0x7fab5413e1a0 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab58820640 1 librados: init done 2024-06-11T19:17:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab58820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab58820640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5409f150 0x7fab5409f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab58820640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab58820640 1 -- 172.21.15.98:0/2028906061 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fab54050830 con 0x7fab5409f150 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab5259c640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5409f150 0x7fab5409f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fab4c01b020 con 0x7fab5413e1a0 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab5259c640 10 monclient: get_auth_request con 0x7fab5409f150 auth_method 0 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab5259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab5259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.540+0000 7fab5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab5259c640 10 monclient(hunting): handle_auth_done global_id 55068 payload 824 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab5259c640 1 -- 172.21.15.98:0/2028906061 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab5409fa70 con 0x7fab5409f150 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fab3403fdb0 0x7fab34042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(343..343 src has 1..343) ==== 6230+0+0 (secure 0 0 0) 0x7fab4c052670 con 0x7fab5413e1a0 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 10 client.55059.objecter ms_dispatch 0x7fab54000b90 osd_map(343..343 src has 1..343) 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 3 client.55059.objecter handle_osd_map got epochs [343,343] > 0 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 3 client.55059.objecter handle_osd_map decoding full epoch 343 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab5259c640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5409f150 0x7fab5409f530 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fab4400fb10 tx=0x7fab44004580 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 20 client.55059.objecter dump_active .. 0 homeless 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 10 client.55059.objecter ms_handle_connect 0x7fab5409f150 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 10 client.55059.objecter resend_mon_ops 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab49d9b640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fab3403fdb0 0x7fab34042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab49d9b640 10 monclient: get_auth_request con 0x7fab3403fdb0 auth_method 0 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab44018020 con 0x7fab5409f150 2024-06-11T19:17:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 10 monclient: dump: 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fab4400db60 con 0x7fab5409f150 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5409f150 msgr2=0x7fab5409f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5409f150 0x7fab5409f530 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fab4400fb10 tx=0x7fab44004580 comp rx=0 tx=0).stop 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5409f150 msgr2=0x7fab5409f530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5409f150 0x7fab5409f530 secure :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fab4400fb10 tx=0x7fab44004580 comp rx=0 tx=0).stop 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab4b7fe640 10 client.55059.objecter ms_handle_connect 0x7fab3403fdb0 2024-06-11T19:17:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.541+0000 7fab49d9b640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fab3403fdb0 0x7fab34042270 secure :-1 s=READY pgs=4165 cs=0 l=1 rev1=1 crypto rx=0x7fab3c035000 tx=0x7fab3c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.548+0000 7fab58820640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:48.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.548+0000 7fab58820640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5413bd30 0x7fab5413c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.548+0000 7fab58820640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.548+0000 7fab58820640 1 -- 172.21.15.98:0/2028906061 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fab541221c0 con 0x7fab5413bd30 2024-06-11T19:17:48.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.548+0000 7fab5259c640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5413bd30 0x7fab5413c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.548+0000 7fab5259c640 10 monclient: get_auth_request con 0x7fab5413bd30 auth_method 0 2024-06-11T19:17:48.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.548+0000 7fab5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.548+0000 7fab5259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.548+0000 7fab5259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.548+0000 7fab5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.548+0000 7fab5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.549+0000 7fab5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.549+0000 7fab5259c640 10 monclient(hunting): handle_auth_done global_id 55077 payload 824 2024-06-11T19:17:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.549+0000 7fab5259c640 1 -- 172.21.15.98:0/2028906061 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fab540a42b0 con 0x7fab5413bd30 2024-06-11T19:17:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.549+0000 7fab4b7fe640 10 client.55059.objecter ms_handle_connect 0x7fab5413bd30 2024-06-11T19:17:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.549+0000 7fab4b7fe640 10 client.55059.objecter resend_mon_ops 2024-06-11T19:17:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.549+0000 7fab5259c640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5413bd30 0x7fab5413c140 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fab44019000 tx=0x7fab44003760 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.552+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab44018040 con 0x7fab5413bd30 2024-06-11T19:17:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.552+0000 7fab4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.552+0000 7fab4b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.552+0000 7fab4b7fe640 10 monclient: dump: 2024-06-11T19:17:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.571 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.572 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.552+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fab44014d50 con 0x7fab5413bd30 2024-06-11T19:17:48.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.552+0000 7fab4b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:48.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.552+0000 7fab4b7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:48.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.552+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5413bd30 msgr2=0x7fab5413c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.552+0000 7fab4b7fe640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5413bd30 0x7fab5413c140 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fab44019000 tx=0x7fab44003760 comp rx=0 tx=0).stop 2024-06-11T19:17:48.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.552+0000 7fab4b7fe640 1 -- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5413bd30 msgr2=0x7fab5413c140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:48.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.552+0000 7fab4b7fe640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5413bd30 0x7fab5413c140 secure :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fab44019000 tx=0x7fab44003760 comp rx=0 tx=0).stop 2024-06-11T19:17:48.572 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 381308, 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-06-11T19:17:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:17:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:48.596 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab58820640 10 librados: watch_flush enter 2024-06-11T19:17:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab58820640 10 librados: watch_flush exit 2024-06-11T19:17:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab58820640 20 client.55059.objecter shutdown clearing up homeless session... 2024-06-11T19:17:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab58820640 10 client.55059.objecter successfully canceled tick 2024-06-11T19:17:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab58820640 1 -- 172.21.15.98:0/2028906061 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fab3403fdb0 msgr2=0x7fab34042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab58820640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fab3403fdb0 0x7fab34042270 secure :-1 s=READY pgs=4165 cs=0 l=1 rev1=1 crypto rx=0x7fab3c035000 tx=0x7fab3c030000 comp rx=0 tx=0).stop 2024-06-11T19:17:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab58820640 10 monclient: shutdown 2024-06-11T19:17:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab58820640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab58820640 1 -- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5413e1a0 msgr2=0x7fab5409ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab58820640 1 --2- 172.21.15.98:0/2028906061 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab5413e1a0 0x7fab5409ec10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00ec00 tx=0x7fab4c00c490 comp rx=0 tx=0).stop 2024-06-11T19:17:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab52d9d640 1 -- 172.21.15.98:0/2028906061 reap_dead start 2024-06-11T19:17:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab52d9d640 1 -- 172.21.15.98:0/2028906061 reap_dead start 2024-06-11T19:17:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab58820640 1 -- 172.21.15.98:0/2028906061 shutdown_connections 2024-06-11T19:17:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.553+0000 7fab58820640 1 -- 172.21.15.98:0/2028906061 >> 172.21.15.98:0/2028906061 conn(0x7fab540ab850 msgr2=0x7fab540a9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.554+0000 7fab58820640 1 -- 172.21.15.98:0/2028906061 shutdown_connections 2024-06-11T19:17:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.554+0000 7fab58820640 1 -- 172.21.15.98:0/2028906061 wait complete. 2024-06-11T19:17:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.554+0000 7fab58820640 1 librados: shutdown 2024-06-11T19:17:48.601 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:48.601 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:17:48.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:17:48.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.702+0000 7f17ca5f0640 10 monclient: get_monmap_and_config 2024-06-11T19:17:48.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.702+0000 7f17ca5f0640 10 monclient: build_initial_monmap 2024-06-11T19:17:48.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.702+0000 7f17ca5f0640 10 monclient: monmap: 2024-06-11T19:17:48.702 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:48.702 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.702 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:48.703878+0000 2024-06-11T19:17:48.702 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:48.703878+0000 2024-06-11T19:17:48.702 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:48.702 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.702 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:48.703 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.703+0000 7f17ca5f0640 1 Processor -- start 2024-06-11T19:17:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.703+0000 7f17ca5f0640 1 -- start start 2024-06-11T19:17:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.703+0000 7f17ca5f0640 10 monclient: init 2024-06-11T19:17:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 10 monclient: _add_conns ranks=[8,7,6,1,9,0,2,4,5,10,3] 2024-06-11T19:17:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c40a4920 0x7f17c40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 10 monclient(hunting): picked mon.noname-e con 0x7f17c40a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:17:48.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f17c40a5240 0x7f17c40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 10 monclient(hunting): picked mon.noname-i con 0x7f17c40a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:17:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c40a2530 0x7f17c40a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 10 monclient(hunting): picked mon.noname-d con 0x7f17c40a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f17c40a7140 con 0x7f17c40a2530 2024-06-11T19:17:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f17c40a2b50 con 0x7f17c40a5240 2024-06-11T19:17:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f17c40a2e30 con 0x7f17c40a4920 2024-06-11T19:17:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17ca5f0640 10 monclient(hunting): authenticate will time out at 1278.511164s 2024-06-11T19:17:48.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c8b8d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c40a2530 0x7f17c40a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c40a4920 0x7f17c40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c40a4920 0x7f17c40a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39030/0 (socket says 172.21.15.98:39030) 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c8b8d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c40a2530 0x7f17c40a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:33100/0 (socket says 172.21.15.98:33100) 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 1 -- 172.21.15.98:0/781076191 learned_addr learned my addr 172.21.15.98:0/781076191 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 10 monclient(hunting): get_auth_request con 0x7f17c40a4920 auth_method 0 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c8b8d640 10 monclient(hunting): get_auth_request con 0x7f17c40a2530 auth_method 0 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c8b8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c8b8d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c8b8d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c8b8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c8b8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c8b8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c37fe640 1 --2- 172.21.15.98:0/781076191 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f17c40a5240 0x7f17c40a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 10 monclient(hunting): handle_auth_done global_id 94139 payload 274 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 1 -- 172.21.15.98:0/781076191 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f17c40a5240 msgr2=0x7f17c40a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 1 --2- 172.21.15.98:0/781076191 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f17c40a5240 0x7f17c40a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 1 -- 172.21.15.98:0/781076191 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c40a2530 msgr2=0x7f17c40a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 1 --2- 172.21.15.98:0/781076191 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c40a2530 0x7f17c40a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 1 monclient: found mon.noname-e 2024-06-11T19:17:48.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c3fff640 1 -- 172.21.15.98:0/781076191 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c40a3110 con 0x7f17c40a4920 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.704+0000 7f17c8b8d640 1 --2- 172.21.15.98:0/781076191 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c40a2530 0x7f17c40a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c3fff640 1 --2- 172.21.15.98:0/781076191 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c40a4920 0x7f17c40a4d00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f17b40098a0 tx=0x7f17b402b7a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=f8168e30c14bfdc7 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 1 -- 172.21.15.98:0/781076191 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17b4038070 con 0x7f17c40a4920 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 10 monclient: dump: 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.717 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 10 monclient: _finish_auth 0 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:17:48.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 1 -- 172.21.15.98:0/781076191 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17b4031970 con 0x7f17c40a4920 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 1 -- 172.21.15.98:0/781076191 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17b4031ba0 con 0x7f17c40a4920 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17ca5f0640 5 monclient: authenticate success, global_id 94139 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17ca5f0640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17ca5f0640 4 set_mon_vals no callback set 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.705+0000 7f17c2ffd640 10 monclient: dump: 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.719 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 10 monclient: shutdown 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:48.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 1 -- 172.21.15.98:0/781076191 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c40a4920 msgr2=0x7f17c40a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 1 --2- 172.21.15.98:0/781076191 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c40a4920 0x7f17c40a4d00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f17b40098a0 tx=0x7f17b402b7a0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 1 -- 172.21.15.98:0/781076191 shutdown_connections 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 1 --2- 172.21.15.98:0/781076191 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c40a2530 0x7f17c40a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 1 --2- 172.21.15.98:0/781076191 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f17c40a5240 0x7f17c40a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 1 --2- 172.21.15.98:0/781076191 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c40a4920 0x7f17c40a4d00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 1 -- 172.21.15.98:0/781076191 >> 172.21.15.98:0/781076191 conn(0x7f17c40ab850 msgr2=0x7f17c40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 1 -- 172.21.15.98:0/781076191 shutdown_connections 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 1 -- 172.21.15.98:0/781076191 wait complete. 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 10 monclient: build_initial_monmap 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 10 monclient: monmap: 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:48.721 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:48.722 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 1 librados: starting msgr at 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.706+0000 7f17ca5f0640 1 librados: starting objecter 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 1 Processor -- start 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 1 -- start start 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 1 librados: setting wanted keys 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 1 librados: calling monclient init 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 10 monclient: init 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 10 monclient: _add_conns ranks=[4,2,6,3,7,1,0,8,5] 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17c40a2530 0x7f17c4137740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 10 monclient(hunting): picked mon.noname-c con 0x7f17c40a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c40a4920 0x7f17c4137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 10 monclient(hunting): picked mon.noname-b con 0x7f17c40a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:48.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c40a5240 0x7f17c41232a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 10 monclient(hunting): picked mon.noname-d con 0x7f17c40a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f17c40a6da0 con 0x7f17c40a4920 2024-06-11T19:17:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f17c4138e50 con 0x7f17c40a2530 2024-06-11T19:17:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f17c40a6420 con 0x7f17c40a5240 2024-06-11T19:17:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17ca5f0640 10 monclient(hunting): authenticate will time out at 1278.514752s 2024-06-11T19:17:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c37fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c40a4920 0x7f17c4137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c8b8d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c40a5240 0x7f17c41232a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c3fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17c40a2530 0x7f17c4137740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c37fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c40a4920 0x7f17c4137c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45166/0 (socket says 172.21.15.98:45166) 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c3fff640 10 monclient(hunting): get_auth_request con 0x7f17c40a2530 auth_method 0 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c37fe640 1 -- 172.21.15.98:0/3110015172 learned_addr learned my addr 172.21.15.98:0/3110015172 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c8b8d640 10 monclient(hunting): get_auth_request con 0x7f17c40a5240 auth_method 0 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c8b8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c8b8d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c8b8d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c37fe640 10 monclient(hunting): get_auth_request con 0x7f17c40a4920 auth_method 0 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c8b8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c8b8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c8b8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.707+0000 7f17c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c3fff640 10 monclient(hunting): handle_auth_done global_id 23944 payload 824 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c3fff640 1 -- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c40a5240 msgr2=0x7f17c41232a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c3fff640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c40a5240 0x7f17c41232a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c3fff640 1 -- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c40a4920 msgr2=0x7f17c4137c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c3fff640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c40a4920 0x7f17c4137c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c3fff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c3fff640 1 monclient: found mon.noname-c 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c3fff640 1 -- 172.21.15.98:0/3110015172 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c41c0d10 con 0x7f17c40a2530 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c8b8d640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f17c40a5240 0x7f17c41232a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c37fe640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17c40a4920 0x7f17c4137c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c3fff640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17c40a2530 0x7f17c4137740 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f17b4033000 tx=0x7f17b40046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 10 client.?.objecter ms_handle_connect 0x7f17c40a2530 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17b4042020 con 0x7f17c40a2530 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 10 monclient: dump: 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.728 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 10 monclient: _finish_auth 0 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17b4031ea0 con 0x7f17c40a2530 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17ca5f0640 5 monclient: authenticate success, global_id 23944 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17ca5f0640 10 monclient: _renew_subs 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17ca5f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17ca5f0640 1 -- 172.21.15.98:0/3110015172 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c4126b80 con 0x7f17c40a2530 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17b4038040 con 0x7f17c40a2530 2024-06-11T19:17:48.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17ca5f0640 10 client.23944.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c0ff9640 10 monclient: dump: 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.730 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17ca5f0640 10 monclient: _renew_subs 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17ca5f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17ca5f0640 1 -- 172.21.15.98:0/3110015172 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c40a92f0 con 0x7f17c40a2530 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17ca5f0640 1 librados: init done 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.708+0000 7f17c27fc640 4 set_mon_vals no callback set 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f17b4045020 con 0x7f17c40a2530 2024-06-11T19:17:48.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17ca5f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17ca5f0640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c4126dc0 0x7f17c4123fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17ca5f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17ca5f0640 1 -- 172.21.15.98:0/3110015172 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f17c4050830 con 0x7f17c4126dc0 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c0ff9640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f17b003fda0 0x7f17b0042260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(344..344 src has 1..344) ==== 5845+0+0 (secure 0 0 0) 0x7f17b4072fd0 con 0x7f17c40a2530 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c0ff9640 10 client.23944.objecter ms_dispatch 0x7f17c4000b90 osd_map(344..344 src has 1..344) 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c0ff9640 3 client.23944.objecter handle_osd_map got epochs [344,344] > 0 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c0ff9640 3 client.23944.objecter handle_osd_map decoding full epoch 344 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c37fe640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c4126dc0 0x7f17c4123fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c37fe640 10 monclient: get_auth_request con 0x7f17c4126dc0 auth_method 0 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c0ff9640 20 client.23944.objecter dump_active .. 0 homeless 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c37fe640 10 monclient(hunting): handle_auth_done global_id 94148 payload 824 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.709+0000 7f17c37fe640 1 -- 172.21.15.98:0/3110015172 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17c4073550 con 0x7f17c4126dc0 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c8b8d640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f17b003fda0 0x7f17b0042260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c8b8d640 10 monclient: get_auth_request con 0x7f17b003fda0 auth_method 0 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 10 client.23944.objecter ms_handle_connect 0x7f17c4126dc0 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 10 client.23944.objecter resend_mon_ops 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c37fe640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c4126dc0 0x7f17c4123fa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f17b800fb10 tx=0x7f17b800d1c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17b800dc40 con 0x7f17c4126dc0 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 10 monclient: dump: 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.734 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.735 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f17b800dc40 con 0x7f17c4126dc0 2024-06-11T19:17:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c8b8d640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f17b003fda0 0x7f17b0042260 secure :-1 s=READY pgs=4166 cs=0 l=1 rev1=1 crypto rx=0x7f17c40a2070 tx=0x7f17bc00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c4126dc0 msgr2=0x7f17c4123fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c4126dc0 0x7f17c4123fa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f17b800fb10 tx=0x7f17b800d1c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c4126dc0 msgr2=0x7f17c4123fa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c4126dc0 0x7f17c4123fa0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f17b800fb10 tx=0x7f17b800d1c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.710+0000 7f17c0ff9640 10 client.23944.objecter ms_handle_connect 0x7f17b003fda0 2024-06-11T19:17:48.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.716+0000 7f17ca5f0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:48.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.716+0000 7f17ca5f0640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c402cb40 0x7f17c402cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.716+0000 7f17ca5f0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.716+0000 7f17ca5f0640 1 -- 172.21.15.98:0/3110015172 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f17c40a66f0 con 0x7f17c402cb40 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c37fe640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c402cb40 0x7f17c402cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c37fe640 10 monclient: get_auth_request con 0x7f17c402cb40 auth_method 0 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c37fe640 10 monclient(hunting): handle_auth_done global_id 94157 payload 824 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c37fe640 1 -- 172.21.15.98:0/3110015172 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f17c4124de0 con 0x7f17c402cb40 2024-06-11T19:17:48.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 10 client.23944.objecter ms_handle_connect 0x7f17c402cb40 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 10 client.23944.objecter resend_mon_ops 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c37fe640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c402cb40 0x7f17c402cf20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f17b800fb10 tx=0x7f17b80046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17b8006830 con 0x7f17c402cb40 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 10 monclient: dump: 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.740 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f17b80063c0 con 0x7f17c402cb40 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c402cb40 msgr2=0x7f17c402cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c402cb40 0x7f17c402cf20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f17b800fb10 tx=0x7f17b80046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 1 -- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c402cb40 msgr2=0x7f17c402cf20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:48.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17c0ff9640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f17c402cb40 0x7f17c402cf20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.742 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 14270, 2024-06-11T19:17:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:48.764 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:48.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.717+0000 7f17ca5f0640 10 librados: watch_flush enter 2024-06-11T19:17:48.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 10 librados: watch_flush exit 2024-06-11T19:17:48.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 20 client.23944.objecter shutdown clearing up homeless session... 2024-06-11T19:17:48.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 10 client.23944.objecter successfully canceled tick 2024-06-11T19:17:48.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 1 -- 172.21.15.98:0/3110015172 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f17b003fda0 msgr2=0x7f17b0042260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f17b003fda0 0x7f17b0042260 secure :-1 s=READY pgs=4166 cs=0 l=1 rev1=1 crypto rx=0x7f17c40a2070 tx=0x7f17bc00b040 comp rx=0 tx=0).stop 2024-06-11T19:17:48.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 10 monclient: shutdown 2024-06-11T19:17:48.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:48.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 1 -- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17c40a2530 msgr2=0x7f17c4137740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 1 --2- 172.21.15.98:0/3110015172 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17c40a2530 0x7f17c4137740 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f17b4033000 tx=0x7f17b40046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17c8b8d640 1 -- 172.21.15.98:0/3110015172 reap_dead start 2024-06-11T19:17:48.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17c8b8d640 1 -- 172.21.15.98:0/3110015172 reap_dead start 2024-06-11T19:17:48.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 1 -- 172.21.15.98:0/3110015172 shutdown_connections 2024-06-11T19:17:48.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 1 -- 172.21.15.98:0/3110015172 >> 172.21.15.98:0/3110015172 conn(0x7f17c40ab850 msgr2=0x7f17c40a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:48.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 1 -- 172.21.15.98:0/3110015172 shutdown_connections 2024-06-11T19:17:48.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 1 -- 172.21.15.98:0/3110015172 wait complete. 2024-06-11T19:17:48.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.718+0000 7f17ca5f0640 1 librados: shutdown 2024-06-11T19:17:48.767 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:48.767 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:17:48.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:17:48.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.867+0000 7fd6cfdd2640 10 monclient: get_monmap_and_config 2024-06-11T19:17:48.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.867+0000 7fd6cfdd2640 10 monclient: build_initial_monmap 2024-06-11T19:17:48.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.867+0000 7fd6cfdd2640 10 monclient: monmap: 2024-06-11T19:17:48.867 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:48.867 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.867 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:48.868832+0000 2024-06-11T19:17:48.867 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:48.868832+0000 2024-06-11T19:17:48.867 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:48.869 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.868+0000 7fd6cfdd2640 1 Processor -- start 2024-06-11T19:17:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.868+0000 7fd6cfdd2640 1 -- start start 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.868+0000 7fd6cfdd2640 10 monclient: init 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 10 monclient: _add_conns ranks=[1,2,4,6,5,7,3,9,10,0,8] 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd6c80a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6c80a2530 0x7fd6c813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 10 monclient(hunting): picked mon.noname-b con 0x7fd6c80a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6c813e1a0 0x7fd6c8140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 10 monclient(hunting): picked mon.noname-c con 0x7fd6c813e1a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c80a7140 con 0x7fd6c80a4920 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd6c80a2b50 con 0x7fd6c80a2530 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd6c80a2e30 con 0x7fd6c813e1a0 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cfdd2640 10 monclient(hunting): authenticate will time out at 1278.676256s 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6c80a2530 0x7fd6c813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6c80a2530 0x7fd6c813dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45172/0 (socket says 172.21.15.98:45172) 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 1 -- 172.21.15.98:0/758408712 learned_addr learned my addr 172.21.15.98:0/758408712 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 10 monclient(hunting): get_auth_request con 0x7fd6c80a2530 auth_method 0 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6ce36f640 1 --2- 172.21.15.98:0/758408712 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6c813e1a0 0x7fd6c8140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6ce36f640 10 monclient(hunting): get_auth_request con 0x7fd6c813e1a0 auth_method 0 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6ce36f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6ce36f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6ce36f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 10 monclient(hunting): handle_auth_done global_id 84395 payload 274 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 1 -- 172.21.15.98:0/758408712 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6c813e1a0 msgr2=0x7fd6c8140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 1 --2- 172.21.15.98:0/758408712 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6c813e1a0 0x7fd6c8140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 1 -- 172.21.15.98:0/758408712 >> v1:172.21.15.117:6789/0 conn(0x7fd6c80a4920 legacy=0x7fd6c80a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 10 monclient: _finish_hunting 0 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 1 monclient: found mon.noname-b 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.869+0000 7fd6cd36d640 1 -- 172.21.15.98:0/758408712 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c80a3110 con 0x7fd6c80a2530 2024-06-11T19:17:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6cd36d640 1 --2- 172.21.15.98:0/758408712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6c80a2530 0x7fd6c813dc60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6b80098a0 tx=0x7fd6b802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d9938bc565538c7 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 1 -- 172.21.15.98:0/758408712 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6b8031a10 con 0x7fd6c80a2530 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 10 monclient: dump: 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 10 monclient: _finish_auth 0 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 10 monclient: _check_auth_tickets 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 1 -- 172.21.15.98:0/758408712 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6b8037070 con 0x7fd6c80a2530 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6cfdd2640 5 monclient: authenticate success, global_id 84395 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6cfdd2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6cfdd2640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 1 -- 172.21.15.98:0/758408712 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6b8031c10 con 0x7fd6c80a2530 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6cfdd2640 4 set_mon_vals no callback set 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.870+0000 7fd6ccb6c640 10 monclient: dump: 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.871+0000 7fd6cfdd2640 10 monclient: shutdown 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.871+0000 7fd6cfdd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.871+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/758408712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6c80a2530 msgr2=0x7fd6c813dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.871+0000 7fd6cfdd2640 1 --2- 172.21.15.98:0/758408712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6c80a2530 0x7fd6c813dc60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6b80098a0 tx=0x7fd6b802b840 comp rx=0 tx=0).stop 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.871+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/758408712 shutdown_connections 2024-06-11T19:17:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.871+0000 7fd6cfdd2640 1 --2- 172.21.15.98:0/758408712 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6c813e1a0 0x7fd6c8140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.871+0000 7fd6cfdd2640 1 --2- 172.21.15.98:0/758408712 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd6c80a2530 0x7fd6c813dc60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.871+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/758408712 >> 172.21.15.98:0/758408712 conn(0x7fd6c80ab850 msgr2=0x7fd6c80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.871+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/758408712 shutdown_connections 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.871+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/758408712 wait complete. 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.871+0000 7fd6cfdd2640 10 monclient: build_initial_monmap 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.871+0000 7fd6cfdd2640 10 monclient: monmap: 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 1 librados: starting msgr at 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 1 librados: starting objecter 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 1 Processor -- start 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 1 -- start start 2024-06-11T19:17:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 1 librados: setting wanted keys 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 1 librados: calling monclient init 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 10 monclient: init 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 10 monclient: _add_conns ranks=[0,6,4,7,5,2,1,8,3] 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80a2530 0x7fd6c81cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 10 monclient(hunting): picked mon.noname-a con 0x7fd6c80a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd6c80a4920 0x7fd6c81cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 10 monclient(hunting): picked mon.noname-d con 0x7fd6c80a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6c813e1a0 0x7fd6c81ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 10 monclient(hunting): picked mon.noname-c con 0x7fd6c813e1a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c8138e50 con 0x7fd6c80a2530 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd6c80a84b0 con 0x7fd6c813e1a0 2024-06-11T19:17:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd6c80a6da0 con 0x7fd6c80a4920 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cfdd2640 10 monclient(hunting): authenticate will time out at 1278.679741s 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cdb6e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80a2530 0x7fd6c81cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cd36d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd6c80a4920 0x7fd6c81cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cdb6e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80a2530 0x7fd6c81cd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56634/0 (socket says 172.21.15.98:56634) 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6ce36f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6c813e1a0 0x7fd6c81ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cd36d640 10 monclient(hunting): get_auth_request con 0x7fd6c80a4920 auth_method 0 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cdb6e640 1 -- 172.21.15.98:0/1401551403 learned_addr learned my addr 172.21.15.98:0/1401551403 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cd36d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cd36d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cd36d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cdb6e640 10 monclient(hunting): get_auth_request con 0x7fd6c80a2530 auth_method 0 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cdb6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cdb6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cdb6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6ce36f640 10 monclient(hunting): get_auth_request con 0x7fd6c813e1a0 auth_method 0 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6ce36f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6ce36f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6ce36f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cd36d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cd36d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cd36d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6ce36f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6ce36f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6ce36f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cdb6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cdb6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.872+0000 7fd6cdb6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cd36d640 10 monclient(hunting): handle_auth_done global_id 55104 payload 824 2024-06-11T19:17:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cd36d640 1 -- 172.21.15.98:0/1401551403 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6c813e1a0 msgr2=0x7fd6c81ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cd36d640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6c813e1a0 0x7fd6c81ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cd36d640 1 -- 172.21.15.98:0/1401551403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80a2530 msgr2=0x7fd6c81cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cd36d640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80a2530 0x7fd6c81cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cd36d640 10 monclient: _finish_hunting 0 2024-06-11T19:17:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cd36d640 1 monclient: found mon.noname-d 2024-06-11T19:17:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cd36d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cd36d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:17:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cd36d640 1 -- 172.21.15.98:0/1401551403 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c80a0450 con 0x7fd6c80a4920 2024-06-11T19:17:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6ce36f640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd6c813e1a0 0x7fd6c81ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cd36d640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd6c80a4920 0x7fd6c81cde20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8033000 tx=0x7fd6b80046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 10 client.?.objecter ms_handle_connect 0x7fd6c80a4920 2024-06-11T19:17:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6b8041070 con 0x7fd6c80a4920 2024-06-11T19:17:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:17:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 10 monclient: dump: 2024-06-11T19:17:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.894 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 10 monclient: _finish_auth 0 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6b80315a0 con 0x7fd6c80a4920 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cfdd2640 5 monclient: authenticate success, global_id 55104 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cfdd2640 10 monclient: _renew_subs 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cfdd2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/1401551403 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c813de10 con 0x7fd6c80a4920 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6b8031bc0 con 0x7fd6c80a4920 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cfdd2640 10 client.55104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6bffff640 4 set_mon_vals no callback set 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6be7fc640 10 monclient: dump: 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cfdd2640 10 monclient: _renew_subs 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cfdd2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/1401551403 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c80a0730 con 0x7fd6c80a4920 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.873+0000 7fd6cfdd2640 1 librados: init done 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6cfdd2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd6b804c060 con 0x7fd6c80a4920 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6cfdd2640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c80a1480 0x7fd6c8133710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6cfdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/1401551403 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c8050830 con 0x7fd6c80a1480 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6be7fc640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6ac03fdb0 0x7fd6ac042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(344..344 src has 1..344) ==== 5845+0+0 (secure 0 0 0) 0x7fd6b802f050 con 0x7fd6c80a4920 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6be7fc640 10 client.55104.objecter ms_dispatch 0x7fd6c8000b90 osd_map(344..344 src has 1..344) 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6be7fc640 3 client.55104.objecter handle_osd_map got epochs [344,344] > 0 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6be7fc640 3 client.55104.objecter handle_osd_map decoding full epoch 344 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6be7fc640 20 client.55104.objecter dump_active .. 0 homeless 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6cdb6e640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c80a1480 0x7fd6c8133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6cdb6e640 10 monclient: get_auth_request con 0x7fd6c80a1480 auth_method 0 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6cdb6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6cdb6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6cdb6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6ce36f640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6ac03fdb0 0x7fd6ac042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6ce36f640 10 monclient: get_auth_request con 0x7fd6ac03fdb0 auth_method 0 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6cdb6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.874+0000 7fd6cdb6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.875+0000 7fd6cdb6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.875+0000 7fd6cdb6e640 10 monclient(hunting): handle_auth_done global_id 32122 payload 824 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.875+0000 7fd6cdb6e640 1 -- 172.21.15.98:0/1401551403 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6c80a4d00 con 0x7fd6c80a1480 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.875+0000 7fd6be7fc640 10 client.55104.objecter ms_handle_connect 0x7fd6ac03fdb0 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.875+0000 7fd6ce36f640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6ac03fdb0 0x7fd6ac042270 secure :-1 s=READY pgs=4167 cs=0 l=1 rev1=1 crypto rx=0x7fd6c809f2f0 tx=0x7fd6c4009780 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.875+0000 7fd6be7fc640 10 client.55104.objecter ms_handle_connect 0x7fd6c80a1480 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.875+0000 7fd6be7fc640 10 client.55104.objecter resend_mon_ops 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.875+0000 7fd6cdb6e640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c80a1480 0x7fd6c8133710 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000f7a0 tx=0x7fd6c000fc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.875+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6c000d700 con 0x7fd6c80a1480 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.875+0000 7fd6be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.875+0000 7fd6be7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.875+0000 7fd6be7fc640 10 monclient: dump: 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.876+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd6c0018070 con 0x7fd6c80a1480 2024-06-11T19:17:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.876+0000 7fd6be7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.876+0000 7fd6be7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.876+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c80a1480 msgr2=0x7fd6c8133710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.876+0000 7fd6be7fc640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c80a1480 0x7fd6c8133710 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000f7a0 tx=0x7fd6c000fc60 comp rx=0 tx=0).stop 2024-06-11T19:17:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.876+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c80a1480 msgr2=0x7fd6c8133710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.876+0000 7fd6be7fc640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c80a1480 0x7fd6c8133710 secure :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000f7a0 tx=0x7fd6c000fc60 comp rx=0 tx=0).stop 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cfdd2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cfdd2640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c8133c50 0x7fd6c8134060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cfdd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/1401551403 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81345a0 con 0x7fd6c8133c50 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cdb6e640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c8133c50 0x7fd6c8134060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cdb6e640 10 monclient: get_auth_request con 0x7fd6c8133c50 auth_method 0 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cdb6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cdb6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cdb6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cdb6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cdb6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cdb6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cdb6e640 10 monclient(hunting): handle_auth_done global_id 32131 payload 824 2024-06-11T19:17:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.882+0000 7fd6cdb6e640 1 -- 172.21.15.98:0/1401551403 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6c8069270 con 0x7fd6c8133c50 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6cdb6e640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c8133c50 0x7fd6c8134060 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000f7a0 tx=0x7fd6c0000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 10 client.55104.objecter ms_handle_connect 0x7fd6c8133c50 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 10 client.55104.objecter resend_mon_ops 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6c000dd80 con 0x7fd6c8133c50 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 10 monclient: dump: 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7fd6c000d910 con 0x7fd6c8133c50 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:48.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c8133c50 msgr2=0x7fd6c8134060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c8133c50 0x7fd6c8134060 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000f7a0 tx=0x7fd6c0000ed0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 1 -- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c8133c50 msgr2=0x7fd6c8134060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:48.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6be7fc640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6c8133c50 0x7fd6c8134060 secure :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000f7a0 tx=0x7fd6c0000ed0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.906 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:17:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 519101, 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:48.930 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:48.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6cfdd2640 10 librados: watch_flush enter 2024-06-11T19:17:48.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6cfdd2640 10 librados: watch_flush exit 2024-06-11T19:17:48.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6cfdd2640 20 client.55104.objecter shutdown clearing up homeless session... 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.883+0000 7fd6cfdd2640 10 client.55104.objecter successfully canceled tick 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6ac03fdb0 msgr2=0x7fd6ac042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6cfdd2640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6ac03fdb0 0x7fd6ac042270 secure :-1 s=READY pgs=4167 cs=0 l=1 rev1=1 crypto rx=0x7fd6c809f2f0 tx=0x7fd6c4009780 comp rx=0 tx=0).stop 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6cfdd2640 10 monclient: shutdown 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6cfdd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/1401551403 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd6c80a4920 msgr2=0x7fd6c81cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6cfdd2640 1 --2- 172.21.15.98:0/1401551403 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd6c80a4920 0x7fd6c81cde20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8033000 tx=0x7fd6b80046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6ce36f640 1 -- 172.21.15.98:0/1401551403 reap_dead start 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6ce36f640 1 -- 172.21.15.98:0/1401551403 reap_dead start 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/1401551403 shutdown_connections 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/1401551403 >> 172.21.15.98:0/1401551403 conn(0x7fd6c80ab850 msgr2=0x7fd6c80a91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/1401551403 shutdown_connections 2024-06-11T19:17:48.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6cfdd2640 1 -- 172.21.15.98:0/1401551403 wait complete. 2024-06-11T19:17:48.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:48.884+0000 7fd6cfdd2640 1 librados: shutdown 2024-06-11T19:17:48.933 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:48.933 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:17:48.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:17:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.034+0000 7f1cd43a4640 10 monclient: get_monmap_and_config 2024-06-11T19:17:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.034+0000 7f1cd43a4640 10 monclient: build_initial_monmap 2024-06-11T19:17:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.034+0000 7f1cd43a4640 10 monclient: monmap: 2024-06-11T19:17:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:49.035362+0000 2024-06-11T19:17:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:49.035362+0000 2024-06-11T19:17:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:49.034 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:49.035 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:49.035 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:49.035 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.034+0000 7f1cd43a4640 1 Processor -- start 2024-06-11T19:17:49.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.034+0000 7f1cd43a4640 1 -- start start 2024-06-11T19:17:49.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.034+0000 7f1cd43a4640 10 monclient: init 2024-06-11T19:17:49.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd43a4640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:49.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd43a4640 10 monclient: _add_conns ranks=[0,10,1,7,4,2,3,6,9,5,8] 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd43a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ccc0a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd43a4640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1ccc0a2530 0x7f1ccc13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd43a4640 10 monclient(hunting): picked mon.noname-f con 0x7f1ccc0a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd43a4640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f1ccc13e1a0 addr v1:172.21.15.117:6789/0 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd43a4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ccc0a7140 con 0x7f1ccc0a4920 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd43a4640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ccc0a2b50 con 0x7f1ccc13e1a0 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd43a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd43a4640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f1ccc0a2e30 con 0x7f1ccc0a2530 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd43a4640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd43a4640 10 monclient(hunting): authenticate will time out at 1278.842631s 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd2140640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1ccc0a4920 0x7f1ccc0a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35750/0 (socket says 172.21.15.98:35750) 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd2140640 1 -- 172.21.15.98:0/236217569 learned_addr learned my addr 172.21.15.98:0/236217569 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:17:49.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.035+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1202102174 0 0) 0x7f1ccc0a7140 con 0x7f1ccc0a4920 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient(hunting): _negotiate 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient(hunting): my global_id is 64980 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cb8003520 con 0x7f1ccc0a4920 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 1 --2- 172.21.15.98:0/236217569 >> v2:172.21.15.117:3300/0 conn(0x7f1ccc0a2530 0x7f1ccc13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): get_auth_request con 0x7f1ccc0a2530 auth_method 0 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1662003784 0 0) 0x7f1cb8003520 con 0x7f1ccc0a4920 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 >> v2:172.21.15.117:3300/0 conn(0x7f1ccc0a2530 msgr2=0x7f1ccc13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 --2- 172.21.15.98:0/236217569 >> v2:172.21.15.117:3300/0 conn(0x7f1ccc0a2530 0x7f1ccc13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 >> v1:172.21.15.117:6789/0 conn(0x7f1ccc13e1a0 legacy=0x7f1ccc140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient: _finish_hunting 0 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ccc0a3110 con 0x7f1ccc0a4920 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient: _finish_auth 0 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient: _check_auth_tickets 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7f1cc8001960 con 0x7f1ccc0a4920 2024-06-11T19:17:49.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd43a4640 5 monclient: authenticate success, global_id 64980 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd43a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient: dump: 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.051 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 >> v1:172.21.15.98:6789/0 conn(0x7f1ccc0a4920 legacy=0x7f1ccc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient: _add_conns ranks=[3,0,8,7,1,6,5,2,4] 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 --2- 172.21.15.98:0/236217569 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cb8004250 0x7f1cb8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient(hunting): picked mon.g con 0x7f1cb8004250 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:49.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 --2- 172.21.15.98:0/236217569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cb8004db0 0x7f1cb8005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient(hunting): picked mon.a con 0x7f1cb8004db0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:17:49.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 --2- 172.21.15.98:0/236217569 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1cb8009c70 0x7f1cb800c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient(hunting): picked mon.e con 0x7f1cb8009c70 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:17:49.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc0a2e30 con 0x7f1cb8004db0 2024-06-11T19:17:49.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f1cb800c9e0 con 0x7f1cb8004250 2024-06-11T19:17:49.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f1cb800cb90 con 0x7f1cb8009c70 2024-06-11T19:17:49.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:49.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd113e640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd43a4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 1 --2- 172.21.15.98:0/236217569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cb8004db0 0x7f1cb8005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): get_auth_request con 0x7f1cb8004db0 auth_method 0 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2941640 1 --2- 172.21.15.98:0/236217569 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1cb8009c70 0x7f1cb800c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2941640 10 monclient(hunting): get_auth_request con 0x7f1cb8009c70 auth_method 0 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2941640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2941640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2941640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2140640 1 --2- 172.21.15.98:0/236217569 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cb8004250 0x7f1cb8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2140640 10 monclient(hunting): get_auth_request con 0x7f1cb8004250 auth_method 0 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2140640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2140640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2941640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2941640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2941640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2140640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2140640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2140640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:17:49.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient(hunting): handle_auth_done global_id 64980 payload 293 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 1 -- 172.21.15.98:0/236217569 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1cb8009c70 msgr2=0x7f1cb800c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 1 --2- 172.21.15.98:0/236217569 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1cb8009c70 0x7f1cb800c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 1 -- 172.21.15.98:0/236217569 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cb8004250 msgr2=0x7f1cb8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 1 --2- 172.21.15.98:0/236217569 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cb8004250 0x7f1cb8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient: _finish_hunting 0 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 1 monclient: found mon.a 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd2941640 1 -- 172.21.15.98:0/236217569 reap_dead start 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 1 -- 172.21.15.98:0/236217569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb8003520 con 0x7f1cb8004db0 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient: _finish_auth 0 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 10 monclient: _check_auth_tickets 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.036+0000 7f1cd193f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd193f640 1 --2- 172.21.15.98:0/236217569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cb8004db0 0x7f1cb8005230 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0009ac0 tx=0x7f1cb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134d7192b58162f2 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1cc0039070 con 0x7f1cb8004db0 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd113e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd113e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd113e640 10 monclient: dump: 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.057 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cc002cb20 con 0x7f1cb8004db0 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd43a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd113e640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd113e640 1 -- 172.21.15.98:0/236217569 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1cc0009690 con 0x7f1cb8004db0 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd43a4640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd43a4640 4 set_mon_vals no callback set 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd113e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd113e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.037+0000 7f1cd113e640 10 monclient: dump: 2024-06-11T19:17:49.058 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.059 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.038+0000 7f1cd43a4640 10 monclient: shutdown 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.038+0000 7f1cd43a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.038+0000 7f1cd43a4640 1 -- 172.21.15.98:0/236217569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cb8004db0 msgr2=0x7f1cb8005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.038+0000 7f1cd43a4640 1 --2- 172.21.15.98:0/236217569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cb8004db0 0x7f1cb8005230 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0009ac0 tx=0x7f1cb8005830 comp rx=0 tx=0).stop 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.038+0000 7f1cd43a4640 1 -- 172.21.15.98:0/236217569 shutdown_connections 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.038+0000 7f1cd43a4640 1 --2- 172.21.15.98:0/236217569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cb8004db0 0x7f1cb8005230 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.038+0000 7f1cd43a4640 1 -- 172.21.15.98:0/236217569 >> 172.21.15.98:0/236217569 conn(0x7f1ccc0ab850 msgr2=0x7f1ccc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.038+0000 7f1cd43a4640 1 -- 172.21.15.98:0/236217569 shutdown_connections 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.038+0000 7f1cd43a4640 1 -- 172.21.15.98:0/236217569 wait complete. 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.038+0000 7f1cd43a4640 10 monclient: build_initial_monmap 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.038+0000 7f1cd43a4640 10 monclient: monmap: 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:49.060 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 1 librados: starting msgr at 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 1 librados: starting objecter 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 1 Processor -- start 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 1 -- start start 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 1 librados: setting wanted keys 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 1 librados: calling monclient init 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 10 monclient: init 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 10 monclient: _add_conns ranks=[6,7,1,2,4,0,5,8,3] 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cb8004db0 0x7f1ccc136b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 10 monclient(hunting): picked mon.noname-d con 0x7f1cb8004db0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1ccc1cd850 0x7f1ccc137060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 10 monclient(hunting): picked mon.noname-i con 0x7f1ccc1cd850 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ccc09fb50 0x7f1ccc09ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 10 monclient(hunting): picked mon.noname-f con 0x7f1ccc09fb50 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc0a6da0 con 0x7f1ccc09fb50 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1ccc138e50 con 0x7f1cb8004db0 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f1ccc0a4d00 con 0x7f1ccc1cd850 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd43a4640 10 monclient(hunting): authenticate will time out at 1278.846711s 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd2140640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cb8004db0 0x7f1ccc136b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd2140640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cb8004db0 0x7f1ccc136b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:33136/0 (socket says 172.21.15.98:33136) 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd2140640 1 -- 172.21.15.98:0/2870490059 learned_addr learned my addr 172.21.15.98:0/2870490059 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd2140640 10 monclient(hunting): get_auth_request con 0x7f1cb8004db0 auth_method 0 2024-06-11T19:17:49.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd2140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd2140640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd2140640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd193f640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1ccc1cd850 0x7f1ccc137060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd193f640 10 monclient(hunting): get_auth_request con 0x7f1ccc1cd850 auth_method 0 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd193f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd193f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd193f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd2140640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd2140640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd2140640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd2941640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ccc09fb50 0x7f1ccc09ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.039+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2941640 10 monclient(hunting): get_auth_request con 0x7f1ccc09fb50 auth_method 0 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2941640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2941640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2941640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2941640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2941640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2941640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2140640 10 monclient(hunting): handle_auth_done global_id 55113 payload 824 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2140640 1 -- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1ccc1cd850 msgr2=0x7f1ccc137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2140640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1ccc1cd850 0x7f1ccc137060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2140640 1 -- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ccc09fb50 msgr2=0x7f1ccc09ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2140640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ccc09fb50 0x7f1ccc09ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2140640 10 monclient: _finish_hunting 0 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2140640 1 monclient: found mon.noname-d 2024-06-11T19:17:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2140640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2140640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2140640 1 -- 172.21.15.98:0/2870490059 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ccc123210 con 0x7f1cb8004db0 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd2140640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cb8004db0 0x7f1ccc136b20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1cc80072f0 tx=0x7f1cc80077b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 10 client.?.objecter ms_handle_connect 0x7f1cb8004db0 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1cc8001960 con 0x7f1cb8004db0 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 10 monclient: dump: 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.066 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 10 monclient: _finish_auth 0 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 10 monclient: _check_auth_tickets 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cc8018070 con 0x7f1cb8004db0 2024-06-11T19:17:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd43a4640 5 monclient: authenticate success, global_id 55113 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd43a4640 10 monclient: _renew_subs 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd43a4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd43a4640 1 -- 172.21.15.98:0/2870490059 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ccc1234f0 con 0x7f1cb8004db0 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1cc800fe30 con 0x7f1cb8004db0 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cbeffd640 10 monclient: dump: 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd43a4640 10 client.55113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd093d640 4 set_mon_vals no callback set 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd43a4640 10 monclient: _renew_subs 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd43a4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd43a4640 1 -- 172.21.15.98:0/2870490059 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ccc0aa520 con 0x7f1cb8004db0 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.040+0000 7f1cd43a4640 1 librados: init done 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd43a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd43a4640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc0a0510 0x7f1ccc13bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd43a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd43a4640 1 -- 172.21.15.98:0/2870490059 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f1ccc13e020 con 0x7f1ccc0a0510 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f1cc800f9c0 con 0x7f1cb8004db0 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd193f640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc0a0510 0x7f1ccc13bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd193f640 10 monclient: get_auth_request con 0x7f1ccc0a0510 auth_method 0 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd193f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd193f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd193f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cbeffd640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1cb003fdb0 0x7f1cb0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(344..344 src has 1..344) ==== 5845+0+0 (secure 0 0 0) 0x7f1cc8010040 con 0x7f1cb8004db0 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cbeffd640 10 client.55113.objecter ms_dispatch 0x7f1ccc000b90 osd_map(344..344 src has 1..344) 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cbeffd640 3 client.55113.objecter handle_osd_map got epochs [344,344] > 0 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cbeffd640 3 client.55113.objecter handle_osd_map decoding full epoch 344 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd193f640 10 monclient(hunting): handle_auth_done global_id 94143 payload 824 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cbeffd640 20 client.55113.objecter dump_active .. 0 homeless 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd193f640 1 -- 172.21.15.98:0/2870490059 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ccc0a4fa0 con 0x7f1ccc0a0510 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd2941640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1cb003fdb0 0x7f1cb0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.041+0000 7f1cd2941640 10 monclient: get_auth_request con 0x7f1cb003fdb0 auth_method 0 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 10 client.55113.objecter ms_handle_connect 0x7f1ccc0a0510 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 10 client.55113.objecter resend_mon_ops 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cd193f640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc0a0510 0x7f1ccc13bdf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0009a90 tx=0x7f1cc0004730 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1cc0044070 con 0x7f1ccc0a0510 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 10 monclient: dump: 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.072 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.073 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.073 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.073 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 10 client.55113.objecter ms_handle_connect 0x7f1cb003fdb0 2024-06-11T19:17:49.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cd2941640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1cb003fdb0 0x7f1cb0042270 secure :-1 s=READY pgs=4168 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc073fc0 tx=0x7f1cc4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1cc003a040 con 0x7f1ccc0a0510 2024-06-11T19:17:49.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:49.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:49.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc0a0510 msgr2=0x7f1ccc13bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc0a0510 0x7f1ccc13bdf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0009a90 tx=0x7f1cc0004730 comp rx=0 tx=0).stop 2024-06-11T19:17:49.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc0a0510 msgr2=0x7f1ccc13bdf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:49.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.042+0000 7f1cbeffd640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc0a0510 0x7f1ccc13bdf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.048+0000 7f1cd43a4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:49.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.048+0000 7f1cd43a4640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc13c330 0x7f1ccc13c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.048+0000 7f1cd43a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.048+0000 7f1cd43a4640 1 -- 172.21.15.98:0/2870490059 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f1ccc050830 con 0x7f1ccc13c330 2024-06-11T19:17:49.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cd193f640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc13c330 0x7f1ccc13c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cd193f640 10 monclient: get_auth_request con 0x7f1ccc13c330 auth_method 0 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cd193f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cd193f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cd193f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cd193f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cd193f640 10 monclient(hunting): handle_auth_done global_id 94152 payload 824 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cd193f640 1 -- 172.21.15.98:0/2870490059 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ccc06c300 con 0x7f1ccc13c330 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cd193f640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc13c330 0x7f1ccc13c740 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00099e0 tx=0x7f1cc0008030 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cbeffd640 10 client.55113.objecter ms_handle_connect 0x7f1ccc13c330 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cbeffd640 10 client.55113.objecter resend_mon_ops 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1cc0039040 con 0x7f1ccc13c330 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cbeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.049+0000 7f1cbeffd640 10 monclient: dump: 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.077 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.078 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.078 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f1cc0003930 con 0x7f1ccc13c330 2024-06-11T19:17:49.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cbeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:49.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cbeffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:49.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc13c330 msgr2=0x7f1ccc13c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cbeffd640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc13c330 0x7f1ccc13c740 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00099e0 tx=0x7f1cc0008030 comp rx=0 tx=0).stop 2024-06-11T19:17:49.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cbeffd640 1 -- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc13c330 msgr2=0x7f1ccc13c740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:49.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cbeffd640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ccc13c330 0x7f1ccc13c740 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.078 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 14598, 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:49.101 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:49.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 10 librados: watch_flush enter 2024-06-11T19:17:49.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 10 librados: watch_flush exit 2024-06-11T19:17:49.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 20 client.55113.objecter shutdown clearing up homeless session... 2024-06-11T19:17:49.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 10 client.55113.objecter successfully canceled tick 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 1 -- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1cb003fdb0 msgr2=0x7f1cb0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1cb003fdb0 0x7f1cb0042270 secure :-1 s=READY pgs=4168 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc073fc0 tx=0x7f1cc4009040 comp rx=0 tx=0).stop 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 10 monclient: shutdown 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 1 -- 172.21.15.98:0/2870490059 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cb8004db0 msgr2=0x7f1ccc136b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 1 --2- 172.21.15.98:0/2870490059 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cb8004db0 0x7f1ccc136b20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1cc80072f0 tx=0x7f1cc80077b0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd2941640 1 -- 172.21.15.98:0/2870490059 reap_dead start 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd2941640 1 -- 172.21.15.98:0/2870490059 reap_dead start 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 1 -- 172.21.15.98:0/2870490059 shutdown_connections 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 1 -- 172.21.15.98:0/2870490059 >> 172.21.15.98:0/2870490059 conn(0x7f1ccc0ab850 msgr2=0x7f1ccc0a9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 1 -- 172.21.15.98:0/2870490059 shutdown_connections 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 1 -- 172.21.15.98:0/2870490059 wait complete. 2024-06-11T19:17:49.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.050+0000 7f1cd43a4640 1 librados: shutdown 2024-06-11T19:17:49.104 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:49.104 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:17:49.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:17:49.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.202+0000 7fc99063e640 10 monclient: get_monmap_and_config 2024-06-11T19:17:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.202+0000 7fc99063e640 10 monclient: build_initial_monmap 2024-06-11T19:17:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.202+0000 7fc99063e640 10 monclient: monmap: 2024-06-11T19:17:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:49.203845+0000 2024-06-11T19:17:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:49.203845+0000 2024-06-11T19:17:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:49.202 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:49.203 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:49.203 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.203+0000 7fc99063e640 1 Processor -- start 2024-06-11T19:17:49.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.203+0000 7fc99063e640 1 -- start start 2024-06-11T19:17:49.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.203+0000 7fc99063e640 10 monclient: init 2024-06-11T19:17:49.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:49.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 10 monclient: _add_conns ranks=[6,7,2,9,3,8,0,10,5,4,1] 2024-06-11T19:17:49.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9880a4920 0x7fc9880a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 10 monclient(hunting): picked mon.noname-d con 0x7fc9880a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:49.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc9880a5240 0x7fc9880a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 10 monclient(hunting): picked mon.noname-i con 0x7fc9880a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:17:49.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9880a2530 0x7fc9880a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 10 monclient(hunting): picked mon.noname-b con 0x7fc9880a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:49.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc9880a7140 con 0x7fc9880a2530 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc9880a2b50 con 0x7fc9880a4920 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fc9880a2e30 con 0x7fc9880a5240 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc99063e640 10 monclient(hunting): authenticate will time out at 1279.011125s 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9880a2530 0x7fc9880a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98e3da640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9880a4920 0x7fc9880a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9880a2530 0x7fc9880a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45184/0 (socket says 172.21.15.98:45184) 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98e3da640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9880a4920 0x7fc9880a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:33142/0 (socket says 172.21.15.98:33142) 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 1 -- 172.21.15.98:0/3232638682 learned_addr learned my addr 172.21.15.98:0/3232638682 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 10 monclient(hunting): get_auth_request con 0x7fc9880a2530 auth_method 0 2024-06-11T19:17:49.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98e3da640 10 monclient(hunting): get_auth_request con 0x7fc9880a4920 auth_method 0 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98e3da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98e3da640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98e3da640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98e3da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98e3da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98e3da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98dbd9640 1 --2- 172.21.15.98:0/3232638682 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc9880a5240 0x7fc9880a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98dbd9640 10 monclient(hunting): get_auth_request con 0x7fc9880a5240 auth_method 0 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98dbd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98dbd9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98dbd9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 10 monclient(hunting): handle_auth_done global_id 84404 payload 274 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 1 -- 172.21.15.98:0/3232638682 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc9880a5240 msgr2=0x7fc9880a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 1 --2- 172.21.15.98:0/3232638682 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc9880a5240 0x7fc9880a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 1 -- 172.21.15.98:0/3232638682 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9880a4920 msgr2=0x7fc9880a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 1 --2- 172.21.15.98:0/3232638682 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9880a4920 0x7fc9880a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 10 monclient: _finish_hunting 0 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 1 monclient: found mon.noname-b 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:17:49.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98ebdb640 1 -- 172.21.15.98:0/3232638682 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9880a3110 con 0x7fc9880a2530 2024-06-11T19:17:49.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.204+0000 7fc98e3da640 1 --2- 172.21.15.98:0/3232638682 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9880a4920 0x7fc9880a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:49.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98ebdb640 1 --2- 172.21.15.98:0/3232638682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9880a2530 0x7fc9880a5ee0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc9780098a0 tx=0x7fc97802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77fd1fb3b666095b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 1 -- 172.21.15.98:0/3232638682 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc978038070 con 0x7fc9880a2530 2024-06-11T19:17:49.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:17:49.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 10 monclient: dump: 2024-06-11T19:17:49.218 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.218 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.218 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.218 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.218 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.218 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 10 monclient: _finish_auth 0 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 10 monclient: _check_auth_tickets 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:49.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc99063e640 5 monclient: authenticate success, global_id 84404 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 1 -- 172.21.15.98:0/3232638682 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc978031970 con 0x7fc9880a2530 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc99063e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc99063e640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc99063e640 4 set_mon_vals no callback set 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 1 -- 172.21.15.98:0/3232638682 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc978031ba0 con 0x7fc9880a2530 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.205+0000 7fc98d3d8640 10 monclient: dump: 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.220 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 10 monclient: shutdown 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 1 -- 172.21.15.98:0/3232638682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9880a2530 msgr2=0x7fc9880a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 1 --2- 172.21.15.98:0/3232638682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9880a2530 0x7fc9880a5ee0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc9780098a0 tx=0x7fc97802b7a0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 1 -- 172.21.15.98:0/3232638682 shutdown_connections 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 1 --2- 172.21.15.98:0/3232638682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc9880a2530 0x7fc9880a5ee0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 1 --2- 172.21.15.98:0/3232638682 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc9880a5240 0x7fc9880a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 1 --2- 172.21.15.98:0/3232638682 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc9880a4920 0x7fc9880a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 1 -- 172.21.15.98:0/3232638682 >> 172.21.15.98:0/3232638682 conn(0x7fc9880ab850 msgr2=0x7fc9880a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 1 -- 172.21.15.98:0/3232638682 shutdown_connections 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 1 -- 172.21.15.98:0/3232638682 wait complete. 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 10 monclient: build_initial_monmap 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 10 monclient: monmap: 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:49.222 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 1 librados: starting msgr at 2024-06-11T19:17:49.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.206+0000 7fc99063e640 1 librados: starting objecter 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 1 Processor -- start 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 1 -- start start 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 1 librados: setting wanted keys 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 1 librados: calling monclient init 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 10 monclient: init 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 10 monclient: _add_conns ranks=[3,5,7,2,6,8,1,4,0] 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9880a2530 0x7fc9881215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 10 monclient(hunting): picked mon.noname-g con 0x7fc9880a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc9880a4920 0x7fc988121b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 10 monclient(hunting): picked mon.noname-h con 0x7fc9880a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc9880a5240 0x7fc988122070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 10 monclient(hunting): picked mon.noname-i con 0x7fc9880a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:17:49.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fc9880a6da0 con 0x7fc9880a2530 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc988138e50 con 0x7fc9880a4920 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fc9880a6420 con 0x7fc9880a5240 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc99063e640 10 monclient(hunting): authenticate will time out at 1279.014643s 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98ebdb640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc9880a5240 0x7fc988122070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98dbd9640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc9880a4920 0x7fc988121b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98dbd9640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc9880a4920 0x7fc988121b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:59148/0 (socket says 172.21.15.98:59148) 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98ebdb640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc9880a5240 0x7fc988122070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:34584/0 (socket says 172.21.15.98:34584) 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98dbd9640 1 -- 172.21.15.98:0/1367045158 learned_addr learned my addr 172.21.15.98:0/1367045158 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98dbd9640 10 monclient(hunting): get_auth_request con 0x7fc9880a4920 auth_method 0 2024-06-11T19:17:49.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98dbd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98dbd9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98dbd9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98e3da640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9880a2530 0x7fc9881215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98ebdb640 10 monclient(hunting): get_auth_request con 0x7fc9880a5240 auth_method 0 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98ebdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98ebdb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98ebdb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98e3da640 10 monclient(hunting): get_auth_request con 0x7fc9880a2530 auth_method 0 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98e3da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98e3da640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.207+0000 7fc98e3da640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98ebdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98ebdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98ebdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98dbd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98dbd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98dbd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 10 monclient(hunting): handle_auth_done global_id 94161 payload 824 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 1 -- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc9880a5240 msgr2=0x7fc988122070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc9880a5240 0x7fc988122070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 1 -- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc9880a4920 msgr2=0x7fc988121b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc9880a4920 0x7fc988121b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 10 monclient: _finish_hunting 0 2024-06-11T19:17:49.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 1 monclient: found mon.noname-g 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 1 -- 172.21.15.98:0/1367045158 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc988122ef0 con 0x7fc9880a2530 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98ebdb640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc9880a5240 0x7fc988122070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98dbd9640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc9880a4920 0x7fc988121b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98e3da640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9880a2530 0x7fc9881215f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc98000e900 tx=0x7fc98000edc0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 10 client.?.objecter ms_handle_connect 0x7fc9880a2530 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc980017070 con 0x7fc9880a2530 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 10 monclient: dump: 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.228 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.229 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 10 monclient: _finish_auth 0 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc99063e640 5 monclient: authenticate success, global_id 94161 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc99063e640 10 monclient: _renew_subs 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc980013470 con 0x7fc9880a2530 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc99063e640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc99063e640 1 -- 172.21.15.98:0/1367045158 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9881235b0 con 0x7fc9880a2530 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc980013a90 con 0x7fc9880a2530 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:17:49.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc986ffd640 10 monclient: dump: 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.231 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc98cbd7640 4 set_mon_vals no callback set 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc99063e640 10 client.94161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc99063e640 10 monclient: _renew_subs 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc99063e640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc99063e640 1 -- 172.21.15.98:0/1367045158 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9881237d0 con 0x7fc9880a2530 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.208+0000 7fc99063e640 1 librados: init done 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc99063e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 <== mon.3 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fc98001a070 con 0x7fc9880a2530 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc99063e640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124840 0x7fc9881cda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc99063e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc99063e640 1 -- 172.21.15.98:0/1367045158 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc988050830 con 0x7fc988124840 2024-06-11T19:17:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc986ffd640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc97403fdb0 0x7fc974042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc98dbd9640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124840 0x7fc9881cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc98dbd9640 10 monclient: get_auth_request con 0x7fc988124840 auth_method 0 2024-06-11T19:17:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc98dbd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc98dbd9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc98dbd9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 <== mon.3 v2:172.21.15.117:3301/0 5 ==== osd_map(344..344 src has 1..344) ==== 5845+0+0 (secure 0 0 0) 0x7fc98000f040 con 0x7fc9880a2530 2024-06-11T19:17:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc986ffd640 10 client.94161.objecter ms_dispatch 0x7fc988000b90 osd_map(344..344 src has 1..344) 2024-06-11T19:17:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc986ffd640 3 client.94161.objecter handle_osd_map got epochs [344,344] > 0 2024-06-11T19:17:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.209+0000 7fc986ffd640 3 client.94161.objecter handle_osd_map decoding full epoch 344 2024-06-11T19:17:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc98dbd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc98dbd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc98dbd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 20 client.94161.objecter dump_active .. 0 homeless 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc98ebdb640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc97403fdb0 0x7fc974042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc98ebdb640 10 monclient: get_auth_request con 0x7fc97403fdb0 auth_method 0 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc98dbd9640 10 monclient(hunting): handle_auth_done global_id 23036 payload 824 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc98dbd9640 1 -- 172.21.15.98:0/1367045158 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9881c0580 con 0x7fc988124840 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc98dbd9640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124840 0x7fc9881cda80 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fc9880a1370 tx=0x7fc97c0077c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 10 client.94161.objecter ms_handle_connect 0x7fc988124840 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 10 client.94161.objecter resend_mon_ops 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 10 client.94161.objecter ms_handle_connect 0x7fc97403fdb0 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc98ebdb640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc97403fdb0 0x7fc974042270 secure :-1 s=READY pgs=4169 cs=0 l=1 rev1=1 crypto rx=0x7fc978035000 tx=0x7fc978008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc97c014a00 con 0x7fc988124840 2024-06-11T19:17:49.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 10 monclient: dump: 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.235 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.236 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc97c01a070 con 0x7fc988124840 2024-06-11T19:17:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124840 msgr2=0x7fc9881cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124840 0x7fc9881cda80 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fc9880a1370 tx=0x7fc97c0077c0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124840 msgr2=0x7fc9881cda80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:49.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.210+0000 7fc986ffd640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124840 0x7fc9881cda80 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc99063e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:49.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc99063e640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124c20 0x7fc9881cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc99063e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc99063e640 1 -- 172.21.15.98:0/1367045158 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc9880a66f0 con 0x7fc988124c20 2024-06-11T19:17:49.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc98dbd9640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124c20 0x7fc9881cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc98dbd9640 10 monclient: get_auth_request con 0x7fc988124c20 auth_method 0 2024-06-11T19:17:49.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc98dbd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc98dbd9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc98dbd9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc98dbd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc98dbd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc98dbd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc98dbd9640 10 monclient(hunting): handle_auth_done global_id 23045 payload 824 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc98dbd9640 1 -- 172.21.15.98:0/1367045158 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc988123300 con 0x7fc988124c20 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc98dbd9640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124c20 0x7fc9881cdfc0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fc9880a1370 tx=0x7fc97c004a40 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc986ffd640 10 client.94161.objecter ms_handle_connect 0x7fc988124c20 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.217+0000 7fc986ffd640 10 client.94161.objecter resend_mon_ops 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc97c015420 con 0x7fc988124c20 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc986ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc986ffd640 10 monclient: dump: 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.239 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.240 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7fc97c014b70 con 0x7fc988124c20 2024-06-11T19:17:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc986ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc986ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124c20 msgr2=0x7fc9881cdfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc986ffd640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124c20 0x7fc9881cdfc0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fc9880a1370 tx=0x7fc97c004a40 comp rx=0 tx=0).stop 2024-06-11T19:17:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc986ffd640 1 -- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124c20 msgr2=0x7fc9881cdfc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:49.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc986ffd640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc988124c20 0x7fc9881cdfc0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.241 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 519412, 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:49.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc99063e640 10 librados: watch_flush enter 2024-06-11T19:17:49.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc99063e640 10 librados: watch_flush exit 2024-06-11T19:17:49.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc99063e640 20 client.94161.objecter shutdown clearing up homeless session... 2024-06-11T19:17:49.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc99063e640 10 client.94161.objecter successfully canceled tick 2024-06-11T19:17:49.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc99063e640 1 -- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc97403fdb0 msgr2=0x7fc974042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc99063e640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc97403fdb0 0x7fc974042270 secure :-1 s=READY pgs=4169 cs=0 l=1 rev1=1 crypto rx=0x7fc978035000 tx=0x7fc978008030 comp rx=0 tx=0).stop 2024-06-11T19:17:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc99063e640 10 monclient: shutdown 2024-06-11T19:17:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc99063e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc99063e640 1 -- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9880a2530 msgr2=0x7fc9881215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc99063e640 1 --2- 172.21.15.98:0/1367045158 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9880a2530 0x7fc9881215f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc98000e900 tx=0x7fc98000edc0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc98ebdb640 1 -- 172.21.15.98:0/1367045158 reap_dead start 2024-06-11T19:17:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc98ebdb640 1 -- 172.21.15.98:0/1367045158 reap_dead start 2024-06-11T19:17:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc99063e640 1 -- 172.21.15.98:0/1367045158 shutdown_connections 2024-06-11T19:17:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.218+0000 7fc99063e640 1 -- 172.21.15.98:0/1367045158 >> 172.21.15.98:0/1367045158 conn(0x7fc9880ab850 msgr2=0x7fc9880a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.219+0000 7fc99063e640 1 -- 172.21.15.98:0/1367045158 shutdown_connections 2024-06-11T19:17:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.219+0000 7fc99063e640 1 -- 172.21.15.98:0/1367045158 wait complete. 2024-06-11T19:17:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.219+0000 7fc99063e640 1 librados: shutdown 2024-06-11T19:17:49.267 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:49.267 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:17:49.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:17:49.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.367+0000 7f062340a640 10 monclient: get_monmap_and_config 2024-06-11T19:17:49.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.367+0000 7f062340a640 10 monclient: build_initial_monmap 2024-06-11T19:17:49.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.367+0000 7f062340a640 10 monclient: monmap: 2024-06-11T19:17:49.366 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:49.366 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.366 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:49.368249+0000 2024-06-11T19:17:49.366 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:49.368249+0000 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:49.367 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.367+0000 7f062340a640 1 Processor -- start 2024-06-11T19:17:49.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.367+0000 7f062340a640 1 -- start start 2024-06-11T19:17:49.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.367+0000 7f062340a640 10 monclient: init 2024-06-11T19:17:49.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:49.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 10 monclient: _add_conns ranks=[9,6,4,10,5,8,7,2,3,0,1] 2024-06-11T19:17:49.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f061c136c90 0x7f061c137070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 10 monclient(hunting): picked mon.noname-a con 0x7f061c136c90 addr v2:172.21.15.98:3300/0 2024-06-11T19:17:49.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c1367d0 0x7f061c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 10 monclient(hunting): picked mon.noname-d con 0x7f061c1367d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:49.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f061c0a3bd0 0x7f061c0a4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 10 monclient(hunting): picked mon.noname-c con 0x7f061c0a3bd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f061c068180 con 0x7f061c136c90 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f061c0a0270 con 0x7f061c0a3bd0 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f061c09fc20 con 0x7f061c1367d0 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f062340a640 10 monclient(hunting): authenticate will time out at 1279.175484s 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06211a6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f061c136c90 0x7f061c137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c1367d0 0x7f061c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c1367d0 0x7f061c0a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:33146/0 (socket says 172.21.15.98:33146) 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06211a6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f061c136c90 0x7f061c137070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56646/0 (socket says 172.21.15.98:56646) 2024-06-11T19:17:49.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 1 -- 172.21.15.98:0/1823282667 learned_addr learned my addr 172.21.15.98:0/1823282667 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:49.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 10 monclient(hunting): get_auth_request con 0x7f061c1367d0 auth_method 0 2024-06-11T19:17:49.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06219a7640 1 --2- 172.21.15.98:0/1823282667 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f061c0a3bd0 0x7f061c0a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06211a6640 10 monclient(hunting): get_auth_request con 0x7f061c136c90 auth_method 0 2024-06-11T19:17:49.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06211a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06211a6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06211a6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06219a7640 10 monclient(hunting): get_auth_request con 0x7f061c0a3bd0 auth_method 0 2024-06-11T19:17:49.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06219a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06219a7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06219a7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06219a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06219a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06219a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06211a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06211a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06211a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 10 monclient(hunting): handle_auth_done global_id 55131 payload 274 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 1 -- 172.21.15.98:0/1823282667 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f061c0a3bd0 msgr2=0x7f061c0a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:49.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 1 --2- 172.21.15.98:0/1823282667 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f061c0a3bd0 0x7f061c0a4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 1 -- 172.21.15.98:0/1823282667 >> v2:172.21.15.98:3300/0 conn(0x7f061c136c90 msgr2=0x7f061c137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 1 --2- 172.21.15.98:0/1823282667 >> v2:172.21.15.98:3300/0 conn(0x7f061c136c90 0x7f061c137070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 10 monclient: _finish_hunting 0 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 1 monclient: found mon.noname-d 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06209a5640 1 -- 172.21.15.98:0/1823282667 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c09ff00 con 0x7f061c1367d0 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06219a7640 1 --2- 172.21.15.98:0/1823282667 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f061c0a3bd0 0x7f061c0a4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.368+0000 7f06211a6640 1 --2- 172.21.15.98:0/1823282667 >> v2:172.21.15.98:3300/0 conn(0x7f061c136c90 0x7f061c137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f06209a5640 1 --2- 172.21.15.98:0/1823282667 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c1367d0 0x7f061c0a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f060c0098a0 tx=0x7f060c02b840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=919dcec3d86e7191 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 1 -- 172.21.15.98:0/1823282667 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f060c031a10 con 0x7f061c1367d0 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:17:49.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 10 monclient: dump: 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.382 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 10 monclient: _finish_auth 0 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 10 monclient: _check_auth_tickets 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f062340a640 5 monclient: authenticate success, global_id 55131 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 1 -- 172.21.15.98:0/1823282667 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060c037070 con 0x7f061c1367d0 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f062340a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f062340a640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 1 -- 172.21.15.98:0/1823282667 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f060c031c10 con 0x7f061c1367d0 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f062340a640 4 set_mon_vals no callback set 2024-06-11T19:17:49.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.369+0000 7f0613fff640 10 monclient: dump: 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.384 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.370+0000 7f062340a640 10 monclient: shutdown 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.370+0000 7f062340a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.370+0000 7f062340a640 1 -- 172.21.15.98:0/1823282667 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c1367d0 msgr2=0x7f061c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.370+0000 7f062340a640 1 --2- 172.21.15.98:0/1823282667 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c1367d0 0x7f061c0a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f060c0098a0 tx=0x7f060c02b840 comp rx=0 tx=0).stop 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.370+0000 7f062340a640 1 -- 172.21.15.98:0/1823282667 shutdown_connections 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.370+0000 7f062340a640 1 --2- 172.21.15.98:0/1823282667 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f061c0a3bd0 0x7f061c0a4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.370+0000 7f062340a640 1 --2- 172.21.15.98:0/1823282667 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c1367d0 0x7f061c0a3690 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.370+0000 7f062340a640 1 --2- 172.21.15.98:0/1823282667 >> v2:172.21.15.98:3300/0 conn(0x7f061c136c90 0x7f061c137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.370+0000 7f062340a640 1 -- 172.21.15.98:0/1823282667 >> 172.21.15.98:0/1823282667 conn(0x7f061c0a4780 msgr2=0x7f061c0a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.370+0000 7f062340a640 1 -- 172.21.15.98:0/1823282667 shutdown_connections 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.370+0000 7f062340a640 1 -- 172.21.15.98:0/1823282667 wait complete. 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient: build_initial_monmap 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient: monmap: 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:49.386 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 1 librados: starting msgr at 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 1 librados: starting objecter 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 1 Processor -- start 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 1 -- start start 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 1 librados: setting wanted keys 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 1 librados: calling monclient init 2024-06-11T19:17:49.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient: init 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient: _add_conns ranks=[2,7,6,1,8,4,0,5,3] 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f061c0a3bd0 0x7f061c1cda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient(hunting): picked mon.noname-b con 0x7f061c0a3bd0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c1367d0 0x7f061c1cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient(hunting): picked mon.noname-i con 0x7f061c1367d0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c136c90 0x7f061c09c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient(hunting): picked mon.noname-d con 0x7f061c136c90 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f061c1375b0 con 0x7f061c0a3bd0 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f061c068560 con 0x7f061c136c90 2024-06-11T19:17:49.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f061c1255a0 con 0x7f061c1367d0 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.371+0000 7f062340a640 10 monclient(hunting): authenticate will time out at 1279.178940s 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f061c0a3bd0 0x7f061c1cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06219a7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c136c90 0x7f061c09c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f061c0a3bd0 0x7f061c1cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45196/0 (socket says 172.21.15.98:45196) 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06219a7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c136c90 0x7f061c09c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:33154/0 (socket says 172.21.15.98:33154) 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 1 -- 172.21.15.98:0/3210705502 learned_addr learned my addr 172.21.15.98:0/3210705502 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 10 monclient(hunting): get_auth_request con 0x7f061c0a3bd0 auth_method 0 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06219a7640 10 monclient(hunting): get_auth_request con 0x7f061c136c90 auth_method 0 2024-06-11T19:17:49.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06219a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06219a7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06219a7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06219a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06219a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06219a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06209a5640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c1367d0 0x7f061c1cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06209a5640 10 monclient(hunting): get_auth_request con 0x7f061c1367d0 auth_method 0 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06209a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06209a5640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06209a5640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 10 monclient(hunting): handle_auth_done global_id 84413 payload 824 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 1 -- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c1367d0 msgr2=0x7f061c1cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c1367d0 0x7f061c1cdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 1 -- 172.21.15.98:0/3210705502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c136c90 msgr2=0x7f061c09c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c136c90 0x7f061c09c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 10 monclient: _finish_hunting 0 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 1 monclient: found mon.noname-b 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 1 -- 172.21.15.98:0/3210705502 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c126d30 con 0x7f061c0a3bd0 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06219a7640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f061c136c90 0x7f061c09c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06211a6640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f061c0a3bd0 0x7f061c1cda80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f061800e990 tx=0x7f061800ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 10 client.?.objecter ms_handle_connect 0x7f061c0a3bd0 2024-06-11T19:17:49.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0618017070 con 0x7f061c0a3bd0 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 10 monclient: dump: 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.392 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 10 monclient: _finish_auth 0 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f061800cdb0 con 0x7f061c0a3bd0 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f062340a640 5 monclient: authenticate success, global_id 84413 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:49.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06180138e0 con 0x7f061c0a3bd0 2024-06-11T19:17:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f062340a640 10 monclient: _renew_subs 2024-06-11T19:17:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f062340a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:17:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f062340a640 1 -- 172.21.15.98:0/3210705502 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f061c0a7850 con 0x7f061c0a3bd0 2024-06-11T19:17:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f06137fe640 4 set_mon_vals no callback set 2024-06-11T19:17:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:17:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f0611ffb640 10 monclient: dump: 2024-06-11T19:17:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.394 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f062340a640 10 client.84413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f062340a640 10 monclient: _renew_subs 2024-06-11T19:17:49.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f062340a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f062340a640 1 -- 172.21.15.98:0/3210705502 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f061c09e630 con 0x7f061c0a3bd0 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.372+0000 7f062340a640 1 librados: init done 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f061801a070 con 0x7f061c0a3bd0 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f062340a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f062340a640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c1266d0 0x7f061c09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f062340a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f062340a640 1 -- 172.21.15.98:0/3210705502 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f061c137730 con 0x7f061c1266d0 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f0611ffb640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f060003fdb0 0x7f0600042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f06209a5640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c1266d0 0x7f061c09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f06209a5640 10 monclient: get_auth_request con 0x7f061c1266d0 auth_method 0 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f06209a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f06209a5640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f06209a5640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(344..344 src has 1..344) ==== 5845+0+0 (secure 0 0 0) 0x7f061800f040 con 0x7f061c0a3bd0 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f0611ffb640 10 client.84413.objecter ms_dispatch 0x7f061c000b90 osd_map(344..344 src has 1..344) 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f0611ffb640 3 client.84413.objecter handle_osd_map got epochs [344,344] > 0 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f0611ffb640 3 client.84413.objecter handle_osd_map decoding full epoch 344 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f06209a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f06209a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f06209a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.373+0000 7f0611ffb640 20 client.84413.objecter dump_active .. 0 homeless 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f06219a7640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f060003fdb0 0x7f0600042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f06219a7640 10 monclient: get_auth_request con 0x7f060003fdb0 auth_method 0 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f06209a5640 10 monclient(hunting): handle_auth_done global_id 84283 payload 824 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f06209a5640 1 -- 172.21.15.98:0/3210705502 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f061c09f2a0 con 0x7f061c1266d0 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f06209a5640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c1266d0 0x7f061c09cf20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f061c0a5b50 tx=0x7f060c0046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 10 client.84413.objecter ms_handle_connect 0x7f061c1266d0 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 10 client.84413.objecter resend_mon_ops 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 10 client.84413.objecter ms_handle_connect 0x7f060003fdb0 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f06219a7640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f060003fdb0 0x7f0600042270 secure :-1 s=READY pgs=4170 cs=0 l=1 rev1=1 crypto rx=0x7f0614003c00 tx=0x7f061400b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f060c003c90 con 0x7f061c1266d0 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 10 monclient: dump: 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.398 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f060c037040 con 0x7f061c1266d0 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c1266d0 msgr2=0x7f061c09cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c1266d0 0x7f061c09cf20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f061c0a5b50 tx=0x7f060c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c1266d0 msgr2=0x7f061c09cf20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:49.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.374+0000 7f0611ffb640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c1266d0 0x7f061c09cf20 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f061c0a5b50 tx=0x7f060c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f062340a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:49.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f062340a640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c09d460 0x7f061c09d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f062340a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f062340a640 1 -- 172.21.15.98:0/3210705502 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f061c050830 con 0x7f061c09d460 2024-06-11T19:17:49.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f06209a5640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c09d460 0x7f061c09d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f06209a5640 10 monclient: get_auth_request con 0x7f061c09d460 auth_method 0 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f06209a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f06209a5640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f06209a5640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f06209a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f06209a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f06209a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f06209a5640 10 monclient(hunting): handle_auth_done global_id 84292 payload 824 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f06209a5640 1 -- 172.21.15.98:0/3210705502 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f061c09dd80 con 0x7f061c09d460 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f06209a5640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c09d460 0x7f061c09d840 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f060c0097c0 tx=0x7f060c031000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f0611ffb640 10 client.84413.objecter ms_handle_connect 0x7f061c09d460 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.381+0000 7f0611ffb640 10 client.84413.objecter resend_mon_ops 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f060c04c2b0 con 0x7f061c09d460 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f0611ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f0611ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f0611ffb640 10 monclient: dump: 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.403 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.404 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f060c04c2b0 con 0x7f061c09d460 2024-06-11T19:17:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f0611ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f0611ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c09d460 msgr2=0x7f061c09d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f0611ffb640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c09d460 0x7f061c09d840 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f060c0097c0 tx=0x7f060c031000 comp rx=0 tx=0).stop 2024-06-11T19:17:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f0611ffb640 1 -- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c09d460 msgr2=0x7f061c09d840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:49.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f0611ffb640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f061c09d460 0x7f061c09d840 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f060c0097c0 tx=0x7f060c031000 comp rx=0 tx=0).stop 2024-06-11T19:17:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 141152, 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:49.427 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:49.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f062340a640 10 librados: watch_flush enter 2024-06-11T19:17:49.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f062340a640 10 librados: watch_flush exit 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f062340a640 20 client.84413.objecter shutdown clearing up homeless session... 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f062340a640 10 client.84413.objecter successfully canceled tick 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f062340a640 1 -- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f060003fdb0 msgr2=0x7f0600042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f062340a640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f060003fdb0 0x7f0600042270 secure :-1 s=READY pgs=4170 cs=0 l=1 rev1=1 crypto rx=0x7f0614003c00 tx=0x7f061400b040 comp rx=0 tx=0).stop 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f062340a640 10 monclient: shutdown 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f062340a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f062340a640 1 -- 172.21.15.98:0/3210705502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f061c0a3bd0 msgr2=0x7f061c1cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f062340a640 1 --2- 172.21.15.98:0/3210705502 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f061c0a3bd0 0x7f061c1cda80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f061800e990 tx=0x7f061800ed90 comp rx=0 tx=0).stop 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.382+0000 7f06219a7640 1 -- 172.21.15.98:0/3210705502 reap_dead start 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.383+0000 7f06219a7640 1 -- 172.21.15.98:0/3210705502 reap_dead start 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.383+0000 7f062340a640 1 -- 172.21.15.98:0/3210705502 shutdown_connections 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.383+0000 7f062340a640 1 -- 172.21.15.98:0/3210705502 >> 172.21.15.98:0/3210705502 conn(0x7f061c0a4780 msgr2=0x7f061c1bc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:49.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.383+0000 7f062340a640 1 -- 172.21.15.98:0/3210705502 shutdown_connections 2024-06-11T19:17:49.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.383+0000 7f062340a640 1 -- 172.21.15.98:0/3210705502 wait complete. 2024-06-11T19:17:49.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.383+0000 7f062340a640 1 librados: shutdown 2024-06-11T19:17:49.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:49.430 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-06-11T19:17:49.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-06-11T19:17:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.529+0000 7fc12e4f1640 10 monclient: get_monmap_and_config 2024-06-11T19:17:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.529+0000 7fc12e4f1640 10 monclient: build_initial_monmap 2024-06-11T19:17:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.529+0000 7fc12e4f1640 10 monclient: monmap: 2024-06-11T19:17:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:49.530222+0000 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:49.530222+0000 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:49.530 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.529+0000 7fc12e4f1640 1 Processor -- start 2024-06-11T19:17:49.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 1 -- start start 2024-06-11T19:17:49.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 10 monclient: init 2024-06-11T19:17:49.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:49.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 10 monclient: _add_conns ranks=[9,1,10,0,7,5,3,2,8,6,4] 2024-06-11T19:17:49.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc1280a52a0 0x7fc1280a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 10 monclient(hunting): picked mon.noname-a con 0x7fc1280a52a0 addr v2:172.21.15.98:3300/0 2024-06-11T19:17:49.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc12809d520 addr v1:172.21.15.117:6789/0 2024-06-11T19:17:49.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc128073830 0x7fc1280a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 10 monclient(hunting): picked mon.noname-f con 0x7fc128073830 addr v2:172.21.15.117:3300/0 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc12809d2e0 con 0x7fc12809d520 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc12809daa0 con 0x7fc1280a52a0 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fc1280a5680 con 0x7fc128073830 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12e4f1640 10 monclient(hunting): authenticate will time out at 1279.337505s 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc1280a52a0 0x7fc1280a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc1280a52a0 0x7fc1280a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56648/0 (socket says 172.21.15.98:56648) 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 1 -- 172.21.15.98:0/1473311692 learned_addr learned my addr 172.21.15.98:0/1473311692 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 10 monclient(hunting): get_auth_request con 0x7fc1280a52a0 auth_method 0 2024-06-11T19:17:49.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12ca8e640 1 --2- 172.21.15.98:0/1473311692 >> v2:172.21.15.117:3300/0 conn(0x7fc128073830 0x7fc1280a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 10 monclient(hunting): handle_auth_done global_id 64998 payload 274 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 1 -- 172.21.15.98:0/1473311692 >> v2:172.21.15.117:3300/0 conn(0x7fc128073830 msgr2=0x7fc1280a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 1 --2- 172.21.15.98:0/1473311692 >> v2:172.21.15.117:3300/0 conn(0x7fc128073830 0x7fc1280a3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 1 -- 172.21.15.98:0/1473311692 >> v1:172.21.15.117:6789/0 conn(0x7fc12809d520 legacy=0x7fc1280a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 1 monclient: found mon.noname-a 2024-06-11T19:17:49.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc127fff640 1 -- 172.21.15.98:0/1473311692 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc12809e9d0 con 0x7fc1280a52a0 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12ca8e640 10 monclient: get_auth_request con 0x7fc128073830 auth_method 0 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.530+0000 7fc12ca8e640 1 --2- 172.21.15.98:0/1473311692 >> v2:172.21.15.117:3300/0 conn(0x7fc128073830 0x7fc1280a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc127fff640 1 --2- 172.21.15.98:0/1473311692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1280a52a0 0x7fc1280a1910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc12809ca00 tx=0x7fc11802dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cd92120b2d1dd2 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 1 -- 172.21.15.98:0/1473311692 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc11803b070 con 0x7fc1280a52a0 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 10 monclient: dump: 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.541 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 10 monclient: _finish_auth 0 2024-06-11T19:17:49.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 1 -- 172.21.15.98:0/1473311692 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc118034340 con 0x7fc1280a52a0 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 1 -- 172.21.15.98:0/1473311692 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc1180349a0 con 0x7fc1280a52a0 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc12e4f1640 5 monclient: authenticate success, global_id 64998 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc12e4f1640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc12e4f1640 4 set_mon_vals no callback set 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.531+0000 7fc126ffd640 10 monclient: dump: 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.543 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.544 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.532+0000 7fc12e4f1640 10 monclient: shutdown 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.532+0000 7fc12e4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.532+0000 7fc12e4f1640 1 -- 172.21.15.98:0/1473311692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1280a52a0 msgr2=0x7fc1280a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.532+0000 7fc12e4f1640 1 --2- 172.21.15.98:0/1473311692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1280a52a0 0x7fc1280a1910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc12809ca00 tx=0x7fc11802dfc0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.532+0000 7fc12e4f1640 1 -- 172.21.15.98:0/1473311692 shutdown_connections 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.532+0000 7fc12e4f1640 1 --2- 172.21.15.98:0/1473311692 >> v2:172.21.15.117:3300/0 conn(0x7fc128073830 0x7fc1280a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.532+0000 7fc12e4f1640 1 --2- 172.21.15.98:0/1473311692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1280a52a0 0x7fc1280a1910 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.532+0000 7fc12e4f1640 1 -- 172.21.15.98:0/1473311692 >> 172.21.15.98:0/1473311692 conn(0x7fc1280a4780 msgr2=0x7fc1280a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.532+0000 7fc12e4f1640 1 -- 172.21.15.98:0/1473311692 shutdown_connections 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.532+0000 7fc12e4f1640 1 -- 172.21.15.98:0/1473311692 wait complete. 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.532+0000 7fc12e4f1640 10 monclient: build_initial_monmap 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.532+0000 7fc12e4f1640 10 monclient: monmap: 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:49.545 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:49.546 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.533+0000 7fc12e4f1640 1 librados: starting msgr at 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.533+0000 7fc12e4f1640 1 librados: starting objecter 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.533+0000 7fc12e4f1640 1 Processor -- start 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.533+0000 7fc12e4f1640 1 -- start start 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.533+0000 7fc12e4f1640 1 librados: setting wanted keys 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.533+0000 7fc12e4f1640 1 librados: calling monclient init 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.533+0000 7fc12e4f1640 10 monclient: init 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.533+0000 7fc12e4f1640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.533+0000 7fc12e4f1640 10 monclient: _add_conns ranks=[0,3,1,8,4,7,5,6,2] 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc128073830 0x7fc1281c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 monclient(hunting): picked mon.noname-a con 0x7fc128073830 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc12809d520 0x7fc1281c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 monclient(hunting): picked mon.noname-g con 0x7fc12809d520 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1280a52a0 0x7fc12811fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 monclient(hunting): picked mon.noname-f con 0x7fc1280a52a0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc1280a4460 con 0x7fc128073830 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc1280a42e0 con 0x7fc1280a52a0 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fc1280a1e50 con 0x7fc12809d520 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 monclient(hunting): authenticate will time out at 1279.341091s 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc128073830 0x7fc1281c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc128073830 0x7fc1281c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56658/0 (socket says 172.21.15.98:56658) 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 1 -- 172.21.15.98:0/367899039 learned_addr learned my addr 172.21.15.98:0/367899039 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 10 monclient(hunting): get_auth_request con 0x7fc128073830 auth_method 0 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12ca8e640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1280a52a0 0x7fc12811fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12ca8e640 10 monclient(hunting): get_auth_request con 0x7fc1280a52a0 auth_method 0 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12ca8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12ca8e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12ca8e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc1277fe640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc12809d520 0x7fc1281c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc1277fe640 10 monclient(hunting): get_auth_request con 0x7fc12809d520 auth_method 0 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc1277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:49.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc1277fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc1277fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 10 monclient(hunting): handle_auth_done global_id 65007 payload 824 2024-06-11T19:17:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 1 -- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc12809d520 msgr2=0x7fc1281c9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc12809d520 0x7fc1281c9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 1 -- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1280a52a0 msgr2=0x7fc12811fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1280a52a0 0x7fc12811fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 1 monclient: found mon.noname-a 2024-06-11T19:17:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:17:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 1 -- 172.21.15.98:0/367899039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1280a1600 con 0x7fc128073830 2024-06-11T19:17:49.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12ca8e640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1280a52a0 0x7fc12811fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc1277fe640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc12809d520 0x7fc1281c9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc127fff640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc128073830 0x7fc1281c9950 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc1180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 10 client.?.objecter ms_handle_connect 0x7fc128073830 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 1 -- 172.21.15.98:0/367899039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc11803b040 con 0x7fc128073830 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 10 monclient: dump: 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.551 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 10 monclient: _finish_auth 0 2024-06-11T19:17:49.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 5 monclient: authenticate success, global_id 65007 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 1 -- 172.21.15.98:0/367899039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc118046070 con 0x7fc128073830 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 monclient: _renew_subs 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 1 -- 172.21.15.98:0/367899039 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc118003c80 con 0x7fc128073830 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 1 -- 172.21.15.98:0/367899039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc128123a80 con 0x7fc128073830 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc1267fc640 4 set_mon_vals no callback set 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 client.65007.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:49.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc124ff9640 10 monclient: dump: 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:49.554 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 monclient: _renew_subs 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 1 -- 172.21.15.98:0/367899039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1281234c0 con 0x7fc128073830 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.534+0000 7fc12e4f1640 1 librados: init done 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.535+0000 7fc124ff9640 1 -- 172.21.15.98:0/367899039 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fc118003810 con 0x7fc128073830 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.535+0000 7fc124ff9640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc11403fdb0 0x7fc114042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.535+0000 7fc12e4f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.535+0000 7fc12e4f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.535+0000 7fc12e4f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:17:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.535+0000 7fc124ff9640 1 -- 172.21.15.98:0/367899039 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(344..344 src has 1..344) ==== 5845+0+0 (secure 0 0 0) 0x7fc118076180 con 0x7fc128073830 2024-06-11T19:17:49.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.535+0000 7fc12e4f1640 1 -- 172.21.15.98:0/367899039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1280a1910 con 0x7fc128073830 2024-06-11T19:17:49.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.535+0000 7fc124ff9640 10 client.65007.objecter ms_dispatch 0x7fc128000b90 osd_map(344..344 src has 1..344) 2024-06-11T19:17:49.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.535+0000 7fc124ff9640 3 client.65007.objecter handle_osd_map got epochs [344,344] > 0 2024-06-11T19:17:49.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.535+0000 7fc124ff9640 3 client.65007.objecter handle_osd_map decoding full epoch 344 2024-06-11T19:17:49.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.535+0000 7fc124ff9640 20 client.65007.objecter dump_active .. 0 homeless 2024-06-11T19:17:49.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.536+0000 7fc1277fe640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc11403fdb0 0x7fc114042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:49.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.536+0000 7fc1277fe640 10 monclient: get_auth_request con 0x7fc11403fdb0 auth_method 0 2024-06-11T19:17:49.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.536+0000 7fc124ff9640 10 client.65007.objecter ms_handle_connect 0x7fc11403fdb0 2024-06-11T19:17:49.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.536+0000 7fc1277fe640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc11403fdb0 0x7fc114042270 secure :-1 s=READY pgs=4171 cs=0 l=1 rev1=1 crypto rx=0x7fc128121220 tx=0x7fc11c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:49.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.540+0000 7fc124ff9640 1 -- 172.21.15.98:0/367899039 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc1280a1600 con 0x7fc128073830 2024-06-11T19:17:49.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.540+0000 7fc124ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:49.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.540+0000 7fc124ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.801+0000 7fc12e4f1640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-06-11T19:17:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.801+0000 7fc12e4f1640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-06-11T19:17:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.801+0000 7fc12e4f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:17:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.801+0000 7fc12e4f1640 1 -- 172.21.15.98:0/367899039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fc128122630 con 0x7fc128073830 2024-06-11T19:17:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.803+0000 7fc124ff9640 1 -- 172.21.15.98:0/367899039 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fc118040d70 con 0x7fc128073830 2024-06-11T19:17:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.803+0000 7fc124ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-06-11T19:17:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.803+0000 7fc124ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:49.804 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.803+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.805 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.803+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=1935681255,config=692619927,health=4226598906,kv=2000936593,logm=3430511090}) 2024-06-11T19:17:49.805 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=10,logm=61} crc {auth=2151276440,config=692619927,health=3162157963,kv=126839869,logm=540262123}) 2024-06-11T19:17:49.805 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.805 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=1935681255,config=692619927,health=4226598906,kv=2000936593,logm=3430511090}) 2024-06-11T19:17:49.805 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=1,health=11,kv=10,logm=62} crc {auth=499469526,config=3678701417,health=3090289264,kv=2000936593,logm=4015961894}) 2024-06-11T19:17:49.805 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.805 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=1935681255,config=692619927,health=4226598906,kv=2000936593,logm=3430511090}) 2024-06-11T19:17:49.805 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=16,config=2,health=10,kv=10,logm=62} crc {auth=3233391901,config=692619927,health=1620792997,kv=2236601199,logm=1901968326}) 2024-06-11T19:17:49.805 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.805 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=1935681255,config=692619927,health=4226598906,kv=2000936593,logm=3430511090}) 2024-06-11T19:17:49.805 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=17,config=2,health=10,kv=9,logm=62} crc {auth=3731324668,config=692619927,health=2078942188,kv=1820642019,logm=1079934826}) 2024-06-11T19:17:49.805 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.806 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=1935681255,config=692619927,health=4226598906,kv=2000936593,logm=3430511090}) 2024-06-11T19:17:49.806 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=14,config=2,health=11,kv=10,logm=63} crc {auth=1866570688,config=692619927,health=3090289264,kv=2000936593,logm=3071253483}) 2024-06-11T19:17:49.806 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.806 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=1935681255,config=692619927,health=4226598906,kv=2000936593,logm=3430511090}) 2024-06-11T19:17:49.806 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=15,config=2,health=12,kv=8,logm=63} crc {auth=47661736,config=692619927,health=2897991795,kv=1720013064,logm=1327877704}) 2024-06-11T19:17:49.806 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.806 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=1935681255,config=692619927,health=4226598906,kv=2000936593,logm=3430511090}) 2024-06-11T19:17:49.806 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=15,config=2,health=11,kv=9,logm=63} crc {auth=8253832,config=692619927,health=3712199308,kv=3604117662,logm=3203832796}) 2024-06-11T19:17:49.806 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.806 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=1935681255,config=692619927,health=4226598906,kv=2000936593,logm=3430511090}) 2024-06-11T19:17:49.806 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.804+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=3521718562,config=692619927,health=4226598906,kv=2000936593,logm=4247200949}) 2024-06-11T19:17:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.805+0000 7fc12e4f1640 10 librados: watch_flush enter 2024-06-11T19:17:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.805+0000 7fc12e4f1640 10 librados: watch_flush exit 2024-06-11T19:17:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.805+0000 7fc12e4f1640 20 client.65007.objecter shutdown clearing up homeless session... 2024-06-11T19:17:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.805+0000 7fc12e4f1640 10 client.65007.objecter successfully canceled tick 2024-06-11T19:17:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 -- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc11403fdb0 msgr2=0x7fc114042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc11403fdb0 0x7fc114042270 secure :-1 s=READY pgs=4171 cs=0 l=1 rev1=1 crypto rx=0x7fc128121220 tx=0x7fc11c009040 comp rx=0 tx=0).stop 2024-06-11T19:17:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 10 monclient: shutdown 2024-06-11T19:17:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 -- 172.21.15.98:0/367899039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc128073830 msgr2=0x7fc1281c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc128073830 0x7fc1281c9950 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc1180046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 -- 172.21.15.98:0/367899039 shutdown_connections 2024-06-11T19:17:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc11403fdb0 0x7fc114042270 unknown :-1 s=CLOSED pgs=4171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1280a52a0 0x7fc12811fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc12809d520 0x7fc1281c9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 --2- 172.21.15.98:0/367899039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc128073830 0x7fc1281c9950 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 -- 172.21.15.98:0/367899039 >> 172.21.15.98:0/367899039 conn(0x7fc1280a4780 msgr2=0x7fc12809f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 -- 172.21.15.98:0/367899039 shutdown_connections 2024-06-11T19:17:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 -- 172.21.15.98:0/367899039 wait complete. 2024-06-11T19:17:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:49.806+0000 7fc12e4f1640 1 librados: shutdown 2024-06-11T19:17:49.821 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-06-11T19:17:49.921 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.921 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122054023}) 2024-06-11T19:17:49.921 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3869773994}) 2024-06-11T19:17:49.921 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.921 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122054023}) 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3662923631}) 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122054023}) 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=194240384}) 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122054023}) 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=268945339}) 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122054023}) 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2694946050}) 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122054023}) 2024-06-11T19:17:49.922 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2048714804}) 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122054023}) 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=762339444}) 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122054023}) 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.806+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3827038251}) 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615406841}) 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502105669}) 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615406841}) 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1866492432}) 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.923 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615406841}) 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=180355129}) 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615406841}) 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4181921691}) 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615406841}) 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2741949465}) 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615406841}) 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=266127170}) 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615406841}) 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1356441111}) 2024-06-11T19:17:49.924 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615406841}) 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.808+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1393051033}) 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410811317}) 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888729932}) 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410811317}) 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152364564}) 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410811317}) 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3852227209}) 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.925 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410811317}) 2024-06-11T19:17:49.926 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1385745442}) 2024-06-11T19:17:49.926 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.926 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410811317}) 2024-06-11T19:17:49.926 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3802287000}) 2024-06-11T19:17:49.926 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.926 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410811317}) 2024-06-11T19:17:49.926 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2174385250}) 2024-06-11T19:17:49.926 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.926 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410811317}) 2024-06-11T19:17:49.926 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=364731611}) 2024-06-11T19:17:49.926 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410811317}) 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.810+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1538989565}) 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488427881}) 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463171900}) 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488427881}) 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1298074543}) 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488427881}) 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1283887712}) 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.927 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488427881}) 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2148896391}) 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488427881}) 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1039640902}) 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488427881}) 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2004960393}) 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488427881}) 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2783052675}) 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488427881}) 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.812+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3142547206}) 2024-06-11T19:17:49.928 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710323351}) 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=827317975}) 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710323351}) 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=567123716}) 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710323351}) 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=573669947}) 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710323351}) 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3082531169}) 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710323351}) 2024-06-11T19:17:49.929 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2817125168}) 2024-06-11T19:17:49.930 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.930 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710323351}) 2024-06-11T19:17:49.930 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=974276408}) 2024-06-11T19:17:49.930 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.930 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710323351}) 2024-06-11T19:17:49.930 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=260273263}) 2024-06-11T19:17:49.930 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.930 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710323351}) 2024-06-11T19:17:49.930 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.813+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2085366433}) 2024-06-11T19:17:49.930 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.930 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382964753}) 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3347327411}) 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382964753}) 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011196734}) 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382964753}) 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1739587622}) 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382964753}) 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3274602398}) 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382964753}) 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1260702822}) 2024-06-11T19:17:49.931 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382964753}) 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2895059122}) 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382964753}) 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2147135607}) 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382964753}) 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.815+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=688383355}) 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603112353}) 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337905073}) 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603112353}) 2024-06-11T19:17:49.932 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1884549787}) 2024-06-11T19:17:49.933 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.933 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603112353}) 2024-06-11T19:17:49.933 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2819015027}) 2024-06-11T19:17:49.933 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.933 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603112353}) 2024-06-11T19:17:49.933 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4283814609}) 2024-06-11T19:17:49.933 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.933 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603112353}) 2024-06-11T19:17:49.933 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=932381759}) 2024-06-11T19:17:49.933 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.933 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603112353}) 2024-06-11T19:17:49.933 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4056691986}) 2024-06-11T19:17:49.934 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.934 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603112353}) 2024-06-11T19:17:49.934 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3069272696}) 2024-06-11T19:17:49.934 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.934 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603112353}) 2024-06-11T19:17:49.934 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.816+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3659889724}) 2024-06-11T19:17:49.934 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.817+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.934 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.817+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983953176}) 2024-06-11T19:17:49.934 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.817+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2900702246}) 2024-06-11T19:17:49.934 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.817+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.934 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.817+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983953176}) 2024-06-11T19:17:49.934 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.817+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3168027749}) 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.817+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.817+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983953176}) 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.817+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=967657576}) 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.817+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.817+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983953176}) 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.817+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2350905916}) 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.818+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.818+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983953176}) 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.818+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=243895903}) 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.818+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.818+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983953176}) 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.818+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=311149607}) 2024-06-11T19:17:49.935 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.818+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.818+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983953176}) 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.818+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3692157648}) 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.818+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.818+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983953176}) 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.818+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2294008665}) 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=867750103}) 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=868413357}) 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=867750103}) 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=873839439}) 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=867750103}) 2024-06-11T19:17:49.936 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3450024665}) 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=867750103}) 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1147200084}) 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=867750103}) 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3072720610}) 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=867750103}) 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=471427901}) 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=867750103}) 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2501060509}) 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.937 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=867750103}) 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.819+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1117728714}) 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873016747}) 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2663980778}) 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873016747}) 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2385312824}) 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873016747}) 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3460797691}) 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873016747}) 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4021960327}) 2024-06-11T19:17:49.938 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873016747}) 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=435200761}) 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873016747}) 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=102190223}) 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873016747}) 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3087283311}) 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873016747}) 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.820+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1771518754}) 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997935741}) 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=507738218}) 2024-06-11T19:17:49.939 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997935741}) 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1052237397}) 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997935741}) 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=183154077}) 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997935741}) 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3219857264}) 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997935741}) 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=220253853}) 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.940 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997935741}) 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=978421744}) 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997935741}) 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1324308233}) 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997935741}) 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.822+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=449816659}) 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314363394}) 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221930596}) 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314363394}) 2024-06-11T19:17:49.941 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1064850161}) 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314363394}) 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1007029660}) 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314363394}) 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3526218455}) 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314363394}) 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1387079255}) 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314363394}) 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=139932994}) 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.942 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314363394}) 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=736496822}) 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314363394}) 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.823+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1427389931}) 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434254337}) 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705662576}) 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434254337}) 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1844560858}) 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434254337}) 2024-06-11T19:17:49.943 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3801107470}) 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434254337}) 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2227013948}) 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434254337}) 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=316781770}) 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434254337}) 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1552655579}) 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434254337}) 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1519079067}) 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.944 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434254337}) 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.825+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2382257721}) 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262932460}) 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=13849385}) 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262932460}) 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3828612560}) 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262932460}) 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3393791635}) 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262932460}) 2024-06-11T19:17:49.945 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3700753242}) 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.826+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.827+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262932460}) 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.827+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=40626716}) 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.827+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.827+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262932460}) 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.827+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2305540301}) 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.827+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.827+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262932460}) 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.827+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2001278876}) 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.827+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.827+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262932460}) 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.827+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3526471204}) 2024-06-11T19:17:49.946 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.947 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196763269}) 2024-06-11T19:17:49.947 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016882933}) 2024-06-11T19:17:49.947 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.947 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196763269}) 2024-06-11T19:17:49.947 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=955863834}) 2024-06-11T19:17:49.947 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.947 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196763269}) 2024-06-11T19:17:49.947 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2979173038}) 2024-06-11T19:17:49.947 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.947 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196763269}) 2024-06-11T19:17:49.947 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1132060197}) 2024-06-11T19:17:49.947 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196763269}) 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=228845799}) 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196763269}) 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1070364678}) 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196763269}) 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2341846744}) 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196763269}) 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.828+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=474321377}) 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909686785}) 2024-06-11T19:17:49.948 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2671089056}) 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909686785}) 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3243843480}) 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909686785}) 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1599557899}) 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909686785}) 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2930573364}) 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909686785}) 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2118024587}) 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909686785}) 2024-06-11T19:17:49.949 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.829+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1562928658}) 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.830+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.830+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909686785}) 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.830+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=955749914}) 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.830+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.830+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909686785}) 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.830+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2481028717}) 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371965446}) 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=480310971}) 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371965446}) 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2073644460}) 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371965446}) 2024-06-11T19:17:49.950 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=289698097}) 2024-06-11T19:17:49.951 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.951 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371965446}) 2024-06-11T19:17:49.951 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=14901959}) 2024-06-11T19:17:49.951 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.951 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371965446}) 2024-06-11T19:17:49.951 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2286638476}) 2024-06-11T19:17:49.951 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.951 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371965446}) 2024-06-11T19:17:49.951 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=940990152}) 2024-06-11T19:17:49.951 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.951 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371965446}) 2024-06-11T19:17:49.951 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3422694746}) 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371965446}) 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.831+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2208772365}) 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261698176}) 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=861625471}) 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261698176}) 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=888879792}) 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261698176}) 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=447952838}) 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261698176}) 2024-06-11T19:17:49.952 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2723738598}) 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261698176}) 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2328505766}) 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261698176}) 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2333727130}) 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261698176}) 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3766487951}) 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261698176}) 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.832+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1026613243}) 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.953 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256337277}) 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284099918}) 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256337277}) 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201682520}) 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256337277}) 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1350080303}) 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256337277}) 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3643246931}) 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256337277}) 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1176230863}) 2024-06-11T19:17:49.954 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256337277}) 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3428376488}) 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256337277}) 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=97376937}) 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256337277}) 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.834+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=366910763}) 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851255269}) 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132177781}) 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.955 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851255269}) 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=253004352}) 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851255269}) 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3270534637}) 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851255269}) 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2659301535}) 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851255269}) 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2054414546}) 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851255269}) 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=235897899}) 2024-06-11T19:17:49.956 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851255269}) 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2696030495}) 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851255269}) 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.835+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1826910711}) 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598100244}) 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1525039596}) 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598100244}) 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3609638187}) 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598100244}) 2024-06-11T19:17:49.957 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1246989719}) 2024-06-11T19:17:49.958 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.958 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598100244}) 2024-06-11T19:17:49.958 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2968424346}) 2024-06-11T19:17:49.958 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.958 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598100244}) 2024-06-11T19:17:49.958 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2845825891}) 2024-06-11T19:17:49.958 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.958 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598100244}) 2024-06-11T19:17:49.958 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3483388849}) 2024-06-11T19:17:49.958 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.958 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598100244}) 2024-06-11T19:17:49.958 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2277898158}) 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598100244}) 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.837+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3123835890}) 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369675056}) 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461924289}) 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369675056}) 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3824538748}) 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369675056}) 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=216483507}) 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369675056}) 2024-06-11T19:17:49.959 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=871584016}) 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369675056}) 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2652629631}) 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369675056}) 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=867858074}) 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369675056}) 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3409690619}) 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369675056}) 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.838+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3618256486}) 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.839+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.839+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217861339}) 2024-06-11T19:17:49.960 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.839+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624141283}) 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.839+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.839+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217861339}) 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.839+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1606386522}) 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.839+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.839+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217861339}) 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.839+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=605600913}) 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.839+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.839+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217861339}) 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3192262114}) 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217861339}) 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2188157326}) 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217861339}) 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2914866427}) 2024-06-11T19:17:49.961 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217861339}) 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3063642112}) 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217861339}) 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.840+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3241346919}) 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3539218614}) 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1904653589}) 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3539218614}) 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=971329032}) 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3539218614}) 2024-06-11T19:17:49.962 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1733024562}) 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3539218614}) 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=218200872}) 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3539218614}) 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3933690603}) 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3539218614}) 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3465728112}) 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3539218614}) 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3477551092}) 2024-06-11T19:17:49.963 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3539218614}) 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.841+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=543259353}) 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913597025}) 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192529528}) 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913597025}) 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3325759628}) 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913597025}) 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1562929666}) 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913597025}) 2024-06-11T19:17:49.964 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3138093141}) 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913597025}) 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2188024010}) 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913597025}) 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=987618806}) 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913597025}) 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3100076367}) 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913597025}) 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.842+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=771296017}) 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149757886}) 2024-06-11T19:17:49.965 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1605466152}) 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149757886}) 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2135374711}) 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149757886}) 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1558326680}) 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149757886}) 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1500811610}) 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149757886}) 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=373070657}) 2024-06-11T19:17:49.966 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149757886}) 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2673307845}) 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149757886}) 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3995177058}) 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149757886}) 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.844+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3007775862}) 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798192039}) 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1202841708}) 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798192039}) 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1486717739}) 2024-06-11T19:17:49.967 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798192039}) 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=389749318}) 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798192039}) 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4286081544}) 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798192039}) 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=789558596}) 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798192039}) 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1460392926}) 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798192039}) 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2811356575}) 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.968 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798192039}) 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.845+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3177922892}) 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.846+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.846+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149981089}) 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.846+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280970733}) 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.846+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.846+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149981089}) 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.846+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3009717047}) 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.846+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.846+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149981089}) 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.846+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3279389175}) 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.846+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.846+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149981089}) 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=59034334}) 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.969 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149981089}) 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3809915}) 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149981089}) 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3614964677}) 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149981089}) 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3447665576}) 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149981089}) 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.847+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2236933767}) 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003678382}) 2024-06-11T19:17:49.970 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204828305}) 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003678382}) 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1720571693}) 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003678382}) 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2344425536}) 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003678382}) 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1761848656}) 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003678382}) 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3606219443}) 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003678382}) 2024-06-11T19:17:49.971 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2202183000}) 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003678382}) 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2678769088}) 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003678382}) 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.848+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=277221858}) 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819459937}) 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835383740}) 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819459937}) 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3707853494}) 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819459937}) 2024-06-11T19:17:49.972 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=354593779}) 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819459937}) 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4189625200}) 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819459937}) 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2137721455}) 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819459937}) 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3461618645}) 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819459937}) 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1574119541}) 2024-06-11T19:17:49.973 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819459937}) 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.849+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2782245425}) 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343252003}) 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102330501}) 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343252003}) 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1721793250}) 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343252003}) 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2306718776}) 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343252003}) 2024-06-11T19:17:49.974 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1896921293}) 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343252003}) 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3269825300}) 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343252003}) 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3769719246}) 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343252003}) 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2142815623}) 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343252003}) 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.851+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1471916851}) 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3282949067}) 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3101050252}) 2024-06-11T19:17:49.975 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3282949067}) 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788716906}) 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3282949067}) 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=808264827}) 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3282949067}) 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1990056936}) 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3282949067}) 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4002548097}) 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3282949067}) 2024-06-11T19:17:49.976 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1510602669}) 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3282949067}) 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3894103681}) 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3282949067}) 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.853+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1399143654}) 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762168878}) 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3766470792}) 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762168878}) 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=660057792}) 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762168878}) 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=415444354}) 2024-06-11T19:17:49.977 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762168878}) 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1605376765}) 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762168878}) 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3041121295}) 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762168878}) 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2812573083}) 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762168878}) 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=808023402}) 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762168878}) 2024-06-11T19:17:49.978 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.854+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1356782492}) 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.855+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.855+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235713782}) 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.855+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2418517252}) 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.855+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.855+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235713782}) 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.855+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232055189}) 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235713782}) 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4256781266}) 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235713782}) 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1160263303}) 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235713782}) 2024-06-11T19:17:49.979 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=389072413}) 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235713782}) 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2874343995}) 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235713782}) 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=931531755}) 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235713782}) 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.856+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3492608909}) 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501732285}) 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1474491312}) 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501732285}) 2024-06-11T19:17:49.980 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3841397804}) 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501732285}) 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=567412085}) 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501732285}) 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3323520902}) 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501732285}) 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=331021782}) 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501732285}) 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2537305754}) 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.981 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501732285}) 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2148379235}) 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501732285}) 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.857+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=6972848}) 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.858+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.858+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505988527}) 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.858+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1654136284}) 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.858+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.858+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505988527}) 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.858+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074359579}) 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.858+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.858+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505988527}) 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.858+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2123136496}) 2024-06-11T19:17:49.982 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.858+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505988527}) 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2025530746}) 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505988527}) 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1688089075}) 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505988527}) 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1883970157}) 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505988527}) 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2354101658}) 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.983 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505988527}) 2024-06-11T19:17:49.984 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.859+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1824814067}) 2024-06-11T19:17:49.984 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.984 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146693862}) 2024-06-11T19:17:49.984 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881236374}) 2024-06-11T19:17:49.984 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.984 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146693862}) 2024-06-11T19:17:49.984 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1145973322}) 2024-06-11T19:17:49.984 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.984 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146693862}) 2024-06-11T19:17:49.984 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=948988536}) 2024-06-11T19:17:49.984 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.984 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146693862}) 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3330374305}) 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146693862}) 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=72140099}) 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146693862}) 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1430746609}) 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146693862}) 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=655235138}) 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146693862}) 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.860+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1489048467}) 2024-06-11T19:17:49.985 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669172060}) 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2599195696}) 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669172060}) 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2892641296}) 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669172060}) 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3914147619}) 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669172060}) 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3718623433}) 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.986 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669172060}) 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2496389800}) 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669172060}) 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3723206868}) 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.861+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669172060}) 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.862+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3778493346}) 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.862+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.862+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669172060}) 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.862+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1625332110}) 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518073240}) 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1118642933}) 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.987 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518073240}) 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2728741383}) 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518073240}) 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821025705}) 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518073240}) 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=907673217}) 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518073240}) 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=968649188}) 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518073240}) 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1146577949}) 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.988 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518073240}) 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1602149652}) 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518073240}) 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.863+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3220889589}) 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1827663207}) 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028350295}) 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1827663207}) 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1466453238}) 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1827663207}) 2024-06-11T19:17:49.989 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=395299333}) 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1827663207}) 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1378083153}) 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1827663207}) 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1436788459}) 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1827663207}) 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2593590260}) 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1827663207}) 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4084913283}) 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1827663207}) 2024-06-11T19:17:49.990 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.864+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2785357774}) 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1666430812}) 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=969182412}) 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1666430812}) 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476669766}) 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1666430812}) 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2158840439}) 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1666430812}) 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1403641737}) 2024-06-11T19:17:49.991 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1666430812}) 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=334817622}) 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1666430812}) 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3088558235}) 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1666430812}) 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1870326979}) 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1666430812}) 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.866+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1433722021}) 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2555414347}) 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3408325820}) 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.992 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2555414347}) 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590549919}) 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2555414347}) 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1789806964}) 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2555414347}) 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=723532600}) 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2555414347}) 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3955241082}) 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2555414347}) 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4208110837}) 2024-06-11T19:17:49.993 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2555414347}) 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2200736233}) 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2555414347}) 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.867+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=887992488}) 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.868+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.868+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366248438}) 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589458771}) 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366248438}) 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2085153165}) 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.994 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366248438}) 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=36548406}) 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366248438}) 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1345556069}) 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366248438}) 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1547763781}) 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366248438}) 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4026237019}) 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366248438}) 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2874078326}) 2024-06-11T19:17:49.995 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366248438}) 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.869+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4123091072}) 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3112051271}) 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4236800069}) 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3112051271}) 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=684061552}) 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3112051271}) 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1498124867}) 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3112051271}) 2024-06-11T19:17:49.996 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2353516973}) 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3112051271}) 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=823977307}) 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3112051271}) 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1350187251}) 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3112051271}) 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4235962846}) 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3112051271}) 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.870+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2306197032}) 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2562341021}) 2024-06-11T19:17:49.997 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3370162010}) 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2562341021}) 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3412950784}) 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2562341021}) 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=625825835}) 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2562341021}) 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=590047685}) 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2562341021}) 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2700670702}) 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2562341021}) 2024-06-11T19:17:49.998 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3042452872}) 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2562341021}) 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2871775478}) 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2562341021}) 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.872+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=264877681}) 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933387014}) 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030490432}) 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933387014}) 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3144510347}) 2024-06-11T19:17:49.999 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933387014}) 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1636693126}) 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933387014}) 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=149845771}) 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933387014}) 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1115129234}) 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933387014}) 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.873+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1580853670}) 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.874+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.874+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933387014}) 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.874+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=159663732}) 2024-06-11T19:17:50.000 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.874+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.874+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933387014}) 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.874+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2564975087}) 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603932067}) 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1031856602}) 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603932067}) 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3136075977}) 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603932067}) 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2881018093}) 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603932067}) 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4277497265}) 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.001 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603932067}) 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2370196404}) 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603932067}) 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3531939313}) 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603932067}) 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=927732806}) 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603932067}) 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.875+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2111447403}) 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=311481522}) 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175746399}) 2024-06-11T19:17:50.002 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=311481522}) 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1646319178}) 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=311481522}) 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=690618501}) 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=311481522}) 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3851184346}) 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=311481522}) 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2427902442}) 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=311481522}) 2024-06-11T19:17:50.003 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3495659977}) 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=311481522}) 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=330580101}) 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=311481522}) 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.876+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1704901448}) 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941197089}) 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1320276510}) 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941197089}) 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2978924995}) 2024-06-11T19:17:50.004 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941197089}) 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3949608564}) 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941197089}) 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2146401889}) 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941197089}) 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1515745649}) 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941197089}) 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4042698303}) 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941197089}) 2024-06-11T19:17:50.005 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3973846890}) 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941197089}) 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.878+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1140652613}) 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987992938}) 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680507058}) 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987992938}) 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=221599989}) 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987992938}) 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2912408452}) 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.006 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987992938}) 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3298146333}) 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987992938}) 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4158594177}) 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987992938}) 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3920240902}) 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987992938}) 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1445880745}) 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987992938}) 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.879+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2311094343}) 2024-06-11T19:17:50.007 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3015658132}) 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185783615}) 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3015658132}) 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3980557908}) 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3015658132}) 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=524295753}) 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3015658132}) 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=5326314}) 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3015658132}) 2024-06-11T19:17:50.008 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2206763165}) 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3015658132}) 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.880+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1028478180}) 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.881+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.881+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3015658132}) 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.881+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1290272158}) 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.881+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.881+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3015658132}) 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.881+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2395880575}) 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153506006}) 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736758026}) 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153506006}) 2024-06-11T19:17:50.009 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557466079}) 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153506006}) 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1656030573}) 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153506006}) 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=571603560}) 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153506006}) 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3217002562}) 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153506006}) 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1785792071}) 2024-06-11T19:17:50.010 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153506006}) 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2439783920}) 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153506006}) 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.882+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2262030919}) 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288003354}) 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3937883290}) 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288003354}) 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1676469892}) 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288003354}) 2024-06-11T19:17:50.011 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1448943381}) 2024-06-11T19:17:50.012 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.012 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288003354}) 2024-06-11T19:17:50.012 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3743965074}) 2024-06-11T19:17:50.012 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.012 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288003354}) 2024-06-11T19:17:50.012 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=865030096}) 2024-06-11T19:17:50.012 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.012 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288003354}) 2024-06-11T19:17:50.012 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2168063551}) 2024-06-11T19:17:50.012 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.012 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288003354}) 2024-06-11T19:17:50.012 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=915121469}) 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288003354}) 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.883+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2930216613}) 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.884+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.884+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3426691902}) 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2221974557}) 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3426691902}) 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1236983553}) 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3426691902}) 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2106063251}) 2024-06-11T19:17:50.013 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3426691902}) 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1549017957}) 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3426691902}) 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2989667372}) 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3426691902}) 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=766059265}) 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3426691902}) 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1663234466}) 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.014 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3426691902}) 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.885+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3178810228}) 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3999541194}) 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165848649}) 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3999541194}) 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3393952775}) 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3999541194}) 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2199133686}) 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3999541194}) 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4093056833}) 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.015 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3999541194}) 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3484494485}) 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3999541194}) 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4288978422}) 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3999541194}) 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2822720152}) 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3999541194}) 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.886+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1298887855}) 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62033908}) 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152572419}) 2024-06-11T19:17:50.016 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62033908}) 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3643421488}) 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62033908}) 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2413260919}) 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62033908}) 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=83980031}) 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.887+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.888+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62033908}) 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.888+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3329952973}) 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.888+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.888+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62033908}) 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.888+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1968665914}) 2024-06-11T19:17:50.017 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.888+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.018 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.888+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62033908}) 2024-06-11T19:17:50.018 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.888+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4016845788}) 2024-06-11T19:17:50.018 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.888+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.018 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.888+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62033908}) 2024-06-11T19:17:50.018 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.888+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3276991239}) 2024-06-11T19:17:50.018 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.018 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083803661}) 2024-06-11T19:17:50.018 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3261950783}) 2024-06-11T19:17:50.018 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.018 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083803661}) 2024-06-11T19:17:50.018 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4158507879}) 2024-06-11T19:17:50.018 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083803661}) 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2771675178}) 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083803661}) 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=208131106}) 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083803661}) 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1666226826}) 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083803661}) 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.889+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3483571183}) 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.890+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.890+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083803661}) 2024-06-11T19:17:50.019 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.890+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1444980142}) 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.890+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.890+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083803661}) 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.890+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3550391275}) 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.895+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.895+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286653050}) 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.895+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1952320727}) 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.895+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.895+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286653050}) 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=306938324}) 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286653050}) 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3346168650}) 2024-06-11T19:17:50.020 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286653050}) 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=689318969}) 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286653050}) 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1845642033}) 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286653050}) 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3745088595}) 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286653050}) 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=275354343}) 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.021 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286653050}) 2024-06-11T19:17:50.022 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.896+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3050459638}) 2024-06-11T19:17:50.022 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.022 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835344050}) 2024-06-11T19:17:50.022 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2972315713}) 2024-06-11T19:17:50.022 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.022 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835344050}) 2024-06-11T19:17:50.022 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3790007969}) 2024-06-11T19:17:50.022 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.022 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835344050}) 2024-06-11T19:17:50.022 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2680989330}) 2024-06-11T19:17:50.022 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.022 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835344050}) 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=137945364}) 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835344050}) 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1146074314}) 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835344050}) 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2443960471}) 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835344050}) 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=173453881}) 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.900+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835344050}) 2024-06-11T19:17:50.023 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.901+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1034834033}) 2024-06-11T19:17:50.024 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.024 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=172115695,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=283088686}) 2024-06-11T19:17:50.024 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=79,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=1} crc {logm=1471623931,mdsmap=2840952586,mgr=2264053585,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1293282039,mon_config_key=3250354011}) 2024-06-11T19:17:50.024 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.024 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=172115695,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=283088686}) 2024-06-11T19:17:50.024 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=5} crc {logm=1125601590,mdsmap=2840952586,mgr=2102212137,mgr_metadata=1826097544,mgrstat=283088685}) 2024-06-11T19:17:50.024 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.024 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=172115695,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=283088686}) 2024-06-11T19:17:50.024 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=78,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=2} crc {logm=1583453231,mdsmap=2840952586,mgr=1864858522,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1293282039,mon_config_key=1555703263}) 2024-06-11T19:17:50.024 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.024 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=172115695,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=283088686}) 2024-06-11T19:17:50.024 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=83,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=2116559155,mdsmap=3140514009,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=283088685}) 2024-06-11T19:17:50.025 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.025 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=172115695,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=283088686}) 2024-06-11T19:17:50.025 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=77,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=3} crc {logm=2986163443,mdsmap=1401145748,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1293282039,mon_config_key=162979876}) 2024-06-11T19:17:50.025 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.025 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=172115695,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=283088686}) 2024-06-11T19:17:50.025 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=85,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=3125024962,mdsmap=2840952586,mgr=120143446,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=585048171}) 2024-06-11T19:17:50.025 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.904+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.025 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.905+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=172115695,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=283088686}) 2024-06-11T19:17:50.025 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.905+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=82,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=484391243,mdsmap=3140514009,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=4114361736}) 2024-06-11T19:17:50.025 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.905+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.025 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.905+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=172115695,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=283088686}) 2024-06-11T19:17:50.025 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.905+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=4256350525,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=283088685}) 2024-06-11T19:17:50.026 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.907+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.026 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=3,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=48} crc {mgrstat=1161434022,mon_config_key=657769230,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=3262333735,osdmap=719867055}) 2024-06-11T19:17:50.026 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=3,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28,osdmap=49} crc {mgrstat=1161434022,mon_config_key=2314039359,monmap=506305176,osd_metadata=1424309990,osd_pg_creating=3129148780,osd_snap=1665068367,osdmap=3616094681}) 2024-06-11T19:17:50.026 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.026 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=3,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=48} crc {mgrstat=1161434022,mon_config_key=657769230,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=3262333735,osdmap=719867055}) 2024-06-11T19:17:50.026 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=3,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=48} crc {mgrstat=1161434023,mon_config_key=518484376,monmap=506305176,osd_metadata=1293217973,osd_pg_creating=3129148780,osd_snap=1820449667,osdmap=1540086779}) 2024-06-11T19:17:50.026 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.026 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=3,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=48} crc {mgrstat=1161434022,mon_config_key=657769230,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=3262333735,osdmap=719867055}) 2024-06-11T19:17:50.026 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mgrstat=3,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=49} crc {mgrstat=1161434022,mon_config_key=1870814644,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=3927735485,osdmap=2178706458}) 2024-06-11T19:17:50.026 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.026 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=3,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=48} crc {mgrstat=1161434022,mon_config_key=657769230,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=3262333735,osdmap=719867055}) 2024-06-11T19:17:50.026 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {mgrstat=3,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=49} crc {mgrstat=1161434022,mon_config_key=845859478,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=2878757323,osdmap=3282071168}) 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=3,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=48} crc {mgrstat=1161434022,mon_config_key=657769230,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=3262333735,osdmap=719867055}) 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {mgrstat=2,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=49} crc {mgrstat=3797824817,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=228323004,osdmap=564235159}) 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=3,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=48} crc {mgrstat=1161434022,mon_config_key=657769230,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=3262333735,osdmap=719867055}) 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {mgrstat=3,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=46} crc {mgrstat=1161434022,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148781,osd_snap=3552937844,osdmap=3230224214}) 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=3,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=48} crc {mgrstat=1161434022,mon_config_key=657769230,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=3262333735,osdmap=719867055}) 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mgrstat=3,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=48} crc {mgrstat=1161434022,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=3517577310,osdmap=4290259541}) 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=3,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=48} crc {mgrstat=1161434022,mon_config_key=657769230,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=3262333735,osdmap=719867055}) 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.908+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mgrstat=3,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27,osdmap=51} crc {mgrstat=1161434022,mon_config_key=2281582573,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=3129148780,osd_snap=3859828433,osdmap=4207286801}) 2024-06-11T19:17:50.027 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1287169658}) 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=297610047}) 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1287169658}) 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=389919484}) 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1287169658}) 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=4135984335}) 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1287169658}) 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4271403306}) 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1287169658}) 2024-06-11T19:17:50.028 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2182659523}) 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.912+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1287169658}) 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.913+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=733089854}) 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.913+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.913+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1287169658}) 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.913+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2634578151}) 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.913+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.913+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1287169658}) 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.913+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1513343439}) 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.915+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.915+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3666689723}) 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=862703175}) 2024-06-11T19:17:50.029 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.030 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3666689723}) 2024-06-11T19:17:50.030 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=32545368}) 2024-06-11T19:17:50.030 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.030 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3666689723}) 2024-06-11T19:17:50.030 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=429890791}) 2024-06-11T19:17:50.030 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.030 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3666689723}) 2024-06-11T19:17:50.030 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3842940126}) 2024-06-11T19:17:50.030 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.030 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3666689723}) 2024-06-11T19:17:50.030 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2352138845}) 2024-06-11T19:17:50.030 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3666689723}) 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2820088016}) 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3666689723}) 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3795814871}) 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3666689723}) 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.916+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2047354670}) 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3949960744}) 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1558351192}) 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.031 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3949960744}) 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=240687273}) 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3949960744}) 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1614135543}) 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3949960744}) 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=281667999}) 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3949960744}) 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2988348836}) 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3949960744}) 2024-06-11T19:17:50.032 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=4017396614}) 2024-06-11T19:17:50.033 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.033 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3949960744}) 2024-06-11T19:17:50.033 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=480530808}) 2024-06-11T19:17:50.033 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.920+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.033 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.921+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3949960744}) 2024-06-11T19:17:50.033 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.921+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=291813582}) 2024-06-11T19:17:50.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.924+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.924+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=826263283}) 2024-06-11T19:17:50.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.924+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2409364705}) 2024-06-11T19:17:50.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=826263283}) 2024-06-11T19:17:50.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3545724408}) 2024-06-11T19:17:50.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=826263283}) 2024-06-11T19:17:50.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1827434109}) 2024-06-11T19:17:50.038 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=826263283}) 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4135996600}) 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=826263283}) 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2542793132}) 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=826263283}) 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3691158106}) 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=826263283}) 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=20844545}) 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=826263283}) 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.925+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3460642865}) 2024-06-11T19:17:50.039 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=59} crc {osdmap=2130985657}) 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=60} crc {osdmap=1307639969}) 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=59} crc {osdmap=2130985657}) 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=58} crc {osdmap=2622135662}) 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=59} crc {osdmap=2130985657}) 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=59} crc {osdmap=2549610262}) 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=59} crc {osdmap=2130985657}) 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=56} crc {osdmap=3432530034}) 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.040 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=59} crc {osdmap=2130985657}) 2024-06-11T19:17:50.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=57} crc {osdmap=850492752}) 2024-06-11T19:17:50.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=59} crc {osdmap=2130985657}) 2024-06-11T19:17:50.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=62} crc {osdmap=3114797074}) 2024-06-11T19:17:50.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=59} crc {osdmap=2130985657}) 2024-06-11T19:17:50.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=56} crc {osdmap=728598041}) 2024-06-11T19:17:50.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:17:50.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=59} crc {osdmap=2130985657}) 2024-06-11T19:17:50.041 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:17:49.941+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=58} crc {osdmap=340418338}) 2024-06-11T19:17:50.822 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:17:50.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:17:50.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.921+0000 7fc43f120640 10 monclient: get_monmap_and_config 2024-06-11T19:17:50.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.921+0000 7fc43f120640 10 monclient: build_initial_monmap 2024-06-11T19:17:50.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.921+0000 7fc43f120640 10 monclient: monmap: 2024-06-11T19:17:50.921 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:50.921 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:50.921 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:50.922951+0000 2024-06-11T19:17:50.921 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:50.922951+0000 2024-06-11T19:17:50.921 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:50.921 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:50.922 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:50.922 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:50.922 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:50.922 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:50.922 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:50.922 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:50.922 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:50.922 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:50.922 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:50.922 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:50.922 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:50.922 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:50.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.922+0000 7fc43f120640 1 Processor -- start 2024-06-11T19:17:50.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.922+0000 7fc43f120640 1 -- start start 2024-06-11T19:17:50.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.922+0000 7fc43f120640 10 monclient: init 2024-06-11T19:17:50.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43f120640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:50.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43f120640 10 monclient: _add_conns ranks=[1,0,3,7,8,4,9,5,10,6,2] 2024-06-11T19:17:50.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43f120640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc4380a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:17:50.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43f120640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4380a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:17:50.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43f120640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43813f380 0x7fc438141770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:50.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43f120640 10 monclient(hunting): picked mon.noname-g con 0x7fc43813f380 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:50.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43f120640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4380a7140 con 0x7fc4380a2530 2024-06-11T19:17:50.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43f120640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4380a2b50 con 0x7fc4380a4920 2024-06-11T19:17:50.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43f120640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:50.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43f120640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fc4380a2e30 con 0x7fc43813f380 2024-06-11T19:17:50.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43f120640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43f120640 10 monclient(hunting): authenticate will time out at 1280.730286s 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc437fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc4380a2530 0x7fc43813dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35760/0 (socket says 172.21.15.98:35760) 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc437fff640 1 -- 172.21.15.98:0/3736500752 learned_addr learned my addr 172.21.15.98:0/3736500752 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 1 --2- 172.21.15.98:0/3736500752 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43813f380 0x7fc438141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 10 monclient(hunting): get_auth_request con 0x7fc43813f380 auth_method 0 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc4377fe640 1 -- 172.21.15.98:0/3736500752 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2959268394 0 0) 0x7fc4380a7140 con 0x7fc4380a2530 2024-06-11T19:17:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc4377fe640 10 monclient(hunting): _negotiate 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc4377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc4377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc4377fe640 10 monclient(hunting): my global_id is 65016 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc4377fe640 1 -- 172.21.15.98:0/3736500752 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc428003520 con 0x7fc4380a2530 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc4377fe640 1 -- 172.21.15.98:0/3736500752 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3157444931 0 0) 0x7fc4380a2b50 con 0x7fc4380a4920 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc4377fe640 10 monclient(hunting): _negotiate 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc4377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc4377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc4377fe640 10 monclient(hunting): my global_id is 32167 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc4377fe640 1 -- 172.21.15.98:0/3736500752 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4380a7140 con 0x7fc4380a4920 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 10 monclient(hunting): handle_auth_done global_id 94179 payload 274 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 1 -- 172.21.15.98:0/3736500752 >> v1:172.21.15.117:6789/0 conn(0x7fc4380a4920 legacy=0x7fc4380a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 1 -- 172.21.15.98:0/3736500752 >> v1:172.21.15.98:6789/0 conn(0x7fc4380a2530 legacy=0x7fc43813dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 10 monclient: _finish_hunting 0 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 1 monclient: found mon.noname-g 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.923+0000 7fc43d6bd640 1 -- 172.21.15.98:0/3736500752 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4380a3110 con 0x7fc43813f380 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc43d6bd640 1 --2- 172.21.15.98:0/3736500752 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43813f380 0x7fc438141770 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc430009dc0 tx=0x7fc43002be00 comp rx=0 tx=0).ready entity=mon.3 client_cookie=feb6d616b06b7f0f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 1 -- 172.21.15.98:0/3736500752 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc430039070 con 0x7fc43813f380 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 10 monclient: dump: 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 10 monclient: _finish_auth 0 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 10 monclient: _check_auth_tickets 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 1 -- 172.21.15.98:0/3736500752 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc43002ce20 con 0x7fc43813f380 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc43f120640 5 monclient: authenticate success, global_id 94179 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc43f120640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 1 -- 172.21.15.98:0/3736500752 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4300327b0 con 0x7fc43813f380 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc4377fe640 10 monclient: dump: 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:50.937 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc43f120640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.924+0000 7fc43f120640 4 set_mon_vals no callback set 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.925+0000 7fc43f120640 10 monclient: shutdown 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.925+0000 7fc43f120640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.925+0000 7fc43f120640 1 -- 172.21.15.98:0/3736500752 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43813f380 msgr2=0x7fc438141770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.925+0000 7fc43f120640 1 --2- 172.21.15.98:0/3736500752 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43813f380 0x7fc438141770 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc430009dc0 tx=0x7fc43002be00 comp rx=0 tx=0).stop 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.925+0000 7fc43f120640 1 -- 172.21.15.98:0/3736500752 shutdown_connections 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.925+0000 7fc43f120640 1 --2- 172.21.15.98:0/3736500752 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43813f380 0x7fc438141770 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.925+0000 7fc43f120640 1 -- 172.21.15.98:0/3736500752 >> 172.21.15.98:0/3736500752 conn(0x7fc4380ab850 msgr2=0x7fc4380a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.925+0000 7fc43f120640 1 -- 172.21.15.98:0/3736500752 shutdown_connections 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.925+0000 7fc43f120640 1 -- 172.21.15.98:0/3736500752 wait complete. 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.925+0000 7fc43f120640 10 monclient: build_initial_monmap 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.925+0000 7fc43f120640 10 monclient: monmap: 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:50.939 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:50.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 1 librados: starting msgr at 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 1 librados: starting objecter 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 1 Processor -- start 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 1 -- start start 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 1 librados: setting wanted keys 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 1 librados: calling monclient init 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 10 monclient: init 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 10 monclient: _add_conns ranks=[5,0,4,1,7,6,8,2,3] 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc4380a2530 0x7fc4381cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 10 monclient(hunting): picked mon.noname-h con 0x7fc4380a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4380a4920 0x7fc4381358e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 10 monclient(hunting): picked mon.noname-a con 0x7fc4380a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:17:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc43813f380 0x7fc43809da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 10 monclient(hunting): picked mon.noname-c con 0x7fc43813f380 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc438138e50 con 0x7fc4380a4920 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc4380a84b0 con 0x7fc43813f380 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc4380a6da0 con 0x7fc4380a2530 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.926+0000 7fc43f120640 10 monclient(hunting): authenticate will time out at 1280.733975s 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4380a4920 0x7fc4381358e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4380a4920 0x7fc4381358e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56664/0 (socket says 172.21.15.98:56664) 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 1 -- 172.21.15.98:0/612645744 learned_addr learned my addr 172.21.15.98:0/612645744 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:50.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 10 monclient(hunting): get_auth_request con 0x7fc4380a4920 auth_method 0 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43cebc640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc4380a2530 0x7fc4381cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43d6bd640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc43813f380 0x7fc43809da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43cebc640 1 -- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc4380a2530 msgr2=0x7fc4381cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43cebc640 1 -- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc4380a2530 msgr2=0x7fc4381cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43cebc640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc4380a2530 0x7fc4381cd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43d6bd640 10 monclient(hunting): get_auth_request con 0x7fc43813f380 auth_method 0 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43d6bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43d6bd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43d6bd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:50.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43cebc640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc4380a2530 0x7fc4381cd6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43d6bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43d6bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43d6bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 10 monclient(hunting): handle_auth_done global_id 65025 payload 824 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 1 -- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc4380a2530 msgr2=0x7fc4381cd6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc4380a2530 0x7fc4381cd6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 1 -- 172.21.15.98:0/612645744 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc43813f380 msgr2=0x7fc43809da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc43813f380 0x7fc43809da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 1 monclient: found mon.noname-a 2024-06-11T19:17:50.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 1 -- 172.21.15.98:0/612645744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc438073410 con 0x7fc4380a4920 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43d6bd640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc43813f380 0x7fc43809da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc437fff640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4380a4920 0x7fc4381358e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc42400c4d0 tx=0x7fc42400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 10 client.?.objecter ms_handle_connect 0x7fc4380a4920 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 1 -- 172.21.15.98:0/612645744 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc424017070 con 0x7fc4380a4920 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 10 monclient: dump: 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:50.945 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:50.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 10 monclient: _finish_auth 0 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 10 monclient: _check_auth_tickets 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 1 -- 172.21.15.98:0/612645744 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc424007070 con 0x7fc4380a4920 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43f120640 5 monclient: authenticate success, global_id 65025 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43f120640 10 monclient: _renew_subs 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43f120640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43f120640 1 -- 172.21.15.98:0/612645744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4380a03f0 con 0x7fc4380a4920 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 1 -- 172.21.15.98:0/612645744 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc424007690 con 0x7fc4380a4920 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc436ffd640 4 set_mon_vals no callback set 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc4357fa640 10 monclient: dump: 2024-06-11T19:17:50.947 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:50.948 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43f120640 10 client.65025.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43f120640 10 monclient: _renew_subs 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43f120640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43f120640 1 -- 172.21.15.98:0/612645744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4380a17b0 con 0x7fc4380a4920 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.927+0000 7fc43f120640 1 librados: init done 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.928+0000 7fc4357fa640 1 -- 172.21.15.98:0/612645744 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fc424007220 con 0x7fc4380a4920 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.928+0000 7fc43f120640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.928+0000 7fc43f120640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.928+0000 7fc43f120640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.928+0000 7fc43f120640 1 -- 172.21.15.98:0/612645744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4380a7360 con 0x7fc4380a4920 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.929+0000 7fc4357fa640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc41c03fdb0 0x7fc41c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:50.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.929+0000 7fc4357fa640 1 -- 172.21.15.98:0/612645744 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(346..346 src has 1..346) ==== 6230+0+0 (secure 0 0 0) 0x7fc424050c60 con 0x7fc4380a4920 2024-06-11T19:17:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.929+0000 7fc4357fa640 10 client.65025.objecter ms_dispatch 0x7fc438000b90 osd_map(346..346 src has 1..346) 2024-06-11T19:17:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.929+0000 7fc4357fa640 3 client.65025.objecter handle_osd_map got epochs [346,346] > 0 2024-06-11T19:17:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.929+0000 7fc4357fa640 3 client.65025.objecter handle_osd_map decoding full epoch 346 2024-06-11T19:17:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.929+0000 7fc4357fa640 20 client.65025.objecter dump_active .. 0 homeless 2024-06-11T19:17:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.929+0000 7fc43cebc640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc41c03fdb0 0x7fc41c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.929+0000 7fc43cebc640 10 monclient: get_auth_request con 0x7fc41c03fdb0 auth_method 0 2024-06-11T19:17:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.929+0000 7fc4357fa640 10 client.65025.objecter ms_handle_connect 0x7fc41c03fdb0 2024-06-11T19:17:50.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.929+0000 7fc43cebc640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc41c03fdb0 0x7fc41c042270 secure :-1 s=READY pgs=4172 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0010a0 tx=0x7fc42c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:50.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.933+0000 7fc4357fa640 1 -- 172.21.15.98:0/612645744 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc4380a7360 con 0x7fc4380a4920 2024-06-11T19:17:50.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.933+0000 7fc4357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:50.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:50.933+0000 7fc4357fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.183+0000 7fc43f120640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:17:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.183+0000 7fc43f120640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:17:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.183+0000 7fc43f120640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:17:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.183+0000 7fc43f120640 1 -- 172.21.15.98:0/612645744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc4380a4d00 con 0x7fc4380a4920 2024-06-11T19:17:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.184+0000 7fc4357fa640 1 -- 172.21.15.98:0/612645744 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fc424011db0 con 0x7fc4380a4920 2024-06-11T19:17:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.184+0000 7fc4357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:17:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.184+0000 7fc4357fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:51.184 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":38,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:17:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 10 librados: watch_flush enter 2024-06-11T19:17:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 10 librados: watch_flush exit 2024-06-11T19:17:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 20 client.65025.objecter shutdown clearing up homeless session... 2024-06-11T19:17:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 10 client.65025.objecter successfully canceled tick 2024-06-11T19:17:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 -- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc41c03fdb0 msgr2=0x7fc41c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc41c03fdb0 0x7fc41c042270 secure :-1 s=READY pgs=4172 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0010a0 tx=0x7fc42c009040 comp rx=0 tx=0).stop 2024-06-11T19:17:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 10 monclient: shutdown 2024-06-11T19:17:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 -- 172.21.15.98:0/612645744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4380a4920 msgr2=0x7fc4381358e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4380a4920 0x7fc4381358e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc42400c4d0 tx=0x7fc42400c990 comp rx=0 tx=0).stop 2024-06-11T19:17:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 -- 172.21.15.98:0/612645744 shutdown_connections 2024-06-11T19:17:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc41c03fdb0 0x7fc41c042270 unknown :-1 s=CLOSED pgs=4172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc43813f380 0x7fc43809da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4380a4920 0x7fc4381358e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 --2- 172.21.15.98:0/612645744 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc4380a2530 0x7fc4381cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 -- 172.21.15.98:0/612645744 >> 172.21.15.98:0/612645744 conn(0x7fc4380ab850 msgr2=0x7fc4380a8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 -- 172.21.15.98:0/612645744 shutdown_connections 2024-06-11T19:17:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 -- 172.21.15.98:0/612645744 wait complete. 2024-06-11T19:17:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.186+0000 7fc43f120640 1 librados: shutdown 2024-06-11T19:17:51.196 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:17:51.196 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-06-11T19:17:51.196 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:17:51.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:17:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.295+0000 7f125636d640 10 monclient: get_monmap_and_config 2024-06-11T19:17:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.295+0000 7f125636d640 10 monclient: build_initial_monmap 2024-06-11T19:17:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.295+0000 7f125636d640 10 monclient: monmap: 2024-06-11T19:17:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:51.296959+0000 2024-06-11T19:17:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:51.296959+0000 2024-06-11T19:17:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:51.298 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:51.298 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:51.298 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.296+0000 7f125636d640 1 Processor -- start 2024-06-11T19:17:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.296+0000 7f125636d640 1 -- start start 2024-06-11T19:17:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.296+0000 7f125636d640 10 monclient: init 2024-06-11T19:17:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 10 monclient: _add_conns ranks=[8,5,6,0,4,3,9,10,1,7,2] 2024-06-11T19:17:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12500a4920 0x7f12500a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 10 monclient(hunting): picked mon.noname-e con 0x7f12500a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:17:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12500a5240 0x7f12500a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 10 monclient(hunting): picked mon.noname-h con 0x7f12500a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f12500a2530 0x7f12500a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 10 monclient(hunting): picked mon.noname-d con 0x7f12500a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f12500a7140 con 0x7f12500a5240 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f12500a2b50 con 0x7f12500a2530 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f12500a2e30 con 0x7f12500a4920 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125636d640 10 monclient(hunting): authenticate will time out at 1281.104263s 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12500a4920 0x7f12500a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125490a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f12500a2530 0x7f12500a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12500a4920 0x7f12500a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39064/0 (socket says 172.21.15.98:39064) 2024-06-11T19:17:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 1 -- 172.21.15.98:0/2510222130 learned_addr learned my addr 172.21.15.98:0/2510222130 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125490a640 10 monclient(hunting): get_auth_request con 0x7f12500a2530 auth_method 0 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125490a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125490a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125490a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 10 monclient(hunting): get_auth_request con 0x7f12500a4920 auth_method 0 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125490a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125490a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f125490a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124f7fe640 1 --2- 172.21.15.98:0/2510222130 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12500a5240 0x7f12500a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124f7fe640 10 monclient(hunting): get_auth_request con 0x7f12500a5240 auth_method 0 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 10 monclient(hunting): handle_auth_done global_id 94166 payload 274 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 1 -- 172.21.15.98:0/2510222130 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f12500a2530 msgr2=0x7f12500a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 1 --2- 172.21.15.98:0/2510222130 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f12500a2530 0x7f12500a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 1 -- 172.21.15.98:0/2510222130 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12500a5240 msgr2=0x7f12500a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 1 --2- 172.21.15.98:0/2510222130 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12500a5240 0x7f12500a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 1 monclient: found mon.noname-e 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124ffff640 1 -- 172.21.15.98:0/2510222130 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12500a3110 con 0x7f12500a4920 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.297+0000 7f124f7fe640 1 --2- 172.21.15.98:0/2510222130 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12500a5240 0x7f12500a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124ffff640 1 --2- 172.21.15.98:0/2510222130 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12500a4920 0x7f12500a4d00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f12400098a0 tx=0x7f124002b7a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=6ba59e405ddc76cf server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 1 -- 172.21.15.98:0/2510222130 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1240038070 con 0x7f12500a4920 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 10 monclient: dump: 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 10 monclient: _finish_auth 0 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 10 monclient: _check_auth_tickets 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 1 -- 172.21.15.98:0/2510222130 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1240031970 con 0x7f12500a4920 2024-06-11T19:17:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f125636d640 5 monclient: authenticate success, global_id 94166 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f125636d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 1 -- 172.21.15.98:0/2510222130 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1240031ba0 con 0x7f12500a4920 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f125636d640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f125636d640 4 set_mon_vals no callback set 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.298+0000 7f124effd640 10 monclient: dump: 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 10 monclient: shutdown 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 1 -- 172.21.15.98:0/2510222130 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12500a4920 msgr2=0x7f12500a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 1 --2- 172.21.15.98:0/2510222130 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12500a4920 0x7f12500a4d00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f12400098a0 tx=0x7f124002b7a0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 1 -- 172.21.15.98:0/2510222130 shutdown_connections 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 1 --2- 172.21.15.98:0/2510222130 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f12500a2530 0x7f12500a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 1 --2- 172.21.15.98:0/2510222130 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f12500a5240 0x7f12500a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 1 --2- 172.21.15.98:0/2510222130 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12500a4920 0x7f12500a4d00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 1 -- 172.21.15.98:0/2510222130 >> 172.21.15.98:0/2510222130 conn(0x7f12500ab850 msgr2=0x7f12500a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 1 -- 172.21.15.98:0/2510222130 shutdown_connections 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 1 -- 172.21.15.98:0/2510222130 wait complete. 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 10 monclient: build_initial_monmap 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 10 monclient: monmap: 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 1 librados: starting msgr at 2024-06-11T19:17:51.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.299+0000 7f125636d640 1 librados: starting objecter 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 1 Processor -- start 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 1 -- start start 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 1 librados: setting wanted keys 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 1 librados: calling monclient init 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 10 monclient: init 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 10 monclient: _add_conns ranks=[2,0,7,8,1,4,3,6,5] 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12500a2530 0x7f1250137a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 10 monclient(hunting): picked mon.noname-b con 0x7f12500a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12500a4920 0x7f1250123280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 10 monclient(hunting): picked mon.noname-a con 0x7f12500a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f12500a5240 0x7f12501237c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 10 monclient(hunting): picked mon.noname-i con 0x7f12500a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f12500a6da0 con 0x7f12500a4920 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1250138e50 con 0x7f12500a2530 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f12500a6420 con 0x7f12500a5240 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125636d640 10 monclient(hunting): authenticate will time out at 1281.107691s 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12500a2530 0x7f1250137a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12500a4920 0x7f1250123280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12500a2530 0x7f1250137a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45212/0 (socket says 172.21.15.98:45212) 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12500a4920 0x7f1250123280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56672/0 (socket says 172.21.15.98:56672) 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124ffff640 1 -- 172.21.15.98:0/17381821 learned_addr learned my addr 172.21.15.98:0/17381821 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124ffff640 10 monclient(hunting): get_auth_request con 0x7f12500a2530 auth_method 0 2024-06-11T19:17:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124f7fe640 10 monclient(hunting): get_auth_request con 0x7f12500a4920 auth_method 0 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f124f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125490a640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f12500a5240 0x7f12501237c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125490a640 10 monclient(hunting): get_auth_request con 0x7f12500a5240 auth_method 0 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125490a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125490a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.300+0000 7f125490a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124ffff640 10 monclient(hunting): handle_auth_done global_id 84422 payload 824 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124ffff640 1 -- 172.21.15.98:0/17381821 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f12500a5240 msgr2=0x7f12501237c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124ffff640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f12500a5240 0x7f12501237c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124ffff640 1 -- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12500a4920 msgr2=0x7f1250123280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124ffff640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12500a4920 0x7f1250123280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124ffff640 1 monclient: found mon.noname-b 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:17:51.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124ffff640 1 -- 172.21.15.98:0/17381821 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1250124eb0 con 0x7f12500a2530 2024-06-11T19:17:51.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f125490a640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f12500a5240 0x7f12501237c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:17:51.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124ffff640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12500a2530 0x7f1250137a30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f12400097c0 tx=0x7f12400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 10 client.?.objecter ms_handle_connect 0x7f12500a2530 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1240042020 con 0x7f12500a2530 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 10 monclient: dump: 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 10 monclient: _finish_auth 0 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 10 monclient: _check_auth_tickets 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1240031ea0 con 0x7f12500a2530 2024-06-11T19:17:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f125636d640 5 monclient: authenticate success, global_id 84422 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1240038040 con 0x7f12500a2530 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f125636d640 10 monclient: _renew_subs 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f125636d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f125636d640 1 -- 172.21.15.98:0/17381821 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12500a8da0 con 0x7f12500a2530 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124cff9640 10 monclient: dump: 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.327 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f124e7fc640 4 set_mon_vals no callback set 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f125636d640 10 client.84422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f125636d640 10 monclient: _renew_subs 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f125636d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f125636d640 1 -- 172.21.15.98:0/17381821 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1250126ec0 con 0x7f12500a2530 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.301+0000 7f125636d640 1 librados: init done 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f1240045020 con 0x7f12500a2530 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f125636d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f125636d640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12501c0c30 0x7f125002cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f125636d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f125636d640 1 -- 172.21.15.98:0/17381821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1250050830 con 0x7f12501c0c30 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124f7fe640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12501c0c30 0x7f125002cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124f7fe640 10 monclient: get_auth_request con 0x7f12501c0c30 auth_method 0 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124cff9640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f123c03fdb0 0x7f123c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(346..346 src has 1..346) ==== 6230+0+0 (secure 0 0 0) 0x7f12400732d0 con 0x7f12500a2530 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124cff9640 10 client.84422.objecter ms_dispatch 0x7f1250000b90 osd_map(346..346 src has 1..346) 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124cff9640 3 client.84422.objecter handle_osd_map got epochs [346,346] > 0 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124cff9640 3 client.84422.objecter handle_osd_map decoding full epoch 346 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124cff9640 20 client.84422.objecter dump_active .. 0 homeless 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f125490a640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f123c03fdb0 0x7f123c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124f7fe640 10 monclient(hunting): handle_auth_done global_id 65043 payload 824 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f124f7fe640 1 -- 172.21.15.98:0/17381821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1250126a90 con 0x7f12501c0c30 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.302+0000 7f125490a640 10 monclient: get_auth_request con 0x7f123c03fdb0 auth_method 0 2024-06-11T19:17:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 10 client.84422.objecter ms_handle_connect 0x7f12501c0c30 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 10 client.84422.objecter resend_mon_ops 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124f7fe640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12501c0c30 0x7f125002cb40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1244006e80 tx=0x7f1244010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1244018020 con 0x7f12501c0c30 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 10 monclient: dump: 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 10 client.84422.objecter ms_handle_connect 0x7f123c03fdb0 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f125490a640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f123c03fdb0 0x7f123c042270 secure :-1 s=READY pgs=4173 cs=0 l=1 rev1=1 crypto rx=0x7f12500a2950 tx=0x7f124800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1244010e70 con 0x7f12501c0c30 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12501c0c30 msgr2=0x7f125002cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12501c0c30 0x7f125002cb40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1244006e80 tx=0x7f1244010c80 comp rx=0 tx=0).stop 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12501c0c30 msgr2=0x7f125002cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.303+0000 7f124cff9640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12501c0c30 0x7f125002cb40 secure :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1244006e80 tx=0x7f1244010c80 comp rx=0 tx=0).stop 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.309+0000 7f125636d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.309+0000 7f125636d640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125002d080 0x7f125002d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.309+0000 7f125636d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.309+0000 7f125636d640 1 -- 172.21.15.98:0/17381821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1250073550 con 0x7f125002d080 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.309+0000 7f124f7fe640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125002d080 0x7f125002d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.309+0000 7f124f7fe640 10 monclient: get_auth_request con 0x7f125002d080 auth_method 0 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.309+0000 7f124f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.309+0000 7f124f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.309+0000 7f124f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124f7fe640 10 monclient(hunting): handle_auth_done global_id 65052 payload 824 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124f7fe640 1 -- 172.21.15.98:0/17381821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f12500a66f0 con 0x7f125002d080 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 10 client.84422.objecter ms_handle_connect 0x7f125002d080 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 10 client.84422.objecter resend_mon_ops 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124f7fe640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125002d080 0x7f125002d460 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1244019000 tx=0x7f12440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1244018040 con 0x7f125002d080 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 10 monclient: dump: 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f124400eb60 con 0x7f125002d080 2024-06-11T19:17:51.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:51.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:51.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125002d080 msgr2=0x7f125002d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125002d080 0x7f125002d460 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1244019000 tx=0x7f12440046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 1 -- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125002d080 msgr2=0x7f125002d460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:51.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f124cff9640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125002d080 0x7f125002d460 secure :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1244019000 tx=0x7f12440046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.336 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:51.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 263138, 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:51.337 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.338 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.339 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:51.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.341 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:51.344 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.348 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:51.349 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:51.350 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:51.352 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.354 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:51.354 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:51.354 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:51.354 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:51.356 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:51.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.310+0000 7f125636d640 10 librados: watch_flush enter 2024-06-11T19:17:51.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 10 librados: watch_flush exit 2024-06-11T19:17:51.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 20 client.84422.objecter shutdown clearing up homeless session... 2024-06-11T19:17:51.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 10 client.84422.objecter successfully canceled tick 2024-06-11T19:17:51.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 1 -- 172.21.15.98:0/17381821 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f123c03fdb0 msgr2=0x7f123c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f123c03fdb0 0x7f123c042270 secure :-1 s=READY pgs=4173 cs=0 l=1 rev1=1 crypto rx=0x7f12500a2950 tx=0x7f124800b040 comp rx=0 tx=0).stop 2024-06-11T19:17:51.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 10 monclient: shutdown 2024-06-11T19:17:51.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:51.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 1 -- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12500a2530 msgr2=0x7f1250137a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12500a2530 0x7f1250137a30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f12400097c0 tx=0x7f12400046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125490a640 1 -- 172.21.15.98:0/17381821 reap_dead start 2024-06-11T19:17:51.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 1 -- 172.21.15.98:0/17381821 shutdown_connections 2024-06-11T19:17:51.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 1 --2- 172.21.15.98:0/17381821 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12500a2530 0x7f1250137a30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 1 -- 172.21.15.98:0/17381821 >> 172.21.15.98:0/17381821 conn(0x7f12500ab850 msgr2=0x7f12500a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:51.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 1 -- 172.21.15.98:0/17381821 shutdown_connections 2024-06-11T19:17:51.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 1 -- 172.21.15.98:0/17381821 wait complete. 2024-06-11T19:17:51.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.311+0000 7f125636d640 1 librados: shutdown 2024-06-11T19:17:51.361 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:51.361 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:17:51.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:17:51.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.462+0000 7f3e9783e640 10 monclient: get_monmap_and_config 2024-06-11T19:17:51.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.462+0000 7f3e9783e640 10 monclient: build_initial_monmap 2024-06-11T19:17:51.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.462+0000 7f3e9783e640 10 monclient: monmap: 2024-06-11T19:17:51.461 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:51.461 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:51.463228+0000 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:51.463228+0000 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:51.463 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.462+0000 7f3e9783e640 1 Processor -- start 2024-06-11T19:17:51.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 1 -- start start 2024-06-11T19:17:51.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 10 monclient: init 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 10 monclient: _add_conns ranks=[6,9,5,0,1,2,7,4,10,3,8] 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e900a6260 0x7f3e900a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 10 monclient(hunting): picked mon.noname-d con 0x7f3e900a6260 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3e9006c890 0x7f3e900a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 10 monclient(hunting): picked mon.noname-a con 0x7f3e9006c890 addr v2:172.21.15.98:3300/0 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3e900a2d00 0x7f3e900a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 10 monclient(hunting): picked mon.noname-h con 0x7f3e900a2d00 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3e900a05b0 con 0x7f3e9006c890 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f3e900a6640 con 0x7f3e900a2d00 2024-06-11T19:17:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3e9006c650 con 0x7f3e900a6260 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e9783e640 10 monclient(hunting): authenticate will time out at 1281.270542s 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e900a6260 0x7f3e900a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e94dd9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3e9006c890 0x7f3e900a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e900a6260 0x7f3e900a1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:33182/0 (socket says 172.21.15.98:33182) 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 1 -- 172.21.15.98:0/1308193972 learned_addr learned my addr 172.21.15.98:0/1308193972 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e94dd9640 10 monclient(hunting): get_auth_request con 0x7f3e9006c890 auth_method 0 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e94dd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e94dd9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e94dd9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 10 monclient(hunting): get_auth_request con 0x7f3e900a6260 auth_method 0 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e94dd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e94dd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e94dd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e95ddb640 1 --2- 172.21.15.98:0/1308193972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3e900a2d00 0x7f3e900a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e95ddb640 10 monclient(hunting): get_auth_request con 0x7f3e900a2d00 auth_method 0 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e95ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e95ddb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e95ddb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 10 monclient(hunting): handle_auth_done global_id 55158 payload 274 2024-06-11T19:17:51.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 1 -- 172.21.15.98:0/1308193972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3e900a2d00 msgr2=0x7f3e900a4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 1 --2- 172.21.15.98:0/1308193972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3e900a2d00 0x7f3e900a4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 1 -- 172.21.15.98:0/1308193972 >> v2:172.21.15.98:3300/0 conn(0x7f3e9006c890 msgr2=0x7f3e900a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 1 --2- 172.21.15.98:0/1308193972 >> v2:172.21.15.98:3300/0 conn(0x7f3e9006c890 0x7f3e900a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 10 monclient: _finish_hunting 0 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 1 monclient: found mon.noname-d 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e955da640 1 -- 172.21.15.98:0/1308193972 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e90073b60 con 0x7f3e900a6260 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.463+0000 7f3e94dd9640 1 --2- 172.21.15.98:0/1308193972 >> v2:172.21.15.98:3300/0 conn(0x7f3e9006c890 0x7f3e900a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e955da640 1 --2- 172.21.15.98:0/1308193972 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e900a6260 0x7f3e900a1d50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3e840098a0 tx=0x7f3e8402b7c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=64a2541c282a56 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 1 -- 172.21.15.98:0/1308193972 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e84038070 con 0x7f3e900a6260 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 10 monclient: dump: 2024-06-11T19:17:51.481 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.482 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 10 monclient: _finish_auth 0 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 10 monclient: _check_auth_tickets 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e9783e640 5 monclient: authenticate success, global_id 55158 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 1 -- 172.21.15.98:0/1308193972 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e84031990 con 0x7f3e900a6260 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e9783e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e9783e640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e9783e640 4 set_mon_vals no callback set 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 1 -- 172.21.15.98:0/1308193972 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e840323b0 con 0x7f3e900a6260 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.464+0000 7f3e83fff640 10 monclient: dump: 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.484 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.465+0000 7f3e9783e640 10 monclient: shutdown 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.465+0000 7f3e9783e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.465+0000 7f3e9783e640 1 -- 172.21.15.98:0/1308193972 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e900a6260 msgr2=0x7f3e900a1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.465+0000 7f3e9783e640 1 --2- 172.21.15.98:0/1308193972 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e900a6260 0x7f3e900a1d50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3e840098a0 tx=0x7f3e8402b7c0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.465+0000 7f3e9783e640 1 -- 172.21.15.98:0/1308193972 shutdown_connections 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.465+0000 7f3e9783e640 1 --2- 172.21.15.98:0/1308193972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f3e900a2d00 0x7f3e900a4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.465+0000 7f3e9783e640 1 --2- 172.21.15.98:0/1308193972 >> v2:172.21.15.98:3300/0 conn(0x7f3e9006c890 0x7f3e900a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.465+0000 7f3e9783e640 1 --2- 172.21.15.98:0/1308193972 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3e900a6260 0x7f3e900a1d50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.465+0000 7f3e9783e640 1 -- 172.21.15.98:0/1308193972 >> 172.21.15.98:0/1308193972 conn(0x7f3e900a95b0 msgr2=0x7f3e900a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.465+0000 7f3e9783e640 1 -- 172.21.15.98:0/1308193972 shutdown_connections 2024-06-11T19:17:51.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.465+0000 7f3e9783e640 1 -- 172.21.15.98:0/1308193972 wait complete. 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.466+0000 7f3e9783e640 10 monclient: build_initial_monmap 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.466+0000 7f3e9783e640 10 monclient: monmap: 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:51.486 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:51.487 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.466+0000 7f3e9783e640 1 librados: starting msgr at 2024-06-11T19:17:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.466+0000 7f3e9783e640 1 librados: starting objecter 2024-06-11T19:17:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.466+0000 7f3e9783e640 1 Processor -- start 2024-06-11T19:17:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.466+0000 7f3e9783e640 1 -- start start 2024-06-11T19:17:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.466+0000 7f3e9783e640 1 librados: setting wanted keys 2024-06-11T19:17:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.466+0000 7f3e9783e640 1 librados: calling monclient init 2024-06-11T19:17:51.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.466+0000 7f3e9783e640 10 monclient: init 2024-06-11T19:17:51.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:51.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 10 monclient: _add_conns ranks=[4,8,7,1,2,3,5,6,0] 2024-06-11T19:17:51.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e9006c890 0x7f3e901cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 10 monclient(hunting): picked mon.noname-c con 0x7f3e9006c890 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:51.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e900a2d00 0x7f3e901ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 10 monclient(hunting): picked mon.noname-e con 0x7f3e900a2d00 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:17:51.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3e900a6260 0x7f3e901a8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 10 monclient(hunting): picked mon.noname-i con 0x7f3e900a6260 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:17:51.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3e900a2410 con 0x7f3e9006c890 2024-06-11T19:17:51.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f3e900a2290 con 0x7f3e900a6260 2024-06-11T19:17:51.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3e900a5330 con 0x7f3e900a2d00 2024-06-11T19:17:51.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:51.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 10 monclient(hunting): authenticate will time out at 1281.274226s 2024-06-11T19:17:51.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e955da640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e9006c890 0x7f3e901cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e900a2d00 0x7f3e901ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e955da640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e9006c890 0x7f3e901cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:50980/0 (socket says 172.21.15.98:50980) 2024-06-11T19:17:51.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e955da640 1 -- 172.21.15.98:0/4189858615 learned_addr learned my addr 172.21.15.98:0/4189858615 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:51.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e955da640 10 monclient(hunting): get_auth_request con 0x7f3e9006c890 auth_method 0 2024-06-11T19:17:51.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e955da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e955da640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e955da640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 10 monclient(hunting): get_auth_request con 0x7f3e900a2d00 auth_method 0 2024-06-11T19:17:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e955da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e955da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e95ddb640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3e900a6260 0x7f3e901a8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e955da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e95ddb640 10 monclient(hunting): get_auth_request con 0x7f3e900a6260 auth_method 0 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e95ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e95ddb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e95ddb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e95ddb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e95ddb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e95ddb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 10 monclient(hunting): handle_auth_done global_id 94175 payload 824 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 1 -- 172.21.15.98:0/4189858615 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3e900a6260 msgr2=0x7f3e901a8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3e900a6260 0x7f3e901a8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 1 -- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e9006c890 msgr2=0x7f3e901cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e9006c890 0x7f3e901cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 10 monclient: _finish_hunting 0 2024-06-11T19:17:51.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 1 monclient: found mon.noname-e 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 1 -- 172.21.15.98:0/4189858615 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e900a6ca0 con 0x7f3e900a2d00 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e955da640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e9006c890 0x7f3e901cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e94dd9640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e900a2d00 0x7f3e901ce080 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00e8d0 tx=0x7f3e8c00ed90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e81ffb640 10 client.?.objecter ms_handle_connect 0x7f3e900a2d00 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e81ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e8c017070 con 0x7f3e900a2d00 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e81ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e81ffb640 10 monclient: dump: 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.492 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e81ffb640 10 monclient: _finish_auth 0 2024-06-11T19:17:51.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e81ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 5 monclient: authenticate success, global_id 94175 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 10 monclient: _renew_subs 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e8c00cdb0 con 0x7f3e900a2d00 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.467+0000 7f3e9783e640 1 -- 172.21.15.98:0/4189858615 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e900a88e0 con 0x7f3e900a2d00 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e81ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e8c0138e0 con 0x7f3e900a2d00 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e9783e640 10 client.94175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e837fe640 4 set_mon_vals no callback set 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e81ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:17:51.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e81ffb640 10 monclient: dump: 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.495 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e9783e640 10 monclient: _renew_subs 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e9783e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e9783e640 1 -- 172.21.15.98:0/4189858615 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e901acd30 con 0x7f3e900a2d00 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e9783e640 1 librados: init done 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3e8c01a070 con 0x7f3e900a2d00 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e9783e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e9783e640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9420 0x7f3e901a9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e9783e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.468+0000 7f3e9783e640 1 -- 172.21.15.98:0/4189858615 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3e901ab360 con 0x7f3e901a9420 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e955da640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9420 0x7f3e901a9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e955da640 10 monclient: get_auth_request con 0x7f3e901a9420 auth_method 0 2024-06-11T19:17:51.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e955da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e955da640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e955da640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3e7403fdb0 0x7f3e74042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e955da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e955da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e955da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(346..346 src has 1..346) ==== 6230+0+0 (secure 0 0 0) 0x7f3e8c00f040 con 0x7f3e900a2d00 2024-06-11T19:17:51.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 10 client.94175.objecter ms_dispatch 0x7f3e90000b90 osd_map(346..346 src has 1..346) 2024-06-11T19:17:51.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 3 client.94175.objecter handle_osd_map got epochs [346,346] > 0 2024-06-11T19:17:51.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 3 client.94175.objecter handle_osd_map decoding full epoch 346 2024-06-11T19:17:51.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e955da640 10 monclient(hunting): handle_auth_done global_id 84431 payload 824 2024-06-11T19:17:51.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 20 client.94175.objecter dump_active .. 0 homeless 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e955da640 1 -- 172.21.15.98:0/4189858615 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e901aa9d0 con 0x7f3e901a9420 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e95ddb640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3e7403fdb0 0x7f3e74042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e95ddb640 10 monclient: get_auth_request con 0x7f3e7403fdb0 auth_method 0 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 10 client.94175.objecter ms_handle_connect 0x7f3e901a9420 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 10 client.94175.objecter resend_mon_ops 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e955da640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9420 0x7f3e901a9800 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3e840090b0 tx=0x7f3e840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e84044070 con 0x7f3e901a9420 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e95ddb640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3e7403fdb0 0x7f3e74042270 secure :-1 s=READY pgs=4174 cs=0 l=1 rev1=1 crypto rx=0x7f3e88003aa0 tx=0x7f3e88009820 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 10 monclient: dump: 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.498 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 10 client.94175.objecter ms_handle_connect 0x7f3e7403fdb0 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3e84039040 con 0x7f3e901a9420 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9420 msgr2=0x7f3e901a9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.469+0000 7f3e81ffb640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9420 0x7f3e901a9800 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3e840090b0 tx=0x7f3e840046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.470+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9420 msgr2=0x7f3e901a9800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.470+0000 7f3e81ffb640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9420 0x7f3e901a9800 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e9783e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e9783e640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9d40 0x7f3e900705e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e9783e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e9783e640 1 -- 172.21.15.98:0/4189858615 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3e90050830 con 0x7f3e901a9d40 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e955da640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9d40 0x7f3e900705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e955da640 10 monclient: get_auth_request con 0x7f3e901a9d40 auth_method 0 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e955da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e955da640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e955da640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e955da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e955da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e955da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e955da640 10 monclient(hunting): handle_auth_done global_id 84440 payload 824 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e955da640 1 -- 172.21.15.98:0/4189858615 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3e900a1d50 con 0x7f3e901a9d40 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e81ffb640 10 client.94175.objecter ms_handle_connect 0x7f3e901a9d40 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e955da640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9d40 0x7f3e900705e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3e8402b800 tx=0x7f3e84008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e81ffb640 10 client.94175.objecter resend_mon_ops 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e84038040 con 0x7f3e901a9d40 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e81ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:17:51.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.476+0000 7f3e81ffb640 10 monclient: dump: 2024-06-11T19:17:51.502 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.502 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.502 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.502 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.502 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.502 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.502 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.502 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.502 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.502 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.503 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.503 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.503 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.503 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.503 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.503 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f3e840499b0 con 0x7f3e901a9d40 2024-06-11T19:17:51.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e81ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:51.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e81ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:51.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9d40 msgr2=0x7f3e900705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e81ffb640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9d40 0x7f3e900705e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3e8402b800 tx=0x7f3e84008030 comp rx=0 tx=0).stop 2024-06-11T19:17:51.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e81ffb640 1 -- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9d40 msgr2=0x7f3e900705e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:51.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e81ffb640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e901a9d40 0x7f3e900705e0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3e8402b800 tx=0x7f3e84008030 comp rx=0 tx=0).stop 2024-06-11T19:17:51.504 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:51.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:51.504 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:51.504 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:51.504 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 143250, 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-06-11T19:17:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.506 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.507 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:51.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.509 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.511 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.513 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.516 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:51.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.522 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:51.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:51.522 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:51.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:51.523 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:51.524 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.525 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:51.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.526 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:51.526 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:51.526 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:51.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.526 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:51.526 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:51.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 10 librados: watch_flush enter 2024-06-11T19:17:51.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 10 librados: watch_flush exit 2024-06-11T19:17:51.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 20 client.94175.objecter shutdown clearing up homeless session... 2024-06-11T19:17:51.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 10 client.94175.objecter successfully canceled tick 2024-06-11T19:17:51.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 1 -- 172.21.15.98:0/4189858615 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3e7403fdb0 msgr2=0x7f3e74042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3e7403fdb0 0x7f3e74042270 secure :-1 s=READY pgs=4174 cs=0 l=1 rev1=1 crypto rx=0x7f3e88003aa0 tx=0x7f3e88009820 comp rx=0 tx=0).stop 2024-06-11T19:17:51.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 10 monclient: shutdown 2024-06-11T19:17:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 1 -- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e900a2d00 msgr2=0x7f3e901ce080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 1 --2- 172.21.15.98:0/4189858615 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e900a2d00 0x7f3e901ce080 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00e8d0 tx=0x7f3e8c00ed90 comp rx=0 tx=0).stop 2024-06-11T19:17:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e95ddb640 1 -- 172.21.15.98:0/4189858615 reap_dead start 2024-06-11T19:17:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e95ddb640 1 -- 172.21.15.98:0/4189858615 reap_dead start 2024-06-11T19:17:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 1 -- 172.21.15.98:0/4189858615 shutdown_connections 2024-06-11T19:17:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 1 -- 172.21.15.98:0/4189858615 >> 172.21.15.98:0/4189858615 conn(0x7f3e900a95b0 msgr2=0x7f3e900a7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 1 -- 172.21.15.98:0/4189858615 shutdown_connections 2024-06-11T19:17:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 1 -- 172.21.15.98:0/4189858615 wait complete. 2024-06-11T19:17:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.477+0000 7f3e9783e640 1 librados: shutdown 2024-06-11T19:17:51.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:51.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:17:51.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:17:51.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.629+0000 7fcfd51e2640 10 monclient: get_monmap_and_config 2024-06-11T19:17:51.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.629+0000 7fcfd51e2640 10 monclient: build_initial_monmap 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.629+0000 7fcfd51e2640 10 monclient: monmap: 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:51.630478+0000 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:51.630478+0000 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:51.630 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 1 Processor -- start 2024-06-11T19:17:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 1 -- start start 2024-06-11T19:17:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 10 monclient: init 2024-06-11T19:17:51.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 10 monclient: _add_conns ranks=[4,1,6,0,5,2,9,7,8,3,10] 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfd0136c90 0x7fcfd0137070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 10 monclient(hunting): picked mon.noname-c con 0x7fcfd0136c90 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fcfd01367d0 addr v1:172.21.15.117:6789/0 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcfd00a3da0 0x7fcfd013eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 10 monclient(hunting): picked mon.noname-d con 0x7fcfd00a3da0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd009fc20 con 0x7fcfd01367d0 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fcfd0068180 con 0x7fcfd0136c90 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fcfd00a0270 con 0x7fcfd00a3da0 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfd51e2640 10 monclient(hunting): authenticate will time out at 1281.437879s 2024-06-11T19:17:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.630+0000 7fcfced9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfd0136c90 0x7fcfd0137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfc7fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fcfd01367d0 0x7fcfd00a3690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:58296/0 (socket says 172.21.15.98:58296) 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfc7fff640 1 -- 172.21.15.98:0/2026321668 learned_addr learned my addr 172.21.15.98:0/2026321668 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 10 monclient(hunting): get_auth_request con 0x7fcfd0136c90 auth_method 0 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfcf59e640 1 --2- 172.21.15.98:0/2026321668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcfd00a3da0 0x7fcfd013eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfcf59e640 10 monclient(hunting): get_auth_request con 0x7fcfd00a3da0 auth_method 0 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfcf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfcf59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfcf59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfcf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfcf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfcf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 10 monclient(hunting): handle_auth_done global_id 23998 payload 274 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 1 -- 172.21.15.98:0/2026321668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcfd00a3da0 msgr2=0x7fcfd013eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 1 --2- 172.21.15.98:0/2026321668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcfd00a3da0 0x7fcfd013eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 1 -- 172.21.15.98:0/2026321668 >> v1:172.21.15.117:6789/0 conn(0x7fcfd01367d0 legacy=0x7fcfd00a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 10 monclient: _finish_hunting 0 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 1 monclient: found mon.noname-c 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfced9d640 1 -- 172.21.15.98:0/2026321668 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd009ff00 con 0x7fcfd0136c90 2024-06-11T19:17:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.631+0000 7fcfcf59e640 1 --2- 172.21.15.98:0/2026321668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcfd00a3da0 0x7fcfd013eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:51.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.632+0000 7fcfced9d640 1 --2- 172.21.15.98:0/2026321668 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfd0136c90 0x7fcfd0137070 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00097d0 tx=0x7fcfc002b810 comp rx=0 tx=0).ready entity=mon.4 client_cookie=77ebd34ad74c37ef server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 1 -- 172.21.15.98:0/2026321668 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcfc00319e0 con 0x7fcfd0136c90 2024-06-11T19:17:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:17:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 10 monclient: dump: 2024-06-11T19:17:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.640 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 10 monclient: _finish_auth 0 2024-06-11T19:17:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 10 monclient: _check_auth_tickets 2024-06-11T19:17:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfd51e2640 5 monclient: authenticate success, global_id 23998 2024-06-11T19:17:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 1 -- 172.21.15.98:0/2026321668 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfc0037070 con 0x7fcfd0136c90 2024-06-11T19:17:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfd51e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfd51e2640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfd51e2640 4 set_mon_vals no callback set 2024-06-11T19:17:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 1 -- 172.21.15.98:0/2026321668 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcfc0031ba0 con 0x7fcfd0136c90 2024-06-11T19:17:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.634+0000 7fcfce59c640 10 monclient: dump: 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.635+0000 7fcfd51e2640 10 monclient: shutdown 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.635+0000 7fcfd51e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.635+0000 7fcfd51e2640 1 -- 172.21.15.98:0/2026321668 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfd0136c90 msgr2=0x7fcfd0137070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.635+0000 7fcfd51e2640 1 --2- 172.21.15.98:0/2026321668 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfd0136c90 0x7fcfd0137070 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00097d0 tx=0x7fcfc002b810 comp rx=0 tx=0).stop 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.635+0000 7fcfd51e2640 1 -- 172.21.15.98:0/2026321668 shutdown_connections 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.635+0000 7fcfd51e2640 1 --2- 172.21.15.98:0/2026321668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcfd00a3da0 0x7fcfd013eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.635+0000 7fcfd51e2640 1 --2- 172.21.15.98:0/2026321668 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfd0136c90 0x7fcfd0137070 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.635+0000 7fcfd51e2640 1 -- 172.21.15.98:0/2026321668 >> 172.21.15.98:0/2026321668 conn(0x7fcfd00a4780 msgr2=0x7fcfd00a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.635+0000 7fcfd51e2640 1 -- 172.21.15.98:0/2026321668 shutdown_connections 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.635+0000 7fcfd51e2640 1 -- 172.21.15.98:0/2026321668 wait complete. 2024-06-11T19:17:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.636+0000 7fcfd51e2640 10 monclient: build_initial_monmap 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.636+0000 7fcfd51e2640 10 monclient: monmap: 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.636+0000 7fcfd51e2640 1 librados: starting msgr at 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.636+0000 7fcfd51e2640 1 librados: starting objecter 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.636+0000 7fcfd51e2640 1 Processor -- start 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.636+0000 7fcfd51e2640 1 -- start start 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.636+0000 7fcfd51e2640 1 librados: setting wanted keys 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.636+0000 7fcfd51e2640 1 librados: calling monclient init 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.636+0000 7fcfd51e2640 10 monclient: init 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.636+0000 7fcfd51e2640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 10 monclient: _add_conns ranks=[4,2,7,1,5,0,6,8,3] 2024-06-11T19:17:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfd00a3da0 0x7fcfd013f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 10 monclient(hunting): picked mon.noname-c con 0x7fcfd00a3da0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd01367d0 0x7fcfd013a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 10 monclient(hunting): picked mon.noname-b con 0x7fcfd01367d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcfd0136c90 0x7fcfd013a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 10 monclient(hunting): picked mon.noname-i con 0x7fcfd0136c90 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd01375b0 con 0x7fcfd01367d0 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fcfd0068560 con 0x7fcfd00a3da0 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fcfd013def0 con 0x7fcfd0136c90 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfd51e2640 10 monclient(hunting): authenticate will time out at 1281.444068s 2024-06-11T19:17:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfced9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfd00a3da0 0x7fcfd013f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd01367d0 0x7fcfd013a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfced9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfd00a3da0 0x7fcfd013f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:51004/0 (socket says 172.21.15.98:51004) 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfced9d640 1 -- 172.21.15.98:0/614792825 learned_addr learned my addr 172.21.15.98:0/614792825 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfced9d640 10 monclient(hunting): get_auth_request con 0x7fcfd00a3da0 auth_method 0 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfced9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd01367d0 0x7fcfd013a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45248/0 (socket says 172.21.15.98:45248) 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfced9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 10 monclient(hunting): get_auth_request con 0x7fcfd01367d0 auth_method 0 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfcf59e640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcfd0136c90 0x7fcfd013a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfcf59e640 10 monclient(hunting): get_auth_request con 0x7fcfd0136c90 auth_method 0 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfcf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfcf59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfcf59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfcf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfcf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfcf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 10 monclient(hunting): handle_auth_done global_id 84449 payload 824 2024-06-11T19:17:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 1 -- 172.21.15.98:0/614792825 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcfd0136c90 msgr2=0x7fcfd013a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcfd0136c90 0x7fcfd013a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 1 -- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfd00a3da0 msgr2=0x7fcfd013f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfd00a3da0 0x7fcfd013f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 1 monclient: found mon.noname-b 2024-06-11T19:17:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:17:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 1 -- 172.21.15.98:0/614792825 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd013d930 con 0x7fcfd01367d0 2024-06-11T19:17:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfced9d640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfd00a3da0 0x7fcfd013f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc7fff640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd01367d0 0x7fcfd013a090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800c400 tx=0x7fcfb800c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc77fe640 10 client.?.objecter ms_handle_connect 0x7fcfd01367d0 2024-06-11T19:17:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.637+0000 7fcfc77fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcfb8017070 con 0x7fcfd01367d0 2024-06-11T19:17:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:17:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 10 monclient: dump: 2024-06-11T19:17:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 10 monclient: _finish_auth 0 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 10 monclient: _check_auth_tickets 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfb80070d0 con 0x7fcfd01367d0 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcfb8007730 con 0x7fcfd01367d0 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfc77fe640 10 monclient: dump: 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfd51e2640 5 monclient: authenticate success, global_id 84449 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfd51e2640 10 monclient: _renew_subs 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfd51e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:17:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfd51e2640 1 -- 172.21.15.98:0/614792825 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd013c6f0 con 0x7fcfd01367d0 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfd51e2640 10 client.84449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfd51e2640 10 monclient: _renew_subs 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfd51e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfd51e2640 1 -- 172.21.15.98:0/614792825 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd00a35a0 con 0x7fcfd01367d0 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.641+0000 7fcfd51e2640 1 librados: init done 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.642+0000 7fcfcdd9b640 4 set_mon_vals no callback set 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.642+0000 7fcfd51e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.642+0000 7fcfd51e2640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4002880 0x7fcfb4004d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.642+0000 7fcfd51e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.642+0000 7fcfd51e2640 1 -- 172.21.15.98:0/614792825 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fcfb4007cb0 con 0x7fcfb4002880 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fcfb80072c0 con 0x7fcfd01367d0 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfced9d640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4002880 0x7fcfb4004d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfced9d640 10 monclient: get_auth_request con 0x7fcfb4002880 auth_method 0 2024-06-11T19:17:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfced9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfced9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcfb003fce0 0x7fcfb00421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(347..347 src has 1..347) ==== 6230+0+0 (secure 0 0 0) 0x7fcfb8050a90 con 0x7fcfd01367d0 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 10 client.84449.objecter ms_dispatch 0x7fcfd0000b90 osd_map(347..347 src has 1..347) 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 3 client.84449.objecter handle_osd_map got epochs [347,347] > 0 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 3 client.84449.objecter handle_osd_map decoding full epoch 347 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfced9d640 10 monclient(hunting): handle_auth_done global_id 24016 payload 824 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfced9d640 1 -- 172.21.15.98:0/614792825 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfb4007eb0 con 0x7fcfb4002880 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 20 client.84449.objecter dump_active .. 0 homeless 2024-06-11T19:17:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 10 client.84449.objecter ms_handle_connect 0x7fcfb4002880 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 10 client.84449.objecter resend_mon_ops 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfced9d640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4002880 0x7fcfb4004d20 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00097a0 tx=0x7fcfc00046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcfc004a070 con 0x7fcfb4002880 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.643+0000 7fcfc77fe640 10 monclient: dump: 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.644+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcfc0037040 con 0x7fcfb4002880 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.644+0000 7fcfc77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.644+0000 7fcfc77fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.644+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4002880 msgr2=0x7fcfb4004d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.644+0000 7fcfc77fe640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4002880 0x7fcfb4004d20 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00097a0 tx=0x7fcfc00046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.644+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4002880 msgr2=0x7fcfb4004d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.644+0000 7fcfc77fe640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4002880 0x7fcfb4004d20 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.644+0000 7fcfcf59e640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcfb003fce0 0x7fcfb00421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.644+0000 7fcfcf59e640 10 monclient: get_auth_request con 0x7fcfb003fce0 auth_method 0 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.644+0000 7fcfc77fe640 10 client.84449.objecter ms_handle_connect 0x7fcfb003fce0 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.644+0000 7fcfcf59e640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcfb003fce0 0x7fcfb00421a0 secure :-1 s=READY pgs=4175 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8011420 tx=0x7fcfc800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfd51e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfd51e2640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4006d20 0x7fcfb40071b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfd51e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfd51e2640 1 -- 172.21.15.98:0/614792825 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fcfb400a1f0 con 0x7fcfb4006d20 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfced9d640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4006d20 0x7fcfb40071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfced9d640 10 monclient: get_auth_request con 0x7fcfb4006d20 auth_method 0 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfced9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfced9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfced9d640 10 monclient(hunting): handle_auth_done global_id 24025 payload 824 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.650+0000 7fcfced9d640 1 -- 172.21.15.98:0/614792825 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcfb40076f0 con 0x7fcfb4006d20 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 10 client.84449.objecter ms_handle_connect 0x7fcfb4006d20 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 10 client.84449.objecter resend_mon_ops 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfced9d640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4006d20 0x7fcfb40071b0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4007e80 tx=0x7fcfc0009190 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcfc004a050 con 0x7fcfb4006d20 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:17:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 10 monclient: dump: 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.668 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fcfc004b040 con 0x7fcfb4006d20 2024-06-11T19:17:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4006d20 msgr2=0x7fcfb40071b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4006d20 0x7fcfb40071b0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4007e80 tx=0x7fcfc0009190 comp rx=0 tx=0).stop 2024-06-11T19:17:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 1 -- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4006d20 msgr2=0x7fcfb40071b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfc77fe640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcfb4006d20 0x7fcfb40071b0 secure :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4007e80 tx=0x7fcfc0009190 comp rx=0 tx=0).stop 2024-06-11T19:17:51.668 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 521869, 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:51.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:51.685 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.689 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:51.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.691 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:51.691 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:51.691 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:17:51.691 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.691 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.691 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:51.691 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:51.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 10 librados: watch_flush enter 2024-06-11T19:17:51.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 10 librados: watch_flush exit 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 20 client.84449.objecter shutdown clearing up homeless session... 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 10 client.84449.objecter successfully canceled tick 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 1 -- 172.21.15.98:0/614792825 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcfb003fce0 msgr2=0x7fcfb00421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcfb003fce0 0x7fcfb00421a0 secure :-1 s=READY pgs=4175 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8011420 tx=0x7fcfc800b040 comp rx=0 tx=0).stop 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 10 monclient: shutdown 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 1 -- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd01367d0 msgr2=0x7fcfd013a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 1 --2- 172.21.15.98:0/614792825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfd01367d0 0x7fcfd013a090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800c400 tx=0x7fcfb800c800 comp rx=0 tx=0).stop 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfcf59e640 1 -- 172.21.15.98:0/614792825 reap_dead start 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfcf59e640 1 -- 172.21.15.98:0/614792825 reap_dead start 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 1 -- 172.21.15.98:0/614792825 shutdown_connections 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 1 -- 172.21.15.98:0/614792825 >> 172.21.15.98:0/614792825 conn(0x7fcfd00a4780 msgr2=0x7fcfd009e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:51.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 1 -- 172.21.15.98:0/614792825 shutdown_connections 2024-06-11T19:17:51.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.651+0000 7fcfd51e2640 1 -- 172.21.15.98:0/614792825 wait complete. 2024-06-11T19:17:51.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.652+0000 7fcfd51e2640 1 librados: shutdown 2024-06-11T19:17:51.694 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:51.694 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:17:51.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:17:51.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.793+0000 7f323d469640 10 monclient: get_monmap_and_config 2024-06-11T19:17:51.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.793+0000 7f323d469640 10 monclient: build_initial_monmap 2024-06-11T19:17:51.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.793+0000 7f323d469640 10 monclient: monmap: 2024-06-11T19:17:51.792 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:51.794413+0000 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:51.794413+0000 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:51.793 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:51.794 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:51.794 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:51.794 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.793+0000 7f323d469640 1 Processor -- start 2024-06-11T19:17:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 1 -- start start 2024-06-11T19:17:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 10 monclient: init 2024-06-11T19:17:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 10 monclient: _add_conns ranks=[1,10,3,6,0,8,2,5,9,4,7] 2024-06-11T19:17:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f32380a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:17:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f32380a2530 0x7f323813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 10 monclient(hunting): picked mon.noname-f con 0x7f32380a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:17:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f323813e1a0 0x7f3238140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 10 monclient(hunting): picked mon.noname-g con 0x7f323813e1a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32380a7140 con 0x7f32380a4920 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f32380a2b50 con 0x7f32380a2530 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f32380a2e30 con 0x7f323813e1a0 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f323d469640 10 monclient(hunting): authenticate will time out at 1281.601527s 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32367fc640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f32380a2530 0x7f323813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32377fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f323813e1a0 0x7f3238140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32367fc640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f32380a2530 0x7f323813dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:41668/0 (socket says 172.21.15.98:41668) 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32377fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f323813e1a0 0x7f3238140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:35792/0 (socket says 172.21.15.98:35792) 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32367fc640 1 -- 172.21.15.98:0/1096642702 learned_addr learned my addr 172.21.15.98:0/1096642702 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f3236ffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f32380a4920 0x7f32380a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:58298/0 (socket says 172.21.15.98:58298) 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32367fc640 10 monclient(hunting): get_auth_request con 0x7f32380a2530 auth_method 0 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32367fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32367fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32377fe640 10 monclient(hunting): get_auth_request con 0x7f323813e1a0 auth_method 0 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.794+0000 7f32367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32367fc640 10 monclient(hunting): handle_auth_done global_id 32176 payload 274 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32367fc640 1 -- 172.21.15.98:0/1096642702 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f323813e1a0 msgr2=0x7f3238140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32367fc640 1 --2- 172.21.15.98:0/1096642702 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f323813e1a0 0x7f3238140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32367fc640 1 -- 172.21.15.98:0/1096642702 >> v1:172.21.15.117:6789/0 conn(0x7f32380a4920 legacy=0x7f32380a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32367fc640 10 monclient: _finish_hunting 0 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32367fc640 1 monclient: found mon.noname-f 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32367fc640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32367fc640 1 -- 172.21.15.98:0/1096642702 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32380a3110 con 0x7f32380a2530 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.795+0000 7f32367fc640 1 --2- 172.21.15.98:0/1096642702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32380a2530 0x7f323813dc60 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f322c009870 tx=0x7f322c02b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95cd43e8d718418e server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 1 -- 172.21.15.98:0/1096642702 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f322c031a30 con 0x7f32380a2530 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:17:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 10 monclient: dump: 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 10 monclient: _finish_auth 0 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 1 -- 172.21.15.98:0/1096642702 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f322c037070 con 0x7f32380a2530 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 1 -- 172.21.15.98:0/1096642702 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f322c031c30 con 0x7f32380a2530 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 5 monclient: authenticate success, global_id 32176 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 4 set_mon_vals no callback set 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:17:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f3235ffb640 10 monclient: dump: 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 10 monclient: shutdown 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 1 -- 172.21.15.98:0/1096642702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32380a2530 msgr2=0x7f323813dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 1 --2- 172.21.15.98:0/1096642702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32380a2530 0x7f323813dc60 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f322c009870 tx=0x7f322c02b860 comp rx=0 tx=0).stop 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 1 -- 172.21.15.98:0/1096642702 shutdown_connections 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 1 --2- 172.21.15.98:0/1096642702 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f323813e1a0 0x7f3238140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 1 --2- 172.21.15.98:0/1096642702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32380a2530 0x7f323813dc60 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 1 -- 172.21.15.98:0/1096642702 >> 172.21.15.98:0/1096642702 conn(0x7f32380ab850 msgr2=0x7f32380a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 1 -- 172.21.15.98:0/1096642702 shutdown_connections 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.796+0000 7f323d469640 1 -- 172.21.15.98:0/1096642702 wait complete. 2024-06-11T19:17:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.797+0000 7f323d469640 10 monclient: build_initial_monmap 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.797+0000 7f323d469640 10 monclient: monmap: 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.797+0000 7f323d469640 1 librados: starting msgr at 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.797+0000 7f323d469640 1 librados: starting objecter 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.797+0000 7f323d469640 1 Processor -- start 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.797+0000 7f323d469640 1 -- start start 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.797+0000 7f323d469640 1 librados: setting wanted keys 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.797+0000 7f323d469640 1 librados: calling monclient init 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.797+0000 7f323d469640 10 monclient: init 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 10 monclient: _add_conns ranks=[0,8,5,1,2,4,7,6,3] 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32380a2530 0x7f32381cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 10 monclient(hunting): picked mon.noname-a con 0x7f32380a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32380a4920 0x7f32381cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 10 monclient(hunting): picked mon.noname-e con 0x7f32380a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f323813e1a0 0x7f32381ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 10 monclient(hunting): picked mon.noname-h con 0x7f323813e1a0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3238138e50 con 0x7f32380a2530 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f32380a84b0 con 0x7f323813e1a0 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f32380a6da0 con 0x7f32380a4920 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f323d469640 10 monclient(hunting): authenticate will time out at 1281.605400s 2024-06-11T19:17:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f3236ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32380a2530 0x7f32381cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32380a4920 0x7f32381cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32380a4920 0x7f32381cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39092/0 (socket says 172.21.15.98:39092) 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f3236ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32380a2530 0x7f32381cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56708/0 (socket says 172.21.15.98:56708) 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 1 -- 172.21.15.98:0/3578418943 learned_addr learned my addr 172.21.15.98:0/3578418943 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 10 monclient(hunting): get_auth_request con 0x7f32380a4920 auth_method 0 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f3236ffd640 10 monclient(hunting): get_auth_request con 0x7f32380a2530 auth_method 0 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f3236ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f3236ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f3236ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32377fe640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f323813e1a0 0x7f32381ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f3236ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f3236ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f3236ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32377fe640 10 monclient(hunting): get_auth_request con 0x7f323813e1a0 auth_method 0 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32377fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32377fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 10 monclient(hunting): handle_auth_done global_id 94184 payload 824 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 1 -- 172.21.15.98:0/3578418943 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f323813e1a0 msgr2=0x7f32381ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f323813e1a0 0x7f32381ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 1 -- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32380a2530 msgr2=0x7f32381cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32380a2530 0x7f32381cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 10 monclient: _finish_hunting 0 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 1 monclient: found mon.noname-e 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f32367fc640 1 -- 172.21.15.98:0/3578418943 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32380a0880 con 0x7f32380a4920 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.798+0000 7f3236ffd640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32380a2530 0x7f32381cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f32367fc640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32380a4920 0x7f32381cdc60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f322c0090b0 tx=0x7f322c0046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 10 client.?.objecter ms_handle_connect 0x7f32380a4920 2024-06-11T19:17:51.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f322c041070 con 0x7f32380a4920 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 10 monclient: dump: 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.817 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f323d469640 5 monclient: authenticate success, global_id 94184 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f322c003e10 con 0x7f32380a4920 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f322c031a30 con 0x7f32380a4920 2024-06-11T19:17:51.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f323d469640 10 monclient: _renew_subs 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f323d469640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f323d469640 1 -- 172.21.15.98:0/3578418943 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f323809cdd0 con 0x7f32380a4920 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f32357fa640 4 set_mon_vals no callback set 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f323d469640 10 client.94184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 10 monclient: dump: 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.819 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f323d469640 10 monclient: _renew_subs 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f323d469640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f323d469640 1 -- 172.21.15.98:0/3578418943 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f323809f090 con 0x7f32380a4920 2024-06-11T19:17:51.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f323d469640 1 librados: init done 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.799+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f322c04c060 con 0x7f32380a4920 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f323d469640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f323d469640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32380a4d00 0x7f3238133810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f323d469640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f323d469640 1 -- 172.21.15.98:0/3578418943 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f323809f3d0 con 0x7f32380a4d00 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f3236ffd640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32380a4d00 0x7f3238133810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f3236ffd640 10 monclient: get_auth_request con 0x7f32380a4d00 auth_method 0 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f3236ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f3236ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f3236ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f3236ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f322403fdb0 0x7f3224042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f3236ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f3236ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(347..347 src has 1..347) ==== 6230+0+0 (secure 0 0 0) 0x7f322c072d50 con 0x7f32380a4920 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 10 client.94184.objecter ms_dispatch 0x7f3238000b90 osd_map(347..347 src has 1..347) 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 3 client.94184.objecter handle_osd_map got epochs [347,347] > 0 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 3 client.94184.objecter handle_osd_map decoding full epoch 347 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f3236ffd640 10 monclient(hunting): handle_auth_done global_id 55176 payload 824 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f3236ffd640 1 -- 172.21.15.98:0/3578418943 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3238073140 con 0x7f32380a4d00 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f32377fe640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f322403fdb0 0x7f3224042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 20 client.94184.objecter dump_active .. 0 homeless 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f32377fe640 10 monclient: get_auth_request con 0x7f322403fdb0 auth_method 0 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 10 client.94184.objecter ms_handle_connect 0x7f32380a4d00 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f3236ffd640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32380a4d00 0x7f3238133810 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3228001880 tx=0x7f3228001c80 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 10 client.94184.objecter resend_mon_ops 2024-06-11T19:17:51.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f322800fe00 con 0x7f32380a4d00 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 10 monclient: dump: 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.823 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 10 client.94184.objecter ms_handle_connect 0x7f322403fdb0 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f32377fe640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f322403fdb0 0x7f3224042270 secure :-1 s=READY pgs=4176 cs=0 l=1 rev1=1 crypto rx=0x7f323813d680 tx=0x7f323000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3228018070 con 0x7f32380a4d00 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.800+0000 7f321f7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.801+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32380a4d00 msgr2=0x7f3238133810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.801+0000 7f321f7fe640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32380a4d00 0x7f3238133810 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3228001880 tx=0x7f3228001c80 comp rx=0 tx=0).stop 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.801+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32380a4d00 msgr2=0x7f3238133810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:51.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.801+0000 7f321f7fe640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32380a4d00 0x7f3238133810 secure :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3228001880 tx=0x7f3228001c80 comp rx=0 tx=0).stop 2024-06-11T19:17:51.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f323d469640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:51.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f323d469640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3238133d50 0x7f3238134160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f323d469640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f323d469640 1 -- 172.21.15.98:0/3578418943 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f32381346a0 con 0x7f3238133d50 2024-06-11T19:17:51.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f3236ffd640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3238133d50 0x7f3238134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f3236ffd640 10 monclient: get_auth_request con 0x7f3238133d50 auth_method 0 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f3236ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f3236ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f3236ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f3236ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f3236ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f3236ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f3236ffd640 10 monclient(hunting): handle_auth_done global_id 55185 payload 824 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f3236ffd640 1 -- 172.21.15.98:0/3578418943 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f323806c300 con 0x7f3238133d50 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f3236ffd640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3238133d50 0x7f3238134160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3228017000 tx=0x7f3228007f90 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f321f7fe640 10 client.94184.objecter ms_handle_connect 0x7f3238133d50 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f321f7fe640 10 client.94184.objecter resend_mon_ops 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3228013a70 con 0x7f3238133d50 2024-06-11T19:17:51.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f321f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f321f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.807+0000 7f321f7fe640 10 monclient: dump: 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.828 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.829 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.829 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.829 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f3228013600 con 0x7f3238133d50 2024-06-11T19:17:51.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f321f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:51.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f321f7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:51.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3238133d50 msgr2=0x7f3238134160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f321f7fe640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3238133d50 0x7f3238134160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3228017000 tx=0x7f3228007f90 comp rx=0 tx=0).stop 2024-06-11T19:17:51.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f321f7fe640 1 -- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3238133d50 msgr2=0x7f3238134160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:51.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f321f7fe640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3238133d50 0x7f3238134160 secure :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3228017000 tx=0x7f3228007f90 comp rx=0 tx=0).stop 2024-06-11T19:17:51.829 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:51.830 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:51.830 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:51.830 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:51.830 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:51.830 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 384563, 2024-06-11T19:17:51.830 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:51.830 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:51.830 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:51.830 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:51.830 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:51.830 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:51.830 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.831 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.832 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:51.833 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.834 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:51.835 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:51.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:51.839 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:51.841 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.843 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.844 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.846 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:51.848 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:51.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:51.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.852 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:51.852 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:51.852 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:17:51.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.852 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.852 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:51.852 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:51.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 10 librados: watch_flush enter 2024-06-11T19:17:51.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 10 librados: watch_flush exit 2024-06-11T19:17:51.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 20 client.94184.objecter shutdown clearing up homeless session... 2024-06-11T19:17:51.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 10 client.94184.objecter successfully canceled tick 2024-06-11T19:17:51.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 1 -- 172.21.15.98:0/3578418943 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f322403fdb0 msgr2=0x7f3224042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f322403fdb0 0x7f3224042270 secure :-1 s=READY pgs=4176 cs=0 l=1 rev1=1 crypto rx=0x7f323813d680 tx=0x7f323000b040 comp rx=0 tx=0).stop 2024-06-11T19:17:51.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 10 monclient: shutdown 2024-06-11T19:17:51.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:51.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 1 -- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32380a4920 msgr2=0x7f32381cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f32377fe640 1 -- 172.21.15.98:0/3578418943 reap_dead start 2024-06-11T19:17:51.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32380a4920 0x7f32381cdc60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f322c0090b0 tx=0x7f322c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 1 -- 172.21.15.98:0/3578418943 shutdown_connections 2024-06-11T19:17:51.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 1 --2- 172.21.15.98:0/3578418943 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32380a4920 0x7f32381cdc60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 1 -- 172.21.15.98:0/3578418943 >> 172.21.15.98:0/3578418943 conn(0x7f32380ab850 msgr2=0x7f32380a8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:51.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 1 -- 172.21.15.98:0/3578418943 shutdown_connections 2024-06-11T19:17:51.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 1 -- 172.21.15.98:0/3578418943 wait complete. 2024-06-11T19:17:51.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.808+0000 7f323d469640 1 librados: shutdown 2024-06-11T19:17:51.855 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:51.855 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:17:51.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:17:51.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.955+0000 7f425b835640 10 monclient: get_monmap_and_config 2024-06-11T19:17:51.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.955+0000 7f425b835640 10 monclient: build_initial_monmap 2024-06-11T19:17:51.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.955+0000 7f425b835640 10 monclient: monmap: 2024-06-11T19:17:51.955 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:51.955 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.955 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:51.956469+0000 2024-06-11T19:17:51.955 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:51.956469+0000 2024-06-11T19:17:51.955 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:51.955 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.956 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:51.956 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:51.956 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:51.956 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:51.956 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:51.956 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:51.956 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:51.956 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:51.956 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:51.956 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:51.956 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:51.956 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 1 Processor -- start 2024-06-11T19:17:51.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 1 -- start start 2024-06-11T19:17:51.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 10 monclient: init 2024-06-11T19:17:51.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:51.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 10 monclient: _add_conns ranks=[9,3,5,10,2,6,8,1,0,4,7] 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f42540a4920 0x7f42540a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 10 monclient(hunting): picked mon.noname-a con 0x7f42540a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42540a5240 0x7f42540a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 10 monclient(hunting): picked mon.noname-g con 0x7f42540a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42540a2530 0x7f42540a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 10 monclient(hunting): picked mon.noname-h con 0x7f42540a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f42540a7140 con 0x7f42540a4920 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f42540a2b50 con 0x7f42540a5240 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f42540a2e30 con 0x7f42540a2530 2024-06-11T19:17:51.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f425b835640 10 monclient(hunting): authenticate will time out at 1281.763893s 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f42595d1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f42540a4920 0x7f42540a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.956+0000 7f42595d1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f42540a4920 0x7f42540a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56722/0 (socket says 172.21.15.98:56722) 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 1 -- 172.21.15.98:0/591872706 learned_addr learned my addr 172.21.15.98:0/591872706 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 10 monclient(hunting): get_auth_request con 0x7f42540a4920 auth_method 0 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f4259dd2640 1 --2- 172.21.15.98:0/591872706 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42540a2530 0x7f42540a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 10 monclient(hunting): handle_auth_done global_id 65079 payload 274 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 1 -- 172.21.15.98:0/591872706 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42540a2530 msgr2=0x7f42540a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 1 --2- 172.21.15.98:0/591872706 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42540a2530 0x7f42540a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 1 -- 172.21.15.98:0/591872706 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42540a5240 msgr2=0x7f42540a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 1 --2- 172.21.15.98:0/591872706 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42540a5240 0x7f42540a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 10 monclient: _finish_hunting 0 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 1 monclient: found mon.noname-a 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 1 -- 172.21.15.98:0/591872706 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42540a3110 con 0x7f42540a4920 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.957+0000 7f42595d1640 1 --2- 172.21.15.98:0/591872706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42540a4920 0x7f42540a4d00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4250009870 tx=0x7f425002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eaaeb9833f93fd2 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 1 -- 172.21.15.98:0/591872706 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4250038070 con 0x7f42540a4920 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 10 monclient: dump: 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 10 monclient: _finish_auth 0 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 10 monclient: _check_auth_tickets 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 5 monclient: authenticate success, global_id 65079 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 1 -- 172.21.15.98:0/591872706 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42500319b0 con 0x7f42540a4920 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 4 set_mon_vals no callback set 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 1 -- 172.21.15.98:0/591872706 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f425002f050 con 0x7f42540a4920 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f4247fff640 10 monclient: dump: 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.971 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 10 monclient: shutdown 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 1 -- 172.21.15.98:0/591872706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42540a4920 msgr2=0x7f42540a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 1 --2- 172.21.15.98:0/591872706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42540a4920 0x7f42540a4d00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4250009870 tx=0x7f425002b7e0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 1 -- 172.21.15.98:0/591872706 shutdown_connections 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 1 --2- 172.21.15.98:0/591872706 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42540a2530 0x7f42540a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 1 --2- 172.21.15.98:0/591872706 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42540a5240 0x7f42540a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 1 --2- 172.21.15.98:0/591872706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42540a4920 0x7f42540a4d00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 1 -- 172.21.15.98:0/591872706 >> 172.21.15.98:0/591872706 conn(0x7f42540ab850 msgr2=0x7f42540a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 1 -- 172.21.15.98:0/591872706 shutdown_connections 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.958+0000 7f425b835640 1 -- 172.21.15.98:0/591872706 wait complete. 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.959+0000 7f425b835640 10 monclient: build_initial_monmap 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.959+0000 7f425b835640 10 monclient: monmap: 2024-06-11T19:17:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:51.973 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.959+0000 7f425b835640 1 librados: starting msgr at 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.959+0000 7f425b835640 1 librados: starting objecter 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.959+0000 7f425b835640 1 Processor -- start 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.959+0000 7f425b835640 1 -- start start 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.959+0000 7f425b835640 1 librados: setting wanted keys 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.959+0000 7f425b835640 1 librados: calling monclient init 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.959+0000 7f425b835640 10 monclient: init 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 10 monclient: _add_conns ranks=[5,1,6,8,3,0,7,4,2] 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42540a2530 0x7f4254137620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 10 monclient(hunting): picked mon.noname-h con 0x7f42540a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42540a4920 0x7f4254137b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 10 monclient(hunting): picked mon.noname-f con 0x7f42540a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:17:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f42540a5240 0x7f42541231a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 10 monclient(hunting): picked mon.noname-d con 0x7f42540a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f42540a6da0 con 0x7f42540a4920 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f4254138e50 con 0x7f42540a2530 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f42540a6420 con 0x7f42540a5240 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 10 monclient(hunting): authenticate will time out at 1281.767069s 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f42540a5240 0x7f42541231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f42540a5240 0x7f42541231a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:33218/0 (socket says 172.21.15.98:33218) 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 1 -- 172.21.15.98:0/390722289 learned_addr learned my addr 172.21.15.98:0/390722289 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 10 monclient(hunting): get_auth_request con 0x7f42540a5240 auth_method 0 2024-06-11T19:17:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f42595d1640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42540a2530 0x7f4254137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f42595d1640 10 monclient(hunting): get_auth_request con 0x7f42540a2530 auth_method 0 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f42595d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f42595d1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f42595d1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4258dd0640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42540a4920 0x7f4254137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f42595d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f42595d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f42595d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4258dd0640 10 monclient(hunting): get_auth_request con 0x7f42540a4920 auth_method 0 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4258dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4258dd0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4258dd0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 10 monclient(hunting): handle_auth_done global_id 55194 payload 824 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 1 -- 172.21.15.98:0/390722289 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42540a2530 msgr2=0x7f4254137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42540a2530 0x7f4254137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 1 -- 172.21.15.98:0/390722289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42540a4920 msgr2=0x7f4254137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42540a4920 0x7f4254137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 10 monclient: _finish_hunting 0 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 1 monclient: found mon.noname-d 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 1 -- 172.21.15.98:0/390722289 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4254127020 con 0x7f42540a5240 2024-06-11T19:17:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4259dd2640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f42540a5240 0x7f42541231a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f424c00cd70 tx=0x7f424c00a660 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 10 client.?.objecter ms_handle_connect 0x7f42540a5240 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f424c00e860 con 0x7f42540a5240 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 10 monclient: dump: 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.979 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 10 monclient: _finish_auth 0 2024-06-11T19:17:51.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 5 monclient: authenticate success, global_id 55194 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f424c00ea60 con 0x7f42540a5240 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 10 monclient: _renew_subs 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f425b835640 1 -- 172.21.15.98:0/390722289 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4254124cc0 con 0x7f42540a5240 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f424c00ec50 con 0x7f42540a5240 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f4245ffb640 10 monclient: dump: 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.980 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.960+0000 7f42477fe640 4 set_mon_vals no callback set 2024-06-11T19:17:51.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.961+0000 7f425b835640 10 client.55194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.961+0000 7f425b835640 10 monclient: _renew_subs 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.961+0000 7f425b835640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.961+0000 7f425b835640 1 -- 172.21.15.98:0/390722289 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f42541380a0 con 0x7f42540a5240 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.961+0000 7f425b835640 1 librados: init done 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.961+0000 7f425b835640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.961+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f424c00e3f0 con 0x7f42540a5240 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.961+0000 7f425b835640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f42541236e0 0x7f4254123ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.961+0000 7f425b835640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.961+0000 7f425b835640 1 -- 172.21.15.98:0/390722289 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4254050830 con 0x7f42541236e0 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f42595d1640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f42541236e0 0x7f4254123ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f42595d1640 10 monclient: get_auth_request con 0x7f42541236e0 auth_method 0 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f42595d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f42595d1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f42595d1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f423803fde0 0x7f42380422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f42595d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f42595d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f42595d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(347..347 src has 1..347) ==== 6230+0+0 (secure 0 0 0) 0x7f424c052440 con 0x7f42540a5240 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 10 client.55194.objecter ms_dispatch 0x7f4254000b90 osd_map(347..347 src has 1..347) 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 3 client.55194.objecter handle_osd_map got epochs [347,347] > 0 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 3 client.55194.objecter handle_osd_map decoding full epoch 347 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4258dd0640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f423803fde0 0x7f42380422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4258dd0640 10 monclient: get_auth_request con 0x7f423803fde0 auth_method 0 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 20 client.55194.objecter dump_active .. 0 homeless 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f42595d1640 10 monclient(hunting): handle_auth_done global_id 94193 payload 824 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f42595d1640 1 -- 172.21.15.98:0/390722289 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42540a4d00 con 0x7f42541236e0 2024-06-11T19:17:51.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f42595d1640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f42541236e0 0x7f4254123ac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4254073de0 tx=0x7f42500046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 10 client.55194.objecter ms_handle_connect 0x7f42541236e0 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 10 client.55194.objecter resend_mon_ops 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4250038040 con 0x7f42541236e0 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4258dd0640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f423803fde0 0x7f42380422a0 secure :-1 s=READY pgs=4177 cs=0 l=1 rev1=1 crypto rx=0x7f42540a5620 tx=0x7f4248008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 10 monclient: dump: 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.984 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.962+0000 7f4245ffb640 10 client.55194.objecter ms_handle_connect 0x7f423803fde0 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.963+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4250031a50 con 0x7f42541236e0 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.963+0000 7f4245ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.963+0000 7f4245ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.963+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f42541236e0 msgr2=0x7f4254123ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.963+0000 7f4245ffb640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f42541236e0 0x7f4254123ac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4254073de0 tx=0x7f42500046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.963+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f42541236e0 msgr2=0x7f4254123ac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:51.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.963+0000 7f4245ffb640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f42541236e0 0x7f4254123ac0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4254073de0 tx=0x7f42500046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:51.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f425b835640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f425b835640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4254124000 0x7f425409e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f425b835640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f425b835640 1 -- 172.21.15.98:0/390722289 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f42540a6720 con 0x7f4254124000 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f42595d1640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4254124000 0x7f425409e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f42595d1640 10 monclient: get_auth_request con 0x7f4254124000 auth_method 0 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f42595d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f42595d1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f42595d1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f42595d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f42595d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f42595d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f42595d1640 10 monclient(hunting): handle_auth_done global_id 94202 payload 824 2024-06-11T19:17:51.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f42595d1640 1 -- 172.21.15.98:0/390722289 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f42540a4f00 con 0x7f4254124000 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f42595d1640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4254124000 0x7f425409e9f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4250009870 tx=0x7f4250031000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f4245ffb640 10 client.55194.objecter ms_handle_connect 0x7f4254124000 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.969+0000 7f4245ffb640 10 client.55194.objecter resend_mon_ops 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4250050070 con 0x7f4254124000 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f4245ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f4245ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f4245ffb640 10 monclient: dump: 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:51.989 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f425004a850 con 0x7f4254124000 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f4245ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f4245ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4254124000 msgr2=0x7f425409e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:51.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f4245ffb640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4254124000 0x7f425409e9f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4250009870 tx=0x7f4250031000 comp rx=0 tx=0).stop 2024-06-11T19:17:51.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f4245ffb640 1 -- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4254124000 msgr2=0x7f425409e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:51.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f4245ffb640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4254124000 0x7f425409e9f0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4250009870 tx=0x7f4250031000 comp rx=0 tx=0).stop 2024-06-11T19:17:51.991 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:51.991 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:51.991 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:51.991 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:51.991 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:51.991 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 17523, 2024-06-11T19:17:51.991 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:51.991 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:51.992 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:51.993 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.994 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:51.995 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:51.997 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:51.998 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:51.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:52.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:52.002 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.003 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:52.005 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:52.007 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.009 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.010 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.011 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.012 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.013 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.013 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:52.013 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.013 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.013 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:52.013 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 10 librados: watch_flush enter 2024-06-11T19:17:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 10 librados: watch_flush exit 2024-06-11T19:17:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 20 client.55194.objecter shutdown clearing up homeless session... 2024-06-11T19:17:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 10 client.55194.objecter successfully canceled tick 2024-06-11T19:17:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 1 -- 172.21.15.98:0/390722289 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f423803fde0 msgr2=0x7f42380422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f423803fde0 0x7f42380422a0 secure :-1 s=READY pgs=4177 cs=0 l=1 rev1=1 crypto rx=0x7f42540a5620 tx=0x7f4248008040 comp rx=0 tx=0).stop 2024-06-11T19:17:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 10 monclient: shutdown 2024-06-11T19:17:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 1 -- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f42540a5240 msgr2=0x7f42541231a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f4259dd2640 1 -- 172.21.15.98:0/390722289 reap_dead start 2024-06-11T19:17:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f42540a5240 0x7f42541231a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f424c00cd70 tx=0x7f424c00a660 comp rx=0 tx=0).stop 2024-06-11T19:17:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 1 -- 172.21.15.98:0/390722289 shutdown_connections 2024-06-11T19:17:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 1 --2- 172.21.15.98:0/390722289 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f42540a5240 0x7f42541231a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 1 -- 172.21.15.98:0/390722289 >> 172.21.15.98:0/390722289 conn(0x7f42540ab850 msgr2=0x7f42541bf2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 1 -- 172.21.15.98:0/390722289 shutdown_connections 2024-06-11T19:17:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 1 -- 172.21.15.98:0/390722289 wait complete. 2024-06-11T19:17:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:51.970+0000 7f425b835640 1 librados: shutdown 2024-06-11T19:17:52.016 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:52.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:17:52.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:17:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.116+0000 7fec94ed2640 10 monclient: get_monmap_and_config 2024-06-11T19:17:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.116+0000 7fec94ed2640 10 monclient: build_initial_monmap 2024-06-11T19:17:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.116+0000 7fec94ed2640 10 monclient: monmap: 2024-06-11T19:17:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:52.115 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:52.117370+0000 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:52.117370+0000 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:52.116 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:52.117 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:52.117 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 1 Processor -- start 2024-06-11T19:17:52.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 1 -- start start 2024-06-11T19:17:52.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 10 monclient: init 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 10 monclient: _add_conns ranks=[2,7,9,0,10,6,3,5,1,8,4] 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec900a4920 0x7fec900a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 10 monclient(hunting): picked mon.noname-b con 0x7fec900a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fec900a5240 0x7fec900a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 10 monclient(hunting): picked mon.noname-i con 0x7fec900a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fec900a2530 0x7fec900a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 10 monclient(hunting): picked mon.noname-a con 0x7fec900a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fec900a7140 con 0x7fec900a2530 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fec900a2b50 con 0x7fec900a4920 2024-06-11T19:17:52.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fec900a2e30 con 0x7fec900a5240 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec94ed2640 10 monclient(hunting): authenticate will time out at 1281.924582s 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec900a4920 0x7fec900a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8ed9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fec900a2530 0x7fec900a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec900a4920 0x7fec900a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45264/0 (socket says 172.21.15.98:45264) 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 1 -- 172.21.15.98:0/3935955573 learned_addr learned my addr 172.21.15.98:0/3935955573 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8ed9d640 1 --2- 172.21.15.98:0/3935955573 >> v2:172.21.15.98:3300/0 conn(0x7fec900a2530 0x7fec900a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56724/0 (socket says 172.21.15.98:56724) 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 10 monclient(hunting): get_auth_request con 0x7fec900a4920 auth_method 0 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8ed9d640 10 monclient(hunting): get_auth_request con 0x7fec900a2530 auth_method 0 2024-06-11T19:17:52.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8ed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec87fff640 1 --2- 172.21.15.98:0/3935955573 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fec900a5240 0x7fec900a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec87fff640 10 monclient(hunting): get_auth_request con 0x7fec900a5240 auth_method 0 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec87fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec87fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 10 monclient(hunting): handle_auth_done global_id 84458 payload 274 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 1 -- 172.21.15.98:0/3935955573 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fec900a5240 msgr2=0x7fec900a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 1 --2- 172.21.15.98:0/3935955573 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fec900a5240 0x7fec900a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 1 -- 172.21.15.98:0/3935955573 >> v2:172.21.15.98:3300/0 conn(0x7fec900a2530 msgr2=0x7fec900a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.117+0000 7fec8e59c640 1 --2- 172.21.15.98:0/3935955573 >> v2:172.21.15.98:3300/0 conn(0x7fec900a2530 0x7fec900a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8e59c640 10 monclient: _finish_hunting 0 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8e59c640 1 monclient: found mon.noname-b 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8e59c640 1 -- 172.21.15.98:0/3935955573 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec900a3110 con 0x7fec900a4920 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8ed9d640 1 --2- 172.21.15.98:0/3935955573 >> v2:172.21.15.98:3300/0 conn(0x7fec900a2530 0x7fec900a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:52.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8e59c640 1 --2- 172.21.15.98:0/3935955573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec900a4920 0x7fec900a4d00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fec80009870 tx=0x7fec8002b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae467906f05050fa server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 1 -- 172.21.15.98:0/3935955573 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec80038070 con 0x7fec900a4920 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 10 monclient: dump: 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.131 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 10 monclient: _finish_auth 0 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 10 monclient: _check_auth_tickets 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 1 -- 172.21.15.98:0/3935955573 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec80031990 con 0x7fec900a4920 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec94ed2640 5 monclient: authenticate success, global_id 84458 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec94ed2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:52.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec94ed2640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 1 -- 172.21.15.98:0/3935955573 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec800323b0 con 0x7fec900a4920 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec94ed2640 4 set_mon_vals no callback set 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.118+0000 7fec8dd9b640 10 monclient: dump: 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.133 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.119+0000 7fec94ed2640 10 monclient: shutdown 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.119+0000 7fec94ed2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.119+0000 7fec94ed2640 1 -- 172.21.15.98:0/3935955573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec900a4920 msgr2=0x7fec900a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.119+0000 7fec94ed2640 1 --2- 172.21.15.98:0/3935955573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec900a4920 0x7fec900a4d00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fec80009870 tx=0x7fec8002b7c0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.119+0000 7fec94ed2640 1 -- 172.21.15.98:0/3935955573 shutdown_connections 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.119+0000 7fec94ed2640 1 --2- 172.21.15.98:0/3935955573 >> v2:172.21.15.98:3300/0 conn(0x7fec900a2530 0x7fec900a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.119+0000 7fec94ed2640 1 --2- 172.21.15.98:0/3935955573 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fec900a5240 0x7fec900a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.119+0000 7fec94ed2640 1 --2- 172.21.15.98:0/3935955573 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec900a4920 0x7fec900a4d00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.119+0000 7fec94ed2640 1 -- 172.21.15.98:0/3935955573 >> 172.21.15.98:0/3935955573 conn(0x7fec900ab850 msgr2=0x7fec900a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.119+0000 7fec94ed2640 1 -- 172.21.15.98:0/3935955573 shutdown_connections 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.119+0000 7fec94ed2640 1 -- 172.21.15.98:0/3935955573 wait complete. 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.120+0000 7fec94ed2640 10 monclient: build_initial_monmap 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.120+0000 7fec94ed2640 10 monclient: monmap: 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:52.135 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.120+0000 7fec94ed2640 1 librados: starting msgr at 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.120+0000 7fec94ed2640 1 librados: starting objecter 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.120+0000 7fec94ed2640 1 Processor -- start 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.120+0000 7fec94ed2640 1 -- start start 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.120+0000 7fec94ed2640 1 librados: setting wanted keys 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.120+0000 7fec94ed2640 1 librados: calling monclient init 2024-06-11T19:17:52.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.120+0000 7fec94ed2640 10 monclient: init 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.120+0000 7fec94ed2640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.120+0000 7fec94ed2640 10 monclient: _add_conns ranks=[3,0,5,1,6,4,7,8,2] 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec900a2530 0x7fec901cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 monclient(hunting): picked mon.noname-g con 0x7fec900a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec900a4920 0x7fec901cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 monclient(hunting): picked mon.noname-a con 0x7fec900a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fec900a5240 0x7fec90125110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 monclient(hunting): picked mon.noname-h con 0x7fec900a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec900a6da0 con 0x7fec900a4920 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fec90138e50 con 0x7fec900a2530 2024-06-11T19:17:52.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fec900a6420 con 0x7fec900a5240 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 monclient(hunting): authenticate will time out at 1281.928066s 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec900a4920 0x7fec901cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec900a4920 0x7fec901cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56738/0 (socket says 172.21.15.98:56738) 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 1 -- 172.21.15.98:0/2163886381 learned_addr learned my addr 172.21.15.98:0/2163886381 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 10 monclient(hunting): get_auth_request con 0x7fec900a4920 auth_method 0 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec8e59c640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec900a2530 0x7fec901cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec8ed9d640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fec900a5240 0x7fec90125110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec8ed9d640 10 monclient(hunting): get_auth_request con 0x7fec900a5240 auth_method 0 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec8ed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec8e59c640 10 monclient(hunting): get_auth_request con 0x7fec900a2530 auth_method 0 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec8e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 10 monclient(hunting): handle_auth_done global_id 65097 payload 824 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 1 -- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fec900a5240 msgr2=0x7fec90125110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fec900a5240 0x7fec90125110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 1 -- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec900a2530 msgr2=0x7fec901cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec900a2530 0x7fec901cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 1 monclient: found mon.noname-a 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 1 -- 172.21.15.98:0/2163886381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec90128b10 con 0x7fec900a4920 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec87fff640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec900a4920 0x7fec901cdfe0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fec7800e870 tx=0x7fec7800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 10 client.?.objecter ms_handle_connect 0x7fec900a4920 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec78017070 con 0x7fec900a4920 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 10 monclient: dump: 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.140 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 10 monclient: _finish_auth 0 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec7800ee60 con 0x7fec900a4920 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 5 monclient: authenticate success, global_id 65097 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 monclient: _renew_subs 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 1 -- 172.21.15.98:0/2163886381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec90127940 con 0x7fec900a4920 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec7800cdb0 con 0x7fec900a4920 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 client.65097.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec8d59a640 4 set_mon_vals no callback set 2024-06-11T19:17:52.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec86ffd640 10 monclient: dump: 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.143 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 monclient: _renew_subs 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 1 -- 172.21.15.98:0/2163886381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec90138570 con 0x7fec900a4920 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.121+0000 7fec94ed2640 1 librados: init done 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec94ed2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec94ed2640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec90125e10 0x7fec9002cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec94ed2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec94ed2640 1 -- 172.21.15.98:0/2163886381 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fec90050830 con 0x7fec90125e10 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fec7801a070 con 0x7fec900a4920 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec8e59c640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec90125e10 0x7fec9002cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec8e59c640 10 monclient: get_auth_request con 0x7fec90125e10 auth_method 0 2024-06-11T19:17:52.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec8e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec86ffd640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fec7003fdb0 0x7fec70042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(347..347 src has 1..347) ==== 6230+0+0 (secure 0 0 0) 0x7fec7800f040 con 0x7fec900a4920 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec86ffd640 10 client.65097.objecter ms_dispatch 0x7fec90000b90 osd_map(347..347 src has 1..347) 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec86ffd640 3 client.65097.objecter handle_osd_map got epochs [347,347] > 0 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec86ffd640 3 client.65097.objecter handle_osd_map decoding full epoch 347 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.122+0000 7fec8e59c640 10 monclient(hunting): handle_auth_done global_id 32203 payload 824 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec8e59c640 1 -- 172.21.15.98:0/2163886381 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec90138830 con 0x7fec90125e10 2024-06-11T19:17:52.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec8ed9d640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fec7003fdb0 0x7fec70042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 20 client.65097.objecter dump_active .. 0 homeless 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec8ed9d640 10 monclient: get_auth_request con 0x7fec7003fdb0 auth_method 0 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec8e59c640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec90125e10 0x7fec9002cb40 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7fec900abf80 tx=0x7fec800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 10 client.65097.objecter ms_handle_connect 0x7fec90125e10 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 10 client.65097.objecter resend_mon_ops 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec80043070 con 0x7fec90125e10 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 10 monclient: dump: 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.146 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 10 client.65097.objecter ms_handle_connect 0x7fec7003fdb0 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec8ed9d640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fec7003fdb0 0x7fec70042270 secure :-1 s=READY pgs=4178 cs=0 l=1 rev1=1 crypto rx=0x7fec90126870 tx=0x7fec8800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fec80003ba0 con 0x7fec90125e10 2024-06-11T19:17:52.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:52.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:52.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec90125e10 msgr2=0x7fec9002cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec90125e10 0x7fec9002cb40 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7fec900abf80 tx=0x7fec800046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec90125e10 msgr2=0x7fec9002cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:52.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.123+0000 7fec86ffd640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec90125e10 0x7fec9002cb40 secure :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7fec900abf80 tx=0x7fec800046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.129+0000 7fec94ed2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:52.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.129+0000 7fec94ed2640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec901261f0 0x7fec9002d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.129+0000 7fec94ed2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.129+0000 7fec94ed2640 1 -- 172.21.15.98:0/2163886381 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fec900a6750 con 0x7fec901261f0 2024-06-11T19:17:52.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec8e59c640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec901261f0 0x7fec9002d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec8e59c640 10 monclient: get_auth_request con 0x7fec901261f0 auth_method 0 2024-06-11T19:17:52.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec8e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec8e59c640 10 monclient(hunting): handle_auth_done global_id 32212 payload 824 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec8e59c640 1 -- 172.21.15.98:0/2163886381 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fec900a0c00 con 0x7fec901261f0 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec8e59c640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec901261f0 0x7fec9002d080 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fec80000c00 tx=0x7fec80008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 10 client.65097.objecter ms_handle_connect 0x7fec901261f0 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 10 client.65097.objecter resend_mon_ops 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec80038040 con 0x7fec901261f0 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 10 monclient: dump: 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.151 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.152 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7fec800093d0 con 0x7fec901261f0 2024-06-11T19:17:52.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:52.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:52.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec901261f0 msgr2=0x7fec9002d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec901261f0 0x7fec9002d080 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fec80000c00 tx=0x7fec80008030 comp rx=0 tx=0).stop 2024-06-11T19:17:52.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 1 -- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec901261f0 msgr2=0x7fec9002d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:52.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.130+0000 7fec86ffd640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec901261f0 0x7fec9002d080 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.153 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:52.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:52.153 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 522348, 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:52.155 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:52.156 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:52.157 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:52.158 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.160 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:52.162 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:52.164 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:52.166 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:52.168 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.173 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.174 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:52.175 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 10 librados: watch_flush enter 2024-06-11T19:17:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 10 librados: watch_flush exit 2024-06-11T19:17:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 20 client.65097.objecter shutdown clearing up homeless session... 2024-06-11T19:17:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 10 client.65097.objecter successfully canceled tick 2024-06-11T19:17:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 1 -- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fec7003fdb0 msgr2=0x7fec70042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fec7003fdb0 0x7fec70042270 secure :-1 s=READY pgs=4178 cs=0 l=1 rev1=1 crypto rx=0x7fec90126870 tx=0x7fec8800b040 comp rx=0 tx=0).stop 2024-06-11T19:17:52.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 10 monclient: shutdown 2024-06-11T19:17:52.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:52.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 1 -- 172.21.15.98:0/2163886381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec900a4920 msgr2=0x7fec901cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec900a4920 0x7fec901cdfe0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fec7800e870 tx=0x7fec7800ed30 comp rx=0 tx=0).stop 2024-06-11T19:17:52.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec8ed9d640 1 -- 172.21.15.98:0/2163886381 reap_dead start 2024-06-11T19:17:52.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 1 -- 172.21.15.98:0/2163886381 shutdown_connections 2024-06-11T19:17:52.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 1 --2- 172.21.15.98:0/2163886381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec900a4920 0x7fec901cdfe0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 1 -- 172.21.15.98:0/2163886381 >> 172.21.15.98:0/2163886381 conn(0x7fec900ab850 msgr2=0x7fec9009c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:52.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 1 -- 172.21.15.98:0/2163886381 shutdown_connections 2024-06-11T19:17:52.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 1 -- 172.21.15.98:0/2163886381 wait complete. 2024-06-11T19:17:52.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.131+0000 7fec94ed2640 1 librados: shutdown 2024-06-11T19:17:52.179 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:52.179 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:17:52.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:17:52.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.278+0000 7fbebcc63640 10 monclient: get_monmap_and_config 2024-06-11T19:17:52.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.278+0000 7fbebcc63640 10 monclient: build_initial_monmap 2024-06-11T19:17:52.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.278+0000 7fbebcc63640 10 monclient: monmap: 2024-06-11T19:17:52.278 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:52.278 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.278 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:52.279812+0000 2024-06-11T19:17:52.278 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:52.279812+0000 2024-06-11T19:17:52.278 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:52.278 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.278 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:52.279 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:52.279 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:52.279 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:52.279 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:52.279 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:52.279 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:52.279 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:52.279 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:52.279 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:52.279 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:52.279 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.279+0000 7fbebcc63640 1 Processor -- start 2024-06-11T19:17:52.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.279+0000 7fbebcc63640 1 -- start start 2024-06-11T19:17:52.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.279+0000 7fbebcc63640 10 monclient: init 2024-06-11T19:17:52.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.279+0000 7fbebcc63640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:52.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.279+0000 7fbebcc63640 10 monclient: _add_conns ranks=[0,2,8,4,5,7,9,1,6,3,10] 2024-06-11T19:17:52.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.279+0000 7fbebcc63640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbeb80a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbebcc63640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbeb80a2530 0x7fbeb813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbebcc63640 10 monclient(hunting): picked mon.noname-b con 0x7fbeb80a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbebcc63640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb813e1a0 0x7fbeb8140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbebcc63640 10 monclient(hunting): picked mon.noname-e con 0x7fbeb813e1a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbebcc63640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbeb80a7140 con 0x7fbeb80a4920 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbebcc63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbebcc63640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb80a2b50 con 0x7fbeb80a2530 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbebcc63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbebcc63640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbeb80a2e30 con 0x7fbeb813e1a0 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbebcc63640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbebcc63640 10 monclient(hunting): authenticate will time out at 1282.087058s 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbeb80a2530 0x7fbeb813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb6d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb813e1a0 0x7fbeb8140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb659c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbeb80a4920 0x7fbeb80a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35772/0 (socket says 172.21.15.98:35772) 2024-06-11T19:17:52.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb659c640 1 -- 172.21.15.98:0/229496538 learned_addr learned my addr 172.21.15.98:0/229496538 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:17:52.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 10 monclient(hunting): get_auth_request con 0x7fbeb80a2530 auth_method 0 2024-06-11T19:17:52.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb6d9d640 10 monclient(hunting): get_auth_request con 0x7fbeb813e1a0 auth_method 0 2024-06-11T19:17:52.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb6d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_done global_id 84467 payload 274 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 1 -- 172.21.15.98:0/229496538 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb813e1a0 msgr2=0x7fbeb8140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb559a640 1 -- 172.21.15.98:0/229496538 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3635039879 0 0) 0x7fbeb80a7140 con 0x7fbeb80a4920 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 1 --2- 172.21.15.98:0/229496538 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb813e1a0 0x7fbeb8140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 1 -- 172.21.15.98:0/229496538 >> v1:172.21.15.98:6789/0 conn(0x7fbeb80a4920 legacy=0x7fbeb80a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 1 monclient: found mon.noname-b 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:52.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb5d9b640 1 -- 172.21.15.98:0/229496538 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb80a3110 con 0x7fbeb80a2530 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb6d9d640 1 --2- 172.21.15.98:0/229496538 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb813e1a0 0x7fbeb8140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.280+0000 7fbeb559a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb5d9b640 1 --2- 172.21.15.98:0/229496538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbeb80a2530 0x7fbeb813dc60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbea80097d0 tx=0x7fbea802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91a2824cc1a97230 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 1 -- 172.21.15.98:0/229496538 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbea8039070 con 0x7fbeb80a2530 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 10 monclient: dump: 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.298 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 10 monclient: _finish_auth 0 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 10 monclient: _check_auth_tickets 2024-06-11T19:17:52.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbebcc63640 5 monclient: authenticate success, global_id 84467 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 1 -- 172.21.15.98:0/229496538 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea8031960 con 0x7fbeb80a2530 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbebcc63640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 1 -- 172.21.15.98:0/229496538 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbea8032380 con 0x7fbeb80a2530 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbebcc63640 4 set_mon_vals no callback set 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbeb559a640 10 monclient: dump: 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.300 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbebcc63640 10 monclient: shutdown 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbebcc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:52.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbebcc63640 1 -- 172.21.15.98:0/229496538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbeb80a2530 msgr2=0x7fbeb813dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.281+0000 7fbebcc63640 1 --2- 172.21.15.98:0/229496538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbeb80a2530 0x7fbeb813dc60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbea80097d0 tx=0x7fbea802b790 comp rx=0 tx=0).stop 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.282+0000 7fbebcc63640 1 -- 172.21.15.98:0/229496538 shutdown_connections 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.282+0000 7fbebcc63640 1 --2- 172.21.15.98:0/229496538 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbeb80a2530 0x7fbeb813dc60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.282+0000 7fbebcc63640 1 --2- 172.21.15.98:0/229496538 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb813e1a0 0x7fbeb8140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.282+0000 7fbebcc63640 1 -- 172.21.15.98:0/229496538 >> 172.21.15.98:0/229496538 conn(0x7fbeb80ab850 msgr2=0x7fbeb80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.282+0000 7fbebcc63640 1 -- 172.21.15.98:0/229496538 shutdown_connections 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.282+0000 7fbebcc63640 1 -- 172.21.15.98:0/229496538 wait complete. 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.282+0000 7fbebcc63640 10 monclient: build_initial_monmap 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.282+0000 7fbebcc63640 10 monclient: monmap: 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.282+0000 7fbebcc63640 1 librados: starting msgr at 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.282+0000 7fbebcc63640 1 librados: starting objecter 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 1 Processor -- start 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 1 -- start start 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 1 librados: setting wanted keys 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 1 librados: calling monclient init 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 10 monclient: init 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 10 monclient: _add_conns ranks=[3,7,1,0,8,2,5,4,6] 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a2530 0x7fbeb8136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 10 monclient(hunting): picked mon.noname-g con 0x7fbeb80a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbeb80a4920 0x7fbeb8137200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 10 monclient(hunting): picked mon.noname-i con 0x7fbeb80a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb813e1a0 0x7fbeb809fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 10 monclient(hunting): picked mon.noname-f con 0x7fbeb813e1a0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:17:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb8138e50 con 0x7fbeb813e1a0 2024-06-11T19:17:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb80a84b0 con 0x7fbeb80a2530 2024-06-11T19:17:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fbeb80a6da0 con 0x7fbeb80a4920 2024-06-11T19:17:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbebcc63640 10 monclient(hunting): authenticate will time out at 1282.090593s 2024-06-11T19:17:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb659c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a2530 0x7fbeb8136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb6d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb813e1a0 0x7fbeb809fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb5d9b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbeb80a4920 0x7fbeb8137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb659c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a2530 0x7fbeb8136cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:35816/0 (socket says 172.21.15.98:35816) 2024-06-11T19:17:52.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb6d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb813e1a0 0x7fbeb809fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:41702/0 (socket says 172.21.15.98:41702) 2024-06-11T19:17:52.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb659c640 1 -- 172.21.15.98:0/3827952675 learned_addr learned my addr 172.21.15.98:0/3827952675 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:52.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb659c640 10 monclient(hunting): get_auth_request con 0x7fbeb80a2530 auth_method 0 2024-06-11T19:17:52.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb6d9d640 10 monclient(hunting): get_auth_request con 0x7fbeb813e1a0 auth_method 0 2024-06-11T19:17:52.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb6d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb5d9b640 10 monclient(hunting): get_auth_request con 0x7fbeb80a4920 auth_method 0 2024-06-11T19:17:52.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb5d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.283+0000 7fbeb6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb659c640 10 monclient(hunting): handle_auth_done global_id 94206 payload 824 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb659c640 1 -- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbeb80a4920 msgr2=0x7fbeb8137200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb659c640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbeb80a4920 0x7fbeb8137200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb659c640 1 -- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb813e1a0 msgr2=0x7fbeb809fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb659c640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb813e1a0 0x7fbeb809fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb659c640 10 monclient: _finish_hunting 0 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb659c640 1 monclient: found mon.noname-g 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb659c640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb659c640 1 -- 172.21.15.98:0/3827952675 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb8123670 con 0x7fbeb80a2530 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb5d9b640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fbeb80a4920 0x7fbeb8137200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:52.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb659c640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a2530 0x7fbeb8136cc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000c7c0 tx=0x7fbeb000cbc0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 10 client.?.objecter ms_handle_connect 0x7fbeb80a2530 2024-06-11T19:17:52.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:52.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbeb0018070 con 0x7fbeb80a2530 2024-06-11T19:17:52.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:17:52.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 10 monclient: dump: 2024-06-11T19:17:52.309 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.309 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.309 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.309 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 10 monclient: _finish_auth 0 2024-06-11T19:17:52.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 10 monclient: _check_auth_tickets 2024-06-11T19:17:52.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:52.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeb000ce10 con 0x7fbeb80a2530 2024-06-11T19:17:52.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbebcc63640 5 monclient: authenticate success, global_id 94206 2024-06-11T19:17:52.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbebcc63640 10 monclient: _renew_subs 2024-06-11T19:17:52.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbebcc63640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:17:52.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbebcc63640 1 -- 172.21.15.98:0/3827952675 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb80aa6d0 con 0x7fbeb80a2530 2024-06-11T19:17:52.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:52.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbeb0004d70 con 0x7fbeb80a2530 2024-06-11T19:17:52.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:17:52.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeb4d99640 4 set_mon_vals no callback set 2024-06-11T19:17:52.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbebcc63640 10 client.94206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:52.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbeaeffd640 10 monclient: dump: 2024-06-11T19:17:52.313 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.313 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.313 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.313 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.313 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.313 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.313 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.313 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.313 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.313 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbebcc63640 10 monclient: _renew_subs 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbebcc63640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbebcc63640 1 -- 172.21.15.98:0/3827952675 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb81230b0 con 0x7fbeb80a2530 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.284+0000 7fbebcc63640 1 librados: init done 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbebcc63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 <== mon.3 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fbeb000a640 con 0x7fbeb80a2530 2024-06-11T19:17:52.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbebcc63640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb81cd920 0x7fbeb80a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbebcc63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbebcc63640 1 -- 172.21.15.98:0/3827952675 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb813df30 con 0x7fbeb81cd920 2024-06-11T19:17:52.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeb5d9b640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb81cd920 0x7fbeb80a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeaeffd640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbe9803fde0 0x7fbe980422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeb5d9b640 10 monclient: get_auth_request con 0x7fbeb81cd920 auth_method 0 2024-06-11T19:17:52.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeb5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeb5d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeb5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 <== mon.3 v2:172.21.15.117:3301/0 5 ==== osd_map(347..347 src has 1..347) ==== 6230+0+0 (secure 0 0 0) 0x7fbeb0050f90 con 0x7fbeb80a2530 2024-06-11T19:17:52.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeaeffd640 10 client.94206.objecter ms_dispatch 0x7fbeb8000b90 osd_map(347..347 src has 1..347) 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeaeffd640 3 client.94206.objecter handle_osd_map got epochs [347,347] > 0 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeaeffd640 3 client.94206.objecter handle_osd_map decoding full epoch 347 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeb6d9d640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbe9803fde0 0x7fbe980422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeaeffd640 20 client.94206.objecter dump_active .. 0 homeless 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeb6d9d640 10 monclient: get_auth_request con 0x7fbe9803fde0 auth_method 0 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_done global_id 94215 payload 824 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.285+0000 7fbeb5d9b640 1 -- 172.21.15.98:0/3827952675 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbeb8069520 con 0x7fbeb81cd920 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 10 client.94206.objecter ms_handle_connect 0x7fbeb81cd920 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 10 client.94206.objecter resend_mon_ops 2024-06-11T19:17:52.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeb5d9b640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb81cd920 0x7fbeb80a02e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8073de0 tx=0x7fbea80046c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 10 client.94206.objecter ms_handle_connect 0x7fbe9803fde0 2024-06-11T19:17:52.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeb6d9d640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbe9803fde0 0x7fbe980422a0 secure :-1 s=READY pgs=4179 cs=0 l=1 rev1=1 crypto rx=0x7fbea00095a0 tx=0x7fbea0009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbea8045070 con 0x7fbeb81cd920 2024-06-11T19:17:52.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:17:52.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 10 monclient: dump: 2024-06-11T19:17:52.317 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.317 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.317 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.317 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.317 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.317 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbea8003b50 con 0x7fbeb81cd920 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:52.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb81cd920 msgr2=0x7fbeb80a02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb81cd920 0x7fbeb80a02e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8073de0 tx=0x7fbea80046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb81cd920 msgr2=0x7fbeb80a02e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.286+0000 7fbeaeffd640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb81cd920 0x7fbeb80a02e0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8073de0 tx=0x7fbea80046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.292+0000 7fbebcc63640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.292+0000 7fbebcc63640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a0820 0x7fbeb813bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.292+0000 7fbebcc63640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.292+0000 7fbebcc63640 1 -- 172.21.15.98:0/3827952675 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb8050830 con 0x7fbeb80a0820 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeb5d9b640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a0820 0x7fbeb813bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeb5d9b640 10 monclient: get_auth_request con 0x7fbeb80a0820 auth_method 0 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeb5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeb5d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeb5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeb5d9b640 10 monclient(hunting): handle_auth_done global_id 94224 payload 824 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeb5d9b640 1 -- 172.21.15.98:0/3827952675 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbeb80a0c30 con 0x7fbeb80a0820 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeb5d9b640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a0820 0x7fbeb813bd30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbea802b7d0 tx=0x7fbea8031000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 10 client.94206.objecter ms_handle_connect 0x7fbeb80a0820 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 10 client.94206.objecter resend_mon_ops 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbea8039040 con 0x7fbeb80a0820 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 10 monclient: dump: 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.320 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.322 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fbea8003dc0 con 0x7fbeb80a0820 2024-06-11T19:17:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a0820 msgr2=0x7fbeb813bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a0820 0x7fbeb813bd30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbea802b7d0 tx=0x7fbea8031000 comp rx=0 tx=0).stop 2024-06-11T19:17:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 1 -- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a0820 msgr2=0x7fbeb813bd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.293+0000 7fbeaeffd640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a0820 0x7fbeb813bd30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.323 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:52.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:52.323 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:52.323 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:52.324 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:52.324 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 17841, 2024-06-11T19:17:52.324 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:52.324 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:52.324 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:52.324 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:52.325 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:52.325 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:52.325 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:52.325 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:52.325 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:52.325 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:52.325 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.326 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-06-11T19:17:52.326 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:52.326 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:52.326 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:52.326 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:52.326 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:52.327 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:52.327 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:52.327 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:52.327 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:52.327 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:52.328 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:52.328 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:52.328 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:52.328 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:52.328 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.328 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:52.329 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:52.329 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:52.329 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:52.329 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:52.329 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:52.329 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:52.330 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:52.330 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:52.330 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:52.330 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:52.330 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:52.330 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:52.331 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.331 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:52.331 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:52.331 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:52.331 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:52.332 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:52.332 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:52.332 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:52.332 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:52.332 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:52.332 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:52.333 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:52.333 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:52.333 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:52.333 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:52.333 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:52.333 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:52.334 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:52.334 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:52.334 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:52.334 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:52.334 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:52.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:52.335 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:52.335 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:52.335 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:52.335 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:52.335 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:52.336 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:52.336 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.336 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:52.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.336 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:52.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.337 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:52.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:52.337 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:52.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:52.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.339 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:52.339 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:52.339 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.340 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.340 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.340 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:52.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:52.341 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:52.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:52.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.342 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:52.342 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:52.342 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.342 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.342 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.342 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:52.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:52.342 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:52.344 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:52.345 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.345 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.345 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:52.345 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:52.345 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.345 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:52.347 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.347 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:52.347 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:52.348 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.355 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:52.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.356 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.356 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:52.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.356 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.356 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.356 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:17:52.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.356 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.356 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:52.356 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:52.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 10 librados: watch_flush enter 2024-06-11T19:17:52.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 10 librados: watch_flush exit 2024-06-11T19:17:52.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 20 client.94206.objecter shutdown clearing up homeless session... 2024-06-11T19:17:52.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 10 client.94206.objecter successfully canceled tick 2024-06-11T19:17:52.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 1 -- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbe9803fde0 msgr2=0x7fbe980422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbe9803fde0 0x7fbe980422a0 secure :-1 s=READY pgs=4179 cs=0 l=1 rev1=1 crypto rx=0x7fbea00095a0 tx=0x7fbea0009190 comp rx=0 tx=0).stop 2024-06-11T19:17:52.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 10 monclient: shutdown 2024-06-11T19:17:52.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:52.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 1 -- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a2530 msgr2=0x7fbeb8136cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 1 --2- 172.21.15.98:0/3827952675 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80a2530 0x7fbeb8136cc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000c7c0 tx=0x7fbeb000cbc0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbeb6d9d640 1 -- 172.21.15.98:0/3827952675 reap_dead start 2024-06-11T19:17:52.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbeb6d9d640 1 -- 172.21.15.98:0/3827952675 reap_dead start 2024-06-11T19:17:52.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 1 -- 172.21.15.98:0/3827952675 shutdown_connections 2024-06-11T19:17:52.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 1 -- 172.21.15.98:0/3827952675 >> 172.21.15.98:0/3827952675 conn(0x7fbeb80ab850 msgr2=0x7fbeb80a8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:52.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 1 -- 172.21.15.98:0/3827952675 shutdown_connections 2024-06-11T19:17:52.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 1 -- 172.21.15.98:0/3827952675 wait complete. 2024-06-11T19:17:52.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.294+0000 7fbebcc63640 1 librados: shutdown 2024-06-11T19:17:52.362 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:52.363 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:17:52.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:17:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.463+0000 7f39ae035640 10 monclient: get_monmap_and_config 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.463+0000 7f39ae035640 10 monclient: build_initial_monmap 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.463+0000 7f39ae035640 10 monclient: monmap: 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:52.464665+0000 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:52.464665+0000 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:52.463 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:52.464 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:52.464 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.464+0000 7f39ae035640 1 Processor -- start 2024-06-11T19:17:52.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.464+0000 7f39ae035640 1 -- start start 2024-06-11T19:17:52.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.464+0000 7f39ae035640 10 monclient: init 2024-06-11T19:17:52.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39ae035640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:52.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39ae035640 10 monclient: _add_conns ranks=[0,9,1,8,7,2,3,6,5,10,4] 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39ae035640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39a80a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39ae035640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f39a80a2530 0x7f39a813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39ae035640 10 monclient(hunting): picked mon.noname-a con 0x7f39a80a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39ae035640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f39a813e1a0 addr v1:172.21.15.117:6789/0 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39ae035640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39a80a7140 con 0x7f39a80a4920 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39ae035640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39a80a2b50 con 0x7f39a813e1a0 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39ae035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39ae035640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f39a80a2e30 con 0x7f39a80a2530 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39ae035640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39ae035640 10 monclient(hunting): authenticate will time out at 1282.272166s 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f39a80a2530 0x7f39a813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f39a80a2530 0x7f39a813dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56746/0 (socket says 172.21.15.98:56746) 2024-06-11T19:17:52.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a77fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f39a80a4920 0x7f39a80a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35778/0 (socket says 172.21.15.98:35778) 2024-06-11T19:17:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 1 -- 172.21.15.98:0/1159800128 learned_addr learned my addr 172.21.15.98:0/1159800128 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 10 monclient(hunting): get_auth_request con 0x7f39a80a2530 auth_method 0 2024-06-11T19:17:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 10 monclient(hunting): handle_auth_done global_id 65115 payload 274 2024-06-11T19:17:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 1 -- 172.21.15.98:0/1159800128 >> v1:172.21.15.117:6789/0 conn(0x7f39a813e1a0 legacy=0x7f39a8140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 1 -- 172.21.15.98:0/1159800128 >> v1:172.21.15.98:6789/0 conn(0x7f39a80a4920 legacy=0x7f39a80a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 1 monclient: found mon.noname-a 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.465+0000 7f39a6ffd640 1 -- 172.21.15.98:0/1159800128 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a80a3110 con 0x7f39a80a2530 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a6ffd640 1 --2- 172.21.15.98:0/1159800128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39a80a2530 0x7f39a813dc60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f399c009870 tx=0x7f399c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d26e7c516175d4 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 1 -- 172.21.15.98:0/1159800128 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f399c035020 con 0x7f39a80a2530 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 10 monclient: dump: 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 10 monclient: _finish_auth 0 2024-06-11T19:17:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 10 monclient: _check_auth_tickets 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 1 -- 172.21.15.98:0/1159800128 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f399c031ae0 con 0x7f39a80a2530 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39ae035640 5 monclient: authenticate success, global_id 65115 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39ae035640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39ae035640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 1 -- 172.21.15.98:0/1159800128 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f399c031d10 con 0x7f39a80a2530 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39ae035640 4 set_mon_vals no callback set 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39a67fc640 10 monclient: dump: 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.466+0000 7f39ae035640 10 monclient: shutdown 2024-06-11T19:17:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.467+0000 7f39ae035640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.467+0000 7f39ae035640 1 -- 172.21.15.98:0/1159800128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39a80a2530 msgr2=0x7f39a813dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.467+0000 7f39ae035640 1 --2- 172.21.15.98:0/1159800128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39a80a2530 0x7f39a813dc60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f399c009870 tx=0x7f399c02b910 comp rx=0 tx=0).stop 2024-06-11T19:17:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.467+0000 7f39ae035640 1 -- 172.21.15.98:0/1159800128 shutdown_connections 2024-06-11T19:17:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.467+0000 7f39ae035640 1 --2- 172.21.15.98:0/1159800128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39a80a2530 0x7f39a813dc60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.467+0000 7f39ae035640 1 -- 172.21.15.98:0/1159800128 >> 172.21.15.98:0/1159800128 conn(0x7f39a80ab850 msgr2=0x7f39a80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.467+0000 7f39ae035640 1 -- 172.21.15.98:0/1159800128 shutdown_connections 2024-06-11T19:17:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.467+0000 7f39ae035640 1 -- 172.21.15.98:0/1159800128 wait complete. 2024-06-11T19:17:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.467+0000 7f39ae035640 10 monclient: build_initial_monmap 2024-06-11T19:17:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.467+0000 7f39ae035640 10 monclient: monmap: 2024-06-11T19:17:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:52.483 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.467+0000 7f39ae035640 1 librados: starting msgr at 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.467+0000 7f39ae035640 1 librados: starting objecter 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 1 Processor -- start 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 1 -- start start 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 1 librados: setting wanted keys 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 1 librados: calling monclient init 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 10 monclient: init 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 10 monclient: _add_conns ranks=[5,3,1,8,6,2,4,0,7] 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a2530 0x7f39a8133d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 10 monclient(hunting): picked mon.noname-h con 0x7f39a80a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f39a80a4920 0x7f39a8134240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 10 monclient(hunting): picked mon.noname-g con 0x7f39a80a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39a813e1a0 0x7f39a81cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 10 monclient(hunting): picked mon.noname-f con 0x7f39a813e1a0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f39a8138e50 con 0x7f39a813e1a0 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f39a80a84b0 con 0x7f39a80a4920 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f39a80a6da0 con 0x7f39a80a2530 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.468+0000 7f39ae035640 10 monclient(hunting): authenticate will time out at 1282.275726s 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39a813e1a0 0x7f39a81cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a77fe640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a2530 0x7f39a8133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a6ffd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f39a80a4920 0x7f39a8134240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a77fe640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a2530 0x7f39a8133d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:59254/0 (socket says 172.21.15.98:59254) 2024-06-11T19:17:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39a813e1a0 0x7f39a81cd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:41718/0 (socket says 172.21.15.98:41718) 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a77fe640 1 -- 172.21.15.98:0/4054599739 learned_addr learned my addr 172.21.15.98:0/4054599739 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a77fe640 10 monclient(hunting): get_auth_request con 0x7f39a80a2530 auth_method 0 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 10 monclient(hunting): get_auth_request con 0x7f39a813e1a0 auth_method 0 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a6ffd640 10 monclient(hunting): get_auth_request con 0x7f39a80a4920 auth_method 0 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 10 monclient(hunting): handle_auth_done global_id 32239 payload 824 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 1 -- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a2530 msgr2=0x7f39a8133d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a2530 0x7f39a8133d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 1 -- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f39a80a4920 msgr2=0x7f39a8134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f39a80a4920 0x7f39a8134240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 1 monclient: found mon.noname-f 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 1 -- 172.21.15.98:0/4054599739 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a809e600 con 0x7f39a813e1a0 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a6ffd640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f39a80a4920 0x7f39a8134240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a77fe640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a2530 0x7f39a8133d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a7fff640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39a813e1a0 0x7f39a81cd510 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f39a000c700 tx=0x7f39a000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 10 client.?.objecter ms_handle_connect 0x7f39a813e1a0 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f39a0003de0 con 0x7f39a813e1a0 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:17:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 10 monclient: dump: 2024-06-11T19:17:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.490 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 10 monclient: _finish_auth 0 2024-06-11T19:17:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:17:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39a00050d0 con 0x7f39a813e1a0 2024-06-11T19:17:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39ae035640 5 monclient: authenticate success, global_id 32239 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f39a00056f0 con 0x7f39a813e1a0 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f398ffff640 10 monclient: dump: 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39ae035640 10 monclient: _renew_subs 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39a5ffb640 4 set_mon_vals no callback set 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39ae035640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39ae035640 1 -- 172.21.15.98:0/4054599739 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a813df70 con 0x7f39a813e1a0 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39ae035640 10 client.32239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39ae035640 10 monclient: _renew_subs 2024-06-11T19:17:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39ae035640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39ae035640 1 -- 172.21.15.98:0/4054599739 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a80a0680 con 0x7f39a813e1a0 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.469+0000 7f39ae035640 1 librados: init done 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.470+0000 7f39ae035640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.470+0000 7f39ae035640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a0850 0x7f39a81cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.470+0000 7f39ae035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.470+0000 7f39ae035640 1 -- 172.21.15.98:0/4054599739 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f39a8050830 con 0x7f39a80a0850 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.470+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f39a0005280 con 0x7f39a813e1a0 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.470+0000 7f39a77fe640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a0850 0x7f39a81cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.470+0000 7f39a77fe640 10 monclient: get_auth_request con 0x7f39a80a0850 auth_method 0 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.470+0000 7f39a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.470+0000 7f39a77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.470+0000 7f39a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f39a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f39a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f39a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f399403fdb0 0x7f3994042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f39a77fe640 10 monclient(hunting): handle_auth_done global_id 23108 payload 824 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(347..347 src has 1..347) ==== 6230+0+0 (secure 0 0 0) 0x7f39a0051400 con 0x7f39a813e1a0 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 10 client.32239.objecter ms_dispatch 0x7f39a8000b90 osd_map(347..347 src has 1..347) 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 3 client.32239.objecter handle_osd_map got epochs [347,347] > 0 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 3 client.32239.objecter handle_osd_map decoding full epoch 347 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f39a77fe640 1 -- 172.21.15.98:0/4054599739 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39a80a4d00 con 0x7f39a80a0850 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 20 client.32239.objecter dump_active .. 0 homeless 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f39a6ffd640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f399403fdb0 0x7f3994042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f39a6ffd640 10 monclient: get_auth_request con 0x7f399403fdb0 auth_method 0 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 10 client.32239.objecter ms_handle_connect 0x7f39a80a0850 2024-06-11T19:17:52.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 10 client.32239.objecter resend_mon_ops 2024-06-11T19:17:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f39a77fe640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a0850 0x7f39a81cda50 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f3998007fd0 tx=0x7f3998004de0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f39980133f0 con 0x7f39a80a0850 2024-06-11T19:17:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:17:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 10 monclient: dump: 2024-06-11T19:17:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.495 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 10 client.32239.objecter ms_handle_connect 0x7f399403fdb0 2024-06-11T19:17:52.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f39a6ffd640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f399403fdb0 0x7f3994042270 secure :-1 s=READY pgs=4180 cs=0 l=1 rev1=1 crypto rx=0x7f399c036000 tx=0x7f399c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3998018070 con 0x7f39a80a0850 2024-06-11T19:17:52.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:52.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:52.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a0850 msgr2=0x7f39a81cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a0850 0x7f39a81cda50 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f3998007fd0 tx=0x7f3998004de0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a0850 msgr2=0x7f39a81cda50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:52.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.471+0000 7f398ffff640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a80a0850 0x7f39a81cda50 secure :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f3998007fd0 tx=0x7f3998004de0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39ae035640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:52.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39ae035640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a81cdf90 0x7f39a8139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39ae035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39ae035640 1 -- 172.21.15.98:0/4054599739 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f39a81ce370 con 0x7f39a81cdf90 2024-06-11T19:17:52.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39a77fe640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a81cdf90 0x7f39a8139cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39a77fe640 10 monclient: get_auth_request con 0x7f39a81cdf90 auth_method 0 2024-06-11T19:17:52.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39a77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39a77fe640 10 monclient(hunting): handle_auth_done global_id 23117 payload 824 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.478+0000 7f39a77fe640 1 -- 172.21.15.98:0/4054599739 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f39a813d9f0 con 0x7f39a81cdf90 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39a77fe640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a81cdf90 0x7f39a8139cd0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f3998015040 tx=0x7f3998000ed0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 10 client.32239.objecter ms_handle_connect 0x7f39a81cdf90 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 10 client.32239.objecter resend_mon_ops 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3998013a70 con 0x7f39a81cdf90 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 10 monclient: dump: 2024-06-11T19:17:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.502 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7f3998013600 con 0x7f39a81cdf90 2024-06-11T19:17:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a81cdf90 msgr2=0x7f39a8139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a81cdf90 0x7f39a8139cd0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f3998015040 tx=0x7f3998000ed0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 1 -- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a81cdf90 msgr2=0x7f39a8139cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f398ffff640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f39a81cdf90 0x7f39a8139cd0 secure :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f3998015040 tx=0x7f3998000ed0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.502 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 522673, 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:52.503 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:52.504 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:52.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:52.507 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:52.507 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:52.507 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:52.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:52.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:52.507 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:52.507 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:52.507 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:52.507 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:52.507 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:52.507 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:52.507 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.508 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.510 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:52.513 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:52.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.515 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:52.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:52.515 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.515 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.515 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.516 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:52.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:52.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:52.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:52.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.517 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:52.519 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.520 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.520 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.520 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:52.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:17:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:17:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:17:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:52.534 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 10 librados: watch_flush enter 2024-06-11T19:17:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 10 librados: watch_flush exit 2024-06-11T19:17:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 20 client.32239.objecter shutdown clearing up homeless session... 2024-06-11T19:17:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 10 client.32239.objecter successfully canceled tick 2024-06-11T19:17:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 1 -- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f399403fdb0 msgr2=0x7f3994042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f399403fdb0 0x7f3994042270 secure :-1 s=READY pgs=4180 cs=0 l=1 rev1=1 crypto rx=0x7f399c036000 tx=0x7f399c008030 comp rx=0 tx=0).stop 2024-06-11T19:17:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 10 monclient: shutdown 2024-06-11T19:17:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 1 -- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39a813e1a0 msgr2=0x7f39a81cd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39a7fff640 1 -- 172.21.15.98:0/4054599739 reap_dead start 2024-06-11T19:17:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39a813e1a0 0x7f39a81cd510 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f39a000c700 tx=0x7f39a000cbc0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 1 -- 172.21.15.98:0/4054599739 shutdown_connections 2024-06-11T19:17:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 1 --2- 172.21.15.98:0/4054599739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39a813e1a0 0x7f39a81cd510 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 1 -- 172.21.15.98:0/4054599739 >> 172.21.15.98:0/4054599739 conn(0x7f39a80ab850 msgr2=0x7f39a8137c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 1 -- 172.21.15.98:0/4054599739 shutdown_connections 2024-06-11T19:17:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.479+0000 7f39ae035640 1 -- 172.21.15.98:0/4054599739 wait complete. 2024-06-11T19:17:52.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.480+0000 7f39ae035640 1 librados: shutdown 2024-06-11T19:17:52.540 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:52.540 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:17:52.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:17:52.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.640+0000 7f5048669640 10 monclient: get_monmap_and_config 2024-06-11T19:17:52.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.640+0000 7f5048669640 10 monclient: build_initial_monmap 2024-06-11T19:17:52.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.640+0000 7f5048669640 10 monclient: monmap: 2024-06-11T19:17:52.640 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:52.640 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.640 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:52.641942+0000 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:52.641942+0000 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:52.641 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:52.642 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.641+0000 7f5048669640 1 Processor -- start 2024-06-11T19:17:52.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.641+0000 7f5048669640 1 -- start start 2024-06-11T19:17:52.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.641+0000 7f5048669640 10 monclient: init 2024-06-11T19:17:52.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:52.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 10 monclient: _add_conns ranks=[2,5,6,9,10,1,4,8,0,7,3] 2024-06-11T19:17:52.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50400a0830 0x7f50401352e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 10 monclient(hunting): picked mon.noname-b con 0x7f50400a0830 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:52.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f50400a0c10 0x7f50400a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 10 monclient(hunting): picked mon.noname-h con 0x7f50400a0c10 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:17:52.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f50400a21f0 0x7f50400a2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 10 monclient(hunting): picked mon.noname-d con 0x7f50400a21f0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:52.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f50400a35a0 con 0x7f50400a0830 2024-06-11T19:17:52.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f504009ea60 con 0x7f50400a0c10 2024-06-11T19:17:52.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f504009ed40 con 0x7f50400a21f0 2024-06-11T19:17:52.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:52.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.642+0000 7f5048669640 10 monclient(hunting): authenticate will time out at 1282.449927s 2024-06-11T19:17:52.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046405640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50400a0830 0x7f50401352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f50400a21f0 0x7f50400a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f50400a21f0 0x7f50400a2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:33226/0 (socket says 172.21.15.98:33226) 2024-06-11T19:17:52.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 1 -- 172.21.15.98:0/3336279250 learned_addr learned my addr 172.21.15.98:0/3336279250 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:52.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 10 monclient(hunting): get_auth_request con 0x7f50400a21f0 auth_method 0 2024-06-11T19:17:52.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046405640 10 monclient(hunting): get_auth_request con 0x7f50400a0830 auth_method 0 2024-06-11T19:17:52.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046405640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046405640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046405640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 10 monclient(hunting): handle_auth_done global_id 55203 payload 274 2024-06-11T19:17:52.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 1 -- 172.21.15.98:0/3336279250 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f50400a0c10 msgr2=0x7f50400a1050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-11T19:17:52.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 1 --2- 172.21.15.98:0/3336279250 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f50400a0c10 0x7f50400a1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 1 -- 172.21.15.98:0/3336279250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50400a0830 msgr2=0x7f50401352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:17:52.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 1 --2- 172.21.15.98:0/3336279250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50400a0830 0x7f50401352e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 10 monclient: _finish_hunting 0 2024-06-11T19:17:52.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 1 monclient: found mon.noname-d 2024-06-11T19:17:52.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:52.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:17:52.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046c06640 1 -- 172.21.15.98:0/3336279250 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50400a4880 con 0x7f50400a21f0 2024-06-11T19:17:52.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.643+0000 7f5046405640 1 --2- 172.21.15.98:0/3336279250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50400a0830 0x7f50401352e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:17:52.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.645+0000 7f5046c06640 1 --2- 172.21.15.98:0/3336279250 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f50400a21f0 0x7f50400a2670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5038009ee0 tx=0x7f503802bd70 comp rx=0 tx=0).ready entity=mon.6 client_cookie=9efa3f8b02dcedcb server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 1 -- 172.21.15.98:0/3336279250 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f503802cb80 con 0x7f50400a21f0 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 10 monclient: dump: 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.656 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 10 monclient: _finish_auth 0 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 10 monclient: _check_auth_tickets 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 1 -- 172.21.15.98:0/3336279250 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503802cd40 con 0x7f50400a21f0 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5048669640 5 monclient: authenticate success, global_id 55203 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5048669640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 1 -- 172.21.15.98:0/3336279250 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5038034860 con 0x7f50400a21f0 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5048669640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5048669640 4 set_mon_vals no callback set 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.646+0000 7f5045403640 10 monclient: dump: 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.659 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.647+0000 7f5048669640 10 monclient: shutdown 2024-06-11T19:17:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.647+0000 7f5048669640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.647+0000 7f5048669640 1 -- 172.21.15.98:0/3336279250 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f50400a21f0 msgr2=0x7f50400a2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.647+0000 7f5048669640 1 --2- 172.21.15.98:0/3336279250 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f50400a21f0 0x7f50400a2670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5038009ee0 tx=0x7f503802bd70 comp rx=0 tx=0).stop 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.647+0000 7f5048669640 1 -- 172.21.15.98:0/3336279250 shutdown_connections 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.647+0000 7f5048669640 1 --2- 172.21.15.98:0/3336279250 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f50400a21f0 0x7f50400a2670 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.647+0000 7f5048669640 1 --2- 172.21.15.98:0/3336279250 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f50400a0c10 0x7f50400a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.647+0000 7f5048669640 1 --2- 172.21.15.98:0/3336279250 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50400a0830 0x7f50401352e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.647+0000 7f5048669640 1 -- 172.21.15.98:0/3336279250 >> 172.21.15.98:0/3336279250 conn(0x7f50400a3a60 msgr2=0x7f50400a3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.648+0000 7f5048669640 1 -- 172.21.15.98:0/3336279250 shutdown_connections 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.648+0000 7f5048669640 1 -- 172.21.15.98:0/3336279250 wait complete. 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.648+0000 7f5048669640 10 monclient: build_initial_monmap 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.648+0000 7f5048669640 10 monclient: monmap: 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:52.662 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.648+0000 7f5048669640 1 librados: starting msgr at 2024-06-11T19:17:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.648+0000 7f5048669640 1 librados: starting objecter 2024-06-11T19:17:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.648+0000 7f5048669640 1 Processor -- start 2024-06-11T19:17:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 1 -- start start 2024-06-11T19:17:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 1 librados: setting wanted keys 2024-06-11T19:17:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 1 librados: calling monclient init 2024-06-11T19:17:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 10 monclient: init 2024-06-11T19:17:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 10 monclient: _add_conns ranks=[4,2,1,7,8,6,5,0,3] 2024-06-11T19:17:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f50400a0830 0x7f504012e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 10 monclient(hunting): picked mon.noname-c con 0x7f50400a0830 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50400a0c10 0x7f504012ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 10 monclient(hunting): picked mon.noname-b con 0x7f50400a0c10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50400a21f0 0x7f5040071c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 10 monclient(hunting): picked mon.noname-f con 0x7f50400a21f0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f50400a2d30 con 0x7f50400a21f0 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f50400a2bb0 con 0x7f50400a0c10 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f50400a2eb0 con 0x7f50400a0830 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5048669640 10 monclient(hunting): authenticate will time out at 1282.456441s 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046405640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f50400a0830 0x7f504012e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046405640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f50400a0830 0x7f504012e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:51036/0 (socket says 172.21.15.98:51036) 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046405640 1 -- 172.21.15.98:0/2400120430 learned_addr learned my addr 172.21.15.98:0/2400120430 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046405640 10 monclient(hunting): get_auth_request con 0x7f50400a0830 auth_method 0 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046405640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046405640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046405640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046c06640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50400a21f0 0x7f5040071c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046c06640 10 monclient(hunting): get_auth_request con 0x7f50400a21f0 auth_method 0 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046c06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046c06640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046c06640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5045c04640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50400a0c10 0x7f504012ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046405640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046405640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046405640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046c06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046c06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.649+0000 7f5046c06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5046405640 10 monclient(hunting): handle_auth_done global_id 24034 payload 824 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5046405640 1 -- 172.21.15.98:0/2400120430 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50400a0c10 msgr2=0x7f504012ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5046405640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f50400a0c10 0x7f504012ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5046405640 1 -- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50400a21f0 msgr2=0x7f5040071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5046405640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50400a21f0 0x7f5040071c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5046405640 10 monclient: _finish_hunting 0 2024-06-11T19:17:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5046405640 1 monclient: found mon.noname-c 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5046405640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5046405640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5046405640 1 -- 172.21.15.98:0/2400120430 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50400737a0 con 0x7f50400a0830 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5046c06640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50400a21f0 0x7f5040071c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5046405640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f50400a0830 0x7f504012e8c0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f503c00e730 tx=0x7f503c00ebf0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5032ffd640 10 client.?.objecter ms_handle_connect 0x7f50400a0830 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.650+0000 7f5032ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f503c00f040 con 0x7f50400a0830 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 10 monclient: dump: 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 10 monclient: _finish_auth 0 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503c00ee40 con 0x7f50400a0830 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f503c0044a0 con 0x7f50400a0830 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5032ffd640 10 monclient: dump: 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5044c02640 4 set_mon_vals no callback set 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5048669640 5 monclient: authenticate success, global_id 24034 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5048669640 10 monclient: _renew_subs 2024-06-11T19:17:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5048669640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5048669640 1 -- 172.21.15.98:0/2400120430 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f504012b370 con 0x7f50400a0830 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5048669640 10 client.24034.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5048669640 10 monclient: _renew_subs 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5048669640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5048669640 1 -- 172.21.15.98:0/2400120430 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f504013f1e0 con 0x7f50400a0830 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.651+0000 7f5048669640 1 librados: init done 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.652+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f503c017c00 con 0x7f50400a0830 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.652+0000 7f5048669640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.652+0000 7f5048669640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f50401cd8e0 0x7f504012abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.652+0000 7f5048669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.652+0000 7f5048669640 1 -- 172.21.15.98:0/2400120430 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f5040050830 con 0x7f50401cd8e0 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.652+0000 7f5032ffd640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f502403fdb0 0x7f5024042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(348..348 src has 1..348) ==== 6230+0+0 (secure 0 0 0) 0x7f503c059a70 con 0x7f50400a0830 2024-06-11T19:17:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5032ffd640 10 client.24034.objecter ms_dispatch 0x7f5040000b90 osd_map(348..348 src has 1..348) 2024-06-11T19:17:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5032ffd640 3 client.24034.objecter handle_osd_map got epochs [348,348] > 0 2024-06-11T19:17:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5032ffd640 3 client.24034.objecter handle_osd_map decoding full epoch 348 2024-06-11T19:17:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5045c04640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f50401cd8e0 0x7f504012abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5045c04640 10 monclient: get_auth_request con 0x7f50401cd8e0 auth_method 0 2024-06-11T19:17:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5045c04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5045c04640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5045c04640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5032ffd640 20 client.24034.objecter dump_active .. 0 homeless 2024-06-11T19:17:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5046c06640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f502403fdb0 0x7f5024042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5046c06640 10 monclient: get_auth_request con 0x7f502403fdb0 auth_method 0 2024-06-11T19:17:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5045c04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5045c04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5045c04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5045c04640 10 monclient(hunting): handle_auth_done global_id 84346 payload 824 2024-06-11T19:17:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5045c04640 1 -- 172.21.15.98:0/2400120430 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5040135400 con 0x7f50401cd8e0 2024-06-11T19:17:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5045c04640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f50401cd8e0 0x7f504012abb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5040077150 tx=0x7f503400c8a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5032ffd640 10 client.24034.objecter ms_handle_connect 0x7f50401cd8e0 2024-06-11T19:17:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5032ffd640 10 client.24034.objecter resend_mon_ops 2024-06-11T19:17:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5046c06640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f502403fdb0 0x7f5024042270 secure :-1 s=READY pgs=4181 cs=0 l=1 rev1=1 crypto rx=0x7f5038035000 tx=0x7f5038030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5032ffd640 10 client.24034.objecter ms_handle_connect 0x7f502403fdb0 2024-06-11T19:17:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5034015070 con 0x7f50401cd8e0 2024-06-11T19:17:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5032ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5032ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.653+0000 7f5032ffd640 10 monclient: dump: 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.674 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.654+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5034011600 con 0x7f50401cd8e0 2024-06-11T19:17:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.654+0000 7f5032ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.654+0000 7f5032ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.654+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f50401cd8e0 msgr2=0x7f504012abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.654+0000 7f5032ffd640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f50401cd8e0 0x7f504012abb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5040077150 tx=0x7f503400c8a0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.654+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f50401cd8e0 msgr2=0x7f504012abb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.654+0000 7f5032ffd640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f50401cd8e0 0x7f504012abb0 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5040077150 tx=0x7f503400c8a0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.660+0000 7f5048669640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:17:52.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.660+0000 7f5048669640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5040072180 0x7f5040072590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.660+0000 7f5048669640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.660+0000 7f5048669640 1 -- 172.21.15.98:0/2400120430 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f5040072ad0 con 0x7f5040072180 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5045c04640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5040072180 0x7f5040072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5045c04640 10 monclient: get_auth_request con 0x7f5040072180 auth_method 0 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5045c04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5045c04640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5045c04640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5045c04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5045c04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5045c04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5045c04640 10 monclient(hunting): handle_auth_done global_id 84355 payload 824 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5045c04640 1 -- 172.21.15.98:0/2400120430 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f50400a0ff0 con 0x7f5040072180 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5032ffd640 10 client.24034.objecter ms_handle_connect 0x7f5040072180 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5032ffd640 10 client.24034.objecter resend_mon_ops 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5045c04640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5040072180 0x7f5040072590 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5034017000 tx=0x7f5034004750 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5034028020 con 0x7f5040072180 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5032ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5032ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.661+0000 7f5032ffd640 10 monclient: dump: 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.679 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f5034023350 con 0x7f5040072180 2024-06-11T19:17:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5032ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:17:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5032ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5040072180 msgr2=0x7f5040072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5032ffd640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5040072180 0x7f5040072590 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5034017000 tx=0x7f5034004750 comp rx=0 tx=0).stop 2024-06-11T19:17:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5032ffd640 1 -- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5040072180 msgr2=0x7f5040072590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5032ffd640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5040072180 0x7f5040072590 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.680 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:52.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:52.680 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:52.680 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:17:52.680 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 38, 2024-06-11T19:17:52.680 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 144431, 2024-06-11T19:17:52.680 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:17:52.680 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:17:52.680 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:17:52.680 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:17:52.680 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:17:52.680 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:17:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:17:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:17:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:17:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:17:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:17:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:17:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:17:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:17:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.697 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:17:52.697 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:17:52.697 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.697 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.697 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.697 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:17:52.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:17:52.698 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:17:52.698 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:17:52.698 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.698 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:17:52.700 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:17:52.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:17:52.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.703 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.703 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:17:52.703 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:17:52.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.703 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.703 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.703 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:52.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.704 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:17:52.704 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:17:52.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:17:52.704 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:17:52.704 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:17:52.704 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:17:52.704 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:17:52.704 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:17:52.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:17:52.704 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:17:52.704 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:52.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 10 librados: watch_flush enter 2024-06-11T19:17:52.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 10 librados: watch_flush exit 2024-06-11T19:17:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 20 client.24034.objecter shutdown clearing up homeless session... 2024-06-11T19:17:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 10 client.24034.objecter successfully canceled tick 2024-06-11T19:17:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 1 -- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f502403fdb0 msgr2=0x7f5024042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f502403fdb0 0x7f5024042270 secure :-1 s=READY pgs=4181 cs=0 l=1 rev1=1 crypto rx=0x7f5038035000 tx=0x7f5038030000 comp rx=0 tx=0).stop 2024-06-11T19:17:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 10 monclient: shutdown 2024-06-11T19:17:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 1 -- 172.21.15.98:0/2400120430 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f50400a0830 msgr2=0x7f504012e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 1 --2- 172.21.15.98:0/2400120430 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f50400a0830 0x7f504012e8c0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f503c00e730 tx=0x7f503c00ebf0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5046c06640 1 -- 172.21.15.98:0/2400120430 reap_dead start 2024-06-11T19:17:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5046c06640 1 -- 172.21.15.98:0/2400120430 reap_dead start 2024-06-11T19:17:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 1 -- 172.21.15.98:0/2400120430 shutdown_connections 2024-06-11T19:17:52.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 1 -- 172.21.15.98:0/2400120430 >> 172.21.15.98:0/2400120430 conn(0x7f50400a3a60 msgr2=0x7f50400a5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:52.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 1 -- 172.21.15.98:0/2400120430 shutdown_connections 2024-06-11T19:17:52.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 1 -- 172.21.15.98:0/2400120430 wait complete. 2024-06-11T19:17:52.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.662+0000 7f5048669640 1 librados: shutdown 2024-06-11T19:17:52.708 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:17:52.708 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['g'] 2024-06-11T19:17:52.708 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['h'] 2024-06-11T19:17:52.708 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-06-11T19:17:52.708 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g store 2024-06-11T19:17:52.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g sync_force --yes-i-really-mean-it 2024-06-11T19:17:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.810+0000 7fece7fe3640 10 monclient: get_monmap_and_config 2024-06-11T19:17:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.810+0000 7fece7fe3640 10 monclient: build_initial_monmap 2024-06-11T19:17:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.810+0000 7fece7fe3640 10 monclient: monmap: 2024-06-11T19:17:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:17:52.811247+0000 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:17:52.811247+0000 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:17:52.811 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:17:52.811 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.810+0000 7fece7fe3640 1 Processor -- start 2024-06-11T19:17:52.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.810+0000 7fece7fe3640 1 -- start start 2024-06-11T19:17:52.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.810+0000 7fece7fe3640 10 monclient: init 2024-06-11T19:17:52.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece7fe3640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:52.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece7fe3640 10 monclient: _add_conns ranks=[9,0,1,10,5,2,7,4,3,6,8] 2024-06-11T19:17:52.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece7fe3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fece00a4920 0x7fece00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece7fe3640 10 monclient(hunting): picked mon.noname-a con 0x7fece00a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:17:52.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece7fe3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fece00a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:17:52.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece7fe3640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fece00a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:17:52.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece7fe3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fece00a7140 con 0x7fece00a5240 2024-06-11T19:17:52.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece7fe3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fece00a67c0 con 0x7fece00a2530 2024-06-11T19:17:52.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece7fe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece7fe3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fece00a2b50 con 0x7fece00a4920 2024-06-11T19:17:52.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece7fe3640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:52.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece7fe3640 10 monclient(hunting): authenticate will time out at 1282.618558s 2024-06-11T19:17:52.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fece00a4920 0x7fece00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fece00a4920 0x7fece00a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56758/0 (socket says 172.21.15.98:56758) 2024-06-11T19:17:52.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece557e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fece00a5240 0x7fece00a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35790/0 (socket says 172.21.15.98:35790) 2024-06-11T19:17:52.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 1 -- 172.21.15.98:0/3646506923 learned_addr learned my addr 172.21.15.98:0/3646506923 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:52.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 10 monclient(hunting): get_auth_request con 0x7fece00a4920 auth_method 0 2024-06-11T19:17:52.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 10 monclient(hunting): handle_auth_done global_id 65133 payload 274 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 1 -- 172.21.15.98:0/3646506923 >> v1:172.21.15.117:6789/0 conn(0x7fece00a2530 legacy=0x7fece00a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 1 -- 172.21.15.98:0/3646506923 >> v1:172.21.15.98:6789/0 conn(0x7fece00a5240 legacy=0x7fece00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 10 monclient: _finish_hunting 0 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 1 monclient: found mon.noname-a 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece4d7d640 1 -- 172.21.15.98:0/3646506923 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1008854384 0 0) 0x7fece00a7140 con 0x7fece00a5240 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.811+0000 7fece5d7f640 1 -- 172.21.15.98:0/3646506923 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece00a2e30 con 0x7fece00a4920 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:17:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece5d7f640 1 --2- 172.21.15.98:0/3646506923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece00a4920 0x7fece00a4d00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009870 tx=0x7fecdc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10efc08114cffb50 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 1 -- 172.21.15.98:0/3646506923 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fecdc035020 con 0x7fece00a4920 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 10 monclient: dump: 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.825 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 10 monclient: _finish_auth 0 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 10 monclient: _check_auth_tickets 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece7fe3640 5 monclient: authenticate success, global_id 65133 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 1 -- 172.21.15.98:0/3646506923 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecdc031ae0 con 0x7fece00a4920 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece7fe3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:17:52.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece7fe3640 10 monclient: get_monmap_and_config success 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece7fe3640 4 set_mon_vals no callback set 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 1 -- 172.21.15.98:0/3646506923 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fecdc031c90 con 0x7fece00a4920 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.812+0000 7fece4d7d640 10 monclient: dump: 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.827 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.813+0000 7fece7fe3640 10 monclient: shutdown 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.813+0000 7fece7fe3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.813+0000 7fece7fe3640 1 -- 172.21.15.98:0/3646506923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece00a4920 msgr2=0x7fece00a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.813+0000 7fece7fe3640 1 --2- 172.21.15.98:0/3646506923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece00a4920 0x7fece00a4d00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009870 tx=0x7fecdc02b910 comp rx=0 tx=0).stop 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.813+0000 7fece7fe3640 1 -- 172.21.15.98:0/3646506923 shutdown_connections 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.813+0000 7fece7fe3640 1 --2- 172.21.15.98:0/3646506923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece00a4920 0x7fece00a4d00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.813+0000 7fece7fe3640 1 -- 172.21.15.98:0/3646506923 >> 172.21.15.98:0/3646506923 conn(0x7fece00ab850 msgr2=0x7fece00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.813+0000 7fece7fe3640 1 -- 172.21.15.98:0/3646506923 shutdown_connections 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.813+0000 7fece7fe3640 1 -- 172.21.15.98:0/3646506923 wait complete. 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient: build_initial_monmap 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient: monmap: 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:17:52.829 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 1 librados: starting msgr at 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 1 librados: starting objecter 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 1 Processor -- start 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 1 -- start start 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 1 librados: setting wanted keys 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 1 librados: calling monclient init 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient: init 2024-06-11T19:17:52.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient: _reopen_session rank -1 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient: _add_conns ranks=[6,2,4,1,5,8,0,7,3] 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fece00a2530 0x7fece01cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient(hunting): picked mon.noname-d con 0x7fece00a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fece00a4920 0x7fece01cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient(hunting): picked mon.noname-b con 0x7fece00a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fece00a5240 0x7fece0123db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient(hunting): picked mon.noname-c con 0x7fece00a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fece0138e50 con 0x7fece00a4920 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fece00a84b0 con 0x7fece00a5240 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fece00a6da0 con 0x7fece00a2530 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient(hunting): _renew_subs 2024-06-11T19:17:52.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.814+0000 7fece7fe3640 10 monclient(hunting): authenticate will time out at 1282.621989s 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fece00a2530 0x7fece01cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fece00a2530 0x7fece01cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:33238/0 (socket says 172.21.15.98:33238) 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 1 -- 172.21.15.98:0/4141687603 learned_addr learned my addr 172.21.15.98:0/4141687603 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 10 monclient(hunting): get_auth_request con 0x7fece00a2530 auth_method 0 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece557e640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fece00a4920 0x7fece01cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece6580640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fece00a5240 0x7fece0123db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece6580640 10 monclient(hunting): get_auth_request con 0x7fece00a5240 auth_method 0 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece6580640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece6580640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece6580640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece557e640 10 monclient(hunting): get_auth_request con 0x7fece00a4920 auth_method 0 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece557e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece557e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece557e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece6580640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece6580640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece6580640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece557e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece557e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece557e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 10 monclient(hunting): handle_auth_done global_id 55212 payload 824 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 1 -- 172.21.15.98:0/4141687603 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fece00a5240 msgr2=0x7fece0123db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fece00a5240 0x7fece0123db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 1 -- 172.21.15.98:0/4141687603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fece00a4920 msgr2=0x7fece01cdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fece00a4920 0x7fece01cdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 10 monclient: _finish_hunting 0 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 1 monclient: found mon.noname-d 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 1 -- 172.21.15.98:0/4141687603 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece01277b0 con 0x7fece00a2530 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece5d7f640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fece00a2530 0x7fece01cda10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fecdc000c00 tx=0x7fecdc0046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 10 client.?.objecter ms_handle_connect 0x7fece00a2530 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fecdc035050 con 0x7fece00a2530 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 10 monclient: dump: 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.835 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 10 monclient: _finish_auth 0 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 10 monclient: _check_auth_tickets 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece7fe3640 5 monclient: authenticate success, global_id 55212 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecdc003c50 con 0x7fece00a2530 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece7fe3640 10 monclient: _renew_subs 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece7fe3640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece7fe3640 1 -- 172.21.15.98:0/4141687603 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece0127a90 con 0x7fece00a2530 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fecdc003e00 con 0x7fece00a2530 2024-06-11T19:17:52.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd67fc640 10 monclient: dump: 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.837 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.838 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.838 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.838 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.838 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.838 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fecd7fff640 4 set_mon_vals no callback set 2024-06-11T19:17:52.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece7fe3640 10 client.55212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:17:52.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece7fe3640 10 monclient: _renew_subs 2024-06-11T19:17:52.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece7fe3640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:17:52.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece7fe3640 1 -- 172.21.15.98:0/4141687603 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fece01265e0 con 0x7fece00a2530 2024-06-11T19:17:52.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.815+0000 7fece7fe3640 1 librados: init done 2024-06-11T19:17:52.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fece7fe3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fece7fe3640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124ab0 0x7fece002cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fece7fe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fece7fe3640 1 -- 172.21.15.98:0/4141687603 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fece0050830 con 0x7fece0124ab0 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fecdc038020 con 0x7fece00a2530 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fecd67fc640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fecc403fdb0 0x7fecc4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fece557e640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124ab0 0x7fece002cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fece557e640 10 monclient: get_auth_request con 0x7fece0124ab0 auth_method 0 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fece557e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(348..348 src has 1..348) ==== 6230+0+0 (secure 0 0 0) 0x7fecdc074090 con 0x7fece00a2530 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fece557e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fece557e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fecd67fc640 10 client.55212.objecter ms_dispatch 0x7fece0000b90 osd_map(348..348 src has 1..348) 2024-06-11T19:17:52.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fecd67fc640 3 client.55212.objecter handle_osd_map got epochs [348,348] > 0 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.816+0000 7fecd67fc640 3 client.55212.objecter handle_osd_map decoding full epoch 348 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fece557e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fece557e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fece557e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 20 client.55212.objecter dump_active .. 0 homeless 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fece6580640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fecc403fdb0 0x7fecc4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fece6580640 10 monclient: get_auth_request con 0x7fecc403fdb0 auth_method 0 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fece557e640 10 monclient(hunting): handle_auth_done global_id 94242 payload 824 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fece557e640 1 -- 172.21.15.98:0/4141687603 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fece00aac90 con 0x7fece0124ab0 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 10 client.55212.objecter ms_handle_connect 0x7fece0124ab0 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 10 client.55212.objecter resend_mon_ops 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fece557e640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124ab0 0x7fece002cb40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fecd0000f60 tx=0x7fecd00018c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 10 client.55212.objecter ms_handle_connect 0x7fecc403fdb0 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fece6580640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fecc403fdb0 0x7fecc4042270 secure :-1 s=READY pgs=4182 cs=0 l=1 rev1=1 crypto rx=0x7fece0136e90 tx=0x7fecd8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fecd0019070 con 0x7fece0124ab0 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 10 monclient: dump: 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.841 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fecd0011580 con 0x7fece0124ab0 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124ab0 msgr2=0x7fece002cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124ab0 0x7fece002cb40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fecd0000f60 tx=0x7fecd00018c0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124ab0 msgr2=0x7fece002cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:52.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.817+0000 7fecd67fc640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124ab0 0x7fece002cb40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.825+0000 7fece7fe3640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:17:52.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.825+0000 7fece7fe3640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124e90 0x7fece002d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:17:52.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.825+0000 7fece7fe3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:17:52.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.825+0000 7fece7fe3640 1 -- 172.21.15.98:0/4141687603 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fece00a4d00 con 0x7fece0124e90 2024-06-11T19:17:52.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.825+0000 7fece557e640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124e90 0x7fece002d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:17:52.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.825+0000 7fece557e640 10 monclient: get_auth_request con 0x7fece0124e90 auth_method 0 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.825+0000 7fece557e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.825+0000 7fece557e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.825+0000 7fece557e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.825+0000 7fece557e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.825+0000 7fece557e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.825+0000 7fece557e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fece557e640 10 monclient(hunting): handle_auth_done global_id 94251 payload 824 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fece557e640 1 -- 172.21.15.98:0/4141687603 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7fece00a0950 con 0x7fece0124e90 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fece557e640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124e90 0x7fece002d080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fecd0007820 tx=0x7fecd0007f90 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fecd67fc640 10 client.55212.objecter ms_handle_connect 0x7fece0124e90 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fecd67fc640 10 client.55212.objecter resend_mon_ops 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fecd0019030 con 0x7fece0124e90 2024-06-11T19:17:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fecd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fecd67fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fecd67fc640 10 monclient: dump: 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:17:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:17:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:17:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:17:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:17:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:17:52.847 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:17:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7fecd001a040 con 0x7fece0124e90 2024-06-11T19:17:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fecd67fc640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:17:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fecd67fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:17:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124e90 msgr2=0x7fece002d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.826+0000 7fecd67fc640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124e90 0x7fece002d080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fecd0007820 tx=0x7fecd0007f90 comp rx=0 tx=0).stop 2024-06-11T19:17:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fecd67fc640 1 -- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124e90 msgr2=0x7fece002d080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:17:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fecd67fc640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fece0124e90 0x7fece002d080 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.848 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:17:52.848 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-06-11T19:17:52.848 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-06-11T19:17:52.848 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:17:52.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 10 librados: watch_flush enter 2024-06-11T19:17:52.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 10 librados: watch_flush exit 2024-06-11T19:17:52.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 20 client.55212.objecter shutdown clearing up homeless session... 2024-06-11T19:17:52.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 10 client.55212.objecter successfully canceled tick 2024-06-11T19:17:52.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 1 -- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fecc403fdb0 msgr2=0x7fecc4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fecc403fdb0 0x7fecc4042270 secure :-1 s=READY pgs=4182 cs=0 l=1 rev1=1 crypto rx=0x7fece0136e90 tx=0x7fecd8009040 comp rx=0 tx=0).stop 2024-06-11T19:17:52.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 10 monclient: shutdown 2024-06-11T19:17:52.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:17:52.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 1 -- 172.21.15.98:0/4141687603 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fece00a2530 msgr2=0x7fece01cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:17:52.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 1 --2- 172.21.15.98:0/4141687603 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fece00a2530 0x7fece01cda10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fecdc000c00 tx=0x7fecdc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:17:52.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece6580640 1 -- 172.21.15.98:0/4141687603 reap_dead start 2024-06-11T19:17:52.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece6580640 1 -- 172.21.15.98:0/4141687603 reap_dead start 2024-06-11T19:17:52.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 1 -- 172.21.15.98:0/4141687603 shutdown_connections 2024-06-11T19:17:52.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 1 -- 172.21.15.98:0/4141687603 >> 172.21.15.98:0/4141687603 conn(0x7fece00ab850 msgr2=0x7fece0138090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:17:52.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 1 -- 172.21.15.98:0/4141687603 shutdown_connections 2024-06-11T19:17:52.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 1 -- 172.21.15.98:0/4141687603 wait complete. 2024-06-11T19:17:52.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:17:52.827+0000 7fece7fe3640 1 librados: shutdown 2024-06-11T19:17:52.851 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-06-11T19:17:52.851 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-06-11T19:17:52.851 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:17:52.864 INFO:tasks.ceph.mon.g:Stopped 2024-06-11T19:17:52.864 INFO:tasks.mon_thrash:Sending STOP to mon h 2024-06-11T19:17:52.865 INFO:tasks.ceph.mon.h:Sent signal 19 2024-06-11T19:17:52.865 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-06-11T19:18:07.865 INFO:tasks.mon_thrash:Sending CONT to mon h 2024-06-11T19:18:07.865 INFO:tasks.ceph.mon.h:Sent signal 18 2024-06-11T19:18:07.866 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-06-11T19:18:07.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:18:07.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.977+0000 7feeabe69640 10 monclient: get_monmap_and_config 2024-06-11T19:18:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.977+0000 7feeabe69640 10 monclient: build_initial_monmap 2024-06-11T19:18:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.977+0000 7feeabe69640 10 monclient: monmap: 2024-06-11T19:18:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:18:07.978698+0000 2024-06-11T19:18:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:18:07.978698+0000 2024-06-11T19:18:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:18:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:18:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:07.977 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:18:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:18:07.978 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:07.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 1 Processor -- start 2024-06-11T19:18:07.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 1 -- start start 2024-06-11T19:18:07.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 10 monclient: init 2024-06-11T19:18:07.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:07.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 10 monclient: _add_conns ranks=[5,3,0,6,10,8,2,7,9,4,1] 2024-06-11T19:18:07.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:07.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 10 monclient(hunting): picked mon.noname-h con 0x7feea40a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:18:07.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea40a5240 0x7feea40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:07.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 10 monclient(hunting): picked mon.noname-g con 0x7feea40a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:18:07.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feea40a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feea40a7140 con 0x7feea40a2530 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7feea40a2b50 con 0x7feea40a5240 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.978+0000 7feeabe69640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7feea40a2e30 con 0x7feea40a4920 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feeabe69640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feeabe69640 10 monclient(hunting): authenticate will time out at 1297.786011s 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feea9404640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea40a5240 msgr2=0x7feea40a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feea9404640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea40a5240 0x7feea40a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feeaa406640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7feea40a2530 0x7feea40a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43474/0 (socket says 172.21.15.98:43474) 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feeaa406640 1 -- 172.21.15.98:0/516879449 learned_addr learned my addr 172.21.15.98:0/516879449 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feea9c05640 10 monclient(hunting): get_auth_request con 0x7feea40a4920 auth_method 0 2024-06-11T19:18:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feea9c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:07.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feea9c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:07.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feea9c05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:07.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 msgr2=0x7feea40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:18:07.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 msgr2=0x7feea40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:18:07.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:07.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:07.979+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:08.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.180+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea40a5240 msgr2=0x7feea40a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:18:08.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.180+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea40a5240 0x7feea40a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:18:08.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.180+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:08.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.180+0000 7feea9c05640 10 monclient(hunting): get_auth_request con 0x7feea40a4920 auth_method 0 2024-06-11T19:18:08.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.180+0000 7feea9c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:08.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.180+0000 7feea9c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:08.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.180+0000 7feea9c05640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:18:08.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.180+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 msgr2=0x7feea40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:18:08.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.180+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 msgr2=0x7feea40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:18:08.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.180+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:08.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.180+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:18:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.581+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea40a5240 msgr2=0x7feea40a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:18:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.581+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea40a5240 0x7feea40a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:18:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.581+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.581+0000 7feea9c05640 10 monclient(hunting): get_auth_request con 0x7feea40a4920 auth_method 0 2024-06-11T19:18:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.581+0000 7feea9c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.581+0000 7feea9c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.581+0000 7feea9c05640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:18:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.581+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 msgr2=0x7feea40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:18:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.581+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 msgr2=0x7feea40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:18:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.581+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:08.581+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:18:09.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:09.382+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea40a5240 msgr2=0x7feea40a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:18:09.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:09.382+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea40a5240 0x7feea40a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:18:09.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:09.382+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:09.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:09.382+0000 7feea9c05640 10 monclient(hunting): get_auth_request con 0x7feea40a4920 auth_method 0 2024-06-11T19:18:09.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:09.382+0000 7feea9c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:09.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:09.382+0000 7feea9c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:09.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:09.382+0000 7feea9c05640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:18:09.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:09.382+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 msgr2=0x7feea40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:18:09.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:09.382+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 msgr2=0x7feea40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:18:09.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:09.382+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:09.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:09.382+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:18:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 10 monclient(hunting): tick 2024-06-11T19:18:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:18:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 1 monclient(hunting): continuing hunt 2024-06-11T19:18:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:18:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 msgr2=0x7feea40a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:18:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7feea40a4920 0x7feea40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea40a5240 msgr2=0x7feea40a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea40a5240 0x7feea40a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 1 -- 172.21.15.98:0/516879449 >> v1:172.21.15.98:6789/0 conn(0x7feea40a2530 legacy=0x7feea40a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 10 monclient: _add_conns ranks=[9,6,1,8,4,2,10,7] 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fee88003820 addr v2:172.21.15.98:3300/0 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 10 monclient(hunting): picked mon.noname-d con 0x7fee88006270 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fee88008da0 addr v1:172.21.15.117:6789/0 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 1 -- 172.21.15.98:0/516879449 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feea40a3110 con 0x7fee88008da0 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 1 -- 172.21.15.98:0/516879449 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7feea40a2b50 con 0x7fee88003820 2024-06-11T19:18:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 1 -- 172.21.15.98:0/516879449 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7feea40a2e30 con 0x7fee88006270 2024-06-11T19:18:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7fee9b7fe640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9c05640 10 monclient(hunting): get_auth_request con 0x7fee88003820 auth_method 0 2024-06-11T19:18:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9c05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 msgr2=0x7fee88005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:18:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 msgr2=0x7fee88005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:18:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9404640 10 monclient(hunting): get_auth_request con 0x7fee88006270 auth_method 0 2024-06-11T19:18:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9404640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9404640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 msgr2=0x7fee880086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:18:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 msgr2=0x7fee880086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:18:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:10.979+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:11.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9c05640 10 monclient(hunting): get_auth_request con 0x7fee88003820 auth_method 0 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9c05640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9404640 10 monclient(hunting): get_auth_request con 0x7fee88006270 auth_method 0 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9404640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 msgr2=0x7fee88005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 msgr2=0x7fee88005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:11.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:18:11.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 msgr2=0x7fee880086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:18:11.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 msgr2=0x7fee880086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:18:11.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:11.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.180+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:18:11.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:11.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9404640 10 monclient(hunting): get_auth_request con 0x7fee88006270 auth_method 0 2024-06-11T19:18:11.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:11.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9404640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:11.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:18:11.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:11.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9c05640 10 monclient(hunting): get_auth_request con 0x7fee88003820 auth_method 0 2024-06-11T19:18:11.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:11.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:11.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9c05640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:18:11.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 msgr2=0x7fee88005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:18:11.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 msgr2=0x7fee88005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:18:11.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:11.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 msgr2=0x7fee880086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:18:11.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:18:11.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 msgr2=0x7fee880086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:18:11.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:11.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:11.581+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:18:12.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:12.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:12.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9404640 10 monclient(hunting): get_auth_request con 0x7fee88006270 auth_method 0 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9404640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9c05640 10 monclient(hunting): get_auth_request con 0x7fee88003820 auth_method 0 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9c05640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 msgr2=0x7fee88005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 msgr2=0x7fee880086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 msgr2=0x7fee88005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 msgr2=0x7fee880086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:12.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:18:12.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.383+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:18:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2201458353 0 0) 0x7feea40a3110 con 0x7fee88008da0 2024-06-11T19:18:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient(hunting): _negotiate 2024-06-11T19:18:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient(hunting): my global_id is 32266 2024-06-11T19:18:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee90003520 con 0x7fee88008da0 2024-06-11T19:18:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 603688727 0 0) 0x7fee90003520 con 0x7fee88008da0 2024-06-11T19:18:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 msgr2=0x7fee880086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:18:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee88006270 0x7fee880086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 msgr2=0x7fee88005c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 1 --2- 172.21.15.98:0/516879449 >> v2:172.21.15.98:3300/0 conn(0x7fee88003820 0x7fee88005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient: _finish_hunting 0 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 1 monclient: found mon.noname-f-legacy 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feeaa406640 1 -- 172.21.15.98:0/516879449 reap_dead start 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.117:6789/0 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8800dc60 con 0x7fee88008da0 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient: _finish_auth 0 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient: _check_auth_tickets 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7fee9c001d20 con 0x7fee88008da0 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feeabe69640 5 monclient: authenticate success, global_id 32266 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feeabe69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:18:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2024-06-11T19:18:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient: dump: 2024-06-11T19:18:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 1 monclient: mon.f has (v2) addrs [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] but i'm connected to v1:172.21.15.117:6789/0, reconnecting 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 >> v1:172.21.15.117:6789/0 conn(0x7fee88008da0 legacy=0x7fee8800b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.887+0000 7feea8c03640 10 monclient: _add_conns ranks=[3,0,8,4,2,5,7,1,6] 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee90004200 0x7fee90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 10 monclient(hunting): picked mon.g con 0x7fee90004200 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee90004e10 0x7fee90005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 10 monclient(hunting): picked mon.a con 0x7fee90004e10 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee90009bf0 0x7fee9000c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 10 monclient(hunting): picked mon.e con 0x7fee90009bf0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feea40a2b50 con 0x7fee90004e10 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7feea40a2e30 con 0x7fee90004200 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fee9000c990 con 0x7fee90009bf0 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea8c03640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feeabe69640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9c05640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee90004200 msgr2=0x7fee90004800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9c05640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee90004200 0x7fee90004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feeaa406640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee90009bf0 0x7fee9000c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee90004e10 0x7fee90005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 10 monclient(hunting): get_auth_request con 0x7fee90004e10 auth_method 0 2024-06-11T19:18:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:18:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feeaa406640 10 monclient(hunting): get_auth_request con 0x7fee90009bf0 auth_method 0 2024-06-11T19:18:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feeaa406640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feeaa406640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feeaa406640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:18:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feeaa406640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feeaa406640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feeaa406640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 10 monclient(hunting): handle_auth_done global_id 32266 payload 293 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee90009bf0 msgr2=0x7fee9000c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee90009bf0 0x7fee9000c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee90004200 msgr2=0x7fee90004800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee90004200 0x7fee90004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 10 monclient: _finish_hunting 0 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 1 monclient: found mon.a 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:18:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 1 -- 172.21.15.98:0/516879449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee90003520 con 0x7fee90004e10 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 10 monclient: _finish_auth 0 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 10 monclient: _check_auth_tickets 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feea9404640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.888+0000 7feeaa406640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee90009bf0 0x7fee9000c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feea9404640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee90004e10 0x7fee90005290 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7feea4073fc0 tx=0x7fee90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dfa5c2567878c65 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee94009ae0 con 0x7fee90004e10 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feea8c03640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feea8c03640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feea8c03640 10 monclient: dump: 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:12.898 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee9402e960 con 0x7fee90004e10 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feeabe69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feea8c03640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feea8c03640 1 -- 172.21.15.98:0/516879449 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee9402d420 con 0x7fee90004e10 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feea8c03640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feea8c03640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feea8c03640 10 monclient: dump: 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:12.900 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feeabe69640 10 monclient: get_monmap_and_config success 2024-06-11T19:18:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.890+0000 7feeabe69640 4 set_mon_vals no callback set 2024-06-11T19:18:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.891+0000 7feeabe69640 10 monclient: shutdown 2024-06-11T19:18:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.891+0000 7feeabe69640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.891+0000 7feeabe69640 1 -- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee90004e10 msgr2=0x7fee90005290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.891+0000 7feeabe69640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee90004e10 0x7fee90005290 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7feea4073fc0 tx=0x7fee90005830 comp rx=0 tx=0).stop 2024-06-11T19:18:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.891+0000 7feeabe69640 1 -- 172.21.15.98:0/516879449 shutdown_connections 2024-06-11T19:18:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.891+0000 7feeabe69640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee90009bf0 0x7fee9000c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.891+0000 7feeabe69640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee90004e10 0x7fee90005290 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.891+0000 7feeabe69640 1 --2- 172.21.15.98:0/516879449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee90004200 0x7fee90004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.891+0000 7feeabe69640 1 -- 172.21.15.98:0/516879449 >> 172.21.15.98:0/516879449 conn(0x7feea40ab850 msgr2=0x7feea40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.891+0000 7feeabe69640 1 -- 172.21.15.98:0/516879449 shutdown_connections 2024-06-11T19:18:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.891+0000 7feeabe69640 1 -- 172.21.15.98:0/516879449 wait complete. 2024-06-11T19:18:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.892+0000 7feeabe69640 10 monclient: build_initial_monmap 2024-06-11T19:18:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.892+0000 7feeabe69640 10 monclient: monmap: 2024-06-11T19:18:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:18:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:18:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:18:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.892+0000 7feeabe69640 1 librados: starting msgr at 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.892+0000 7feeabe69640 1 librados: starting objecter 2024-06-11T19:18:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 1 Processor -- start 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 1 -- start start 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 1 librados: setting wanted keys 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 1 librados: calling monclient init 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 10 monclient: init 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 10 monclient: _add_conns ranks=[0,1,3,6,2,4,8,5,7] 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee88008da0 0x7feea41c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 10 monclient(hunting): picked mon.noname-a con 0x7fee88008da0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee90004200 0x7feea4137e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 10 monclient(hunting): picked mon.noname-f con 0x7fee90004200 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee90004e10 0x7feea41cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 10 monclient(hunting): picked mon.noname-g con 0x7fee90004e10 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feea40a6da0 con 0x7fee88008da0 2024-06-11T19:18:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feea4138e50 con 0x7fee90004200 2024-06-11T19:18:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7feea40a4d00 con 0x7fee90004e10 2024-06-11T19:18:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeabe69640 10 monclient(hunting): authenticate will time out at 1302.700644s 2024-06-11T19:18:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeaa406640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee90004e10 msgr2=0x7feea41cd3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:18:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9c05640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee88008da0 0x7feea41c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feeaa406640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee90004e10 0x7feea41cd3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9c05640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee88008da0 0x7feea41c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40250/0 (socket says 172.21.15.98:40250) 2024-06-11T19:18:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9c05640 1 -- 172.21.15.98:0/1663799587 learned_addr learned my addr 172.21.15.98:0/1663799587 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9c05640 10 monclient(hunting): get_auth_request con 0x7fee88008da0 auth_method 0 2024-06-11T19:18:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9c05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9404640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee90004200 0x7feea4137e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9404640 10 monclient(hunting): get_auth_request con 0x7fee90004200 auth_method 0 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9404640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9404640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9c05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9c05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.893+0000 7feea9c05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9404640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9404640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9404640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9c05640 10 monclient(hunting): handle_auth_done global_id 65151 payload 824 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9c05640 1 -- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee90004e10 msgr2=0x7feea41cd3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9c05640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee90004e10 0x7feea41cd3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9c05640 1 -- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee90004200 msgr2=0x7feea4137e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9c05640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee90004200 0x7feea4137e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9c05640 10 monclient: _finish_hunting 0 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9c05640 1 monclient: found mon.noname-a 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9c05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9c05640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9c05640 1 -- 172.21.15.98:0/1663799587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea41ca910 con 0x7fee88008da0 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feea9c05640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee88008da0 0x7feea41c9eb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feea0019a50 tx=0x7feea0019f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 10 client.?.objecter ms_handle_connect 0x7fee88008da0 2024-06-11T19:18:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 1 -- 172.21.15.98:0/1663799587 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feea0027070 con 0x7fee88008da0 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 10 monclient: dump: 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 1 -- 172.21.15.98:0/1663799587 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea00044a0 con 0x7fee88008da0 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feeabe69640 5 monclient: authenticate success, global_id 65151 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 1 -- 172.21.15.98:0/1663799587 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feea0016d20 con 0x7fee88008da0 2024-06-11T19:18:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feeabe69640 10 monclient: _renew_subs 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feeabe69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feeabe69640 1 -- 172.21.15.98:0/1663799587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea41257c0 con 0x7fee88008da0 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9a7fc640 10 monclient: dump: 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7fee9bfff640 4 set_mon_vals no callback set 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feeabe69640 10 client.65151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feeabe69640 10 monclient: _renew_subs 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feeabe69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feeabe69640 1 -- 172.21.15.98:0/1663799587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feea40a44c0 con 0x7fee88008da0 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.894+0000 7feeabe69640 1 librados: init done 2024-06-11T19:18:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.895+0000 7fee9a7fc640 1 -- 172.21.15.98:0/1663799587 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7feea0004030 con 0x7fee88008da0 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.895+0000 7fee9a7fc640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fee88040eb0 0x7fee88043370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.895+0000 7fee9a7fc640 1 -- 172.21.15.98:0/1663799587 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(348..348 src has 1..348) ==== 6230+0+0 (secure 0 0 0) 0x7feea001f030 con 0x7fee88008da0 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.895+0000 7fee9a7fc640 10 client.65151.objecter ms_dispatch 0x7feea4000b90 osd_map(348..348 src has 1..348) 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.895+0000 7fee9a7fc640 3 client.65151.objecter handle_osd_map got epochs [348,348] > 0 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.895+0000 7fee9a7fc640 3 client.65151.objecter handle_osd_map decoding full epoch 348 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.896+0000 7fee9a7fc640 20 client.65151.objecter dump_active .. 0 homeless 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.896+0000 7feea9404640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fee88040eb0 0x7fee88043370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.896+0000 7feeabe69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.896+0000 7feeabe69640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.896+0000 7feeabe69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.896+0000 7feeabe69640 1 -- 172.21.15.98:0/1663799587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea40a5000 con 0x7fee88008da0 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.896+0000 7feea9404640 10 monclient: get_auth_request con 0x7fee88040eb0 auth_method 0 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.896+0000 7fee9a7fc640 10 client.65151.objecter ms_handle_connect 0x7fee88040eb0 2024-06-11T19:18:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.896+0000 7feea9404640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fee88040eb0 0x7fee88043370 secure :-1 s=READY pgs=4183 cs=0 l=1 rev1=1 crypto rx=0x7fee94038000 tx=0x7fee94033000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.901+0000 7fee9a7fc640 1 -- 172.21.15.98:0/1663799587 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7feea41ca910 con 0x7fee88008da0 2024-06-11T19:18:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.901+0000 7fee9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:12.901+0000 7fee9a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:18:13.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.164+0000 7feeabe69640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:18:13.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.164+0000 7feeabe69640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:18:13.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.164+0000 7feeabe69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:18:13.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.164+0000 7feeabe69640 1 -- 172.21.15.98:0/1663799587 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feea41234d0 con 0x7fee88008da0 2024-06-11T19:18:13.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.164+0000 7fee9a7fc640 1 -- 172.21.15.98:0/1663799587 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7feea0030070 con 0x7fee88008da0 2024-06-11T19:18:13.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.164+0000 7fee9a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:18:13.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.164+0000 7fee9a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:18:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":50,"quorum":[0,1,2,4,5,6,7,8],"quorum_names":["a","f","b","c","h","d","i","e"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:18:13.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.166+0000 7feeabe69640 10 librados: watch_flush enter 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 10 librados: watch_flush exit 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 20 client.65151.objecter shutdown clearing up homeless session... 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 10 client.65151.objecter successfully canceled tick 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 -- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fee88040eb0 msgr2=0x7fee88043370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fee88040eb0 0x7fee88043370 secure :-1 s=READY pgs=4183 cs=0 l=1 rev1=1 crypto rx=0x7fee94038000 tx=0x7fee94033000 comp rx=0 tx=0).stop 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 10 monclient: shutdown 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 -- 172.21.15.98:0/1663799587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee88008da0 msgr2=0x7feea41c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee88008da0 0x7feea41c9eb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feea0019a50 tx=0x7feea0019f10 comp rx=0 tx=0).stop 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 -- 172.21.15.98:0/1663799587 shutdown_connections 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fee88040eb0 0x7fee88043370 unknown :-1 s=CLOSED pgs=4183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee90004e10 0x7feea41cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee90004200 0x7feea4137e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 --2- 172.21.15.98:0/1663799587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee88008da0 0x7feea41c9eb0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 -- 172.21.15.98:0/1663799587 >> 172.21.15.98:0/1663799587 conn(0x7feea40ab850 msgr2=0x7feea41c0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:13.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 -- 172.21.15.98:0/1663799587 shutdown_connections 2024-06-11T19:18:13.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 -- 172.21.15.98:0/1663799587 wait complete. 2024-06-11T19:18:13.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.167+0000 7feeabe69640 1 librados: shutdown 2024-06-11T19:18:13.177 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-06-11T19:18:13.177 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-06-11T19:18:13.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:18:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.278+0000 7f6d0e384640 10 monclient: get_monmap_and_config 2024-06-11T19:18:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.278+0000 7f6d0e384640 10 monclient: build_initial_monmap 2024-06-11T19:18:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.278+0000 7f6d0e384640 10 monclient: monmap: 2024-06-11T19:18:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:18:13.279112+0000 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:18:13.279112+0000 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:13.278 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:13.279 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:18:13.279 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:18:13.279 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.278+0000 7f6d0e384640 1 Processor -- start 2024-06-11T19:18:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.278+0000 7f6d0e384640 1 -- start start 2024-06-11T19:18:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.278+0000 7f6d0e384640 10 monclient: init 2024-06-11T19:18:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 10 monclient: _add_conns ranks=[2,7,9,10,8,0,6,1,5,3,4] 2024-06-11T19:18:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d081347f0 0x7f6d0809e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 10 monclient(hunting): picked mon.noname-b con 0x7f6d081347f0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:18:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d08134bd0 0x7f6d080a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 10 monclient(hunting): picked mon.noname-i con 0x7f6d08134bd0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:18:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d08135010 0x7f6d080a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 10 monclient(hunting): picked mon.noname-a con 0x7f6d08135010 addr v2:172.21.15.98:3300/0 2024-06-11T19:18:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6d080687f0 con 0x7f6d08135010 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0809d410 con 0x7f6d081347f0 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f6d0809d6f0 con 0x7f6d08134bd0 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0e384640 10 monclient(hunting): authenticate will time out at 1303.086487s 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d081347f0 0x7f6d0809e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0c921640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d08135010 0x7f6d080a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d081347f0 0x7f6d0809e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45660/0 (socket says 172.21.15.98:45660) 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 1 -- 172.21.15.98:0/3775361962 learned_addr learned my addr 172.21.15.98:0/3775361962 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0c921640 10 monclient(hunting): get_auth_request con 0x7f6d08135010 auth_method 0 2024-06-11T19:18:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0c921640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0c921640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0c921640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 10 monclient(hunting): get_auth_request con 0x7f6d081347f0 auth_method 0 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0c921640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0c921640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0c921640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 10 monclient(hunting): handle_auth_done global_id 84494 payload 274 2024-06-11T19:18:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 1 -- 172.21.15.98:0/3775361962 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d08134bd0 msgr2=0x7f6d080a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:18:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d077fe640 1 --2- 172.21.15.98:0/3775361962 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d08134bd0 0x7f6d080a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 1 --2- 172.21.15.98:0/3775361962 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d08134bd0 0x7f6d080a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 1 -- 172.21.15.98:0/3775361962 >> v2:172.21.15.98:3300/0 conn(0x7f6d08135010 msgr2=0x7f6d080a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:18:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 1 --2- 172.21.15.98:0/3775361962 >> v2:172.21.15.98:3300/0 conn(0x7f6d08135010 0x7f6d080a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 10 monclient: _finish_hunting 0 2024-06-11T19:18:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 1 monclient: found mon.noname-b 2024-06-11T19:18:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:18:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d07fff640 1 -- 172.21.15.98:0/3775361962 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0809d9d0 con 0x7f6d081347f0 2024-06-11T19:18:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.279+0000 7f6d0c921640 1 --2- 172.21.15.98:0/3775361962 >> v2:172.21.15.98:3300/0 conn(0x7f6d08135010 0x7f6d080a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:18:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d07fff640 1 --2- 172.21.15.98:0/3775361962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d081347f0 0x7f6d0809e6b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009870 tx=0x7f6cf802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=979ae5f09672f012 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 1 -- 172.21.15.98:0/3775361962 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6cf8031a30 con 0x7f6d081347f0 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 10 monclient: dump: 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 10 monclient: _finish_auth 0 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d0e384640 5 monclient: authenticate success, global_id 84494 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 1 -- 172.21.15.98:0/3775361962 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf8037070 con 0x7f6d081347f0 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d0e384640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:18:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d0e384640 10 monclient: get_monmap_and_config success 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 1 -- 172.21.15.98:0/3775361962 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6cf8031c30 con 0x7f6d081347f0 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d0e384640 4 set_mon_vals no callback set 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.280+0000 7f6d06ffd640 10 monclient: dump: 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.295 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.281+0000 7f6d0e384640 10 monclient: shutdown 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.281+0000 7f6d0e384640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.281+0000 7f6d0e384640 1 -- 172.21.15.98:0/3775361962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d081347f0 msgr2=0x7f6d0809e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.281+0000 7f6d0e384640 1 --2- 172.21.15.98:0/3775361962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d081347f0 0x7f6d0809e6b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009870 tx=0x7f6cf802b860 comp rx=0 tx=0).stop 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.281+0000 7f6d0e384640 1 -- 172.21.15.98:0/3775361962 shutdown_connections 2024-06-11T19:18:13.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.281+0000 7f6d0e384640 1 --2- 172.21.15.98:0/3775361962 >> v2:172.21.15.98:3300/0 conn(0x7f6d08135010 0x7f6d080a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.281+0000 7f6d0e384640 1 --2- 172.21.15.98:0/3775361962 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d08134bd0 0x7f6d080a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.281+0000 7f6d0e384640 1 --2- 172.21.15.98:0/3775361962 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6d081347f0 0x7f6d0809e6b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.281+0000 7f6d0e384640 1 -- 172.21.15.98:0/3775361962 >> 172.21.15.98:0/3775361962 conn(0x7f6d080a4780 msgr2=0x7f6d080a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.281+0000 7f6d0e384640 1 -- 172.21.15.98:0/3775361962 shutdown_connections 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.281+0000 7f6d0e384640 1 -- 172.21.15.98:0/3775361962 wait complete. 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.281+0000 7f6d0e384640 10 monclient: build_initial_monmap 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 10 monclient: monmap: 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:13.297 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 1 librados: starting msgr at 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 1 librados: starting objecter 2024-06-11T19:18:13.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 1 Processor -- start 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 1 -- start start 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 1 librados: setting wanted keys 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 1 librados: calling monclient init 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 10 monclient: init 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 10 monclient: _add_conns ranks=[7,4,1,2,0,6,8,5,3] 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d081347f0 0x7f6d081cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 10 monclient(hunting): picked mon.noname-i con 0x7f6d081347f0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d08134bd0 0x7f6d081ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 10 monclient(hunting): picked mon.noname-c con 0x7f6d08134bd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d08135010 0x7f6d0811ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 10 monclient(hunting): picked mon.noname-f con 0x7f6d08135010 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6d08068c40 con 0x7f6d08135010 2024-06-11T19:18:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f6d08068ac0 con 0x7f6d08134bd0 2024-06-11T19:18:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f6d080a4110 con 0x7f6d081347f0 2024-06-11T19:18:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.282+0000 7f6d0e384640 10 monclient(hunting): authenticate will time out at 1303.089994s 2024-06-11T19:18:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d08134bd0 0x7f6d081ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d08134bd0 0x7f6d081ce450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:35254/0 (socket says 172.21.15.98:35254) 2024-06-11T19:18:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 1 -- 172.21.15.98:0/3554171804 learned_addr learned my addr 172.21.15.98:0/3554171804 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 10 monclient(hunting): get_auth_request con 0x7f6d08134bd0 auth_method 0 2024-06-11T19:18:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d07fff640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d081347f0 0x7f6d081cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d07fff640 10 monclient(hunting): get_auth_request con 0x7f6d081347f0 auth_method 0 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d07fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0c921640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d08135010 0x7f6d0811ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0c921640 10 monclient(hunting): get_auth_request con 0x7f6d08135010 auth_method 0 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0c921640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0c921640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0c921640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 10 monclient(hunting): handle_auth_done global_id 24052 payload 824 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 1 -- 172.21.15.98:0/3554171804 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d081347f0 msgr2=0x7f6d081cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d081347f0 0x7f6d081cdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 1 -- 172.21.15.98:0/3554171804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d08135010 msgr2=0x7f6d0811ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d08135010 0x7f6d0811ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 10 monclient: _finish_hunting 0 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 1 monclient: found mon.noname-c 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 1 -- 172.21.15.98:0/3554171804 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d08136160 con 0x7f6d08134bd0 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0c921640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d08135010 0x7f6d0811ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d077fe640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d08134bd0 0x7f6d081ce450 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00c820 tx=0x7f6cfc00cce0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 10 client.?.objecter ms_handle_connect 0x7f6d08134bd0 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6cfc015070 con 0x7f6d08134bd0 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 10 monclient: dump: 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 10 monclient: _finish_auth 0 2024-06-11T19:18:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cfc00ce10 con 0x7f6d08134bd0 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0e384640 5 monclient: authenticate success, global_id 24052 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0e384640 10 monclient: _renew_subs 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0e384640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0e384640 1 -- 172.21.15.98:0/3554171804 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d080a36a0 con 0x7f6d08134bd0 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6cfc007c40 con 0x7f6d08134bd0 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d04ff9640 10 monclient: dump: 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0e384640 10 client.24052.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0e384640 10 monclient: _renew_subs 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0e384640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0e384640 1 -- 172.21.15.98:0/3554171804 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d08122ba0 con 0x7f6d08134bd0 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d0e384640 1 librados: init done 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.283+0000 7f6d067fc640 4 set_mon_vals no callback set 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d0e384640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d0e384640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d08123570 0x7f6d0811f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d0e384640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d0e384640 1 -- 172.21.15.98:0/3554171804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d08050830 con 0x7f6d08123570 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f6cfc0077d0 con 0x7f6d08134bd0 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d07fff640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d08123570 0x7f6d0811f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d07fff640 10 monclient: get_auth_request con 0x7f6d08123570 auth_method 0 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d07fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d04ff9640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6cf403fdb0 0x7f6cf4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(348..348 src has 1..348) ==== 6230+0+0 (secure 0 0 0) 0x7f6cfc0514b0 con 0x7f6d08134bd0 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d04ff9640 10 client.24052.objecter ms_dispatch 0x7f6d08000b90 osd_map(348..348 src has 1..348) 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d04ff9640 3 client.24052.objecter handle_osd_map got epochs [348,348] > 0 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d04ff9640 3 client.24052.objecter handle_osd_map decoding full epoch 348 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d07fff640 10 monclient(hunting): handle_auth_done global_id 65169 payload 824 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d07fff640 1 -- 172.21.15.98:0/3554171804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d0809e6b0 con 0x7f6d08123570 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d04ff9640 20 client.24052.objecter dump_active .. 0 homeless 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.284+0000 7f6d0c921640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6cf403fdb0 0x7f6cf4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d0c921640 10 monclient: get_auth_request con 0x7f6cf403fdb0 auth_method 0 2024-06-11T19:18:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 10 client.24052.objecter ms_handle_connect 0x7f6d08123570 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 10 client.24052.objecter resend_mon_ops 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d07fff640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d08123570 0x7f6d0811f120 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6d080a8d40 tx=0x7f6cf8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6cf804a070 con 0x7f6d08123570 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 10 monclient: dump: 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 10 client.24052.objecter ms_handle_connect 0x7f6cf403fdb0 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d0c921640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6cf403fdb0 0x7f6cf4042270 secure :-1 s=READY pgs=4184 cs=0 l=1 rev1=1 crypto rx=0x7f6d08120340 tx=0x7f6d0000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6cf8037040 con 0x7f6d08123570 2024-06-11T19:18:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:18:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d08123570 msgr2=0x7f6d0811f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d08123570 0x7f6d0811f120 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6d080a8d40 tx=0x7f6cf8004730 comp rx=0 tx=0).stop 2024-06-11T19:18:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d08123570 msgr2=0x7f6d0811f120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.285+0000 7f6d04ff9640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d08123570 0x7f6d0811f120 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.291+0000 7f6d0e384640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:18:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.291+0000 7f6d0e384640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d0811f660 0x7f6d0811fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.291+0000 7f6d0e384640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.291+0000 7f6d0e384640 1 -- 172.21.15.98:0/3554171804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0811ffb0 con 0x7f6d0811f660 2024-06-11T19:18:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.291+0000 7f6d07fff640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d0811f660 0x7f6d0811fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.291+0000 7f6d07fff640 10 monclient: get_auth_request con 0x7f6d0811f660 auth_method 0 2024-06-11T19:18:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.291+0000 7f6d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.291+0000 7f6d07fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.291+0000 7f6d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.291+0000 7f6d07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.291+0000 7f6d07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d07fff640 10 monclient(hunting): handle_auth_done global_id 65178 payload 824 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d07fff640 1 -- 172.21.15.98:0/3554171804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6d0806d540 con 0x7f6d0811f660 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 10 client.24052.objecter ms_handle_connect 0x7f6d0811f660 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 10 client.24052.objecter resend_mon_ops 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d07fff640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d0811f660 0x7f6d0811fa70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80097c0 tx=0x7f6cf8031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6cf804a040 con 0x7f6d0811f660 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 10 monclient: dump: 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9262 (secure 0 0 0) 0x7f6cf8031c40 con 0x7f6d0811f660 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:18:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:18:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d0811f660 msgr2=0x7f6d0811fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d0811f660 0x7f6d0811fa70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80097c0 tx=0x7f6cf8031000 comp rx=0 tx=0).stop 2024-06-11T19:18:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 1 -- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d0811f660 msgr2=0x7f6d0811fa70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d04ff9640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d0811f660 0x7f6d0811fa70 secure :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80097c0 tx=0x7f6cf8031000 comp rx=0 tx=0).stop 2024-06-11T19:18:13.316 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:18:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:18:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:18:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:18:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 50, 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 285119, 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:18:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:18:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:18:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:18:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:18:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:18:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:18:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:18:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:18:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:18:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:18:13.339 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:18:13.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d0e384640 10 librados: watch_flush enter 2024-06-11T19:18:13.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d0e384640 10 librados: watch_flush exit 2024-06-11T19:18:13.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d0e384640 20 client.24052.objecter shutdown clearing up homeless session... 2024-06-11T19:18:13.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.292+0000 7f6d0e384640 10 client.24052.objecter successfully canceled tick 2024-06-11T19:18:13.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0e384640 1 -- 172.21.15.98:0/3554171804 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6cf403fdb0 msgr2=0x7f6cf4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0e384640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6cf403fdb0 0x7f6cf4042270 secure :-1 s=READY pgs=4184 cs=0 l=1 rev1=1 crypto rx=0x7f6d08120340 tx=0x7f6d0000b040 comp rx=0 tx=0).stop 2024-06-11T19:18:13.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0e384640 10 monclient: shutdown 2024-06-11T19:18:13.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0e384640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:13.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0e384640 1 -- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d08134bd0 msgr2=0x7f6d081ce450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0e384640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d08134bd0 0x7f6d081ce450 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00c820 tx=0x7f6cfc00cce0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0c921640 1 -- 172.21.15.98:0/3554171804 reap_dead start 2024-06-11T19:18:13.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0e384640 1 -- 172.21.15.98:0/3554171804 shutdown_connections 2024-06-11T19:18:13.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0e384640 1 --2- 172.21.15.98:0/3554171804 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d08134bd0 0x7f6d081ce450 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0e384640 1 -- 172.21.15.98:0/3554171804 >> 172.21.15.98:0/3554171804 conn(0x7f6d080a4780 msgr2=0x7f6d081bf710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:13.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0e384640 1 -- 172.21.15.98:0/3554171804 shutdown_connections 2024-06-11T19:18:13.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0e384640 1 -- 172.21.15.98:0/3554171804 wait complete. 2024-06-11T19:18:13.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.293+0000 7f6d0e384640 1 librados: shutdown 2024-06-11T19:18:13.342 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:18:13.342 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-06-11T19:18:13.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:18:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.442+0000 7f2d74292640 10 monclient: get_monmap_and_config 2024-06-11T19:18:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.442+0000 7f2d74292640 10 monclient: build_initial_monmap 2024-06-11T19:18:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.442+0000 7f2d74292640 10 monclient: monmap: 2024-06-11T19:18:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:18:13.443430+0000 2024-06-11T19:18:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:18:13.443430+0000 2024-06-11T19:18:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:18:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:18:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:18:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:18:13.444 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.442+0000 7f2d74292640 1 Processor -- start 2024-06-11T19:18:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 1 -- start start 2024-06-11T19:18:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 10 monclient: init 2024-06-11T19:18:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 10 monclient: _add_conns ranks=[2,1,4,5,6,10,0,9,3,8,7] 2024-06-11T19:18:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a4920 0x7f2d6c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 10 monclient(hunting): picked mon.noname-b con 0x7f2d6c0a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:18:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f2d6c0a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:18:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2d6c0a2530 0x7f2d6c0a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 10 monclient(hunting): picked mon.noname-c con 0x7f2d6c0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d6c0a7140 con 0x7f2d6c0a5240 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6c0a2b50 con 0x7f2d6c0a4920 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2d6c0a2e30 con 0x7f2d6c0a2530 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d74292640 10 monclient(hunting): authenticate will time out at 1303.250707s 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7202e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a4920 0x7f2d6c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7282f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2d6c0a2530 0x7f2d6c0a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7202e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a4920 0x7f2d6c0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45664/0 (socket says 172.21.15.98:45664) 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7202e640 1 -- 172.21.15.98:0/2354174625 learned_addr learned my addr 172.21.15.98:0/2354174625 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7202e640 10 monclient(hunting): get_auth_request con 0x7f2d6c0a4920 auth_method 0 2024-06-11T19:18:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7202e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7202e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7202e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7282f640 10 monclient(hunting): get_auth_request con 0x7f2d6c0a2530 auth_method 0 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7282f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7282f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7282f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7202e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7202e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7202e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7282f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7282f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.443+0000 7f2d7282f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.444+0000 7f2d7282f640 10 monclient(hunting): handle_auth_done global_id 24061 payload 274 2024-06-11T19:18:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.444+0000 7f2d7282f640 1 -- 172.21.15.98:0/2354174625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a4920 msgr2=0x7f2d6c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.444+0000 7f2d7282f640 1 --2- 172.21.15.98:0/2354174625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a4920 0x7f2d6c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.444+0000 7f2d7282f640 1 -- 172.21.15.98:0/2354174625 >> v1:172.21.15.117:6789/0 conn(0x7f2d6c0a5240 legacy=0x7f2d6c0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.444+0000 7f2d7282f640 10 monclient: _finish_hunting 0 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.444+0000 7f2d7282f640 1 monclient: found mon.noname-c 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.444+0000 7f2d7282f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.444+0000 7f2d7282f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.444+0000 7f2d7282f640 1 -- 172.21.15.98:0/2354174625 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d6c0a3110 con 0x7f2d6c0a2530 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.444+0000 7f2d7202e640 1 --2- 172.21.15.98:0/2354174625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a4920 0x7f2d6c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.444+0000 7f2d7282f640 1 --2- 172.21.15.98:0/2354174625 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2d6c0a2530 0x7f2d6c0a60b0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009dc0 tx=0x7f2d6402bd50 comp rx=0 tx=0).ready entity=mon.4 client_cookie=aa1edac676e18c92 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 1 -- 172.21.15.98:0/2354174625 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2d64039070 con 0x7f2d6c0a2530 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:18:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 10 monclient: dump: 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 10 monclient: _finish_auth 0 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 10 monclient: _check_auth_tickets 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 1 -- 172.21.15.98:0/2354174625 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d6402cd70 con 0x7f2d6c0a2530 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 5 monclient: authenticate success, global_id 24061 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 10 monclient: get_monmap_and_config success 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 1 -- 172.21.15.98:0/2354174625 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2d640327b0 con 0x7f2d6c0a2530 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 4 set_mon_vals no callback set 2024-06-11T19:18:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d7102c640 10 monclient: dump: 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 10 monclient: shutdown 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 1 -- 172.21.15.98:0/2354174625 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2d6c0a2530 msgr2=0x7f2d6c0a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 1 --2- 172.21.15.98:0/2354174625 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2d6c0a2530 0x7f2d6c0a60b0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009dc0 tx=0x7f2d6402bd50 comp rx=0 tx=0).stop 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 1 -- 172.21.15.98:0/2354174625 shutdown_connections 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 1 --2- 172.21.15.98:0/2354174625 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2d6c0a2530 0x7f2d6c0a60b0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 1 --2- 172.21.15.98:0/2354174625 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a4920 0x7f2d6c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 1 -- 172.21.15.98:0/2354174625 >> 172.21.15.98:0/2354174625 conn(0x7f2d6c0ab850 msgr2=0x7f2d6c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 1 -- 172.21.15.98:0/2354174625 shutdown_connections 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.445+0000 7f2d74292640 1 -- 172.21.15.98:0/2354174625 wait complete. 2024-06-11T19:18:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient: build_initial_monmap 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient: monmap: 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 1 librados: starting msgr at 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 1 librados: starting objecter 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 1 Processor -- start 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 1 -- start start 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 1 librados: setting wanted keys 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 1 librados: calling monclient init 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient: init 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient: _add_conns ranks=[2,3,0,8,7,4,6,5,1] 2024-06-11T19:18:13.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a2530 0x7f2d6c1374e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient(hunting): picked mon.noname-b con 0x7f2d6c0a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d6c0a4920 0x7f2d6c137a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient(hunting): picked mon.noname-g con 0x7f2d6c0a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d6c0a5240 0x7f2d6c122f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient(hunting): picked mon.noname-a con 0x7f2d6c0a5240 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c0a6da0 con 0x7f2d6c0a5240 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6c138e50 con 0x7f2d6c0a2530 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6c0a65f0 con 0x7f2d6c0a4920 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:13.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.446+0000 7f2d74292640 10 monclient(hunting): authenticate will time out at 1303.253995s 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7182d640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d6c0a4920 msgr2=0x7f2d6c137a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a2530 0x7f2d6c1374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7182d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d6c0a4920 0x7f2d6c137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a2530 0x7f2d6c1374e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45674/0 (socket says 172.21.15.98:45674) 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 1 -- 172.21.15.98:0/3254051660 learned_addr learned my addr 172.21.15.98:0/3254051660 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 10 monclient(hunting): get_auth_request con 0x7f2d6c0a2530 auth_method 0 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7282f640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d6c0a5240 0x7f2d6c122f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7282f640 10 monclient(hunting): get_auth_request con 0x7f2d6c0a5240 auth_method 0 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7282f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7282f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7282f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7282f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7282f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7282f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 10 monclient(hunting): handle_auth_done global_id 84512 payload 824 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 1 -- 172.21.15.98:0/3254051660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d6c0a4920 msgr2=0x7f2d6c137a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d6c0a4920 0x7f2d6c137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 1 -- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d6c0a5240 msgr2=0x7f2d6c122f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d6c0a5240 0x7f2d6c122f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 10 monclient: _finish_hunting 0 2024-06-11T19:18:13.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 1 monclient: found mon.noname-b 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 1 -- 172.21.15.98:0/3254051660 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d6c124bd0 con 0x7f2d6c0a2530 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7282f640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d6c0a5240 0x7f2d6c122f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7202e640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a2530 0x7f2d6c1374e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000e9d0 tx=0x7f2d6000ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 10 client.?.objecter ms_handle_connect 0x7f2d6c0a2530 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2d60017070 con 0x7f2d6c0a2530 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 10 monclient: dump: 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 10 monclient: _finish_auth 0 2024-06-11T19:18:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 10 monclient: _check_auth_tickets 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d60013470 con 0x7f2d6c0a2530 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d74292640 5 monclient: authenticate success, global_id 84512 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d74292640 10 monclient: _renew_subs 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2d60013a90 con 0x7f2d6c0a2530 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d74292640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d74292640 1 -- 172.21.15.98:0/3254051660 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d6c126ec0 con 0x7f2d6c0a2530 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d7082b640 4 set_mon_vals no callback set 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d5effd640 10 monclient: dump: 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.470 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d74292640 10 client.84512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d74292640 10 monclient: _renew_subs 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d74292640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d74292640 1 -- 172.21.15.98:0/3254051660 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d6c0a91a0 con 0x7f2d6c0a2530 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.447+0000 7f2d74292640 1 librados: init done 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d74292640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d74292640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123480 0x7f2d6c123860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d74292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d74292640 1 -- 172.21.15.98:0/3254051660 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6c050830 con 0x7f2d6c123480 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d7182d640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123480 0x7f2d6c123860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d7182d640 10 monclient: get_auth_request con 0x7f2d6c123480 auth_method 0 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d7182d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d7182d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d7182d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f2d6001a070 con 0x7f2d6c0a2530 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d7182d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d7182d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d7182d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d7182d640 10 monclient(hunting): handle_auth_done global_id 84521 payload 824 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d7182d640 1 -- 172.21.15.98:0/3254051660 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d6c0a4d00 con 0x7f2d6c123480 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.448+0000 7f2d5effd640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2d5003fdb0 0x7f2d50042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(348..348 src has 1..348) ==== 6230+0+0 (secure 0 0 0) 0x7f2d6000f040 con 0x7f2d6c0a2530 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 10 client.84512.objecter ms_dispatch 0x7f2d6c000b90 osd_map(348..348 src has 1..348) 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 3 client.84512.objecter handle_osd_map got epochs [348,348] > 0 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 3 client.84512.objecter handle_osd_map decoding full epoch 348 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d7182d640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123480 0x7f2d6c123860 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c0a5620 tx=0x7f2d6800c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 20 client.84512.objecter dump_active .. 0 homeless 2024-06-11T19:18:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 10 client.84512.objecter ms_handle_connect 0x7f2d6c123480 2024-06-11T19:18:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 10 client.84512.objecter resend_mon_ops 2024-06-11T19:18:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2d680057c0 con 0x7f2d6c123480 2024-06-11T19:18:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:18:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 10 monclient: dump: 2024-06-11T19:18:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.474 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d7282f640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2d5003fdb0 0x7f2d50042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d7282f640 10 monclient: get_auth_request con 0x7f2d5003fdb0 auth_method 0 2024-06-11T19:18:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2d680072e0 con 0x7f2d6c123480 2024-06-11T19:18:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:18:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123480 msgr2=0x7f2d6c123860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123480 0x7f2d6c123860 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c0a5620 tx=0x7f2d6800c7a0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123480 msgr2=0x7f2d6c123860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123480 0x7f2d6c123860 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d5effd640 10 client.84512.objecter ms_handle_connect 0x7f2d5003fdb0 2024-06-11T19:18:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.449+0000 7f2d7282f640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2d5003fdb0 0x7f2d50042270 secure :-1 s=READY pgs=4185 cs=0 l=1 rev1=1 crypto rx=0x7f2d64036000 tx=0x7f2d64030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.455+0000 7f2d74292640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:18:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.455+0000 7f2d74292640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123da0 0x7f2d6c09e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.455+0000 7f2d74292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.455+0000 7f2d74292640 1 -- 172.21.15.98:0/3254051660 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6c125480 con 0x7f2d6c123da0 2024-06-11T19:18:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d7182d640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123da0 0x7f2d6c09e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d7182d640 10 monclient: get_auth_request con 0x7f2d6c123da0 auth_method 0 2024-06-11T19:18:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d7182d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d7182d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d7182d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d7182d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d7182d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d7182d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d7182d640 10 monclient(hunting): handle_auth_done global_id 84530 payload 824 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d7182d640 1 -- 172.21.15.98:0/3254051660 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d6c06c300 con 0x7f2d6c123da0 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d7182d640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123da0 0x7f2d6c09e9f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2d68000fb0 tx=0x7f2d68000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 10 client.84512.objecter ms_handle_connect 0x7f2d6c123da0 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 10 client.84512.objecter resend_mon_ops 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2d68012400 con 0x7f2d6c123da0 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:18:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 10 monclient: dump: 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.480 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9262 (secure 0 0 0) 0x7f2d68012400 con 0x7f2d6c123da0 2024-06-11T19:18:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:18:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:18:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123da0 msgr2=0x7f2d6c09e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123da0 0x7f2d6c09e9f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2d68000fb0 tx=0x7f2d68000ed0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 1 -- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123da0 msgr2=0x7f2d6c09e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d5effd640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c123da0 0x7f2d6c09e9f0 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2d68000fb0 tx=0x7f2d68000ed0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.480 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:18:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:18:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 50, 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 165230, 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:18:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:18:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.491 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.491 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:18:13.491 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:18:13.491 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.491 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.491 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.491 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:18:13.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:18:13.491 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.494 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:18:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:18:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:18:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:18:13.502 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:18:13.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.456+0000 7f2d74292640 10 librados: watch_flush enter 2024-06-11T19:18:13.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 10 librados: watch_flush exit 2024-06-11T19:18:13.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 20 client.84512.objecter shutdown clearing up homeless session... 2024-06-11T19:18:13.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 10 client.84512.objecter successfully canceled tick 2024-06-11T19:18:13.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 1 -- 172.21.15.98:0/3254051660 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2d5003fdb0 msgr2=0x7f2d50042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2d5003fdb0 0x7f2d50042270 secure :-1 s=READY pgs=4185 cs=0 l=1 rev1=1 crypto rx=0x7f2d64036000 tx=0x7f2d64030000 comp rx=0 tx=0).stop 2024-06-11T19:18:13.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 10 monclient: shutdown 2024-06-11T19:18:13.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:13.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 1 -- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a2530 msgr2=0x7f2d6c1374e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 1 --2- 172.21.15.98:0/3254051660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2d6c0a2530 0x7f2d6c1374e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000e9d0 tx=0x7f2d6000ee90 comp rx=0 tx=0).stop 2024-06-11T19:18:13.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d7282f640 1 -- 172.21.15.98:0/3254051660 reap_dead start 2024-06-11T19:18:13.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d7282f640 1 -- 172.21.15.98:0/3254051660 reap_dead start 2024-06-11T19:18:13.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 1 -- 172.21.15.98:0/3254051660 shutdown_connections 2024-06-11T19:18:13.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 1 -- 172.21.15.98:0/3254051660 >> 172.21.15.98:0/3254051660 conn(0x7f2d6c0ab850 msgr2=0x7f2d6c1bff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:13.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 1 -- 172.21.15.98:0/3254051660 shutdown_connections 2024-06-11T19:18:13.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 1 -- 172.21.15.98:0/3254051660 wait complete. 2024-06-11T19:18:13.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.457+0000 7f2d74292640 1 librados: shutdown 2024-06-11T19:18:13.505 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:18:13.505 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-06-11T19:18:13.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:18:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.605+0000 7fb598d0b640 10 monclient: get_monmap_and_config 2024-06-11T19:18:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.605+0000 7fb598d0b640 10 monclient: build_initial_monmap 2024-06-11T19:18:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.605+0000 7fb598d0b640 10 monclient: monmap: 2024-06-11T19:18:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:18:13.606724+0000 2024-06-11T19:18:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:18:13.606724+0000 2024-06-11T19:18:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.606 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:18:13.606 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:18:13.606 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:13.606 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:13.606 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:13.606 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:13.606 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:13.606 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:13.606 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:13.606 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:18:13.606 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:18:13.607 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 1 Processor -- start 2024-06-11T19:18:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 1 -- start start 2024-06-11T19:18:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 10 monclient: init 2024-06-11T19:18:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 10 monclient: _add_conns ranks=[6,2,10,7,5,0,4,3,1,9,8] 2024-06-11T19:18:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5940a4920 0x7fb5940a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 10 monclient(hunting): picked mon.noname-d con 0x7fb5940a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a5240 0x7fb5940a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 10 monclient(hunting): picked mon.noname-b con 0x7fb5940a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb5940a2530 0x7fb5940a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 10 monclient(hunting): picked mon.noname-f con 0x7fb5940a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fb5940a7140 con 0x7fb5940a2530 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb5940a2b50 con 0x7fb5940a5240 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb5940a2e30 con 0x7fb5940a4920 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.606+0000 7fb598d0b640 10 monclient(hunting): authenticate will time out at 1303.413942s 2024-06-11T19:18:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a5240 0x7fb5940a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb59259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5940a4920 0x7fb5940a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a5240 0x7fb5940a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45692/0 (socket says 172.21.15.98:45692) 2024-06-11T19:18:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 1 -- 172.21.15.98:0/34560763 learned_addr learned my addr 172.21.15.98:0/34560763 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 10 monclient(hunting): get_auth_request con 0x7fb5940a5240 auth_method 0 2024-06-11T19:18:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb59259c640 10 monclient(hunting): get_auth_request con 0x7fb5940a4920 auth_method 0 2024-06-11T19:18:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb59259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb59259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb59259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb59259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb59259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb59259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb592d9d640 1 --2- 172.21.15.98:0/34560763 >> v2:172.21.15.117:3300/0 conn(0x7fb5940a2530 0x7fb5940a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb592d9d640 10 monclient(hunting): get_auth_request con 0x7fb5940a2530 auth_method 0 2024-06-11T19:18:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb592d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb592d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb592d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 10 monclient(hunting): handle_auth_done global_id 84539 payload 274 2024-06-11T19:18:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 1 -- 172.21.15.98:0/34560763 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5940a4920 msgr2=0x7fb5940a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:18:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 1 --2- 172.21.15.98:0/34560763 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5940a4920 0x7fb5940a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 1 -- 172.21.15.98:0/34560763 >> v2:172.21.15.117:3300/0 conn(0x7fb5940a2530 msgr2=0x7fb5940a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:18:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 1 --2- 172.21.15.98:0/34560763 >> v2:172.21.15.117:3300/0 conn(0x7fb5940a2530 0x7fb5940a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:18:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 1 monclient: found mon.noname-b 2024-06-11T19:18:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:18:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb58bfff640 1 -- 172.21.15.98:0/34560763 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5940a3110 con 0x7fb5940a5240 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.607+0000 7fb59259c640 1 --2- 172.21.15.98:0/34560763 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5940a4920 0x7fb5940a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb58bfff640 1 --2- 172.21.15.98:0/34560763 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a5240 0x7fb5940a59a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009870 tx=0x7fb57c02b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8c741be403ff53c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 1 -- 172.21.15.98:0/34560763 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb57c038070 con 0x7fb5940a5240 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 10 monclient: dump: 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.622 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 10 monclient: _finish_auth 0 2024-06-11T19:18:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:18:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 1 -- 172.21.15.98:0/34560763 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57c031990 con 0x7fb5940a5240 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 5 monclient: authenticate success, global_id 84539 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 10 monclient: get_monmap_and_config success 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 1 -- 172.21.15.98:0/34560763 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb57c0323b0 con 0x7fb5940a5240 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 4 set_mon_vals no callback set 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb591d9b640 10 monclient: dump: 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.624 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 10 monclient: shutdown 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 1 -- 172.21.15.98:0/34560763 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a5240 msgr2=0x7fb5940a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 1 --2- 172.21.15.98:0/34560763 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a5240 0x7fb5940a59a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009870 tx=0x7fb57c02b7c0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 1 -- 172.21.15.98:0/34560763 shutdown_connections 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 1 --2- 172.21.15.98:0/34560763 >> v2:172.21.15.117:3300/0 conn(0x7fb5940a2530 0x7fb5940a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 1 --2- 172.21.15.98:0/34560763 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a5240 0x7fb5940a59a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 1 --2- 172.21.15.98:0/34560763 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5940a4920 0x7fb5940a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 1 -- 172.21.15.98:0/34560763 >> 172.21.15.98:0/34560763 conn(0x7fb5940ab850 msgr2=0x7fb5940a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 1 -- 172.21.15.98:0/34560763 shutdown_connections 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.608+0000 7fb598d0b640 1 -- 172.21.15.98:0/34560763 wait complete. 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.609+0000 7fb598d0b640 10 monclient: build_initial_monmap 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.609+0000 7fb598d0b640 10 monclient: monmap: 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:13.627 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.609+0000 7fb598d0b640 1 librados: starting msgr at 2024-06-11T19:18:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.609+0000 7fb598d0b640 1 librados: starting objecter 2024-06-11T19:18:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.609+0000 7fb598d0b640 1 Processor -- start 2024-06-11T19:18:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.609+0000 7fb598d0b640 1 -- start start 2024-06-11T19:18:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.609+0000 7fb598d0b640 1 librados: setting wanted keys 2024-06-11T19:18:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.609+0000 7fb598d0b640 1 librados: calling monclient init 2024-06-11T19:18:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.609+0000 7fb598d0b640 10 monclient: init 2024-06-11T19:18:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 10 monclient: _add_conns ranks=[2,5,7,4,3,1,0,6,8] 2024-06-11T19:18:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a2530 0x7fb594137760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 10 monclient(hunting): picked mon.noname-b con 0x7fb5940a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb5940a4920 0x7fb594137ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 10 monclient(hunting): picked mon.noname-h con 0x7fb5940a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb5940a5240 0x7fb5941232a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 10 monclient(hunting): picked mon.noname-i con 0x7fb5940a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb5940a6da0 con 0x7fb5940a2530 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fb594138e50 con 0x7fb5940a4920 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fb5940a6420 con 0x7fb5940a5240 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb598d0b640 10 monclient(hunting): authenticate will time out at 1303.417461s 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a2530 0x7fb594137760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a2530 0x7fb594137760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45708/0 (socket says 172.21.15.98:45708) 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 1 -- 172.21.15.98:0/2041635342 learned_addr learned my addr 172.21.15.98:0/2041635342 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 10 monclient(hunting): get_auth_request con 0x7fb5940a2530 auth_method 0 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 10 monclient(hunting): handle_auth_done global_id 84548 payload 824 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 1 -- 172.21.15.98:0/2041635342 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb5940a5240 msgr2=0x7fb5941232a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb5940a5240 0x7fb5941232a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 1 -- 172.21.15.98:0/2041635342 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb5940a4920 msgr2=0x7fb594137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb5940a4920 0x7fb594137ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 10 monclient: _finish_hunting 0 2024-06-11T19:18:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 1 monclient: found mon.noname-b 2024-06-11T19:18:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:18:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 1 -- 172.21.15.98:0/2041635342 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5941c0d10 con 0x7fb5940a2530 2024-06-11T19:18:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb59259c640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a2530 0x7fb594137760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb58400ebe0 tx=0x7fb58400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb58affd640 10 client.?.objecter ms_handle_connect 0x7fb5940a2530 2024-06-11T19:18:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.610+0000 7fb58affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:18:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb584017070 con 0x7fb5940a2530 2024-06-11T19:18:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 10 monclient: dump: 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.632 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 10 monclient: _finish_auth 0 2024-06-11T19:18:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 10 monclient: _check_auth_tickets 2024-06-11T19:18:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb598d0b640 5 monclient: authenticate success, global_id 84548 2024-06-11T19:18:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb58400cc30 con 0x7fb5940a2530 2024-06-11T19:18:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb58400cde0 con 0x7fb5940a2530 2024-06-11T19:18:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb598d0b640 10 monclient: _renew_subs 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb598d0b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb598d0b640 1 -- 172.21.15.98:0/2041635342 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5940a9160 con 0x7fb5940a2530 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb59159a640 4 set_mon_vals no callback set 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb58affd640 10 monclient: dump: 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb598d0b640 10 client.84548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb598d0b640 10 monclient: _renew_subs 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb598d0b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:18:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb598d0b640 1 -- 172.21.15.98:0/2041635342 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5941259b0 con 0x7fb5940a2530 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.611+0000 7fb598d0b640 1 librados: init done 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb598d0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fb58401a070 con 0x7fb5940a2530 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb598d0b640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5941237e0 0x7fb594123bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb598d0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb598d0b640 1 -- 172.21.15.98:0/2041635342 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb594050830 con 0x7fb5941237e0 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58bfff640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5941237e0 0x7fb594123bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58bfff640 10 monclient: get_auth_request con 0x7fb5941237e0 auth_method 0 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb57403fde0 0x7fb5740422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(348..348 src has 1..348) ==== 6230+0+0 (secure 0 0 0) 0x7fb58400f040 con 0x7fb5940a2530 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 10 client.84548.objecter ms_dispatch 0x7fb594000b90 osd_map(348..348 src has 1..348) 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 3 client.84548.objecter handle_osd_map got epochs [348,348] > 0 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 3 client.84548.objecter handle_osd_map decoding full epoch 348 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 20 client.84548.objecter dump_active .. 0 homeless 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58bfff640 10 monclient(hunting): handle_auth_done global_id 24070 payload 824 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb592d9d640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb57403fde0 0x7fb5740422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58bfff640 1 -- 172.21.15.98:0/2041635342 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5940a6750 con 0x7fb5941237e0 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb592d9d640 10 monclient: get_auth_request con 0x7fb57403fde0 auth_method 0 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 10 client.84548.objecter ms_handle_connect 0x7fb5941237e0 2024-06-11T19:18:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 10 client.84548.objecter resend_mon_ops 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58bfff640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5941237e0 0x7fb594123bc0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fb5940abf80 tx=0x7fb57c003b40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb57c038050 con 0x7fb5941237e0 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 10 monclient: dump: 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb58affd640 10 client.84548.objecter ms_handle_connect 0x7fb57403fde0 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.612+0000 7fb592d9d640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb57403fde0 0x7fb5740422a0 secure :-1 s=READY pgs=4186 cs=0 l=1 rev1=1 crypto rx=0x7fb58c0070b0 tx=0x7fb58c007040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.613+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb57c039040 con 0x7fb5941237e0 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.613+0000 7fb58affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.613+0000 7fb58affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:18:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.613+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5941237e0 msgr2=0x7fb594123bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.613+0000 7fb58affd640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5941237e0 0x7fb594123bc0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fb5940abf80 tx=0x7fb57c003b40 comp rx=0 tx=0).stop 2024-06-11T19:18:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.613+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5941237e0 msgr2=0x7fb594123bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.613+0000 7fb58affd640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5941237e0 0x7fb594123bc0 secure :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fb5940abf80 tx=0x7fb57c003b40 comp rx=0 tx=0).stop 2024-06-11T19:18:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb598d0b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:18:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb598d0b640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb594124100 0x7fb59409e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb598d0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb598d0b640 1 -- 172.21.15.98:0/2041635342 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb59406c300 con 0x7fb594124100 2024-06-11T19:18:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb58bfff640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb594124100 0x7fb59409e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb58bfff640 10 monclient: get_auth_request con 0x7fb594124100 auth_method 0 2024-06-11T19:18:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb58bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb58bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb58bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb58bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb58bfff640 10 monclient(hunting): handle_auth_done global_id 24079 payload 824 2024-06-11T19:18:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.619+0000 7fb58bfff640 1 -- 172.21.15.98:0/2041635342 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb5941256c0 con 0x7fb594124100 2024-06-11T19:18:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 10 client.84548.objecter ms_handle_connect 0x7fb594124100 2024-06-11T19:18:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 10 client.84548.objecter resend_mon_ops 2024-06-11T19:18:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58bfff640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb594124100 0x7fb59409e9f0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fb57c02b800 tx=0x7fb57c008030 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb57c031ba0 con 0x7fb594124100 2024-06-11T19:18:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:18:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 10 monclient: dump: 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.644 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fb57c031730 con 0x7fb594124100 2024-06-11T19:18:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:18:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:18:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb594124100 msgr2=0x7fb59409e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb594124100 0x7fb59409e9f0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fb57c02b800 tx=0x7fb57c008030 comp rx=0 tx=0).stop 2024-06-11T19:18:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 1 -- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb594124100 msgr2=0x7fb59409e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb58affd640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb594124100 0x7fb59409e9f0 secure :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fb57c02b800 tx=0x7fb57c008030 comp rx=0 tx=0).stop 2024-06-11T19:18:13.644 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 50, 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 543838, 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:18:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:18:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:18:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:18:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:18:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:18:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:18:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:18:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:18:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:18:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:18:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:18:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:18:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:18:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:18:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:18:13.667 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:18:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 10 librados: watch_flush enter 2024-06-11T19:18:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 10 librados: watch_flush exit 2024-06-11T19:18:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 20 client.84548.objecter shutdown clearing up homeless session... 2024-06-11T19:18:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 10 client.84548.objecter successfully canceled tick 2024-06-11T19:18:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 1 -- 172.21.15.98:0/2041635342 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb57403fde0 msgr2=0x7fb5740422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb57403fde0 0x7fb5740422a0 secure :-1 s=READY pgs=4186 cs=0 l=1 rev1=1 crypto rx=0x7fb58c0070b0 tx=0x7fb58c007040 comp rx=0 tx=0).stop 2024-06-11T19:18:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 10 monclient: shutdown 2024-06-11T19:18:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 1 -- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a2530 msgr2=0x7fb594137760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 1 --2- 172.21.15.98:0/2041635342 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5940a2530 0x7fb594137760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb58400ebe0 tx=0x7fb58400c520 comp rx=0 tx=0).stop 2024-06-11T19:18:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb592d9d640 1 -- 172.21.15.98:0/2041635342 reap_dead start 2024-06-11T19:18:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb592d9d640 1 -- 172.21.15.98:0/2041635342 reap_dead start 2024-06-11T19:18:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 1 -- 172.21.15.98:0/2041635342 shutdown_connections 2024-06-11T19:18:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 1 -- 172.21.15.98:0/2041635342 >> 172.21.15.98:0/2041635342 conn(0x7fb5940ab850 msgr2=0x7fb5941be740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 1 -- 172.21.15.98:0/2041635342 shutdown_connections 2024-06-11T19:18:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 1 -- 172.21.15.98:0/2041635342 wait complete. 2024-06-11T19:18:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.620+0000 7fb598d0b640 1 librados: shutdown 2024-06-11T19:18:13.671 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:18:13.671 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon d 2024-06-11T19:18:13.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:18:13.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.771+0000 7f2922e17640 10 monclient: get_monmap_and_config 2024-06-11T19:18:13.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.771+0000 7f2922e17640 10 monclient: build_initial_monmap 2024-06-11T19:18:13.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.771+0000 7f2922e17640 10 monclient: monmap: 2024-06-11T19:18:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:18:13.772385+0000 2024-06-11T19:18:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:18:13.772385+0000 2024-06-11T19:18:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:18:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:18:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:18:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:18:13.772 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 1 Processor -- start 2024-06-11T19:18:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 1 -- start start 2024-06-11T19:18:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 10 monclient: init 2024-06-11T19:18:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 10 monclient: _add_conns ranks=[0,7,6,5,4,10,1,2,9,8,3] 2024-06-11T19:18:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f291c0a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:18:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f291c0a2530 0x7f291c13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 10 monclient(hunting): picked mon.noname-i con 0x7f291c0a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:18:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c13e1a0 0x7f291c140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 10 monclient(hunting): picked mon.noname-d con 0x7f291c13e1a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:18:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291c0a7140 con 0x7f291c0a4920 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f291c0a2b50 con 0x7f291c13e1a0 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f291c0a2e30 con 0x7f291c0a2530 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2922e17640 10 monclient(hunting): authenticate will time out at 1303.579695s 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2920bb3640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f291c0a4920 0x7f291c0a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46530/0 (socket says 172.21.15.98:46530) 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f29213b4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c13e1a0 0x7f291c140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f2920bb3640 1 -- 172.21.15.98:0/1247218423 learned_addr learned my addr 172.21.15.98:0/1247218423 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f29213b4640 10 monclient(hunting): get_auth_request con 0x7f291c13e1a0 auth_method 0 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f29213b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f29213b4640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f29213b4640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f29213b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f29213b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.772+0000 7f29213b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f291bfff640 1 --2- 172.21.15.98:0/1247218423 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f291c0a2530 0x7f291c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f291bfff640 10 monclient(hunting): get_auth_request con 0x7f291c0a2530 auth_method 0 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f291bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f291bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f291bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f29213b4640 10 monclient(hunting): handle_auth_done global_id 55230 payload 274 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f291b7fe640 1 -- 172.21.15.98:0/1247218423 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 38606027 0 0) 0x7f291c0a7140 con 0x7f291c0a4920 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f29213b4640 1 -- 172.21.15.98:0/1247218423 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f291c0a2530 msgr2=0x7f291c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f29213b4640 1 --2- 172.21.15.98:0/1247218423 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f291c0a2530 0x7f291c13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f29213b4640 1 -- 172.21.15.98:0/1247218423 >> v1:172.21.15.98:6789/0 conn(0x7f291c0a4920 legacy=0x7f291c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f29213b4640 10 monclient: _finish_hunting 0 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f29213b4640 1 monclient: found mon.noname-d 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f29213b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f29213b4640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f29213b4640 1 -- 172.21.15.98:0/1247218423 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291c0a3110 con 0x7f291c13e1a0 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f291b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f29213b4640 1 --2- 172.21.15.98:0/1247218423 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c13e1a0 0x7f291c140590 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2914009dc0 tx=0x7f291402bd50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=aa26c22604904cb5 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f291b7fe640 1 -- 172.21.15.98:0/1247218423 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2914039070 con 0x7f291c13e1a0 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f291b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f291b7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.773+0000 7f291b7fe640 10 monclient: dump: 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f291b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f291b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f291b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f291b7fe640 1 -- 172.21.15.98:0/1247218423 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f291402cd70 con 0x7f291c13e1a0 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 5 monclient: authenticate success, global_id 55230 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f291b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 10 monclient: get_monmap_and_config success 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 4 set_mon_vals no callback set 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f291b7fe640 1 -- 172.21.15.98:0/1247218423 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2914032340 con 0x7f291c13e1a0 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f291b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f291b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f291b7fe640 10 monclient: dump: 2024-06-11T19:18:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 10 monclient: shutdown 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 1 -- 172.21.15.98:0/1247218423 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c13e1a0 msgr2=0x7f291c140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 1 --2- 172.21.15.98:0/1247218423 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c13e1a0 0x7f291c140590 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2914009dc0 tx=0x7f291402bd50 comp rx=0 tx=0).stop 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 1 -- 172.21.15.98:0/1247218423 shutdown_connections 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 1 --2- 172.21.15.98:0/1247218423 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c13e1a0 0x7f291c140590 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 1 --2- 172.21.15.98:0/1247218423 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f291c0a2530 0x7f291c13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 1 -- 172.21.15.98:0/1247218423 >> 172.21.15.98:0/1247218423 conn(0x7f291c0ab850 msgr2=0x7f291c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 1 -- 172.21.15.98:0/1247218423 shutdown_connections 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.774+0000 7f2922e17640 1 -- 172.21.15.98:0/1247218423 wait complete. 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.775+0000 7f2922e17640 10 monclient: build_initial_monmap 2024-06-11T19:18:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.775+0000 7f2922e17640 10 monclient: monmap: 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.775+0000 7f2922e17640 1 librados: starting msgr at 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.775+0000 7f2922e17640 1 librados: starting objecter 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.775+0000 7f2922e17640 1 Processor -- start 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.775+0000 7f2922e17640 1 -- start start 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.775+0000 7f2922e17640 1 librados: setting wanted keys 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.775+0000 7f2922e17640 1 librados: calling monclient init 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.775+0000 7f2922e17640 10 monclient: init 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.775+0000 7f2922e17640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.775+0000 7f2922e17640 10 monclient: _add_conns ranks=[1,5,3,0,7,6,4,2,8] 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f291c0a2530 0x7f291c1cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 10 monclient(hunting): picked mon.noname-f con 0x7f291c0a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:18:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f291c0a4920 0x7f291c136a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 10 monclient(hunting): picked mon.noname-h con 0x7f291c0a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f291c13e1a0 0x7f291c136f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 10 monclient(hunting): picked mon.noname-g con 0x7f291c13e1a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f291c138e50 con 0x7f291c0a2530 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f291c0a84b0 con 0x7f291c13e1a0 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f291c0a6da0 con 0x7f291c0a4920 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2922e17640 10 monclient(hunting): authenticate will time out at 1303.583059s 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f29213b4640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f291c13e1a0 msgr2=0x7f291c136f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f29213b4640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f291c13e1a0 0x7f291c136f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2920bb3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f291c0a2530 0x7f291c1cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f291c0a4920 0x7f291c136a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f291c0a4920 0x7f291c136a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:35864/0 (socket says 172.21.15.98:35864) 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 1 -- 172.21.15.98:0/15968933 learned_addr learned my addr 172.21.15.98:0/15968933 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 10 monclient(hunting): get_auth_request con 0x7f291c0a4920 auth_method 0 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2920bb3640 10 monclient(hunting): get_auth_request con 0x7f291c0a2530 auth_method 0 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2920bb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2920bb3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2920bb3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2920bb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2920bb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f2920bb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 10 monclient(hunting): handle_auth_done global_id 23126 payload 824 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 1 -- 172.21.15.98:0/15968933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f291c13e1a0 msgr2=0x7f291c136f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f291c13e1a0 0x7f291c136f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 1 -- 172.21.15.98:0/15968933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f291c0a2530 msgr2=0x7f291c1cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f291c0a2530 0x7f291c1cd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 1 monclient: found mon.noname-h 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 1 -- 172.21.15.98:0/15968933 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291c123400 con 0x7f291c0a4920 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f291bfff640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f291c0a4920 0x7f291c136a50 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f290800d830 tx=0x7f290800dcf0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f29197fa640 10 client.?.objecter ms_handle_connect 0x7f291c0a4920 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.776+0000 7f29197fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f290800bb30 con 0x7f291c0a4920 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 10 monclient: dump: 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 10 monclient: _finish_auth 0 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 10 monclient: _check_auth_tickets 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2908019070 con 0x7f291c0a4920 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 5 monclient: authenticate success, global_id 23126 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 10 monclient: _renew_subs 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f290800bd10 con 0x7f291c0a4920 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 1 -- 172.21.15.98:0/15968933 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f291c0aaa10 con 0x7f291c0a4920 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f29197fa640 10 monclient: dump: 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f291affd640 4 set_mon_vals no callback set 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 10 client.23126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:18:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 10 monclient: _renew_subs 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 1 -- 172.21.15.98:0/15968933 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f291c0a1690 con 0x7f291c0a4920 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 1 librados: init done 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c09f9c0 0x7f291c09fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.777+0000 7f2922e17640 1 -- 172.21.15.98:0/15968933 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f291c050830 con 0x7f291c09f9c0 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f2920bb3640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c09f9c0 0x7f291c09fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f2920bb3640 10 monclient: get_auth_request con 0x7f291c09f9c0 auth_method 0 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f2920bb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f2920bb3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f2920bb3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f2920bb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f2920bb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f2920bb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 <== mon.5 v2:172.21.15.117:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f290801e460 con 0x7f291c0a4920 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f2920bb3640 10 monclient(hunting): handle_auth_done global_id 55239 payload 824 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f2920bb3640 1 -- 172.21.15.98:0/15968933 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f291c0a5230 con 0x7f291c09f9c0 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f290003fdb0 0x7f2900042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 <== mon.5 v2:172.21.15.117:3302/0 5 ==== osd_map(348..348 src has 1..348) ==== 6230+0+0 (secure 0 0 0) 0x7f290800f040 con 0x7f291c0a4920 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 10 client.23126.objecter ms_dispatch 0x7f291c000b90 osd_map(348..348 src has 1..348) 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 3 client.23126.objecter handle_osd_map got epochs [348,348] > 0 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 3 client.23126.objecter handle_osd_map decoding full epoch 348 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f2920bb3640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c09f9c0 0x7f291c09fda0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2910000c00 tx=0x7f29100019b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 20 client.23126.objecter dump_active .. 0 homeless 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29213b4640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f290003fdb0 0x7f2900042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 10 client.23126.objecter ms_handle_connect 0x7f291c09f9c0 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 10 client.23126.objecter resend_mon_ops 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29213b4640 10 monclient: get_auth_request con 0x7f290003fdb0 auth_method 0 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29100119f0 con 0x7f291c09f9c0 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 10 monclient: dump: 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2910018070 con 0x7f291c09f9c0 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:18:13.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c09f9c0 msgr2=0x7f291c09fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c09f9c0 0x7f291c09fda0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2910000c00 tx=0x7f29100019b0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c09f9c0 msgr2=0x7f291c09fda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.778+0000 7f29197fa640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c09f9c0 0x7f291c09fda0 secure :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2910000c00 tx=0x7f29100019b0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.779+0000 7f29197fa640 10 client.23126.objecter ms_handle_connect 0x7f290003fdb0 2024-06-11T19:18:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.779+0000 7f29213b4640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f290003fdb0 0x7f2900042270 secure :-1 s=READY pgs=4187 cs=0 l=1 rev1=1 crypto rx=0x7f2914036000 tx=0x7f2914030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2922e17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:18:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2922e17640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c0a02e0 0x7f291c13bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2922e17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2922e17640 1 -- 172.21.15.98:0/15968933 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f291c072d40 con 0x7f291c0a02e0 2024-06-11T19:18:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2920bb3640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c0a02e0 0x7f291c13bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2920bb3640 10 monclient: get_auth_request con 0x7f291c0a02e0 auth_method 0 2024-06-11T19:18:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2920bb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2920bb3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2920bb3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2920bb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2920bb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2920bb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2920bb3640 10 monclient(hunting): handle_auth_done global_id 55248 payload 824 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2920bb3640 1 -- 172.21.15.98:0/15968933 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f291c0a4d00 con 0x7f291c0a02e0 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f29197fa640 10 client.23126.objecter ms_handle_connect 0x7f291c0a02e0 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f29197fa640 10 client.23126.objecter resend_mon_ops 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f2920bb3640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c0a02e0 0x7f291c13bd50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2910000fb0 tx=0x7f2910007cd0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2910011db0 con 0x7f291c0a02e0 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f29197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f29197fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.785+0000 7f29197fa640 10 monclient: dump: 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9432 (secure 0 0 0) 0x7f2910011db0 con 0x7f291c0a02e0 2024-06-11T19:18:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f29197fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:18:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f29197fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:18:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c0a02e0 msgr2=0x7f291c13bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f29197fa640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c0a02e0 0x7f291c13bd50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2910000fb0 tx=0x7f2910007cd0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f29197fa640 1 -- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c0a02e0 msgr2=0x7f291c13bd50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f29197fa640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f291c0a02e0 0x7f291c13bd50 secure :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2910000fb0 tx=0x7f2910007cd0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.808 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:18:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:18:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:18:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 50, 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 406541, 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:18:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:18:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:18:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:18:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:18:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:18:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:18:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:18:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:18:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:18:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:18:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:18:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:18:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:18:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:18:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:18:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:18:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:18:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:18:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:18:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:18:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:18:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:13.831 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:18:13.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:13.832 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:13.832 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:13.832 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:18:13.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:13.832 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:13.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:13.832 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:18:13.832 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:18:13.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 10 librados: watch_flush enter 2024-06-11T19:18:13.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 10 librados: watch_flush exit 2024-06-11T19:18:13.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 20 client.23126.objecter shutdown clearing up homeless session... 2024-06-11T19:18:13.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 10 client.23126.objecter successfully canceled tick 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 1 -- 172.21.15.98:0/15968933 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f290003fdb0 msgr2=0x7f2900042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f290003fdb0 0x7f2900042270 secure :-1 s=READY pgs=4187 cs=0 l=1 rev1=1 crypto rx=0x7f2914036000 tx=0x7f2914030000 comp rx=0 tx=0).stop 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 10 monclient: shutdown 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 1 -- 172.21.15.98:0/15968933 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f291c0a4920 msgr2=0x7f291c136a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 1 --2- 172.21.15.98:0/15968933 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f291c0a4920 0x7f291c136a50 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f290800d830 tx=0x7f290800dcf0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f29213b4640 1 -- 172.21.15.98:0/15968933 reap_dead start 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f29213b4640 1 -- 172.21.15.98:0/15968933 reap_dead start 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 1 -- 172.21.15.98:0/15968933 shutdown_connections 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 1 -- 172.21.15.98:0/15968933 >> 172.21.15.98:0/15968933 conn(0x7f291c0ab850 msgr2=0x7f291c072ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 1 -- 172.21.15.98:0/15968933 shutdown_connections 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 1 -- 172.21.15.98:0/15968933 wait complete. 2024-06-11T19:18:13.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.786+0000 7f2922e17640 1 librados: shutdown 2024-06-11T19:18:13.836 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:18:13.836 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-06-11T19:18:13.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:18:13.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.942+0000 7f4c3c79c640 10 monclient: get_monmap_and_config 2024-06-11T19:18:13.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.942+0000 7f4c3c79c640 10 monclient: build_initial_monmap 2024-06-11T19:18:13.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.942+0000 7f4c3c79c640 10 monclient: monmap: 2024-06-11T19:18:13.942 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:13.942 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:18:13.943593+0000 2024-06-11T19:18:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:18:13.943593+0000 2024-06-11T19:18:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:18:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:18:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:13.943 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:18:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:18:13.944 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 1 Processor -- start 2024-06-11T19:18:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 1 -- start start 2024-06-11T19:18:13.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 10 monclient: init 2024-06-11T19:18:13.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:13.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 10 monclient: _add_conns ranks=[6,2,8,1,5,10,3,4,9,7,0] 2024-06-11T19:18:13.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c340a2f40 0x7f4c340a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 10 monclient(hunting): picked mon.noname-d con 0x7f4c340a2f40 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:18:13.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c340a4df0 0x7f4c340a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 10 monclient(hunting): picked mon.noname-b con 0x7f4c340a4df0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:18:13.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c341366d0 0x7f4c34136af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 10 monclient(hunting): picked mon.noname-e con 0x7f4c341366d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:18:13.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4c340a5710 con 0x7f4c340a4df0 2024-06-11T19:18:13.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4c340a3320 con 0x7f4c340a2f40 2024-06-11T19:18:13.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4c340a2460 con 0x7f4c341366d0 2024-06-11T19:18:13.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:13.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3c79c640 10 monclient(hunting): authenticate will time out at 1303.750856s 2024-06-11T19:18:13.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3a538640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c340a2f40 0x7f4c340a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c39d37640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c340a4df0 0x7f4c340a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3a538640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c340a2f40 0x7f4c340a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49818/0 (socket says 172.21.15.98:49818) 2024-06-11T19:18:13.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3a538640 1 -- 172.21.15.98:0/718538414 learned_addr learned my addr 172.21.15.98:0/718538414 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:13.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c3ad39640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c341366d0 0x7f4c34136af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c39d37640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c340a4df0 0x7f4c340a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45710/0 (socket says 172.21.15.98:45710) 2024-06-11T19:18:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.943+0000 7f4c39d37640 10 monclient(hunting): get_auth_request con 0x7f4c340a4df0 auth_method 0 2024-06-11T19:18:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3a538640 10 monclient(hunting): get_auth_request con 0x7f4c340a2f40 auth_method 0 2024-06-11T19:18:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3a538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3a538640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3a538640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3ad39640 10 monclient(hunting): get_auth_request con 0x7f4c341366d0 auth_method 0 2024-06-11T19:18:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3ad39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3ad39640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3ad39640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3ad39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3ad39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3ad39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3a538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3a538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3a538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 10 monclient(hunting): handle_auth_done global_id 84557 payload 274 2024-06-11T19:18:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 1 -- 172.21.15.98:0/718538414 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c341366d0 msgr2=0x7f4c34136af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:18:13.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 1 --2- 172.21.15.98:0/718538414 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c341366d0 0x7f4c34136af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 1 -- 172.21.15.98:0/718538414 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c340a2f40 msgr2=0x7f4c340a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 1 --2- 172.21.15.98:0/718538414 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c340a2f40 0x7f4c340a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 10 monclient: _finish_hunting 0 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 1 monclient: found mon.noname-b 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c39d37640 1 -- 172.21.15.98:0/718538414 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c340a0ed0 con 0x7f4c340a4df0 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3ad39640 1 --2- 172.21.15.98:0/718538414 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c341366d0 0x7f4c34136af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.944+0000 7f4c3a538640 1 --2- 172.21.15.98:0/718538414 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c340a2f40 0x7f4c340a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39d37640 1 --2- 172.21.15.98:0/718538414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c340a4df0 0x7f4c340a51d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4c24009800 tx=0x7f4c2402b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67de893f6690dda5 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 1 -- 172.21.15.98:0/718538414 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4c24037070 con 0x7f4c340a4df0 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:18:13.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 10 monclient: dump: 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.957 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.957 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 10 monclient: _finish_auth 0 2024-06-11T19:18:13.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 10 monclient: _check_auth_tickets 2024-06-11T19:18:13.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:13.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 1 -- 172.21.15.98:0/718538414 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c24030980 con 0x7f4c340a4df0 2024-06-11T19:18:13.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 5 monclient: authenticate success, global_id 84557 2024-06-11T19:18:13.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:18:13.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:13.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 1 -- 172.21.15.98:0/718538414 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4c24030bb0 con 0x7f4c340a4df0 2024-06-11T19:18:13.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c39536640 10 monclient: dump: 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:13.958 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 10 monclient: get_monmap_and_config success 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 4 set_mon_vals no callback set 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 10 monclient: shutdown 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 1 -- 172.21.15.98:0/718538414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c340a4df0 msgr2=0x7f4c340a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 1 --2- 172.21.15.98:0/718538414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c340a4df0 0x7f4c340a51d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4c24009800 tx=0x7f4c2402b6d0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 1 -- 172.21.15.98:0/718538414 shutdown_connections 2024-06-11T19:18:13.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 1 --2- 172.21.15.98:0/718538414 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c341366d0 0x7f4c34136af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 1 --2- 172.21.15.98:0/718538414 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4c340a4df0 0x7f4c340a51d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 1 --2- 172.21.15.98:0/718538414 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c340a2f40 0x7f4c340a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 1 -- 172.21.15.98:0/718538414 >> 172.21.15.98:0/718538414 conn(0x7f4c340a95b0 msgr2=0x7f4c340a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 1 -- 172.21.15.98:0/718538414 shutdown_connections 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.945+0000 7f4c3c79c640 1 -- 172.21.15.98:0/718538414 wait complete. 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.946+0000 7f4c3c79c640 10 monclient: build_initial_monmap 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.946+0000 7f4c3c79c640 10 monclient: monmap: 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.946+0000 7f4c3c79c640 1 librados: starting msgr at 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.946+0000 7f4c3c79c640 1 librados: starting objecter 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.946+0000 7f4c3c79c640 1 Processor -- start 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.946+0000 7f4c3c79c640 1 -- start start 2024-06-11T19:18:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.946+0000 7f4c3c79c640 1 librados: setting wanted keys 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.946+0000 7f4c3c79c640 1 librados: calling monclient init 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.946+0000 7f4c3c79c640 10 monclient: init 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 10 monclient: _add_conns ranks=[8,3,5,7,6,0,2,4,1] 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340a2f40 0x7f4c34135710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 10 monclient(hunting): picked mon.noname-e con 0x7f4c340a2f40 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c340a4df0 0x7f4c34135c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 10 monclient(hunting): picked mon.noname-g con 0x7f4c340a4df0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4c341366d0 0x7f4c341211f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 10 monclient(hunting): picked mon.noname-h con 0x7f4c341366d0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f4c340a1250 con 0x7f4c340a4df0 2024-06-11T19:18:13.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f4c34137030 con 0x7f4c341366d0 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4c341bea00 con 0x7f4c340a2f40 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3c79c640 10 monclient(hunting): authenticate will time out at 1303.754309s 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340a2f40 0x7f4c34135710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340a2f40 0x7f4c34135710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:41554/0 (socket says 172.21.15.98:41554) 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 -- 172.21.15.98:0/1017110886 learned_addr learned my addr 172.21.15.98:0/1017110886 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 10 monclient(hunting): get_auth_request con 0x7f4c340a2f40 auth_method 0 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c39d37640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c340a4df0 msgr2=0x7f4c34135c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c39d37640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c340a4df0 0x7f4c34135c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3ad39640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4c341366d0 0x7f4c341211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3ad39640 10 monclient(hunting): get_auth_request con 0x7f4c341366d0 auth_method 0 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3ad39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3ad39640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3ad39640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 10 monclient(hunting): handle_auth_done global_id 94229 payload 824 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4c341366d0 msgr2=0x7f4c341211f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4c341366d0 0x7f4c341211f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c340a4df0 msgr2=0x7f4c34135c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:18:13.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c340a4df0 0x7f4c34135c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:13.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 10 monclient: _finish_hunting 0 2024-06-11T19:18:13.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 monclient: found mon.noname-e 2024-06-11T19:18:13.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:13.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:18:13.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 -- 172.21.15.98:0/1017110886 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c34124e10 con 0x7f4c340a2f40 2024-06-11T19:18:13.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340a2f40 msgr2=0x7f4c34135710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:18:13.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340a2f40 msgr2=0x7f4c34135710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:18:13.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340a2f40 0x7f4c34135710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000e9d0 tx=0x7f4c3000ee90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:18:13.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:13.947+0000 7f4c3a538640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340a2f40 0x7f4c34135710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:14.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340a2f40 0x7f4c34135710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 10 monclient: get_auth_request con 0x7f4c340a2f40 auth_method 0 2024-06-11T19:18:14.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 0 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340a2f40 0x7f4c34135710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-11T19:18:14.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340a2f40 0x7f4c34135710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:18:14.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:14.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340a2f40 msgr2=0x7f4c34135710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:14.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340a2f40 0x7f4c34135710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 10 monclient: _add_conns ranks=[6,7,1,4,2,0] 2024-06-11T19:18:14.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c180035d0 0x7f4c18005a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 10 monclient(hunting): picked mon.noname-d con 0x7f4c180035d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:18:14.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4c18006570 0x7f4c18008a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 10 monclient(hunting): picked mon.noname-i con 0x7f4c18006570 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:18:14.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c180095a0 0x7f4c1800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 10 monclient(hunting): picked mon.noname-f con 0x7f4c180095a0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:18:14.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1800c650 con 0x7f4c180095a0 2024-06-11T19:18:14.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4c1800c7d0 con 0x7f4c180035d0 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f4c1800c9d0 con 0x7f4c18006570 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c180035d0 0x7f4c18005a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 10 monclient(hunting): get_auth_request con 0x7f4c180035d0 auth_method 0 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 10 monclient(hunting): handle_auth_done global_id 94229 payload 293 2024-06-11T19:18:14.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4c18006570 msgr2=0x7f4c18008a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4c18006570 0x7f4c18008a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c180095a0 msgr2=0x7f4c1800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c180095a0 0x7f4c1800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 10 monclient: _finish_hunting 0 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 1 monclient: found mon.noname-d 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3ad39640 1 -- 172.21.15.98:0/1017110886 reap_dead start 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 1 -- 172.21.15.98:0/1017110886 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1800cc90 con 0x7f4c180035d0 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c3a538640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c180035d0 0x7f4c18005a20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4c34125570 tx=0x7f4c30009240 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 10 client.?.objecter ms_handle_connect 0x7f4c180035d0 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.148+0000 7f4c2affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4c3000cbc0 con 0x7f4c180035d0 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:18:14.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 10 monclient: dump: 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.156 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 10 monclient: _finish_auth 0 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 10 monclient: _check_auth_tickets 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c30017040 con 0x7f4c180035d0 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 5 monclient: authenticate success, global_id 94229 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 10 monclient: _renew_subs 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 1 -- 172.21.15.98:0/1017110886 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c340a4310 con 0x7f4c180035d0 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4c300044a0 con 0x7f4c180035d0 2024-06-11T19:18:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 10 monclient: dump: 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c38d35640 4 set_mon_vals no callback set 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 10 client.94229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 10 monclient: _renew_subs 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 1 -- 172.21.15.98:0/1017110886 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c34122e00 con 0x7f4c180035d0 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 1 librados: init done 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f4c30004030 con 0x7f4c180035d0 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c34135710 0x7f4c34135af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.149+0000 7f4c3c79c640 1 -- 172.21.15.98:0/1017110886 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4c340a13d0 con 0x7f4c34135710 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c39d37640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c34135710 0x7f4c34135af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c39d37640 10 monclient: get_auth_request con 0x7f4c34135710 auth_method 0 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c39d37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c39d37640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c39d37640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c39d37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c39d37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c39d37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4c18044610 0x7f4c18046ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(349..349 src has 1..349) ==== 5845+0+0 (secure 0 0 0) 0x7f4c3000f040 con 0x7f4c180035d0 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 10 client.94229.objecter ms_dispatch 0x7f4c34000b90 osd_map(349..349 src has 1..349) 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 3 client.94229.objecter handle_osd_map got epochs [349,349] > 0 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 3 client.94229.objecter handle_osd_map decoding full epoch 349 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c39d37640 10 monclient(hunting): handle_auth_done global_id 94238 payload 824 2024-06-11T19:18:14.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c39d37640 1 -- 172.21.15.98:0/1017110886 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c34124850 con 0x7f4c34135710 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 20 client.94229.objecter dump_active .. 0 homeless 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c3ad39640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4c18044610 0x7f4c18046ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c3ad39640 10 monclient: get_auth_request con 0x7f4c18044610 auth_method 0 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 10 client.94229.objecter ms_handle_connect 0x7f4c34135710 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 10 client.94229.objecter resend_mon_ops 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c39d37640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c34135710 0x7f4c34135af0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4c240097d0 tx=0x7f4c240044c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4c24041030 con 0x7f4c34135710 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 10 monclient: dump: 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.161 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c2affd640 10 client.94229.objecter ms_handle_connect 0x7f4c18044610 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.150+0000 7f4c3ad39640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4c18044610 0x7f4c18046ad0 secure :-1 s=READY pgs=4188 cs=0 l=1 rev1=1 crypto rx=0x7f4c34124760 tx=0x7f4c2c004ab0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.151+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4c24030eb0 con 0x7f4c34135710 2024-06-11T19:18:14.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.151+0000 7f4c2affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:14.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.151+0000 7f4c2affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:18:14.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.151+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c34135710 msgr2=0x7f4c34135af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.151+0000 7f4c2affd640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c34135710 0x7f4c34135af0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4c240097d0 tx=0x7f4c240044c0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.151+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c34135710 msgr2=0x7f4c34135af0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:14.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.151+0000 7f4c2affd640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c34135710 0x7f4c34135af0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4c240097d0 tx=0x7f4c240044c0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c3c79c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c3c79c640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340705e0 0x7f4c340709f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c3c79c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c3c79c640 1 -- 172.21.15.98:0/1017110886 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4c34050830 con 0x7f4c340705e0 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c39d37640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340705e0 0x7f4c340709f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c39d37640 10 monclient: get_auth_request con 0x7f4c340705e0 auth_method 0 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c39d37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c39d37640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c39d37640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c39d37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c39d37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c39d37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c39d37640 10 monclient(hunting): handle_auth_done global_id 94247 payload 824 2024-06-11T19:18:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.157+0000 7f4c39d37640 1 -- 172.21.15.98:0/1017110886 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4c341373c0 con 0x7f4c340705e0 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c39d37640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340705e0 0x7f4c340709f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4c240096f0 tx=0x7f4c24030000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 10 client.94229.objecter ms_handle_connect 0x7f4c340705e0 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 10 client.94229.objecter resend_mon_ops 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4c24047e60 con 0x7f4c340705e0 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 10 monclient: dump: 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9261 (secure 0 0 0) 0x7f4c24009300 con 0x7f4c340705e0 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:18:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340705e0 msgr2=0x7f4c340709f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340705e0 0x7f4c340709f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4c240096f0 tx=0x7f4c24030000 comp rx=0 tx=0).stop 2024-06-11T19:18:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340705e0 msgr2=0x7f4c340709f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c2affd640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340705e0 0x7f4c340709f0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4c240096f0 tx=0x7f4c24030000 comp rx=0 tx=0).stop 2024-06-11T19:18:14.168 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:18:14.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:18:14.168 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:18:14.168 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:18:14.168 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 50, 2024-06-11T19:18:14.168 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 39711, 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:18:14.169 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:18:14.170 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:18:14.171 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:18:14.172 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:14.173 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.176 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:18:14.178 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.180 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:18:14.183 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:18:14.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:18:14.187 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:18:14.187 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.187 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.187 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.187 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:18:14.187 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:18:14.187 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.188 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:18:14.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.190 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:14.190 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:14.190 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:14.190 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.190 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.190 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:18:14.190 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:18:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c3c79c640 10 librados: watch_flush enter 2024-06-11T19:18:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c3c79c640 10 librados: watch_flush exit 2024-06-11T19:18:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c3c79c640 20 client.94229.objecter shutdown clearing up homeless session... 2024-06-11T19:18:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c3c79c640 10 client.94229.objecter successfully canceled tick 2024-06-11T19:18:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c3c79c640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4c18044610 msgr2=0x7f4c18046ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c3c79c640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4c18044610 0x7f4c18046ad0 secure :-1 s=READY pgs=4188 cs=0 l=1 rev1=1 crypto rx=0x7f4c34124760 tx=0x7f4c2c004ab0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c3c79c640 10 monclient: shutdown 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c3c79c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c3c79c640 1 -- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c180035d0 msgr2=0x7f4c18005a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.158+0000 7f4c3c79c640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c180035d0 0x7f4c18005a20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4c34125570 tx=0x7f4c30009240 comp rx=0 tx=0).stop 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.159+0000 7f4c3c79c640 1 -- 172.21.15.98:0/1017110886 shutdown_connections 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.159+0000 7f4c3c79c640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4c18044610 0x7f4c18046ad0 unknown :-1 s=CLOSED pgs=4188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.159+0000 7f4c3c79c640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4c180035d0 0x7f4c18005a20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.159+0000 7f4c3c79c640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c340705e0 0x7f4c340709f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.159+0000 7f4c3c79c640 1 --2- 172.21.15.98:0/1017110886 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4c34135710 0x7f4c34135af0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.159+0000 7f4c3c79c640 1 -- 172.21.15.98:0/1017110886 >> 172.21.15.98:0/1017110886 conn(0x7f4c340a95b0 msgr2=0x7f4c340a4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.159+0000 7f4c3c79c640 1 -- 172.21.15.98:0/1017110886 shutdown_connections 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.159+0000 7f4c3c79c640 1 -- 172.21.15.98:0/1017110886 wait complete. 2024-06-11T19:18:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.159+0000 7f4c3c79c640 1 librados: shutdown 2024-06-11T19:18:14.194 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:18:14.194 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-06-11T19:18:14.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:18:14.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.294+0000 7f3198784640 10 monclient: get_monmap_and_config 2024-06-11T19:18:14.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.294+0000 7f3198784640 10 monclient: build_initial_monmap 2024-06-11T19:18:14.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.294+0000 7f3198784640 10 monclient: monmap: 2024-06-11T19:18:14.294 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:14.294 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.294 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:18:14.295204+0000 2024-06-11T19:18:14.294 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:18:14.295204+0000 2024-06-11T19:18:14.294 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:14.294 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.294+0000 7f3198784640 1 Processor -- start 2024-06-11T19:18:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.294+0000 7f3198784640 1 -- start start 2024-06-11T19:18:14.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.294+0000 7f3198784640 10 monclient: init 2024-06-11T19:18:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 10 monclient: _add_conns ranks=[6,1,2,10,3,9,4,7,0,8,5] 2024-06-11T19:18:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31901347f0 0x7f319009e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 10 monclient(hunting): picked mon.noname-d con 0x7f31901347f0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:18:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3190134bd0 addr v1:172.21.15.117:6789/0 2024-06-11T19:18:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3190135010 0x7f31900a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 10 monclient(hunting): picked mon.noname-b con 0x7f3190135010 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:18:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f319009d410 con 0x7f3190134bd0 2024-06-11T19:18:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f31900687f0 con 0x7f3190135010 2024-06-11T19:18:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f319009d6f0 con 0x7f31901347f0 2024-06-11T19:18:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3198784640 10 monclient(hunting): authenticate will time out at 1304.102470s 2024-06-11T19:18:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196520640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31901347f0 0x7f319009e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3190135010 0x7f31900a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196520640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31901347f0 0x7f319009e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49836/0 (socket says 172.21.15.98:49836) 2024-06-11T19:18:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196520640 1 -- 172.21.15.98:0/1390725361 learned_addr learned my addr 172.21.15.98:0/1390725361 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196520640 10 monclient(hunting): get_auth_request con 0x7f31901347f0 auth_method 0 2024-06-11T19:18:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196520640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196520640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 10 monclient(hunting): get_auth_request con 0x7f3190135010 auth_method 0 2024-06-11T19:18:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 10 monclient(hunting): handle_auth_done global_id 84566 payload 274 2024-06-11T19:18:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 1 -- 172.21.15.98:0/1390725361 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31901347f0 msgr2=0x7f319009e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:18:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 1 --2- 172.21.15.98:0/1390725361 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31901347f0 0x7f319009e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 1 -- 172.21.15.98:0/1390725361 >> v1:172.21.15.117:6789/0 conn(0x7f3190134bd0 legacy=0x7f31900a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:18:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 10 monclient: _finish_hunting 0 2024-06-11T19:18:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 1 monclient: found mon.noname-b 2024-06-11T19:18:14.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196d21640 1 -- 172.21.15.98:0/1390725361 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319009d9d0 con 0x7f3190135010 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.295+0000 7f3196520640 1 --2- 172.21.15.98:0/1390725361 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31901347f0 0x7f319009e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f3196d21640 1 --2- 172.21.15.98:0/1390725361 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3190135010 0x7f31900a3da0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f31800097d0 tx=0x7f318002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4175a3421ec08a9 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 1 -- 172.21.15.98:0/1390725361 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31800319d0 con 0x7f3190135010 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 10 monclient: dump: 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.309 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 10 monclient: _finish_auth 0 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 10 monclient: _check_auth_tickets 2024-06-11T19:18:14.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 1 -- 172.21.15.98:0/1390725361 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3180037070 con 0x7f3190135010 2024-06-11T19:18:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f3198784640 5 monclient: authenticate success, global_id 84566 2024-06-11T19:18:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f3198784640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:18:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f3198784640 10 monclient: get_monmap_and_config success 2024-06-11T19:18:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 1 -- 172.21.15.98:0/1390725361 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3180031bd0 con 0x7f3190135010 2024-06-11T19:18:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f3198784640 4 set_mon_vals no callback set 2024-06-11T19:18:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:18:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.296+0000 7f319551e640 10 monclient: dump: 2024-06-11T19:18:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.311 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.312 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.297+0000 7f3198784640 10 monclient: shutdown 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.297+0000 7f3198784640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.297+0000 7f3198784640 1 -- 172.21.15.98:0/1390725361 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3190135010 msgr2=0x7f31900a3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.297+0000 7f3198784640 1 --2- 172.21.15.98:0/1390725361 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3190135010 0x7f31900a3da0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f31800097d0 tx=0x7f318002b800 comp rx=0 tx=0).stop 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.297+0000 7f3198784640 1 -- 172.21.15.98:0/1390725361 shutdown_connections 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.297+0000 7f3198784640 1 --2- 172.21.15.98:0/1390725361 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3190135010 0x7f31900a3da0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.297+0000 7f3198784640 1 --2- 172.21.15.98:0/1390725361 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31901347f0 0x7f319009e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.297+0000 7f3198784640 1 -- 172.21.15.98:0/1390725361 >> 172.21.15.98:0/1390725361 conn(0x7f31900a4780 msgr2=0x7f31900a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.297+0000 7f3198784640 1 -- 172.21.15.98:0/1390725361 shutdown_connections 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.297+0000 7f3198784640 1 -- 172.21.15.98:0/1390725361 wait complete. 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.297+0000 7f3198784640 10 monclient: build_initial_monmap 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 10 monclient: monmap: 2024-06-11T19:18:14.313 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:14.314 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 1 librados: starting msgr at 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 1 librados: starting objecter 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 1 Processor -- start 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 1 -- start start 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 1 librados: setting wanted keys 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 1 librados: calling monclient init 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 10 monclient: init 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 10 monclient: _add_conns ranks=[2,4,0,1,8,3,7,6,5] 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901347f0 0x7f31901336d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 10 monclient(hunting): picked mon.noname-b con 0x7f31901347f0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3190134bd0 0x7f319011ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 10 monclient(hunting): picked mon.noname-c con 0x7f3190134bd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3190135010 0x7f319011f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 10 monclient(hunting): picked mon.noname-a con 0x7f3190135010 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3190068c40 con 0x7f3190135010 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3190068ac0 con 0x7f31901347f0 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f31900a42e0 con 0x7f3190134bd0 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3198784640 10 monclient(hunting): authenticate will time out at 1304.105863s 2024-06-11T19:18:14.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3196520640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901347f0 0x7f31901336d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3196520640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901347f0 0x7f31901336d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45720/0 (socket says 172.21.15.98:45720) 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3196520640 1 -- 172.21.15.98:0/2039201004 learned_addr learned my addr 172.21.15.98:0/2039201004 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3196520640 10 monclient(hunting): get_auth_request con 0x7f31901347f0 auth_method 0 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3196520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3196520640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3196520640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3195d1f640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3190134bd0 0x7f319011ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3196d21640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3190135010 0x7f319011f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3195d1f640 10 monclient(hunting): get_auth_request con 0x7f3190134bd0 auth_method 0 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3195d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3195d1f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.298+0000 7f3195d1f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196d21640 10 monclient(hunting): get_auth_request con 0x7f3190135010 auth_method 0 2024-06-11T19:18:14.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196d21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196d21640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196d21640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3195d1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3195d1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3195d1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196d21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196d21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196d21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 10 monclient(hunting): handle_auth_done global_id 84575 payload 824 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 1 -- 172.21.15.98:0/2039201004 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3190134bd0 msgr2=0x7f319011ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3190134bd0 0x7f319011ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 1 -- 172.21.15.98:0/2039201004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3190135010 msgr2=0x7f319011f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3190135010 0x7f319011f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 10 monclient: _finish_hunting 0 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 1 monclient: found mon.noname-b 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 1 -- 172.21.15.98:0/2039201004 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31901ce960 con 0x7f31901347f0 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3196520640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901347f0 0x7f31901336d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f318800efc0 tx=0x7f318800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 10 client.?.objecter ms_handle_connect 0x7f31901347f0 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31880090d0 con 0x7f31901347f0 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 10 monclient: dump: 2024-06-11T19:18:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.320 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 10 monclient: _finish_auth 0 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 10 monclient: _check_auth_tickets 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3188019070 con 0x7f31901347f0 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3198784640 5 monclient: authenticate success, global_id 84575 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3198784640 10 monclient: _renew_subs 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3198784640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3198784640 1 -- 172.21.15.98:0/2039201004 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31900a3680 con 0x7f31901347f0 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3188014d60 con 0x7f31901347f0 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f318effd640 10 monclient: dump: 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.322 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3198784640 10 client.84575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3198784640 10 monclient: _renew_subs 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3198784640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3198784640 1 -- 172.21.15.98:0/2039201004 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f31901216b0 con 0x7f31901347f0 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3198784640 1 librados: init done 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.299+0000 7f3194d1d640 4 set_mon_vals no callback set 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3198784640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3198784640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f31901c0c90 0x7f319011f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3198784640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3198784640 1 -- 172.21.15.98:0/2039201004 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3190050830 con 0x7f31901c0c90 2024-06-11T19:18:14.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3188005d10 con 0x7f31901347f0 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3195d1f640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f31901c0c90 0x7f319011f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3195d1f640 10 monclient: get_auth_request con 0x7f31901c0c90 auth_method 0 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3195d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3195d1f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3195d1f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3195d1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3195d1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3195d1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f318effd640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f317c03fdb0 0x7f317c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(349..349 src has 1..349) ==== 5845+0+0 (secure 0 0 0) 0x7f3188051ad0 con 0x7f31901347f0 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f318effd640 10 client.84575.objecter ms_dispatch 0x7f3190000b90 osd_map(349..349 src has 1..349) 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f318effd640 3 client.84575.objecter handle_osd_map got epochs [349,349] > 0 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f318effd640 3 client.84575.objecter handle_osd_map decoding full epoch 349 2024-06-11T19:18:14.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3195d1f640 10 monclient(hunting): handle_auth_done global_id 32320 payload 824 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3195d1f640 1 -- 172.21.15.98:0/2039201004 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f319009cda0 con 0x7f31901c0c90 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.300+0000 7f3196d21640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f317c03fdb0 0x7f317c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 20 client.84575.objecter dump_active .. 0 homeless 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f3196d21640 10 monclient: get_auth_request con 0x7f317c03fdb0 auth_method 0 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 10 client.84575.objecter ms_handle_connect 0x7f31901c0c90 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 10 client.84575.objecter resend_mon_ops 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f3195d1f640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f31901c0c90 0x7f319011f990 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f3190068a00 tx=0x7f318400f9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3184019070 con 0x7f31901c0c90 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 10 monclient: dump: 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.325 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 10 client.84575.objecter ms_handle_connect 0x7f317c03fdb0 2024-06-11T19:18:14.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f3196d21640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f317c03fdb0 0x7f317c042270 secure :-1 s=READY pgs=4189 cs=0 l=1 rev1=1 crypto rx=0x7f3180036000 tx=0x7f3180008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3184005320 con 0x7f31901c0c90 2024-06-11T19:18:14.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:14.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:18:14.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f31901c0c90 msgr2=0x7f319011f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f31901c0c90 0x7f319011f990 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f3190068a00 tx=0x7f318400f9d0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f31901c0c90 msgr2=0x7f319011f990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:14.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.301+0000 7f318effd640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f31901c0c90 0x7f319011f990 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.307+0000 7f3198784640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:18:14.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3198784640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f319011fed0 0x7f319009f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3198784640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3198784640 1 -- 172.21.15.98:0/2039201004 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f31901202b0 con 0x7f319011fed0 2024-06-11T19:18:14.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3195d1f640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f319011fed0 0x7f319009f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3195d1f640 10 monclient: get_auth_request con 0x7f319011fed0 auth_method 0 2024-06-11T19:18:14.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3195d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3195d1f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3195d1f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3195d1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3195d1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3195d1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3195d1f640 10 monclient(hunting): handle_auth_done global_id 32329 payload 824 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3195d1f640 1 -- 172.21.15.98:0/2039201004 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f319009e6b0 con 0x7f319011fed0 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f3195d1f640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f319011fed0 0x7f319009f080 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f3190068a00 tx=0x7f3184000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f318effd640 10 client.84575.objecter ms_handle_connect 0x7f319011fed0 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f318effd640 10 client.84575.objecter resend_mon_ops 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3184019040 con 0x7f319011fed0 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f318effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f318effd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.308+0000 7f318effd640 10 monclient: dump: 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.330 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9432 (secure 0 0 0) 0x7f3184006ac0 con 0x7f319011fed0 2024-06-11T19:18:14.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f318effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:18:14.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f318effd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:18:14.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f319011fed0 msgr2=0x7f319009f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f318effd640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f319011fed0 0x7f319009f080 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f3190068a00 tx=0x7f3184000ed0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f318effd640 1 -- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f319011fed0 msgr2=0x7f319009f080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:14.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f318effd640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f319011fed0 0x7f319009f080 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.332 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:18:14.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:18:14.332 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:18:14.332 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:18:14.332 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 50, 2024-06-11T19:18:14.332 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 544527, 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:18:14.333 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:18:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:18:14.335 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:18:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:18:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.338 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.340 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.342 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:18:14.342 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:18:14.342 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.342 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.342 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.342 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:18:14.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:18:14.342 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:18:14.343 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:18:14.343 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.343 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.343 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:18:14.345 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:18:14.345 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.345 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.345 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:18:14.346 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:18:14.351 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.352 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:14.353 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:18:14.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.354 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.354 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:18:14.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.354 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:14.354 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:14.354 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:18:14.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.354 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.354 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:18:14.354 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:18:14.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 10 librados: watch_flush enter 2024-06-11T19:18:14.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 10 librados: watch_flush exit 2024-06-11T19:18:14.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 20 client.84575.objecter shutdown clearing up homeless session... 2024-06-11T19:18:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 10 client.84575.objecter successfully canceled tick 2024-06-11T19:18:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 1 -- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f317c03fdb0 msgr2=0x7f317c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f317c03fdb0 0x7f317c042270 secure :-1 s=READY pgs=4189 cs=0 l=1 rev1=1 crypto rx=0x7f3180036000 tx=0x7f3180008030 comp rx=0 tx=0).stop 2024-06-11T19:18:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 10 monclient: shutdown 2024-06-11T19:18:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 1 -- 172.21.15.98:0/2039201004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901347f0 msgr2=0x7f31901336d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 1 --2- 172.21.15.98:0/2039201004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901347f0 0x7f31901336d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f318800efc0 tx=0x7f318800c490 comp rx=0 tx=0).stop 2024-06-11T19:18:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3196d21640 1 -- 172.21.15.98:0/2039201004 reap_dead start 2024-06-11T19:18:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3196d21640 1 -- 172.21.15.98:0/2039201004 reap_dead start 2024-06-11T19:18:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 1 -- 172.21.15.98:0/2039201004 shutdown_connections 2024-06-11T19:18:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 1 -- 172.21.15.98:0/2039201004 >> 172.21.15.98:0/2039201004 conn(0x7f31900a4780 msgr2=0x7f31901bf5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 1 -- 172.21.15.98:0/2039201004 shutdown_connections 2024-06-11T19:18:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 1 -- 172.21.15.98:0/2039201004 wait complete. 2024-06-11T19:18:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.309+0000 7f3198784640 1 librados: shutdown 2024-06-11T19:18:14.358 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:18:14.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-06-11T19:18:14.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:18:14.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.461+0000 7f368f654640 10 monclient: get_monmap_and_config 2024-06-11T19:18:14.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.461+0000 7f368f654640 10 monclient: build_initial_monmap 2024-06-11T19:18:14.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.461+0000 7f368f654640 10 monclient: monmap: 2024-06-11T19:18:14.461 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:14.461 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.461 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:18:14.462858+0000 2024-06-11T19:18:14.461 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:18:14.462858+0000 2024-06-11T19:18:14.461 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:14.461 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.461 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:18:14.462 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:18:14.462 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:14.462 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:14.462 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:14.462 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:14.462 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:14.462 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:14.462 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:14.462 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:18:14.462 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:18:14.463 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.462+0000 7f368f654640 1 Processor -- start 2024-06-11T19:18:14.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.462+0000 7f368f654640 1 -- start start 2024-06-11T19:18:14.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.462+0000 7f368f654640 10 monclient: init 2024-06-11T19:18:14.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:14.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 10 monclient: _add_conns ranks=[0,3,9,7,6,8,4,10,1,5,2] 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36880a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36880a2530 0x7f368813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 10 monclient(hunting): picked mon.noname-g con 0x7f36880a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f368813e1a0 0x7f3688140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 10 monclient(hunting): picked mon.noname-a con 0x7f368813e1a0 addr v2:172.21.15.98:3300/0 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36880a7140 con 0x7f36880a4920 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f36880a2b50 con 0x7f368813e1a0 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f36880a2e30 con 0x7f36880a2530 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368f654640 10 monclient(hunting): authenticate will time out at 1304.270261s 2024-06-11T19:18:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f368813e1a0 0x7f3688140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368d3f0640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f36880a4920 0x7f36880a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46538/0 (socket says 172.21.15.98:46538) 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368d3f0640 1 -- 172.21.15.98:0/567935987 learned_addr learned my addr 172.21.15.98:0/567935987 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f368813e1a0 0x7f3688140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34482/0 (socket says 172.21.15.98:34482) 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 10 monclient(hunting): get_auth_request con 0x7f368813e1a0 auth_method 0 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368cbef640 1 -- 172.21.15.98:0/567935987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36880a2530 msgr2=0x7f368813dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368cbef640 1 --2- 172.21.15.98:0/567935987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36880a2530 0x7f368813dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 10 monclient(hunting): handle_auth_done global_id 65214 payload 274 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 1 -- 172.21.15.98:0/567935987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36880a2530 msgr2=0x7f368813dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 1 --2- 172.21.15.98:0/567935987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36880a2530 0x7f368813dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 1 -- 172.21.15.98:0/567935987 >> v1:172.21.15.98:6789/0 conn(0x7f36880a4920 legacy=0x7f36880a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 10 monclient: _finish_hunting 0 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 1 monclient: found mon.noname-a 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f368dbf1640 1 -- 172.21.15.98:0/567935987 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36880a3110 con 0x7f368813e1a0 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f367ffff640 1 -- 172.21.15.98:0/567935987 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2032036245 0 0) 0x7f36880a7140 con 0x7f36880a4920 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.463+0000 7f367ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f368dbf1640 1 --2- 172.21.15.98:0/567935987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f368813e1a0 0x7f3688140590 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3680009d90 tx=0x7f368002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef1020aa5b95b4b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 1 -- 172.21.15.98:0/567935987 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3680039070 con 0x7f368813e1a0 2024-06-11T19:18:14.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 10 monclient: dump: 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.475 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 10 monclient: _finish_auth 0 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f368f654640 5 monclient: authenticate success, global_id 65214 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f368f654640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 1 -- 172.21.15.98:0/567935987 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f368002cdd0 con 0x7f368813e1a0 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f368f654640 10 monclient: get_monmap_and_config success 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f368f654640 4 set_mon_vals no callback set 2024-06-11T19:18:14.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 1 -- 172.21.15.98:0/567935987 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f36800327b0 con 0x7f368813e1a0 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.464+0000 7f367ffff640 10 monclient: dump: 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.465+0000 7f368f654640 10 monclient: shutdown 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.465+0000 7f368f654640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.465+0000 7f368f654640 1 -- 172.21.15.98:0/567935987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f368813e1a0 msgr2=0x7f3688140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.465+0000 7f368f654640 1 --2- 172.21.15.98:0/567935987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f368813e1a0 0x7f3688140590 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3680009d90 tx=0x7f368002bdb0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.465+0000 7f368f654640 1 -- 172.21.15.98:0/567935987 shutdown_connections 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.465+0000 7f368f654640 1 --2- 172.21.15.98:0/567935987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f368813e1a0 0x7f3688140590 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.465+0000 7f368f654640 1 --2- 172.21.15.98:0/567935987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36880a2530 0x7f368813dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.465+0000 7f368f654640 1 -- 172.21.15.98:0/567935987 >> 172.21.15.98:0/567935987 conn(0x7f36880ab850 msgr2=0x7f36880a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:14.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.465+0000 7f368f654640 1 -- 172.21.15.98:0/567935987 shutdown_connections 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.465+0000 7f368f654640 1 -- 172.21.15.98:0/567935987 wait complete. 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.465+0000 7f368f654640 10 monclient: build_initial_monmap 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.465+0000 7f368f654640 10 monclient: monmap: 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:14.479 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 1 librados: starting msgr at 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 1 librados: starting objecter 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 1 Processor -- start 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 1 -- start start 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 1 librados: setting wanted keys 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 1 librados: calling monclient init 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 10 monclient: init 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:14.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 10 monclient: _add_conns ranks=[8,4,1,5,7,2,0,3,6] 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f36880a2530 0x7f36881cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 10 monclient(hunting): picked mon.noname-e con 0x7f36880a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f36880a4920 0x7f3688136930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 10 monclient(hunting): picked mon.noname-c con 0x7f36880a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f368813e1a0 0x7f3688136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 10 monclient(hunting): picked mon.noname-f con 0x7f368813e1a0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3688138e50 con 0x7f368813e1a0 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f36880a84b0 con 0x7f36880a4920 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f36880a6da0 con 0x7f36880a2530 2024-06-11T19:18:14.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368f654640 10 monclient(hunting): authenticate will time out at 1304.273631s 2024-06-11T19:18:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f36880a2530 0x7f36881cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f36880a2530 0x7f36881cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:41576/0 (socket says 172.21.15.98:41576) 2024-06-11T19:18:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 1 -- 172.21.15.98:0/3527974843 learned_addr learned my addr 172.21.15.98:0/3527974843 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 10 monclient(hunting): get_auth_request con 0x7f36880a2530 auth_method 0 2024-06-11T19:18:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368cbef640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f36880a4920 0x7f3688136930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368cbef640 10 monclient(hunting): get_auth_request con 0x7f36880a4920 auth_method 0 2024-06-11T19:18:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368cbef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368cbef640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368cbef640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368cbef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368cbef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368cbef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 10 monclient(hunting): handle_auth_done global_id 94256 payload 824 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 1 -- 172.21.15.98:0/3527974843 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f36880a4920 msgr2=0x7f3688136930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f36880a4920 0x7f3688136930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 1 -- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f368813e1a0 msgr2=0x7f3688136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f368813e1a0 0x7f3688136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 10 monclient: _finish_hunting 0 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 1 monclient: found mon.noname-e 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.466+0000 7f368d3f0640 1 -- 172.21.15.98:0/3527974843 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3688122ab0 con 0x7f36880a2530 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f368d3f0640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f36880a2530 0x7f36881cd4a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f368400c450 tx=0x7f368400c910 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 10 client.?.objecter ms_handle_connect 0x7f36880a2530 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3684018070 con 0x7f36880a2530 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 10 monclient: dump: 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.484 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.485 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 10 monclient: _finish_auth 0 2024-06-11T19:18:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 10 monclient: _check_auth_tickets 2024-06-11T19:18:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36840047e0 con 0x7f36880a2530 2024-06-11T19:18:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f368f654640 5 monclient: authenticate success, global_id 94256 2024-06-11T19:18:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f368f654640 10 monclient: _renew_subs 2024-06-11T19:18:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f368f654640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:18:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f368f654640 1 -- 172.21.15.98:0/3527974843 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3688122d20 con 0x7f36880a2530 2024-06-11T19:18:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3684004e00 con 0x7f36880a2530 2024-06-11T19:18:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367dffb640 10 monclient: dump: 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f367f7fe640 4 set_mon_vals no callback set 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f368f654640 10 client.94256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f368f654640 10 monclient: _renew_subs 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f368f654640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f368f654640 1 -- 172.21.15.98:0/3527974843 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f36881cd9e0 con 0x7f36880a2530 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.467+0000 7f368f654640 1 librados: init done 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3684004990 con 0x7f36880a2530 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368f654640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368f654640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813bbc0 0x7f368813bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368f654640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368f654640 1 -- 172.21.15.98:0/3527974843 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f3688050830 con 0x7f368813bbc0 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f367dffb640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f366c03fdb0 0x7f366c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(349..349 src has 1..349) ==== 5845+0+0 (secure 0 0 0) 0x7f3684050ba0 con 0x7f36880a2530 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f367dffb640 10 client.94256.objecter ms_dispatch 0x7f3688000b90 osd_map(349..349 src has 1..349) 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f367dffb640 3 client.94256.objecter handle_osd_map got epochs [349,349] > 0 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f367dffb640 3 client.94256.objecter handle_osd_map decoding full epoch 349 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368cbef640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813bbc0 0x7f368813bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368cbef640 10 monclient: get_auth_request con 0x7f368813bbc0 auth_method 0 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368cbef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368cbef640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368cbef640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368dbf1640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f366c03fdb0 0x7f366c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f367dffb640 20 client.94256.objecter dump_active .. 0 homeless 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368dbf1640 10 monclient: get_auth_request con 0x7f366c03fdb0 auth_method 0 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368cbef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368cbef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.468+0000 7f368cbef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f368cbef640 10 monclient(hunting): handle_auth_done global_id 23144 payload 824 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f368cbef640 1 -- 172.21.15.98:0/3527974843 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f36880a0a70 con 0x7f368813bbc0 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 10 client.94256.objecter ms_handle_connect 0x7f366c03fdb0 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f368dbf1640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f366c03fdb0 0x7f366c042270 secure :-1 s=READY pgs=4190 cs=0 l=1 rev1=1 crypto rx=0x7f3680036000 tx=0x7f368002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 10 client.94256.objecter ms_handle_connect 0x7f368813bbc0 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 10 client.94256.objecter resend_mon_ops 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f368cbef640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813bbc0 0x7f368813bfa0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f3688073fc0 tx=0x7f3678009b10 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3678016070 con 0x7f368813bbc0 2024-06-11T19:18:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 10 monclient: dump: 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.492 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f367800e770 con 0x7f368813bbc0 2024-06-11T19:18:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:18:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813bbc0 msgr2=0x7f368813bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813bbc0 0x7f368813bfa0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f3688073fc0 tx=0x7f3678009b10 comp rx=0 tx=0).stop 2024-06-11T19:18:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813bbc0 msgr2=0x7f368813bfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.469+0000 7f367dffb640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813bbc0 0x7f368813bfa0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368f654640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:18:14.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368f654640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813c4e0 0x7f368809f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368f654640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368f654640 1 -- 172.21.15.98:0/3527974843 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f3688121710 con 0x7f368813c4e0 2024-06-11T19:18:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368cbef640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813c4e0 0x7f368809f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368cbef640 10 monclient: get_auth_request con 0x7f368813c4e0 auth_method 0 2024-06-11T19:18:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368cbef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368cbef640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368cbef640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368cbef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368cbef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368cbef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368cbef640 10 monclient(hunting): handle_auth_done global_id 23153 payload 824 2024-06-11T19:18:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368cbef640 1 -- 172.21.15.98:0/3527974843 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f36880a4d00 con 0x7f368813c4e0 2024-06-11T19:18:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f368cbef640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813c4e0 0x7f368809f8f0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f3688073fc0 tx=0x7f36780059d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f367dffb640 10 client.94256.objecter ms_handle_connect 0x7f368813c4e0 2024-06-11T19:18:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.476+0000 7f367dffb640 10 client.94256.objecter resend_mon_ops 2024-06-11T19:18:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3678016040 con 0x7f368813c4e0 2024-06-11T19:18:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f367dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f367dffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:18:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f367dffb640 10 monclient: dump: 2024-06-11T19:18:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9602 (secure 0 0 0) 0x7f367801d0d0 con 0x7f368813c4e0 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f367dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f367dffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:18:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813c4e0 msgr2=0x7f368809f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f367dffb640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813c4e0 0x7f368809f8f0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f3688073fc0 tx=0x7f36780059d0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f367dffb640 1 -- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813c4e0 msgr2=0x7f368809f8f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:14.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f367dffb640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f368813c4e0 0x7f368809f8f0 secure :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f3688073fc0 tx=0x7f36780059d0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.498 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:18:14.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:18:14.498 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:18:14.498 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:18:14.498 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 50, 2024-06-11T19:18:14.498 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 544671, 2024-06-11T19:18:14.498 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:18:14.499 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.500 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:18:14.502 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:14.503 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.505 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:18:14.511 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.513 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:18:14.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:18:14.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:18:14.513 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.514 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:18:14.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:18:14.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:18:14.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:18:14.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:18:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:18:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:18:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.519 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:14.520 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:14.521 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:14.522 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:14.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.522 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.522 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:18:14.522 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:18:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 10 librados: watch_flush enter 2024-06-11T19:18:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 10 librados: watch_flush exit 2024-06-11T19:18:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 20 client.94256.objecter shutdown clearing up homeless session... 2024-06-11T19:18:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 10 client.94256.objecter successfully canceled tick 2024-06-11T19:18:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 1 -- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f366c03fdb0 msgr2=0x7f366c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f366c03fdb0 0x7f366c042270 secure :-1 s=READY pgs=4190 cs=0 l=1 rev1=1 crypto rx=0x7f3680036000 tx=0x7f368002e000 comp rx=0 tx=0).stop 2024-06-11T19:18:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 10 monclient: shutdown 2024-06-11T19:18:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 1 -- 172.21.15.98:0/3527974843 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f36880a2530 msgr2=0x7f36881cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 1 --2- 172.21.15.98:0/3527974843 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f36880a2530 0x7f36881cd4a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f368400c450 tx=0x7f368400c910 comp rx=0 tx=0).stop 2024-06-11T19:18:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368dbf1640 1 -- 172.21.15.98:0/3527974843 reap_dead start 2024-06-11T19:18:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368dbf1640 1 -- 172.21.15.98:0/3527974843 reap_dead start 2024-06-11T19:18:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 1 -- 172.21.15.98:0/3527974843 shutdown_connections 2024-06-11T19:18:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 1 -- 172.21.15.98:0/3527974843 >> 172.21.15.98:0/3527974843 conn(0x7f36880ab850 msgr2=0x7f36880a8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 1 -- 172.21.15.98:0/3527974843 shutdown_connections 2024-06-11T19:18:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 1 -- 172.21.15.98:0/3527974843 wait complete. 2024-06-11T19:18:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.477+0000 7f368f654640 1 librados: shutdown 2024-06-11T19:18:14.525 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:18:14.525 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-06-11T19:18:14.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:18:14.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.625+0000 7fd571b3f640 10 monclient: get_monmap_and_config 2024-06-11T19:18:14.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.625+0000 7fd571b3f640 10 monclient: build_initial_monmap 2024-06-11T19:18:14.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.625+0000 7fd571b3f640 10 monclient: monmap: 2024-06-11T19:18:14.625 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:14.625 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.625 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:18:14.626838+0000 2024-06-11T19:18:14.625 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:18:14.626838+0000 2024-06-11T19:18:14.625 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:14.625 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.625 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:18:14.625 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:18:14.626 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:14.626 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:14.626 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:14.626 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:14.626 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:14.626 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:14.626 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:14.626 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:18:14.626 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:18:14.626 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.626+0000 7fd571b3f640 1 Processor -- start 2024-06-11T19:18:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.626+0000 7fd571b3f640 1 -- start start 2024-06-11T19:18:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.626+0000 7fd571b3f640 10 monclient: init 2024-06-11T19:18:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 10 monclient: _add_conns ranks=[3,5,7,1,9,8,0,2,6,10,4] 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd56c0a4920 0x7fd56c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 10 monclient(hunting): picked mon.noname-g con 0x7fd56c0a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd56c0a5240 0x7fd56c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 10 monclient(hunting): picked mon.noname-h con 0x7fd56c0a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c0a2530 0x7fd56c0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 10 monclient(hunting): picked mon.noname-i con 0x7fd56c0a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd56c0a7140 con 0x7fd56c0a4920 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd56c0a2b50 con 0x7fd56c0a5240 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd56c0a2e30 con 0x7fd56c0a2530 2024-06-11T19:18:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd571b3f640 10 monclient(hunting): authenticate will time out at 1304.434251s 2024-06-11T19:18:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56b7fe640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd56c0a4920 msgr2=0x7fd56c0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:18:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56b7fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd56c0a4920 0x7fd56c0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56affd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd56c0a5240 0x7fd56c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c0a2530 0x7fd56c0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56affd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd56c0a5240 0x7fd56c0a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:35894/0 (socket says 172.21.15.98:35894) 2024-06-11T19:18:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 10 monclient(hunting): get_auth_request con 0x7fd56c0a2530 auth_method 0 2024-06-11T19:18:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56affd640 1 -- 172.21.15.98:0/1108748941 learned_addr learned my addr 172.21.15.98:0/1108748941 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56affd640 10 monclient(hunting): get_auth_request con 0x7fd56c0a5240 auth_method 0 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 10 monclient(hunting): handle_auth_done global_id 84382 payload 274 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 1 -- 172.21.15.98:0/1108748941 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd56c0a5240 msgr2=0x7fd56c0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:18:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 1 --2- 172.21.15.98:0/1108748941 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd56c0a5240 0x7fd56c0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 1 -- 172.21.15.98:0/1108748941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd56c0a4920 msgr2=0x7fd56c0a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:18:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 1 --2- 172.21.15.98:0/1108748941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd56c0a4920 0x7fd56c0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:18:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 1 monclient: found mon.noname-i 2024-06-11T19:18:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:18:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56bfff640 1 -- 172.21.15.98:0/1108748941 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56c0a3110 con 0x7fd56c0a2530 2024-06-11T19:18:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.627+0000 7fd56affd640 1 --2- 172.21.15.98:0/1108748941 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd56c0a5240 0x7fd56c0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:18:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd56bfff640 1 --2- 172.21.15.98:0/1108748941 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c0a2530 0x7fd56c0a5ee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd564009dc0 tx=0x7fd56402bce0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=5e4bf4d4203faf4 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd56a7fc640 1 -- 172.21.15.98:0/1108748941 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd564039070 con 0x7fd56c0a2530 2024-06-11T19:18:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd56a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd56a7fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd56a7fc640 10 monclient: dump: 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd56a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd56a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd56a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd571b3f640 5 monclient: authenticate success, global_id 84382 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd56a7fc640 1 -- 172.21.15.98:0/1108748941 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd56402cd00 con 0x7fd56c0a2530 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd571b3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd56a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd571b3f640 10 monclient: get_monmap_and_config success 2024-06-11T19:18:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd571b3f640 4 set_mon_vals no callback set 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd56a7fc640 1 -- 172.21.15.98:0/1108748941 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5640327b0 con 0x7fd56c0a2530 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.628+0000 7fd56a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd56a7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd56a7fc640 10 monclient: dump: 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd571b3f640 10 monclient: shutdown 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd571b3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd571b3f640 1 -- 172.21.15.98:0/1108748941 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c0a2530 msgr2=0x7fd56c0a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd571b3f640 1 --2- 172.21.15.98:0/1108748941 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c0a2530 0x7fd56c0a5ee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd564009dc0 tx=0x7fd56402bce0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd571b3f640 1 -- 172.21.15.98:0/1108748941 shutdown_connections 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd571b3f640 1 --2- 172.21.15.98:0/1108748941 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c0a2530 0x7fd56c0a5ee0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd571b3f640 1 --2- 172.21.15.98:0/1108748941 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd56c0a5240 0x7fd56c0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd571b3f640 1 --2- 172.21.15.98:0/1108748941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd56c0a4920 0x7fd56c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd571b3f640 1 -- 172.21.15.98:0/1108748941 >> 172.21.15.98:0/1108748941 conn(0x7fd56c0ab850 msgr2=0x7fd56c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd571b3f640 1 -- 172.21.15.98:0/1108748941 shutdown_connections 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.629+0000 7fd571b3f640 1 -- 172.21.15.98:0/1108748941 wait complete. 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 10 monclient: build_initial_monmap 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 10 monclient: monmap: 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:18:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:18:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:18:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:18:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:18:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:18:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:18:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:18:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:18:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:18:14.646 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 1 librados: starting msgr at 2024-06-11T19:18:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 1 librados: starting objecter 2024-06-11T19:18:14.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 1 Processor -- start 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 1 -- start start 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 1 librados: setting wanted keys 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 1 librados: calling monclient init 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 10 monclient: init 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 10 monclient: _reopen_session rank -1 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 10 monclient: _add_conns ranks=[5,2,3,1,7,0,8,4,6] 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd56c0a2530 0x7fd56c1cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 10 monclient(hunting): picked mon.noname-h con 0x7fd56c0a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd56c0a4920 0x7fd56c1ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.630+0000 7fd571b3f640 10 monclient(hunting): picked mon.noname-b con 0x7fd56c0a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd56c0a5240 0x7fd56c1250c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 10 monclient(hunting): picked mon.noname-g con 0x7fd56c0a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:18:14.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd56c0a6da0 con 0x7fd56c0a4920 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd56c138e50 con 0x7fd56c0a5240 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd56c0a6420 con 0x7fd56c0a2530 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 10 monclient(hunting): _renew_subs 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 10 monclient(hunting): authenticate will time out at 1304.438032s 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56bfff640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd56c0a5240 msgr2=0x7fd56c1250c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56bfff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd56c0a5240 0x7fd56c1250c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd56c0a4920 0x7fd56c1ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56b7fe640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd56c0a2530 0x7fd56c1cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd56c0a4920 0x7fd56c1ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45722/0 (socket says 172.21.15.98:45722) 2024-06-11T19:18:14.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 1 -- 172.21.15.98:0/90412434 learned_addr learned my addr 172.21.15.98:0/90412434 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 10 monclient(hunting): get_auth_request con 0x7fd56c0a4920 auth_method 0 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56b7fe640 10 monclient(hunting): get_auth_request con 0x7fd56c0a2530 auth_method 0 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 10 monclient(hunting): handle_auth_done global_id 84584 payload 824 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 1 -- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd56c0a2530 msgr2=0x7fd56c1cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd56c0a2530 0x7fd56c1cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 1 -- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd56c0a5240 msgr2=0x7fd56c1250c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd56c0a5240 0x7fd56c1250c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 10 monclient: _finish_hunting 0 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 1 monclient: found mon.noname-b 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 1 -- 172.21.15.98:0/90412434 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56c129a60 con 0x7fd56c0a4920 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56b7fe640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd56c0a2530 0x7fd56c1cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd56affd640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd56c0a4920 0x7fd56c1ce010 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00e8d0 tx=0x7fd55c00ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 10 client.?.objecter ms_handle_connect 0x7fd56c0a4920 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd55c0044a0 con 0x7fd56c0a4920 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 10 monclient: dump: 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.651 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 10 monclient: _finish_auth 0 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 10 monclient: _check_auth_tickets 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd55c00c940 con 0x7fd56c0a4920 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 5 monclient: authenticate success, global_id 84584 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 10 monclient: _renew_subs 2024-06-11T19:18:14.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 1 -- 172.21.15.98:0/90412434 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd56c1278f0 con 0x7fd56c0a4920 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd55c003610 con 0x7fd56c0a4920 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd569ffb640 4 set_mon_vals no callback set 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd563fff640 10 monclient: dump: 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.653 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 10 client.84584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 10 monclient: _renew_subs 2024-06-11T19:18:14.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 1 -- 172.21.15.98:0/90412434 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd56c0a90a0 con 0x7fd56c0a4920 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.631+0000 7fd571b3f640 1 librados: init done 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.632+0000 7fd571b3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.632+0000 7fd571b3f640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c128a20 0x7fd56c125600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.632+0000 7fd571b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.632+0000 7fd571b3f640 1 -- 172.21.15.98:0/90412434 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd56c0a6260 con 0x7fd56c128a20 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.632+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd55c004030 con 0x7fd56c0a4920 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.632+0000 7fd56b7fe640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c128a20 0x7fd56c125600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.632+0000 7fd56b7fe640 10 monclient: get_auth_request con 0x7fd56c128a20 auth_method 0 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.632+0000 7fd56b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.632+0000 7fd56b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.632+0000 7fd56b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd55003fdb0 0x7fd550042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd56b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd56b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd56b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(349..349 src has 1..349) ==== 5845+0+0 (secure 0 0 0) 0x7fd55c00f040 con 0x7fd56c0a4920 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 10 client.84584.objecter ms_dispatch 0x7fd56c000b90 osd_map(349..349 src has 1..349) 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 3 client.84584.objecter handle_osd_map got epochs [349,349] > 0 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 3 client.84584.objecter handle_osd_map decoding full epoch 349 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 20 client.84584.objecter dump_active .. 0 homeless 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd56b7fe640 10 monclient(hunting): handle_auth_done global_id 84391 payload 824 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd56bfff640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd55003fdb0 0x7fd550042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd56b7fe640 1 -- 172.21.15.98:0/90412434 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd56c1387f0 con 0x7fd56c128a20 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd56bfff640 10 monclient: get_auth_request con 0x7fd55003fdb0 auth_method 0 2024-06-11T19:18:14.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd56b7fe640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c128a20 0x7fd56c125600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd554003930 tx=0x7fd554005850 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 10 client.84584.objecter ms_handle_connect 0x7fd56c128a20 2024-06-11T19:18:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 10 client.84584.objecter resend_mon_ops 2024-06-11T19:18:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd554016070 con 0x7fd56c128a20 2024-06-11T19:18:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:18:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 10 monclient: dump: 2024-06-11T19:18:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.657 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 10 client.84584.objecter ms_handle_connect 0x7fd55003fdb0 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd56bfff640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd55003fdb0 0x7fd550042270 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 crypto rx=0x7fd564036000 tx=0x7fd564030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd55400ea20 con 0x7fd56c128a20 2024-06-11T19:18:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:18:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:18:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c128a20 msgr2=0x7fd56c125600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c128a20 0x7fd56c125600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd554003930 tx=0x7fd554005850 comp rx=0 tx=0).stop 2024-06-11T19:18:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c128a20 msgr2=0x7fd56c125600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:14.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.633+0000 7fd563fff640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c128a20 0x7fd56c125600 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd554003930 tx=0x7fd554005850 comp rx=0 tx=0).stop 2024-06-11T19:18:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd571b3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:18:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd571b3f640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c125b40 0x7fd56c125f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:18:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd571b3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:18:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd571b3f640 1 -- 172.21.15.98:0/90412434 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd56c069750 con 0x7fd56c125b40 2024-06-11T19:18:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd56b7fe640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c125b40 0x7fd56c125f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:18:14.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd56b7fe640 10 monclient: get_auth_request con 0x7fd56c125b40 auth_method 0 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd56b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd56b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd56b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd56b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd56b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd56b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd56b7fe640 10 monclient(hunting): handle_auth_done global_id 84400 payload 824 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.640+0000 7fd56b7fe640 1 -- 172.21.15.98:0/90412434 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd56c0a66f0 con 0x7fd56c125b40 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 10 client.84584.objecter ms_handle_connect 0x7fd56c125b40 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 10 client.84584.objecter resend_mon_ops 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd56b7fe640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c125b40 0x7fd56c125f50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd554003930 tx=0x7fd5540059d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd554016040 con 0x7fd56c125b40 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:18:14.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 10 monclient: dump: 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:18:14.664 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:18:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:18:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:18:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:18:14.665 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:18:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9262 (secure 0 0 0) 0x7fd554005a40 con 0x7fd56c125b40 2024-06-11T19:18:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:18:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:18:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c125b40 msgr2=0x7fd56c125f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c125b40 0x7fd56c125f50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd554003930 tx=0x7fd5540059d0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 1 -- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c125b40 msgr2=0x7fd56c125f50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:18:14.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd563fff640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd56c125b40 0x7fd56c125f50 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd554003930 tx=0x7fd5540059d0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.666 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:18:14.666 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:18:14.666 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:18:14.666 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:18:14.666 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 50, 2024-06-11T19:18:14.666 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 166411, 2024-06-11T19:18:14.666 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:18:14.666 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:18:14.666 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:18:14.666 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:18:14.666 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:18:14.666 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:14.667 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:14.668 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:18:14.669 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:18:14.670 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.671 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:18:14.672 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:18:14.673 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.676 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.678 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.679 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:18:14.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:18:14.686 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.687 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:18:14.688 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:18:14.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:18:14.688 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:18:14.688 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:18:14.688 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:18:14.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:18:14.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:18:14.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:18:14.688 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:18:14.688 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:18:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd571b3f640 10 librados: watch_flush enter 2024-06-11T19:18:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd571b3f640 10 librados: watch_flush exit 2024-06-11T19:18:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd571b3f640 20 client.84584.objecter shutdown clearing up homeless session... 2024-06-11T19:18:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.641+0000 7fd571b3f640 10 client.84584.objecter successfully canceled tick 2024-06-11T19:18:14.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd571b3f640 1 -- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd55003fdb0 msgr2=0x7fd550042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd571b3f640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd55003fdb0 0x7fd550042270 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 crypto rx=0x7fd564036000 tx=0x7fd564030000 comp rx=0 tx=0).stop 2024-06-11T19:18:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd571b3f640 10 monclient: shutdown 2024-06-11T19:18:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd571b3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:18:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd571b3f640 1 -- 172.21.15.98:0/90412434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd56c0a4920 msgr2=0x7fd56c1ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:18:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd571b3f640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd56c0a4920 0x7fd56c1ce010 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00e8d0 tx=0x7fd55c00ed90 comp rx=0 tx=0).stop 2024-06-11T19:18:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd56bfff640 1 -- 172.21.15.98:0/90412434 reap_dead start 2024-06-11T19:18:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd571b3f640 1 -- 172.21.15.98:0/90412434 shutdown_connections 2024-06-11T19:18:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd571b3f640 1 --2- 172.21.15.98:0/90412434 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd56c0a4920 0x7fd56c1ce010 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:18:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd571b3f640 1 -- 172.21.15.98:0/90412434 >> 172.21.15.98:0/90412434 conn(0x7fd56c0ab850 msgr2=0x7fd56c09c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:18:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd571b3f640 1 -- 172.21.15.98:0/90412434 shutdown_connections 2024-06-11T19:18:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd571b3f640 1 -- 172.21.15.98:0/90412434 wait complete. 2024-06-11T19:18:14.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:18:14.642+0000 7fd571b3f640 1 librados: shutdown 2024-06-11T19:18:14.692 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-06-11T19:18:14.692 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-06-11T19:18:46.916 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-06-11T19:18:46.916+0000 7f87b8390640 -1 snap_mapper.add_oid found existing snaps mapped on 300:98cba08d:test-rados-api-smithi117-36854-80::foo:2, removing 2024-06-11T19:18:46.917 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-06-11T19:18:46.916+0000 7fc6c5e8a640 -1 snap_mapper.add_oid found existing snaps mapped on 300:98cba08d:test-rados-api-smithi117-36854-80::foo:2, removing 2024-06-11T19:19:17.896 INFO:tasks.workunit.client.0.smithi117.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (19000 ms) 2024-06-11T19:19:17.896 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-06-11T19:19:17.896 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (37979 ms) 2024-06-11T19:19:17.897 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-06-11T19:19:17.897 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3014 ms) 2024-06-11T19:19:17.897 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-06-11T19:19:17.897 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.897 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3006 ms) 2024-06-11T19:19:17.897 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-06-11T19:19:17.897 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (2967 ms) 2024-06-11T19:19:17.897 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-06-11T19:19:17.897 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.897 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3015 ms) 2024-06-11T19:19:17.897 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4012 ms) 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (37980 ms) 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17098 ms) 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4052 ms) 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-06-11T19:19:17.898 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3007 ms) 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24510 ms) 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (25356 ms) 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5022 ms) 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3930 ms) 2024-06-11T19:19:17.899 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: waiting for scrubs... 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: done waiting 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25040 ms) 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9981 ms) 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (8985 ms) 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9003 ms) 2024-06-11T19:19:17.900 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-06-11T19:19:44.693 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-06-11T19:19:44.694 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-06-11T19:19:44.694 INFO:tasks.ceph.mon.g:Restarting daemon 2024-06-11T19:19:44.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-06-11T19:19:44.698 INFO:tasks.ceph.mon.g:Started 2024-06-11T19:19:44.698 INFO:tasks.mon_thrash:Sending STOP to mon h 2024-06-11T19:19:44.698 INFO:tasks.ceph.mon.h:Sent signal 19 2024-06-11T19:19:44.699 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-06-11T19:19:44.810 INFO:tasks.ceph.mon.g.smithi117.stderr:2024-06-11T19:19:44.809+0000 7fec606a1b00 -1 mon.g@-1(???).log v0 log_external_backlog rewinding external_log_to from 370 -> 0 (sync_force? mon rebuild?) 2024-06-11T19:19:59.699 INFO:tasks.mon_thrash:Sending CONT to mon h 2024-06-11T19:19:59.700 INFO:tasks.ceph.mon.h:Sent signal 18 2024-06-11T19:19:59.700 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:19:59.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:19:59.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.817+0000 7fc9508a1640 10 monclient: get_monmap_and_config 2024-06-11T19:19:59.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.817+0000 7fc9508a1640 10 monclient: build_initial_monmap 2024-06-11T19:19:59.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.817+0000 7fc9508a1640 10 monclient: monmap: 2024-06-11T19:19:59.816 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:19:59.816 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:19:59.816 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:19:59.818101+0000 2024-06-11T19:19:59.816 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:19:59.818101+0000 2024-06-11T19:19:59.817 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:19:59.817 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:19:59.817 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:19:59.817 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:19:59.817 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:19:59.817 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:19:59.817 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:19:59.817 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:19:59.817 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:19:59.817 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:19:59.817 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:19:59.818 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:19:59.818 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:19:59.818 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:19:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.817+0000 7fc9508a1640 1 Processor -- start 2024-06-11T19:19:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.817+0000 7fc9508a1640 1 -- start start 2024-06-11T19:19:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.817+0000 7fc9508a1640 10 monclient: init 2024-06-11T19:19:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:19:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 10 monclient: _add_conns ranks=[6,10,2,3,8,9,0,1,7,5,4] 2024-06-11T19:19:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc94c1347f0 0x7fc94c09e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:19:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 10 monclient(hunting): picked mon.noname-d con 0x7fc94c1347f0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:19:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc94c134bd0 0x7fc94c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:19:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 10 monclient(hunting): picked mon.noname-f con 0x7fc94c134bd0 addr v2:172.21.15.117:3300/0 2024-06-11T19:19:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 0x7fc94c0a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:19:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 10 monclient(hunting): picked mon.noname-b con 0x7fc94c135010 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:19:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:19:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fc94c0687f0 con 0x7fc94c134bd0 2024-06-11T19:19:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:19:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc94c09d410 con 0x7fc94c135010 2024-06-11T19:19:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:19:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc94c09d6f0 con 0x7fc94c1347f0 2024-06-11T19:19:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:19:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc9508a1640 10 monclient(hunting): authenticate will time out at 1409.625455s 2024-06-11T19:19:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc94c1347f0 0x7fc94c09e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:19:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94ad9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 0x7fc94c0a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:19:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94ad9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 0x7fc94c0a3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:46142/0 (socket says 172.21.15.98:46142) 2024-06-11T19:19:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94ad9d640 1 -- 172.21.15.98:0/1043825025 learned_addr learned my addr 172.21.15.98:0/1043825025 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:19:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94ad9d640 10 monclient(hunting): get_auth_request con 0x7fc94c135010 auth_method 0 2024-06-11T19:19:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:19:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:19:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:19:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 10 monclient(hunting): get_auth_request con 0x7fc94c1347f0 auth_method 0 2024-06-11T19:19:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:19:59.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:19:59.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:19:59.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:19:59.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:19:59.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:19:59.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:19:59.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:19:59.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:19:59.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc949d9b640 1 --2- 172.21.15.98:0/1043825025 >> v2:172.21.15.117:3300/0 conn(0x7fc94c134bd0 0x7fc94c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:19:59.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc949d9b640 10 monclient(hunting): get_auth_request con 0x7fc94c134bd0 auth_method 0 2024-06-11T19:19:59.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc949d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:19:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc949d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:19:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc949d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:19:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 10 monclient(hunting): handle_auth_done global_id 55275 payload 274 2024-06-11T19:19:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 1 -- 172.21.15.98:0/1043825025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 msgr2=0x7fc94c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:19:59.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 1 --2- 172.21.15.98:0/1043825025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 0x7fc94c0a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:19:59.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 1 -- 172.21.15.98:0/1043825025 >> v2:172.21.15.117:3300/0 conn(0x7fc94c134bd0 msgr2=0x7fc94c0a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:19:59.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 1 --2- 172.21.15.98:0/1043825025 >> v2:172.21.15.117:3300/0 conn(0x7fc94c134bd0 0x7fc94c0a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:19:59.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 10 monclient: _finish_hunting 0 2024-06-11T19:19:59.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 1 monclient: found mon.noname-d 2024-06-11T19:19:59.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:19:59.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:19:59.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94a59c640 1 -- 172.21.15.98:0/1043825025 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc94c09d9d0 con 0x7fc94c1347f0 2024-06-11T19:19:59.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.818+0000 7fc94ad9d640 1 --2- 172.21.15.98:0/1043825025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 0x7fc94c0a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:19:59.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94a59c640 1 --2- 172.21.15.98:0/1043825025 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc94c1347f0 0x7fc94c09e6b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009d90 tx=0x7fc93c02bd70 comp rx=0 tx=0).ready entity=mon.6 client_cookie=763ac3f1bf433b79 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:19:59.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 1 -- 172.21.15.98:0/1043825025 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc93c039070 con 0x7fc94c1347f0 2024-06-11T19:19:59.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:19:59.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:19:59.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 10 monclient: dump: 2024-06-11T19:19:59.835 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:19:59.836 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:19:59.836 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:19:59.836 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:19:59.836 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:19:59.836 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:19:59.836 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:19:59.837 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:19:59.837 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:19:59.837 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:19:59.837 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:19:59.837 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:19:59.837 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:19:59.838 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:19:59.838 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:19:59.838 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:19:59.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 10 monclient: _finish_auth 0 2024-06-11T19:19:59.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 10 monclient: _check_auth_tickets 2024-06-11T19:19:59.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:19:59.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 1 -- 172.21.15.98:0/1043825025 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc93c02cd90 con 0x7fc94c1347f0 2024-06-11T19:19:59.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:19:59.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 1 -- 172.21.15.98:0/1043825025 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc93c0327b0 con 0x7fc94c1347f0 2024-06-11T19:19:59.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc9508a1640 5 monclient: authenticate success, global_id 55275 2024-06-11T19:19:59.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc9508a1640 10 monclient: get_monmap_and_config success 2024-06-11T19:19:59.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc9508a1640 4 set_mon_vals no callback set 2024-06-11T19:19:59.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:19:59.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:19:59.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.819+0000 7fc94959a640 10 monclient: dump: 2024-06-11T19:19:59.840 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:19:59.840 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:19:59.840 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:19:59.840 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:19:59.840 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:19:59.841 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:19:59.841 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:19:59.841 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:19:59.841 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:19:59.841 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:19:59.841 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:19:59.841 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:19:59.841 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:19:59.841 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:19:59.841 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:19:59.841 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:19:59.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.820+0000 7fc9508a1640 10 monclient: shutdown 2024-06-11T19:19:59.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.820+0000 7fc9508a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:19:59.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.820+0000 7fc9508a1640 1 -- 172.21.15.98:0/1043825025 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc94c1347f0 msgr2=0x7fc94c09e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:19:59.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.820+0000 7fc9508a1640 1 --2- 172.21.15.98:0/1043825025 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc94c1347f0 0x7fc94c09e6b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009d90 tx=0x7fc93c02bd70 comp rx=0 tx=0).stop 2024-06-11T19:19:59.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.820+0000 7fc9508a1640 1 -- 172.21.15.98:0/1043825025 shutdown_connections 2024-06-11T19:19:59.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.820+0000 7fc9508a1640 1 --2- 172.21.15.98:0/1043825025 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 0x7fc94c0a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:19:59.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.820+0000 7fc9508a1640 1 --2- 172.21.15.98:0/1043825025 >> v2:172.21.15.117:3300/0 conn(0x7fc94c134bd0 0x7fc94c0a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:19:59.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.820+0000 7fc9508a1640 1 --2- 172.21.15.98:0/1043825025 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc94c1347f0 0x7fc94c09e6b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:19:59.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.820+0000 7fc9508a1640 1 -- 172.21.15.98:0/1043825025 >> 172.21.15.98:0/1043825025 conn(0x7fc94c0a4780 msgr2=0x7fc94c0a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:19:59.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.820+0000 7fc9508a1640 1 -- 172.21.15.98:0/1043825025 shutdown_connections 2024-06-11T19:19:59.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.820+0000 7fc9508a1640 1 -- 172.21.15.98:0/1043825025 wait complete. 2024-06-11T19:19:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.821+0000 7fc9508a1640 10 monclient: build_initial_monmap 2024-06-11T19:19:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.821+0000 7fc9508a1640 10 monclient: monmap: 2024-06-11T19:19:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:19:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:19:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:19:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:19:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:19:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:19:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:19:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:19:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:19:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.821+0000 7fc9508a1640 1 librados: starting msgr at 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.821+0000 7fc9508a1640 1 librados: starting objecter 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.821+0000 7fc9508a1640 1 Processor -- start 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.821+0000 7fc9508a1640 1 -- start start 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.821+0000 7fc9508a1640 1 librados: setting wanted keys 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.821+0000 7fc9508a1640 1 librados: calling monclient init 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.821+0000 7fc9508a1640 10 monclient: init 2024-06-11T19:19:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:19:59.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 10 monclient: _add_conns ranks=[4,5,2,7,1,0,6,3,8] 2024-06-11T19:19:59.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc94c1347f0 0x7fc94c1cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:19:59.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 10 monclient(hunting): picked mon.noname-c con 0x7fc94c1347f0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:19:59.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc94c134bd0 0x7fc94c133580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:19:59.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 10 monclient(hunting): picked mon.noname-h con 0x7fc94c134bd0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:19:59.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 0x7fc94c133ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:19:59.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 10 monclient(hunting): picked mon.noname-b con 0x7fc94c135010 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:19:59.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:19:59.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc94c068c40 con 0x7fc94c135010 2024-06-11T19:19:59.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:19:59.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc94c068ac0 con 0x7fc94c1347f0 2024-06-11T19:19:59.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:19:59.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc94c0a4110 con 0x7fc94c134bd0 2024-06-11T19:19:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:19:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc9508a1640 10 monclient(hunting): authenticate will time out at 1409.629303s 2024-06-11T19:19:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc94c1347f0 0x7fc94c1cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:19:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc94c1347f0 0x7fc94c1cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:41070/0 (socket says 172.21.15.98:41070) 2024-06-11T19:19:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94ad9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 0x7fc94c133ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:19:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 1 -- 172.21.15.98:0/3285468988 learned_addr learned my addr 172.21.15.98:0/3285468988 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:19:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 10 monclient(hunting): get_auth_request con 0x7fc94c1347f0 auth_method 0 2024-06-11T19:19:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:19:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:19:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:19:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94ad9d640 10 monclient(hunting): get_auth_request con 0x7fc94c135010 auth_method 0 2024-06-11T19:19:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:19:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:19:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:19:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc949d9b640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc94c134bd0 0x7fc94c133580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:19:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc949d9b640 10 monclient(hunting): get_auth_request con 0x7fc94c134bd0 auth_method 0 2024-06-11T19:19:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc949d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:19:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc949d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc949d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc949d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc949d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc949d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 10 monclient(hunting): handle_auth_done global_id 24106 payload 824 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 1 -- 172.21.15.98:0/3285468988 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc94c134bd0 msgr2=0x7fc94c133580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:19:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc94c134bd0 0x7fc94c133580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:19:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 1 -- 172.21.15.98:0/3285468988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 msgr2=0x7fc94c133ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:19:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 0x7fc94c133ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:19:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 10 monclient: _finish_hunting 0 2024-06-11T19:19:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 1 monclient: found mon.noname-c 2024-06-11T19:19:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:19:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:19:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 1 -- 172.21.15.98:0/3285468988 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc94c136160 con 0x7fc94c1347f0 2024-06-11T19:19:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94ad9d640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 0x7fc94c133ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:19:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc94a59c640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc94c1347f0 0x7fc94c1cde70 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fc93c037040 tx=0x7fc93c004a40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:19:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc93affd640 10 client.?.objecter ms_handle_connect 0x7fc94c1347f0 2024-06-11T19:19:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.822+0000 7fc93affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:19:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 1 -- 172.21.15.98:0/3285468988 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc93c039050 con 0x7fc94c1347f0 2024-06-11T19:19:59.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:19:59.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:19:59.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 10 monclient: dump: 2024-06-11T19:19:59.851 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:19:59.851 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:19:59.851 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:19:59.851 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:19:59.851 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:19:59.852 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:19:59.852 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:19:59.852 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:19:59.852 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:19:59.852 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:19:59.852 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:19:59.852 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:19:59.852 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:19:59.852 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:19:59.852 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:19:59.852 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 10 monclient: _finish_auth 0 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 10 monclient: _check_auth_tickets 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 1 -- 172.21.15.98:0/3285468988 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc93c045070 con 0x7fc94c1347f0 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc9508a1640 5 monclient: authenticate success, global_id 24106 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 1 -- 172.21.15.98:0/3285468988 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc93c0046c0 con 0x7fc94c1347f0 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc9508a1640 10 monclient: _renew_subs 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc9508a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc9508a1640 1 -- 172.21.15.98:0/3285468988 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc94c122c80 con 0x7fc94c1347f0 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:19:59.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc948d99640 4 set_mon_vals no callback set 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc93affd640 10 monclient: dump: 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:19:59.854 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:19:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:19:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:19:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:19:59.855 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:19:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc9508a1640 10 client.24106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:19:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc9508a1640 10 monclient: _renew_subs 2024-06-11T19:19:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc9508a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:19:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc9508a1640 1 -- 172.21.15.98:0/3285468988 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc94c1229a0 con 0x7fc94c1347f0 2024-06-11T19:19:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.823+0000 7fc9508a1640 1 librados: init done 2024-06-11T19:19:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc9508a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:19:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc9508a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:19:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc9508a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:19:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc93affd640 1 -- 172.21.15.98:0/3285468988 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fc93c004250 con 0x7fc94c1347f0 2024-06-11T19:19:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc9508a1640 1 -- 172.21.15.98:0/3285468988 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc94c074f50 con 0x7fc94c1347f0 2024-06-11T19:19:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc93affd640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc92c03fdb0 0x7fc92c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:19:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc93affd640 1 -- 172.21.15.98:0/3285468988 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(419..419 src has 1..419) ==== 6258+0+0 (secure 0 0 0) 0x7fc93c074090 con 0x7fc94c1347f0 2024-06-11T19:19:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc93affd640 10 client.24106.objecter ms_dispatch 0x7fc94c000b90 osd_map(419..419 src has 1..419) 2024-06-11T19:19:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc93affd640 3 client.24106.objecter handle_osd_map got epochs [419,419] > 0 2024-06-11T19:19:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc93affd640 3 client.24106.objecter handle_osd_map decoding full epoch 419 2024-06-11T19:19:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc93affd640 20 client.24106.objecter dump_active .. 0 homeless 2024-06-11T19:19:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc949d9b640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc92c03fdb0 0x7fc92c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:19:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.824+0000 7fc949d9b640 10 monclient: get_auth_request con 0x7fc92c03fdb0 auth_method 0 2024-06-11T19:19:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.825+0000 7fc93affd640 10 client.24106.objecter ms_handle_connect 0x7fc92c03fdb0 2024-06-11T19:19:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.825+0000 7fc949d9b640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc92c03fdb0 0x7fc92c042270 secure :-1 s=READY pgs=4195 cs=0 l=1 rev1=1 crypto rx=0x7fc94c122e90 tx=0x7fc940008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:19:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.829+0000 7fc93affd640 1 -- 172.21.15.98:0/3285468988 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc94c074f50 con 0x7fc94c1347f0 2024-06-11T19:19:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.829+0000 7fc93affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:19:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:19:59.829+0000 7fc93affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:00.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.085+0000 7fc9508a1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:20:00.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.086+0000 7fc9508a1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:20:00.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.086+0000 7fc9508a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:00.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.086+0000 7fc9508a1640 1 -- 172.21.15.98:0/3285468988 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc94c09e6b0 con 0x7fc94c1347f0 2024-06-11T19:20:00.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.086+0000 7fc93affd640 1 -- 172.21.15.98:0/3285468988 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fc93c04f020 con 0x7fc94c1347f0 2024-06-11T19:20:00.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.086+0000 7fc93affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:20:00.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.086+0000 7fc93affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:00.086 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":54,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:20:00.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 10 librados: watch_flush enter 2024-06-11T19:20:00.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 10 librados: watch_flush exit 2024-06-11T19:20:00.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 20 client.24106.objecter shutdown clearing up homeless session... 2024-06-11T19:20:00.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 10 client.24106.objecter successfully canceled tick 2024-06-11T19:20:00.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 -- 172.21.15.98:0/3285468988 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc92c03fdb0 msgr2=0x7fc92c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc92c03fdb0 0x7fc92c042270 secure :-1 s=READY pgs=4195 cs=0 l=1 rev1=1 crypto rx=0x7fc94c122e90 tx=0x7fc940008040 comp rx=0 tx=0).stop 2024-06-11T19:20:00.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 10 monclient: shutdown 2024-06-11T19:20:00.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:00.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 -- 172.21.15.98:0/3285468988 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc94c1347f0 msgr2=0x7fc94c1cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc94c1347f0 0x7fc94c1cde70 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fc93c037040 tx=0x7fc93c004a40 comp rx=0 tx=0).stop 2024-06-11T19:20:00.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 -- 172.21.15.98:0/3285468988 shutdown_connections 2024-06-11T19:20:00.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc92c03fdb0 0x7fc92c042270 unknown :-1 s=CLOSED pgs=4195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc94c135010 0x7fc94c133ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc94c134bd0 0x7fc94c133580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 --2- 172.21.15.98:0/3285468988 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc94c1347f0 0x7fc94c1cde70 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 -- 172.21.15.98:0/3285468988 >> 172.21.15.98:0/3285468988 conn(0x7fc94c0a4780 msgr2=0x7fc94c1bf770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:00.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 -- 172.21.15.98:0/3285468988 shutdown_connections 2024-06-11T19:20:00.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 -- 172.21.15.98:0/3285468988 wait complete. 2024-06-11T19:20:00.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.088+0000 7fc9508a1640 1 librados: shutdown 2024-06-11T19:20:00.098 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:20:00.098 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:20:00.099 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:20:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.200+0000 7f7dee087640 10 monclient: get_monmap_and_config 2024-06-11T19:20:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.200+0000 7f7dee087640 10 monclient: build_initial_monmap 2024-06-11T19:20:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.200+0000 7f7dee087640 10 monclient: monmap: 2024-06-11T19:20:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:00.201311+0000 2024-06-11T19:20:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:00.201311+0000 2024-06-11T19:20:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:00.202 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 1 Processor -- start 2024-06-11T19:20:00.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 1 -- start start 2024-06-11T19:20:00.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 10 monclient: init 2024-06-11T19:20:00.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:00.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 10 monclient: _add_conns ranks=[7,0,2,3,6,9,8,5,4,1,10] 2024-06-11T19:20:00.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7de809d9f0 0x7f7de80a1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 10 monclient(hunting): picked mon.noname-i con 0x7f7de809d9f0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:00.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7de80a1cc0 addr v1:172.21.15.98:6789/0 2024-06-11T19:20:00.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7de80a20a0 0x7f7de813cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 10 monclient(hunting): picked mon.noname-b con 0x7f7de80a20a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:00.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7de809ddd0 con 0x7f7de80a1cc0 2024-06-11T19:20:00.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7de809f400 con 0x7f7de80a20a0 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f7de809f9c0 con 0x7f7de809d9f0 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7dee087640 10 monclient(hunting): authenticate will time out at 1410.008614s 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7de80a20a0 0x7f7de813cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de6ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7de80a1cc0 0x7f7de8139570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58070/0 (socket says 172.21.15.98:58070) 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de6ffd640 1 -- 172.21.15.98:0/1652125473 learned_addr learned my addr 172.21.15.98:0/1652125473 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de7fff640 10 monclient(hunting): get_auth_request con 0x7f7de80a20a0 auth_method 0 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de7fff640 10 monclient(hunting): handle_auth_done global_id 84611 payload 274 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de67fc640 1 -- 172.21.15.98:0/1652125473 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2685250995 0 0) 0x7f7de809ddd0 con 0x7f7de80a1cc0 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de7fff640 1 -- 172.21.15.98:0/1652125473 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7de809d9f0 msgr2=0x7f7de80a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de7fff640 1 --2- 172.21.15.98:0/1652125473 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7de809d9f0 0x7f7de80a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.201+0000 7f7de7fff640 1 -- 172.21.15.98:0/1652125473 >> v1:172.21.15.98:6789/0 conn(0x7f7de80a1cc0 legacy=0x7f7de8139570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de7fff640 1 monclient: found mon.noname-b 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de7fff640 1 -- 172.21.15.98:0/1652125473 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de809f120 con 0x7f7de80a20a0 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de7fff640 1 --2- 172.21.15.98:0/1652125473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7de80a20a0 0x7f7de813cca0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009dc0 tx=0x7f7de002bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=850d907be6dd4a38 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 1 -- 172.21.15.98:0/1652125473 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7de0039070 con 0x7f7de80a20a0 2024-06-11T19:20:00.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:20:00.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 10 monclient: dump: 2024-06-11T19:20:00.219 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.219 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.219 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.219 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.219 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.219 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.220 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.220 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.220 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.220 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.220 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.220 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.220 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.220 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.220 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.220 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 10 monclient: _finish_auth 0 2024-06-11T19:20:00.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 10 monclient: _check_auth_tickets 2024-06-11T19:20:00.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:00.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 1 -- 172.21.15.98:0/1652125473 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7de002cd70 con 0x7f7de80a20a0 2024-06-11T19:20:00.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7dee087640 5 monclient: authenticate success, global_id 84611 2024-06-11T19:20:00.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7dee087640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:00.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:00.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 1 -- 172.21.15.98:0/1652125473 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7de00327b0 con 0x7f7de80a20a0 2024-06-11T19:20:00.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7dee087640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:00.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7dee087640 4 set_mon_vals no callback set 2024-06-11T19:20:00.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:00.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.202+0000 7f7de67fc640 10 monclient: dump: 2024-06-11T19:20:00.222 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.222 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.222 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.222 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.222 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.222 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.222 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.222 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.222 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.222 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.223 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.223 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.223 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.223 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.223 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.223 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.203+0000 7f7dee087640 10 monclient: shutdown 2024-06-11T19:20:00.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.203+0000 7f7dee087640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:00.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.203+0000 7f7dee087640 1 -- 172.21.15.98:0/1652125473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7de80a20a0 msgr2=0x7f7de813cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.203+0000 7f7dee087640 1 --2- 172.21.15.98:0/1652125473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7de80a20a0 0x7f7de813cca0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009dc0 tx=0x7f7de002bdb0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.203+0000 7f7dee087640 1 -- 172.21.15.98:0/1652125473 shutdown_connections 2024-06-11T19:20:00.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.203+0000 7f7dee087640 1 --2- 172.21.15.98:0/1652125473 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7de80a20a0 0x7f7de813cca0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.203+0000 7f7dee087640 1 --2- 172.21.15.98:0/1652125473 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7de809d9f0 0x7f7de80a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.203+0000 7f7dee087640 1 -- 172.21.15.98:0/1652125473 >> 172.21.15.98:0/1652125473 conn(0x7f7de80a73b0 msgr2=0x7f7de80a3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:00.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.203+0000 7f7dee087640 1 -- 172.21.15.98:0/1652125473 shutdown_connections 2024-06-11T19:20:00.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.203+0000 7f7dee087640 1 -- 172.21.15.98:0/1652125473 wait complete. 2024-06-11T19:20:00.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.204+0000 7f7dee087640 10 monclient: build_initial_monmap 2024-06-11T19:20:00.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.204+0000 7f7dee087640 10 monclient: monmap: 2024-06-11T19:20:00.224 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:00.224 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:00.225 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:00.225 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:00.225 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:00.225 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:00.225 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:00.225 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:00.225 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:00.225 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:00.225 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:00.225 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:00.226 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:00.226 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:00.226 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:00.226 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.204+0000 7f7dee087640 1 librados: starting msgr at 2024-06-11T19:20:00.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.204+0000 7f7dee087640 1 librados: starting objecter 2024-06-11T19:20:00.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.204+0000 7f7dee087640 1 Processor -- start 2024-06-11T19:20:00.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.204+0000 7f7dee087640 1 -- start start 2024-06-11T19:20:00.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.204+0000 7f7dee087640 1 librados: setting wanted keys 2024-06-11T19:20:00.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.204+0000 7f7dee087640 1 librados: calling monclient init 2024-06-11T19:20:00.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.204+0000 7f7dee087640 10 monclient: init 2024-06-11T19:20:00.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.204+0000 7f7dee087640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:00.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.204+0000 7f7dee087640 10 monclient: _add_conns ranks=[5,8,1,6,0,2,3,4,7] 2024-06-11T19:20:00.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de809d9f0 0x7f7de81cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 monclient(hunting): picked mon.noname-h con 0x7f7de809d9f0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:00.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a1cc0 0x7f7de813c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 monclient(hunting): picked mon.noname-e con 0x7f7de80a1cc0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:00.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7de80a20a0 0x7f7de81c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 monclient(hunting): picked mon.noname-f con 0x7f7de80a20a0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:00.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7de80a2f70 con 0x7f7de80a20a0 2024-06-11T19:20:00.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f7de80a0440 con 0x7f7de809d9f0 2024-06-11T19:20:00.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f7de80a32e0 con 0x7f7de80a1cc0 2024-06-11T19:20:00.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:00.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 monclient(hunting): authenticate will time out at 1410.012057s 2024-06-11T19:20:00.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a1cc0 0x7f7de813c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a1cc0 0x7f7de813c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44016/0 (socket says 172.21.15.98:44016) 2024-06-11T19:20:00.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 1 -- 172.21.15.98:0/2241005310 learned_addr learned my addr 172.21.15.98:0/2241005310 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:00.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 10 monclient(hunting): get_auth_request con 0x7f7de80a1cc0 auth_method 0 2024-06-11T19:20:00.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de7fff640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7de80a20a0 0x7f7de81c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de77fe640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de809d9f0 0x7f7de81cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de77fe640 10 monclient(hunting): get_auth_request con 0x7f7de809d9f0 auth_method 0 2024-06-11T19:20:00.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de7fff640 10 monclient(hunting): get_auth_request con 0x7f7de80a20a0 auth_method 0 2024-06-11T19:20:00.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 10 monclient(hunting): handle_auth_done global_id 94265 payload 824 2024-06-11T19:20:00.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 1 -- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de809d9f0 msgr2=0x7f7de81cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de809d9f0 0x7f7de81cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 1 -- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7de80a20a0 msgr2=0x7f7de81c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7de80a20a0 0x7f7de81c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:20:00.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 1 monclient: found mon.noname-e 2024-06-11T19:20:00.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:00.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:20:00.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 1 -- 172.21.15.98:0/2241005310 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de8123e50 con 0x7f7de80a1cc0 2024-06-11T19:20:00.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de77fe640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7de809d9f0 0x7f7de81cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:20:00.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de7fff640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7de80a20a0 0x7f7de81c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:20:00.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de6ffd640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a1cc0 0x7f7de813c810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00c410 tx=0x7f7ddc00c8d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 10 client.?.objecter ms_handle_connect 0x7f7de80a1cc0 2024-06-11T19:20:00.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:00.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ddc0047e0 con 0x7f7de80a1cc0 2024-06-11T19:20:00.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:20:00.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 10 monclient: dump: 2024-06-11T19:20:00.232 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.232 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.232 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.232 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.232 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.232 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.232 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.233 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.233 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.233 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.233 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.233 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.233 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.233 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.233 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.233 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 10 monclient: _finish_auth 0 2024-06-11T19:20:00.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 10 monclient: _check_auth_tickets 2024-06-11T19:20:00.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:00.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ddc0049e0 con 0x7f7de80a1cc0 2024-06-11T19:20:00.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 5 monclient: authenticate success, global_id 94265 2024-06-11T19:20:00.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:00.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ddc0133c0 con 0x7f7de80a1cc0 2024-06-11T19:20:00.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 monclient: _renew_subs 2024-06-11T19:20:00.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:00.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 1 -- 172.21.15.98:0/2241005310 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de80a0a60 con 0x7f7de80a1cc0 2024-06-11T19:20:00.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dcffff640 10 monclient: dump: 2024-06-11T19:20:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.236 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.236 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.236 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.236 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.236 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.236 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7de5ffb640 4 set_mon_vals no callback set 2024-06-11T19:20:00.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 client.94265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:00.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 monclient: _renew_subs 2024-06-11T19:20:00.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:00.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 1 -- 172.21.15.98:0/2241005310 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de809e070 con 0x7f7de80a1cc0 2024-06-11T19:20:00.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.205+0000 7f7dee087640 1 librados: init done 2024-06-11T19:20:00.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7dee087640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:00.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7dee087640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de813cd50 0x7f7de8125e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7dee087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f7ddc011070 con 0x7f7de80a1cc0 2024-06-11T19:20:00.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7dee087640 1 -- 172.21.15.98:0/2241005310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7de8050830 con 0x7f7de813cd50 2024-06-11T19:20:00.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7de77fe640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de813cd50 0x7f7de8125e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7de77fe640 10 monclient: get_auth_request con 0x7f7de813cd50 auth_method 0 2024-06-11T19:20:00.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7de77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7de77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7de77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.206+0000 7f7de77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7dd403fdb0 0x7f7dd4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7f7ddc051650 con 0x7f7de80a1cc0 2024-06-11T19:20:00.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 10 client.94265.objecter ms_dispatch 0x7f7de8000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:00.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 3 client.94265.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:00.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 3 client.94265.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:00.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7de77fe640 10 monclient(hunting): handle_auth_done global_id 65241 payload 824 2024-06-11T19:20:00.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7de77fe640 1 -- 172.21.15.98:0/2241005310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7de8068e20 con 0x7f7de813cd50 2024-06-11T19:20:00.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7de7fff640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7dd403fdb0 0x7f7dd4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 20 client.94265.objecter dump_active .. 0 homeless 2024-06-11T19:20:00.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7de7fff640 10 monclient: get_auth_request con 0x7f7dd403fdb0 auth_method 0 2024-06-11T19:20:00.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 10 client.94265.objecter ms_handle_connect 0x7f7de813cd50 2024-06-11T19:20:00.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 10 client.94265.objecter resend_mon_ops 2024-06-11T19:20:00.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7de77fe640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de813cd50 0x7f7de8125e30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800d520 tx=0x7f7dd800d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7dd8012a60 con 0x7f7de813cd50 2024-06-11T19:20:00.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:00.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 10 monclient: dump: 2024-06-11T19:20:00.239 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.239 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.240 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.241 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 10 client.94265.objecter ms_handle_connect 0x7f7dd403fdb0 2024-06-11T19:20:00.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7de7fff640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7dd403fdb0 0x7f7dd4042270 secure :-1 s=READY pgs=4196 cs=0 l=1 rev1=1 crypto rx=0x7f7de0036000 tx=0x7f7de0030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7dd8019070 con 0x7f7de813cd50 2024-06-11T19:20:00.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:00.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:00.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de813cd50 msgr2=0x7f7de8125e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de813cd50 0x7f7de8125e30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800d520 tx=0x7f7dd800d9e0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de813cd50 msgr2=0x7f7de8125e30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:00.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.207+0000 7f7dcffff640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de813cd50 0x7f7de8125e30 secure :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800d520 tx=0x7f7dd800d9e0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dee087640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:00.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dee087640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de81c86f0 0x7f7de81c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dee087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dee087640 1 -- 172.21.15.98:0/2241005310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7de81c9040 con 0x7f7de81c86f0 2024-06-11T19:20:00.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7de77fe640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de81c86f0 0x7f7de81c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7de77fe640 10 monclient: get_auth_request con 0x7f7de81c86f0 auth_method 0 2024-06-11T19:20:00.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7de77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7de77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7de77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7de77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7de77fe640 10 monclient(hunting): handle_auth_done global_id 65250 payload 824 2024-06-11T19:20:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7de77fe640 1 -- 172.21.15.98:0/2241005310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7de8071e00 con 0x7f7de81c86f0 2024-06-11T19:20:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 10 client.94265.objecter ms_handle_connect 0x7f7de81c86f0 2024-06-11T19:20:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 10 client.94265.objecter resend_mon_ops 2024-06-11T19:20:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7de77fe640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de81c86f0 0x7f7de81c8b00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800d520 tx=0x7f7dd80047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7dd8012c00 con 0x7f7de81c86f0 2024-06-11T19:20:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 10 monclient: dump: 2024-06-11T19:20:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.243 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.244 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.244 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.244 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.244 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.244 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.244 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.244 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.244 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.244 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.244 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.244 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.245 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.245 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.245 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f7dd8012790 con 0x7f7de81c86f0 2024-06-11T19:20:00.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:00.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:00.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de81c86f0 msgr2=0x7f7de81c8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de81c86f0 0x7f7de81c8b00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800d520 tx=0x7f7dd80047d0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 1 -- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de81c86f0 msgr2=0x7f7de81c8b00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:00.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.214+0000 7f7dcffff640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7de81c86f0 0x7f7de81c8b00 secure :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800d520 tx=0x7f7dd80047d0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.246 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 392042, 2024-06-11T19:20:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:20:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:20:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:00.272 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:00.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 10 librados: watch_flush enter 2024-06-11T19:20:00.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 10 librados: watch_flush exit 2024-06-11T19:20:00.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 20 client.94265.objecter shutdown clearing up homeless session... 2024-06-11T19:20:00.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 10 client.94265.objecter successfully canceled tick 2024-06-11T19:20:00.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 1 -- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7dd403fdb0 msgr2=0x7f7dd4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7dd403fdb0 0x7f7dd4042270 secure :-1 s=READY pgs=4196 cs=0 l=1 rev1=1 crypto rx=0x7f7de0036000 tx=0x7f7de0030000 comp rx=0 tx=0).stop 2024-06-11T19:20:00.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 10 monclient: shutdown 2024-06-11T19:20:00.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:00.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 1 -- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a1cc0 msgr2=0x7f7de813c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 1 --2- 172.21.15.98:0/2241005310 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7de80a1cc0 0x7f7de813c810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00c410 tx=0x7f7ddc00c8d0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7de7fff640 1 -- 172.21.15.98:0/2241005310 reap_dead start 2024-06-11T19:20:00.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7de7fff640 1 -- 172.21.15.98:0/2241005310 reap_dead start 2024-06-11T19:20:00.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 1 -- 172.21.15.98:0/2241005310 shutdown_connections 2024-06-11T19:20:00.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 1 -- 172.21.15.98:0/2241005310 >> 172.21.15.98:0/2241005310 conn(0x7f7de80a73b0 msgr2=0x7f7de80716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:00.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 1 -- 172.21.15.98:0/2241005310 shutdown_connections 2024-06-11T19:20:00.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 1 -- 172.21.15.98:0/2241005310 wait complete. 2024-06-11T19:20:00.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.215+0000 7f7dee087640 1 librados: shutdown 2024-06-11T19:20:00.276 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:00.276 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:20:00.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:20:00.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.378+0000 7f18de6ff640 10 monclient: get_monmap_and_config 2024-06-11T19:20:00.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.378+0000 7f18de6ff640 10 monclient: build_initial_monmap 2024-06-11T19:20:00.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.378+0000 7f18de6ff640 10 monclient: monmap: 2024-06-11T19:20:00.377 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:00.377 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.377 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:00.379119+0000 2024-06-11T19:20:00.378 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:00.379119+0000 2024-06-11T19:20:00.378 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:00.378 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.378 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:00.378 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:00.378 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:00.378 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:00.378 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:00.378 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:00.378 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:00.378 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:00.378 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:00.379 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:00.379 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:00.379 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.378+0000 7f18de6ff640 1 Processor -- start 2024-06-11T19:20:00.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.378+0000 7f18de6ff640 1 -- start start 2024-06-11T19:20:00.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.378+0000 7f18de6ff640 10 monclient: init 2024-06-11T19:20:00.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:00.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 10 monclient: _add_conns ranks=[9,2,7,6,5,3,10,0,1,8,4] 2024-06-11T19:20:00.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f18d81347f0 0x7f18d809e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 10 monclient(hunting): picked mon.noname-a con 0x7f18d81347f0 addr v2:172.21.15.98:3300/0 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d8134bd0 0x7f18d80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 10 monclient(hunting): picked mon.noname-b con 0x7f18d8134bd0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d8135010 0x7f18d80a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 10 monclient(hunting): picked mon.noname-i con 0x7f18d8135010 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f18d80687f0 con 0x7f18d81347f0 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f18d809d410 con 0x7f18d8134bd0 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f18d809d6f0 con 0x7f18d8135010 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:00.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18de6ff640 10 monclient(hunting): authenticate will time out at 1410.186448s 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d77fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d8134bd0 0x7f18d80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f18d81347f0 0x7f18d809e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d77fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d8134bd0 0x7f18d80a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:46162/0 (socket says 172.21.15.98:46162) 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d77fe640 1 -- 172.21.15.98:0/2922988561 learned_addr learned my addr 172.21.15.98:0/2922988561 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 10 monclient(hunting): get_auth_request con 0x7f18d81347f0 auth_method 0 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d77fe640 10 monclient(hunting): get_auth_request con 0x7f18d8134bd0 auth_method 0 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18dcc9c640 1 --2- 172.21.15.98:0/2922988561 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d8135010 0x7f18d80a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 10 monclient(hunting): handle_auth_done global_id 65259 payload 274 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 1 -- 172.21.15.98:0/2922988561 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d8135010 msgr2=0x7f18d80a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 1 --2- 172.21.15.98:0/2922988561 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d8135010 0x7f18d80a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 1 -- 172.21.15.98:0/2922988561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d8134bd0 msgr2=0x7f18d80a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 1 --2- 172.21.15.98:0/2922988561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d8134bd0 0x7f18d80a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 1 monclient: found mon.noname-a 2024-06-11T19:20:00.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:00.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:00.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d7fff640 1 -- 172.21.15.98:0/2922988561 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d809d9d0 con 0x7f18d81347f0 2024-06-11T19:20:00.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18dcc9c640 10 monclient: get_auth_request con 0x7f18d8135010 auth_method 0 2024-06-11T19:20:00.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18d77fe640 1 --2- 172.21.15.98:0/2922988561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d8134bd0 0x7f18d80a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:00.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.379+0000 7f18dcc9c640 1 --2- 172.21.15.98:0/2922988561 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d8135010 0x7f18d80a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:20:00.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d7fff640 1 --2- 172.21.15.98:0/2922988561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18d81347f0 0x7f18d809e6b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f18c80097a0 tx=0x7f18c802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f525a1789e8523ed server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 1 -- 172.21.15.98:0/2922988561 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18c8039070 con 0x7f18d81347f0 2024-06-11T19:20:00.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:00.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 10 monclient: dump: 2024-06-11T19:20:00.395 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.395 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.395 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.395 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.395 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 10 monclient: _finish_auth 0 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:20:00.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 1 -- 172.21.15.98:0/2922988561 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c8031960 con 0x7f18d81347f0 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18de6ff640 5 monclient: authenticate success, global_id 65259 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18de6ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18de6ff640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 1 -- 172.21.15.98:0/2922988561 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18c8032380 con 0x7f18d81347f0 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18de6ff640 4 set_mon_vals no callback set 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.380+0000 7f18d6ffd640 10 monclient: dump: 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.397 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.398 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.398 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.398 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.398 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.398 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.398 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.398 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.398 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.398 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.398 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.398 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.398 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.399 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 10 monclient: shutdown 2024-06-11T19:20:00.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:00.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 1 -- 172.21.15.98:0/2922988561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18d81347f0 msgr2=0x7f18d809e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 1 --2- 172.21.15.98:0/2922988561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18d81347f0 0x7f18d809e6b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f18c80097a0 tx=0x7f18c802b790 comp rx=0 tx=0).stop 2024-06-11T19:20:00.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 1 -- 172.21.15.98:0/2922988561 shutdown_connections 2024-06-11T19:20:00.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 1 --2- 172.21.15.98:0/2922988561 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d8135010 0x7f18d80a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 1 --2- 172.21.15.98:0/2922988561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d8134bd0 0x7f18d80a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 1 --2- 172.21.15.98:0/2922988561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18d81347f0 0x7f18d809e6b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 1 -- 172.21.15.98:0/2922988561 >> 172.21.15.98:0/2922988561 conn(0x7f18d80a4780 msgr2=0x7f18d80a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:00.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 1 -- 172.21.15.98:0/2922988561 shutdown_connections 2024-06-11T19:20:00.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 1 -- 172.21.15.98:0/2922988561 wait complete. 2024-06-11T19:20:00.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 10 monclient: build_initial_monmap 2024-06-11T19:20:00.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.381+0000 7f18de6ff640 10 monclient: monmap: 2024-06-11T19:20:00.400 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:00.400 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:00.400 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:00.400 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:00.400 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 1 librados: starting msgr at 2024-06-11T19:20:00.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 1 librados: starting objecter 2024-06-11T19:20:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 1 Processor -- start 2024-06-11T19:20:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 1 -- start start 2024-06-11T19:20:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 1 librados: setting wanted keys 2024-06-11T19:20:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 1 librados: calling monclient init 2024-06-11T19:20:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 10 monclient: init 2024-06-11T19:20:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 10 monclient: _add_conns ranks=[7,1,5,4,2,6,0,3,8] 2024-06-11T19:20:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d81347f0 0x7f18d81cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 10 monclient(hunting): picked mon.noname-i con 0x7f18d81347f0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18d8134bd0 0x7f18d81ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 10 monclient(hunting): picked mon.noname-f con 0x7f18d8134bd0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:00.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f18d8135010 0x7f18d811eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 10 monclient(hunting): picked mon.noname-h con 0x7f18d8135010 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:00.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f18d8068c40 con 0x7f18d8134bd0 2024-06-11T19:20:00.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f18d8068ac0 con 0x7f18d8135010 2024-06-11T19:20:00.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f18d80a4110 con 0x7f18d81347f0 2024-06-11T19:20:00.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:00.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.382+0000 7f18de6ff640 10 monclient(hunting): authenticate will time out at 1410.189991s 2024-06-11T19:20:00.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d81347f0 0x7f18d81cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d81347f0 0x7f18d81cdc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:45748/0 (socket says 172.21.15.98:45748) 2024-06-11T19:20:00.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18dcc9c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f18d8135010 0x7f18d811eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 1 -- 172.21.15.98:0/511391082 learned_addr learned my addr 172.21.15.98:0/511391082 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d77fe640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18d8134bd0 0x7f18d81ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 10 monclient(hunting): get_auth_request con 0x7f18d81347f0 auth_method 0 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18dcc9c640 10 monclient(hunting): get_auth_request con 0x7f18d8135010 auth_method 0 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18dcc9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18dcc9c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18dcc9c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d77fe640 10 monclient(hunting): get_auth_request con 0x7f18d8134bd0 auth_method 0 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18dcc9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18dcc9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18dcc9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 10 monclient(hunting): handle_auth_done global_id 84409 payload 824 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 1 -- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f18d8135010 msgr2=0x7f18d811eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f18d8135010 0x7f18d811eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 1 -- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18d8134bd0 msgr2=0x7f18d81ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18d8134bd0 0x7f18d81ce1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 1 monclient: found mon.noname-i 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 1 -- 172.21.15.98:0/511391082 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d8121260 con 0x7f18d81347f0 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18dcc9c640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f18d8135010 0x7f18d811eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d77fe640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18d8134bd0 0x7f18d81ce1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d7fff640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d81347f0 0x7f18d81cdc90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f18c80096f0 tx=0x7f18c80046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d4ff9640 10 client.?.objecter ms_handle_connect 0x7f18d81347f0 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d4ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18c8043030 con 0x7f18d81347f0 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.383+0000 7f18d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d4ff9640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d4ff9640 10 monclient: dump: 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.407 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d4ff9640 10 monclient: _finish_auth 0 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d4ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c80044b0 con 0x7f18d81347f0 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 5 monclient: authenticate success, global_id 84409 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 10 monclient: _renew_subs 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 1 -- 172.21.15.98:0/511391082 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d81229f0 con 0x7f18d81347f0 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d4ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18c8039040 con 0x7f18d81347f0 2024-06-11T19:20:00.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d4ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d4ff9640 10 monclient: dump: 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.409 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 10 client.84409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 10 monclient: _renew_subs 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.117:3303/0 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 1 -- 172.21.15.98:0/511391082 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d8122430 con 0x7f18d81347f0 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 1 librados: init done 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d67fc640 4 set_mon_vals no callback set 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f050 0x7f18d811f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18de6ff640 1 -- 172.21.15.98:0/511391082 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f18d8121480 con 0x7f18d811f050 2024-06-11T19:20:00.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d77fe640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f050 0x7f18d811f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d77fe640 10 monclient: get_auth_request con 0x7f18d811f050 auth_method 0 2024-06-11T19:20:00.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.384+0000 7f18d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 <== mon.7 v2:172.21.15.117:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f18c8046070 con 0x7f18d81347f0 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d77fe640 10 monclient(hunting): handle_auth_done global_id 84629 payload 824 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d77fe640 1 -- 172.21.15.98:0/511391082 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18d809e6b0 con 0x7f18d811f050 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f18c403fdb0 0x7f18c4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 <== mon.7 v2:172.21.15.117:3303/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7f18c8044070 con 0x7f18d81347f0 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 10 client.84409.objecter ms_dispatch 0x7f18d8000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 3 client.84409.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 3 client.84409.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d77fe640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f050 0x7f18d811f430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f18d8120230 tx=0x7f18cc00f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 20 client.84409.objecter dump_active .. 0 homeless 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 10 client.84409.objecter ms_handle_connect 0x7f18d811f050 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 10 client.84409.objecter resend_mon_ops 2024-06-11T19:20:00.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18cc0148b0 con 0x7f18d811f050 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18dcc9c640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f18c403fdb0 0x7f18c4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 10 monclient: dump: 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.414 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18dcc9c640 10 monclient: get_auth_request con 0x7f18c403fdb0 auth_method 0 2024-06-11T19:20:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f18cc014440 con 0x7f18d811f050 2024-06-11T19:20:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f050 msgr2=0x7f18d811f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f050 0x7f18d811f430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f18d8120230 tx=0x7f18cc00f610 comp rx=0 tx=0).stop 2024-06-11T19:20:00.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f050 msgr2=0x7f18d811f430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:00.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f050 0x7f18d811f430 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f18d8120230 tx=0x7f18cc00f610 comp rx=0 tx=0).stop 2024-06-11T19:20:00.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18d4ff9640 10 client.84409.objecter ms_handle_connect 0x7f18c403fdb0 2024-06-11T19:20:00.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.385+0000 7f18dcc9c640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f18c403fdb0 0x7f18c4042270 secure :-1 s=READY pgs=4197 cs=0 l=1 rev1=1 crypto rx=0x7f18d0007fd0 tx=0x7f18d00076e0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18de6ff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18de6ff640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f970 0x7f18d809f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18de6ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18de6ff640 1 -- 172.21.15.98:0/511391082 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f18d8050830 con 0x7f18d811f970 2024-06-11T19:20:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d77fe640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f970 0x7f18d809f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d77fe640 10 monclient: get_auth_request con 0x7f18d811f970 auth_method 0 2024-06-11T19:20:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d77fe640 10 monclient(hunting): handle_auth_done global_id 84638 payload 824 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d77fe640 1 -- 172.21.15.98:0/511391082 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f18d8120f50 con 0x7f18d811f970 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 10 client.84409.objecter ms_handle_connect 0x7f18d811f970 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 10 client.84409.objecter resend_mon_ops 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d77fe640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f970 0x7f18d809f080 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f18d8120230 tx=0x7f18cc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18cc004490 con 0x7f18d811f970 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 10 monclient: dump: 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f18cc004020 con 0x7f18d811f970 2024-06-11T19:20:00.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:00.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:00.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f970 msgr2=0x7f18d809f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f970 0x7f18d809f080 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f18d8120230 tx=0x7f18cc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 1 -- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f970 msgr2=0x7f18d809f080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:00.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.392+0000 7f18d4ff9640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f18d811f970 0x7f18d809f080 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f18d8120230 tx=0x7f18cc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.421 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:00.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:00.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:00.422 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:00.422 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:00.422 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 272166, 2024-06-11T19:20:00.422 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:00.422 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:00.422 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:00.422 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:00.422 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:00.422 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:00.422 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.424 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:00.425 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:00.426 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:00.426 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:00.426 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:00.426 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:00.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:00.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:00.426 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:00.426 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:00.426 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:00.426 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:00.426 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:00.426 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.427 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:00.430 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:00.432 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:00.433 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.433 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.433 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:00.433 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:00.434 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:00.435 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.435 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.435 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:00.435 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:00.435 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.435 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:00.438 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.439 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:00.439 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:00.439 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.439 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.439 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:00.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.440 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.441 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:00.441 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:00.441 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.441 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.441 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:00.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:00.441 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.441 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.442 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:00.442 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.442 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:00.442 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.442 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.442 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.443 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:00.443 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:00.443 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.443 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.443 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:00.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:00.444 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.444 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.444 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:00.444 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.445 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.446 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:00.447 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.447 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.447 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:00.447 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 10 librados: watch_flush enter 2024-06-11T19:20:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 10 librados: watch_flush exit 2024-06-11T19:20:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 20 client.84409.objecter shutdown clearing up homeless session... 2024-06-11T19:20:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 10 client.84409.objecter successfully canceled tick 2024-06-11T19:20:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 1 -- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f18c403fdb0 msgr2=0x7f18c4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f18c403fdb0 0x7f18c4042270 secure :-1 s=READY pgs=4197 cs=0 l=1 rev1=1 crypto rx=0x7f18d0007fd0 tx=0x7f18d00076e0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 10 monclient: shutdown 2024-06-11T19:20:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 1 -- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d81347f0 msgr2=0x7f18d81cdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d81347f0 0x7f18d81cdc90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f18c80096f0 tx=0x7f18c80046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18dcc9c640 1 -- 172.21.15.98:0/511391082 reap_dead start 2024-06-11T19:20:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 1 -- 172.21.15.98:0/511391082 shutdown_connections 2024-06-11T19:20:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 1 --2- 172.21.15.98:0/511391082 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f18d81347f0 0x7f18d81cdc90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 1 -- 172.21.15.98:0/511391082 >> 172.21.15.98:0/511391082 conn(0x7f18d80a4780 msgr2=0x7f18d8137e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 1 -- 172.21.15.98:0/511391082 shutdown_connections 2024-06-11T19:20:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 1 -- 172.21.15.98:0/511391082 wait complete. 2024-06-11T19:20:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.393+0000 7f18de6ff640 1 librados: shutdown 2024-06-11T19:20:00.450 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:00.450 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:20:00.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:20:00.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.550+0000 7fe6c8346640 10 monclient: get_monmap_and_config 2024-06-11T19:20:00.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.550+0000 7fe6c8346640 10 monclient: build_initial_monmap 2024-06-11T19:20:00.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.550+0000 7fe6c8346640 10 monclient: monmap: 2024-06-11T19:20:00.550 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:00.550 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.550 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:00.551372+0000 2024-06-11T19:20:00.550 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:00.551372+0000 2024-06-11T19:20:00.550 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:00.550 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.551 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:00.551 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:00.551 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:00.551 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:00.551 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:00.551 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:00.551 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:00.551 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:00.552 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:00.552 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:00.552 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:00.552 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.550+0000 7fe6c8346640 1 Processor -- start 2024-06-11T19:20:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.550+0000 7fe6c8346640 1 -- start start 2024-06-11T19:20:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.550+0000 7fe6c8346640 10 monclient: init 2024-06-11T19:20:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 10 monclient: _add_conns ranks=[7,10,4,3,2,6,9,8,5,0,1] 2024-06-11T19:20:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe6c00a4920 0x7fe6c00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 10 monclient(hunting): picked mon.noname-i con 0x7fe6c00a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:00.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe6c00a5240 0x7fe6c00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 10 monclient(hunting): picked mon.noname-f con 0x7fe6c00a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c00a2530 0x7fe6c00a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 10 monclient(hunting): picked mon.noname-c con 0x7fe6c00a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fe6c00a7140 con 0x7fe6c00a5240 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe6c00a2b50 con 0x7fe6c00a2530 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fe6c00a2e30 con 0x7fe6c00a4920 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c8346640 10 monclient(hunting): authenticate will time out at 1410.358613s 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c68e3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c00a2530 0x7fe6c00a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c68e3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c00a2530 0x7fe6c00a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:41076/0 (socket says 172.21.15.98:41076) 2024-06-11T19:20:00.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c68e3640 1 -- 172.21.15.98:0/571945078 learned_addr learned my addr 172.21.15.98:0/571945078 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c68e3640 10 monclient(hunting): get_auth_request con 0x7fe6c00a2530 auth_method 0 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c68e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c68e3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c68e3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c68e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c68e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c68e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c60e2640 1 --2- 172.21.15.98:0/571945078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe6c00a4920 0x7fe6c00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c58e1640 1 --2- 172.21.15.98:0/571945078 >> v2:172.21.15.117:3300/0 conn(0x7fe6c00a5240 0x7fe6c00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c60e2640 10 monclient(hunting): get_auth_request con 0x7fe6c00a4920 auth_method 0 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c60e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c60e2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c60e2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c58e1640 10 monclient(hunting): get_auth_request con 0x7fe6c00a5240 auth_method 0 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c58e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c58e1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c58e1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.551+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c58e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c58e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c58e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c68e3640 10 monclient(hunting): handle_auth_done global_id 24115 payload 274 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c68e3640 1 -- 172.21.15.98:0/571945078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe6c00a4920 msgr2=0x7fe6c00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c68e3640 1 --2- 172.21.15.98:0/571945078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe6c00a4920 0x7fe6c00a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c68e3640 1 -- 172.21.15.98:0/571945078 >> v2:172.21.15.117:3300/0 conn(0x7fe6c00a5240 msgr2=0x7fe6c00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:00.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c68e3640 1 --2- 172.21.15.98:0/571945078 >> v2:172.21.15.117:3300/0 conn(0x7fe6c00a5240 0x7fe6c00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c68e3640 10 monclient: _finish_hunting 0 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c68e3640 1 monclient: found mon.noname-c 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c68e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c68e3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c68e3640 1 -- 172.21.15.98:0/571945078 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c00a3110 con 0x7fe6c00a2530 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c68e3640 1 --2- 172.21.15.98:0/571945078 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c00a2530 0x7fe6c00a5ee0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009e50 tx=0x7fe6b802bdb0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=776e45c6f396c346 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c50e0640 1 -- 172.21.15.98:0/571945078 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6b802cbc0 con 0x7fe6c00a2530 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c50e0640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c50e0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.552+0000 7fe6c50e0640 10 monclient: dump: 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.575 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c50e0640 10 monclient: _finish_auth 0 2024-06-11T19:20:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c50e0640 10 monclient: _check_auth_tickets 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c50e0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c50e0640 1 -- 172.21.15.98:0/571945078 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b802cd80 con 0x7fe6c00a2530 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 5 monclient: authenticate success, global_id 24115 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c50e0640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c50e0640 1 -- 172.21.15.98:0/571945078 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6b8034860 con 0x7fe6c00a2530 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 4 set_mon_vals no callback set 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c50e0640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c50e0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c50e0640 10 monclient: dump: 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.577 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 10 monclient: shutdown 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 1 -- 172.21.15.98:0/571945078 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c00a2530 msgr2=0x7fe6c00a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 1 --2- 172.21.15.98:0/571945078 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c00a2530 0x7fe6c00a5ee0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009e50 tx=0x7fe6b802bdb0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 1 -- 172.21.15.98:0/571945078 shutdown_connections 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 1 --2- 172.21.15.98:0/571945078 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c00a2530 0x7fe6c00a5ee0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 1 --2- 172.21.15.98:0/571945078 >> v2:172.21.15.117:3300/0 conn(0x7fe6c00a5240 0x7fe6c00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 1 --2- 172.21.15.98:0/571945078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe6c00a4920 0x7fe6c00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 1 -- 172.21.15.98:0/571945078 >> 172.21.15.98:0/571945078 conn(0x7fe6c00ab850 msgr2=0x7fe6c00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 1 -- 172.21.15.98:0/571945078 shutdown_connections 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.553+0000 7fe6c8346640 1 -- 172.21.15.98:0/571945078 wait complete. 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.554+0000 7fe6c8346640 10 monclient: build_initial_monmap 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.554+0000 7fe6c8346640 10 monclient: monmap: 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:00.580 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.554+0000 7fe6c8346640 1 librados: starting msgr at 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.554+0000 7fe6c8346640 1 librados: starting objecter 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.554+0000 7fe6c8346640 1 Processor -- start 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.554+0000 7fe6c8346640 1 -- start start 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.554+0000 7fe6c8346640 1 librados: setting wanted keys 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.554+0000 7fe6c8346640 1 librados: calling monclient init 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.554+0000 7fe6c8346640 10 monclient: init 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 10 monclient: _add_conns ranks=[1,2,0,6,5,7,3,4,8] 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6c00a2530 0x7fe6c0137200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 10 monclient(hunting): picked mon.noname-f con 0x7fe6c00a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00a4920 0x7fe6c0137740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 10 monclient(hunting): picked mon.noname-b con 0x7fe6c00a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00a5240 0x7fe6c0123160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 10 monclient(hunting): picked mon.noname-a con 0x7fe6c00a5240 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c00a6da0 con 0x7fe6c00a5240 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c0138e50 con 0x7fe6c00a2530 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe6c00a6420 con 0x7fe6c00a4920 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 10 monclient(hunting): authenticate will time out at 1410.362176s 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00a5240 0x7fe6c0123160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c58e1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00a4920 0x7fe6c0137740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00a5240 0x7fe6c0123160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44808/0 (socket says 172.21.15.98:44808) 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c58e1640 10 monclient(hunting): get_auth_request con 0x7fe6c00a4920 auth_method 0 2024-06-11T19:20:00.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 1 -- 172.21.15.98:0/3496951354 learned_addr learned my addr 172.21.15.98:0/3496951354 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c58e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c58e1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c58e1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 10 monclient(hunting): get_auth_request con 0x7fe6c00a5240 auth_method 0 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c60e2640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6c00a2530 0x7fe6c0137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c60e2640 10 monclient(hunting): get_auth_request con 0x7fe6c00a2530 auth_method 0 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c60e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c60e2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c60e2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c58e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c58e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c58e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 10 monclient(hunting): handle_auth_done global_id 65268 payload 824 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 1 -- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00a4920 msgr2=0x7fe6c0137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00a4920 0x7fe6c0137740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 1 -- 172.21.15.98:0/3496951354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6c00a2530 msgr2=0x7fe6c0137200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6c00a2530 0x7fe6c0137200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 10 monclient: _finish_hunting 0 2024-06-11T19:20:00.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 1 monclient: found mon.noname-a 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 1 -- 172.21.15.98:0/3496951354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c0126df0 con 0x7fe6c00a5240 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c58e1640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe6c00a4920 0x7fe6c0137740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c68e3640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00a5240 0x7fe6c0123160 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8036040 tx=0x7fe6b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6b6ffd640 10 client.?.objecter ms_handle_connect 0x7fe6c00a5240 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6b6ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6b8042020 con 0x7fe6c00a5240 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6b6ffd640 10 monclient: dump: 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.585 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6b6ffd640 10 monclient: _finish_auth 0 2024-06-11T19:20:00.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6b6ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b802cbd0 con 0x7fe6c00a5240 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 5 monclient: authenticate success, global_id 65268 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 10 monclient: _renew_subs 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.555+0000 7fe6c8346640 1 -- 172.21.15.98:0/3496951354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6c01270d0 con 0x7fe6c00a5240 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6b6ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6b80046c0 con 0x7fe6c00a5240 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6b6ffd640 10 monclient: dump: 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.587 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c48df640 4 set_mon_vals no callback set 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c8346640 10 client.65268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c8346640 10 monclient: _renew_subs 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c8346640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c8346640 1 -- 172.21.15.98:0/3496951354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6c0125940 con 0x7fe6c00a5240 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c8346640 1 librados: init done 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c8346640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c8346640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0123e60 0x7fe6c002cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c8346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c8346640 1 -- 172.21.15.98:0/3496951354 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe6c0050830 con 0x7fe6c0123e60 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fe6b8045020 con 0x7fe6c00a5240 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c60e2640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0123e60 0x7fe6c002cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c60e2640 10 monclient: get_auth_request con 0x7fe6c0123e60 auth_method 0 2024-06-11T19:20:00.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c60e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c60e2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.556+0000 7fe6c60e2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_done global_id 24124 payload 824 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6c60e2640 1 -- 172.21.15.98:0/3496951354 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6c0077a80 con 0x7fe6c0123e60 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe6a403fde0 0x7fe6a40422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7fe6b803a040 con 0x7fe6c00a5240 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 10 client.65268.objecter ms_dispatch 0x7fe6c0000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 3 client.65268.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 3 client.65268.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6c60e2640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0123e60 0x7fe6c002cb40 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0118e0 tx=0x7fe6bc011da0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 20 client.65268.objecter dump_active .. 0 homeless 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 10 client.65268.objecter ms_handle_connect 0x7fe6c0123e60 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 10 client.65268.objecter resend_mon_ops 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6bc019070 con 0x7fe6c0123e60 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6c58e1640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe6a403fde0 0x7fe6a40422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 10 monclient: dump: 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.591 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6c58e1640 10 monclient: get_auth_request con 0x7fe6a403fde0 auth_method 0 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe6bc004300 con 0x7fe6c0123e60 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0123e60 msgr2=0x7fe6c002cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0123e60 0x7fe6c002cb40 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0118e0 tx=0x7fe6bc011da0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0123e60 msgr2=0x7fe6c002cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0123e60 0x7fe6c002cb40 secure :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0118e0 tx=0x7fe6bc011da0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6b6ffd640 10 client.65268.objecter ms_handle_connect 0x7fe6a403fde0 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.557+0000 7fe6c58e1640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe6a403fde0 0x7fe6a40422a0 secure :-1 s=READY pgs=4198 cs=0 l=1 rev1=1 crypto rx=0x7fe6c01248c0 tx=0x7fe6b0009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c8346640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c8346640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0124240 0x7fe6c002d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c8346640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c8346640 1 -- 172.21.15.98:0/3496951354 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe6c0069740 con 0x7fe6c0124240 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c60e2640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0124240 0x7fe6c002d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c60e2640 10 monclient: get_auth_request con 0x7fe6c0124240 auth_method 0 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c60e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c60e2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c60e2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c60e2640 10 monclient(hunting): handle_auth_done global_id 24133 payload 824 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c60e2640 1 -- 172.21.15.98:0/3496951354 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe6c00a4d00 con 0x7fe6c0124240 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6c60e2640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0124240 0x7fe6c002d080 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0118e0 tx=0x7fe6bc0046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 10 client.65268.objecter ms_handle_connect 0x7fe6c0124240 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 10 client.65268.objecter resend_mon_ops 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6bc019040 con 0x7fe6c0124240 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 10 monclient: dump: 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.594 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.595 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7fe6bc01e450 con 0x7fe6c0124240 2024-06-11T19:20:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0124240 msgr2=0x7fe6c002d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0124240 0x7fe6c002d080 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0118e0 tx=0x7fe6bc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 1 -- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0124240 msgr2=0x7fe6c002d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:00.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.564+0000 7fe6b6ffd640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6c0124240 0x7fe6c002d080 secure :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0118e0 tx=0x7fe6bc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.596 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:00.596 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 650782, 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:00.597 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.598 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.599 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:00.600 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.601 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:00.603 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.605 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.608 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:00.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:00.612 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:00.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.615 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.615 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:00.615 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:00.615 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.615 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.615 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.615 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:00.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:00.616 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:00.616 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:00.616 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:00.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:00.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.619 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.619 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.619 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:20:00.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.619 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:00.619 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 10 librados: watch_flush enter 2024-06-11T19:20:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 10 librados: watch_flush exit 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 20 client.65268.objecter shutdown clearing up homeless session... 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 10 client.65268.objecter successfully canceled tick 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 1 -- 172.21.15.98:0/3496951354 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe6a403fde0 msgr2=0x7fe6a40422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe6a403fde0 0x7fe6a40422a0 secure :-1 s=READY pgs=4198 cs=0 l=1 rev1=1 crypto rx=0x7fe6c01248c0 tx=0x7fe6b0009190 comp rx=0 tx=0).stop 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 10 monclient: shutdown 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 1 -- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00a5240 msgr2=0x7fe6c0123160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 1 --2- 172.21.15.98:0/3496951354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00a5240 0x7fe6c0123160 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8036040 tx=0x7fe6b8004a40 comp rx=0 tx=0).stop 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c68e3640 1 -- 172.21.15.98:0/3496951354 reap_dead start 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c68e3640 1 -- 172.21.15.98:0/3496951354 reap_dead start 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 1 -- 172.21.15.98:0/3496951354 shutdown_connections 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 1 -- 172.21.15.98:0/3496951354 >> 172.21.15.98:0/3496951354 conn(0x7fe6c00ab850 msgr2=0x7fe6c01bf010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 1 -- 172.21.15.98:0/3496951354 shutdown_connections 2024-06-11T19:20:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 1 -- 172.21.15.98:0/3496951354 wait complete. 2024-06-11T19:20:00.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.565+0000 7fe6c8346640 1 librados: shutdown 2024-06-11T19:20:00.624 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:00.624 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:20:00.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:20:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.727+0000 7fd8162b9640 10 monclient: get_monmap_and_config 2024-06-11T19:20:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.727+0000 7fd8162b9640 10 monclient: build_initial_monmap 2024-06-11T19:20:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.727+0000 7fd8162b9640 10 monclient: monmap: 2024-06-11T19:20:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:00.728934+0000 2024-06-11T19:20:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:00.728934+0000 2024-06-11T19:20:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:00.728 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.728+0000 7fd8162b9640 1 Processor -- start 2024-06-11T19:20:00.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.728+0000 7fd8162b9640 1 -- start start 2024-06-11T19:20:00.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.728+0000 7fd8162b9640 10 monclient: init 2024-06-11T19:20:00.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:00.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 10 monclient: _add_conns ranks=[4,0,9,1,7,10,6,8,2,5,3] 2024-06-11T19:20:00.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd8100a4920 0x7fd8100a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 10 monclient(hunting): picked mon.noname-c con 0x7fd8100a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:00.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8100a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:20:00.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8100a2530 0x7fd8100a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 10 monclient(hunting): picked mon.noname-a con 0x7fd8100a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:20:00.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8100a7140 con 0x7fd8100a5240 2024-06-11T19:20:00.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd8100a2b50 con 0x7fd8100a2530 2024-06-11T19:20:00.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd8100a2e30 con 0x7fd8100a4920 2024-06-11T19:20:00.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:00.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd8162b9640 10 monclient(hunting): authenticate will time out at 1410.536464s 2024-06-11T19:20:00.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd80ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd8100a4920 0x7fd8100a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd807fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd8100a5240 0x7fd8100a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58074/0 (socket says 172.21.15.98:58074) 2024-06-11T19:20:00.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd80ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd8100a4920 0x7fd8100a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:41102/0 (socket says 172.21.15.98:41102) 2024-06-11T19:20:00.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd807fff640 1 -- 172.21.15.98:0/1982835309 learned_addr learned my addr 172.21.15.98:0/1982835309 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:20:00.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 1 --2- 172.21.15.98:0/1982835309 >> v2:172.21.15.98:3300/0 conn(0x7fd8100a2530 0x7fd8100a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd80ffff640 10 monclient(hunting): get_auth_request con 0x7fd8100a4920 auth_method 0 2024-06-11T19:20:00.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd80ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd80ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd80ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 10 monclient(hunting): get_auth_request con 0x7fd8100a2530 auth_method 0 2024-06-11T19:20:00.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd80ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd80ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd80ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 10 monclient(hunting): handle_auth_done global_id 65277 payload 274 2024-06-11T19:20:00.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 1 -- 172.21.15.98:0/1982835309 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd8100a4920 msgr2=0x7fd8100a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:00.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 1 --2- 172.21.15.98:0/1982835309 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd8100a4920 0x7fd8100a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 1 -- 172.21.15.98:0/1982835309 >> v1:172.21.15.98:6789/0 conn(0x7fd8100a5240 legacy=0x7fd8100a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 10 monclient: _finish_hunting 0 2024-06-11T19:20:00.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 1 monclient: found mon.noname-a 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd814856640 1 -- 172.21.15.98:0/1982835309 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8100a3110 con 0x7fd8100a2530 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd80f7fe640 1 -- 172.21.15.98:0/1982835309 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090953796 0 0) 0x7fd8100a7140 con 0x7fd8100a5240 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd80f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.729+0000 7fd80ffff640 1 --2- 172.21.15.98:0/1982835309 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd8100a4920 0x7fd8100a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd814856640 1 --2- 172.21.15.98:0/1982835309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8100a2530 0x7fd8100a60b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd808009d90 tx=0x7fd80802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e525b5428c9347 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 1 -- 172.21.15.98:0/1982835309 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd808039070 con 0x7fd8100a2530 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 10 monclient: dump: 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.748 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.750 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.750 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.750 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:20:00.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:20:00.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:00.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd8162b9640 5 monclient: authenticate success, global_id 65277 2024-06-11T19:20:00.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd8162b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:00.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 1 -- 172.21.15.98:0/1982835309 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd80802cda0 con 0x7fd8100a2530 2024-06-11T19:20:00.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:00.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd8162b9640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:00.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd8162b9640 4 set_mon_vals no callback set 2024-06-11T19:20:00.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 1 -- 172.21.15.98:0/1982835309 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8080327b0 con 0x7fd8100a2530 2024-06-11T19:20:00.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:00.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.730+0000 7fd80f7fe640 10 monclient: dump: 2024-06-11T19:20:00.751 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.751 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.751 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.751 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.751 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.731+0000 7fd8162b9640 10 monclient: shutdown 2024-06-11T19:20:00.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.731+0000 7fd8162b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:00.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.731+0000 7fd8162b9640 1 -- 172.21.15.98:0/1982835309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8100a2530 msgr2=0x7fd8100a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.731+0000 7fd8162b9640 1 --2- 172.21.15.98:0/1982835309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8100a2530 0x7fd8100a60b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd808009d90 tx=0x7fd80802bd80 comp rx=0 tx=0).stop 2024-06-11T19:20:00.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.731+0000 7fd8162b9640 1 -- 172.21.15.98:0/1982835309 shutdown_connections 2024-06-11T19:20:00.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.731+0000 7fd8162b9640 1 --2- 172.21.15.98:0/1982835309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8100a2530 0x7fd8100a60b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.731+0000 7fd8162b9640 1 --2- 172.21.15.98:0/1982835309 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd8100a4920 0x7fd8100a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.731+0000 7fd8162b9640 1 -- 172.21.15.98:0/1982835309 >> 172.21.15.98:0/1982835309 conn(0x7fd8100ab850 msgr2=0x7fd8100a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:00.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.731+0000 7fd8162b9640 1 -- 172.21.15.98:0/1982835309 shutdown_connections 2024-06-11T19:20:00.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.731+0000 7fd8162b9640 1 -- 172.21.15.98:0/1982835309 wait complete. 2024-06-11T19:20:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.731+0000 7fd8162b9640 10 monclient: build_initial_monmap 2024-06-11T19:20:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.731+0000 7fd8162b9640 10 monclient: monmap: 2024-06-11T19:20:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 1 librados: starting msgr at 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 1 librados: starting objecter 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 1 Processor -- start 2024-06-11T19:20:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 1 -- start start 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 1 librados: setting wanted keys 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 1 librados: calling monclient init 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 10 monclient: init 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 10 monclient: _add_conns ranks=[8,6,7,5,2,1,3,4,0] 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8100a2530 0x7fd810137820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 10 monclient(hunting): picked mon.noname-e con 0x7fd8100a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8100a4920 0x7fd810137d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 10 monclient(hunting): picked mon.noname-d con 0x7fd8100a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8100a5240 0x7fd810125380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 10 monclient(hunting): picked mon.noname-i con 0x7fd8100a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd8100a6da0 con 0x7fd8100a4920 2024-06-11T19:20:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd810138e50 con 0x7fd8100a5240 2024-06-11T19:20:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd8100a65f0 con 0x7fd8100a2530 2024-06-11T19:20:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd8162b9640 10 monclient(hunting): authenticate will time out at 1410.539819s 2024-06-11T19:20:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd80ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8100a2530 0x7fd810137820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd807fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8100a4920 0x7fd810137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd80ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8100a2530 0x7fd810137820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44030/0 (socket says 172.21.15.98:44030) 2024-06-11T19:20:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd80ffff640 1 -- 172.21.15.98:0/2587663520 learned_addr learned my addr 172.21.15.98:0/2587663520 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd80ffff640 10 monclient(hunting): get_auth_request con 0x7fd8100a2530 auth_method 0 2024-06-11T19:20:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd80ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd80ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd80ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd807fff640 10 monclient(hunting): get_auth_request con 0x7fd8100a4920 auth_method 0 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd807fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd807fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd80ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.732+0000 7fd80ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd807fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd807fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd807fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80ffff640 10 monclient(hunting): handle_auth_done global_id 94274 payload 824 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd814856640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8100a5240 0x7fd810125380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80ffff640 1 -- 172.21.15.98:0/2587663520 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8100a5240 msgr2=0x7fd810125380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80ffff640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd8100a5240 0x7fd810125380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80ffff640 1 -- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8100a4920 msgr2=0x7fd810137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80ffff640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8100a4920 0x7fd810137d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80ffff640 1 monclient: found mon.noname-e 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80ffff640 1 -- 172.21.15.98:0/2587663520 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd810128c60 con 0x7fd8100a2530 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd807fff640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8100a4920 0x7fd810137d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80ffff640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8100a2530 0x7fd810137820 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd80000ec10 tx=0x7fd80000c520 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 10 client.?.objecter ms_handle_connect 0x7fd8100a2530 2024-06-11T19:20:00.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:00.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd800017070 con 0x7fd8100a2530 2024-06-11T19:20:00.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:20:00.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 10 monclient: dump: 2024-06-11T19:20:00.760 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.760 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.760 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.760 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.760 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.761 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.761 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.761 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.761 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.761 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.761 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.761 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.761 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.761 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.761 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.761 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 10 monclient: _finish_auth 0 2024-06-11T19:20:00.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:20:00.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:00.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd8162b9640 5 monclient: authenticate success, global_id 94274 2024-06-11T19:20:00.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd80000cc30 con 0x7fd8100a2530 2024-06-11T19:20:00.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd8162b9640 10 monclient: _renew_subs 2024-06-11T19:20:00.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd8162b9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:00.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd8162b9640 1 -- 172.21.15.98:0/2587663520 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd810128f40 con 0x7fd8100a2530 2024-06-11T19:20:00.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:00.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd80000cde0 con 0x7fd8100a2530 2024-06-11T19:20:00.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd8162b9640 10 client.94274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:00.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:00.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80effd640 4 set_mon_vals no callback set 2024-06-11T19:20:00.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd80d7fa640 10 monclient: dump: 2024-06-11T19:20:00.763 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.763 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.763 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.763 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd8162b9640 10 monclient: _renew_subs 2024-06-11T19:20:00.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd8162b9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:00.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd8162b9640 1 -- 172.21.15.98:0/2587663520 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fd810128980 con 0x7fd8100a2530 2024-06-11T19:20:00.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.733+0000 7fd8162b9640 1 librados: init done 2024-06-11T19:20:00.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd8162b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:00.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd80001a070 con 0x7fd8100a2530 2024-06-11T19:20:00.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd8162b9640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8101cdae0 0x7fd81002cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd8162b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd8162b9640 1 -- 172.21.15.98:0/2587663520 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd810050830 con 0x7fd8101cdae0 2024-06-11T19:20:00.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd807fff640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8101cdae0 0x7fd81002cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd807fff640 10 monclient: get_auth_request con 0x7fd8101cdae0 auth_method 0 2024-06-11T19:20:00.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd807fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd807fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd807fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd807fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd80d7fa640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd7f003fdb0 0x7fd7f0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd807fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7fd80000f040 con 0x7fd8100a2530 2024-06-11T19:20:00.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd80d7fa640 10 client.94274.objecter ms_dispatch 0x7fd810000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:00.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd80d7fa640 3 client.94274.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:00.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd80d7fa640 3 client.94274.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:00.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd807fff640 10 monclient(hunting): handle_auth_done global_id 55293 payload 824 2024-06-11T19:20:00.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd807fff640 1 -- 172.21.15.98:0/2587663520 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd8101ce7b0 con 0x7fd8101cdae0 2024-06-11T19:20:00.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd80d7fa640 20 client.94274.objecter dump_active .. 0 homeless 2024-06-11T19:20:00.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd80d7fa640 10 client.94274.objecter ms_handle_connect 0x7fd8101cdae0 2024-06-11T19:20:00.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd80d7fa640 10 client.94274.objecter resend_mon_ops 2024-06-11T19:20:00.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd807fff640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8101cdae0 0x7fd81002cb40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd8100a14e0 tx=0x7fd7f8001c70 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.734+0000 7fd814856640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd7f003fdb0 0x7fd7f0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd814856640 10 monclient: get_auth_request con 0x7fd7f003fdb0 auth_method 0 2024-06-11T19:20:00.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7f800fda0 con 0x7fd8101cdae0 2024-06-11T19:20:00.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd80d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd80d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:00.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd80d7fa640 10 monclient: dump: 2024-06-11T19:20:00.768 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.768 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.768 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.768 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.768 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.768 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd7f8018070 con 0x7fd8101cdae0 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd80d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd80d7fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:00.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8101cdae0 msgr2=0x7fd81002cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd80d7fa640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8101cdae0 0x7fd81002cb40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd8100a14e0 tx=0x7fd7f8001c70 comp rx=0 tx=0).stop 2024-06-11T19:20:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8101cdae0 msgr2=0x7fd81002cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd80d7fa640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd8101cdae0 0x7fd81002cb40 secure :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd8100a14e0 tx=0x7fd7f8001c70 comp rx=0 tx=0).stop 2024-06-11T19:20:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd80d7fa640 10 client.94274.objecter ms_handle_connect 0x7fd7f003fdb0 2024-06-11T19:20:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.735+0000 7fd814856640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd7f003fdb0 0x7fd7f0042270 secure :-1 s=READY pgs=4199 cs=0 l=1 rev1=1 crypto rx=0x7fd808036000 tx=0x7fd808030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.741+0000 7fd8162b9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.741+0000 7fd8162b9640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd81002d080 0x7fd81002d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.741+0000 7fd8162b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.741+0000 7fd8162b9640 1 -- 172.21.15.98:0/2587663520 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd810127cb0 con 0x7fd81002d080 2024-06-11T19:20:00.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.741+0000 7fd807fff640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd81002d080 0x7fd81002d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.741+0000 7fd807fff640 10 monclient: get_auth_request con 0x7fd81002d080 auth_method 0 2024-06-11T19:20:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.741+0000 7fd807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.741+0000 7fd807fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.741+0000 7fd807fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd807fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd807fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd807fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd807fff640 10 monclient(hunting): handle_auth_done global_id 55302 payload 824 2024-06-11T19:20:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd807fff640 1 -- 172.21.15.98:0/2587663520 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd810129340 con 0x7fd81002d080 2024-06-11T19:20:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 10 client.94274.objecter ms_handle_connect 0x7fd81002d080 2024-06-11T19:20:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 10 client.94274.objecter resend_mon_ops 2024-06-11T19:20:00.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd807fff640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd81002d080 0x7fd81002d490 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8017000 tx=0x7fd7f8007f90 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7f8013a70 con 0x7fd81002d080 2024-06-11T19:20:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 10 monclient: dump: 2024-06-11T19:20:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.772 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.773 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fd7f8013600 con 0x7fd81002d080 2024-06-11T19:20:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd81002d080 msgr2=0x7fd81002d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd81002d080 0x7fd81002d490 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8017000 tx=0x7fd7f8007f90 comp rx=0 tx=0).stop 2024-06-11T19:20:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 1 -- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd81002d080 msgr2=0x7fd81002d490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd80d7fa640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd81002d080 0x7fd81002d490 secure :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8017000 tx=0x7fd7f8007f90 comp rx=0 tx=0).stop 2024-06-11T19:20:00.775 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 513499, 2024-06-11T19:20:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:20:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.802 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.803 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:00.803 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd8162b9640 10 librados: watch_flush enter 2024-06-11T19:20:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd8162b9640 10 librados: watch_flush exit 2024-06-11T19:20:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd8162b9640 20 client.94274.objecter shutdown clearing up homeless session... 2024-06-11T19:20:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.742+0000 7fd8162b9640 10 client.94274.objecter successfully canceled tick 2024-06-11T19:20:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd8162b9640 1 -- 172.21.15.98:0/2587663520 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd7f003fdb0 msgr2=0x7fd7f0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd8162b9640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd7f003fdb0 0x7fd7f0042270 secure :-1 s=READY pgs=4199 cs=0 l=1 rev1=1 crypto rx=0x7fd808036000 tx=0x7fd808030000 comp rx=0 tx=0).stop 2024-06-11T19:20:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd8162b9640 10 monclient: shutdown 2024-06-11T19:20:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd8162b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd8162b9640 1 -- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8100a2530 msgr2=0x7fd810137820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd8162b9640 1 --2- 172.21.15.98:0/2587663520 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd8100a2530 0x7fd810137820 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd80000ec10 tx=0x7fd80000c520 comp rx=0 tx=0).stop 2024-06-11T19:20:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd814856640 1 -- 172.21.15.98:0/2587663520 reap_dead start 2024-06-11T19:20:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd814856640 1 -- 172.21.15.98:0/2587663520 reap_dead start 2024-06-11T19:20:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd8162b9640 1 -- 172.21.15.98:0/2587663520 shutdown_connections 2024-06-11T19:20:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd8162b9640 1 -- 172.21.15.98:0/2587663520 >> 172.21.15.98:0/2587663520 conn(0x7fd8100ab850 msgr2=0x7fd81009cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd8162b9640 1 -- 172.21.15.98:0/2587663520 shutdown_connections 2024-06-11T19:20:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd8162b9640 1 -- 172.21.15.98:0/2587663520 wait complete. 2024-06-11T19:20:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.743+0000 7fd8162b9640 1 librados: shutdown 2024-06-11T19:20:00.806 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:00.806 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:20:00.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:20:00.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.909+0000 7fa6b92b9640 10 monclient: get_monmap_and_config 2024-06-11T19:20:00.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.909+0000 7fa6b92b9640 10 monclient: build_initial_monmap 2024-06-11T19:20:00.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.909+0000 7fa6b92b9640 10 monclient: monmap: 2024-06-11T19:20:00.909 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:00.909 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.909 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:00.910964+0000 2024-06-11T19:20:00.909 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:00.910964+0000 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:00.910 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:00.911 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.910+0000 7fa6b92b9640 1 Processor -- start 2024-06-11T19:20:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.910+0000 7fa6b92b9640 1 -- start start 2024-06-11T19:20:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.910+0000 7fa6b92b9640 10 monclient: init 2024-06-11T19:20:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 10 monclient: _add_conns ranks=[4,10,8,3,6,0,7,2,1,5,9] 2024-06-11T19:20:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a4920 0x7fa6b40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 10 monclient(hunting): picked mon.noname-c con 0x7fa6b40a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa6b40a5240 0x7fa6b40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 10 monclient(hunting): picked mon.noname-f con 0x7fa6b40a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b40a2530 0x7fa6b40a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 10 monclient(hunting): picked mon.noname-e con 0x7fa6b40a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fa6b40a7140 con 0x7fa6b40a5240 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa6b40a2b50 con 0x7fa6b40a4920 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa6b40a2e30 con 0x7fa6b40a2530 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b92b9640 10 monclient(hunting): authenticate will time out at 1410.718113s 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b40a2530 0x7fa6b40a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b2ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a4920 0x7fa6b40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b2ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a4920 0x7fa6b40a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:41110/0 (socket says 172.21.15.98:41110) 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b40a2530 0x7fa6b40a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44034/0 (socket says 172.21.15.98:44034) 2024-06-11T19:20:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b2ffd640 1 -- 172.21.15.98:0/2177910162 learned_addr learned my addr 172.21.15.98:0/2177910162 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:00.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b2ffd640 10 monclient(hunting): get_auth_request con 0x7fa6b40a4920 auth_method 0 2024-06-11T19:20:00.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 10 monclient(hunting): get_auth_request con 0x7fa6b40a2530 auth_method 0 2024-06-11T19:20:00.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 10 monclient(hunting): handle_auth_done global_id 94283 payload 274 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b27fc640 1 --2- 172.21.15.98:0/2177910162 >> v2:172.21.15.117:3300/0 conn(0x7fa6b40a5240 0x7fa6b40a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 1 -- 172.21.15.98:0/2177910162 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a4920 msgr2=0x7fa6b40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 1 --2- 172.21.15.98:0/2177910162 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a4920 0x7fa6b40a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 1 -- 172.21.15.98:0/2177910162 >> v2:172.21.15.117:3300/0 conn(0x7fa6b40a5240 msgr2=0x7fa6b40a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 1 --2- 172.21.15.98:0/2177910162 >> v2:172.21.15.117:3300/0 conn(0x7fa6b40a5240 0x7fa6b40a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 10 monclient: _finish_hunting 0 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 1 monclient: found mon.noname-e 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:00.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b37fe640 1 -- 172.21.15.98:0/2177910162 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b40a3110 con 0x7fa6b40a2530 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.911+0000 7fa6b2ffd640 1 --2- 172.21.15.98:0/2177910162 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a4920 0x7fa6b40a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b37fe640 1 --2- 172.21.15.98:0/2177910162 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b40a2530 0x7fa6b40a5ee0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009d90 tx=0x7fa6ac02bce0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=68fa8c8c8e5a3a0a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 1 -- 172.21.15.98:0/2177910162 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6ac039070 con 0x7fa6b40a2530 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 10 monclient: dump: 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.924 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 10 monclient: _finish_auth 0 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 1 -- 172.21.15.98:0/2177910162 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6ac02cd00 con 0x7fa6b40a2530 2024-06-11T19:20:00.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 5 monclient: authenticate success, global_id 94283 2024-06-11T19:20:00.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:00.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:00.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:00.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 1 -- 172.21.15.98:0/2177910162 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6ac0327b0 con 0x7fa6b40a2530 2024-06-11T19:20:00.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 4 set_mon_vals no callback set 2024-06-11T19:20:00.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:00.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b1ffb640 10 monclient: dump: 2024-06-11T19:20:00.926 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.926 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.926 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.926 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 10 monclient: shutdown 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2177910162 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b40a2530 msgr2=0x7fa6b40a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 1 --2- 172.21.15.98:0/2177910162 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b40a2530 0x7fa6b40a5ee0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009d90 tx=0x7fa6ac02bce0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2177910162 shutdown_connections 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 1 --2- 172.21.15.98:0/2177910162 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b40a2530 0x7fa6b40a5ee0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 1 --2- 172.21.15.98:0/2177910162 >> v2:172.21.15.117:3300/0 conn(0x7fa6b40a5240 0x7fa6b40a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 1 --2- 172.21.15.98:0/2177910162 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a4920 0x7fa6b40a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.912+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2177910162 >> 172.21.15.98:0/2177910162 conn(0x7fa6b40ab850 msgr2=0x7fa6b40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.913+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2177910162 shutdown_connections 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.913+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2177910162 wait complete. 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.913+0000 7fa6b92b9640 10 monclient: build_initial_monmap 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.913+0000 7fa6b92b9640 10 monclient: monmap: 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:00.928 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:00.929 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.913+0000 7fa6b92b9640 1 librados: starting msgr at 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.913+0000 7fa6b92b9640 1 librados: starting objecter 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 1 Processor -- start 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 1 -- start start 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 1 librados: setting wanted keys 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 1 librados: calling monclient init 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 10 monclient: init 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 10 monclient: _add_conns ranks=[3,1,4,2,5,0,8,6,7] 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa6b40a2530 0x7fa6b41cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 10 monclient(hunting): picked mon.noname-g con 0x7fa6b40a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b40a4920 0x7fa6b41ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 10 monclient(hunting): picked mon.noname-f con 0x7fa6b40a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:00.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a5240 0x7fa6b4122e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 10 monclient(hunting): picked mon.noname-c con 0x7fa6b40a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b40a6da0 con 0x7fa6b40a4920 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fa6b4138e50 con 0x7fa6b40a2530 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa6b40a6420 con 0x7fa6b40a5240 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b92b9640 10 monclient(hunting): authenticate will time out at 1410.721508s 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a5240 0x7fa6b4122e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a5240 0x7fa6b4122e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:41122/0 (socket says 172.21.15.98:41122) 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 1 -- 172.21.15.98:0/2440003314 learned_addr learned my addr 172.21.15.98:0/2440003314 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:00.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 10 monclient(hunting): get_auth_request con 0x7fa6b40a5240 auth_method 0 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 10 monclient(hunting): handle_auth_done global_id 24160 payload 824 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 1 -- 172.21.15.98:0/2440003314 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa6b40a2530 msgr2=0x7fa6b41cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa6b40a2530 0x7fa6b41cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 1 -- 172.21.15.98:0/2440003314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b40a4920 msgr2=0x7fa6b41ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b40a4920 0x7fa6b41ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 10 monclient: _finish_hunting 0 2024-06-11T19:20:00.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 1 monclient: found mon.noname-c 2024-06-11T19:20:00.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:00.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:20:00.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 1 -- 172.21.15.98:0/2440003314 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b40a90a0 con 0x7fa6b40a5240 2024-06-11T19:20:00.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa6b37fe640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a5240 0x7fa6b4122e40 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac037040 tx=0x7fa6ac004a40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.914+0000 7fa69b7fe640 10 client.?.objecter ms_handle_connect 0x7fa6b40a5240 2024-06-11T19:20:00.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:00.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6ac039050 con 0x7fa6b40a5240 2024-06-11T19:20:00.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:20:00.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 10 monclient: dump: 2024-06-11T19:20:00.933 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.933 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.934 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6ac03a040 con 0x7fa6b40a5240 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa6b92b9640 5 monclient: authenticate success, global_id 24160 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa6b92b9640 10 monclient: _renew_subs 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa6b92b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2440003314 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6b4124b10 con 0x7fa6b40a5240 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6ac0046c0 con 0x7fa6b40a5240 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:00.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa69b7fe640 10 monclient: dump: 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.936 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa6b92b9640 10 client.24160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa6b92b9640 10 monclient: _renew_subs 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa6b92b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2440003314 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6b4126b20 con 0x7fa6b40a5240 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa6b17fa640 4 set_mon_vals no callback set 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.915+0000 7fa6b92b9640 1 librados: init done 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b92b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b92b9640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123380 0x7fa6b4123760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b92b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2440003314 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa6b4050830 con 0x7fa6b4123380 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fa6ac047030 con 0x7fa6b40a5240 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b2ffd640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123380 0x7fa6b4123760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b2ffd640 10 monclient: get_auth_request con 0x7fa6b4123380 auth_method 0 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa6a003fdb0 0x7fa6a0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b2ffd640 10 monclient(hunting): handle_auth_done global_id 94292 payload 824 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7fa6ac045020 con 0x7fa6b40a5240 2024-06-11T19:20:00.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 10 client.24160.objecter ms_dispatch 0x7fa6b4000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 3 client.24160.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 3 client.24160.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b2ffd640 1 -- 172.21.15.98:0/2440003314 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa6b40a6750 con 0x7fa6b4123380 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 20 client.24160.objecter dump_active .. 0 homeless 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 10 client.24160.objecter ms_handle_connect 0x7fa6b4123380 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 10 client.24160.objecter resend_mon_ops 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b2ffd640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123380 0x7fa6b4123760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400e7a0 tx=0x7fa6a400eba0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b27fc640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa6a003fdb0 0x7fa6a0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa6b27fc640 10 monclient: get_auth_request con 0x7fa6a003fdb0 auth_method 0 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6a4017070 con 0x7fa6b4123380 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.916+0000 7fa69b7fe640 10 monclient: dump: 2024-06-11T19:20:00.939 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.917+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa6a400c750 con 0x7fa6b4123380 2024-06-11T19:20:00.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.917+0000 7fa69b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:00.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.917+0000 7fa69b7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:00.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.917+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123380 msgr2=0x7fa6b4123760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.917+0000 7fa69b7fe640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123380 0x7fa6b4123760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400e7a0 tx=0x7fa6a400eba0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.917+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123380 msgr2=0x7fa6b4123760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:00.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.917+0000 7fa69b7fe640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123380 0x7fa6b4123760 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400e7a0 tx=0x7fa6a400eba0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.917+0000 7fa69b7fe640 10 client.24160.objecter ms_handle_connect 0x7fa6a003fdb0 2024-06-11T19:20:00.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.917+0000 7fa6b27fc640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa6a003fdb0 0x7fa6a0042270 secure :-1 s=READY pgs=4200 cs=0 l=1 rev1=1 crypto rx=0x7fa6b41245a0 tx=0x7fa6a8008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b92b9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:00.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b92b9640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123ca0 0x7fa6b409e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:00.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b92b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:00.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2440003314 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa6b40a4f00 con 0x7fa6b4123ca0 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b2ffd640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123ca0 0x7fa6b409e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b2ffd640 10 monclient: get_auth_request con 0x7fa6b4123ca0 auth_method 0 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b2ffd640 10 monclient(hunting): handle_auth_done global_id 94301 payload 824 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b2ffd640 1 -- 172.21.15.98:0/2440003314 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa6b4137d90 con 0x7fa6b4123ca0 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa69b7fe640 10 client.24160.objecter ms_handle_connect 0x7fa6b4123ca0 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa6b2ffd640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123ca0 0x7fa6b409e9f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400e7a0 tx=0x7fa6a40046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:00.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.923+0000 7fa69b7fe640 10 client.24160.objecter resend_mon_ops 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6a4017070 con 0x7fa6b4123ca0 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa69b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa69b7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa69b7fe640 10 monclient: dump: 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:00.945 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fa6a4006390 con 0x7fa6b4123ca0 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa69b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa69b7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123ca0 msgr2=0x7fa6b409e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa69b7fe640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123ca0 0x7fa6b409e9f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400e7a0 tx=0x7fa6a40046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa69b7fe640 1 -- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123ca0 msgr2=0x7fa6b409e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:00.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa69b7fe640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa6b4123ca0 0x7fa6b409e9f0 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400e7a0 tx=0x7fa6a40046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.947 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:00.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:00.947 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:00.947 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:00.947 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:00.947 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 146478, 2024-06-11T19:20:00.947 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:00.947 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:00.947 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.948 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.949 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:00.950 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:00.951 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:00.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:00.954 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.954 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:00.954 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:00.954 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.954 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.954 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.954 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:00.959 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.962 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.964 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:00.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.968 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:00.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.969 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:00.969 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:00.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:00.969 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:00.969 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:00.969 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:00.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:00.969 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:00.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:00.969 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:00.969 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:00.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 10 librados: watch_flush enter 2024-06-11T19:20:00.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 10 librados: watch_flush exit 2024-06-11T19:20:00.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 20 client.24160.objecter shutdown clearing up homeless session... 2024-06-11T19:20:00.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 10 client.24160.objecter successfully canceled tick 2024-06-11T19:20:00.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2440003314 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa6a003fdb0 msgr2=0x7fa6a0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa6a003fdb0 0x7fa6a0042270 secure :-1 s=READY pgs=4200 cs=0 l=1 rev1=1 crypto rx=0x7fa6b41245a0 tx=0x7fa6a8008040 comp rx=0 tx=0).stop 2024-06-11T19:20:00.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 10 monclient: shutdown 2024-06-11T19:20:00.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:00.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a5240 msgr2=0x7fa6b4122e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:00.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a5240 0x7fa6b4122e40 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac037040 tx=0x7fa6ac004a40 comp rx=0 tx=0).stop 2024-06-11T19:20:00.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b37fe640 1 -- 172.21.15.98:0/2440003314 reap_dead start 2024-06-11T19:20:00.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2440003314 shutdown_connections 2024-06-11T19:20:00.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 1 --2- 172.21.15.98:0/2440003314 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa6b40a5240 0x7fa6b4122e40 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:00.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2440003314 >> 172.21.15.98:0/2440003314 conn(0x7fa6b40ab850 msgr2=0x7fa6b41bffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:00.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2440003314 shutdown_connections 2024-06-11T19:20:00.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 1 -- 172.21.15.98:0/2440003314 wait complete. 2024-06-11T19:20:00.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:00.924+0000 7fa6b92b9640 1 librados: shutdown 2024-06-11T19:20:00.973 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:00.973 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:20:00.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:20:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.073+0000 7fd9ddfd2640 10 monclient: get_monmap_and_config 2024-06-11T19:20:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.073+0000 7fd9ddfd2640 10 monclient: build_initial_monmap 2024-06-11T19:20:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.073+0000 7fd9ddfd2640 10 monclient: monmap: 2024-06-11T19:20:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:01.074199+0000 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:01.074199+0000 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:01.074 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 1 Processor -- start 2024-06-11T19:20:01.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 1 -- start start 2024-06-11T19:20:01.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 10 monclient: init 2024-06-11T19:20:01.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:01.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 10 monclient: _add_conns ranks=[9,8,5,6,0,3,4,7,2,1,10] 2024-06-11T19:20:01.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9d80a4920 0x7fd9d80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d80a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:20:01.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd9d80a5240 0x7fd9d80a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 10 monclient(hunting): picked mon.noname-e con 0x7fd9d80a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:01.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd9d80a2530 0x7fd9d80a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 10 monclient(hunting): picked mon.noname-h con 0x7fd9d80a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:01.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd9d80a7140 con 0x7fd9d80a4920 2024-06-11T19:20:01.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd9d80a2b50 con 0x7fd9d80a2530 2024-06-11T19:20:01.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd9d80a2e30 con 0x7fd9d80a5240 2024-06-11T19:20:01.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:01.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9ddfd2640 10 monclient(hunting): authenticate will time out at 1410.881622s 2024-06-11T19:20:01.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9cffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd9d80a5240 0x7fd9d80a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d77fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9d80a4920 0x7fd9d80a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9cffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd9d80a5240 0x7fd9d80a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44046/0 (socket says 172.21.15.98:44046) 2024-06-11T19:20:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d77fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9d80a4920 0x7fd9d80a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44820/0 (socket says 172.21.15.98:44820) 2024-06-11T19:20:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9cffff640 1 -- 172.21.15.98:0/2190645008 learned_addr learned my addr 172.21.15.98:0/2190645008 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9cffff640 10 monclient(hunting): get_auth_request con 0x7fd9d80a5240 auth_method 0 2024-06-11T19:20:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9cffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9cffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d77fe640 10 monclient(hunting): get_auth_request con 0x7fd9d80a4920 auth_method 0 2024-06-11T19:20:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d7fff640 1 --2- 172.21.15.98:0/2190645008 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd9d80a2530 0x7fd9d80a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d7fff640 10 monclient(hunting): get_auth_request con 0x7fd9d80a2530 auth_method 0 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.074+0000 7fd9d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_done global_id 65295 payload 274 2024-06-11T19:20:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d77fe640 1 -- 172.21.15.98:0/2190645008 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd9d80a5240 msgr2=0x7fd9d80a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d77fe640 1 --2- 172.21.15.98:0/2190645008 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd9d80a5240 0x7fd9d80a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d77fe640 1 -- 172.21.15.98:0/2190645008 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd9d80a2530 msgr2=0x7fd9d80a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d77fe640 1 --2- 172.21.15.98:0/2190645008 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd9d80a2530 0x7fd9d80a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d77fe640 10 monclient: _finish_hunting 0 2024-06-11T19:20:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d77fe640 1 monclient: found mon.noname-a 2024-06-11T19:20:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d77fe640 1 -- 172.21.15.98:0/2190645008 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d80a3110 con 0x7fd9d80a4920 2024-06-11T19:20:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d7fff640 1 --2- 172.21.15.98:0/2190645008 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd9d80a2530 0x7fd9d80a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:20:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d77fe640 1 --2- 172.21.15.98:0/2190645008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d80a4920 0x7fd9d80a4d00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80097a0 tx=0x7fd9c802b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a47cb5b9dd0d2a08 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d6ffd640 1 -- 172.21.15.98:0/2190645008 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd9c8037070 con 0x7fd9d80a4920 2024-06-11T19:20:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d6ffd640 10 monclient: dump: 2024-06-11T19:20:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.090 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d6ffd640 10 monclient: _finish_auth 0 2024-06-11T19:20:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d6ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:20:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d6ffd640 1 -- 172.21.15.98:0/2190645008 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c8030980 con 0x7fd9d80a4920 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9ddfd2640 5 monclient: authenticate success, global_id 65295 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9ddfd2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d6ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d6ffd640 1 -- 172.21.15.98:0/2190645008 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd9c8030bb0 con 0x7fd9d80a4920 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9ddfd2640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9ddfd2640 4 set_mon_vals no callback set 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.075+0000 7fd9d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9d6ffd640 10 monclient: dump: 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9ddfd2640 10 monclient: shutdown 2024-06-11T19:20:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9ddfd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/2190645008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d80a4920 msgr2=0x7fd9d80a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9ddfd2640 1 --2- 172.21.15.98:0/2190645008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d80a4920 0x7fd9d80a4d00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80097a0 tx=0x7fd9c802b6f0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/2190645008 shutdown_connections 2024-06-11T19:20:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9ddfd2640 1 --2- 172.21.15.98:0/2190645008 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd9d80a2530 0x7fd9d80a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9ddfd2640 1 --2- 172.21.15.98:0/2190645008 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd9d80a5240 0x7fd9d80a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9ddfd2640 1 --2- 172.21.15.98:0/2190645008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d80a4920 0x7fd9d80a4d00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/2190645008 >> 172.21.15.98:0/2190645008 conn(0x7fd9d80ab850 msgr2=0x7fd9d80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/2190645008 shutdown_connections 2024-06-11T19:20:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.076+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/2190645008 wait complete. 2024-06-11T19:20:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 10 monclient: build_initial_monmap 2024-06-11T19:20:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 10 monclient: monmap: 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 1 librados: starting msgr at 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 1 librados: starting objecter 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 1 Processor -- start 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 1 -- start start 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 1 librados: setting wanted keys 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 1 librados: calling monclient init 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 10 monclient: init 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 10 monclient: _add_conns ranks=[1,4,2,7,6,8,0,3,5] 2024-06-11T19:20:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d80a2530 0x7fd9d8137860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 10 monclient(hunting): picked mon.noname-f con 0x7fd9d80a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd9d80a4920 0x7fd9d8137da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d80a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.077+0000 7fd9ddfd2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d80a5240 0x7fd9d8123280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d80a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d80a6da0 con 0x7fd9d80a2530 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d8138e50 con 0x7fd9d80a5240 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd9d80a6420 con 0x7fd9d80a4920 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 10 monclient(hunting): authenticate will time out at 1410.885029s 2024-06-11T19:20:01.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9cffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd9d80a4920 0x7fd9d8137da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d80a5240 0x7fd9d8123280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9cffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd9d80a4920 0x7fd9d8137da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:41138/0 (socket says 172.21.15.98:41138) 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9cffff640 1 -- 172.21.15.98:0/3331067561 learned_addr learned my addr 172.21.15.98:0/3331067561 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9cffff640 10 monclient(hunting): get_auth_request con 0x7fd9d80a4920 auth_method 0 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9cffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9cffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 10 monclient(hunting): get_auth_request con 0x7fd9d80a5240 auth_method 0 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d77fe640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d80a2530 0x7fd9d8137860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d77fe640 10 monclient(hunting): get_auth_request con 0x7fd9d80a2530 auth_method 0 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 10 monclient(hunting): handle_auth_done global_id 84656 payload 824 2024-06-11T19:20:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 1 -- 172.21.15.98:0/3331067561 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd9d80a4920 msgr2=0x7fd9d8137da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd9d80a4920 0x7fd9d8137da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 1 -- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d80a2530 msgr2=0x7fd9d8137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d80a2530 0x7fd9d8137860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 1 monclient: found mon.noname-b 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 1 -- 172.21.15.98:0/3331067561 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d8126f60 con 0x7fd9d80a5240 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9cffff640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd9d80a4920 0x7fd9d8137da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d7fff640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d80a5240 0x7fd9d8123280 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0007ef0 tx=0x7fd9d000d4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 10 client.?.objecter ms_handle_connect 0x7fd9d80a5240 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd9d001a070 con 0x7fd9d80a5240 2024-06-11T19:20:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 10 monclient: dump: 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 10 monclient: _finish_auth 0 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d0003e00 con 0x7fd9d80a5240 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 5 monclient: authenticate success, global_id 84656 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 10 monclient: _renew_subs 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/3331067561 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d8075480 con 0x7fd9d80a5240 2024-06-11T19:20:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd9d00044a0 con 0x7fd9d80a5240 2024-06-11T19:20:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d4ff9640 10 monclient: dump: 2024-06-11T19:20:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9d67fc640 4 set_mon_vals no callback set 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 10 client.84656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 10 monclient: _renew_subs 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:20:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/3331067561 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d8127240 con 0x7fd9d80a5240 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.078+0000 7fd9ddfd2640 1 librados: init done 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd9d0004640 con 0x7fd9d80a5240 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9ddfd2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9ddfd2640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d81269a0 0x7fd9d81237c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9ddfd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/3331067561 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d81c0c40 con 0x7fd9d81269a0 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9d4ff9640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd9b803fdb0 0x7fd9b8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7fd9d0051c50 con 0x7fd9d80a5240 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9d4ff9640 10 client.84656.objecter ms_dispatch 0x7fd9d8000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9d4ff9640 3 client.84656.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9d4ff9640 3 client.84656.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9d77fe640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d81269a0 0x7fd9d81237c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9d77fe640 10 monclient: get_auth_request con 0x7fd9d81269a0 auth_method 0 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9d77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.079+0000 7fd9d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 20 client.84656.objecter dump_active .. 0 homeless 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9cffff640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd9b803fdb0 0x7fd9b8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9cffff640 10 monclient: get_auth_request con 0x7fd9b803fdb0 auth_method 0 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_done global_id 32392 payload 824 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d77fe640 1 -- 172.21.15.98:0/3331067561 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9d8075280 con 0x7fd9d81269a0 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 10 client.84656.objecter ms_handle_connect 0x7fd9d81269a0 2024-06-11T19:20:01.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 10 client.84656.objecter resend_mon_ops 2024-06-11T19:20:01.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d77fe640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d81269a0 0x7fd9d81237c0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8073de0 tx=0x7fd9c80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 10 client.84656.objecter ms_handle_connect 0x7fd9b803fdb0 2024-06-11T19:20:01.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9cffff640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd9b803fdb0 0x7fd9b8042270 secure :-1 s=READY pgs=4201 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8124d80 tx=0x7fd9c0009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd9c8037040 con 0x7fd9d81269a0 2024-06-11T19:20:01.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:20:01.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 10 monclient: dump: 2024-06-11T19:20:01.106 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.106 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.106 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.106 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.106 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd9c8038040 con 0x7fd9d81269a0 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:01.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:01.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d81269a0 msgr2=0x7fd9d81237c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d81269a0 0x7fd9d81237c0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8073de0 tx=0x7fd9c80046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d81269a0 msgr2=0x7fd9d81237c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:01.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.080+0000 7fd9d4ff9640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d81269a0 0x7fd9d81237c0 secure :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8073de0 tx=0x7fd9c80046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9ddfd2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:01.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9ddfd2640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d8123d00 0x7fd9d8124110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9ddfd2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/3331067561 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d8050830 con 0x7fd9d8123d00 2024-06-11T19:20:01.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9d77fe640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d8123d00 0x7fd9d8124110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9d77fe640 10 monclient: get_auth_request con 0x7fd9d8123d00 auth_method 0 2024-06-11T19:20:01.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9d77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9d77fe640 10 monclient(hunting): handle_auth_done global_id 32401 payload 824 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.087+0000 7fd9d77fe640 1 -- 172.21.15.98:0/3331067561 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd9d80a6750 con 0x7fd9d8123d00 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d77fe640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d8123d00 0x7fd9d8124110 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80096f0 tx=0x7fd9c8030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 10 client.84656.objecter ms_handle_connect 0x7fd9d8123d00 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 10 client.84656.objecter resend_mon_ops 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd9c8037040 con 0x7fd9d8123d00 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 10 monclient: dump: 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.111 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fd9c8030eb0 con 0x7fd9d8123d00 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d8123d00 msgr2=0x7fd9d8124110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d8123d00 0x7fd9d8124110 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80096f0 tx=0x7fd9c8030000 comp rx=0 tx=0).stop 2024-06-11T19:20:01.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 1 -- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d8123d00 msgr2=0x7fd9d8124110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:01.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9d4ff9640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d8123d00 0x7fd9d8124110 secure :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80096f0 tx=0x7fd9c8030000 comp rx=0 tx=0).stop 2024-06-11T19:20:01.113 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:01.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:01.113 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:01.113 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:01.113 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:01.113 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 651306, 2024-06-11T19:20:01.113 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:01.113 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:01.114 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:01.114 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:01.114 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:01.114 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:01.114 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:01.114 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:01.114 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:01.114 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:01.114 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.114 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:20:01.114 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:01.114 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.115 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:01.117 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:01.118 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:01.118 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:01.118 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:01.118 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:01.118 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:01.118 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:01.118 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:01.118 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:01.118 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:01.118 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:01.118 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:01.118 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.120 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:01.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.125 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.126 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:01.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:01.126 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:01.126 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:01.127 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.127 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.127 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:01.127 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:01.127 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.127 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.127 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:01.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.129 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.129 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:01.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:01.129 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.129 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.129 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.129 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:01.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:01.130 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:01.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:01.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.134 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:01.135 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:01.136 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.136 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.136 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:01.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.136 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:01.136 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:01.136 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:20:01.136 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.136 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.136 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:01.136 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:01.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9ddfd2640 10 librados: watch_flush enter 2024-06-11T19:20:01.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9ddfd2640 10 librados: watch_flush exit 2024-06-11T19:20:01.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9ddfd2640 20 client.84656.objecter shutdown clearing up homeless session... 2024-06-11T19:20:01.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.088+0000 7fd9ddfd2640 10 client.84656.objecter successfully canceled tick 2024-06-11T19:20:01.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd9b803fdb0 msgr2=0x7fd9b8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9ddfd2640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd9b803fdb0 0x7fd9b8042270 secure :-1 s=READY pgs=4201 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8124d80 tx=0x7fd9c0009240 comp rx=0 tx=0).stop 2024-06-11T19:20:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9ddfd2640 10 monclient: shutdown 2024-06-11T19:20:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9ddfd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/3331067561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d80a5240 msgr2=0x7fd9d8123280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9ddfd2640 1 --2- 172.21.15.98:0/3331067561 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9d80a5240 0x7fd9d8123280 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0007ef0 tx=0x7fd9d000d4a0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9d7fff640 1 -- 172.21.15.98:0/3331067561 reap_dead start 2024-06-11T19:20:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9d7fff640 1 -- 172.21.15.98:0/3331067561 reap_dead start 2024-06-11T19:20:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/3331067561 shutdown_connections 2024-06-11T19:20:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/3331067561 >> 172.21.15.98:0/3331067561 conn(0x7fd9d80ab850 msgr2=0x7fd9d81be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/3331067561 shutdown_connections 2024-06-11T19:20:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9ddfd2640 1 -- 172.21.15.98:0/3331067561 wait complete. 2024-06-11T19:20:01.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.089+0000 7fd9ddfd2640 1 librados: shutdown 2024-06-11T19:20:01.140 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:01.140 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:20:01.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:20:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.241+0000 7f1e78f0b640 10 monclient: get_monmap_and_config 2024-06-11T19:20:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.241+0000 7f1e78f0b640 10 monclient: build_initial_monmap 2024-06-11T19:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.241+0000 7f1e78f0b640 10 monclient: monmap: 2024-06-11T19:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:01.242474+0000 2024-06-11T19:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:01.242474+0000 2024-06-11T19:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:01.242 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 1 Processor -- start 2024-06-11T19:20:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 1 -- start start 2024-06-11T19:20:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 10 monclient: init 2024-06-11T19:20:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 10 monclient: _add_conns ranks=[5,7,3,2,1,9,0,6,4,8,10] 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1e740a4920 0x7f1e740a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 10 monclient(hunting): picked mon.noname-h con 0x7f1e740a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1e740a5240 0x7f1e740a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 10 monclient(hunting): picked mon.noname-i con 0x7f1e740a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e740a2530 0x7f1e740a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 10 monclient(hunting): picked mon.noname-g con 0x7f1e740a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f1e740a7140 con 0x7f1e740a2530 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f1e740a2b50 con 0x7f1e740a4920 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f1e740a2e30 con 0x7f1e740a5240 2024-06-11T19:20:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.242+0000 7f1e78f0b640 10 monclient(hunting): authenticate will time out at 1411.049807s 2024-06-11T19:20:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e71d9b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1e740a5240 0x7f1e740a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1e740a4920 0x7f1e740a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e72d9d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e740a2530 0x7f1e740a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1e740a4920 0x7f1e740a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:37016/0 (socket says 172.21.15.98:37016) 2024-06-11T19:20:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e71d9b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1e740a5240 0x7f1e740a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:45770/0 (socket says 172.21.15.98:45770) 2024-06-11T19:20:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 1 -- 172.21.15.98:0/842446272 learned_addr learned my addr 172.21.15.98:0/842446272 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 10 monclient(hunting): get_auth_request con 0x7f1e740a4920 auth_method 0 2024-06-11T19:20:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e71d9b640 10 monclient(hunting): get_auth_request con 0x7f1e740a5240 auth_method 0 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e71d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e71d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e72d9d640 10 monclient(hunting): get_auth_request con 0x7f1e740a2530 auth_method 0 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e72d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e72d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e72d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e71d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e71d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e71d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e72d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e72d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e72d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 10 monclient(hunting): handle_auth_done global_id 23216 payload 274 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 1 -- 172.21.15.98:0/842446272 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1e740a5240 msgr2=0x7f1e740a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 1 --2- 172.21.15.98:0/842446272 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1e740a5240 0x7f1e740a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 1 -- 172.21.15.98:0/842446272 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e740a2530 msgr2=0x7f1e740a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 1 --2- 172.21.15.98:0/842446272 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e740a2530 0x7f1e740a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 10 monclient: _finish_hunting 0 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 1 monclient: found mon.noname-h 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e7259c640 1 -- 172.21.15.98:0/842446272 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e740a3110 con 0x7f1e740a4920 2024-06-11T19:20:01.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e71d9b640 1 --2- 172.21.15.98:0/842446272 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1e740a5240 0x7f1e740a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.243+0000 7f1e72d9d640 1 --2- 172.21.15.98:0/842446272 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e740a2530 0x7f1e740a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7259c640 1 --2- 172.21.15.98:0/842446272 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1e740a4920 0x7f1e740a4d00 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c009e80 tx=0x7f1e6c02bd70 comp rx=0 tx=0).ready entity=mon.5 client_cookie=54b2c538769a494f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 1 -- 172.21.15.98:0/842446272 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e6c02cb80 con 0x7f1e740a4920 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 10 monclient: dump: 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.257 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 10 monclient: _finish_auth 0 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 10 monclient: _check_auth_tickets 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:01.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e78f0b640 5 monclient: authenticate success, global_id 23216 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 1 -- 172.21.15.98:0/842446272 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6c02cd40 con 0x7f1e740a4920 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e78f0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e78f0b640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e78f0b640 4 set_mon_vals no callback set 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 1 -- 172.21.15.98:0/842446272 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e6c034860 con 0x7f1e740a4920 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e7159a640 10 monclient: dump: 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.259 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e78f0b640 10 monclient: shutdown 2024-06-11T19:20:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e78f0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e78f0b640 1 -- 172.21.15.98:0/842446272 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1e740a4920 msgr2=0x7f1e740a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.244+0000 7f1e78f0b640 1 --2- 172.21.15.98:0/842446272 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1e740a4920 0x7f1e740a4d00 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c009e80 tx=0x7f1e6c02bd70 comp rx=0 tx=0).stop 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.245+0000 7f1e78f0b640 1 -- 172.21.15.98:0/842446272 shutdown_connections 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.245+0000 7f1e78f0b640 1 --2- 172.21.15.98:0/842446272 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e740a2530 0x7f1e740a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.245+0000 7f1e78f0b640 1 --2- 172.21.15.98:0/842446272 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1e740a5240 0x7f1e740a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.245+0000 7f1e78f0b640 1 --2- 172.21.15.98:0/842446272 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1e740a4920 0x7f1e740a4d00 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.245+0000 7f1e78f0b640 1 -- 172.21.15.98:0/842446272 >> 172.21.15.98:0/842446272 conn(0x7f1e740ab850 msgr2=0x7f1e740a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.245+0000 7f1e78f0b640 1 -- 172.21.15.98:0/842446272 shutdown_connections 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.245+0000 7f1e78f0b640 1 -- 172.21.15.98:0/842446272 wait complete. 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.245+0000 7f1e78f0b640 10 monclient: build_initial_monmap 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.245+0000 7f1e78f0b640 10 monclient: monmap: 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:01.262 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.245+0000 7f1e78f0b640 1 librados: starting msgr at 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.245+0000 7f1e78f0b640 1 librados: starting objecter 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 1 Processor -- start 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 1 -- start start 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 1 librados: setting wanted keys 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 1 librados: calling monclient init 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 10 monclient: init 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 10 monclient: _add_conns ranks=[0,3,4,8,6,2,1,5,7] 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740a2530 0x7f1e741cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 10 monclient(hunting): picked mon.noname-a con 0x7f1e740a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e740a4920 0x7f1e741ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 10 monclient(hunting): picked mon.noname-g con 0x7f1e740a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e740a5240 0x7f1e74122e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 10 monclient(hunting): picked mon.noname-c con 0x7f1e740a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e740a6da0 con 0x7f1e740a2530 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f1e74138e50 con 0x7f1e740a4920 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1e740a6420 con 0x7f1e740a5240 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e78f0b640 10 monclient(hunting): authenticate will time out at 1411.053570s 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e7259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740a2530 0x7f1e741cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e740a5240 0x7f1e74122e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e7259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740a2530 0x7f1e741cddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44832/0 (socket says 172.21.15.98:44832) 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e7259c640 1 -- 172.21.15.98:0/1116151699 learned_addr learned my addr 172.21.15.98:0/1116151699 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e7259c640 10 monclient(hunting): get_auth_request con 0x7f1e740a2530 auth_method 0 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e7259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e7259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 10 monclient(hunting): get_auth_request con 0x7f1e740a5240 auth_method 0 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e71d9b640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e740a4920 0x7f1e741ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e71d9b640 10 monclient(hunting): get_auth_request con 0x7f1e740a4920 auth_method 0 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e71d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e71d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 10 monclient(hunting): handle_auth_done global_id 24178 payload 824 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 1 -- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e740a4920 msgr2=0x7f1e741ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e740a4920 0x7f1e741ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 1 -- 172.21.15.98:0/1116151699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740a2530 msgr2=0x7f1e741cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740a2530 0x7f1e741cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:20:01.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.246+0000 7f1e72d9d640 1 monclient: found mon.noname-c 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e72d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e72d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e72d9d640 1 -- 172.21.15.98:0/1116151699 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e74126e40 con 0x7f1e740a5240 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e7259c640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e740a2530 0x7f1e741cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e72d9d640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e740a5240 0x7f1e74122e80 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400e850 tx=0x7f1e6400ed10 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 10 client.?.objecter ms_handle_connect 0x7f1e740a5240 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e64017070 con 0x7f1e740a5240 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 10 monclient: dump: 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.267 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 10 monclient: _finish_auth 0 2024-06-11T19:20:01.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 10 monclient: _check_auth_tickets 2024-06-11T19:20:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e78f0b640 5 monclient: authenticate success, global_id 24178 2024-06-11T19:20:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6400cdb0 con 0x7f1e740a5240 2024-06-11T19:20:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e640138e0 con 0x7f1e740a5240 2024-06-11T19:20:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e78f0b640 10 monclient: _renew_subs 2024-06-11T19:20:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e78f0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e78f0b640 1 -- 172.21.15.98:0/1116151699 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e740a9160 con 0x7f1e740a5240 2024-06-11T19:20:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e70d99640 4 set_mon_vals no callback set 2024-06-11T19:20:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:01.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e78f0b640 10 client.24178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e6affd640 10 monclient: dump: 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.270 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e78f0b640 10 monclient: _renew_subs 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e78f0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e78f0b640 1 -- 172.21.15.98:0/1116151699 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e74124b50 con 0x7f1e740a5240 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.247+0000 7f1e78f0b640 1 librados: init done 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e78f0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e78f0b640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74127810 0x7f1e741233c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e78f0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e78f0b640 1 -- 172.21.15.98:0/1116151699 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f1e74050830 con 0x7f1e74127810 2024-06-11T19:20:01.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f1e64013470 con 0x7f1e740a5240 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e7259c640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74127810 0x7f1e741233c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e7259c640 10 monclient: get_auth_request con 0x7f1e74127810 auth_method 0 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e7259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e7259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e6affd640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1e5403fde0 0x7f1e540422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7f1e6400f040 con 0x7f1e740a5240 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e6affd640 10 client.24178.objecter ms_dispatch 0x7f1e74000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e6affd640 3 client.24178.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:01.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e6affd640 3 client.24178.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e6affd640 20 client.24178.objecter dump_active .. 0 homeless 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e7259c640 10 monclient(hunting): handle_auth_done global_id 104124 payload 824 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e71d9b640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1e5403fde0 0x7f1e540422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e7259c640 1 -- 172.21.15.98:0/1116151699 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e740a6720 con 0x7f1e74127810 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e71d9b640 10 monclient: get_auth_request con 0x7f1e5403fde0 auth_method 0 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e6affd640 10 client.24178.objecter ms_handle_connect 0x7f1e74127810 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e6affd640 10 client.24178.objecter resend_mon_ops 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.248+0000 7f1e7259c640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74127810 0x7f1e741233c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c009e80 tx=0x7f1e6c004a40 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e6c043030 con 0x7f1e74127810 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e71d9b640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1e5403fde0 0x7f1e540422a0 secure :-1 s=READY pgs=4202 cs=0 l=1 rev1=1 crypto rx=0x7f1e74125c30 tx=0x7f1e5c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e6affd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:20:01.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e6affd640 10 monclient: dump: 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.274 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.275 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.275 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e6affd640 10 client.24178.objecter ms_handle_connect 0x7f1e5403fde0 2024-06-11T19:20:01.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1e6c004250 con 0x7f1e74127810 2024-06-11T19:20:01.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e6affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:01.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e6affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:01.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74127810 msgr2=0x7f1e741233c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e6affd640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74127810 0x7f1e741233c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c009e80 tx=0x7f1e6c004a40 comp rx=0 tx=0).stop 2024-06-11T19:20:01.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74127810 msgr2=0x7f1e741233c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:01.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.249+0000 7f1e6affd640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74127810 0x7f1e741233c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.255+0000 7f1e78f0b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:01.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.255+0000 7f1e78f0b640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74123900 0x7f1e74123d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.255+0000 7f1e78f0b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.255+0000 7f1e78f0b640 1 -- 172.21.15.98:0/1116151699 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f1e74069740 con 0x7f1e74123900 2024-06-11T19:20:01.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e7259c640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74123900 0x7f1e74123d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e7259c640 10 monclient: get_auth_request con 0x7f1e74123900 auth_method 0 2024-06-11T19:20:01.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e7259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e7259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e7259c640 10 monclient(hunting): handle_auth_done global_id 104133 payload 824 2024-06-11T19:20:01.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e7259c640 1 -- 172.21.15.98:0/1116151699 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1e74124250 con 0x7f1e74123900 2024-06-11T19:20:01.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e7259c640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74123900 0x7f1e74123d10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1e74068dd0 tx=0x7f1e6c009000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e6affd640 10 client.24178.objecter ms_handle_connect 0x7f1e74123900 2024-06-11T19:20:01.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e6affd640 10 client.24178.objecter resend_mon_ops 2024-06-11T19:20:01.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e6c043030 con 0x7f1e74123900 2024-06-11T19:20:01.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e6affd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:20:01.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.256+0000 7f1e6affd640 10 monclient: dump: 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.279 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.280 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.280 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f1e6c052470 con 0x7f1e74123900 2024-06-11T19:20:01.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e6affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:01.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e6affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:01.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74123900 msgr2=0x7f1e74123d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e6affd640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74123900 0x7f1e74123d10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1e74068dd0 tx=0x7f1e6c009000 comp rx=0 tx=0).stop 2024-06-11T19:20:01.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e6affd640 1 -- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74123900 msgr2=0x7f1e74123d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:01.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e6affd640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e74123900 0x7f1e74123d10 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1e74068dd0 tx=0x7f1e6c009000 comp rx=0 tx=0).stop 2024-06-11T19:20:01.280 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 16461, 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:01.281 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:01.282 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:01.282 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:01.282 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.282 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:20:01.282 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:01.282 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:01.282 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:01.282 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:01.282 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:01.282 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:01.282 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:01.282 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:01.283 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:01.284 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:01.285 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:01.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:01.288 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.290 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:01.290 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:01.290 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.290 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.290 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.290 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:01.290 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:01.290 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.290 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:01.291 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:01.294 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:01.295 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.298 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.300 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:01.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:01.301 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:01.301 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:01.301 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.304 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:01.304 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:01.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 10 librados: watch_flush enter 2024-06-11T19:20:01.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 10 librados: watch_flush exit 2024-06-11T19:20:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 20 client.24178.objecter shutdown clearing up homeless session... 2024-06-11T19:20:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 10 client.24178.objecter successfully canceled tick 2024-06-11T19:20:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 1 -- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1e5403fde0 msgr2=0x7f1e540422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1e5403fde0 0x7f1e540422a0 secure :-1 s=READY pgs=4202 cs=0 l=1 rev1=1 crypto rx=0x7f1e74125c30 tx=0x7f1e5c008040 comp rx=0 tx=0).stop 2024-06-11T19:20:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 10 monclient: shutdown 2024-06-11T19:20:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 1 -- 172.21.15.98:0/1116151699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e740a5240 msgr2=0x7f1e74122e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 1 --2- 172.21.15.98:0/1116151699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e740a5240 0x7f1e74122e80 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400e850 tx=0x7f1e6400ed10 comp rx=0 tx=0).stop 2024-06-11T19:20:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e72d9d640 1 -- 172.21.15.98:0/1116151699 reap_dead start 2024-06-11T19:20:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e72d9d640 1 -- 172.21.15.98:0/1116151699 reap_dead start 2024-06-11T19:20:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 1 -- 172.21.15.98:0/1116151699 shutdown_connections 2024-06-11T19:20:01.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 1 -- 172.21.15.98:0/1116151699 >> 172.21.15.98:0/1116151699 conn(0x7f1e740ab850 msgr2=0x7f1e741bfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:01.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 1 -- 172.21.15.98:0/1116151699 shutdown_connections 2024-06-11T19:20:01.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 1 -- 172.21.15.98:0/1116151699 wait complete. 2024-06-11T19:20:01.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.257+0000 7f1e78f0b640 1 librados: shutdown 2024-06-11T19:20:01.307 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:01.307 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:20:01.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:20:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.407+0000 7ff10fb3f640 10 monclient: get_monmap_and_config 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.407+0000 7ff10fb3f640 10 monclient: build_initial_monmap 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.407+0000 7ff10fb3f640 10 monclient: monmap: 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:01.408150+0000 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:01.408150+0000 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:01.408 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.407+0000 7ff10fb3f640 1 Processor -- start 2024-06-11T19:20:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.407+0000 7ff10fb3f640 1 -- start start 2024-06-11T19:20:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.407+0000 7ff10fb3f640 10 monclient: init 2024-06-11T19:20:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 10 monclient: _add_conns ranks=[5,2,1,4,8,9,7,0,10,6,3] 2024-06-11T19:20:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff1080a4920 0x7ff1080a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 10 monclient(hunting): picked mon.noname-h con 0x7ff1080a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1080a5240 0x7ff1080a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 10 monclient(hunting): picked mon.noname-b con 0x7ff1080a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff1080a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:20:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1080a7140 con 0x7ff1080a2530 2024-06-11T19:20:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff1080a2b50 con 0x7ff1080a5240 2024-06-11T19:20:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ff1080a2e30 con 0x7ff1080a4920 2024-06-11T19:20:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10fb3f640 10 monclient(hunting): authenticate will time out at 1411.215494s 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1080a5240 0x7ff1080a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1080a5240 0x7ff1080a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:46230/0 (socket says 172.21.15.98:46230) 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 1 -- 172.21.15.98:0/361610180 learned_addr learned my addr 172.21.15.98:0/361610180 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 10 monclient(hunting): get_auth_request con 0x7ff1080a5240 auth_method 0 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d8db640 1 --2- 172.21.15.98:0/361610180 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff1080a4920 0x7ff1080a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 10 monclient(hunting): handle_auth_done global_id 84665 payload 274 2024-06-11T19:20:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 1 -- 172.21.15.98:0/361610180 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff1080a4920 msgr2=0x7ff1080a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 1 --2- 172.21.15.98:0/361610180 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff1080a4920 0x7ff1080a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 1 -- 172.21.15.98:0/361610180 >> v1:172.21.15.117:6789/0 conn(0x7ff1080a2530 legacy=0x7ff1080a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 10 monclient: _finish_hunting 0 2024-06-11T19:20:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 1 monclient: found mon.noname-b 2024-06-11T19:20:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:20:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d0da640 1 -- 172.21.15.98:0/361610180 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1080a3110 con 0x7ff1080a5240 2024-06-11T19:20:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d8db640 10 monclient: get_auth_request con 0x7ff1080a4920 auth_method 0 2024-06-11T19:20:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.408+0000 7ff10d8db640 1 --2- 172.21.15.98:0/361610180 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff1080a4920 0x7ff1080a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:20:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10d0da640 1 --2- 172.21.15.98:0/361610180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1080a5240 0x7ff1080a59a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80098a0 tx=0x7ff0f802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afd83c4ffb6120b0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 1 -- 172.21.15.98:0/361610180 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0f8038070 con 0x7ff1080a5240 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 10 monclient: dump: 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 10 monclient: _finish_auth 0 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 10 monclient: _check_auth_tickets 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 1 -- 172.21.15.98:0/361610180 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f8031970 con 0x7ff1080a5240 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10fb3f640 5 monclient: authenticate success, global_id 84665 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10fb3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10fb3f640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 1 -- 172.21.15.98:0/361610180 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0f8031ba0 con 0x7ff1080a5240 2024-06-11T19:20:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10fb3f640 4 set_mon_vals no callback set 2024-06-11T19:20:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.409+0000 7ff10c8d9640 10 monclient: dump: 2024-06-11T19:20:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.410+0000 7ff10fb3f640 10 monclient: shutdown 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.410+0000 7ff10fb3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.410+0000 7ff10fb3f640 1 -- 172.21.15.98:0/361610180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1080a5240 msgr2=0x7ff1080a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.410+0000 7ff10fb3f640 1 --2- 172.21.15.98:0/361610180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1080a5240 0x7ff1080a59a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80098a0 tx=0x7ff0f802b7a0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.410+0000 7ff10fb3f640 1 -- 172.21.15.98:0/361610180 shutdown_connections 2024-06-11T19:20:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.410+0000 7ff10fb3f640 1 --2- 172.21.15.98:0/361610180 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff1080a5240 0x7ff1080a59a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.410+0000 7ff10fb3f640 1 --2- 172.21.15.98:0/361610180 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff1080a4920 0x7ff1080a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.410+0000 7ff10fb3f640 1 -- 172.21.15.98:0/361610180 >> 172.21.15.98:0/361610180 conn(0x7ff1080ab850 msgr2=0x7ff1080a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.410+0000 7ff10fb3f640 1 -- 172.21.15.98:0/361610180 shutdown_connections 2024-06-11T19:20:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.410+0000 7ff10fb3f640 1 -- 172.21.15.98:0/361610180 wait complete. 2024-06-11T19:20:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.410+0000 7ff10fb3f640 10 monclient: build_initial_monmap 2024-06-11T19:20:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.410+0000 7ff10fb3f640 10 monclient: monmap: 2024-06-11T19:20:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 1 librados: starting msgr at 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 1 librados: starting objecter 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 1 Processor -- start 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 1 -- start start 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 1 librados: setting wanted keys 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 1 librados: calling monclient init 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 10 monclient: init 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 10 monclient: _add_conns ranks=[4,3,6,5,0,8,1,7,2] 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff1080a2530 0x7ff108137620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 10 monclient(hunting): picked mon.noname-c con 0x7ff1080a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1080a4920 0x7ff108137b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 10 monclient(hunting): picked mon.noname-g con 0x7ff1080a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff1080a5240 0x7ff1081230e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 10 monclient(hunting): picked mon.noname-d con 0x7ff1080a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ff1080a6da0 con 0x7ff1080a4920 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff108138e50 con 0x7ff1080a2530 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff1080a65f0 con 0x7ff1080a5240 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10fb3f640 10 monclient(hunting): authenticate will time out at 1411.218772s 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10d8db640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff1080a2530 0x7ff108137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10d8db640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff1080a2530 0x7ff108137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:41158/0 (socket says 172.21.15.98:41158) 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10d8db640 1 -- 172.21.15.98:0/1599950000 learned_addr learned my addr 172.21.15.98:0/1599950000 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10d8db640 10 monclient(hunting): get_auth_request con 0x7ff1080a2530 auth_method 0 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10d8db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10d8db640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10d8db640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10e0dc640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff1080a5240 0x7ff1081230e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10e0dc640 10 monclient(hunting): get_auth_request con 0x7ff1080a5240 auth_method 0 2024-06-11T19:20:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10e0dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10e0dc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10e0dc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10d8db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10d8db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10d8db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10e0dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.411+0000 7ff10e0dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10e0dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d0da640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1080a4920 0x7ff108137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d0da640 10 monclient(hunting): get_auth_request con 0x7ff1080a4920 auth_method 0 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d0da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d0da640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d0da640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d8db640 10 monclient(hunting): handle_auth_done global_id 24187 payload 824 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d8db640 1 -- 172.21.15.98:0/1599950000 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff1080a5240 msgr2=0x7ff1081230e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d8db640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff1080a5240 0x7ff1081230e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d8db640 1 -- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1080a4920 msgr2=0x7ff108137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d8db640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1080a4920 0x7ff108137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d8db640 10 monclient: _finish_hunting 0 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d8db640 1 monclient: found mon.noname-c 2024-06-11T19:20:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d8db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d8db640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d8db640 1 -- 172.21.15.98:0/1599950000 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1080a8da0 con 0x7ff1080a2530 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10d8db640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff1080a2530 0x7ff108137620 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7ff10400c7f0 tx=0x7ff10400cbf0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 10 client.?.objecter ms_handle_connect 0x7ff1080a2530 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff104015070 con 0x7ff1080a2530 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 10 monclient: dump: 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 10 monclient: _finish_auth 0 2024-06-11T19:20:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:20:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff10400ce40 con 0x7ff1080a2530 2024-06-11T19:20:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10fb3f640 5 monclient: authenticate success, global_id 24187 2024-06-11T19:20:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10fb3f640 10 monclient: _renew_subs 2024-06-11T19:20:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10fb3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10fb3f640 1 -- 172.21.15.98:0/1599950000 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1080a9280 con 0x7ff1080a2530 2024-06-11T19:20:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff104007ca0 con 0x7ff1080a2530 2024-06-11T19:20:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fe7fc640 10 monclient: dump: 2024-06-11T19:20:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.436 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10fb3f640 10 client.24187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10fb3f640 10 monclient: _renew_subs 2024-06-11T19:20:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10fb3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10fb3f640 1 -- 172.21.15.98:0/1599950000 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ff108126d20 con 0x7ff1080a2530 2024-06-11T19:20:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff10fb3f640 1 librados: init done 2024-06-11T19:20:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.412+0000 7ff0fffff640 4 set_mon_vals no callback set 2024-06-11T19:20:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10fb3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10fb3f640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff108063180 0x7ff108063560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10fb3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10fb3f640 1 -- 172.21.15.98:0/1599950000 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ff108050830 con 0x7ff108063180 2024-06-11T19:20:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff104018070 con 0x7ff1080a2530 2024-06-11T19:20:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10d0da640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff108063180 0x7ff108063560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10d0da640 10 monclient: get_auth_request con 0x7ff108063180 auth_method 0 2024-06-11T19:20:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10d0da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10d0da640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10d0da640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10d0da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10d0da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff0fe7fc640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0ec03fde0 0x7ff0ec0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10d0da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7ff104051960 con 0x7ff1080a2530 2024-06-11T19:20:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff0fe7fc640 10 client.24187.objecter ms_dispatch 0x7ff108000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff0fe7fc640 3 client.24187.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:01.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff0fe7fc640 3 client.24187.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:01.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10d0da640 10 monclient(hunting): handle_auth_done global_id 23225 payload 824 2024-06-11T19:20:01.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10d0da640 1 -- 172.21.15.98:0/1599950000 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1080a4d00 con 0x7ff108063180 2024-06-11T19:20:01.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff0fe7fc640 20 client.24187.objecter dump_active .. 0 homeless 2024-06-11T19:20:01.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10e0dc640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0ec03fde0 0x7ff0ec0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.413+0000 7ff10e0dc640 10 monclient: get_auth_request con 0x7ff0ec03fde0 auth_method 0 2024-06-11T19:20:01.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 10 client.24187.objecter ms_handle_connect 0x7ff108063180 2024-06-11T19:20:01.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 10 client.24187.objecter resend_mon_ops 2024-06-11T19:20:01.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff10d0da640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff108063180 0x7ff108063560 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7ff1080abf80 tx=0x7ff0f80046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0f8043070 con 0x7ff108063180 2024-06-11T19:20:01.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:20:01.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 10 monclient: dump: 2024-06-11T19:20:01.439 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.441 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.442 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 10 client.24187.objecter ms_handle_connect 0x7ff0ec03fde0 2024-06-11T19:20:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff10e0dc640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0ec03fde0 0x7ff0ec0422a0 secure :-1 s=READY pgs=4203 cs=0 l=1 rev1=1 crypto rx=0x7ff1080a2950 tx=0x7ff100003d10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff0f8003ad0 con 0x7ff108063180 2024-06-11T19:20:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff108063180 msgr2=0x7ff108063560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff108063180 0x7ff108063560 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7ff1080abf80 tx=0x7ff0f80046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff108063180 msgr2=0x7ff108063560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:01.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.414+0000 7ff0fe7fc640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff108063180 0x7ff108063560 secure :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7ff1080abf80 tx=0x7ff0f80046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.420+0000 7ff10fb3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10fb3f640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff10802cb40 0x7ff10802cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10fb3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10fb3f640 1 -- 172.21.15.98:0/1599950000 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ff1080a0100 con 0x7ff10802cb40 2024-06-11T19:20:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10d0da640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff10802cb40 0x7ff10802cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10d0da640 10 monclient: get_auth_request con 0x7ff10802cb40 auth_method 0 2024-06-11T19:20:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10d0da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10d0da640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10d0da640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10d0da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10d0da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10d0da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10d0da640 10 monclient(hunting): handle_auth_done global_id 23234 payload 824 2024-06-11T19:20:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10d0da640 1 -- 172.21.15.98:0/1599950000 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff108126450 con 0x7ff10802cb40 2024-06-11T19:20:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.421+0000 7ff10d0da640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff10802cb40 0x7ff10802cf50 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097c0 tx=0x7ff0f8031000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 10 client.24187.objecter ms_handle_connect 0x7ff10802cb40 2024-06-11T19:20:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 10 client.24187.objecter resend_mon_ops 2024-06-11T19:20:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0f8038040 con 0x7ff10802cb40 2024-06-11T19:20:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:20:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 10 monclient: dump: 2024-06-11T19:20:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.449 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7ff0f803e440 con 0x7ff10802cb40 2024-06-11T19:20:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff10802cb40 msgr2=0x7ff10802cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff10802cb40 0x7ff10802cf50 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097c0 tx=0x7ff0f8031000 comp rx=0 tx=0).stop 2024-06-11T19:20:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 1 -- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff10802cb40 msgr2=0x7ff10802cf50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff0fe7fc640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff10802cb40 0x7ff10802cf50 secure :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097c0 tx=0x7ff0f8031000 comp rx=0 tx=0).stop 2024-06-11T19:20:01.449 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 651616, 2024-06-11T19:20:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:20:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:01.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.467 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.469 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:01.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.472 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.473 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.474 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:01.474 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:01.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.422+0000 7ff10fb3f640 10 librados: watch_flush enter 2024-06-11T19:20:01.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 10 librados: watch_flush exit 2024-06-11T19:20:01.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 20 client.24187.objecter shutdown clearing up homeless session... 2024-06-11T19:20:01.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 10 client.24187.objecter successfully canceled tick 2024-06-11T19:20:01.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 1 -- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0ec03fde0 msgr2=0x7ff0ec0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0ec03fde0 0x7ff0ec0422a0 secure :-1 s=READY pgs=4203 cs=0 l=1 rev1=1 crypto rx=0x7ff1080a2950 tx=0x7ff100003d10 comp rx=0 tx=0).stop 2024-06-11T19:20:01.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 10 monclient: shutdown 2024-06-11T19:20:01.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:01.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 1 -- 172.21.15.98:0/1599950000 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff1080a2530 msgr2=0x7ff108137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 1 --2- 172.21.15.98:0/1599950000 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff1080a2530 0x7ff108137620 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7ff10400c7f0 tx=0x7ff10400cbf0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10e0dc640 1 -- 172.21.15.98:0/1599950000 reap_dead start 2024-06-11T19:20:01.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10e0dc640 1 -- 172.21.15.98:0/1599950000 reap_dead start 2024-06-11T19:20:01.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 1 -- 172.21.15.98:0/1599950000 shutdown_connections 2024-06-11T19:20:01.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 1 -- 172.21.15.98:0/1599950000 >> 172.21.15.98:0/1599950000 conn(0x7ff1080ab850 msgr2=0x7ff1080a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:01.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 1 -- 172.21.15.98:0/1599950000 shutdown_connections 2024-06-11T19:20:01.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 1 -- 172.21.15.98:0/1599950000 wait complete. 2024-06-11T19:20:01.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.423+0000 7ff10fb3f640 1 librados: shutdown 2024-06-11T19:20:01.477 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:01.477 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:20:01.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:20:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.578+0000 7f45897f3640 10 monclient: get_monmap_and_config 2024-06-11T19:20:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.578+0000 7f45897f3640 10 monclient: build_initial_monmap 2024-06-11T19:20:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.578+0000 7f45897f3640 10 monclient: monmap: 2024-06-11T19:20:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:01.579540+0000 2024-06-11T19:20:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:01.579540+0000 2024-06-11T19:20:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:01.579 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 1 Processor -- start 2024-06-11T19:20:01.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 1 -- start start 2024-06-11T19:20:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 10 monclient: init 2024-06-11T19:20:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 10 monclient: _add_conns ranks=[10,4,1,3,8,9,0,6,7,2,5] 2024-06-11T19:20:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f45840a2f40 0x7f45840a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 10 monclient(hunting): picked mon.noname-f con 0x7f45840a2f40 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 0x7f45840a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 10 monclient(hunting): picked mon.noname-c con 0x7f45840a4df0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f45841366d0 addr v1:172.21.15.117:6789/0 2024-06-11T19:20:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45840a0ed0 con 0x7f45841366d0 2024-06-11T19:20:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f45840a5710 con 0x7f45840a2f40 2024-06-11T19:20:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f45840a3320 con 0x7f45840a4df0 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45897f3640 10 monclient(hunting): authenticate will time out at 1411.386732s 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45827fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 0x7f45840a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45827fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 0x7f45840a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:41164/0 (socket says 172.21.15.98:41164) 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45827fc640 1 -- 172.21.15.98:0/2229842682 learned_addr learned my addr 172.21.15.98:0/2229842682 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45827fc640 10 monclient(hunting): get_auth_request con 0x7f45840a4df0 auth_method 0 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45827fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45827fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.579+0000 7f45827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f4582ffd640 1 --2- 172.21.15.98:0/2229842682 >> v2:172.21.15.117:3300/0 conn(0x7f45840a2f40 0x7f45840a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f45827fc640 10 monclient(hunting): handle_auth_done global_id 24196 payload 274 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f45827fc640 1 -- 172.21.15.98:0/2229842682 >> v2:172.21.15.117:3300/0 conn(0x7f45840a2f40 msgr2=0x7f45840a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f45827fc640 1 --2- 172.21.15.98:0/2229842682 >> v2:172.21.15.117:3300/0 conn(0x7f45840a2f40 0x7f45840a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f45827fc640 1 -- 172.21.15.98:0/2229842682 >> v1:172.21.15.117:6789/0 conn(0x7f45841366d0 legacy=0x7f4584136ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f45827fc640 10 monclient: _finish_hunting 0 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f45827fc640 1 monclient: found mon.noname-c 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f45827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f45827fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f45827fc640 1 -- 172.21.15.98:0/2229842682 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45840a2180 con 0x7f45840a4df0 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f4582ffd640 10 monclient: get_auth_request con 0x7f45840a2f40 auth_method 0 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f4582ffd640 1 --2- 172.21.15.98:0/2229842682 >> v2:172.21.15.117:3300/0 conn(0x7f45840a2f40 0x7f45840a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.580+0000 7f45827fc640 1 --2- 172.21.15.98:0/2229842682 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 0x7f45840a51d0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f4578009870 tx=0x7f457802b7c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=2f598cd802f5f24e server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 1 -- 172.21.15.98:0/2229842682 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4578038070 con 0x7f45840a4df0 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 10 monclient: dump: 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 10 monclient: _finish_auth 0 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 1 -- 172.21.15.98:0/2229842682 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4578031990 con 0x7f45840a4df0 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 5 monclient: authenticate success, global_id 24196 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 1 -- 172.21.15.98:0/2229842682 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45780323b0 con 0x7f45840a4df0 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 4 set_mon_vals no callback set 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f4581ffb640 10 monclient: dump: 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 10 monclient: shutdown 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 1 -- 172.21.15.98:0/2229842682 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 msgr2=0x7f45840a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 1 --2- 172.21.15.98:0/2229842682 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 0x7f45840a51d0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f4578009870 tx=0x7f457802b7c0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 1 -- 172.21.15.98:0/2229842682 shutdown_connections 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 1 --2- 172.21.15.98:0/2229842682 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 0x7f45840a51d0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 1 --2- 172.21.15.98:0/2229842682 >> v2:172.21.15.117:3300/0 conn(0x7f45840a2f40 0x7f45840a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 1 -- 172.21.15.98:0/2229842682 >> 172.21.15.98:0/2229842682 conn(0x7f45840a95b0 msgr2=0x7f45840a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 1 -- 172.21.15.98:0/2229842682 shutdown_connections 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.581+0000 7f45897f3640 1 -- 172.21.15.98:0/2229842682 wait complete. 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.582+0000 7f45897f3640 10 monclient: build_initial_monmap 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.582+0000 7f45897f3640 10 monclient: monmap: 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.582+0000 7f45897f3640 1 librados: starting msgr at 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.582+0000 7f45897f3640 1 librados: starting objecter 2024-06-11T19:20:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.582+0000 7f45897f3640 1 Processor -- start 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.582+0000 7f45897f3640 1 -- start start 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.582+0000 7f45897f3640 1 librados: setting wanted keys 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.582+0000 7f45897f3640 1 librados: calling monclient init 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.582+0000 7f45897f3640 10 monclient: init 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient: _add_conns ranks=[7,4,6,5,2,3,0,1,8] 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45840a2f40 0x7f45841cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient(hunting): picked mon.noname-i con 0x7f45840a2f40 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 0x7f45841cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient(hunting): picked mon.noname-c con 0x7f45840a4df0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f45841366d0 0x7f4584120da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient(hunting): picked mon.noname-d con 0x7f45841366d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f45841371f0 con 0x7f45840a4df0 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f45840a2460 con 0x7f45841366d0 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f4584137370 con 0x7f45840a2f40 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient(hunting): authenticate will time out at 1411.390154s 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 0x7f45841cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 0x7f45841cc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:41174/0 (socket says 172.21.15.98:41174) 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45837fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f45841366d0 0x7f4584120da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 1 -- 172.21.15.98:0/1954276809 learned_addr learned my addr 172.21.15.98:0/1954276809 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 10 monclient(hunting): get_auth_request con 0x7f45840a4df0 auth_method 0 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45837fe640 10 monclient(hunting): get_auth_request con 0x7f45841366d0 auth_method 0 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45837fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45837fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f4582ffd640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45840a2f40 0x7f45841cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f4582ffd640 10 monclient(hunting): get_auth_request con 0x7f45840a2f40 auth_method 0 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f4582ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f4582ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f4582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 10 monclient(hunting): handle_auth_done global_id 24205 payload 824 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 1 -- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45840a2f40 msgr2=0x7f45841cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f45840a2f40 0x7f45841cbaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 1 -- 172.21.15.98:0/1954276809 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f45841366d0 msgr2=0x7f4584120da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f45841366d0 0x7f4584120da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 10 monclient: _finish_hunting 0 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 1 monclient: found mon.noname-c 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:20:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 1 -- 172.21.15.98:0/1954276809 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4584123530 con 0x7f45840a4df0 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45837fe640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f45841366d0 0x7f4584120da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45827fc640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 0x7f45841cc030 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f4578000c00 tx=0x7f45780046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 10 client.?.objecter ms_handle_connect 0x7f45840a4df0 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4578042020 con 0x7f45840a4df0 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 10 monclient: dump: 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:20:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 5 monclient: authenticate success, global_id 24205 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45780043f0 con 0x7f45840a4df0 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4578038040 con 0x7f45840a4df0 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f456b7fe640 10 monclient: dump: 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient: _renew_subs 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45817fa640 4 set_mon_vals no callback set 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 1 -- 172.21.15.98:0/1954276809 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45840a8170 con 0x7f45840a4df0 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 client.24205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient: _renew_subs 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 1 -- 172.21.15.98:0/1954276809 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f4584124700 con 0x7f45840a4df0 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.583+0000 7f45897f3640 1 librados: init done 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.584+0000 7f45897f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.584+0000 7f45897f3640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4584124940 0x7f4584121aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.584+0000 7f45897f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.584+0000 7f45897f3640 1 -- 172.21.15.98:0/1954276809 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f4584050830 con 0x7f4584124940 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.584+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f4578045020 con 0x7f45840a4df0 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.584+0000 7f456b7fe640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f457003fdb0 0x7f4570042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f4582ffd640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4584124940 0x7f4584121aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f4582ffd640 10 monclient: get_auth_request con 0x7f4584124940 auth_method 0 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f4582ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f4582ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f4582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7f4578030070 con 0x7f45840a4df0 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f456b7fe640 10 client.24205.objecter ms_dispatch 0x7f4584000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f456b7fe640 3 client.24205.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f456b7fe640 3 client.24205.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f456b7fe640 20 client.24205.objecter dump_active .. 0 homeless 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f4582ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f4582ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f4582ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f45837fe640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f457003fdb0 0x7f4570042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f45837fe640 10 monclient: get_auth_request con 0x7f457003fdb0 auth_method 0 2024-06-11T19:20:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f4582ffd640 10 monclient(hunting): handle_auth_done global_id 84445 payload 824 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f4582ffd640 1 -- 172.21.15.98:0/1954276809 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45840a48b0 con 0x7f4584124940 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f456b7fe640 10 client.24205.objecter ms_handle_connect 0x7f4584124940 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f456b7fe640 10 client.24205.objecter resend_mon_ops 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f4582ffd640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4584124940 0x7f4584121aa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f457400f490 tx=0x7f457400f950 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f456b7fe640 10 client.24205.objecter ms_handle_connect 0x7f457003fdb0 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f45837fe640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f457003fdb0 0x7f4570042270 secure :-1 s=READY pgs=4204 cs=0 l=1 rev1=1 crypto rx=0x7f457c00b7a0 tx=0x7f457c00d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4574018070 con 0x7f4584124940 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f456b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f456b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.585+0000 7f456b7fe640 10 monclient: dump: 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.586+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4574014690 con 0x7f4584124940 2024-06-11T19:20:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.586+0000 7f456b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.586+0000 7f456b7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.586+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4584124940 msgr2=0x7f4584121aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.586+0000 7f456b7fe640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4584124940 0x7f4584121aa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f457400f490 tx=0x7f457400f950 comp rx=0 tx=0).stop 2024-06-11T19:20:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.586+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4584124940 msgr2=0x7f4584121aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.586+0000 7f456b7fe640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4584124940 0x7f4584121aa0 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f457400f490 tx=0x7f457400f950 comp rx=0 tx=0).stop 2024-06-11T19:20:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.592+0000 7f45897f3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.592+0000 7f45897f3640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f458402cb40 0x7f458402cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.592+0000 7f45897f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.592+0000 7f45897f3640 1 -- 172.21.15.98:0/1954276809 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f4584071cf0 con 0x7f458402cb40 2024-06-11T19:20:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.592+0000 7f4582ffd640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f458402cb40 0x7f458402cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.592+0000 7f4582ffd640 10 monclient: get_auth_request con 0x7f458402cb40 auth_method 0 2024-06-11T19:20:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.592+0000 7f4582ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.592+0000 7f4582ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.592+0000 7f4582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.592+0000 7f4582ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.592+0000 7f4582ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.592+0000 7f4582ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f4582ffd640 10 monclient(hunting): handle_auth_done global_id 84454 payload 824 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f4582ffd640 1 -- 172.21.15.98:0/1954276809 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4584123280 con 0x7f458402cb40 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 10 client.24205.objecter ms_handle_connect 0x7f458402cb40 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 10 client.24205.objecter resend_mon_ops 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f4582ffd640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f458402cb40 0x7f458402cf50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f457400f490 tx=0x7f45740046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4574003c90 con 0x7f458402cb40 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 10 monclient: dump: 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f4574003820 con 0x7f458402cb40 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f458402cb40 msgr2=0x7f458402cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f458402cb40 0x7f458402cf50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f457400f490 tx=0x7f45740046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 1 -- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f458402cb40 msgr2=0x7f458402cf50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f456b7fe640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f458402cb40 0x7f458402cf50 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f457400f490 tx=0x7f45740046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.615 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 273363, 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:01.638 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.593+0000 7f45897f3640 10 librados: watch_flush enter 2024-06-11T19:20:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 10 librados: watch_flush exit 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 20 client.24205.objecter shutdown clearing up homeless session... 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 10 client.24205.objecter successfully canceled tick 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 1 -- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f457003fdb0 msgr2=0x7f4570042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f457003fdb0 0x7f4570042270 secure :-1 s=READY pgs=4204 cs=0 l=1 rev1=1 crypto rx=0x7f457c00b7a0 tx=0x7f457c00d040 comp rx=0 tx=0).stop 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 10 monclient: shutdown 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 1 -- 172.21.15.98:0/1954276809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 msgr2=0x7f45841cc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 1 --2- 172.21.15.98:0/1954276809 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f45840a4df0 0x7f45841cc030 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f4578000c00 tx=0x7f45780046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45837fe640 1 -- 172.21.15.98:0/1954276809 reap_dead start 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45837fe640 1 -- 172.21.15.98:0/1954276809 reap_dead start 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 1 -- 172.21.15.98:0/1954276809 shutdown_connections 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 1 -- 172.21.15.98:0/1954276809 >> 172.21.15.98:0/1954276809 conn(0x7f45840a95b0 msgr2=0x7f45841bd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 1 -- 172.21.15.98:0/1954276809 shutdown_connections 2024-06-11T19:20:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 1 -- 172.21.15.98:0/1954276809 wait complete. 2024-06-11T19:20:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.594+0000 7f45897f3640 1 librados: shutdown 2024-06-11T19:20:01.641 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:01.641 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-06-11T19:20:01.641 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-06-11T19:20:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.740+0000 7f61506ca640 10 monclient: get_monmap_and_config 2024-06-11T19:20:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.740+0000 7f61506ca640 10 monclient: build_initial_monmap 2024-06-11T19:20:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.741+0000 7f61506ca640 10 monclient: monmap: 2024-06-11T19:20:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:01.742004+0000 2024-06-11T19:20:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:01.742004+0000 2024-06-11T19:20:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:01.741 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.741+0000 7f61506ca640 1 Processor -- start 2024-06-11T19:20:01.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.741+0000 7f61506ca640 1 -- start start 2024-06-11T19:20:01.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.741+0000 7f61506ca640 10 monclient: init 2024-06-11T19:20:01.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:01.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 10 monclient: _add_conns ranks=[10,7,5,1,4,3,2,8,0,9,6] 2024-06-11T19:20:01.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f61480a52a0 0x7f61480a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 10 monclient(hunting): picked mon.noname-f con 0x7f61480a52a0 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f614809d520 0x7f61480a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 10 monclient(hunting): picked mon.noname-i con 0x7f614809d520 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6148073830 0x7f61480a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 10 monclient(hunting): picked mon.noname-h con 0x7f6148073830 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f614809daa0 con 0x7f61480a52a0 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f614809d2e0 con 0x7f6148073830 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f61480a5680 con 0x7f614809d520 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f61506ca640 10 monclient(hunting): authenticate will time out at 1411.549468s 2024-06-11T19:20:01.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614dc65640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f614809d520 0x7f61480a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614ec67640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6148073830 0x7f61480a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614e466640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f61480a52a0 0x7f61480a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614ec67640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6148073830 0x7f61480a3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:37066/0 (socket says 172.21.15.98:37066) 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614dc65640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f614809d520 0x7f61480a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:45820/0 (socket says 172.21.15.98:45820) 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614ec67640 1 -- 172.21.15.98:0/2664236777 learned_addr learned my addr 172.21.15.98:0/2664236777 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614e466640 10 monclient(hunting): get_auth_request con 0x7f61480a52a0 auth_method 0 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614e466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614e466640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614e466640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614ec67640 10 monclient(hunting): get_auth_request con 0x7f6148073830 auth_method 0 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614ec67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614ec67640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614ec67640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614dc65640 10 monclient(hunting): get_auth_request con 0x7f614809d520 auth_method 0 2024-06-11T19:20:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614dc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614dc65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614dc65640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614e466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614e466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614e466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614dc65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614dc65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614dc65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614ec67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614ec67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.742+0000 7f614ec67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.743+0000 7f614dc65640 10 monclient(hunting): handle_auth_done global_id 84463 payload 274 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.743+0000 7f614dc65640 1 -- 172.21.15.98:0/2664236777 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6148073830 msgr2=0x7f61480a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.743+0000 7f614dc65640 1 --2- 172.21.15.98:0/2664236777 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6148073830 0x7f61480a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.743+0000 7f614dc65640 1 -- 172.21.15.98:0/2664236777 >> v2:172.21.15.117:3300/0 conn(0x7f61480a52a0 msgr2=0x7f61480a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.743+0000 7f614dc65640 1 --2- 172.21.15.98:0/2664236777 >> v2:172.21.15.117:3300/0 conn(0x7f61480a52a0 0x7f61480a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.743+0000 7f614dc65640 10 monclient: _finish_hunting 0 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.743+0000 7f614dc65640 1 monclient: found mon.noname-i 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.743+0000 7f614dc65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.743+0000 7f614dc65640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.743+0000 7f614dc65640 1 -- 172.21.15.98:0/2664236777 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614809e9d0 con 0x7f614809d520 2024-06-11T19:20:01.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.743+0000 7f614e466640 1 --2- 172.21.15.98:0/2664236777 >> v2:172.21.15.117:3300/0 conn(0x7f61480a52a0 0x7f61480a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.743+0000 7f614dc65640 1 --2- 172.21.15.98:0/2664236777 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f614809d520 0x7f61480a3690 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f61380097d0 tx=0x7f613802b6e0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=ace3479a475a45b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 1 -- 172.21.15.98:0/2664236777 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6138037070 con 0x7f614809d520 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 10 monclient: dump: 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 10 monclient: _finish_auth 0 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 10 monclient: _check_auth_tickets 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 1 -- 172.21.15.98:0/2664236777 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6138030980 con 0x7f614809d520 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 5 monclient: authenticate success, global_id 84463 2024-06-11T19:20:01.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 1 -- 172.21.15.98:0/2664236777 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6138030bb0 con 0x7f614809d520 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 4 set_mon_vals no callback set 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f614d464640 10 monclient: dump: 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.757 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 10 monclient: shutdown 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 1 -- 172.21.15.98:0/2664236777 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f614809d520 msgr2=0x7f61480a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 1 --2- 172.21.15.98:0/2664236777 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f614809d520 0x7f61480a3690 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f61380097d0 tx=0x7f613802b6e0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 1 -- 172.21.15.98:0/2664236777 shutdown_connections 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 1 --2- 172.21.15.98:0/2664236777 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f614809d520 0x7f61480a3690 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 1 --2- 172.21.15.98:0/2664236777 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f6148073830 0x7f61480a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 1 --2- 172.21.15.98:0/2664236777 >> v2:172.21.15.117:3300/0 conn(0x7f61480a52a0 0x7f61480a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 1 -- 172.21.15.98:0/2664236777 >> 172.21.15.98:0/2664236777 conn(0x7f61480a4780 msgr2=0x7f61480a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 1 -- 172.21.15.98:0/2664236777 shutdown_connections 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.744+0000 7f61506ca640 1 -- 172.21.15.98:0/2664236777 wait complete. 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.745+0000 7f61506ca640 10 monclient: build_initial_monmap 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.745+0000 7f61506ca640 10 monclient: monmap: 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.745+0000 7f61506ca640 1 librados: starting msgr at 2024-06-11T19:20:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.745+0000 7f61506ca640 1 librados: starting objecter 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.745+0000 7f61506ca640 1 Processor -- start 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.745+0000 7f61506ca640 1 -- start start 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.745+0000 7f61506ca640 1 librados: setting wanted keys 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.745+0000 7f61506ca640 1 librados: calling monclient init 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.745+0000 7f61506ca640 10 monclient: init 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 10 monclient: _add_conns ranks=[2,8,3,0,4,6,5,1,7] 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6148073830 0x7f61481cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 10 monclient(hunting): picked mon.noname-b con 0x7f6148073830 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f614809d520 0x7f61481349a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 10 monclient(hunting): picked mon.noname-e con 0x7f614809d520 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f61480a52a0 0x7f614806f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 10 monclient(hunting): picked mon.noname-g con 0x7f61480a52a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f61480a1fd0 con 0x7f6148073830 2024-06-11T19:20:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f61480a1e50 con 0x7f61480a52a0 2024-06-11T19:20:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f61480a2150 con 0x7f614809d520 2024-06-11T19:20:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 10 monclient(hunting): authenticate will time out at 1411.553198s 2024-06-11T19:20:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614e466640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6148073830 0x7f61481cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f614809d520 0x7f61481349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614e466640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6148073830 0x7f61481cdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:46232/0 (socket says 172.21.15.98:46232) 2024-06-11T19:20:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f614809d520 0x7f61481349a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44056/0 (socket says 172.21.15.98:44056) 2024-06-11T19:20:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614e466640 1 -- 172.21.15.98:0/1525717082 learned_addr learned my addr 172.21.15.98:0/1525717082 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:01.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614e466640 10 monclient(hunting): get_auth_request con 0x7f6148073830 auth_method 0 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614e466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614e466640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614e466640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 10 monclient(hunting): get_auth_request con 0x7f614809d520 auth_method 0 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614ec67640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f61480a52a0 0x7f614806f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614ec67640 10 monclient(hunting): get_auth_request con 0x7f61480a52a0 auth_method 0 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614ec67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:01.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614ec67640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614ec67640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614e466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614e466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614e466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614ec67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614ec67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614ec67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 10 monclient(hunting): handle_auth_done global_id 94319 payload 824 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 1 -- 172.21.15.98:0/1525717082 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f61480a52a0 msgr2=0x7f614806f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f61480a52a0 0x7f614806f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 1 -- 172.21.15.98:0/1525717082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6148073830 msgr2=0x7f61481cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6148073830 0x7f61481cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 10 monclient: _finish_hunting 0 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 1 monclient: found mon.noname-e 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 1 -- 172.21.15.98:0/1525717082 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61480a1360 con 0x7f614809d520 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614e466640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6148073830 0x7f61481cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f614dc65640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f614809d520 0x7f61481349a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6138032000 tx=0x7f61380046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f613effd640 10 client.?.objecter ms_handle_connect 0x7f614809d520 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f613effd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f613effd640 1 -- 172.21.15.98:0/1525717082 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6138040020 con 0x7f614809d520 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f613effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f613effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f613effd640 10 monclient: dump: 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f613effd640 10 monclient: _finish_auth 0 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f613effd640 10 monclient: _check_auth_tickets 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f613effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f613effd640 1 -- 172.21.15.98:0/1525717082 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6138038040 con 0x7f614809d520 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.746+0000 7f61506ca640 5 monclient: authenticate success, global_id 94319 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f61506ca640 10 monclient: _renew_subs 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f61506ca640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f61506ca640 1 -- 172.21.15.98:0/1525717082 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61480717b0 con 0x7f614809d520 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f613effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f613effd640 1 -- 172.21.15.98:0/1525717082 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6138037070 con 0x7f614809d520 2024-06-11T19:20:01.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f613effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f613effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f613effd640 10 monclient: dump: 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f614cc63640 4 set_mon_vals no callback set 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f61506ca640 10 client.94319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f61506ca640 10 monclient: _renew_subs 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f61506ca640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f61506ca640 1 -- 172.21.15.98:0/1525717082 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f61481ce140 con 0x7f614809d520 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f61506ca640 1 librados: init done 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f613effd640 1 -- 172.21.15.98:0/1525717082 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f6138043020 con 0x7f614809d520 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.747+0000 7f61506ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f61506ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f61506ca640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f61506ca640 1 -- 172.21.15.98:0/1525717082 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61480a1590 con 0x7f614809d520 2024-06-11T19:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f613effd640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f612c03fdb0 0x7f612c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f613effd640 1 -- 172.21.15.98:0/1525717082 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7f6138072ea0 con 0x7f614809d520 2024-06-11T19:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f613effd640 10 client.94319.objecter ms_dispatch 0x7f6148000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f613effd640 3 client.94319.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f613effd640 3 client.94319.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f613effd640 20 client.94319.objecter dump_active .. 0 homeless 2024-06-11T19:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f614e466640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f612c03fdb0 0x7f612c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f614e466640 10 monclient: get_auth_request con 0x7f612c03fdb0 auth_method 0 2024-06-11T19:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f613effd640 10 client.94319.objecter ms_handle_connect 0x7f612c03fdb0 2024-06-11T19:20:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.748+0000 7f614e466640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f612c03fdb0 0x7f612c042270 secure :-1 s=READY pgs=4205 cs=0 l=1 rev1=1 crypto rx=0x7f61440095e0 tx=0x7f6144009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.752+0000 7f613effd640 1 -- 172.21.15.98:0/1525717082 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f61480a1590 con 0x7f614809d520 2024-06-11T19:20:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.752+0000 7f613effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:01.752+0000 7f613effd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.010+0000 7f61506ca640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-06-11T19:20:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.010+0000 7f61506ca640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-06-11T19:20:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.010+0000 7f61506ca640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.010+0000 7f61506ca640 1 -- 172.21.15.98:0/1525717082 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f6148135310 con 0x7f614809d520 2024-06-11T19:20:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.012+0000 7f613effd640 1 -- 172.21.15.98:0/1525717082 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f613804a030 con 0x7f614809d520 2024-06-11T19:20:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.012+0000 7f613effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-06-11T19:20:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.012+0000 7f613effd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:02.013 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.013 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=3904342069,config=692619927,health=1515119969,kv=2000936593,logm=3826404806}) 2024-06-11T19:20:02.013 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=1,health=12,kv=9,logm=61} crc {auth=867134848,config=3678701417,health=700719562,kv=1820642019,logm=4007805060}) 2024-06-11T19:20:02.013 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.013 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=3904342069,config=692619927,health=1515119969,kv=2000936593,logm=3826404806}) 2024-06-11T19:20:02.013 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=12,kv=9,logm=59} crc {auth=3904342069,config=692619927,health=700719562,kv=1322560318,logm=1408335802}) 2024-06-11T19:20:02.013 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.013 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=3904342069,config=692619927,health=1515119969,kv=2000936593,logm=3826404806}) 2024-06-11T19:20:02.013 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=3904342069,config=692619927,health=700719562,kv=2000936593,logm=1182346359}) 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=3904342069,config=692619927,health=1515119969,kv=2000936593,logm=3826404806}) 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=18,config=2,health=12,kv=9,logm=59} crc {auth=3904342069,config=692619927,health=700719562,kv=14585854,logm=3139742734}) 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=3904342069,config=692619927,health=1515119969,kv=2000936593,logm=3826404806}) 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=18,config=1,health=11,kv=9,logm=61} crc {auth=3904342069,config=3678701417,health=258690236,kv=3371622858,logm=3086873796}) 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=3904342069,config=692619927,health=1515119969,kv=2000936593,logm=3826404806}) 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=17,config=2,health=11,kv=8,logm=62} crc {auth=3252922620,config=692619927,health=2181456127,kv=2507104616,logm=807475531}) 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=3904342069,config=692619927,health=1515119969,kv=2000936593,logm=3826404806}) 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=17,config=2,health=10,kv=10,logm=61} crc {auth=978722142,config=692619927,health=3617194422,kv=2000936593,logm=1780272882}) 2024-06-11T19:20:02.014 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.015 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=3904342069,config=692619927,health=1515119969,kv=2000936593,logm=3826404806}) 2024-06-11T19:20:02.015 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.013+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=17,config=2,health=12,kv=9,logm=60} crc {auth=3530908496,config=692619927,health=700719562,kv=3584025909,logm=1613469685}) 2024-06-11T19:20:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.014+0000 7f61506ca640 10 librados: watch_flush enter 2024-06-11T19:20:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.014+0000 7f61506ca640 10 librados: watch_flush exit 2024-06-11T19:20:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.014+0000 7f61506ca640 20 client.94319.objecter shutdown clearing up homeless session... 2024-06-11T19:20:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.014+0000 7f61506ca640 10 client.94319.objecter successfully canceled tick 2024-06-11T19:20:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 -- 172.21.15.98:0/1525717082 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f612c03fdb0 msgr2=0x7f612c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f612c03fdb0 0x7f612c042270 secure :-1 s=READY pgs=4205 cs=0 l=1 rev1=1 crypto rx=0x7f61440095e0 tx=0x7f6144009190 comp rx=0 tx=0).stop 2024-06-11T19:20:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 10 monclient: shutdown 2024-06-11T19:20:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 -- 172.21.15.98:0/1525717082 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f614809d520 msgr2=0x7f61481349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f614809d520 0x7f61481349a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6138032000 tx=0x7f61380046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 -- 172.21.15.98:0/1525717082 shutdown_connections 2024-06-11T19:20:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f612c03fdb0 0x7f612c042270 unknown :-1 s=CLOSED pgs=4205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f61480a52a0 0x7f614806f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f614809d520 0x7f61481349a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 --2- 172.21.15.98:0/1525717082 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6148073830 0x7f61481cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 -- 172.21.15.98:0/1525717082 >> 172.21.15.98:0/1525717082 conn(0x7f61480a4780 msgr2=0x7f61480a2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 -- 172.21.15.98:0/1525717082 shutdown_connections 2024-06-11T19:20:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 -- 172.21.15.98:0/1525717082 wait complete. 2024-06-11T19:20:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f61506ca640 1 librados: shutdown 2024-06-11T19:20:02.031 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.031 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=542015037}) 2024-06-11T19:20:02.031 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2831427638}) 2024-06-11T19:20:02.031 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.031 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=542015037}) 2024-06-11T19:20:02.031 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2267377657}) 2024-06-11T19:20:02.031 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.031 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=542015037}) 2024-06-11T19:20:02.031 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3941440076}) 2024-06-11T19:20:02.032 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.032 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=542015037}) 2024-06-11T19:20:02.032 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3986681304}) 2024-06-11T19:20:02.032 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.032 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=542015037}) 2024-06-11T19:20:02.032 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.015+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1127830288}) 2024-06-11T19:20:02.032 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.016+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.032 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.016+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=542015037}) 2024-06-11T19:20:02.032 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.016+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3708883803}) 2024-06-11T19:20:02.032 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.016+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.032 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.016+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=542015037}) 2024-06-11T19:20:02.032 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.016+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3561062721}) 2024-06-11T19:20:02.033 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.016+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.033 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.016+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=542015037}) 2024-06-11T19:20:02.033 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.016+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2767362475}) 2024-06-11T19:20:02.033 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.033 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135529140}) 2024-06-11T19:20:02.033 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3695594958}) 2024-06-11T19:20:02.033 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.033 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135529140}) 2024-06-11T19:20:02.033 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3770024527}) 2024-06-11T19:20:02.033 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.033 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135529140}) 2024-06-11T19:20:02.033 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=429633921}) 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135529140}) 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3178226319}) 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135529140}) 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4095768413}) 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135529140}) 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3270200446}) 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135529140}) 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=254574299}) 2024-06-11T19:20:02.034 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.035 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135529140}) 2024-06-11T19:20:02.035 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.018+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3298762353}) 2024-06-11T19:20:02.035 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.035 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3944887851}) 2024-06-11T19:20:02.035 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350637393}) 2024-06-11T19:20:02.035 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.035 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3944887851}) 2024-06-11T19:20:02.035 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4286697134}) 2024-06-11T19:20:02.035 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.035 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3944887851}) 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2841319738}) 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3944887851}) 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1807644996}) 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3944887851}) 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2149162963}) 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3944887851}) 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=505061851}) 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3944887851}) 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3836045758}) 2024-06-11T19:20:02.036 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3944887851}) 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.020+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1359106372}) 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1054023949}) 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254887183}) 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1054023949}) 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3654633951}) 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1054023949}) 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=454768776}) 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1054023949}) 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2102080909}) 2024-06-11T19:20:02.037 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.038 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1054023949}) 2024-06-11T19:20:02.038 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1819692565}) 2024-06-11T19:20:02.038 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.038 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1054023949}) 2024-06-11T19:20:02.038 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1190089736}) 2024-06-11T19:20:02.038 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.038 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1054023949}) 2024-06-11T19:20:02.038 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3135111274}) 2024-06-11T19:20:02.038 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.038 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1054023949}) 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.022+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3445332267}) 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.023+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.023+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2102625993}) 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.023+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581368724}) 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.023+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.023+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2102625993}) 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.023+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4038957484}) 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.023+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.023+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2102625993}) 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.023+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1109244216}) 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.023+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.023+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2102625993}) 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.023+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1998952084}) 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.024+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.039 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.024+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2102625993}) 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.024+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3156723576}) 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.024+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.024+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2102625993}) 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.024+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=985315086}) 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.024+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.024+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2102625993}) 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.024+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3279058415}) 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.024+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.024+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2102625993}) 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.024+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=418744701}) 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1468423681}) 2024-06-11T19:20:02.040 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049178643}) 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1468423681}) 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3849940412}) 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1468423681}) 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2585668731}) 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1468423681}) 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3219152668}) 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1468423681}) 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=779566173}) 2024-06-11T19:20:02.041 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1468423681}) 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1605126700}) 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1468423681}) 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=840894468}) 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1468423681}) 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.025+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4009233267}) 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2172788742}) 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900887190}) 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.042 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2172788742}) 2024-06-11T19:20:02.043 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=882498167}) 2024-06-11T19:20:02.043 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.043 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2172788742}) 2024-06-11T19:20:02.043 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=86496911}) 2024-06-11T19:20:02.043 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.043 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2172788742}) 2024-06-11T19:20:02.043 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2868404981}) 2024-06-11T19:20:02.043 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.043 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2172788742}) 2024-06-11T19:20:02.043 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2470539075}) 2024-06-11T19:20:02.043 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.043 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2172788742}) 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2903001396}) 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2172788742}) 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=767844722}) 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2172788742}) 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.026+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3394026471}) 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2905953482}) 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1616085589}) 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2905953482}) 2024-06-11T19:20:02.044 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1660117521}) 2024-06-11T19:20:02.045 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.045 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2905953482}) 2024-06-11T19:20:02.045 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3257962720}) 2024-06-11T19:20:02.045 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.045 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2905953482}) 2024-06-11T19:20:02.045 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4161428045}) 2024-06-11T19:20:02.045 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.045 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2905953482}) 2024-06-11T19:20:02.045 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2275968610}) 2024-06-11T19:20:02.045 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.045 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2905953482}) 2024-06-11T19:20:02.045 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=527621441}) 2024-06-11T19:20:02.046 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.046 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2905953482}) 2024-06-11T19:20:02.046 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867115995}) 2024-06-11T19:20:02.046 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.046 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2905953482}) 2024-06-11T19:20:02.046 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.028+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3055133827}) 2024-06-11T19:20:02.047 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-06-11T19:20:02.213 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.213 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2911800754}) 2024-06-11T19:20:02.213 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410286218}) 2024-06-11T19:20:02.213 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.213 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2911800754}) 2024-06-11T19:20:02.214 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3417579512}) 2024-06-11T19:20:02.214 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.214 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2911800754}) 2024-06-11T19:20:02.214 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1524643307}) 2024-06-11T19:20:02.214 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.214 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2911800754}) 2024-06-11T19:20:02.214 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=891488453}) 2024-06-11T19:20:02.214 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.214 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2911800754}) 2024-06-11T19:20:02.214 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1329960319}) 2024-06-11T19:20:02.214 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.215 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2911800754}) 2024-06-11T19:20:02.215 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2038229549}) 2024-06-11T19:20:02.215 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.215 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.029+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2911800754}) 2024-06-11T19:20:02.215 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.030+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2919425660}) 2024-06-11T19:20:02.215 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.030+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.215 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.030+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2911800754}) 2024-06-11T19:20:02.215 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.030+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=90690395}) 2024-06-11T19:20:02.215 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.215 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=27238822}) 2024-06-11T19:20:02.215 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285811558}) 2024-06-11T19:20:02.215 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=27238822}) 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2542383682}) 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=27238822}) 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2195062101}) 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=27238822}) 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2925834035}) 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=27238822}) 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2917217229}) 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=27238822}) 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2330877625}) 2024-06-11T19:20:02.216 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=27238822}) 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3766640191}) 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=27238822}) 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.031+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2436374387}) 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=293505812}) 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3173996374}) 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=293505812}) 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1159333788}) 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=293505812}) 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159697100}) 2024-06-11T19:20:02.217 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=293505812}) 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2981899652}) 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=293505812}) 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2943314549}) 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=293505812}) 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1299704980}) 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=293505812}) 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1492990587}) 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.218 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=293505812}) 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.033+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2406889219}) 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.034+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.034+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3974089524}) 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.034+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013136435}) 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.034+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.034+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3974089524}) 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.034+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=15101966}) 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3974089524}) 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954728588}) 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3974089524}) 2024-06-11T19:20:02.219 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1382249696}) 2024-06-11T19:20:02.220 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.220 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3974089524}) 2024-06-11T19:20:02.220 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=563086020}) 2024-06-11T19:20:02.220 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.220 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3974089524}) 2024-06-11T19:20:02.220 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1521290361}) 2024-06-11T19:20:02.220 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.220 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3974089524}) 2024-06-11T19:20:02.220 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=544894997}) 2024-06-11T19:20:02.220 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.220 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3974089524}) 2024-06-11T19:20:02.220 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.035+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1690227011}) 2024-06-11T19:20:02.221 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.221 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3126144740}) 2024-06-11T19:20:02.221 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716722772}) 2024-06-11T19:20:02.221 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.221 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3126144740}) 2024-06-11T19:20:02.221 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1522772885}) 2024-06-11T19:20:02.221 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.221 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3126144740}) 2024-06-11T19:20:02.221 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1140711881}) 2024-06-11T19:20:02.221 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.221 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3126144740}) 2024-06-11T19:20:02.221 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3022472433}) 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3126144740}) 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1659925760}) 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3126144740}) 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1810575198}) 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3126144740}) 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2681932962}) 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3126144740}) 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.036+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4204216444}) 2024-06-11T19:20:02.222 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=357065140}) 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729354215}) 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=357065140}) 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3899013070}) 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=357065140}) 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2749461326}) 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=357065140}) 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3494653559}) 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=357065140}) 2024-06-11T19:20:02.223 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2708062051}) 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=357065140}) 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.037+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1376371002}) 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.038+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.038+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=357065140}) 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.038+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1645894503}) 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.038+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.038+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=357065140}) 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.038+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2821790753}) 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=806429242}) 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=185992115}) 2024-06-11T19:20:02.224 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=806429242}) 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1438969547}) 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=806429242}) 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3170999781}) 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=806429242}) 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1204066075}) 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=806429242}) 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2224876431}) 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=806429242}) 2024-06-11T19:20:02.225 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1170457120}) 2024-06-11T19:20:02.226 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.226 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=806429242}) 2024-06-11T19:20:02.226 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2424453447}) 2024-06-11T19:20:02.226 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.226 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=806429242}) 2024-06-11T19:20:02.226 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.039+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3021316232}) 2024-06-11T19:20:02.226 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.226 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=637322102}) 2024-06-11T19:20:02.226 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1865238382}) 2024-06-11T19:20:02.226 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.226 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=637322102}) 2024-06-11T19:20:02.226 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2639831553}) 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=637322102}) 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3669814797}) 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=637322102}) 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3742176736}) 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=637322102}) 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2564834760}) 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=637322102}) 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1526214130}) 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=637322102}) 2024-06-11T19:20:02.227 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1005859367}) 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=637322102}) 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.040+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3259677464}) 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=705995786}) 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385148520}) 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=705995786}) 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4163661426}) 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=705995786}) 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3047025780}) 2024-06-11T19:20:02.228 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=705995786}) 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1519235064}) 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=705995786}) 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.041+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=632120425}) 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.042+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.042+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=705995786}) 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.042+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1466395421}) 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.042+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.042+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=705995786}) 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.042+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=902048375}) 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.042+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.229 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.042+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=705995786}) 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.042+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2067694315}) 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1094700204}) 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244553446}) 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1094700204}) 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3047251810}) 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1094700204}) 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1235935549}) 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1094700204}) 2024-06-11T19:20:02.230 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=835810112}) 2024-06-11T19:20:02.231 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.231 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1094700204}) 2024-06-11T19:20:02.231 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3826593792}) 2024-06-11T19:20:02.231 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.231 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1094700204}) 2024-06-11T19:20:02.231 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3746217006}) 2024-06-11T19:20:02.231 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.231 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1094700204}) 2024-06-11T19:20:02.231 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1738697523}) 2024-06-11T19:20:02.231 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.231 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1094700204}) 2024-06-11T19:20:02.231 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.043+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2633624934}) 2024-06-11T19:20:02.232 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.232 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=471415276}) 2024-06-11T19:20:02.232 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=712346108}) 2024-06-11T19:20:02.232 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.232 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=471415276}) 2024-06-11T19:20:02.232 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=348692494}) 2024-06-11T19:20:02.232 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.232 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=471415276}) 2024-06-11T19:20:02.232 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=920009311}) 2024-06-11T19:20:02.232 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.232 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=471415276}) 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2678705297}) 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=471415276}) 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3416257669}) 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=471415276}) 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2169609215}) 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=471415276}) 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2199679532}) 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=471415276}) 2024-06-11T19:20:02.233 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.044+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1946566428}) 2024-06-11T19:20:02.234 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.045+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.234 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.045+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135764787}) 2024-06-11T19:20:02.234 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.045+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179271640}) 2024-06-11T19:20:02.234 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.045+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.234 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.045+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135764787}) 2024-06-11T19:20:02.234 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.045+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2929692384}) 2024-06-11T19:20:02.234 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.234 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135764787}) 2024-06-11T19:20:02.234 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458985643}) 2024-06-11T19:20:02.234 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.234 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135764787}) 2024-06-11T19:20:02.234 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=415089407}) 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135764787}) 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4284540104}) 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135764787}) 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=922778224}) 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135764787}) 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3619931530}) 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135764787}) 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.046+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2285785618}) 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.235 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3337972018}) 2024-06-11T19:20:02.236 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505392859}) 2024-06-11T19:20:02.236 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.236 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3337972018}) 2024-06-11T19:20:02.236 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1008746165}) 2024-06-11T19:20:02.236 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.236 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3337972018}) 2024-06-11T19:20:02.236 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620947605}) 2024-06-11T19:20:02.236 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.236 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3337972018}) 2024-06-11T19:20:02.236 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3617559803}) 2024-06-11T19:20:02.236 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3337972018}) 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1804366834}) 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3337972018}) 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=491172631}) 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.047+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.048+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3337972018}) 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.048+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3837322231}) 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.048+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.048+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3337972018}) 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.048+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3552548286}) 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=149381879}) 2024-06-11T19:20:02.237 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2231360104}) 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=149381879}) 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=699444953}) 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=149381879}) 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4243054557}) 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=149381879}) 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=379136766}) 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=149381879}) 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2745126468}) 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.238 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=149381879}) 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1917533696}) 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=149381879}) 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2026950622}) 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=149381879}) 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.049+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=85562115}) 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.050+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.050+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1953514185}) 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.050+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169570535}) 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.050+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.050+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1953514185}) 2024-06-11T19:20:02.239 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.050+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=39863155}) 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.050+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.050+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1953514185}) 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.050+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4151621275}) 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.050+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.050+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1953514185}) 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.050+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=650200363}) 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.051+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.051+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1953514185}) 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.051+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3667704846}) 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.051+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.051+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1953514185}) 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.051+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3077308122}) 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.051+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.240 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.051+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1953514185}) 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.051+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1249361986}) 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.051+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.051+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1953514185}) 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.051+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3287357449}) 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1268655987}) 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853013123}) 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1268655987}) 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2922992305}) 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1268655987}) 2024-06-11T19:20:02.241 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=147203502}) 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1268655987}) 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2361962095}) 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1268655987}) 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4170655400}) 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1268655987}) 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3985264485}) 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1268655987}) 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3334634099}) 2024-06-11T19:20:02.242 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1268655987}) 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.052+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=562505431}) 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=831803045}) 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252562915}) 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=831803045}) 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4159150226}) 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=831803045}) 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2743413160}) 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=831803045}) 2024-06-11T19:20:02.243 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1421983969}) 2024-06-11T19:20:02.244 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.244 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=831803045}) 2024-06-11T19:20:02.244 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1486836324}) 2024-06-11T19:20:02.244 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.244 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=831803045}) 2024-06-11T19:20:02.244 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2823277699}) 2024-06-11T19:20:02.244 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.244 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=831803045}) 2024-06-11T19:20:02.244 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2463307159}) 2024-06-11T19:20:02.244 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.244 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=831803045}) 2024-06-11T19:20:02.244 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.053+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3368643049}) 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.054+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.054+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2192978030}) 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.054+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3994050915}) 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.054+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.054+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2192978030}) 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.054+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2208696050}) 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2192978030}) 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714993031}) 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2192978030}) 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3940317757}) 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.245 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2192978030}) 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4266680038}) 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2192978030}) 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2170292659}) 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2192978030}) 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2261174198}) 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2192978030}) 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.055+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3056971259}) 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2201514251}) 2024-06-11T19:20:02.246 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2221376868}) 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2201514251}) 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2588896101}) 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2201514251}) 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4216138175}) 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2201514251}) 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2767659775}) 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2201514251}) 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1088389335}) 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.247 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2201514251}) 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3389999536}) 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2201514251}) 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3971467128}) 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2201514251}) 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.056+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4019652091}) 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2792992235}) 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728163866}) 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2792992235}) 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3935086857}) 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2792992235}) 2024-06-11T19:20:02.248 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2763046002}) 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2792992235}) 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1446277534}) 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2792992235}) 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3561554891}) 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2792992235}) 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4096876295}) 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2792992235}) 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=878188593}) 2024-06-11T19:20:02.249 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.250 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2792992235}) 2024-06-11T19:20:02.250 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.057+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4223512592}) 2024-06-11T19:20:02.250 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.058+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.250 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.058+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2555099343}) 2024-06-11T19:20:02.250 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.058+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303954772}) 2024-06-11T19:20:02.250 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.058+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.250 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.058+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2555099343}) 2024-06-11T19:20:02.250 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.058+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1535229419}) 2024-06-11T19:20:02.250 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.058+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.250 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.058+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2555099343}) 2024-06-11T19:20:02.250 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.058+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=769665760}) 2024-06-11T19:20:02.250 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2555099343}) 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2179634940}) 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2555099343}) 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=562074932}) 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2555099343}) 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1417027610}) 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2555099343}) 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=546077126}) 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.251 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2555099343}) 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.059+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=208081939}) 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3893925991}) 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406940331}) 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3893925991}) 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157050719}) 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3893925991}) 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=897350915}) 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3893925991}) 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3730422323}) 2024-06-11T19:20:02.252 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.253 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3893925991}) 2024-06-11T19:20:02.253 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1312519183}) 2024-06-11T19:20:02.253 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.253 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3893925991}) 2024-06-11T19:20:02.253 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1864982919}) 2024-06-11T19:20:02.253 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.253 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3893925991}) 2024-06-11T19:20:02.253 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2013977864}) 2024-06-11T19:20:02.253 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.253 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3893925991}) 2024-06-11T19:20:02.253 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.060+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3175017728}) 2024-06-11T19:20:02.254 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.254 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3308872802}) 2024-06-11T19:20:02.254 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1623514649}) 2024-06-11T19:20:02.254 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.254 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3308872802}) 2024-06-11T19:20:02.254 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3332256500}) 2024-06-11T19:20:02.254 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.254 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3308872802}) 2024-06-11T19:20:02.254 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1721615081}) 2024-06-11T19:20:02.254 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.254 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3308872802}) 2024-06-11T19:20:02.254 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2148006762}) 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3308872802}) 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2777720228}) 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3308872802}) 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1661959080}) 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3308872802}) 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3555811185}) 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3308872802}) 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.061+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1484942838}) 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.255 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3090285253}) 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1556484453}) 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3090285253}) 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3096537356}) 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3090285253}) 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2350520751}) 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3090285253}) 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=430329699}) 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.062+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3090285253}) 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.063+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1513779708}) 2024-06-11T19:20:02.256 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.063+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.063+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3090285253}) 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.063+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2495711933}) 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.063+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.063+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3090285253}) 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.063+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1608762087}) 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.063+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.063+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3090285253}) 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.063+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4021903177}) 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=775963964}) 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3149538505}) 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.257 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=775963964}) 2024-06-11T19:20:02.258 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1059660069}) 2024-06-11T19:20:02.258 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.258 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=775963964}) 2024-06-11T19:20:02.258 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3360391542}) 2024-06-11T19:20:02.258 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.258 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=775963964}) 2024-06-11T19:20:02.258 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=910754056}) 2024-06-11T19:20:02.258 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.258 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=775963964}) 2024-06-11T19:20:02.258 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3496453637}) 2024-06-11T19:20:02.258 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.258 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=775963964}) 2024-06-11T19:20:02.259 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=592133891}) 2024-06-11T19:20:02.259 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.259 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=775963964}) 2024-06-11T19:20:02.259 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1821067570}) 2024-06-11T19:20:02.259 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.259 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=775963964}) 2024-06-11T19:20:02.259 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.064+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2026475256}) 2024-06-11T19:20:02.259 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.259 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=889902864}) 2024-06-11T19:20:02.261 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3319473905}) 2024-06-11T19:20:02.261 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.261 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=889902864}) 2024-06-11T19:20:02.261 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2402906189}) 2024-06-11T19:20:02.261 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.261 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=889902864}) 2024-06-11T19:20:02.261 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3129589147}) 2024-06-11T19:20:02.261 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=889902864}) 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2427868443}) 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=889902864}) 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3190201743}) 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=889902864}) 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2794628328}) 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=889902864}) 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3628069360}) 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.262 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=889902864}) 2024-06-11T19:20:02.263 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.065+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3700998382}) 2024-06-11T19:20:02.263 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.263 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2159767183}) 2024-06-11T19:20:02.263 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3878441050}) 2024-06-11T19:20:02.263 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.263 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2159767183}) 2024-06-11T19:20:02.263 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4059083075}) 2024-06-11T19:20:02.263 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.263 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2159767183}) 2024-06-11T19:20:02.263 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=742643291}) 2024-06-11T19:20:02.263 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.263 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2159767183}) 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2212752487}) 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2159767183}) 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1519342866}) 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2159767183}) 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=84360968}) 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2159767183}) 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2354755104}) 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2159767183}) 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.067+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3007897662}) 2024-06-11T19:20:02.264 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1695554045}) 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252306859}) 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1695554045}) 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2400627300}) 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1695554045}) 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2037104250}) 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1695554045}) 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3058512297}) 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1695554045}) 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2247638743}) 2024-06-11T19:20:02.265 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1695554045}) 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1395423156}) 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1695554045}) 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3762674587}) 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1695554045}) 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.068+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1557493521}) 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3705716781}) 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3460876902}) 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.266 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3705716781}) 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105042727}) 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3705716781}) 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2499897001}) 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3705716781}) 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=93651729}) 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3705716781}) 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=284767894}) 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3705716781}) 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3766229984}) 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.267 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3705716781}) 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=37805204}) 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.069+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.070+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3705716781}) 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.070+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=841383690}) 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1239224763}) 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2465692370}) 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1239224763}) 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3563807232}) 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1239224763}) 2024-06-11T19:20:02.268 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3718429830}) 2024-06-11T19:20:02.269 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.269 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1239224763}) 2024-06-11T19:20:02.269 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=939989507}) 2024-06-11T19:20:02.269 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.269 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1239224763}) 2024-06-11T19:20:02.269 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1318924939}) 2024-06-11T19:20:02.269 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.269 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1239224763}) 2024-06-11T19:20:02.269 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4207928438}) 2024-06-11T19:20:02.269 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.269 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1239224763}) 2024-06-11T19:20:02.269 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3464079040}) 2024-06-11T19:20:02.270 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.270 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1239224763}) 2024-06-11T19:20:02.270 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.071+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3819183261}) 2024-06-11T19:20:02.270 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.270 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=673386989}) 2024-06-11T19:20:02.270 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2294457953}) 2024-06-11T19:20:02.270 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.270 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=673386989}) 2024-06-11T19:20:02.270 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2764193700}) 2024-06-11T19:20:02.270 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.270 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=673386989}) 2024-06-11T19:20:02.270 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2517497938}) 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=673386989}) 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3441641017}) 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=673386989}) 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1360748910}) 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=673386989}) 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4036897108}) 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=673386989}) 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1229658723}) 2024-06-11T19:20:02.271 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=673386989}) 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.072+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=192912135}) 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1604851037}) 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255189259}) 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1604851037}) 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926383232}) 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1604851037}) 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=289953258}) 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.272 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1604851037}) 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3451273701}) 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1604851037}) 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1838451724}) 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1604851037}) 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=522744947}) 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1604851037}) 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2848162717}) 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1604851037}) 2024-06-11T19:20:02.273 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.074+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1838541524}) 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3296677189}) 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3690336531}) 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3296677189}) 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1488639565}) 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3296677189}) 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3885503380}) 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3296677189}) 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=564572238}) 2024-06-11T19:20:02.274 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3296677189}) 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2023847302}) 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3296677189}) 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1796710898}) 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3296677189}) 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1594066710}) 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3296677189}) 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.075+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3869001179}) 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.076+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.076+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3062161709}) 2024-06-11T19:20:02.275 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.076+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3831011695}) 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.076+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.076+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3062161709}) 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.076+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=821322767}) 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.076+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.076+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3062161709}) 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=909683394}) 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3062161709}) 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=747090550}) 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3062161709}) 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2834295835}) 2024-06-11T19:20:02.276 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3062161709}) 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1541949781}) 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3062161709}) 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3509364904}) 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3062161709}) 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.077+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=363405651}) 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2521739299}) 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042257847}) 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2521739299}) 2024-06-11T19:20:02.277 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2839581756}) 2024-06-11T19:20:02.278 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.278 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2521739299}) 2024-06-11T19:20:02.278 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4115857019}) 2024-06-11T19:20:02.278 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.278 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2521739299}) 2024-06-11T19:20:02.278 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=482007219}) 2024-06-11T19:20:02.278 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.278 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2521739299}) 2024-06-11T19:20:02.278 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3010757118}) 2024-06-11T19:20:02.278 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.278 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2521739299}) 2024-06-11T19:20:02.279 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4126301218}) 2024-06-11T19:20:02.279 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.279 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2521739299}) 2024-06-11T19:20:02.279 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=100535382}) 2024-06-11T19:20:02.279 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.279 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2521739299}) 2024-06-11T19:20:02.279 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.078+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3564417996}) 2024-06-11T19:20:02.279 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.279 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2493189772}) 2024-06-11T19:20:02.279 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428255967}) 2024-06-11T19:20:02.279 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.279 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2493189772}) 2024-06-11T19:20:02.280 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3739465877}) 2024-06-11T19:20:02.280 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.280 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2493189772}) 2024-06-11T19:20:02.280 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2525189708}) 2024-06-11T19:20:02.280 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.280 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2493189772}) 2024-06-11T19:20:02.280 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3743976644}) 2024-06-11T19:20:02.280 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.280 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2493189772}) 2024-06-11T19:20:02.280 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=434679490}) 2024-06-11T19:20:02.280 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.280 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2493189772}) 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2454156713}) 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2493189772}) 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=523357688}) 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2493189772}) 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.080+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=414917665}) 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2501020489}) 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058306920}) 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2501020489}) 2024-06-11T19:20:02.281 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4126309931}) 2024-06-11T19:20:02.282 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.282 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2501020489}) 2024-06-11T19:20:02.282 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434158368}) 2024-06-11T19:20:02.282 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.282 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2501020489}) 2024-06-11T19:20:02.282 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2171990272}) 2024-06-11T19:20:02.282 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.282 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2501020489}) 2024-06-11T19:20:02.282 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1776708578}) 2024-06-11T19:20:02.282 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.282 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2501020489}) 2024-06-11T19:20:02.282 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2876588077}) 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.081+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.082+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2501020489}) 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.082+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3192728493}) 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.082+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.082+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2501020489}) 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.082+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2368434007}) 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4211414574}) 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64999848}) 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4211414574}) 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122806855}) 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.283 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4211414574}) 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=474354817}) 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4211414574}) 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1717444011}) 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4211414574}) 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1381307616}) 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4211414574}) 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3612746139}) 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4211414574}) 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3966412510}) 2024-06-11T19:20:02.284 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4211414574}) 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.083+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1971072137}) 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.084+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.084+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2317524773}) 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.084+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1635901659}) 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.084+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.084+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2317524773}) 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3693713704}) 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2317524773}) 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=631618985}) 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.285 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2317524773}) 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1449235065}) 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2317524773}) 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2953167012}) 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2317524773}) 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2494270571}) 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2317524773}) 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2732553071}) 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2317524773}) 2024-06-11T19:20:02.286 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.085+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2575577778}) 2024-06-11T19:20:02.287 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.287 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=284898636}) 2024-06-11T19:20:02.287 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727720493}) 2024-06-11T19:20:02.287 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.287 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=284898636}) 2024-06-11T19:20:02.287 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=909114261}) 2024-06-11T19:20:02.287 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.287 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=284898636}) 2024-06-11T19:20:02.287 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3412289070}) 2024-06-11T19:20:02.287 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.287 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=284898636}) 2024-06-11T19:20:02.287 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4123628336}) 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=284898636}) 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=927617530}) 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=284898636}) 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1361826566}) 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=284898636}) 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3531626616}) 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=284898636}) 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.086+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1009596485}) 2024-06-11T19:20:02.288 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=329962734}) 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306416964}) 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=329962734}) 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2088401255}) 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=329962734}) 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2588879598}) 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=329962734}) 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=543528787}) 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.289 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=329962734}) 2024-06-11T19:20:02.290 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3211959644}) 2024-06-11T19:20:02.290 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.290 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=329962734}) 2024-06-11T19:20:02.290 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1366591919}) 2024-06-11T19:20:02.290 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.290 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=329962734}) 2024-06-11T19:20:02.290 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3895896582}) 2024-06-11T19:20:02.290 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.290 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.087+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=329962734}) 2024-06-11T19:20:02.290 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.088+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1100794919}) 2024-06-11T19:20:02.290 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.290 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4224534204}) 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661073188}) 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4224534204}) 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1385103108}) 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4224534204}) 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2216575962}) 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4224534204}) 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3157326511}) 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4224534204}) 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=183284688}) 2024-06-11T19:20:02.291 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4224534204}) 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2588747198}) 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4224534204}) 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=519082746}) 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4224534204}) 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.089+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2733352423}) 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.090+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.090+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2976231394}) 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.090+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3064746466}) 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.090+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.292 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.090+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2976231394}) 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.090+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1269508228}) 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.090+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.090+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2976231394}) 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.090+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3105693846}) 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2976231394}) 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2849409122}) 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2976231394}) 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4122985112}) 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2976231394}) 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4001310698}) 2024-06-11T19:20:02.293 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.294 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2976231394}) 2024-06-11T19:20:02.294 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2967743197}) 2024-06-11T19:20:02.294 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.294 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2976231394}) 2024-06-11T19:20:02.294 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.091+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1843765757}) 2024-06-11T19:20:02.294 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.294 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=233099240}) 2024-06-11T19:20:02.294 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1617604257}) 2024-06-11T19:20:02.294 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.294 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=233099240}) 2024-06-11T19:20:02.294 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3912454575}) 2024-06-11T19:20:02.294 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=233099240}) 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4089692761}) 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=233099240}) 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=431984062}) 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=233099240}) 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1854750921}) 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=233099240}) 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1456779040}) 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=233099240}) 2024-06-11T19:20:02.295 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2256022234}) 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=233099240}) 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.092+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4005388993}) 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.093+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.093+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3953832260}) 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.093+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705459937}) 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.093+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.093+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3953832260}) 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.093+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=517666228}) 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.093+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.093+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3953832260}) 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.093+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=436064455}) 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.093+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.296 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3953832260}) 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3006495471}) 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3953832260}) 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=401262920}) 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3953832260}) 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1383221049}) 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3953832260}) 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4284962441}) 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3953832260}) 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.094+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=550141397}) 2024-06-11T19:20:02.297 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.298 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=532410472}) 2024-06-11T19:20:02.298 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=476038289}) 2024-06-11T19:20:02.298 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.298 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=532410472}) 2024-06-11T19:20:02.298 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924471950}) 2024-06-11T19:20:02.298 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.298 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=532410472}) 2024-06-11T19:20:02.298 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439906914}) 2024-06-11T19:20:02.298 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.298 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=532410472}) 2024-06-11T19:20:02.298 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4022923733}) 2024-06-11T19:20:02.298 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=532410472}) 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3926044140}) 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=532410472}) 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1282554721}) 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=532410472}) 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3746689309}) 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=532410472}) 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.095+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=17970301}) 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1032889097}) 2024-06-11T19:20:02.299 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3245759062}) 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1032889097}) 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1328203907}) 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1032889097}) 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3067037918}) 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1032889097}) 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2632148286}) 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.096+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1032889097}) 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.097+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2382481358}) 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.097+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.300 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.097+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1032889097}) 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.097+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3163570870}) 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.097+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.097+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1032889097}) 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.097+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2747697199}) 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.097+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.097+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1032889097}) 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.097+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1972712059}) 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2719204907}) 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4089751337}) 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2719204907}) 2024-06-11T19:20:02.301 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=234607626}) 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2719204907}) 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2996660038}) 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2719204907}) 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=337004921}) 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2719204907}) 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3705862358}) 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2719204907}) 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=422669236}) 2024-06-11T19:20:02.302 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2719204907}) 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=141841471}) 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2719204907}) 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.098+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2069494111}) 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.099+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.099+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=334560603}) 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.099+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131017554}) 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.099+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.099+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=334560603}) 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.099+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397015930}) 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.099+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.303 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.099+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=334560603}) 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2196937977}) 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=334560603}) 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4023171082}) 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=334560603}) 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2441164223}) 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=334560603}) 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3560217714}) 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=334560603}) 2024-06-11T19:20:02.304 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4291314449}) 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=334560603}) 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.100+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3555449182}) 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=222950945}) 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177606877}) 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=222950945}) 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1376563411}) 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=222950945}) 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2562554040}) 2024-06-11T19:20:02.305 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=222950945}) 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3346694362}) 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=222950945}) 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=171107469}) 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=222950945}) 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3895711003}) 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=222950945}) 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1702305192}) 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=222950945}) 2024-06-11T19:20:02.306 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.101+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3174202119}) 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=660540041}) 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021851869}) 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=660540041}) 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4030848254}) 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=660540041}) 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3076949199}) 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=660540041}) 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1307540674}) 2024-06-11T19:20:02.307 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=660540041}) 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2733484011}) 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=660540041}) 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=244190079}) 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=660540041}) 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1315376283}) 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=660540041}) 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.102+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3818964669}) 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.103+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.103+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=528181259}) 2024-06-11T19:20:02.308 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.103+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735350123}) 2024-06-11T19:20:02.309 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.103+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.309 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.103+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=528181259}) 2024-06-11T19:20:02.309 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.103+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=439276701}) 2024-06-11T19:20:02.309 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.103+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.309 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=528181259}) 2024-06-11T19:20:02.309 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=870132776}) 2024-06-11T19:20:02.309 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.309 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=528181259}) 2024-06-11T19:20:02.309 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=791405667}) 2024-06-11T19:20:02.309 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.309 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=528181259}) 2024-06-11T19:20:02.309 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3906177239}) 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=528181259}) 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=241364185}) 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=528181259}) 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=399396267}) 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=528181259}) 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.104+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3228281574}) 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1402811344}) 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4114473574}) 2024-06-11T19:20:02.310 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1402811344}) 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859349611}) 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1402811344}) 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3553671583}) 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1402811344}) 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2410180734}) 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1402811344}) 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=595052337}) 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.311 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1402811344}) 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2390562927}) 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1402811344}) 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3074480267}) 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1402811344}) 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.105+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3652095902}) 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.106+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.106+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2106965106}) 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.106+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4251056889}) 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.106+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.106+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2106965106}) 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.106+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2030870160}) 2024-06-11T19:20:02.312 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.106+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.313 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.106+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2106965106}) 2024-06-11T19:20:02.313 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.106+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3493826565}) 2024-06-11T19:20:02.313 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.106+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.313 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.106+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2106965106}) 2024-06-11T19:20:02.313 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3982928778}) 2024-06-11T19:20:02.313 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.313 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2106965106}) 2024-06-11T19:20:02.313 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3698113924}) 2024-06-11T19:20:02.313 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.313 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2106965106}) 2024-06-11T19:20:02.313 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3888437035}) 2024-06-11T19:20:02.313 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.314 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2106965106}) 2024-06-11T19:20:02.314 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1894577337}) 2024-06-11T19:20:02.314 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.314 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2106965106}) 2024-06-11T19:20:02.314 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.107+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=83364955}) 2024-06-11T19:20:02.314 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.314 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1586190253}) 2024-06-11T19:20:02.314 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3219000923}) 2024-06-11T19:20:02.314 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.314 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1586190253}) 2024-06-11T19:20:02.314 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=303457290}) 2024-06-11T19:20:02.314 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1586190253}) 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=141409484}) 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1586190253}) 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2006768672}) 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1586190253}) 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2044364407}) 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1586190253}) 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1442946574}) 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.315 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1586190253}) 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3969725439}) 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1586190253}) 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.108+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2567162385}) 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.109+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.109+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1334119972}) 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.109+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1257134257}) 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.109+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.109+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1334119972}) 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2867736428}) 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1334119972}) 2024-06-11T19:20:02.316 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1927196270}) 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1334119972}) 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3110709008}) 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1334119972}) 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2175749393}) 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1334119972}) 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1705753627}) 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1334119972}) 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2739839515}) 2024-06-11T19:20:02.317 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1334119972}) 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.110+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2119732662}) 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1189841716}) 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350779781}) 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1189841716}) 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1386786714}) 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1189841716}) 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1455280167}) 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.318 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1189841716}) 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1026196462}) 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1189841716}) 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1716546758}) 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1189841716}) 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2624862763}) 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1189841716}) 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2816557946}) 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1189841716}) 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.111+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1938380794}) 2024-06-11T19:20:02.319 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2504809726}) 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4173196242}) 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2504809726}) 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4154617904}) 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2504809726}) 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2930626602}) 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2504809726}) 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4011913033}) 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.320 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2504809726}) 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=812075495}) 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2504809726}) 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=772771099}) 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2504809726}) 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1466571175}) 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2504809726}) 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.112+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3855026225}) 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3414506134}) 2024-06-11T19:20:02.321 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=210643021}) 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3414506134}) 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=859422691}) 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3414506134}) 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1717919688}) 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3414506134}) 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1824269456}) 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3414506134}) 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4055545561}) 2024-06-11T19:20:02.322 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3414506134}) 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2566113769}) 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3414506134}) 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1977670424}) 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3414506134}) 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.114+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3790287620}) 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=459322467}) 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=604160211}) 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.323 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=459322467}) 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3323527447}) 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=459322467}) 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010981458}) 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=459322467}) 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=949112377}) 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=459322467}) 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=673962514}) 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=459322467}) 2024-06-11T19:20:02.324 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=468543393}) 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=459322467}) 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2212167413}) 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=459322467}) 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.115+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=576704872}) 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2663615129}) 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010665500}) 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2663615129}) 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2369184927}) 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.325 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2663615129}) 2024-06-11T19:20:02.326 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224139335}) 2024-06-11T19:20:02.326 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.326 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2663615129}) 2024-06-11T19:20:02.326 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=947298714}) 2024-06-11T19:20:02.326 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.326 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2663615129}) 2024-06-11T19:20:02.326 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2295679712}) 2024-06-11T19:20:02.326 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.326 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2663615129}) 2024-06-11T19:20:02.326 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=805123611}) 2024-06-11T19:20:02.326 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.326 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2663615129}) 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1413294071}) 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2663615129}) 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.116+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2197048110}) 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=765819001}) 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1117518531}) 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=765819001}) 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3092187303}) 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=765819001}) 2024-06-11T19:20:02.327 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=112861919}) 2024-06-11T19:20:02.328 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.328 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=765819001}) 2024-06-11T19:20:02.328 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1906932144}) 2024-06-11T19:20:02.328 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.328 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=765819001}) 2024-06-11T19:20:02.328 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1443758017}) 2024-06-11T19:20:02.328 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.328 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=765819001}) 2024-06-11T19:20:02.328 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=803829824}) 2024-06-11T19:20:02.328 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.328 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=765819001}) 2024-06-11T19:20:02.329 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3005716880}) 2024-06-11T19:20:02.329 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.329 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=765819001}) 2024-06-11T19:20:02.329 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.118+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=348026250}) 2024-06-11T19:20:02.329 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.329 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=84015483}) 2024-06-11T19:20:02.329 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3887023663}) 2024-06-11T19:20:02.329 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.329 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=84015483}) 2024-06-11T19:20:02.329 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2377539838}) 2024-06-11T19:20:02.329 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.329 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=84015483}) 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3751772887}) 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=84015483}) 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2729055762}) 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=84015483}) 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2908295506}) 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=84015483}) 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1001643139}) 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=84015483}) 2024-06-11T19:20:02.330 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1363029863}) 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=84015483}) 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.119+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2740664446}) 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1645349095}) 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2525338518}) 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1645349095}) 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621715355}) 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1645349095}) 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2585546680}) 2024-06-11T19:20:02.331 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1645349095}) 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3883821142}) 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1645349095}) 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2536896863}) 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1645349095}) 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1241837280}) 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1645349095}) 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2031241385}) 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1645349095}) 2024-06-11T19:20:02.332 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.121+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=140719096}) 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3234579621}) 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2644171975}) 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3234579621}) 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3836377463}) 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3234579621}) 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3062579124}) 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3234579621}) 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1628006851}) 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.333 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3234579621}) 2024-06-11T19:20:02.334 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=711714625}) 2024-06-11T19:20:02.334 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.334 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3234579621}) 2024-06-11T19:20:02.334 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2482990629}) 2024-06-11T19:20:02.334 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.334 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3234579621}) 2024-06-11T19:20:02.334 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1936403778}) 2024-06-11T19:20:02.334 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.334 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3234579621}) 2024-06-11T19:20:02.334 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.122+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4035856464}) 2024-06-11T19:20:02.334 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.334 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2852911542}) 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789865881}) 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2852911542}) 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1740634705}) 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2852911542}) 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2292656625}) 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2852911542}) 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2669394253}) 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2852911542}) 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1524457539}) 2024-06-11T19:20:02.335 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2852911542}) 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1772060742}) 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2852911542}) 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1395681478}) 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2852911542}) 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.124+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1444522469}) 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.125+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.125+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=913009572}) 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.125+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3783979409}) 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.125+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.125+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=913009572}) 2024-06-11T19:20:02.336 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.125+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=982300429}) 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.125+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.125+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=913009572}) 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.125+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3216868839}) 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=913009572}) 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1687146579}) 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=913009572}) 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2735839256}) 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=913009572}) 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=597271550}) 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.337 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=913009572}) 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1992649978}) 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=913009572}) 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.126+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3347402124}) 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.127+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.127+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1572632705}) 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.127+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3734518508}) 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.127+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.127+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1572632705}) 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.127+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=349175}) 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.127+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.127+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1572632705}) 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.127+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715066914}) 2024-06-11T19:20:02.338 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.127+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.339 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.127+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1572632705}) 2024-06-11T19:20:02.339 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.127+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=409737569}) 2024-06-11T19:20:02.339 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.128+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.339 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.128+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1572632705}) 2024-06-11T19:20:02.339 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.128+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=484802417}) 2024-06-11T19:20:02.339 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.128+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.339 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.128+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1572632705}) 2024-06-11T19:20:02.339 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.128+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2668496398}) 2024-06-11T19:20:02.339 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.128+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.339 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.128+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1572632705}) 2024-06-11T19:20:02.339 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.128+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3792426606}) 2024-06-11T19:20:02.339 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.128+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.128+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1572632705}) 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.128+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3110791661}) 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=809112222}) 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2021615884}) 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=809112222}) 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3031879049}) 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=809112222}) 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1853629443}) 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=809112222}) 2024-06-11T19:20:02.340 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2403809634}) 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=809112222}) 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=246217424}) 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=809112222}) 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2888127526}) 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=809112222}) 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1430509329}) 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=809112222}) 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.129+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1931941332}) 2024-06-11T19:20:02.341 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=30305089}) 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1126014186}) 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=30305089}) 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=969527533}) 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=30305089}) 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3669982349}) 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=30305089}) 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=6040583}) 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.342 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=30305089}) 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3346638880}) 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=30305089}) 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2749563869}) 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=30305089}) 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=977021299}) 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=30305089}) 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.131+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3250506252}) 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.132+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.132+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3230262670}) 2024-06-11T19:20:02.343 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.132+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846165906}) 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.132+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.132+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3230262670}) 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.132+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=988493062}) 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.132+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.132+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3230262670}) 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.132+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2393816372}) 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.132+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.132+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3230262670}) 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=865901182}) 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3230262670}) 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2577332218}) 2024-06-11T19:20:02.344 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3230262670}) 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2883043887}) 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3230262670}) 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2220377546}) 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3230262670}) 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.133+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=994857908}) 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1885052376}) 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2760724192}) 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.345 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1885052376}) 2024-06-11T19:20:02.346 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3323676336}) 2024-06-11T19:20:02.346 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.346 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1885052376}) 2024-06-11T19:20:02.346 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=405255194}) 2024-06-11T19:20:02.346 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.346 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1885052376}) 2024-06-11T19:20:02.346 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1988655112}) 2024-06-11T19:20:02.346 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.346 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1885052376}) 2024-06-11T19:20:02.346 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=921766253}) 2024-06-11T19:20:02.346 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.346 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1885052376}) 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1437519494}) 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1885052376}) 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3377515359}) 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1885052376}) 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.134+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=570972064}) 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=39,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=17} crc {logm=2105628197,mdsmap=2840952586,mgr=120143446,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1878955872,mon_config_key=2314039360,monmap=3140514009,osd_metadata=1308157559,osd_pg_creating=2334215655,osd_snap=4053696371}) 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=17} crc {logm=3691457525,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3377113717,mon_config_key=1597087497,monmap=506305176,osd_metadata=1424309990,osd_pg_creating=2334215655,osd_snap=775003910}) 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=39,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=17} crc {logm=2105628197,mdsmap=2840952586,mgr=120143446,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1878955872,mon_config_key=2314039360,monmap=3140514009,osd_metadata=1308157559,osd_pg_creating=2334215655,osd_snap=4053696371}) 2024-06-11T19:20:02.347 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=17} crc {logm=3866381327,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2582894825,mon_config_key=2314039359,monmap=506305176,osd_metadata=1390437783,osd_pg_creating=2334215655,osd_snap=2870165881}) 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=39,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=17} crc {logm=2105628197,mdsmap=2840952586,mgr=120143446,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1878955872,mon_config_key=2314039360,monmap=3140514009,osd_metadata=1308157559,osd_pg_creating=2334215655,osd_snap=4053696371}) 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=38,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_snap=19} crc {logm=971210748,mdsmap=1401145748,mgr=3024510395,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2749271841,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_snap=3208312269}) 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=39,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=17} crc {logm=2105628197,mdsmap=2840952586,mgr=120143446,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1878955872,mon_config_key=2314039360,monmap=3140514009,osd_metadata=1308157559,osd_pg_creating=2334215655,osd_snap=4053696371}) 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=17} crc {logm=4160619615,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=574711221,mon_config_key=1252604422,monmap=1530261226,osd_metadata=1603283398,osd_pg_creating=2334215655,osd_snap=2785399272}) 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=39,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=17} crc {logm=2105628197,mdsmap=2840952586,mgr=120143446,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1878955872,mon_config_key=2314039360,monmap=3140514009,osd_metadata=1308157559,osd_pg_creating=2334215655,osd_snap=4053696371}) 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=18} crc {logm=973323890,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=574711221,mon_config_key=2488442063,monmap=1530261226,osd_metadata=1603283398,osd_pg_creating=2334215655,osd_snap=2184000730}) 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=39,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=17} crc {logm=2105628197,mdsmap=2840952586,mgr=120143446,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1878955872,mon_config_key=2314039360,monmap=3140514009,osd_metadata=1308157559,osd_pg_creating=2334215655,osd_snap=4053696371}) 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=39,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17} crc {logm=2366618538,mdsmap=2840952586,mgr=2264053585,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=993713124,mon_config_key=2314039360,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2334215655,osd_snap=212198954}) 2024-06-11T19:20:02.348 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=39,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=17} crc {logm=2105628197,mdsmap=2840952586,mgr=120143446,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1878955872,mon_config_key=2314039360,monmap=3140514009,osd_metadata=1308157559,osd_pg_creating=2334215655,osd_snap=4053696371}) 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14} crc {logm=2196806010,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1285252858,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2334215655,osd_snap=2394138405}) 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=39,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=17} crc {logm=2105628197,mdsmap=2840952586,mgr=120143446,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1878955872,mon_config_key=2314039360,monmap=3140514009,osd_metadata=1308157559,osd_pg_creating=2334215655,osd_snap=4053696371}) 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.136+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17} crc {logm=1588178195,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1503260998,mon_config_key=124724172,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=2334215655,osd_snap=2831158253}) 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=1444747563,osdmap=255466800}) 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=829497260,osdmap=252157372}) 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=1444747563,osdmap=255466800}) 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=4238703506,osdmap=3682455621}) 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.349 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=1444747563,osdmap=255466800}) 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=3306277471,osdmap=905508509}) 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=1444747563,osdmap=255466800}) 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=3828108547,osdmap=2085604794}) 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=1444747563,osdmap=255466800}) 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=829497260,osdmap=1349098715}) 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=1444747563,osdmap=255466800}) 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=829497260,osdmap=1364707584}) 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=1444747563,osdmap=255466800}) 2024-06-11T19:20:02.350 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=1154137079,osdmap=769138315}) 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=1444747563,osdmap=255466800}) 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.138+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=829497260,osdmap=2098329077}) 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1858209762}) 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1652265625}) 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1858209762}) 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1682399670}) 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1858209762}) 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=788882170}) 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.351 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1858209762}) 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1063806078}) 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1858209762}) 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2089162576}) 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1858209762}) 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3743825744}) 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1858209762}) 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1224522635}) 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1858209762}) 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.140+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=989223527}) 2024-06-11T19:20:02.352 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.141+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.353 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=513662912}) 2024-06-11T19:20:02.353 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1922050468}) 2024-06-11T19:20:02.353 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.353 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=513662912}) 2024-06-11T19:20:02.353 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3622391789}) 2024-06-11T19:20:02.353 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.353 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=513662912}) 2024-06-11T19:20:02.353 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1829222480}) 2024-06-11T19:20:02.353 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.353 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=513662912}) 2024-06-11T19:20:02.353 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2120098278}) 2024-06-11T19:20:02.353 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=513662912}) 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2683686887}) 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=513662912}) 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1643394772}) 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=513662912}) 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2767843300}) 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=513662912}) 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.142+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2935989269}) 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.143+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.354 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.143+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1975196758}) 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.143+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2595469578}) 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.143+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1975196758}) 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1316000828}) 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1975196758}) 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1674911858}) 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1975196758}) 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2293335135}) 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1975196758}) 2024-06-11T19:20:02.355 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2223664509}) 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1975196758}) 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3733725314}) 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1975196758}) 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3196375306}) 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1975196758}) 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.144+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3896567423}) 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.146+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.146+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4110545100}) 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.146+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3879808750}) 2024-06-11T19:20:02.356 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.146+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.146+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4110545100}) 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.146+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3995722821}) 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.146+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.146+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4110545100}) 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.146+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3991188789}) 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.146+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.146+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4110545100}) 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3909515040}) 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4110545100}) 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3402003147}) 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.357 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4110545100}) 2024-06-11T19:20:02.358 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3277885671}) 2024-06-11T19:20:02.358 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.358 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4110545100}) 2024-06-11T19:20:02.358 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1098658435}) 2024-06-11T19:20:02.358 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.358 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4110545100}) 2024-06-11T19:20:02.358 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.147+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1531901847}) 2024-06-11T19:20:02.358 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.358 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=47826356}) 2024-06-11T19:20:02.359 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4090343295}) 2024-06-11T19:20:02.359 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.359 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=47826356}) 2024-06-11T19:20:02.359 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1597504056}) 2024-06-11T19:20:02.359 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.359 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=47826356}) 2024-06-11T19:20:02.359 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3863810773}) 2024-06-11T19:20:02.359 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.359 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=47826356}) 2024-06-11T19:20:02.359 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1892221523}) 2024-06-11T19:20:02.359 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.359 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=47826356}) 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1920207414}) 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.149+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=47826356}) 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.150+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2942908300}) 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.150+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.150+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=47826356}) 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.150+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1830210088}) 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.150+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.150+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=47826356}) 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.150+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3203217240}) 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=29} crc {osdmap=2568459862}) 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=2234371208}) 2024-06-11T19:20:02.360 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=29} crc {osdmap=2568459862}) 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=28} crc {osdmap=1169119008}) 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=29} crc {osdmap=2568459862}) 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=29} crc {osdmap=1851931805}) 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=29} crc {osdmap=2568459862}) 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=29} crc {osdmap=789471079}) 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=29} crc {osdmap=2568459862}) 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=29} crc {osdmap=2573630446}) 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.361 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=29} crc {osdmap=2568459862}) 2024-06-11T19:20:02.362 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=24} crc {osdmap=1758139231}) 2024-06-11T19:20:02.362 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.362 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=29} crc {osdmap=2568459862}) 2024-06-11T19:20:02.362 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=29} crc {osdmap=3014246618}) 2024-06-11T19:20:02.362 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:20:02.362 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=29} crc {osdmap=2568459862}) 2024-06-11T19:20:02.362 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:20:02.158+0000 7f78d4952640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=30} crc {osdmap=14240481}) 2024-06-11T19:20:03.048 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:20:03.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:20:03.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.147+0000 7fce01403640 10 monclient: get_monmap_and_config 2024-06-11T19:20:03.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.147+0000 7fce01403640 10 monclient: build_initial_monmap 2024-06-11T19:20:03.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.147+0000 7fce01403640 10 monclient: monmap: 2024-06-11T19:20:03.146 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:03.146 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:03.148227+0000 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:03.148227+0000 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:03.147 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:03.148 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:03.148 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:03.148 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.147+0000 7fce01403640 1 Processor -- start 2024-06-11T19:20:03.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.147+0000 7fce01403640 1 -- start start 2024-06-11T19:20:03.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.147+0000 7fce01403640 10 monclient: init 2024-06-11T19:20:03.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:03.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 10 monclient: _add_conns ranks=[1,2,9,5,10,4,8,7,3,6,0] 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fcdfc0a4740 addr v1:172.21.15.117:6789/0 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdfc0a2350 0x7fcdfc13db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 10 monclient(hunting): picked mon.noname-b con 0x7fcdfc0a2350 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcdfc13e060 0x7fcdfc140450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 10 monclient(hunting): picked mon.noname-a con 0x7fcdfc13e060 addr v2:172.21.15.98:3300/0 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdfc0a2970 con 0x7fcdfc0a4740 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcdfc0a2c50 con 0x7fcdfc13e060 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcdfc0a2f30 con 0x7fcdfc0a2350 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fce01403640 10 monclient(hunting): authenticate will time out at 1412.955312s 2024-06-11T19:20:03.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcdfc13e060 0x7fcdfc140450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfa7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdfc0a2350 0x7fcdfc13db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcdfc13e060 0x7fcdfc140450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54252/0 (socket says 172.21.15.98:54252) 2024-06-11T19:20:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfa7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdfc0a2350 0x7fcdfc13db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40054/0 (socket says 172.21.15.98:40054) 2024-06-11T19:20:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 1 -- 172.21.15.98:0/2467682306 learned_addr learned my addr 172.21.15.98:0/2467682306 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 10 monclient(hunting): get_auth_request con 0x7fcdfc13e060 auth_method 0 2024-06-11T19:20:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfa7fc640 10 monclient(hunting): get_auth_request con 0x7fcdfc0a2350 auth_method 0 2024-06-11T19:20:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfa7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 10 monclient(hunting): handle_auth_done global_id 65313 payload 274 2024-06-11T19:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 1 -- 172.21.15.98:0/2467682306 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdfc0a2350 msgr2=0x7fcdfc13db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 1 --2- 172.21.15.98:0/2467682306 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdfc0a2350 0x7fcdfc13db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 1 -- 172.21.15.98:0/2467682306 >> v1:172.21.15.117:6789/0 conn(0x7fcdfc0a4740 legacy=0x7fcdfc0a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 1 monclient: found mon.noname-a 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfb7fe640 1 -- 172.21.15.98:0/2467682306 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdfc0a5230 con 0x7fcdfc13e060 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.148+0000 7fcdfa7fc640 1 --2- 172.21.15.98:0/2467682306 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdfc0a2350 0x7fcdfc13db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdfb7fe640 1 --2- 172.21.15.98:0/2467682306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdfc13e060 0x7fcdfc140450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009e80 tx=0x7fcdf402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f5feb22a70f5024 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 1 -- 172.21.15.98:0/2467682306 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcdf402cc30 con 0x7fcdfc13e060 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 10 monclient: dump: 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 10 monclient: _finish_auth 0 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:20:03.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fce01403640 5 monclient: authenticate success, global_id 65313 2024-06-11T19:20:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 1 -- 172.21.15.98:0/2467682306 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdf402cdf0 con 0x7fcdfc13e060 2024-06-11T19:20:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fce01403640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fce01403640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fce01403640 4 set_mon_vals no callback set 2024-06-11T19:20:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 1 -- 172.21.15.98:0/2467682306 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcdf4034860 con 0x7fcdfc13e060 2024-06-11T19:20:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.149+0000 7fcdf9ffb640 10 monclient: dump: 2024-06-11T19:20:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.162 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 10 monclient: shutdown 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 1 -- 172.21.15.98:0/2467682306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdfc13e060 msgr2=0x7fcdfc140450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 1 --2- 172.21.15.98:0/2467682306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdfc13e060 0x7fcdfc140450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009e80 tx=0x7fcdf402be20 comp rx=0 tx=0).stop 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 1 -- 172.21.15.98:0/2467682306 shutdown_connections 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 1 --2- 172.21.15.98:0/2467682306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdfc13e060 0x7fcdfc140450 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 1 --2- 172.21.15.98:0/2467682306 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcdfc0a2350 0x7fcdfc13db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 1 -- 172.21.15.98:0/2467682306 >> 172.21.15.98:0/2467682306 conn(0x7fcdfc0ab6d0 msgr2=0x7fcdfc0a78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 1 -- 172.21.15.98:0/2467682306 shutdown_connections 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 1 -- 172.21.15.98:0/2467682306 wait complete. 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 10 monclient: build_initial_monmap 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 10 monclient: monmap: 2024-06-11T19:20:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 1 librados: starting msgr at 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.150+0000 7fce01403640 1 librados: starting objecter 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 1 Processor -- start 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 1 -- start start 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 1 librados: setting wanted keys 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 1 librados: calling monclient init 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 10 monclient: init 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 10 monclient: _add_conns ranks=[5,1,3,8,0,6,7,4,2] 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcdfc0a2350 0x7fcdfc1cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 10 monclient(hunting): picked mon.noname-h con 0x7fcdfc0a2350 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc0a4740 0x7fcdfc1362d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 10 monclient(hunting): picked mon.noname-f con 0x7fcdfc0a4740 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcdfc13e060 0x7fcdfc136810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 10 monclient(hunting): picked mon.noname-g con 0x7fcdfc13e060 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc0a6b50 con 0x7fcdfc0a4740 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fcdfc138d10 con 0x7fcdfc13e060 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fcdfc13db00 con 0x7fcdfc0a2350 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.151+0000 7fce01403640 10 monclient(hunting): authenticate will time out at 1412.958750s 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc0a4740 0x7fcdfc1362d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfb7fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcdfc13e060 0x7fcdfc136810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfaffd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcdfc0a2350 0x7fcdfc1cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc0a4740 0x7fcdfc1362d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:55144/0 (socket says 172.21.15.98:55144) 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 1 -- 172.21.15.98:0/2838196143 learned_addr learned my addr 172.21.15.98:0/2838196143 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfaffd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcdfc0a2350 0x7fcdfc1cd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:43254/0 (socket says 172.21.15.98:43254) 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 10 monclient(hunting): get_auth_request con 0x7fcdfc0a4740 auth_method 0 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfb7fe640 10 monclient(hunting): get_auth_request con 0x7fcdfc13e060 auth_method 0 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfaffd640 10 monclient(hunting): get_auth_request con 0x7fcdfc0a2350 auth_method 0 2024-06-11T19:20:03.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfaffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 10 monclient(hunting): handle_auth_done global_id 32428 payload 824 2024-06-11T19:20:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 1 -- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcdfc0a2350 msgr2=0x7fcdfc1cd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 1 --2- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcdfc0a2350 0x7fcdfc1cd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 1 -- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcdfc13e060 msgr2=0x7fcdfc136810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 1 --2- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcdfc13e060 0x7fcdfc136810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:20:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 1 monclient: found mon.noname-f 2024-06-11T19:20:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:20:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 1 -- 172.21.15.98:0/2838196143 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdfc0a1160 con 0x7fcdfc0a4740 2024-06-11T19:20:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdfa7fc640 1 --2- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc0a4740 0x7fcdfc1362d0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00ebe0 tx=0x7fcdec00c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 10 client.?.objecter ms_handle_connect 0x7fcdfc0a4740 2024-06-11T19:20:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 1 -- 172.21.15.98:0/2838196143 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcdec017070 con 0x7fcdfc0a4740 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 10 monclient: dump: 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 10 monclient: _finish_auth 0 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 10 monclient: _check_auth_tickets 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 1 -- 172.21.15.98:0/2838196143 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdec00cc30 con 0x7fcdfc0a4740 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fce01403640 5 monclient: authenticate success, global_id 32428 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fce01403640 10 monclient: _renew_subs 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fce01403640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:20:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fce01403640 1 -- 172.21.15.98:0/2838196143 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdfc122c80 con 0x7fcdfc0a4740 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 1 -- 172.21.15.98:0/2838196143 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcdec00cde0 con 0x7fcdfc0a4740 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf37fe640 10 monclient: dump: 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fce01403640 10 client.32428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fce01403640 10 monclient: _renew_subs 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fce01403640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fce01403640 1 -- 172.21.15.98:0/2838196143 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdfc121700 con 0x7fcdfc0a4740 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fcdf97fa640 4 set_mon_vals no callback set 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.152+0000 7fce01403640 1 librados: init done 2024-06-11T19:20:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.153+0000 7fce01403640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.153+0000 7fce01403640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.153+0000 7fce01403640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.153+0000 7fce01403640 1 -- 172.21.15.98:0/2838196143 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdfc0a4b20 con 0x7fcdfc0a4740 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.153+0000 7fcdf37fe640 1 -- 172.21.15.98:0/2838196143 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fcdec01a070 con 0x7fcdfc0a4740 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.154+0000 7fcdf37fe640 1 --2- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcddc03fdb0 0x7fcddc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.154+0000 7fcdf37fe640 1 -- 172.21.15.98:0/2838196143 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7fcdec00f040 con 0x7fcdfc0a4740 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.154+0000 7fcdf37fe640 10 client.32428.objecter ms_dispatch 0x7fcdfc000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.154+0000 7fcdf37fe640 3 client.32428.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.154+0000 7fcdf37fe640 3 client.32428.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.154+0000 7fcdf37fe640 20 client.32428.objecter dump_active .. 0 homeless 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.154+0000 7fcdfaffd640 1 --2- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcddc03fdb0 0x7fcddc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.154+0000 7fcdfaffd640 10 monclient: get_auth_request con 0x7fcddc03fdb0 auth_method 0 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.154+0000 7fcdf37fe640 10 client.32428.objecter ms_handle_connect 0x7fcddc03fdb0 2024-06-11T19:20:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.154+0000 7fcdfaffd640 1 --2- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcddc03fdb0 0x7fcddc042270 secure :-1 s=READY pgs=4206 cs=0 l=1 rev1=1 crypto rx=0x7fcde40017c0 tx=0x7fcde4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.158+0000 7fcdf37fe640 1 -- 172.21.15.98:0/2838196143 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcdec013c70 con 0x7fcdfc0a4740 2024-06-11T19:20:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.158+0000 7fcdf37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.158+0000 7fcdf37fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:03.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.407+0000 7fce01403640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:20:03.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.407+0000 7fce01403640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:20:03.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.407+0000 7fce01403640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:20:03.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.407+0000 7fce01403640 1 -- 172.21.15.98:0/2838196143 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcdfc063be0 con 0x7fcdfc0a4740 2024-06-11T19:20:03.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.408+0000 7fcdf37fe640 1 -- 172.21.15.98:0/2838196143 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fcdec01ed40 con 0x7fcdfc0a4740 2024-06-11T19:20:03.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.408+0000 7fcdf37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:20:03.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.408+0000 7fcdf37fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:03.408 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":54,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:20:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 10 librados: watch_flush enter 2024-06-11T19:20:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 10 librados: watch_flush exit 2024-06-11T19:20:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 20 client.32428.objecter shutdown clearing up homeless session... 2024-06-11T19:20:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 10 client.32428.objecter successfully canceled tick 2024-06-11T19:20:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 -- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcddc03fdb0 msgr2=0x7fcddc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 --2- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcddc03fdb0 0x7fcddc042270 secure :-1 s=READY pgs=4206 cs=0 l=1 rev1=1 crypto rx=0x7fcde40017c0 tx=0x7fcde4009040 comp rx=0 tx=0).stop 2024-06-11T19:20:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 10 monclient: shutdown 2024-06-11T19:20:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 -- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc0a4740 msgr2=0x7fcdfc1362d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 --2- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc0a4740 0x7fcdfc1362d0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00ebe0 tx=0x7fcdec00c520 comp rx=0 tx=0).stop 2024-06-11T19:20:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 -- 172.21.15.98:0/2838196143 shutdown_connections 2024-06-11T19:20:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 --2- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcddc03fdb0 0x7fcddc042270 unknown :-1 s=CLOSED pgs=4206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 --2- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcdfc13e060 0x7fcdfc136810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 --2- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc0a4740 0x7fcdfc1362d0 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 --2- 172.21.15.98:0/2838196143 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcdfc0a2350 0x7fcdfc1cd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 -- 172.21.15.98:0/2838196143 >> 172.21.15.98:0/2838196143 conn(0x7fcdfc0ab6d0 msgr2=0x7fcdfc0a9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 -- 172.21.15.98:0/2838196143 shutdown_connections 2024-06-11T19:20:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 -- 172.21.15.98:0/2838196143 wait complete. 2024-06-11T19:20:03.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.410+0000 7fce01403640 1 librados: shutdown 2024-06-11T19:20:03.420 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:20:03.420 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-06-11T19:20:03.420 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:20:03.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:20:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.520+0000 7f307f292640 10 monclient: get_monmap_and_config 2024-06-11T19:20:03.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.520+0000 7f307f292640 10 monclient: build_initial_monmap 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.520+0000 7f307f292640 10 monclient: monmap: 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:03.521578+0000 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:03.521578+0000 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:03.521 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 1 Processor -- start 2024-06-11T19:20:03.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 1 -- start start 2024-06-11T19:20:03.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 10 monclient: init 2024-06-11T19:20:03.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:03.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 10 monclient: _add_conns ranks=[8,5,6,4,0,9,10,7,3,2,1] 2024-06-11T19:20:03.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f30780a4920 0x7f30780a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 10 monclient(hunting): picked mon.noname-e con 0x7f30780a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:03.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f30780a5240 0x7f30780a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 10 monclient(hunting): picked mon.noname-h con 0x7f30780a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f30780a2530 0x7f30780a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 10 monclient(hunting): picked mon.noname-d con 0x7f30780a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f30780a7140 con 0x7f30780a5240 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f30780a2b50 con 0x7f30780a2530 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f30780a2e30 con 0x7f30780a4920 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.521+0000 7f307f292640 10 monclient(hunting): authenticate will time out at 1413.328931s 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f30780a4920 0x7f30780a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d82f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f30780a2530 0x7f30780a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f30780a4920 0x7f30780a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:60686/0 (socket says 172.21.15.98:60686) 2024-06-11T19:20:03.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 1 -- 172.21.15.98:0/1718011875 learned_addr learned my addr 172.21.15.98:0/1718011875 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 10 monclient(hunting): get_auth_request con 0x7f30780a4920 auth_method 0 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d82f640 10 monclient(hunting): get_auth_request con 0x7f30780a2530 auth_method 0 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d82f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d82f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d82f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d82f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d82f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307c82d640 1 --2- 172.21.15.98:0/1718011875 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f30780a5240 0x7f30780a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d82f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307c82d640 10 monclient(hunting): get_auth_request con 0x7f30780a5240 auth_method 0 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307c82d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307c82d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307c82d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307c82d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307c82d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307c82d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 10 monclient(hunting): handle_auth_done global_id 94328 payload 274 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 1 -- 172.21.15.98:0/1718011875 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f30780a2530 msgr2=0x7f30780a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 1 --2- 172.21.15.98:0/1718011875 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f30780a2530 0x7f30780a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 1 -- 172.21.15.98:0/1718011875 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f30780a5240 msgr2=0x7f30780a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 1 --2- 172.21.15.98:0/1718011875 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f30780a5240 0x7f30780a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 10 monclient: _finish_hunting 0 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 1 monclient: found mon.noname-e 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d02e640 1 -- 172.21.15.98:0/1718011875 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30780a3110 con 0x7f30780a4920 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.522+0000 7f307d82f640 1 --2- 172.21.15.98:0/1718011875 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f30780a2530 0x7f30780a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307d02e640 1 --2- 172.21.15.98:0/1718011875 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f30780a4920 0x7f30780a4d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f30740098a0 tx=0x7f307402b7a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=5c24dcf1ffb91911 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 1 -- 172.21.15.98:0/1718011875 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3074038070 con 0x7f30780a4920 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 10 monclient: dump: 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 10 monclient: _finish_auth 0 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 5 monclient: authenticate success, global_id 94328 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 1 -- 172.21.15.98:0/1718011875 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3074031970 con 0x7f30780a4920 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 4 set_mon_vals no callback set 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 1 -- 172.21.15.98:0/1718011875 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3074031ba0 con 0x7f30780a4920 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f306ffff640 10 monclient: dump: 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.537 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.538 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 10 monclient: shutdown 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 1 -- 172.21.15.98:0/1718011875 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f30780a4920 msgr2=0x7f30780a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 1 --2- 172.21.15.98:0/1718011875 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f30780a4920 0x7f30780a4d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f30740098a0 tx=0x7f307402b7a0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 1 -- 172.21.15.98:0/1718011875 shutdown_connections 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 1 --2- 172.21.15.98:0/1718011875 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f30780a2530 0x7f30780a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 1 --2- 172.21.15.98:0/1718011875 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f30780a5240 0x7f30780a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 1 --2- 172.21.15.98:0/1718011875 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f30780a4920 0x7f30780a4d00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 1 -- 172.21.15.98:0/1718011875 >> 172.21.15.98:0/1718011875 conn(0x7f30780ab850 msgr2=0x7f30780a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 1 -- 172.21.15.98:0/1718011875 shutdown_connections 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.523+0000 7f307f292640 1 -- 172.21.15.98:0/1718011875 wait complete. 2024-06-11T19:20:03.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.524+0000 7f307f292640 10 monclient: build_initial_monmap 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.524+0000 7f307f292640 10 monclient: monmap: 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:03.540 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.524+0000 7f307f292640 1 librados: starting msgr at 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.524+0000 7f307f292640 1 librados: starting objecter 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.524+0000 7f307f292640 1 Processor -- start 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.524+0000 7f307f292640 1 -- start start 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.524+0000 7f307f292640 1 librados: setting wanted keys 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.524+0000 7f307f292640 1 librados: calling monclient init 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.524+0000 7f307f292640 10 monclient: init 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 10 monclient: _add_conns ranks=[7,1,6,2,3,5,8,4,0] 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f30780a2530 0x7f3078137740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 10 monclient(hunting): picked mon.noname-i con 0x7f30780a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30780a4920 0x7f3078137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 10 monclient(hunting): picked mon.noname-f con 0x7f30780a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f30780a5240 0x7f30781232a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 10 monclient(hunting): picked mon.noname-d con 0x7f30780a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f30780a6da0 con 0x7f30780a4920 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3078138e50 con 0x7f30780a5240 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f30780a6420 con 0x7f30780a2530 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 10 monclient(hunting): authenticate will time out at 1413.332228s 2024-06-11T19:20:03.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f30780a5240 0x7f30781232a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f30780a5240 0x7f30781232a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:37810/0 (socket says 172.21.15.98:37810) 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 1 -- 172.21.15.98:0/2313810951 learned_addr learned my addr 172.21.15.98:0/2313810951 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 10 monclient(hunting): get_auth_request con 0x7f30780a5240 auth_method 0 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d02e640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f30780a2530 0x7f3078137740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d02e640 10 monclient(hunting): get_auth_request con 0x7f30780a2530 auth_method 0 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d02e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d02e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d02e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307c82d640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30780a4920 0x7f3078137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307c82d640 10 monclient(hunting): get_auth_request con 0x7f30780a4920 auth_method 0 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307c82d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307c82d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307c82d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 10 monclient(hunting): handle_auth_done global_id 55338 payload 824 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 1 -- 172.21.15.98:0/2313810951 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f30780a2530 msgr2=0x7f3078137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f30780a2530 0x7f3078137740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 1 -- 172.21.15.98:0/2313810951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30780a4920 msgr2=0x7f3078137c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30780a4920 0x7f3078137c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 10 monclient: _finish_hunting 0 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 1 monclient: found mon.noname-d 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:03.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 1 -- 172.21.15.98:0/2313810951 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30781c0d10 con 0x7f30780a5240 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307d82f640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f30780a5240 0x7f30781232a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f307000efb0 tx=0x7f307000c490 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 10 client.?.objecter ms_handle_connect 0x7f30780a5240 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3070018020 con 0x7f30780a5240 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 10 monclient: dump: 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 10 monclient: _finish_auth 0 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 10 monclient: _check_auth_tickets 2024-06-11T19:20:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 5 monclient: authenticate success, global_id 55338 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f307000cdf0 con 0x7f30780a5240 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3070014d60 con 0x7f30780a5240 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 10 monclient: _renew_subs 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f307f292640 1 -- 172.21.15.98:0/2313810951 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3078126b80 con 0x7f30780a5240 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.525+0000 7f306f7fe640 4 set_mon_vals no callback set 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.526+0000 7f306dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.526+0000 7f306dffb640 10 monclient: dump: 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.526+0000 7f307f292640 10 client.55338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.526+0000 7f307f292640 10 monclient: _renew_subs 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.526+0000 7f307f292640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.526+0000 7f307f292640 1 -- 172.21.15.98:0/2313810951 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f30780a92f0 con 0x7f30780a5240 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.526+0000 7f307f292640 1 librados: init done 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.526+0000 7f307f292640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.526+0000 7f307f292640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3078126dc0 0x7f3078123fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.526+0000 7f307f292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.526+0000 7f307f292640 1 -- 172.21.15.98:0/2313810951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3078050830 con 0x7f3078126dc0 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3070005d10 con 0x7f30780a5240 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307d02e640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3078126dc0 0x7f3078123fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307d02e640 10 monclient: get_auth_request con 0x7f3078126dc0 auth_method 0 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307d02e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307d02e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307d02e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307d02e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307d02e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307d02e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f305c03fdb0 0x7f305c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307d02e640 10 monclient(hunting): handle_auth_done global_id 65322 payload 824 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7f3070052510 con 0x7f30780a5240 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307d02e640 1 -- 172.21.15.98:0/2313810951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3078124e40 con 0x7f3078126dc0 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 10 client.55338.objecter ms_dispatch 0x7f3078000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 3 client.55338.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 3 client.55338.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 20 client.55338.objecter dump_active .. 0 homeless 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307d02e640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3078126dc0 0x7f3078123fa0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3074009870 tx=0x7f30740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 10 client.55338.objecter ms_handle_connect 0x7f3078126dc0 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 10 client.55338.objecter resend_mon_ops 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307c82d640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f305c03fdb0 0x7f305c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307c82d640 10 monclient: get_auth_request con 0x7f305c03fdb0 auth_method 0 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3074043070 con 0x7f3078126dc0 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 10 monclient: dump: 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 10 client.55338.objecter ms_handle_connect 0x7f305c03fdb0 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f307c82d640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f305c03fdb0 0x7f305c042270 secure :-1 s=READY pgs=4207 cs=0 l=1 rev1=1 crypto rx=0x7f30780a5620 tx=0x7f3068008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3074031ea0 con 0x7f3078126dc0 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:03.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3078126dc0 msgr2=0x7f3078123fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.527+0000 7f306dffb640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3078126dc0 0x7f3078123fa0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3074009870 tx=0x7f30740046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.528+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3078126dc0 msgr2=0x7f3078123fa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:03.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.528+0000 7f306dffb640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3078126dc0 0x7f3078123fa0 secure :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3074009870 tx=0x7f30740046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307f292640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:03.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307f292640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307802cb40 0x7f307802cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307f292640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307f292640 1 -- 172.21.15.98:0/2313810951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3078073550 con 0x7f307802cb40 2024-06-11T19:20:03.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307d02e640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307802cb40 0x7f307802cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307d02e640 10 monclient: get_auth_request con 0x7f307802cb40 auth_method 0 2024-06-11T19:20:03.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307d02e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307d02e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307d02e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307d02e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307d02e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307d02e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307d02e640 10 monclient(hunting): handle_auth_done global_id 65331 payload 824 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307d02e640 1 -- 172.21.15.98:0/2313810951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f30780a66f0 con 0x7f307802cb40 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f307d02e640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307802cb40 0x7f307802cf50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f30740098a0 tx=0x7f3074031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f306dffb640 10 client.55338.objecter ms_handle_connect 0x7f307802cb40 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.534+0000 7f306dffb640 10 client.55338.objecter resend_mon_ops 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3074038040 con 0x7f307802cb40 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f306dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f306dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f306dffb640 10 monclient: dump: 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.556 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f3074003c10 con 0x7f307802cb40 2024-06-11T19:20:03.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f306dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:03.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f306dffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:03.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307802cb40 msgr2=0x7f307802cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f306dffb640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307802cb40 0x7f307802cf50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f30740098a0 tx=0x7f3074031000 comp rx=0 tx=0).stop 2024-06-11T19:20:03.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f306dffb640 1 -- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307802cb40 msgr2=0x7f307802cf50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:03.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f306dffb640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307802cb40 0x7f307802cf50 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.558 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:03.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:03.558 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:03.558 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 395363, 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:03.559 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:03.560 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:03.561 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:03.562 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:03.563 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.565 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.567 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:03.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.570 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:03.572 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.573 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.573 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.573 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:03.573 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:03.573 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.573 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.573 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.573 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:03.573 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.574 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:03.574 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.574 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.574 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.574 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:03.574 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:03.574 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.574 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.575 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.576 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.577 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.579 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:03.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.581 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:03.581 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:03.581 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:20:03.581 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.581 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.581 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:03.581 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:03.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 10 librados: watch_flush enter 2024-06-11T19:20:03.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 10 librados: watch_flush exit 2024-06-11T19:20:03.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 20 client.55338.objecter shutdown clearing up homeless session... 2024-06-11T19:20:03.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 10 client.55338.objecter successfully canceled tick 2024-06-11T19:20:03.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 1 -- 172.21.15.98:0/2313810951 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f305c03fdb0 msgr2=0x7f305c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f305c03fdb0 0x7f305c042270 secure :-1 s=READY pgs=4207 cs=0 l=1 rev1=1 crypto rx=0x7f30780a5620 tx=0x7f3068008040 comp rx=0 tx=0).stop 2024-06-11T19:20:03.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 10 monclient: shutdown 2024-06-11T19:20:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 1 -- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f30780a5240 msgr2=0x7f30781232a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 1 --2- 172.21.15.98:0/2313810951 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f30780a5240 0x7f30781232a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f307000efb0 tx=0x7f307000c490 comp rx=0 tx=0).stop 2024-06-11T19:20:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307d82f640 1 -- 172.21.15.98:0/2313810951 reap_dead start 2024-06-11T19:20:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307d82f640 1 -- 172.21.15.98:0/2313810951 reap_dead start 2024-06-11T19:20:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 1 -- 172.21.15.98:0/2313810951 shutdown_connections 2024-06-11T19:20:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 1 -- 172.21.15.98:0/2313810951 >> 172.21.15.98:0/2313810951 conn(0x7f30780ab850 msgr2=0x7f30780a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 1 -- 172.21.15.98:0/2313810951 shutdown_connections 2024-06-11T19:20:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 1 -- 172.21.15.98:0/2313810951 wait complete. 2024-06-11T19:20:03.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.535+0000 7f307f292640 1 librados: shutdown 2024-06-11T19:20:03.585 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:03.585 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:20:03.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:20:03.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.685+0000 7fe403ca4640 10 monclient: get_monmap_and_config 2024-06-11T19:20:03.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.685+0000 7fe403ca4640 10 monclient: build_initial_monmap 2024-06-11T19:20:03.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.685+0000 7fe403ca4640 10 monclient: monmap: 2024-06-11T19:20:03.695 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:03.695 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.695 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:03.686321+0000 2024-06-11T19:20:03.695 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:03.686321+0000 2024-06-11T19:20:03.695 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:03.696 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.685+0000 7fe403ca4640 1 Processor -- start 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 1 -- start start 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 10 monclient: init 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 10 monclient: _add_conns ranks=[0,7,6,2,9,8,1,4,3,5,10] 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3fc0a6260 addr v1:172.21.15.98:6789/0 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe3fc06c890 0x7fe3fc13da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 10 monclient(hunting): picked mon.noname-i con 0x7fe3fc06c890 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe3fc0a2d00 0x7fe3fc13ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 10 monclient(hunting): picked mon.noname-d con 0x7fe3fc0a2d00 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3fc0a2460 con 0x7fe3fc0a6260 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe3fc0a05b0 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fe3fc0a6640 con 0x7fe3fc06c890 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe403ca4640 10 monclient(hunting): authenticate will time out at 1413.493576s 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe3fc0a2d00 0x7fe3fc13ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe401a40640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe3fc0a6260 0x7fe3fc0a1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55842/0 (socket says 172.21.15.98:55842) 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe401a40640 1 -- 172.21.15.98:0/2796685995 learned_addr learned my addr 172.21.15.98:0/2796685995 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 10 monclient(hunting): get_auth_request con 0x7fe3fc0a2d00 auth_method 0 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe40123f640 1 --2- 172.21.15.98:0/2796685995 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe3fc06c890 0x7fe3fc13da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe40123f640 10 monclient(hunting): get_auth_request con 0x7fe3fc06c890 auth_method 0 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe40123f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe40123f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe40123f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 10 monclient(hunting): handle_auth_done global_id 55347 payload 274 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 1 -- 172.21.15.98:0/2796685995 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe3fc06c890 msgr2=0x7fe3fc13da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 1 --2- 172.21.15.98:0/2796685995 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe3fc06c890 0x7fe3fc13da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe400a3e640 1 -- 172.21.15.98:0/2796685995 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2372816598 0 0) 0x7fe3fc0a2460 con 0x7fe3fc0a6260 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 1 -- 172.21.15.98:0/2796685995 >> v1:172.21.15.98:6789/0 conn(0x7fe3fc0a6260 legacy=0x7fe3fc0a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 10 monclient: _finish_hunting 0 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 1 monclient: found mon.noname-d 2024-06-11T19:20:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe402241640 1 -- 172.21.15.98:0/2796685995 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3fc06c650 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.686+0000 7fe400a3e640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe402241640 1 --2- 172.21.15.98:0/2796685995 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe3fc0a2d00 0x7fe3fc13ffd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009df0 tx=0x7fe3f402bd50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=b2b2c669a3381b7c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 1 -- 172.21.15.98:0/2796685995 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3f4039070 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 10 monclient: dump: 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.700 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 10 monclient: _finish_auth 0 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 10 monclient: _check_auth_tickets 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 1 -- 172.21.15.98:0/2796685995 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f402cd70 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe403ca4640 5 monclient: authenticate success, global_id 55347 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe403ca4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe403ca4640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 1 -- 172.21.15.98:0/2796685995 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3f40327b0 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe403ca4640 4 set_mon_vals no callback set 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.687+0000 7fe400a3e640 10 monclient: dump: 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.688+0000 7fe403ca4640 10 monclient: shutdown 2024-06-11T19:20:03.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.688+0000 7fe403ca4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.688+0000 7fe403ca4640 1 -- 172.21.15.98:0/2796685995 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe3fc0a2d00 msgr2=0x7fe3fc13ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.688+0000 7fe403ca4640 1 --2- 172.21.15.98:0/2796685995 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe3fc0a2d00 0x7fe3fc13ffd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009df0 tx=0x7fe3f402bd50 comp rx=0 tx=0).stop 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.688+0000 7fe403ca4640 1 -- 172.21.15.98:0/2796685995 shutdown_connections 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.688+0000 7fe403ca4640 1 --2- 172.21.15.98:0/2796685995 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe3fc0a2d00 0x7fe3fc13ffd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.688+0000 7fe403ca4640 1 --2- 172.21.15.98:0/2796685995 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe3fc06c890 0x7fe3fc13da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.688+0000 7fe403ca4640 1 -- 172.21.15.98:0/2796685995 >> 172.21.15.98:0/2796685995 conn(0x7fe3fc0a95b0 msgr2=0x7fe3fc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.688+0000 7fe403ca4640 1 -- 172.21.15.98:0/2796685995 shutdown_connections 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.688+0000 7fe403ca4640 1 -- 172.21.15.98:0/2796685995 wait complete. 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient: build_initial_monmap 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient: monmap: 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:03.704 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 1 librados: starting msgr at 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 1 librados: starting objecter 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 1 Processor -- start 2024-06-11T19:20:03.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 1 -- start start 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 1 librados: setting wanted keys 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 1 librados: calling monclient init 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient: init 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient: _add_conns ranks=[1,0,8,3,2,5,4,7,6] 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3fc06c890 0x7fe3fc13d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient(hunting): picked mon.noname-f con 0x7fe3fc06c890 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc0a2d00 0x7fe3fc13d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient(hunting): picked mon.noname-a con 0x7fe3fc0a2d00 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe3fc0a6260 0x7fe3fc13dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient(hunting): picked mon.noname-e con 0x7fe3fc0a6260 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc140510 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc06cd10 con 0x7fe3fc06c890 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe3fc140690 con 0x7fe3fc0a6260 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.689+0000 7fe403ca4640 10 monclient(hunting): authenticate will time out at 1413.496942s 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc0a2d00 0x7fe3fc13d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe402241640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe3fc0a6260 0x7fe3fc13dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe402241640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe3fc0a6260 0x7fe3fc13dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:60688/0 (socket says 172.21.15.98:60688) 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe402241640 1 -- 172.21.15.98:0/661190456 learned_addr learned my addr 172.21.15.98:0/661190456 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe402241640 10 monclient(hunting): get_auth_request con 0x7fe3fc0a6260 auth_method 0 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe402241640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe402241640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe402241640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 10 monclient(hunting): get_auth_request con 0x7fe3fc0a2d00 auth_method 0 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe402241640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe402241640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe401a40640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3fc06c890 0x7fe3fc13d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe402241640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe401a40640 10 monclient(hunting): get_auth_request con 0x7fe3fc06c890 auth_method 0 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe401a40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe401a40640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe401a40640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 10 monclient(hunting): handle_auth_done global_id 65349 payload 824 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 1 -- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe3fc0a6260 msgr2=0x7fe3fc13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe3fc0a6260 0x7fe3fc13dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 1 -- 172.21.15.98:0/661190456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3fc06c890 msgr2=0x7fe3fc13d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3fc06c890 0x7fe3fc13d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 10 monclient: _finish_hunting 0 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 1 monclient: found mon.noname-a 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 1 -- 172.21.15.98:0/661190456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3fc13f9e0 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe401a40640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3fc06c890 0x7fe3fc13d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:20:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe40123f640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc0a2d00 0x7fe3fc13d730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00d5a0 tx=0x7fe3ec00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 10 client.?.objecter ms_handle_connect 0x7fe3fc0a2d00 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3ec019070 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 10 monclient: dump: 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 10 monclient: _finish_auth 0 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 10 monclient: _check_auth_tickets 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe403ca4640 5 monclient: authenticate success, global_id 65349 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3ec004240 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3ec0043f0 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe403ca4640 10 monclient: _renew_subs 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe403ca4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe403ca4640 1 -- 172.21.15.98:0/661190456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3fc13e7a0 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f27fc640 10 monclient: dump: 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.712 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.690+0000 7fe3f3fff640 4 set_mon_vals no callback set 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.691+0000 7fe403ca4640 10 client.65349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.691+0000 7fe403ca4640 10 monclient: _renew_subs 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.691+0000 7fe403ca4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.691+0000 7fe403ca4640 1 -- 172.21.15.98:0/661190456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3fc0a5b50 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.691+0000 7fe403ca4640 1 librados: init done 2024-06-11T19:20:03.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.691+0000 7fe403ca4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:03.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.691+0000 7fe403ca4640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138350 0x7fe3fc138730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.691+0000 7fe403ca4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.691+0000 7fe403ca4640 1 -- 172.21.15.98:0/661190456 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe3fc13e9c0 con 0x7fe3fc138350 2024-06-11T19:20:03.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.691+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fe3ec003610 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe401a40640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138350 0x7fe3fc138730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe401a40640 10 monclient: get_auth_request con 0x7fe3fc138350 auth_method 0 2024-06-11T19:20:03.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe401a40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe401a40640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe401a40640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe401a40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe401a40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe401a40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe3e003fdb0 0x7fe3e0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7fe3ec00f040 con 0x7fe3fc0a2d00 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 10 client.65349.objecter ms_dispatch 0x7fe3fc000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 3 client.65349.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 3 client.65349.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe401a40640 10 monclient(hunting): handle_auth_done global_id 84692 payload 824 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe401a40640 1 -- 172.21.15.98:0/661190456 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3fc0a1d50 con 0x7fe3fc138350 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 20 client.65349.objecter dump_active .. 0 homeless 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe402241640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe3e003fdb0 0x7fe3e0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe402241640 10 monclient: get_auth_request con 0x7fe3e003fdb0 auth_method 0 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe401a40640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138350 0x7fe3fc138730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8000f60 tx=0x7fe3f800e980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 10 client.65349.objecter ms_handle_connect 0x7fe3fc138350 2024-06-11T19:20:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 10 client.65349.objecter resend_mon_ops 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3f80070d0 con 0x7fe3fc138350 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 10 monclient: dump: 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 10 client.65349.objecter ms_handle_connect 0x7fe3e003fdb0 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe402241640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe3e003fdb0 0x7fe3e0042270 secure :-1 s=READY pgs=4208 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009bd0 tx=0x7fe3f4030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe3f8018070 con 0x7fe3fc138350 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138350 msgr2=0x7fe3fc138730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138350 0x7fe3fc138730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8000f60 tx=0x7fe3f800e980 comp rx=0 tx=0).stop 2024-06-11T19:20:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138350 msgr2=0x7fe3fc138730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:03.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.692+0000 7fe3f27fc640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138350 0x7fe3fc138730 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe403ca4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:03.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe403ca4640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138c70 0x7fe3fc070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe403ca4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe403ca4640 1 -- 172.21.15.98:0/661190456 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe3fc050830 con 0x7fe3fc138c70 2024-06-11T19:20:03.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe401a40640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138c70 0x7fe3fc070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe401a40640 10 monclient: get_auth_request con 0x7fe3fc138c70 auth_method 0 2024-06-11T19:20:03.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe401a40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe401a40640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe401a40640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe401a40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe401a40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe401a40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe401a40640 10 monclient(hunting): handle_auth_done global_id 84701 payload 824 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe401a40640 1 -- 172.21.15.98:0/661190456 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe3fc13e520 con 0x7fe3fc138c70 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe3f27fc640 10 client.65349.objecter ms_handle_connect 0x7fe3fc138c70 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe3f27fc640 10 client.65349.objecter resend_mon_ops 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe401a40640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138c70 0x7fe3fc070c10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8017000 tx=0x7fe3f8000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3f801de40 con 0x7fe3fc138c70 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe3f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe3f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.699+0000 7fe3f27fc640 10 monclient: dump: 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.722 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fe3f801d9d0 con 0x7fe3fc138c70 2024-06-11T19:20:03.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe3f27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:03.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe3f27fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:03.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138c70 msgr2=0x7fe3fc070c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe3f27fc640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138c70 0x7fe3fc070c10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8017000 tx=0x7fe3f8000ed0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe3f27fc640 1 -- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138c70 msgr2=0x7fe3fc070c10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:03.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe3f27fc640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3fc138c70 0x7fe3fc070c10 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8017000 tx=0x7fe3f8000ed0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.723 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 275473, 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:03.724 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:03.725 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:03.726 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:03.727 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:03.728 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:03.729 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:03.729 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:03.729 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:03.729 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:03.729 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:03.729 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:03.729 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:03.729 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:03.729 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:03.729 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:03.729 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.729 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.731 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:03.733 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:03.746 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:03.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 10 librados: watch_flush enter 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 10 librados: watch_flush exit 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 20 client.65349.objecter shutdown clearing up homeless session... 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 10 client.65349.objecter successfully canceled tick 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 1 -- 172.21.15.98:0/661190456 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe3e003fdb0 msgr2=0x7fe3e0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe3e003fdb0 0x7fe3e0042270 secure :-1 s=READY pgs=4208 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009bd0 tx=0x7fe3f4030000 comp rx=0 tx=0).stop 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 10 monclient: shutdown 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 1 -- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc0a2d00 msgr2=0x7fe3fc13d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 1 --2- 172.21.15.98:0/661190456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3fc0a2d00 0x7fe3fc13d730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00d5a0 tx=0x7fe3ec00da60 comp rx=0 tx=0).stop 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe402241640 1 -- 172.21.15.98:0/661190456 reap_dead start 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe402241640 1 -- 172.21.15.98:0/661190456 reap_dead start 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 1 -- 172.21.15.98:0/661190456 shutdown_connections 2024-06-11T19:20:03.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 1 -- 172.21.15.98:0/661190456 >> 172.21.15.98:0/661190456 conn(0x7fe3fc0a95b0 msgr2=0x7fe3fc0a6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:03.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 1 -- 172.21.15.98:0/661190456 shutdown_connections 2024-06-11T19:20:03.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 1 -- 172.21.15.98:0/661190456 wait complete. 2024-06-11T19:20:03.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.700+0000 7fe403ca4640 1 librados: shutdown 2024-06-11T19:20:03.749 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:03.749 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:20:03.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:20:03.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.849+0000 7fe501a7a640 10 monclient: get_monmap_and_config 2024-06-11T19:20:03.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.849+0000 7fe501a7a640 10 monclient: build_initial_monmap 2024-06-11T19:20:03.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.849+0000 7fe501a7a640 10 monclient: monmap: 2024-06-11T19:20:03.849 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:03.849 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.849 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:03.850818+0000 2024-06-11T19:20:03.849 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:03.850818+0000 2024-06-11T19:20:03.849 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:03.849 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.849 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:03.849 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:03.850 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:03.850 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:03.850 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:03.850 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:03.850 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:03.850 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:03.850 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:03.850 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:03.850 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:03.850 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.850+0000 7fe501a7a640 1 Processor -- start 2024-06-11T19:20:03.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.850+0000 7fe501a7a640 1 -- start start 2024-06-11T19:20:03.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.850+0000 7fe501a7a640 10 monclient: init 2024-06-11T19:20:03.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:03.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 10 monclient: _add_conns ranks=[5,10,0,8,6,9,2,3,7,4,1] 2024-06-11T19:20:03.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4fc0a2f40 0x7fe4fc0a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 10 monclient(hunting): picked mon.noname-h con 0x7fe4fc0a2f40 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:03.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe4fc0a4df0 0x7fe4fc0a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 10 monclient(hunting): picked mon.noname-f con 0x7fe4fc0a4df0 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:03.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4fc1366d0 addr v1:172.21.15.98:6789/0 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4fc0a0ed0 con 0x7fe4fc1366d0 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fe4fc0a5710 con 0x7fe4fc0a4df0 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fe4fc0a3320 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe501a7a640 10 monclient(hunting): authenticate will time out at 1413.658497s 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4fbfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe4fc1366d0 0x7fe4fc136ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55848/0 (socket says 172.21.15.98:55848) 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4fbfff640 1 -- 172.21.15.98:0/1196387493 learned_addr learned my addr 172.21.15.98:0/1196387493 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4fb7fe640 1 --2- 172.21.15.98:0/1196387493 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4fc0a2f40 0x7fe4fc0a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4f3fff640 1 --2- 172.21.15.98:0/1196387493 >> v2:172.21.15.117:3300/0 conn(0x7fe4fc0a4df0 0x7fe4fc0a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4fb7fe640 10 monclient(hunting): get_auth_request con 0x7fe4fc0a2f40 auth_method 0 2024-06-11T19:20:03.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4fb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4f3fff640 10 monclient(hunting): get_auth_request con 0x7fe4fc0a4df0 auth_method 0 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4f3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4faffd640 1 -- 172.21.15.98:0/1196387493 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2835472672 0 0) 0x7fe4fc0a0ed0 con 0x7fe4fc1366d0 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4faffd640 10 monclient(hunting): _negotiate 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4faffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4faffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4faffd640 10 monclient(hunting): my global_id is 65358 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.851+0000 7fe4faffd640 1 -- 172.21.15.98:0/1196387493 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4e8003520 con 0x7fe4fc1366d0 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.852+0000 7fe4fb7fe640 10 monclient(hunting): handle_auth_done global_id 23270 payload 274 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.852+0000 7fe4fb7fe640 1 -- 172.21.15.98:0/1196387493 >> v2:172.21.15.117:3300/0 conn(0x7fe4fc0a4df0 msgr2=0x7fe4fc0a51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.852+0000 7fe4fb7fe640 1 --2- 172.21.15.98:0/1196387493 >> v2:172.21.15.117:3300/0 conn(0x7fe4fc0a4df0 0x7fe4fc0a51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.852+0000 7fe4fb7fe640 1 -- 172.21.15.98:0/1196387493 >> v1:172.21.15.98:6789/0 conn(0x7fe4fc1366d0 legacy=0x7fe4fc136ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.852+0000 7fe4fb7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.852+0000 7fe4fb7fe640 1 monclient: found mon.noname-h 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.852+0000 7fe4fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:03.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.852+0000 7fe4fb7fe640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.852+0000 7fe4fb7fe640 1 -- 172.21.15.98:0/1196387493 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4fc0a2180 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.852+0000 7fe4f3fff640 1 --2- 172.21.15.98:0/1196387493 >> v2:172.21.15.117:3300/0 conn(0x7fe4fc0a4df0 0x7fe4fc0a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.852+0000 7fe4fb7fe640 1 --2- 172.21.15.98:0/1196387493 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4fc0a2f40 0x7fe4fc0a48b0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec0097a0 tx=0x7fe4ec02b7a0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=2735042976751c03 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 1 -- 172.21.15.98:0/1196387493 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4ec038070 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 10 monclient: dump: 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.865 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 10 monclient: _finish_auth 0 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 10 monclient: _check_auth_tickets 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 5 monclient: authenticate success, global_id 23270 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 1 -- 172.21.15.98:0/1196387493 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4ec031970 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 1 -- 172.21.15.98:0/1196387493 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4ec032390 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 4 set_mon_vals no callback set 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe4faffd640 10 monclient: dump: 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.867 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 10 monclient: shutdown 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 1 -- 172.21.15.98:0/1196387493 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4fc0a2f40 msgr2=0x7fe4fc0a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 1 --2- 172.21.15.98:0/1196387493 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4fc0a2f40 0x7fe4fc0a48b0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec0097a0 tx=0x7fe4ec02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 1 -- 172.21.15.98:0/1196387493 shutdown_connections 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 1 --2- 172.21.15.98:0/1196387493 >> v2:172.21.15.117:3300/0 conn(0x7fe4fc0a4df0 0x7fe4fc0a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 1 --2- 172.21.15.98:0/1196387493 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe4fc0a2f40 0x7fe4fc0a48b0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 1 -- 172.21.15.98:0/1196387493 >> 172.21.15.98:0/1196387493 conn(0x7fe4fc0a95b0 msgr2=0x7fe4fc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 1 -- 172.21.15.98:0/1196387493 shutdown_connections 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.853+0000 7fe501a7a640 1 -- 172.21.15.98:0/1196387493 wait complete. 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.854+0000 7fe501a7a640 10 monclient: build_initial_monmap 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.854+0000 7fe501a7a640 10 monclient: monmap: 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:03.869 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.854+0000 7fe501a7a640 1 librados: starting msgr at 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.854+0000 7fe501a7a640 1 librados: starting objecter 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.854+0000 7fe501a7a640 1 Processor -- start 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.854+0000 7fe501a7a640 1 -- start start 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.854+0000 7fe501a7a640 1 librados: setting wanted keys 2024-06-11T19:20:03.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.854+0000 7fe501a7a640 1 librados: calling monclient init 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.854+0000 7fe501a7a640 10 monclient: init 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 10 monclient: _add_conns ranks=[8,0,7,4,6,5,3,2,1] 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4fc0a2f40 0x7fe4fc1cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 10 monclient(hunting): picked mon.noname-e con 0x7fe4fc0a2f40 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0a4df0 0x7fe4fc1cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 10 monclient(hunting): picked mon.noname-a con 0x7fe4fc0a4df0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4fc1366d0 0x7fe4fc123110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 10 monclient(hunting): picked mon.noname-i con 0x7fe4fc1366d0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe4fc1371f0 con 0x7fe4fc0a4df0 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fe4fc0a2460 con 0x7fe4fc1366d0 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe4fc137370 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe501a7a640 10 monclient(hunting): authenticate will time out at 1413.662473s 2024-06-11T19:20:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4fc0a2f40 0x7fe4fc1cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4fc0a2f40 0x7fe4fc1cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:60704/0 (socket says 172.21.15.98:60704) 2024-06-11T19:20:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 1 -- 172.21.15.98:0/2560677910 learned_addr learned my addr 172.21.15.98:0/2560677910 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 10 monclient(hunting): get_auth_request con 0x7fe4fc0a2f40 auth_method 0 2024-06-11T19:20:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0a4df0 0x7fe4fc1cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4f3fff640 10 monclient(hunting): get_auth_request con 0x7fe4fc0a4df0 auth_method 0 2024-06-11T19:20:03.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4f3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fbfff640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4fc1366d0 0x7fe4fc123110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fbfff640 10 monclient(hunting): get_auth_request con 0x7fe4fc1366d0 auth_method 0 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fbfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 10 monclient(hunting): handle_auth_done global_id 94346 payload 824 2024-06-11T19:20:03.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 1 -- 172.21.15.98:0/2560677910 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4fc1366d0 msgr2=0x7fe4fc123110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4fc1366d0 0x7fe4fc123110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 1 -- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0a4df0 msgr2=0x7fe4fc1cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0a4df0 0x7fe4fc1cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:20:03.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 1 monclient: found mon.noname-e 2024-06-11T19:20:03.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:03.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4fb7fe640 1 -- 172.21.15.98:0/2560677910 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4fc126a90 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.855+0000 7fe4f3fff640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4fc0a4df0 0x7fe4fc1cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4fb7fe640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4fc0a2f40 0x7fe4fc1cb7a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec036040 tx=0x7fe4ec0046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 10 client.?.objecter ms_handle_connect 0x7fe4fc0a2f40 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4ec038050 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 10 monclient: dump: 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.875 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 10 monclient: _finish_auth 0 2024-06-11T19:20:03.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4ec044070 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe501a7a640 5 monclient: authenticate success, global_id 94346 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe501a7a640 10 monclient: _renew_subs 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe501a7a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe501a7a640 1 -- 172.21.15.98:0/2560677910 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4fc125190 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4ec004410 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4fa7fc640 4 set_mon_vals no callback set 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe501a7a640 10 client.94346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe4f8ff9640 10 monclient: dump: 2024-06-11T19:20:03.877 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.878 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe501a7a640 10 monclient: _renew_subs 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe501a7a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe501a7a640 1 -- 172.21.15.98:0/2560677910 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4fc126d70 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.856+0000 7fe501a7a640 1 librados: init done 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fe4ec003610 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe501a7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe501a7a640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123650 0x7fe4fc123a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe501a7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe501a7a640 1 -- 172.21.15.98:0/2560677910 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe4fc050830 con 0x7fe4fc123650 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f3fff640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123650 0x7fe4fc123a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f3fff640 10 monclient: get_auth_request con 0x7fe4fc123650 auth_method 0 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe4dc03fde0 0x7fe4dc0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(420..420 src has 1..420) ==== 6258+0+0 (secure 0 0 0) 0x7fe4ec073790 con 0x7fe4fc0a2f40 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 10 client.94346.objecter ms_dispatch 0x7fe4fc000b90 osd_map(420..420 src has 1..420) 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 3 client.94346.objecter handle_osd_map got epochs [420,420] > 0 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 3 client.94346.objecter handle_osd_map decoding full epoch 420 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 20 client.94346.objecter dump_active .. 0 homeless 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_done global_id 24214 payload 824 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f3fff640 1 -- 172.21.15.98:0/2560677910 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe4fc0a59a0 con 0x7fe4fc123650 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4fbfff640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe4dc03fde0 0x7fe4dc0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4fbfff640 10 monclient: get_auth_request con 0x7fe4dc03fde0 auth_method 0 2024-06-11T19:20:03.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 10 client.94346.objecter ms_handle_connect 0x7fe4fc123650 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 10 client.94346.objecter resend_mon_ops 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f3fff640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123650 0x7fe4fc123a30 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4011cf0 tx=0x7fe4e400f610 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4e4019070 con 0x7fe4fc123650 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.857+0000 7fe4f8ff9640 10 monclient: dump: 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.881 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.858+0000 7fe4f8ff9640 10 client.94346.objecter ms_handle_connect 0x7fe4dc03fde0 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.858+0000 7fe4fbfff640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe4dc03fde0 0x7fe4dc0422a0 secure :-1 s=READY pgs=4209 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc124870 tx=0x7fe4f4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.858+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe4e4004030 con 0x7fe4fc123650 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.858+0000 7fe4f8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.858+0000 7fe4f8ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.858+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123650 msgr2=0x7fe4fc123a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.858+0000 7fe4f8ff9640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123650 0x7fe4fc123a30 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4011cf0 tx=0x7fe4e400f610 comp rx=0 tx=0).stop 2024-06-11T19:20:03.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.858+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123650 msgr2=0x7fe4fc123a30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:03.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.858+0000 7fe4f8ff9640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123650 0x7fe4fc123a30 secure :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4011cf0 tx=0x7fe4e400f610 comp rx=0 tx=0).stop 2024-06-11T19:20:03.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.864+0000 7fe501a7a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:03.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.864+0000 7fe501a7a640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123f70 0x7fe4fc09cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.864+0000 7fe501a7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.864+0000 7fe501a7a640 1 -- 172.21.15.98:0/2560677910 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe4fc0692c0 con 0x7fe4fc123f70 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.864+0000 7fe4f3fff640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123f70 0x7fe4fc09cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.864+0000 7fe4f3fff640 10 monclient: get_auth_request con 0x7fe4fc123f70 auth_method 0 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.864+0000 7fe4f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.864+0000 7fe4f3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.864+0000 7fe4f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.864+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.864+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.864+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f3fff640 10 monclient(hunting): handle_auth_done global_id 24223 payload 824 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f3fff640 1 -- 172.21.15.98:0/2560677910 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe4fc0a5df0 con 0x7fe4fc123f70 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f3fff640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123f70 0x7fe4fc09cc20 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4011cf0 tx=0x7fe4e40046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:03.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 10 client.94346.objecter ms_handle_connect 0x7fe4fc123f70 2024-06-11T19:20:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 10 client.94346.objecter resend_mon_ops 2024-06-11T19:20:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4e4019040 con 0x7fe4fc123f70 2024-06-11T19:20:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 10 monclient: dump: 2024-06-11T19:20:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:03.886 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7fe4e401e660 con 0x7fe4fc123f70 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:03.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123f70 msgr2=0x7fe4fc09cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123f70 0x7fe4fc09cc20 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4011cf0 tx=0x7fe4e40046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 1 -- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123f70 msgr2=0x7fe4fc09cc20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:03.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4f8ff9640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4fc123f70 0x7fe4fc09cc20 secure :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4011cf0 tx=0x7fe4e40046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.888 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:20:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 654083, 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:20:03.889 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:03.890 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:03.891 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:03.892 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:03.893 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:03.894 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.897 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.899 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.900 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:03.911 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe501a7a640 10 librados: watch_flush enter 2024-06-11T19:20:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe501a7a640 10 librados: watch_flush exit 2024-06-11T19:20:03.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe501a7a640 20 client.94346.objecter shutdown clearing up homeless session... 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe501a7a640 10 client.94346.objecter successfully canceled tick 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe501a7a640 1 -- 172.21.15.98:0/2560677910 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe4dc03fde0 msgr2=0x7fe4dc0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe501a7a640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe4dc03fde0 0x7fe4dc0422a0 secure :-1 s=READY pgs=4209 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc124870 tx=0x7fe4f4009040 comp rx=0 tx=0).stop 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe501a7a640 10 monclient: shutdown 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe501a7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe501a7a640 1 -- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4fc0a2f40 msgr2=0x7fe4fc1cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe501a7a640 1 --2- 172.21.15.98:0/2560677910 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4fc0a2f40 0x7fe4fc1cb7a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec036040 tx=0x7fe4ec0046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.865+0000 7fe4fbfff640 1 -- 172.21.15.98:0/2560677910 reap_dead start 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.866+0000 7fe4fbfff640 1 -- 172.21.15.98:0/2560677910 reap_dead start 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.866+0000 7fe501a7a640 1 -- 172.21.15.98:0/2560677910 shutdown_connections 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.866+0000 7fe501a7a640 1 -- 172.21.15.98:0/2560677910 >> 172.21.15.98:0/2560677910 conn(0x7fe4fc0a95b0 msgr2=0x7fe4fc0a0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.866+0000 7fe501a7a640 1 -- 172.21.15.98:0/2560677910 shutdown_connections 2024-06-11T19:20:03.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.866+0000 7fe501a7a640 1 -- 172.21.15.98:0/2560677910 wait complete. 2024-06-11T19:20:03.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:03.866+0000 7fe501a7a640 1 librados: shutdown 2024-06-11T19:20:03.915 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:03.915 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:20:03.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:20:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.020+0000 7fe89ed60640 10 monclient: get_monmap_and_config 2024-06-11T19:20:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.020+0000 7fe89ed60640 10 monclient: build_initial_monmap 2024-06-11T19:20:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.021+0000 7fe89ed60640 10 monclient: monmap: 2024-06-11T19:20:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:04.022033+0000 2024-06-11T19:20:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:04.022033+0000 2024-06-11T19:20:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:04.021 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.021+0000 7fe89ed60640 1 Processor -- start 2024-06-11T19:20:04.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.021+0000 7fe89ed60640 1 -- start start 2024-06-11T19:20:04.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.021+0000 7fe89ed60640 10 monclient: init 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 10 monclient: _add_conns ranks=[5,6,3,0,7,8,9,4,2,1,10] 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe89809d330 0x7fe89809d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 10 monclient(hunting): picked mon.noname-h con 0x7fe89809d330 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 0x7fe898136770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 10 monclient(hunting): picked mon.noname-d con 0x7fe89809dc50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe8980a2850 0x7fe898136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 10 monclient(hunting): picked mon.noname-g con 0x7fe8980a2850 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fe89809bb20 con 0x7fe8980a2850 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fe89809b560 con 0x7fe89809d330 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe89809b840 con 0x7fe89809dc50 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89ed60640 10 monclient(hunting): authenticate will time out at 1413.829337s 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 0x7fe898136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 0x7fe898136770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:37840/0 (socket says 172.21.15.98:37840) 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 1 -- 172.21.15.98:0/3302162169 learned_addr learned my addr 172.21.15.98:0/3302162169 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 10 monclient(hunting): get_auth_request con 0x7fe89809dc50 auth_method 0 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89cafc640 1 --2- 172.21.15.98:0/3302162169 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe89809d330 0x7fe89809d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89cafc640 10 monclient(hunting): get_auth_request con 0x7fe89809d330 auth_method 0 2024-06-11T19:20:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89cafc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89d2fd640 1 --2- 172.21.15.98:0/3302162169 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe8980a2850 0x7fe898136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89cafc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89cafc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89d2fd640 10 monclient(hunting): get_auth_request con 0x7fe8980a2850 auth_method 0 2024-06-11T19:20:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89d2fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89d2fd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89d2fd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 10 monclient(hunting): handle_auth_done global_id 55356 payload 274 2024-06-11T19:20:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 1 -- 172.21.15.98:0/3302162169 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe89809d330 msgr2=0x7fe89809d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 1 --2- 172.21.15.98:0/3302162169 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe89809d330 0x7fe89809d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 1 -- 172.21.15.98:0/3302162169 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe8980a2850 msgr2=0x7fe898136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 1 --2- 172.21.15.98:0/3302162169 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe8980a2850 0x7fe898136cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 10 monclient: _finish_hunting 0 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 1 monclient: found mon.noname-d 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe897fff640 1 -- 172.21.15.98:0/3302162169 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8980769a0 con 0x7fe89809dc50 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.022+0000 7fe89cafc640 1 --2- 172.21.15.98:0/3302162169 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe89809d330 0x7fe89809d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe897fff640 1 --2- 172.21.15.98:0/3302162169 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 0x7fe898136770 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe884009880 tx=0x7fe88402b790 comp rx=0 tx=0).ready entity=mon.6 client_cookie=d50a83ae766fc6f5 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 1 -- 172.21.15.98:0/3302162169 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe884038070 con 0x7fe89809dc50 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 10 monclient: dump: 2024-06-11T19:20:04.035 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.036 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 10 monclient: _finish_auth 0 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 10 monclient: _check_auth_tickets 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe89ed60640 5 monclient: authenticate success, global_id 55356 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 1 -- 172.21.15.98:0/3302162169 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8840044a0 con 0x7fe89809dc50 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe89ed60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe89ed60640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe89ed60640 4 set_mon_vals no callback set 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 1 -- 172.21.15.98:0/3302162169 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe884003a80 con 0x7fe89809dc50 2024-06-11T19:20:04.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.023+0000 7fe8977fe640 10 monclient: dump: 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.038 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 10 monclient: shutdown 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 1 -- 172.21.15.98:0/3302162169 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 msgr2=0x7fe898136770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 1 --2- 172.21.15.98:0/3302162169 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 0x7fe898136770 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe884009880 tx=0x7fe88402b790 comp rx=0 tx=0).stop 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 1 -- 172.21.15.98:0/3302162169 shutdown_connections 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 1 --2- 172.21.15.98:0/3302162169 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe8980a2850 0x7fe898136cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 1 --2- 172.21.15.98:0/3302162169 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 0x7fe898136770 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 1 --2- 172.21.15.98:0/3302162169 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fe89809d330 0x7fe89809d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 1 -- 172.21.15.98:0/3302162169 >> 172.21.15.98:0/3302162169 conn(0x7fe8980a7850 msgr2=0x7fe8980a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 1 -- 172.21.15.98:0/3302162169 shutdown_connections 2024-06-11T19:20:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 1 -- 172.21.15.98:0/3302162169 wait complete. 2024-06-11T19:20:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 10 monclient: build_initial_monmap 2024-06-11T19:20:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.024+0000 7fe89ed60640 10 monclient: monmap: 2024-06-11T19:20:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 1 librados: starting msgr at 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 1 librados: starting objecter 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 1 Processor -- start 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 1 -- start start 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 1 librados: setting wanted keys 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 1 librados: calling monclient init 2024-06-11T19:20:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 10 monclient: init 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 10 monclient: _add_conns ranks=[6,2,7,0,5,1,4,8,3] 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809d330 0x7fe8981cb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 10 monclient(hunting): picked mon.noname-d con 0x7fe89809d330 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8980a2850 0x7fe8981cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 10 monclient(hunting): picked mon.noname-b con 0x7fe8980a2850 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe898132910 0x7fe898130700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 10 monclient(hunting): picked mon.noname-i con 0x7fe898132910 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe8981371f0 con 0x7fe8980a2850 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe89809e270 con 0x7fe89809d330 2024-06-11T19:20:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fe898137370 con 0x7fe898132910 2024-06-11T19:20:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89ed60640 10 monclient(hunting): authenticate will time out at 1413.832784s 2024-06-11T19:20:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89cafc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809d330 0x7fe8981cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89cafc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809d330 0x7fe8981cb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:37844/0 (socket says 172.21.15.98:37844) 2024-06-11T19:20:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89cafc640 1 -- 172.21.15.98:0/2964985146 learned_addr learned my addr 172.21.15.98:0/2964985146 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89cafc640 10 monclient(hunting): get_auth_request con 0x7fe89809d330 auth_method 0 2024-06-11T19:20:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89cafc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89cafc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe897fff640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8980a2850 0x7fe8981cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89cafc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe897fff640 10 monclient(hunting): get_auth_request con 0x7fe8980a2850 auth_method 0 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe897fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe897fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89cafc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89cafc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe89cafc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.025+0000 7fe897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89d2fd640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe898132910 0x7fe898130700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe897fff640 10 monclient(hunting): handle_auth_done global_id 84710 payload 824 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe897fff640 1 -- 172.21.15.98:0/2964985146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe898132910 msgr2=0x7fe898130700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe897fff640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe898132910 0x7fe898130700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe897fff640 1 -- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809d330 msgr2=0x7fe8981cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe897fff640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809d330 0x7fe8981cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe897fff640 10 monclient: _finish_hunting 0 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe897fff640 1 monclient: found mon.noname-b 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe897fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe897fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe897fff640 1 -- 172.21.15.98:0/2964985146 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8981338e0 con 0x7fe8980a2850 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89d2fd640 10 monclient: get_auth_request con 0x7fe898132910 auth_method 0 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89cafc640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809d330 0x7fe8981cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89d2fd640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe898132910 0x7fe898130700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe897fff640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8980a2850 0x7fe8981cbd40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe884004000 tx=0x7fe8840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 10 client.?.objecter ms_handle_connect 0x7fe8980a2850 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe884042030 con 0x7fe8980a2850 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 10 monclient: dump: 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 10 monclient: _finish_auth 0 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 10 monclient: _check_auth_tickets 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe884003e10 con 0x7fe8980a2850 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89ed60640 5 monclient: authenticate success, global_id 84710 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89ed60640 10 monclient: _renew_subs 2024-06-11T19:20:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89ed60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:20:04.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89ed60640 1 -- 172.21.15.98:0/2964985146 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8980a4900 con 0x7fe8980a2850 2024-06-11T19:20:04.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:04.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe884038040 con 0x7fe8980a2850 2024-06-11T19:20:04.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:04.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe8957fa640 10 monclient: dump: 2024-06-11T19:20:04.048 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.048 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.048 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.048 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.048 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.049 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.049 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.049 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.049 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.049 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.049 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.049 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.049 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.049 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.049 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.049 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89ed60640 10 client.84710.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:04.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89ed60640 10 monclient: _renew_subs 2024-06-11T19:20:04.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89ed60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:20:04.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe896ffd640 4 set_mon_vals no callback set 2024-06-11T19:20:04.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89ed60640 1 -- 172.21.15.98:0/2964985146 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe898133e30 con 0x7fe8980a2850 2024-06-11T19:20:04.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.026+0000 7fe89ed60640 1 librados: init done 2024-06-11T19:20:04.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89ed60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:04.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fe884045070 con 0x7fe8980a2850 2024-06-11T19:20:04.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89ed60640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 0x7fe898130c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89ed60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89ed60640 1 -- 172.21.15.98:0/2964985146 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe898050830 con 0x7fe89809dc50 2024-06-11T19:20:04.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89cafc640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 0x7fe898130c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89cafc640 10 monclient: get_auth_request con 0x7fe89809dc50 auth_method 0 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89cafc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89cafc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89cafc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe8957fa640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe87c03fdb0 0x7fe87c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89cafc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89cafc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89cafc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(421..421 src has 1..421) ==== 5845+0+0 (secure 0 0 0) 0x7fe884043070 con 0x7fe8980a2850 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe8957fa640 10 client.84710.objecter ms_dispatch 0x7fe898000b90 osd_map(421..421 src has 1..421) 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe8957fa640 3 client.84710.objecter handle_osd_map got epochs [421,421] > 0 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe8957fa640 3 client.84710.objecter handle_osd_map decoding full epoch 421 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe8957fa640 20 client.84710.objecter dump_active .. 0 homeless 2024-06-11T19:20:04.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89cafc640 10 monclient(hunting): handle_auth_done global_id 55374 payload 824 2024-06-11T19:20:04.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89cafc640 1 -- 172.21.15.98:0/2964985146 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe89809d710 con 0x7fe89809dc50 2024-06-11T19:20:04.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe8957fa640 10 client.84710.objecter ms_handle_connect 0x7fe89809dc50 2024-06-11T19:20:04.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe8957fa640 10 client.84710.objecter resend_mon_ops 2024-06-11T19:20:04.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.027+0000 7fe89cafc640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 0x7fe898130c40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009fd0 tx=0x7fe88c011f10 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe89d2fd640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe87c03fdb0 0x7fe87c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe89d2fd640 10 monclient: get_auth_request con 0x7fe87c03fdb0 auth_method 0 2024-06-11T19:20:04.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe88c00f040 con 0x7fe89809dc50 2024-06-11T19:20:04.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe8957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe8957fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:04.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe8957fa640 10 monclient: dump: 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.053 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.054 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.054 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.054 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe88c00fc60 con 0x7fe89809dc50 2024-06-11T19:20:04.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe8957fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:04.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe8957fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:04.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 msgr2=0x7fe898130c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe8957fa640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 0x7fe898130c40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009fd0 tx=0x7fe88c011f10 comp rx=0 tx=0).stop 2024-06-11T19:20:04.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 msgr2=0x7fe898130c40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:04.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe8957fa640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe89809dc50 0x7fe898130c40 secure :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009fd0 tx=0x7fe88c011f10 comp rx=0 tx=0).stop 2024-06-11T19:20:04.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe8957fa640 10 client.84710.objecter ms_handle_connect 0x7fe87c03fdb0 2024-06-11T19:20:04.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.028+0000 7fe89d2fd640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe87c03fdb0 0x7fe87c042270 secure :-1 s=READY pgs=4210 cs=0 l=1 rev1=1 crypto rx=0x7fe8980a45d0 tx=0x7fe89000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.034+0000 7fe89ed60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:04.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.034+0000 7fe89ed60640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe898131180 0x7fe898131590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.034+0000 7fe89ed60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.034+0000 7fe89ed60640 1 -- 172.21.15.98:0/2964985146 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe898131ad0 con 0x7fe898131180 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.034+0000 7fe89cafc640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe898131180 0x7fe898131590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.034+0000 7fe89cafc640 10 monclient: get_auth_request con 0x7fe898131180 auth_method 0 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.034+0000 7fe89cafc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.034+0000 7fe89cafc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.034+0000 7fe89cafc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe89cafc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe89cafc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe89cafc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe89cafc640 10 monclient(hunting): handle_auth_done global_id 55383 payload 824 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe89cafc640 1 -- 172.21.15.98:0/2964985146 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe8980008d0 con 0x7fe898131180 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe89cafc640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe898131180 0x7fe898131590 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009fd0 tx=0x7fe88c004730 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 10 client.84710.objecter ms_handle_connect 0x7fe898131180 2024-06-11T19:20:04.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 10 client.84710.objecter resend_mon_ops 2024-06-11T19:20:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe88c0158d0 con 0x7fe898131180 2024-06-11T19:20:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 10 monclient: dump: 2024-06-11T19:20:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.058 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fe88c015460 con 0x7fe898131180 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe898131180 msgr2=0x7fe898131590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe898131180 0x7fe898131590 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009fd0 tx=0x7fe88c004730 comp rx=0 tx=0).stop 2024-06-11T19:20:04.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 1 -- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe898131180 msgr2=0x7fe898131590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:04.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe8957fa640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe898131180 0x7fe898131590 secure :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009fd0 tx=0x7fe88c004730 comp rx=0 tx=0).stop 2024-06-11T19:20:04.060 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:04.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:04.060 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:04.060 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:04.060 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:04.060 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 516792, 2024-06-11T19:20:04.060 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:04.060 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:04.060 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:04.061 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:04.061 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:04.061 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:04.061 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:04.061 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:04.061 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:04.061 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:04.061 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.061 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:20:04.061 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:04.061 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:04.061 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.062 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.063 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:04.064 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:04.067 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:04.069 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.069 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.069 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:04.069 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:04.069 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:04.070 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.071 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:04.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:04.076 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.078 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:04.081 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.082 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:20:04.083 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.084 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.084 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:04.084 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe89ed60640 10 librados: watch_flush enter 2024-06-11T19:20:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe89ed60640 10 librados: watch_flush exit 2024-06-11T19:20:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe89ed60640 20 client.84710.objecter shutdown clearing up homeless session... 2024-06-11T19:20:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.035+0000 7fe89ed60640 10 client.84710.objecter successfully canceled tick 2024-06-11T19:20:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89ed60640 1 -- 172.21.15.98:0/2964985146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe87c03fdb0 msgr2=0x7fe87c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89ed60640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe87c03fdb0 0x7fe87c042270 secure :-1 s=READY pgs=4210 cs=0 l=1 rev1=1 crypto rx=0x7fe8980a45d0 tx=0x7fe89000b040 comp rx=0 tx=0).stop 2024-06-11T19:20:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89ed60640 10 monclient: shutdown 2024-06-11T19:20:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89ed60640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89ed60640 1 -- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8980a2850 msgr2=0x7fe8981cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89d2fd640 1 -- 172.21.15.98:0/2964985146 reap_dead start 2024-06-11T19:20:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89ed60640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8980a2850 0x7fe8981cbd40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe884004000 tx=0x7fe8840046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89ed60640 1 -- 172.21.15.98:0/2964985146 shutdown_connections 2024-06-11T19:20:04.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89ed60640 1 --2- 172.21.15.98:0/2964985146 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe8980a2850 0x7fe8981cbd40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89ed60640 1 -- 172.21.15.98:0/2964985146 >> 172.21.15.98:0/2964985146 conn(0x7fe8980a7850 msgr2=0x7fe898072ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:04.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89ed60640 1 -- 172.21.15.98:0/2964985146 shutdown_connections 2024-06-11T19:20:04.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89ed60640 1 -- 172.21.15.98:0/2964985146 wait complete. 2024-06-11T19:20:04.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.036+0000 7fe89ed60640 1 librados: shutdown 2024-06-11T19:20:04.087 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:04.087 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:20:04.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:20:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.188+0000 7fc11aea4640 10 monclient: get_monmap_and_config 2024-06-11T19:20:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.188+0000 7fc11aea4640 10 monclient: build_initial_monmap 2024-06-11T19:20:04.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.188+0000 7fc11aea4640 10 monclient: monmap: 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:04.189456+0000 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:04.189456+0000 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:04.188 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:04.189 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:04.189 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 1 Processor -- start 2024-06-11T19:20:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 1 -- start start 2024-06-11T19:20:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 10 monclient: init 2024-06-11T19:20:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 10 monclient: _add_conns ranks=[5,10,0,8,2,3,4,7,9,6,1] 2024-06-11T19:20:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc1140a4920 0x7fc1140a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 10 monclient(hunting): picked mon.noname-h con 0x7fc1140a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc1140a5240 0x7fc1140a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 10 monclient(hunting): picked mon.noname-f con 0x7fc1140a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc1140a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1140a7140 con 0x7fc1140a2530 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fc1140a2b50 con 0x7fc1140a5240 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc1140a2e30 con 0x7fc1140a4920 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc11aea4640 10 monclient(hunting): authenticate will time out at 1413.996717s 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc119441640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc1140a2530 0x7fc1140a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55864/0 (socket says 172.21.15.98:55864) 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.189+0000 7fc119441640 1 -- 172.21.15.98:0/224947042 learned_addr learned my addr 172.21.15.98:0/224947042 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 1 --2- 172.21.15.98:0/224947042 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc1140a4920 0x7fc1140a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 1 --2- 172.21.15.98:0/224947042 >> v2:172.21.15.117:3300/0 conn(0x7fc1140a5240 0x7fc1140a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3699356909 0 0) 0x7fc1140a7140 con 0x7fc1140a2530 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient(hunting): _negotiate 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient(hunting): my global_id is 65376 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0fc003520 con 0x7fc1140a2530 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): get_auth_request con 0x7fc1140a5240 auth_method 0 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): get_auth_request con 0x7fc1140a4920 auth_method 0 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2743438846 0 0) 0x7fc0fc003520 con 0x7fc1140a2530 2024-06-11T19:20:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc1140a4920 msgr2=0x7fc1140a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 --2- 172.21.15.98:0/224947042 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc1140a4920 0x7fc1140a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 >> v2:172.21.15.117:3300/0 conn(0x7fc1140a5240 msgr2=0x7fc1140a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 --2- 172.21.15.98:0/224947042 >> v2:172.21.15.117:3300/0 conn(0x7fc1140a5240 0x7fc1140a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:20:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:20:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:20:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1140a3110 con 0x7fc1140a2530 2024-06-11T19:20:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:20:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:20:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7fc10c001960 con 0x7fc1140a2530 2024-06-11T19:20:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc11aea4640 5 monclient: authenticate success, global_id 65376 2024-06-11T19:20:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc11aea4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient: dump: 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 >> v1:172.21.15.98:6789/0 conn(0x7fc1140a2530 legacy=0x7fc1140a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient: _add_conns ranks=[1,5,0,7,3,4,8,6,2] 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 --2- 172.21.15.98:0/224947042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0fc004200 0x7fc0fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient(hunting): picked mon.f con 0x7fc0fc004200 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 --2- 172.21.15.98:0/224947042 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc1140a4920 0x7fc0fc004df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient(hunting): picked mon.h con 0x7fc1140a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 --2- 172.21.15.98:0/224947042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0fc0053a0 0x7fc0fc00b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient(hunting): picked mon.a con 0x7fc0fc0053a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:20:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc1140a2b50 con 0x7fc0fc0053a0 2024-06-11T19:20:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc1140a2e30 con 0x7fc0fc004200 2024-06-11T19:20:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc0fc00c260 con 0x7fc1140a4920 2024-06-11T19:20:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10b7fe640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:20:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc11aea4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 1 --2- 172.21.15.98:0/224947042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0fc0053a0 0x7fc0fc00b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 10 monclient(hunting): get_auth_request con 0x7fc0fc0053a0 auth_method 0 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 1 --2- 172.21.15.98:0/224947042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0fc004200 0x7fc0fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): get_auth_request con 0x7fc0fc004200 auth_method 0 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 1 --2- 172.21.15.98:0/224947042 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc1140a4920 0x7fc0fc004df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): get_auth_request con 0x7fc1140a4920 auth_method 0 2024-06-11T19:20:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 10 monclient(hunting): handle_auth_done global_id 65376 payload 293 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 1 -- 172.21.15.98:0/224947042 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc1140a4920 msgr2=0x7fc0fc004df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 1 --2- 172.21.15.98:0/224947042 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc1140a4920 0x7fc0fc004df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 1 -- 172.21.15.98:0/224947042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0fc004200 msgr2=0x7fc0fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 1 --2- 172.21.15.98:0/224947042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0fc004200 0x7fc0fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 10 monclient: _finish_hunting 0 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 1 monclient: found mon.a 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 1 -- 172.21.15.98:0/224947042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0fc003520 con 0x7fc0fc0053a0 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 10 monclient: _finish_auth 0 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 10 monclient: _check_auth_tickets 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.190+0000 7fc119441640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc119441640 1 --2- 172.21.15.98:0/224947042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0fc0053a0 0x7fc0fc00b9e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc10c007f30 tx=0x7fc0fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de983f11404ba460 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc10c007b80 con 0x7fc0fc0053a0 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc10b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc10b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc10b7fe640 10 monclient: dump: 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc11aea4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc10c030070 con 0x7fc0fc0053a0 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc10b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc10b7fe640 1 -- 172.21.15.98:0/224947042 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc10c005d40 con 0x7fc0fc0053a0 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc11aea4640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc11aea4640 4 set_mon_vals no callback set 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc10b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc10b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.191+0000 7fc10b7fe640 10 monclient: dump: 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.192+0000 7fc11aea4640 10 monclient: shutdown 2024-06-11T19:20:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.192+0000 7fc11aea4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.192+0000 7fc11aea4640 1 -- 172.21.15.98:0/224947042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0fc0053a0 msgr2=0x7fc0fc00b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.192+0000 7fc11aea4640 1 --2- 172.21.15.98:0/224947042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0fc0053a0 0x7fc0fc00b9e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc10c007f30 tx=0x7fc0fc005830 comp rx=0 tx=0).stop 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.192+0000 7fc119441640 1 -- 172.21.15.98:0/224947042 reap_dead start 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.192+0000 7fc11aea4640 1 -- 172.21.15.98:0/224947042 shutdown_connections 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.192+0000 7fc11aea4640 1 -- 172.21.15.98:0/224947042 >> 172.21.15.98:0/224947042 conn(0x7fc1140ab850 msgr2=0x7fc1140a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.192+0000 7fc11aea4640 1 -- 172.21.15.98:0/224947042 shutdown_connections 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.192+0000 7fc11aea4640 1 -- 172.21.15.98:0/224947042 wait complete. 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.192+0000 7fc11aea4640 10 monclient: build_initial_monmap 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.192+0000 7fc11aea4640 10 monclient: monmap: 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 1 librados: starting msgr at 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 1 librados: starting objecter 2024-06-11T19:20:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 1 Processor -- start 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 1 -- start start 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 1 librados: setting wanted keys 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 1 librados: calling monclient init 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 10 monclient: init 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 10 monclient: _add_conns ranks=[5,7,4,1,6,0,3,8,2] 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc0fc0053a0 0x7fc1141cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 10 monclient(hunting): picked mon.noname-h con 0x7fc0fc0053a0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc114137bd0 0x7fc1141216a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 10 monclient(hunting): picked mon.noname-i con 0x7fc114137bd0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc114137ff0 0x7fc114121be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 10 monclient(hunting): picked mon.noname-c con 0x7fc114137ff0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:04.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc1140a6da0 con 0x7fc114137ff0 2024-06-11T19:20:04.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc114138e50 con 0x7fc0fc0053a0 2024-06-11T19:20:04.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fc1140a65f0 con 0x7fc114137bd0 2024-06-11T19:20:04.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:04.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.193+0000 7fc11aea4640 10 monclient(hunting): authenticate will time out at 1414.000865s 2024-06-11T19:20:04.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc114137ff0 0x7fc114121be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc118c40640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc0fc0053a0 0x7fc1141cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc10bfff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc114137bd0 0x7fc1141216a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc114137ff0 0x7fc114121be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55316/0 (socket says 172.21.15.98:55316) 2024-06-11T19:20:04.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 1 -- 172.21.15.98:0/2940341189 learned_addr learned my addr 172.21.15.98:0/2940341189 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc10bfff640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc114137bd0 0x7fc1141216a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:48344/0 (socket says 172.21.15.98:48344) 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 10 monclient(hunting): get_auth_request con 0x7fc114137ff0 auth_method 0 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc118c40640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc0fc0053a0 0x7fc1141cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:43312/0 (socket says 172.21.15.98:43312) 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc10bfff640 10 monclient(hunting): get_auth_request con 0x7fc114137bd0 auth_method 0 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc10bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc10bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc10bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc118c40640 10 monclient(hunting): get_auth_request con 0x7fc0fc0053a0 auth_method 0 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc118c40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc118c40640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc118c40640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 10 monclient(hunting): handle_auth_done global_id 24232 payload 824 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 1 -- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc114137bd0 msgr2=0x7fc1141216a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc114137bd0 0x7fc1141216a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 1 -- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc0fc0053a0 msgr2=0x7fc1141cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc0fc0053a0 0x7fc1141cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 10 monclient: _finish_hunting 0 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 1 monclient: found mon.noname-c 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 1 -- 172.21.15.98:0/2940341189 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc114124fd0 con 0x7fc114137ff0 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc10bfff640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fc114137bd0 0x7fc1141216a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc118c40640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc0fc0053a0 0x7fc1141cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc119441640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc114137ff0 0x7fc114121be0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fc10c02c2f0 tx=0x7fc10c001cd0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 10 client.?.objecter ms_handle_connect 0x7fc114137ff0 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc10c007b80 con 0x7fc114137ff0 2024-06-11T19:20:04.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 10 monclient: dump: 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.224 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 10 monclient: _finish_auth 0 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 10 monclient: _check_auth_tickets 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc10c001960 con 0x7fc114137ff0 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc11aea4640 5 monclient: authenticate success, global_id 24232 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc11aea4640 10 monclient: _renew_subs 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc11aea4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:04.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc11aea4640 1 -- 172.21.15.98:0/2940341189 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1141252b0 con 0x7fc114137ff0 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc10c007530 con 0x7fc114137ff0 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc1097fa640 10 monclient: dump: 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.226 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc10affd640 4 set_mon_vals no callback set 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc11aea4640 10 client.24232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc11aea4640 10 monclient: _renew_subs 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc11aea4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc11aea4640 1 -- 172.21.15.98:0/2940341189 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc114125590 con 0x7fc114137ff0 2024-06-11T19:20:04.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.194+0000 7fc11aea4640 1 librados: init done 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc11aea4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc11aea4640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc114122120 0x7fc11409d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc11aea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc11aea4640 1 -- 172.21.15.98:0/2940341189 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc114050830 con 0x7fc114122120 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc118c40640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc114122120 0x7fc11409d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc118c40640 10 monclient: get_auth_request con 0x7fc114122120 auth_method 0 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc118c40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc118c40640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc118c40640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc118c40640 10 monclient(hunting): handle_auth_done global_id 94355 payload 824 2024-06-11T19:20:04.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc118c40640 1 -- 172.21.15.98:0/2940341189 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc114124cf0 con 0x7fc114122120 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.195+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fc10c02c680 con 0x7fc114137ff0 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc118c40640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc114122120 0x7fc11409d0a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc110014580 tx=0x7fc110014a40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc10003fdb0 0x7fc100042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 10 client.24232.objecter ms_handle_connect 0x7fc114122120 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 10 client.24232.objecter resend_mon_ops 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(421..421 src has 1..421) ==== 5845+0+0 (secure 0 0 0) 0x7fc10c072900 con 0x7fc114137ff0 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 10 client.24232.objecter ms_dispatch 0x7fc114000b90 osd_map(421..421 src has 1..421) 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 3 client.24232.objecter handle_osd_map got epochs [421,421] > 0 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 3 client.24232.objecter handle_osd_map decoding full epoch 421 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 20 client.24232.objecter dump_active .. 0 homeless 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc110016c50 con 0x7fc114122120 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 10 monclient: dump: 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.230 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.231 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.231 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.231 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.231 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc10bfff640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc10003fdb0 0x7fc100042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc10bfff640 10 monclient: get_auth_request con 0x7fc10003fdb0 auth_method 0 2024-06-11T19:20:04.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc1100167e0 con 0x7fc114122120 2024-06-11T19:20:04.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:04.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:04.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc114122120 msgr2=0x7fc11409d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc114122120 0x7fc11409d0a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc110014580 tx=0x7fc110014a40 comp rx=0 tx=0).stop 2024-06-11T19:20:04.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc114122120 msgr2=0x7fc11409d0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:04.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc114122120 0x7fc11409d0a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc1097fa640 10 client.24232.objecter ms_handle_connect 0x7fc10003fdb0 2024-06-11T19:20:04.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.196+0000 7fc10bfff640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc10003fdb0 0x7fc100042270 secure :-1 s=READY pgs=4211 cs=0 l=1 rev1=1 crypto rx=0x7fc1141215f0 tx=0x7fc104009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.202+0000 7fc11aea4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:04.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.202+0000 7fc11aea4640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc11409d5e0 0x7fc11409d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.202+0000 7fc11aea4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.202+0000 7fc11aea4640 1 -- 172.21.15.98:0/2940341189 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc114122500 con 0x7fc11409d5e0 2024-06-11T19:20:04.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.202+0000 7fc118c40640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc11409d5e0 0x7fc11409d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.202+0000 7fc118c40640 10 monclient: get_auth_request con 0x7fc11409d5e0 auth_method 0 2024-06-11T19:20:04.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.202+0000 7fc118c40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.202+0000 7fc118c40640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.202+0000 7fc118c40640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc118c40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc118c40640 10 monclient(hunting): handle_auth_done global_id 94364 payload 824 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc118c40640 1 -- 172.21.15.98:0/2940341189 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc114138540 con 0x7fc11409d5e0 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc118c40640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc11409d5e0 0x7fc11409d9f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc11001f000 tx=0x7fc110004ab0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 10 client.24232.objecter ms_handle_connect 0x7fc11409d5e0 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 10 client.24232.objecter resend_mon_ops 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc11001ed70 con 0x7fc11409d5e0 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 10 monclient: dump: 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.233 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.234 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.234 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.234 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.234 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.234 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.234 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.234 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.234 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.234 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.234 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.234 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.234 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.235 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.235 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fc11001ed70 con 0x7fc11409d5e0 2024-06-11T19:20:04.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:04.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:04.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc11409d5e0 msgr2=0x7fc11409d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc11409d5e0 0x7fc11409d9f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc11001f000 tx=0x7fc110004ab0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 1 -- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc11409d5e0 msgr2=0x7fc11409d9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:04.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc1097fa640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc11409d5e0 0x7fc11409d9f0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc11001f000 tx=0x7fc110004ab0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.235 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:04.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:04.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:04.236 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:04.236 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:04.236 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 149757, 2024-06-11T19:20:04.236 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:04.236 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:04.236 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:04.236 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:04.236 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:04.236 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:04.236 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.237 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.238 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:04.239 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:04.240 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.241 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:04.243 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.246 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:04.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.251 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:04.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.254 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.256 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:04.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.258 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.258 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.258 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:04.258 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.258 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.258 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:04.258 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:04.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc11aea4640 10 librados: watch_flush enter 2024-06-11T19:20:04.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.203+0000 7fc11aea4640 10 librados: watch_flush exit 2024-06-11T19:20:04.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 20 client.24232.objecter shutdown clearing up homeless session... 2024-06-11T19:20:04.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 10 client.24232.objecter successfully canceled tick 2024-06-11T19:20:04.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 1 -- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc10003fdb0 msgr2=0x7fc100042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc10003fdb0 0x7fc100042270 secure :-1 s=READY pgs=4211 cs=0 l=1 rev1=1 crypto rx=0x7fc1141215f0 tx=0x7fc104009190 comp rx=0 tx=0).stop 2024-06-11T19:20:04.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 10 monclient: shutdown 2024-06-11T19:20:04.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:04.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 1 -- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc114137ff0 msgr2=0x7fc114121be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc119441640 1 -- 172.21.15.98:0/2940341189 reap_dead start 2024-06-11T19:20:04.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc114137ff0 0x7fc114121be0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fc10c02c2f0 tx=0x7fc10c001cd0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 1 -- 172.21.15.98:0/2940341189 shutdown_connections 2024-06-11T19:20:04.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 1 --2- 172.21.15.98:0/2940341189 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc114137ff0 0x7fc114121be0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 1 -- 172.21.15.98:0/2940341189 >> 172.21.15.98:0/2940341189 conn(0x7fc1140ab850 msgr2=0x7fc1140726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:04.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 1 -- 172.21.15.98:0/2940341189 shutdown_connections 2024-06-11T19:20:04.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 1 -- 172.21.15.98:0/2940341189 wait complete. 2024-06-11T19:20:04.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.204+0000 7fc11aea4640 1 librados: shutdown 2024-06-11T19:20:04.262 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:04.262 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:20:04.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:20:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.361+0000 7f854d50b640 10 monclient: get_monmap_and_config 2024-06-11T19:20:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.361+0000 7f854d50b640 10 monclient: build_initial_monmap 2024-06-11T19:20:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.361+0000 7f854d50b640 10 monclient: monmap: 2024-06-11T19:20:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:04.362833+0000 2024-06-11T19:20:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:04.362833+0000 2024-06-11T19:20:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:04.362 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 1 Processor -- start 2024-06-11T19:20:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 1 -- start start 2024-06-11T19:20:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 10 monclient: init 2024-06-11T19:20:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 10 monclient: _add_conns ranks=[5,8,7,1,2,4,9,0,6,3,10] 2024-06-11T19:20:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f85480a4920 0x7f85480a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 10 monclient(hunting): picked mon.noname-h con 0x7f85480a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a5240 0x7f85480a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 10 monclient(hunting): picked mon.noname-e con 0x7f85480a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85480a2530 0x7f85480a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 10 monclient(hunting): picked mon.noname-i con 0x7f85480a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f85480a7140 con 0x7f85480a4920 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f85480a2b50 con 0x7f85480a2530 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f85480a2e30 con 0x7f85480a5240 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.362+0000 7f854d50b640 10 monclient(hunting): authenticate will time out at 1414.169947s 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a5240 0x7f85480a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a5240 0x7f85480a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:60728/0 (socket says 172.21.15.98:60728) 2024-06-11T19:20:04.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 1 -- 172.21.15.98:0/589507342 learned_addr learned my addr 172.21.15.98:0/589507342 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 10 monclient(hunting): get_auth_request con 0x7f85480a5240 auth_method 0 2024-06-11T19:20:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85477fe640 1 --2- 172.21.15.98:0/589507342 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85480a2530 0x7f85480a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f8546ffd640 1 --2- 172.21.15.98:0/589507342 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f85480a4920 0x7f85480a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85477fe640 10 monclient(hunting): get_auth_request con 0x7f85480a2530 auth_method 0 2024-06-11T19:20:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85477fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85477fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f8546ffd640 10 monclient(hunting): get_auth_request con 0x7f85480a4920 auth_method 0 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f8546ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f8546ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f8546ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f8546ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f8546ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f8546ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 10 monclient(hunting): handle_auth_done global_id 94373 payload 274 2024-06-11T19:20:04.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 1 -- 172.21.15.98:0/589507342 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85480a2530 msgr2=0x7f85480a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 1 --2- 172.21.15.98:0/589507342 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85480a2530 0x7f85480a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 1 -- 172.21.15.98:0/589507342 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f85480a4920 msgr2=0x7f85480a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 1 --2- 172.21.15.98:0/589507342 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f85480a4920 0x7f85480a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 10 monclient: _finish_hunting 0 2024-06-11T19:20:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 1 monclient: found mon.noname-e 2024-06-11T19:20:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:20:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.363+0000 7f85467fc640 1 -- 172.21.15.98:0/589507342 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85480a3110 con 0x7f85480a5240 2024-06-11T19:20:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f85467fc640 1 --2- 172.21.15.98:0/589507342 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a5240 0x7f85480a59a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f85300098a0 tx=0x7f853002b7a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=75eed8bacff41845 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 1 -- 172.21.15.98:0/589507342 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8530038070 con 0x7f85480a5240 2024-06-11T19:20:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 10 monclient: dump: 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.377 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 10 monclient: _finish_auth 0 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 1 -- 172.21.15.98:0/589507342 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8530031970 con 0x7f85480a5240 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 5 monclient: authenticate success, global_id 94373 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 1 -- 172.21.15.98:0/589507342 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8530031ba0 con 0x7f85480a5240 2024-06-11T19:20:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 4 set_mon_vals no callback set 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f8545ffb640 10 monclient: dump: 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.379 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 10 monclient: shutdown 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 1 -- 172.21.15.98:0/589507342 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a5240 msgr2=0x7f85480a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 1 --2- 172.21.15.98:0/589507342 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a5240 0x7f85480a59a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f85300098a0 tx=0x7f853002b7a0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 1 -- 172.21.15.98:0/589507342 shutdown_connections 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 1 --2- 172.21.15.98:0/589507342 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f85480a2530 0x7f85480a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 1 --2- 172.21.15.98:0/589507342 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a5240 0x7f85480a59a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 1 --2- 172.21.15.98:0/589507342 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f85480a4920 0x7f85480a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 1 -- 172.21.15.98:0/589507342 >> 172.21.15.98:0/589507342 conn(0x7f85480ab850 msgr2=0x7f85480a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 1 -- 172.21.15.98:0/589507342 shutdown_connections 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.364+0000 7f854d50b640 1 -- 172.21.15.98:0/589507342 wait complete. 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.365+0000 7f854d50b640 10 monclient: build_initial_monmap 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.365+0000 7f854d50b640 10 monclient: monmap: 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:04.381 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:04.382 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:04.382 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:04.382 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:04.382 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:04.382 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:04.382 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:04.382 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.365+0000 7f854d50b640 1 librados: starting msgr at 2024-06-11T19:20:04.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.365+0000 7f854d50b640 1 librados: starting objecter 2024-06-11T19:20:04.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.365+0000 7f854d50b640 1 Processor -- start 2024-06-11T19:20:04.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.365+0000 7f854d50b640 1 -- start start 2024-06-11T19:20:04.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.365+0000 7f854d50b640 1 librados: setting wanted keys 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.365+0000 7f854d50b640 1 librados: calling monclient init 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.365+0000 7f854d50b640 10 monclient: init 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 10 monclient: _add_conns ranks=[8,6,1,3,0,5,4,7,2] 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a2530 0x7f85481cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 10 monclient(hunting): picked mon.noname-e con 0x7f85480a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f85480a4920 0x7f85481ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 10 monclient(hunting): picked mon.noname-d con 0x7f85480a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85480a5240 0x7f8548124fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 10 monclient(hunting): picked mon.noname-f con 0x7f85480a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f85480a6da0 con 0x7f85480a5240 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8548138e50 con 0x7f85480a4920 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f85480a6420 con 0x7f85480a2530 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f854d50b640 10 monclient(hunting): authenticate will time out at 1414.173369s 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a2530 0x7f85481cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a2530 0x7f85481cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:60734/0 (socket says 172.21.15.98:60734) 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f85467fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f85480a4920 0x7f85481ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 1 -- 172.21.15.98:0/1239177525 learned_addr learned my addr 172.21.15.98:0/1239177525 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 10 monclient(hunting): get_auth_request con 0x7f85480a2530 auth_method 0 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f85467fc640 10 monclient(hunting): get_auth_request con 0x7f85480a4920 auth_method 0 2024-06-11T19:20:04.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f85467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f85467fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f85467fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f85467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f85467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f85467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 10 monclient(hunting): handle_auth_done global_id 94382 payload 824 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 1 -- 172.21.15.98:0/1239177525 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f85480a4920 msgr2=0x7f85481ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f85480a4920 0x7f85481ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f85477fe640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85480a5240 0x7f8548124fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 1 -- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85480a5240 msgr2=0x7f8548124fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85480a5240 0x7f8548124fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:20:04.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 1 monclient: found mon.noname-e 2024-06-11T19:20:04.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:04.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:20:04.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 1 -- 172.21.15.98:0/1239177525 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8548127770 con 0x7f85480a2530 2024-06-11T19:20:04.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f8546ffd640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a2530 0x7f85481cdad0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8538009db0 tx=0x7f853800c490 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f853f7fe640 10 client.?.objecter ms_handle_connect 0x7f85480a2530 2024-06-11T19:20:04.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.366+0000 7f853f7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:04.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8538018020 con 0x7f85480a2530 2024-06-11T19:20:04.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 10 monclient: dump: 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.387 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853800cdb0 con 0x7f85480a2530 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8538014d60 con 0x7f85480a2530 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 5 monclient: authenticate success, global_id 94382 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 10 monclient: _renew_subs 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:04.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 1 -- 172.21.15.98:0/1239177525 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8548128f00 con 0x7f85480a2530 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f85457fa640 4 set_mon_vals no callback set 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 10 monclient: dump: 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.389 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 10 client.94382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 10 monclient: _renew_subs 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 1 -- 172.21.15.98:0/1239177525 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f85481387f0 con 0x7f85480a2530 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 1 librados: init done 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:04.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548128860 0x7f8548125520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f854d50b640 1 -- 172.21.15.98:0/1239177525 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8548128330 con 0x7f8548128860 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.367+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8538005d10 con 0x7f85480a2530 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85467fc640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548128860 0x7f8548125520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85467fc640 10 monclient: get_auth_request con 0x7f8548128860 auth_method 0 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85467fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85467fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f852803fdb0 0x7f8528042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(421..421 src has 1..421) ==== 5845+0+0 (secure 0 0 0) 0x7f85380520a0 con 0x7f85480a2530 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 10 client.94382.objecter ms_dispatch 0x7f8548000b90 osd_map(421..421 src has 1..421) 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 3 client.94382.objecter handle_osd_map got epochs [421,421] > 0 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 3 client.94382.objecter handle_osd_map decoding full epoch 421 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 20 client.94382.objecter dump_active .. 0 homeless 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85467fc640 10 monclient(hunting): handle_auth_done global_id 32473 payload 824 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85467fc640 1 -- 172.21.15.98:0/1239177525 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f85480a66f0 con 0x7f8548128860 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85477fe640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f852803fdb0 0x7f8528042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85477fe640 10 monclient: get_auth_request con 0x7f852803fdb0 auth_method 0 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 10 client.94382.objecter ms_handle_connect 0x7f8548128860 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 10 client.94382.objecter resend_mon_ops 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85467fc640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548128860 0x7f8548125520 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f8530009870 tx=0x7f85300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8530038040 con 0x7f8548128860 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:20:04.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 10 monclient: dump: 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.394 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f853f7fe640 10 client.94382.objecter ms_handle_connect 0x7f852803fdb0 2024-06-11T19:20:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.368+0000 7f85477fe640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f852803fdb0 0x7f8528042270 secure :-1 s=READY pgs=4212 cs=0 l=1 rev1=1 crypto rx=0x7f8548127d90 tx=0x7f8540008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.369+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8530039040 con 0x7f8548128860 2024-06-11T19:20:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.369+0000 7f853f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.369+0000 7f853f7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.369+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548128860 msgr2=0x7f8548125520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.369+0000 7f853f7fe640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548128860 0x7f8548125520 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f8530009870 tx=0x7f85300046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.369+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548128860 msgr2=0x7f8548125520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.369+0000 7f853f7fe640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548128860 0x7f8548125520 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.375+0000 7f854d50b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.375+0000 7f854d50b640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548125a60 0x7f8548125e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.375+0000 7f854d50b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.375+0000 7f854d50b640 1 -- 172.21.15.98:0/1239177525 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f85480a6140 con 0x7f8548125a60 2024-06-11T19:20:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.375+0000 7f85467fc640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548125a60 0x7f8548125e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.375+0000 7f85467fc640 10 monclient: get_auth_request con 0x7f8548125a60 auth_method 0 2024-06-11T19:20:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.375+0000 7f85467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.375+0000 7f85467fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.375+0000 7f85467fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.375+0000 7f85467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.375+0000 7f85467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.375+0000 7f85467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.376+0000 7f85467fc640 10 monclient(hunting): handle_auth_done global_id 32482 payload 824 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.376+0000 7f85467fc640 1 -- 172.21.15.98:0/1239177525 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f85481263b0 con 0x7f8548125a60 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.376+0000 7f85467fc640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548125a60 0x7f8548125e70 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f853002b7e0 tx=0x7f8530031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.376+0000 7f853f7fe640 10 client.94382.objecter ms_handle_connect 0x7f8548125a60 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.376+0000 7f853f7fe640 10 client.94382.objecter resend_mon_ops 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.376+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f853004d070 con 0x7f8548125a60 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.376+0000 7f853f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.376+0000 7f853f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.376+0000 7f853f7fe640 10 monclient: dump: 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.399 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f8530049a00 con 0x7f8548125a60 2024-06-11T19:20:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f853f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f853f7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548125a60 msgr2=0x7f8548125e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f853f7fe640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548125a60 0x7f8548125e70 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f853002b7e0 tx=0x7f8530031000 comp rx=0 tx=0).stop 2024-06-11T19:20:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f853f7fe640 1 -- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548125a60 msgr2=0x7f8548125e70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f853f7fe640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8548125a60 0x7f8548125e70 secure :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f853002b7e0 tx=0x7f8530031000 comp rx=0 tx=0).stop 2024-06-11T19:20:04.399 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:04.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:04.399 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 654594, 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:20:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:04.423 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:04.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 10 librados: watch_flush enter 2024-06-11T19:20:04.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 10 librados: watch_flush exit 2024-06-11T19:20:04.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 20 client.94382.objecter shutdown clearing up homeless session... 2024-06-11T19:20:04.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 10 client.94382.objecter successfully canceled tick 2024-06-11T19:20:04.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 1 -- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f852803fdb0 msgr2=0x7f8528042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f852803fdb0 0x7f8528042270 secure :-1 s=READY pgs=4212 cs=0 l=1 rev1=1 crypto rx=0x7f8548127d90 tx=0x7f8540008040 comp rx=0 tx=0).stop 2024-06-11T19:20:04.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 10 monclient: shutdown 2024-06-11T19:20:04.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:04.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 1 -- 172.21.15.98:0/1239177525 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a2530 msgr2=0x7f85481cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 1 --2- 172.21.15.98:0/1239177525 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f85480a2530 0x7f85481cdad0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8538009db0 tx=0x7f853800c490 comp rx=0 tx=0).stop 2024-06-11T19:20:04.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f85477fe640 1 -- 172.21.15.98:0/1239177525 reap_dead start 2024-06-11T19:20:04.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f85477fe640 1 -- 172.21.15.98:0/1239177525 reap_dead start 2024-06-11T19:20:04.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 1 -- 172.21.15.98:0/1239177525 shutdown_connections 2024-06-11T19:20:04.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 1 -- 172.21.15.98:0/1239177525 >> 172.21.15.98:0/1239177525 conn(0x7f85480ab850 msgr2=0x7f854809c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:04.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 1 -- 172.21.15.98:0/1239177525 shutdown_connections 2024-06-11T19:20:04.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 1 -- 172.21.15.98:0/1239177525 wait complete. 2024-06-11T19:20:04.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.377+0000 7f854d50b640 1 librados: shutdown 2024-06-11T19:20:04.427 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:04.427 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:20:04.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:20:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.527+0000 7f0889b20640 10 monclient: get_monmap_and_config 2024-06-11T19:20:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.527+0000 7f0889b20640 10 monclient: build_initial_monmap 2024-06-11T19:20:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.527+0000 7f0889b20640 10 monclient: monmap: 2024-06-11T19:20:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:04.528806+0000 2024-06-11T19:20:04.527 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:04.528806+0000 2024-06-11T19:20:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:04.528 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:04.529 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:04.529 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.528+0000 7f0889b20640 1 Processor -- start 2024-06-11T19:20:04.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.528+0000 7f0889b20640 1 -- start start 2024-06-11T19:20:04.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.528+0000 7f0889b20640 10 monclient: init 2024-06-11T19:20:04.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:04.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 10 monclient: _add_conns ranks=[7,8,9,6,10,3,5,2,4,1,0] 2024-06-11T19:20:04.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08840a4920 0x7f08840a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 10 monclient(hunting): picked mon.noname-i con 0x7f08840a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:04.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f08840a5240 0x7f08840a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 10 monclient(hunting): picked mon.noname-e con 0x7f08840a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:04.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08840a2530 0x7f08840a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 10 monclient(hunting): picked mon.noname-a con 0x7f08840a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:20:04.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f08840a7140 con 0x7f08840a2530 2024-06-11T19:20:04.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f08840a2b50 con 0x7f08840a4920 2024-06-11T19:20:04.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f08840a2e30 con 0x7f08840a5240 2024-06-11T19:20:04.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:04.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0889b20640 10 monclient(hunting): authenticate will time out at 1414.336155s 2024-06-11T19:20:04.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0882ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f08840a5240 0x7f08840a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08840a2530 0x7f08840a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0882ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f08840a5240 0x7f08840a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:60748/0 (socket says 172.21.15.98:60748) 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0882ffd640 1 -- 172.21.15.98:0/1108108048 learned_addr learned my addr 172.21.15.98:0/1108108048 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0882ffd640 10 monclient(hunting): get_auth_request con 0x7f08840a5240 auth_method 0 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0882ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0882ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 10 monclient(hunting): get_auth_request con 0x7f08840a2530 auth_method 0 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0882ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0882ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0882ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f08837fe640 1 --2- 172.21.15.98:0/1108108048 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08840a4920 0x7f08840a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 10 monclient(hunting): handle_auth_done global_id 65385 payload 274 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 1 -- 172.21.15.98:0/1108108048 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f08840a5240 msgr2=0x7f08840a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 1 --2- 172.21.15.98:0/1108108048 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f08840a5240 0x7f08840a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 1 -- 172.21.15.98:0/1108108048 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08840a4920 msgr2=0x7f08840a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 1 --2- 172.21.15.98:0/1108108048 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08840a4920 0x7f08840a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 10 monclient: _finish_hunting 0 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 1 monclient: found mon.noname-a 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:04.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0883fff640 1 -- 172.21.15.98:0/1108108048 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08840a3110 con 0x7f08840a2530 2024-06-11T19:20:04.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f08837fe640 10 monclient: get_auth_request con 0x7f08840a4920 auth_method 0 2024-06-11T19:20:04.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f0882ffd640 1 --2- 172.21.15.98:0/1108108048 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f08840a5240 0x7f08840a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:04.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.529+0000 7f08837fe640 1 --2- 172.21.15.98:0/1108108048 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08840a4920 0x7f08840a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:20:04.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f0883fff640 1 --2- 172.21.15.98:0/1108108048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08840a2530 0x7f08840a5ee0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f087c009d90 tx=0x7f087c02bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17926e273eb267c0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 1 -- 172.21.15.98:0/1108108048 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f087c039070 con 0x7f08840a2530 2024-06-11T19:20:04.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:04.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 10 monclient: dump: 2024-06-11T19:20:04.542 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.542 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.542 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 10 monclient: _finish_auth 0 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 10 monclient: _check_auth_tickets 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 1 -- 172.21.15.98:0/1108108048 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f087c02cd00 con 0x7f08840a2530 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f0889b20640 5 monclient: authenticate success, global_id 65385 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f0889b20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 1 -- 172.21.15.98:0/1108108048 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f087c0327b0 con 0x7f08840a2530 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f08827fc640 10 monclient: dump: 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f0889b20640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f0889b20640 4 set_mon_vals no callback set 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f0889b20640 10 monclient: shutdown 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f0889b20640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f0889b20640 1 -- 172.21.15.98:0/1108108048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08840a2530 msgr2=0x7f08840a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.530+0000 7f0889b20640 1 --2- 172.21.15.98:0/1108108048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08840a2530 0x7f08840a5ee0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f087c009d90 tx=0x7f087c02bce0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.531+0000 7f0889b20640 1 -- 172.21.15.98:0/1108108048 shutdown_connections 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.531+0000 7f0889b20640 1 --2- 172.21.15.98:0/1108108048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08840a2530 0x7f08840a5ee0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.531+0000 7f0889b20640 1 --2- 172.21.15.98:0/1108108048 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f08840a5240 0x7f08840a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.531+0000 7f0889b20640 1 --2- 172.21.15.98:0/1108108048 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08840a4920 0x7f08840a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.531+0000 7f0889b20640 1 -- 172.21.15.98:0/1108108048 >> 172.21.15.98:0/1108108048 conn(0x7f08840ab850 msgr2=0x7f08840a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.531+0000 7f0889b20640 1 -- 172.21.15.98:0/1108108048 shutdown_connections 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.531+0000 7f0889b20640 1 -- 172.21.15.98:0/1108108048 wait complete. 2024-06-11T19:20:04.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.531+0000 7f0889b20640 10 monclient: build_initial_monmap 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.531+0000 7f0889b20640 10 monclient: monmap: 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:04.547 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.531+0000 7f0889b20640 1 librados: starting msgr at 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.531+0000 7f0889b20640 1 librados: starting objecter 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.531+0000 7f0889b20640 1 Processor -- start 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 1 -- start start 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 1 librados: setting wanted keys 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 1 librados: calling monclient init 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 10 monclient: init 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 10 monclient: _add_conns ranks=[1,6,7,2,4,5,8,3,0] 2024-06-11T19:20:04.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08840a2530 0x7f08841cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 10 monclient(hunting): picked mon.noname-f con 0x7f08840a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08840a4920 0x7f08841ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 10 monclient(hunting): picked mon.noname-d con 0x7f08840a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08840a5240 0x7f0884124e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 10 monclient(hunting): picked mon.noname-i con 0x7f08840a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f08840a6da0 con 0x7f08840a2530 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0884138e50 con 0x7f08840a4920 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f08840a6420 con 0x7f08840a5240 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0889b20640 10 monclient(hunting): authenticate will time out at 1414.339661s 2024-06-11T19:20:04.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0882ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08840a4920 0x7f08841ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0882ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08840a4920 0x7f08841ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:37886/0 (socket says 172.21.15.98:37886) 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0882ffd640 1 -- 172.21.15.98:0/4038008043 learned_addr learned my addr 172.21.15.98:0/4038008043 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0882ffd640 10 monclient(hunting): get_auth_request con 0x7f08840a4920 auth_method 0 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0882ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0882ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0882ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0882ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0882ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0883fff640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08840a5240 0x7f0884124e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0883fff640 10 monclient(hunting): get_auth_request con 0x7f08840a5240 auth_method 0 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0883fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0883fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f08837fe640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08840a2530 0x7f08841cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f08837fe640 10 monclient(hunting): get_auth_request con 0x7f08840a2530 auth_method 0 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f08837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f08837fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f08837fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0883fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0883fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0883fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.532+0000 7f0882ffd640 10 monclient(hunting): handle_auth_done global_id 55401 payload 824 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0882ffd640 1 -- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08840a5240 msgr2=0x7f0884124e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0882ffd640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08840a5240 0x7f0884124e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0882ffd640 1 -- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08840a2530 msgr2=0x7f08841cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0882ffd640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08840a2530 0x7f08841cdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0882ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0882ffd640 1 monclient: found mon.noname-d 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0882ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0882ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0882ffd640 1 -- 172.21.15.98:0/4038008043 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0884128870 con 0x7f08840a4920 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f08837fe640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08840a2530 0x7f08841cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0882ffd640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08840a4920 0x7f08841ce040 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f087800e8d0 tx=0x7f087800ed90 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 10 client.?.objecter ms_handle_connect 0x7f08840a4920 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08780044a0 con 0x7f08840a4920 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:20:04.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 10 monclient: dump: 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.553 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 10 monclient: _finish_auth 0 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f087800c8e0 con 0x7f08840a4920 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0889b20640 5 monclient: authenticate success, global_id 55401 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0889b20640 10 monclient: _renew_subs 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0889b20640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0889b20640 1 -- 172.21.15.98:0/4038008043 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0884128b50 con 0x7f08840a4920 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0878003610 con 0x7f08840a4920 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f086bfff640 10 monclient: dump: 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.555 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0889b20640 10 client.55401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0889b20640 10 monclient: _renew_subs 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0889b20640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0889b20640 1 -- 172.21.15.98:0/4038008043 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f08841387f0 con 0x7f08840a4920 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0881ffb640 4 set_mon_vals no callback set 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.533+0000 7f0889b20640 1 librados: init done 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f0889b20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f0889b20640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08841253b0 0x7f0884125790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f0889b20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f0889b20640 1 -- 172.21.15.98:0/4038008043 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f0884050830 con 0x7f08841253b0 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f0878004030 con 0x7f08840a4920 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f086bfff640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f087003fdb0 0x7f0870042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f08837fe640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08841253b0 0x7f0884125790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(421..421 src has 1..421) ==== 5845+0+0 (secure 0 0 0) 0x7f087800f040 con 0x7f08840a4920 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f086bfff640 10 client.55401.objecter ms_dispatch 0x7f0884000b90 osd_map(421..421 src has 1..421) 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f086bfff640 3 client.55401.objecter handle_osd_map got epochs [421,421] > 0 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f086bfff640 3 client.55401.objecter handle_osd_map decoding full epoch 421 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f08837fe640 10 monclient: get_auth_request con 0x7f08841253b0 auth_method 0 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f08837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f08837fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f08837fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f086bfff640 20 client.55401.objecter dump_active .. 0 homeless 2024-06-11T19:20:04.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f08837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f08837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f08837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f0883fff640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f087003fdb0 0x7f0870042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f0883fff640 10 monclient: get_auth_request con 0x7f087003fdb0 auth_method 0 2024-06-11T19:20:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f08837fe640 10 monclient(hunting): handle_auth_done global_id 104178 payload 824 2024-06-11T19:20:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f08837fe640 1 -- 172.21.15.98:0/4038008043 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08840a9090 con 0x7f08841253b0 2024-06-11T19:20:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.534+0000 7f08837fe640 1 -- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08841253b0 msgr2=0x7f0884125790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:20:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.535+0000 7f08837fe640 1 -- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08841253b0 msgr2=0x7f0884125790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:20:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.535+0000 7f08837fe640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08841253b0 0x7f0884125790 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f087400f510 tx=0x7f087400f9d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:20:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.535+0000 7f08837fe640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08841253b0 0x7f0884125790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.535+0000 7f086bfff640 10 client.55401.objecter ms_handle_connect 0x7f087003fdb0 2024-06-11T19:20:04.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.535+0000 7f0883fff640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f087003fdb0 0x7f0870042270 secure :-1 s=READY pgs=4213 cs=0 l=1 rev1=1 crypto rx=0x7f087c036000 tx=0x7f087c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f08837fe640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08841253b0 0x7f0884125790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f08837fe640 10 monclient: get_auth_request con 0x7f08841253b0 auth_method 0 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f08837fe640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f08837fe640 10 monclient: _init_auth method 2 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f08837fe640 10 monclient: _init_auth already have auth, reseting 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f08837fe640 10 monclient: handle_auth_reply_more payload 9 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f08837fe640 10 monclient: handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f08837fe640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f08837fe640 10 monclient: handle_auth_done global_id 104178 payload 293 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f08837fe640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08841253b0 0x7f0884125790 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f087400f510 tx=0x7f0874008040 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f086bfff640 10 client.55401.objecter ms_handle_connect 0x7f08841253b0 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f086bfff640 10 client.55401.objecter resend_mon_ops 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0874018070 con 0x7f08841253b0 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f086bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f086bfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:20:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.736+0000 7f086bfff640 10 monclient: dump: 2024-06-11T19:20:04.738 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.738 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.738 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.738 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.738 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.738 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.738 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.738 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.738 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.738 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.738 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.738 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.739 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.739 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.739 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.739 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.737+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0874024070 con 0x7f08841253b0 2024-06-11T19:20:04.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.737+0000 7f086bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:04.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.737+0000 7f086bfff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:04.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.737+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08841253b0 msgr2=0x7f0884125790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.737+0000 7f086bfff640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08841253b0 0x7f0884125790 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f087400f510 tx=0x7f0874008040 comp rx=0 tx=0).stop 2024-06-11T19:20:04.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.737+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08841253b0 msgr2=0x7f0884125790 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:04.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.737+0000 7f086bfff640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08841253b0 0x7f0884125790 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f087400f510 tx=0x7f0874008040 comp rx=0 tx=0).stop 2024-06-11T19:20:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.743+0000 7f0889b20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.743+0000 7f0889b20640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0884125cd0 0x7f088409f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.743+0000 7f0889b20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.743+0000 7f0889b20640 1 -- 172.21.15.98:0/4038008043 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f08840a6750 con 0x7f0884125cd0 2024-06-11T19:20:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f08837fe640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0884125cd0 0x7f088409f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f08837fe640 10 monclient: get_auth_request con 0x7f0884125cd0 auth_method 0 2024-06-11T19:20:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f08837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f08837fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f08837fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f08837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f08837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f08837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f08837fe640 10 monclient(hunting): handle_auth_done global_id 104187 payload 824 2024-06-11T19:20:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f08837fe640 1 -- 172.21.15.98:0/4038008043 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f08841ce580 con 0x7f0884125cd0 2024-06-11T19:20:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f08837fe640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0884125cd0 0x7f088409f3f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f087400f510 tx=0x7f08740046c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 10 client.55401.objecter ms_handle_connect 0x7f0884125cd0 2024-06-11T19:20:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 10 client.55401.objecter resend_mon_ops 2024-06-11T19:20:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0874018050 con 0x7f0884125cd0 2024-06-11T19:20:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:20:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 10 monclient: dump: 2024-06-11T19:20:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.747 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f0874005a70 con 0x7f0884125cd0 2024-06-11T19:20:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0884125cd0 msgr2=0x7f088409f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0884125cd0 0x7f088409f3f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f087400f510 tx=0x7f08740046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 1 -- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0884125cd0 msgr2=0x7f088409f3f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.744+0000 7f086bfff640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0884125cd0 0x7f088409f3f0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f087400f510 tx=0x7f08740046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:04.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:04.748 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:04.748 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:04.748 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:04.748 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 19949, 2024-06-11T19:20:04.748 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:04.748 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:04.748 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:04.748 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:04.748 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:04.748 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.749 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.750 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:04.751 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:04.752 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.753 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.756 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:04.761 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:04.768 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:04.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.771 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:04.771 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:04.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 10 librados: watch_flush enter 2024-06-11T19:20:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 10 librados: watch_flush exit 2024-06-11T19:20:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 20 client.55401.objecter shutdown clearing up homeless session... 2024-06-11T19:20:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 10 client.55401.objecter successfully canceled tick 2024-06-11T19:20:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 1 -- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f087003fdb0 msgr2=0x7f0870042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f087003fdb0 0x7f0870042270 secure :-1 s=READY pgs=4213 cs=0 l=1 rev1=1 crypto rx=0x7f087c036000 tx=0x7f087c030000 comp rx=0 tx=0).stop 2024-06-11T19:20:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 10 monclient: shutdown 2024-06-11T19:20:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 1 -- 172.21.15.98:0/4038008043 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08840a4920 msgr2=0x7f08841ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 1 --2- 172.21.15.98:0/4038008043 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08840a4920 0x7f08841ce040 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f087800e8d0 tx=0x7f087800ed90 comp rx=0 tx=0).stop 2024-06-11T19:20:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0883fff640 1 -- 172.21.15.98:0/4038008043 reap_dead start 2024-06-11T19:20:04.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0883fff640 1 -- 172.21.15.98:0/4038008043 reap_dead start 2024-06-11T19:20:04.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 1 -- 172.21.15.98:0/4038008043 shutdown_connections 2024-06-11T19:20:04.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 1 -- 172.21.15.98:0/4038008043 >> 172.21.15.98:0/4038008043 conn(0x7f08840ab850 msgr2=0x7f088409c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:04.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 1 -- 172.21.15.98:0/4038008043 shutdown_connections 2024-06-11T19:20:04.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 1 -- 172.21.15.98:0/4038008043 wait complete. 2024-06-11T19:20:04.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.745+0000 7f0889b20640 1 librados: shutdown 2024-06-11T19:20:04.774 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:04.774 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:20:04.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:20:04.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.879+0000 7f0627d60640 10 monclient: get_monmap_and_config 2024-06-11T19:20:04.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.879+0000 7f0627d60640 10 monclient: build_initial_monmap 2024-06-11T19:20:04.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.879+0000 7f0627d60640 10 monclient: monmap: 2024-06-11T19:20:04.878 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:04.880547+0000 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:04.880547+0000 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:04.879 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:04.880 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:04.880 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 1 Processor -- start 2024-06-11T19:20:04.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 1 -- start start 2024-06-11T19:20:04.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 10 monclient: init 2024-06-11T19:20:04.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:04.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 10 monclient: _add_conns ranks=[9,0,10,6,4,3,8,7,5,1,2] 2024-06-11T19:20:04.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f06200a0cd0 0x7f06200a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 10 monclient(hunting): picked mon.noname-a con 0x7f06200a0cd0 addr v2:172.21.15.98:3300/0 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0620138c80 addr v1:172.21.15.98:6789/0 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0620139060 0x7f06200a2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 10 monclient(hunting): picked mon.noname-f con 0x7f0620139060 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f062009efd0 con 0x7f0620138c80 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f06200739b0 con 0x7f06200a0cd0 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f062009ecf0 con 0x7f0620139060 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0627d60640 10 monclient(hunting): authenticate will time out at 1414.687823s 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0625afc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f06200a0cd0 0x7f06200a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0625afc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f06200a0cd0 0x7f06200a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54310/0 (socket says 172.21.15.98:54310) 2024-06-11T19:20:04.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f06252fb640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0620138c80 0x7f06200a2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55880/0 (socket says 172.21.15.98:55880) 2024-06-11T19:20:04.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0625afc640 1 -- 172.21.15.98:0/913733896 learned_addr learned my addr 172.21.15.98:0/913733896 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:04.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0625afc640 10 monclient(hunting): get_auth_request con 0x7f06200a0cd0 auth_method 0 2024-06-11T19:20:04.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0625afc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0625afc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.880+0000 7f0625afc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0625afc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0625afc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0625afc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f06262fd640 1 --2- 172.21.15.98:0/913733896 >> v2:172.21.15.117:3300/0 conn(0x7f0620139060 0x7f06200a2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0625afc640 10 monclient(hunting): handle_auth_done global_id 65394 payload 274 2024-06-11T19:20:04.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0625afc640 1 -- 172.21.15.98:0/913733896 >> v2:172.21.15.117:3300/0 conn(0x7f0620139060 msgr2=0x7f06200a2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:04.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0625afc640 1 --2- 172.21.15.98:0/913733896 >> v2:172.21.15.117:3300/0 conn(0x7f0620139060 0x7f06200a2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0625afc640 1 -- 172.21.15.98:0/913733896 >> v1:172.21.15.98:6789/0 conn(0x7f0620138c80 legacy=0x7f06200a2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0625afc640 10 monclient: _finish_hunting 0 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0625afc640 1 monclient: found mon.noname-a 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0624afa640 1 -- 172.21.15.98:0/913733896 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182038420 0 0) 0x7f06100011d0 con 0x7f0620138c80 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0625afc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0625afc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0625afc640 1 -- 172.21.15.98:0/913733896 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062009f2b0 con 0x7f06200a0cd0 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f06262fd640 10 monclient: get_auth_request con 0x7f0620139060 auth_method 0 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f06262fd640 1 --2- 172.21.15.98:0/913733896 >> v2:172.21.15.117:3300/0 conn(0x7f0620139060 0x7f06200a2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.881+0000 7f0624afa640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0625afc640 1 --2- 172.21.15.98:0/913733896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06200a0cd0 0x7f06200a10b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f062009fa40 tx=0x7f061c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b237d1e21815d4 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 1 -- 172.21.15.98:0/913733896 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f061c02e070 con 0x7f06200a0cd0 2024-06-11T19:20:04.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 10 monclient: dump: 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.890 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 10 monclient: _finish_auth 0 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 10 monclient: _check_auth_tickets 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 1 -- 172.21.15.98:0/913733896 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f061c037070 con 0x7f06200a0cd0 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 5 monclient: authenticate success, global_id 65394 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 1 -- 172.21.15.98:0/913733896 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f061c032470 con 0x7f06200a0cd0 2024-06-11T19:20:04.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 4 set_mon_vals no callback set 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0624afa640 10 monclient: dump: 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.892 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 10 monclient: shutdown 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 1 -- 172.21.15.98:0/913733896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06200a0cd0 msgr2=0x7f06200a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 1 --2- 172.21.15.98:0/913733896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06200a0cd0 0x7f06200a10b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f062009fa40 tx=0x7f061c02b880 comp rx=0 tx=0).stop 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 1 -- 172.21.15.98:0/913733896 shutdown_connections 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 1 --2- 172.21.15.98:0/913733896 >> v2:172.21.15.117:3300/0 conn(0x7f0620139060 0x7f06200a2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 1 --2- 172.21.15.98:0/913733896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06200a0cd0 0x7f06200a10b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 1 -- 172.21.15.98:0/913733896 >> 172.21.15.98:0/913733896 conn(0x7f06200a7850 msgr2=0x7f06200a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 1 -- 172.21.15.98:0/913733896 shutdown_connections 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.882+0000 7f0627d60640 1 -- 172.21.15.98:0/913733896 wait complete. 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.883+0000 7f0627d60640 10 monclient: build_initial_monmap 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.883+0000 7f0627d60640 10 monclient: monmap: 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:04.894 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.883+0000 7f0627d60640 1 librados: starting msgr at 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.883+0000 7f0627d60640 1 librados: starting objecter 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.883+0000 7f0627d60640 1 Processor -- start 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.884+0000 7f0627d60640 1 -- start start 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.884+0000 7f0627d60640 1 librados: setting wanted keys 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.884+0000 7f0627d60640 1 librados: calling monclient init 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.884+0000 7f0627d60640 10 monclient: init 2024-06-11T19:20:04.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.884+0000 7f0627d60640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.884+0000 7f0627d60640 10 monclient: _add_conns ranks=[4,0,3,2,7,6,8,1,5] 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.884+0000 7f0627d60640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06200a0cd0 0x7f06201cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.884+0000 7f0627d60640 10 monclient(hunting): picked mon.noname-c con 0x7f06200a0cd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0627d60640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0620138c80 0x7f06201ce150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0627d60640 10 monclient(hunting): picked mon.noname-a con 0x7f0620138c80 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0627d60640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0620139060 0x7f062012e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0627d60640 10 monclient(hunting): picked mon.noname-g con 0x7f0620139060 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0627d60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0627d60640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06200a1770 con 0x7f0620138c80 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0627d60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0627d60640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f06200a15f0 con 0x7f0620139060 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0627d60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0627d60640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f06200a2ea0 con 0x7f06200a0cd0 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0627d60640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0627d60640 10 monclient(hunting): authenticate will time out at 1414.692077s 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0625afc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06200a0cd0 0x7f06201cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f06262fd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0620139060 0x7f062012e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0625afc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06200a0cd0 0x7f06201cdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55328/0 (socket says 172.21.15.98:55328) 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0625afc640 1 -- 172.21.15.98:0/2334647772 learned_addr learned my addr 172.21.15.98:0/2334647772 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0625afc640 10 monclient(hunting): get_auth_request con 0x7f06200a0cd0 auth_method 0 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0625afc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0625afc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0625afc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f06262fd640 10 monclient(hunting): get_auth_request con 0x7f0620139060 auth_method 0 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f06262fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f06262fd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f06262fd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0625afc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0625afc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f0625afc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f06262fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f06262fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.885+0000 7f06262fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f0625afc640 10 monclient(hunting): handle_auth_done global_id 24241 payload 824 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f0625afc640 1 -- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0620139060 msgr2=0x7f062012e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f0625afc640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0620139060 0x7f062012e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f0625afc640 1 -- 172.21.15.98:0/2334647772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0620138c80 msgr2=0x7f06201ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f0625afc640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0620138c80 0x7f06201ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f0625afc640 10 monclient: _finish_hunting 0 2024-06-11T19:20:04.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f0625afc640 1 monclient: found mon.noname-c 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f0625afc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f0625afc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f0625afc640 1 -- 172.21.15.98:0/2334647772 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0620131d00 con 0x7f06200a0cd0 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06262fd640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0620139060 0x7f062012e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f0625afc640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06200a0cd0 0x7f06201cdc10 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f061c000c00 tx=0x7f061c0046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 10 client.?.objecter ms_handle_connect 0x7f06200a0cd0 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f061c041070 con 0x7f06200a0cd0 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 10 monclient: dump: 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.899 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.900 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 10 monclient: _finish_auth 0 2024-06-11T19:20:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 10 monclient: _check_auth_tickets 2024-06-11T19:20:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f061c003610 con 0x7f06200a0cd0 2024-06-11T19:20:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f061c003c30 con 0x7f06200a0cd0 2024-06-11T19:20:04.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.886+0000 7f06167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.887+0000 7f06167fc640 10 monclient: dump: 2024-06-11T19:20:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.887+0000 7f0617fff640 4 set_mon_vals no callback set 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.888+0000 7f0627d60640 5 monclient: authenticate success, global_id 24241 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.888+0000 7f0627d60640 10 monclient: _renew_subs 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.888+0000 7f0627d60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.888+0000 7f0627d60640 1 -- 172.21.15.98:0/2334647772 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0620131fe0 con 0x7f06200a0cd0 2024-06-11T19:20:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.888+0000 7f0627d60640 10 client.24241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.888+0000 7f0627d60640 10 monclient: _renew_subs 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.888+0000 7f0627d60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.888+0000 7f0627d60640 1 -- 172.21.15.98:0/2334647772 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f0620136a10 con 0x7f06200a0cd0 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.888+0000 7f0627d60640 1 librados: init done 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.889+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f061c04c060 con 0x7f06200a0cd0 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.889+0000 7f0627d60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.889+0000 7f0627d60640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012e560 0x7f062012e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.889+0000 7f06167fc640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f060403fdb0 0x7f0604042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(421..421 src has 1..421) ==== 5845+0+0 (secure 0 0 0) 0x7f061c073040 con 0x7f06200a0cd0 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06167fc640 10 client.24241.objecter ms_dispatch 0x7f0620000b90 osd_map(421..421 src has 1..421) 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06167fc640 3 client.24241.objecter handle_osd_map got epochs [421,421] > 0 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06167fc640 3 client.24241.objecter handle_osd_map decoding full epoch 421 2024-06-11T19:20:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06167fc640 20 client.24241.objecter dump_active .. 0 homeless 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f0627d60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f0627d60640 1 -- 172.21.15.98:0/2334647772 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f0620050830 con 0x7f062012e560 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06252fb640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012e560 0x7f062012e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06252fb640 10 monclient: get_auth_request con 0x7f062012e560 auth_method 0 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06252fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06252fb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06252fb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06262fd640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f060403fdb0 0x7f0604042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06262fd640 10 monclient: get_auth_request con 0x7f060403fdb0 auth_method 0 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06252fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06252fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06252fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06252fb640 10 monclient(hunting): handle_auth_done global_id 23315 payload 824 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06252fb640 1 -- 172.21.15.98:0/2334647772 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06200a10b0 con 0x7f062012e560 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06167fc640 10 client.24241.objecter ms_handle_connect 0x7f060403fdb0 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.890+0000 7f06262fd640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f060403fdb0 0x7f0604042270 secure :-1 s=READY pgs=4214 cs=0 l=1 rev1=1 crypto rx=0x7f0618007fd0 tx=0x7f06180076e0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 10 client.24241.objecter ms_handle_connect 0x7f062012e560 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 10 client.24241.objecter resend_mon_ops 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06252fb640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012e560 0x7f062012e940 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f061000c2f0 tx=0x7f061000c7b0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0610017070 con 0x7f062012e560 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 10 monclient: dump: 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f06100072e0 con 0x7f062012e560 2024-06-11T19:20:04.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:04.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:04.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012e560 msgr2=0x7f062012e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012e560 0x7f062012e940 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f061000c2f0 tx=0x7f061000c7b0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012e560 msgr2=0x7f062012e940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:04.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.891+0000 7f06167fc640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012e560 0x7f062012e940 secure :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f061000c2f0 tx=0x7f061000c7b0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f0627d60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:04.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f0627d60640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012ee80 0x7f062009c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:04.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f0627d60640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:04.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f0627d60640 1 -- 172.21.15.98:0/2334647772 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f0620132a10 con 0x7f062012ee80 2024-06-11T19:20:04.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f06252fb640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012ee80 0x7f062009c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:04.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f06252fb640 10 monclient: get_auth_request con 0x7f062012ee80 auth_method 0 2024-06-11T19:20:04.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f06252fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:04.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f06252fb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:04.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f06252fb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:04.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f06252fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:04.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f06252fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:04.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f06252fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:04.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f06252fb640 10 monclient(hunting): handle_auth_done global_id 23324 payload 824 2024-06-11T19:20:04.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.906+0000 7f06252fb640 1 -- 172.21.15.98:0/2334647772 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0620068c60 con 0x7f062012ee80 2024-06-11T19:20:04.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06252fb640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012ee80 0x7f062009c340 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f0610014000 tx=0x7f0610000f70 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:04.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 10 client.24241.objecter ms_handle_connect 0x7f062012ee80 2024-06-11T19:20:04.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 10 client.24241.objecter resend_mon_ops 2024-06-11T19:20:04.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0610017040 con 0x7f062012ee80 2024-06-11T19:20:04.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:04.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:20:04.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 10 monclient: dump: 2024-06-11T19:20:04.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:04.914 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:04.914 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.914 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:04.915 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:04.915 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:04.915 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:04.915 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:04.915 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:04.915 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:04.915 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:04.915 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:04.915 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:04.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:04.915 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:04.915 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:04.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7f0610022410 con 0x7f062012ee80 2024-06-11T19:20:04.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:04.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:04.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012ee80 msgr2=0x7f062009c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012ee80 0x7f062009c340 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f0610014000 tx=0x7f0610000f70 comp rx=0 tx=0).stop 2024-06-11T19:20:04.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 1 -- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012ee80 msgr2=0x7f062009c340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:04.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.907+0000 7f06167fc640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f062012ee80 0x7f062009c340 secure :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f0610014000 tx=0x7f0610000f70 comp rx=0 tx=0).stop 2024-06-11T19:20:04.916 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:04.916 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:04.916 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:04.917 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:04.917 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:04.917 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 655101, 2024-06-11T19:20:04.917 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:04.917 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:04.917 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:04.917 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:04.917 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:04.917 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:04.917 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:04.917 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:04.917 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:04.918 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:04.918 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.918 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:20:04.918 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:04.918 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:04.918 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:04.918 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.918 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.918 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.918 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.918 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.918 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.919 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:04.920 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:04.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.922 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.923 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.924 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:04.925 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.926 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.926 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.926 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:04.926 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:04.926 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.926 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.926 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:20:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:04.940 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 10 librados: watch_flush enter 2024-06-11T19:20:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 10 librados: watch_flush exit 2024-06-11T19:20:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 20 client.24241.objecter shutdown clearing up homeless session... 2024-06-11T19:20:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 10 client.24241.objecter successfully canceled tick 2024-06-11T19:20:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 1 -- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f060403fdb0 msgr2=0x7f0604042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f060403fdb0 0x7f0604042270 secure :-1 s=READY pgs=4214 cs=0 l=1 rev1=1 crypto rx=0x7f0618007fd0 tx=0x7f06180076e0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 10 monclient: shutdown 2024-06-11T19:20:04.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 1 -- 172.21.15.98:0/2334647772 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06200a0cd0 msgr2=0x7f06201cdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 1 --2- 172.21.15.98:0/2334647772 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f06200a0cd0 0x7f06201cdc10 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f061c000c00 tx=0x7f061c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f06262fd640 1 -- 172.21.15.98:0/2334647772 reap_dead start 2024-06-11T19:20:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f06262fd640 1 -- 172.21.15.98:0/2334647772 reap_dead start 2024-06-11T19:20:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 1 -- 172.21.15.98:0/2334647772 shutdown_connections 2024-06-11T19:20:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 1 -- 172.21.15.98:0/2334647772 >> 172.21.15.98:0/2334647772 conn(0x7f06200a7850 msgr2=0x7f06201be6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 1 -- 172.21.15.98:0/2334647772 shutdown_connections 2024-06-11T19:20:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 1 -- 172.21.15.98:0/2334647772 wait complete. 2024-06-11T19:20:04.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:04.908+0000 7f0627d60640 1 librados: shutdown 2024-06-11T19:20:04.943 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:04.944 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:20:04.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:20:05.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.051+0000 7efcabfbc640 10 monclient: get_monmap_and_config 2024-06-11T19:20:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.051+0000 7efcabfbc640 10 monclient: build_initial_monmap 2024-06-11T19:20:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.051+0000 7efcabfbc640 10 monclient: monmap: 2024-06-11T19:20:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:05.052787+0000 2024-06-11T19:20:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:05.052787+0000 2024-06-11T19:20:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:05.052 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.052+0000 7efcabfbc640 1 Processor -- start 2024-06-11T19:20:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.052+0000 7efcabfbc640 1 -- start start 2024-06-11T19:20:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.052+0000 7efcabfbc640 10 monclient: init 2024-06-11T19:20:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.052+0000 7efcabfbc640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.052+0000 7efcabfbc640 10 monclient: _add_conns ranks=[2,0,4,7,8,9,5,3,6,10,1] 2024-06-11T19:20:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.052+0000 7efcabfbc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efca40a4920 0x7efca40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.052+0000 7efcabfbc640 10 monclient(hunting): picked mon.noname-b con 0x7efca40a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.052+0000 7efcabfbc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efca40a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:20:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.052+0000 7efcabfbc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efca40a2530 0x7efca40a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.052+0000 7efcabfbc640 10 monclient(hunting): picked mon.noname-c con 0x7efca40a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcabfbc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efca40a7140 con 0x7efca40a5240 2024-06-11T19:20:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcabfbc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcabfbc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efca40a2b50 con 0x7efca40a4920 2024-06-11T19:20:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcabfbc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcabfbc640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7efca40a2e30 con 0x7efca40a2530 2024-06-11T19:20:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcabfbc640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcabfbc640 10 monclient(hunting): authenticate will time out at 1414.860021s 2024-06-11T19:20:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efca40a4920 0x7efca40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcaa559640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efca40a2530 0x7efca40a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9557640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efca40a5240 0x7efca40a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55894/0 (socket says 172.21.15.98:55894) 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9557640 1 -- 172.21.15.98:0/343991052 learned_addr learned my addr 172.21.15.98:0/343991052 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 10 monclient(hunting): get_auth_request con 0x7efca40a4920 auth_method 0 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcaa559640 10 monclient(hunting): get_auth_request con 0x7efca40a2530 auth_method 0 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcaa559640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcaa559640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcaa559640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcaa559640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcaa559640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcaa559640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 10 monclient(hunting): handle_auth_done global_id 84719 payload 274 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 1 -- 172.21.15.98:0/343991052 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efca40a2530 msgr2=0x7efca40a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 1 --2- 172.21.15.98:0/343991052 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efca40a2530 0x7efca40a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 1 -- 172.21.15.98:0/343991052 >> v1:172.21.15.98:6789/0 conn(0x7efca40a5240 legacy=0x7efca40a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 10 monclient: _finish_hunting 0 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 1 monclient: found mon.noname-b 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca8d56640 1 -- 172.21.15.98:0/343991052 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 57426709 0 0) 0x7efca40a7140 con 0x7efca40a5240 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca9d58640 1 -- 172.21.15.98:0/343991052 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca40a3110 con 0x7efca40a4920 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efcaa559640 1 --2- 172.21.15.98:0/343991052 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efca40a2530 0x7efca40a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.053+0000 7efca8d56640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:20:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca9d58640 1 --2- 172.21.15.98:0/343991052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efca40a4920 0x7efca40a4d00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efca00097d0 tx=0x7efca002b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7234def411610bec server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 1 -- 172.21.15.98:0/343991052 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efca0039070 con 0x7efca40a4920 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 10 monclient: dump: 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 10 monclient: _finish_auth 0 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 10 monclient: _check_auth_tickets 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 1 -- 172.21.15.98:0/343991052 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca0030070 con 0x7efca40a4920 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 5 monclient: authenticate success, global_id 84719 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 4 set_mon_vals no callback set 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 1 -- 172.21.15.98:0/343991052 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efca0031db0 con 0x7efca40a4920 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efca8d56640 10 monclient: dump: 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 10 monclient: shutdown 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 1 -- 172.21.15.98:0/343991052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efca40a4920 msgr2=0x7efca40a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 1 --2- 172.21.15.98:0/343991052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efca40a4920 0x7efca40a4d00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efca00097d0 tx=0x7efca002b770 comp rx=0 tx=0).stop 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 1 -- 172.21.15.98:0/343991052 shutdown_connections 2024-06-11T19:20:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 1 --2- 172.21.15.98:0/343991052 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efca40a2530 0x7efca40a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 1 --2- 172.21.15.98:0/343991052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efca40a4920 0x7efca40a4d00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 1 -- 172.21.15.98:0/343991052 >> 172.21.15.98:0/343991052 conn(0x7efca40ab850 msgr2=0x7efca40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.054+0000 7efcabfbc640 1 -- 172.21.15.98:0/343991052 shutdown_connections 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.055+0000 7efcabfbc640 1 -- 172.21.15.98:0/343991052 wait complete. 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.055+0000 7efcabfbc640 10 monclient: build_initial_monmap 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.055+0000 7efcabfbc640 10 monclient: monmap: 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.055+0000 7efcabfbc640 1 librados: starting msgr at 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.055+0000 7efcabfbc640 1 librados: starting objecter 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.055+0000 7efcabfbc640 1 Processor -- start 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.055+0000 7efcabfbc640 1 -- start start 2024-06-11T19:20:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.055+0000 7efcabfbc640 1 librados: setting wanted keys 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.055+0000 7efcabfbc640 1 librados: calling monclient init 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.055+0000 7efcabfbc640 10 monclient: init 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 10 monclient: _add_conns ranks=[8,3,5,6,2,0,4,1,7] 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efca40a2530 0x7efca41cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 10 monclient(hunting): picked mon.noname-e con 0x7efca40a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efca40a4920 0x7efca41cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 10 monclient(hunting): picked mon.noname-g con 0x7efca40a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efca40a5240 0x7efca4124070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 10 monclient(hunting): picked mon.noname-h con 0x7efca40a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7efca40a6da0 con 0x7efca40a4920 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7efca4138e50 con 0x7efca40a5240 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7efca40a65f0 con 0x7efca40a2530 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 10 monclient(hunting): authenticate will time out at 1414.863362s 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efca40a2530 0x7efca41cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efca40a2530 0x7efca41cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:60754/0 (socket says 172.21.15.98:60754) 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 1 -- 172.21.15.98:0/2410797810 learned_addr learned my addr 172.21.15.98:0/2410797810 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 10 monclient(hunting): get_auth_request con 0x7efca40a2530 auth_method 0 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 10 monclient(hunting): handle_auth_done global_id 94400 payload 824 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 1 -- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efca40a5240 msgr2=0x7efca4124070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7efca40a5240 0x7efca4124070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 1 -- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efca40a4920 msgr2=0x7efca41cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efca40a4920 0x7efca41cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 10 monclient: _finish_hunting 0 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 1 monclient: found mon.noname-e 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 1 -- 172.21.15.98:0/2410797810 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca4127a70 con 0x7efca40a2530 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efca9d58640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efca40a2530 0x7efca41cdaa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efca00096f0 tx=0x7efca00046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efc9a7fc640 10 client.?.objecter ms_handle_connect 0x7efca40a2530 2024-06-11T19:20:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efc9a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efca0039050 con 0x7efca40a2530 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efc9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efc9a7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efc9a7fc640 10 monclient: dump: 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efc9a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efc9a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efc9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efcabfbc640 5 monclient: authenticate success, global_id 94400 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.056+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca0045070 con 0x7efca40a2530 2024-06-11T19:20:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efc9a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efca0031e50 con 0x7efca40a2530 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efcabfbc640 10 monclient: _renew_subs 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efcabfbc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efcabfbc640 1 -- 172.21.15.98:0/2410797810 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca41261e0 con 0x7efca40a2530 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efc9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efc9a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efc9bfff640 4 set_mon_vals no callback set 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efcabfbc640 10 client.94400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efc9a7fc640 10 monclient: dump: 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efcabfbc640 10 monclient: _renew_subs 2024-06-11T19:20:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efcabfbc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efcabfbc640 1 -- 172.21.15.98:0/2410797810 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7efca4127d50 con 0x7efca40a2530 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efcabfbc640 1 librados: init done 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efcabfbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efcabfbc640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4124d70 0x7efca402cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7efca0032070 con 0x7efca40a2530 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efcabfbc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.057+0000 7efcabfbc640 1 -- 172.21.15.98:0/2410797810 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7efca4050830 con 0x7efca4124d70 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efca9557640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4124d70 0x7efca402cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efca9557640 10 monclient: get_auth_request con 0x7efca4124d70 auth_method 0 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efca9557640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efca9557640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efca9557640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efc8803fdb0 0x7efc88042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(422..422 src has 1..422) ==== 6217+0+0 (secure 0 0 0) 0x7efca0073700 con 0x7efca40a2530 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 10 client.94400.objecter ms_dispatch 0x7efca4000b90 osd_map(422..422 src has 1..422) 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 3 client.94400.objecter handle_osd_map got epochs [422,422] > 0 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 3 client.94400.objecter handle_osd_map decoding full epoch 422 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efca9557640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efca9557640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efca9557640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 20 client.94400.objecter dump_active .. 0 homeless 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efcaa559640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efc8803fdb0 0x7efc88042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efcaa559640 10 monclient: get_auth_request con 0x7efc8803fdb0 auth_method 0 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efca9557640 10 monclient(hunting): handle_auth_done global_id 84526 payload 824 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efca9557640 1 -- 172.21.15.98:0/2410797810 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efca40aad10 con 0x7efca4124d70 2024-06-11T19:20:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 10 client.94400.objecter ms_handle_connect 0x7efca4124d70 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 10 client.94400.objecter resend_mon_ops 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efca9557640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4124d70 0x7efca402cb40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efca41257d0 tx=0x7efc9400c7a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 10 client.94400.objecter ms_handle_connect 0x7efc8803fdb0 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efcaa559640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efc8803fdb0 0x7efc88042270 secure :-1 s=READY pgs=4215 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009bd0 tx=0x7efc9c00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc940047e0 con 0x7efca4124d70 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.058+0000 7efc9a7fc640 10 monclient: dump: 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.059+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efc940073a0 con 0x7efca4124d70 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.059+0000 7efc9a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.059+0000 7efc9a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.059+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4124d70 msgr2=0x7efca402cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.059+0000 7efc9a7fc640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4124d70 0x7efca402cb40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efca41257d0 tx=0x7efc9400c7a0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.059+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4124d70 msgr2=0x7efca402cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:05.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.059+0000 7efc9a7fc640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4124d70 0x7efca402cb40 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efca41257d0 tx=0x7efc9400c7a0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efcabfbc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:05.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efcabfbc640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4125150 0x7efca402d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efcabfbc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efcabfbc640 1 -- 172.21.15.98:0/2410797810 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7efca4069740 con 0x7efca4125150 2024-06-11T19:20:05.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efca9557640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4125150 0x7efca402d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efca9557640 10 monclient: get_auth_request con 0x7efca4125150 auth_method 0 2024-06-11T19:20:05.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efca9557640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efca9557640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efca9557640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efca9557640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efca9557640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efca9557640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efca9557640 10 monclient(hunting): handle_auth_done global_id 84535 payload 824 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.065+0000 7efca9557640 1 -- 172.21.15.98:0/2410797810 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efca41387f0 con 0x7efca4125150 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efca9557640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4125150 0x7efca402d080 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc94000c00 tx=0x7efc94000ed0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 10 client.94400.objecter ms_handle_connect 0x7efca4125150 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 10 client.94400.objecter resend_mon_ops 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc94019d50 con 0x7efca4125150 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 10 monclient: dump: 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.085 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7efc94019040 con 0x7efca4125150 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:05.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:05.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4125150 msgr2=0x7efca402d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4125150 0x7efca402d080 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc94000c00 tx=0x7efc94000ed0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 1 -- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4125150 msgr2=0x7efca402d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:05.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efc9a7fc640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7efca4125150 0x7efca402d080 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc94000c00 tx=0x7efc94000ed0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.087 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:05.087 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:05.087 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:05.087 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:05.087 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 54, 2024-06-11T19:20:05.087 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 276836, 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:05.088 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:05.089 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:05.091 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:05.092 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:05.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:05.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:05.096 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:05.098 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.099 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:05.100 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:05.100 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:05.100 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:05.100 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:05.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.100 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:05.100 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:05.100 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:05.100 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:05.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:05.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:05.106 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.108 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:05.110 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:05.110 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 10 librados: watch_flush enter 2024-06-11T19:20:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 10 librados: watch_flush exit 2024-06-11T19:20:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 20 client.94400.objecter shutdown clearing up homeless session... 2024-06-11T19:20:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 10 client.94400.objecter successfully canceled tick 2024-06-11T19:20:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 1 -- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efc8803fdb0 msgr2=0x7efc88042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7efc8803fdb0 0x7efc88042270 secure :-1 s=READY pgs=4215 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009bd0 tx=0x7efc9c00b040 comp rx=0 tx=0).stop 2024-06-11T19:20:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 10 monclient: shutdown 2024-06-11T19:20:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 1 -- 172.21.15.98:0/2410797810 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efca40a2530 msgr2=0x7efca41cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 1 --2- 172.21.15.98:0/2410797810 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efca40a2530 0x7efca41cdaa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efca00096f0 tx=0x7efca00046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcaa559640 1 -- 172.21.15.98:0/2410797810 reap_dead start 2024-06-11T19:20:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcaa559640 1 -- 172.21.15.98:0/2410797810 reap_dead start 2024-06-11T19:20:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 1 -- 172.21.15.98:0/2410797810 shutdown_connections 2024-06-11T19:20:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 1 -- 172.21.15.98:0/2410797810 >> 172.21.15.98:0/2410797810 conn(0x7efca40ab850 msgr2=0x7efca41380b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 1 -- 172.21.15.98:0/2410797810 shutdown_connections 2024-06-11T19:20:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 1 -- 172.21.15.98:0/2410797810 wait complete. 2024-06-11T19:20:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.066+0000 7efcabfbc640 1 librados: shutdown 2024-06-11T19:20:05.115 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:20:05.115 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['e', 'g', 'c'] 2024-06-11T19:20:05.116 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-06-11T19:20:05.116 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-06-11T19:20:05.116 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e store 2024-06-11T19:20:05.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e sync_force --yes-i-really-mean-it 2024-06-11T19:20:05.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.217+0000 7f08465c6640 10 monclient: get_monmap_and_config 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.217+0000 7f08465c6640 10 monclient: build_initial_monmap 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.217+0000 7f08465c6640 10 monclient: monmap: 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:05.218603+0000 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:05.218603+0000 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:05.217 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:05.218 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:05.218 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:05.218 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:05.218 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:05.218 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 1 Processor -- start 2024-06-11T19:20:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 1 -- start start 2024-06-11T19:20:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 10 monclient: init 2024-06-11T19:20:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 10 monclient: _add_conns ranks=[7,2,6,3,0,10,1,8,9,5,4] 2024-06-11T19:20:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08400a4920 0x7f08400a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 10 monclient(hunting): picked mon.noname-i con 0x7f08400a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400a5240 0x7f08400a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 10 monclient(hunting): picked mon.noname-b con 0x7f08400a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08400a2530 0x7f08400a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 10 monclient(hunting): picked mon.noname-d con 0x7f08400a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f08400a7140 con 0x7f08400a5240 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f08400a2b50 con 0x7f08400a2530 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f08400a2e30 con 0x7f08400a4920 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.218+0000 7f08465c6640 10 monclient(hunting): authenticate will time out at 1415.025908s 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08400a2530 0x7f08400a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400a5240 0x7f08400a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08400a2530 0x7f08400a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:37894/0 (socket says 172.21.15.98:37894) 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 1 -- 172.21.15.98:0/1231723403 learned_addr learned my addr 172.21.15.98:0/1231723403 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083f7fe640 10 monclient(hunting): get_auth_request con 0x7f08400a5240 auth_method 0 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 10 monclient(hunting): get_auth_request con 0x7f08400a2530 auth_method 0 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083ffff640 1 --2- 172.21.15.98:0/1231723403 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08400a4920 0x7f08400a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083ffff640 10 monclient(hunting): get_auth_request con 0x7f08400a4920 auth_method 0 2024-06-11T19:20:05.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 10 monclient(hunting): handle_auth_done global_id 55410 payload 274 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 1 -- 172.21.15.98:0/1231723403 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08400a4920 msgr2=0x7f08400a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 1 --2- 172.21.15.98:0/1231723403 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08400a4920 0x7f08400a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 1 -- 172.21.15.98:0/1231723403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400a5240 msgr2=0x7f08400a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 1 --2- 172.21.15.98:0/1231723403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400a5240 0x7f08400a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 10 monclient: _finish_hunting 0 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 1 monclient: found mon.noname-d 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f0844b63640 1 -- 172.21.15.98:0/1231723403 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08400a3110 con 0x7f08400a2530 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.219+0000 7f083f7fe640 1 --2- 172.21.15.98:0/1231723403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400a5240 0x7f08400a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:05.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f0844b63640 1 --2- 172.21.15.98:0/1231723403 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08400a2530 0x7f08400a5ee0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0838009e80 tx=0x7f083802bd70 comp rx=0 tx=0).ready entity=mon.6 client_cookie=6b53648ab2912fd server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 1 -- 172.21.15.98:0/1231723403 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f083802cb80 con 0x7f08400a2530 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 10 monclient: dump: 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.232 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 10 monclient: _finish_auth 0 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 10 monclient: _check_auth_tickets 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 1 -- 172.21.15.98:0/1231723403 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f083802cd40 con 0x7f08400a2530 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 5 monclient: authenticate success, global_id 55410 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:05.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 4 set_mon_vals no callback set 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 1 -- 172.21.15.98:0/1231723403 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0838034860 con 0x7f08400a2530 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f083effd640 10 monclient: dump: 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.234 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 10 monclient: shutdown 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 1 -- 172.21.15.98:0/1231723403 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08400a2530 msgr2=0x7f08400a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 1 --2- 172.21.15.98:0/1231723403 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08400a2530 0x7f08400a5ee0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0838009e80 tx=0x7f083802bd70 comp rx=0 tx=0).stop 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 1 -- 172.21.15.98:0/1231723403 shutdown_connections 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 1 --2- 172.21.15.98:0/1231723403 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f08400a5240 0x7f08400a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 1 --2- 172.21.15.98:0/1231723403 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f08400a2530 0x7f08400a5ee0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 1 --2- 172.21.15.98:0/1231723403 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08400a4920 0x7f08400a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 1 -- 172.21.15.98:0/1231723403 >> 172.21.15.98:0/1231723403 conn(0x7f08400ab850 msgr2=0x7f08400a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 1 -- 172.21.15.98:0/1231723403 shutdown_connections 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.220+0000 7f08465c6640 1 -- 172.21.15.98:0/1231723403 wait complete. 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.221+0000 7f08465c6640 10 monclient: build_initial_monmap 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.221+0000 7f08465c6640 10 monclient: monmap: 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:05.236 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.221+0000 7f08465c6640 1 librados: starting msgr at 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.221+0000 7f08465c6640 1 librados: starting objecter 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.221+0000 7f08465c6640 1 Processor -- start 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 1 -- start start 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 1 librados: setting wanted keys 2024-06-11T19:20:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 1 librados: calling monclient init 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 10 monclient: init 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 10 monclient: _add_conns ranks=[7,4,3,5,8,2,6,1,0] 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08400a2530 0x7f08401cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 10 monclient(hunting): picked mon.noname-i con 0x7f08400a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f08400a4920 0x7f08401cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 10 monclient(hunting): picked mon.noname-c con 0x7f08400a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08400a5240 0x7f0840121e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 10 monclient(hunting): picked mon.noname-g con 0x7f08400a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f08400a6da0 con 0x7f08400a5240 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0840138e50 con 0x7f08400a4920 2024-06-11T19:20:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f08400a6420 con 0x7f08400a2530 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f08465c6640 10 monclient(hunting): authenticate will time out at 1415.029452s 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f08400a4920 0x7f08401cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f08400a4920 0x7f08401cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55348/0 (socket says 172.21.15.98:55348) 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 1 -- 172.21.15.98:0/1302860104 learned_addr learned my addr 172.21.15.98:0/1302860104 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 10 monclient(hunting): get_auth_request con 0x7f08400a4920 auth_method 0 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083ffff640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08400a2530 0x7f08401cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f0844b63640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08400a5240 0x7f0840121e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083ffff640 10 monclient(hunting): get_auth_request con 0x7f08400a2530 auth_method 0 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f0844b63640 10 monclient(hunting): get_auth_request con 0x7f08400a5240 auth_method 0 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f0844b63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f0844b63640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f0844b63640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 10 monclient(hunting): handle_auth_done global_id 24259 payload 824 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 1 -- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08400a2530 msgr2=0x7f08401cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08400a2530 0x7f08401cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 1 -- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08400a5240 msgr2=0x7f0840121e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08400a5240 0x7f0840121e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 1 monclient: found mon.noname-c 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 1 -- 172.21.15.98:0/1302860104 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0840138760 con 0x7f08400a4920 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083ffff640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f08400a2530 0x7f08401cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f0844b63640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08400a5240 0x7f0840121e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083f7fe640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f08400a4920 0x7f08401cdfe0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f083400e8d0 tx=0x7f083400ed90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083cff9640 10 client.?.objecter ms_handle_connect 0x7f08400a4920 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.222+0000 7f083cff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0834017070 con 0x7f08400a4920 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 10 monclient: dump: 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 10 monclient: _finish_auth 0 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 10 monclient: _check_auth_tickets 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f083400cdb0 con 0x7f08400a4920 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f08465c6640 5 monclient: authenticate success, global_id 24259 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f08465c6640 10 monclient: _renew_subs 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f08465c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f08465c6640 1 -- 172.21.15.98:0/1302860104 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0840125870 con 0x7f08400a4920 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08340138e0 con 0x7f08400a4920 2024-06-11T19:20:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f08465c6640 10 client.24259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083cff9640 10 monclient: dump: 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f08465c6640 10 monclient: _renew_subs 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f083e7fc640 4 set_mon_vals no callback set 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f08465c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f08465c6640 1 -- 172.21.15.98:0/1302860104 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f0840125e30 con 0x7f08400a4920 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.223+0000 7f08465c6640 1 librados: init done 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f08465c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f08465c6640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122b70 0x7f084002cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f08465c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f08465c6640 1 -- 172.21.15.98:0/1302860104 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0840124050 con 0x7f0840122b70 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083ffff640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122b70 0x7f084002cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f083401a070 con 0x7f08400a4920 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083ffff640 10 monclient: get_auth_request con 0x7f0840122b70 auth_method 0 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083ffff640 10 monclient(hunting): handle_auth_done global_id 94409 payload 824 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083ffff640 1 -- 172.21.15.98:0/1302860104 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0840123b40 con 0x7f0840122b70 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f082c03fdb0 0x7f082c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(422..422 src has 1..422) ==== 6217+0+0 (secure 0 0 0) 0x7f083400f040 con 0x7f08400a4920 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 10 client.24259.objecter ms_dispatch 0x7f0840000b90 osd_map(422..422 src has 1..422) 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 3 client.24259.objecter handle_osd_map got epochs [422,422] > 0 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 3 client.24259.objecter handle_osd_map decoding full epoch 422 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083ffff640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122b70 0x7f084002cb40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f083000f550 tx=0x7f083000fa10 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 20 client.24259.objecter dump_active .. 0 homeless 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 10 client.24259.objecter ms_handle_connect 0x7f0840122b70 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 10 client.24259.objecter resend_mon_ops 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f0844b63640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f082c03fdb0 0x7f082c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f0844b63640 10 monclient: get_auth_request con 0x7f082c03fdb0 auth_method 0 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08300044a0 con 0x7f0840122b70 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:05.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 10 monclient: dump: 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.249 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0830004030 con 0x7f0840122b70 2024-06-11T19:20:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122b70 msgr2=0x7f084002cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122b70 0x7f084002cb40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f083000f550 tx=0x7f083000fa10 comp rx=0 tx=0).stop 2024-06-11T19:20:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.224+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122b70 msgr2=0x7f084002cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.225+0000 7f083cff9640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122b70 0x7f084002cb40 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f083000f550 tx=0x7f083000fa10 comp rx=0 tx=0).stop 2024-06-11T19:20:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.225+0000 7f0844b63640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f082c03fdb0 0x7f082c042270 secure :-1 s=READY pgs=4216 cs=0 l=1 rev1=1 crypto rx=0x7f0838035000 tx=0x7f0838030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.225+0000 7f083cff9640 10 client.24259.objecter ms_handle_connect 0x7f082c03fdb0 2024-06-11T19:20:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f08465c6640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:20:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f08465c6640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122f50 0x7f084002d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f08465c6640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f08465c6640 1 -- 172.21.15.98:0/1302860104 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0840050830 con 0x7f0840122f50 2024-06-11T19:20:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083ffff640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122f50 0x7f084002d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083ffff640 10 monclient: get_auth_request con 0x7f0840122f50 auth_method 0 2024-06-11T19:20:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083ffff640 10 monclient(hunting): handle_auth_done global_id 94418 payload 824 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083ffff640 1 -- 172.21.15.98:0/1302860104 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f08400a6720 con 0x7f0840122f50 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083cff9640 10 client.24259.objecter ms_handle_connect 0x7f0840122f50 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083cff9640 10 client.24259.objecter resend_mon_ops 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083ffff640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122f50 0x7f084002d080 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f083000f550 tx=0x7f08300046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0830003610 con 0x7f0840122f50 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083cff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:20:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.232+0000 7f083cff9640 10 monclient: dump: 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.254 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f0830014a20 con 0x7f0840122f50 2024-06-11T19:20:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f083cff9640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:20:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f083cff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122f50 msgr2=0x7f084002d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f083cff9640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122f50 0x7f084002d080 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f083000f550 tx=0x7f08300046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f083cff9640 1 -- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122f50 msgr2=0x7f084002d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f083cff9640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0840122f50 0x7f084002d080 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f083000f550 tx=0x7f08300046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.254 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:05.254 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-06-11T19:20:05.255 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-06-11T19:20:05.255 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 10 librados: watch_flush enter 2024-06-11T19:20:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 10 librados: watch_flush exit 2024-06-11T19:20:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 20 client.24259.objecter shutdown clearing up homeless session... 2024-06-11T19:20:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 10 client.24259.objecter successfully canceled tick 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 1 -- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f082c03fdb0 msgr2=0x7f082c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f082c03fdb0 0x7f082c042270 secure :-1 s=READY pgs=4216 cs=0 l=1 rev1=1 crypto rx=0x7f0838035000 tx=0x7f0838030000 comp rx=0 tx=0).stop 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 10 monclient: shutdown 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 1 -- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f08400a4920 msgr2=0x7f08401cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f08400a4920 0x7f08401cdfe0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f083400e8d0 tx=0x7f083400ed90 comp rx=0 tx=0).stop 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f0844b63640 1 -- 172.21.15.98:0/1302860104 reap_dead start 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 1 -- 172.21.15.98:0/1302860104 shutdown_connections 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 1 --2- 172.21.15.98:0/1302860104 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f08400a4920 0x7f08401cdfe0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 1 -- 172.21.15.98:0/1302860104 >> 172.21.15.98:0/1302860104 conn(0x7f08400ab850 msgr2=0x7f08400a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 1 -- 172.21.15.98:0/1302860104 shutdown_connections 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 1 -- 172.21.15.98:0/1302860104 wait complete. 2024-06-11T19:20:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.233+0000 7f08465c6640 1 librados: shutdown 2024-06-11T19:20:05.258 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-06-11T19:20:05.258 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-06-11T19:20:05.258 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:20:05.270 INFO:tasks.ceph.mon.e:Stopped 2024-06-11T19:20:05.270 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-06-11T19:20:05.270 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-06-11T19:20:05.270 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-06-11T19:20:05.270 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:20:05.281 INFO:tasks.ceph.mon.g:Stopped 2024-06-11T19:20:05.281 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-06-11T19:20:05.282 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c store 2024-06-11T19:20:05.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c sync_force --yes-i-really-mean-it 2024-06-11T19:20:05.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.411+0000 7f1a0e55c640 10 monclient: get_monmap_and_config 2024-06-11T19:20:05.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.411+0000 7f1a0e55c640 10 monclient: build_initial_monmap 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.411+0000 7f1a0e55c640 10 monclient: monmap: 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:05.412589+0000 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:05.412589+0000 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:05.411 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:05.412 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:05.412 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:05.412 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:05.412 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:05.412 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 1 Processor -- start 2024-06-11T19:20:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 1 -- start start 2024-06-11T19:20:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 10 monclient: init 2024-06-11T19:20:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 10 monclient: _add_conns ranks=[8,4,7,6,5,0,1,2,9,3,10] 2024-06-11T19:20:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a080a4920 0x7f1a080a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 10 monclient(hunting): picked mon.noname-e con 0x7f1a080a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a5240 0x7f1a080a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 10 monclient(hunting): picked mon.noname-c con 0x7f1a080a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1a080a2530 0x7f1a080a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 10 monclient(hunting): picked mon.noname-i con 0x7f1a080a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:05.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1a080a7140 con 0x7f1a080a5240 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f1a080a2b50 con 0x7f1a080a2530 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f1a080a2e30 con 0x7f1a080a4920 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a0e55c640 10 monclient(hunting): authenticate will time out at 1415.219897s 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a07fff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a080a4920 msgr2=0x7f1a080a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a080a4920 0x7f1a080a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.412+0000 7f1a077fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a5240 0x7f1a080a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a5240 0x7f1a080a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55364/0 (socket says 172.21.15.98:55364) 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 1 -- 172.21.15.98:0/2081912148 learned_addr learned my addr 172.21.15.98:0/2081912148 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 10 monclient(hunting): get_auth_request con 0x7f1a080a5240 auth_method 0 2024-06-11T19:20:05.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a0caf9640 1 --2- 172.21.15.98:0/2081912148 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1a080a2530 0x7f1a080a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 10 monclient(hunting): handle_auth_done global_id 24268 payload 274 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 1 -- 172.21.15.98:0/2081912148 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a080a4920 msgr2=0x7f1a080a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 1 --2- 172.21.15.98:0/2081912148 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a080a4920 0x7f1a080a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 1 -- 172.21.15.98:0/2081912148 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1a080a2530 msgr2=0x7f1a080a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 1 --2- 172.21.15.98:0/2081912148 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1a080a2530 0x7f1a080a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 10 monclient: _finish_hunting 0 2024-06-11T19:20:05.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 1 monclient: found mon.noname-c 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 1 -- 172.21.15.98:0/2081912148 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a080a3110 con 0x7f1a080a5240 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a0caf9640 10 monclient: get_auth_request con 0x7f1a080a2530 auth_method 0 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a0caf9640 1 --2- 172.21.15.98:0/2081912148 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1a080a2530 0x7f1a080a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.413+0000 7f1a077fe640 1 --2- 172.21.15.98:0/2081912148 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a5240 0x7f1a080a59a0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f19f00098a0 tx=0x7f19f002b7c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=e3d135d2e908022a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 1 -- 172.21.15.98:0/2081912148 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f19f0038070 con 0x7f1a080a5240 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 10 monclient: dump: 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.425 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.426 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.426 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.426 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.426 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.426 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.426 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.426 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.426 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.426 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.426 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.426 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.426 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 10 monclient: _finish_auth 0 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 1 -- 172.21.15.98:0/2081912148 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19f0031990 con 0x7f1a080a5240 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 5 monclient: authenticate success, global_id 24268 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 1 -- 172.21.15.98:0/2081912148 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f19f00323b0 con 0x7f1a080a5240 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 4 set_mon_vals no callback set 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:05.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a06ffd640 10 monclient: dump: 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.428 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 10 monclient: shutdown 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 1 -- 172.21.15.98:0/2081912148 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a5240 msgr2=0x7f1a080a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 1 --2- 172.21.15.98:0/2081912148 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a5240 0x7f1a080a59a0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f19f00098a0 tx=0x7f19f002b7c0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 1 -- 172.21.15.98:0/2081912148 shutdown_connections 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 1 --2- 172.21.15.98:0/2081912148 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1a080a2530 0x7f1a080a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 1 --2- 172.21.15.98:0/2081912148 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a5240 0x7f1a080a59a0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 1 --2- 172.21.15.98:0/2081912148 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1a080a4920 0x7f1a080a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 1 -- 172.21.15.98:0/2081912148 >> 172.21.15.98:0/2081912148 conn(0x7f1a080ab850 msgr2=0x7f1a080a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 1 -- 172.21.15.98:0/2081912148 shutdown_connections 2024-06-11T19:20:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.414+0000 7f1a0e55c640 1 -- 172.21.15.98:0/2081912148 wait complete. 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.415+0000 7f1a0e55c640 10 monclient: build_initial_monmap 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.415+0000 7f1a0e55c640 10 monclient: monmap: 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.415+0000 7f1a0e55c640 1 librados: starting msgr at 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.415+0000 7f1a0e55c640 1 librados: starting objecter 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.415+0000 7f1a0e55c640 1 Processor -- start 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.415+0000 7f1a0e55c640 1 -- start start 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.415+0000 7f1a0e55c640 1 librados: setting wanted keys 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.415+0000 7f1a0e55c640 1 librados: calling monclient init 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.415+0000 7f1a0e55c640 10 monclient: init 2024-06-11T19:20:05.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 10 monclient: _add_conns ranks=[1,4,2,3,7,6,0,8,5] 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a080a2530 0x7f1a081cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 10 monclient(hunting): picked mon.noname-f con 0x7f1a080a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a4920 0x7f1a081ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 10 monclient(hunting): picked mon.noname-c con 0x7f1a080a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a080a5240 0x7f1a08123160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 10 monclient(hunting): picked mon.noname-b con 0x7f1a080a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1a080a6da0 con 0x7f1a080a2530 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1a08138e50 con 0x7f1a080a5240 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1a080a6420 con 0x7f1a080a4920 2024-06-11T19:20:05.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 10 monclient(hunting): authenticate will time out at 1415.223228s 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a4920 0x7f1a081ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a4920 0x7f1a081ce1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55374/0 (socket says 172.21.15.98:55374) 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 1 -- 172.21.15.98:0/1798864950 learned_addr learned my addr 172.21.15.98:0/1798864950 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 10 monclient(hunting): get_auth_request con 0x7f1a080a4920 auth_method 0 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0caf9640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a080a5240 0x7f1a08123160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0caf9640 10 monclient(hunting): get_auth_request con 0x7f1a080a5240 auth_method 0 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0caf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0caf9640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0caf9640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0caf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0caf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0caf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a07fff640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a080a2530 0x7f1a081cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 10 monclient(hunting): handle_auth_done global_id 24277 payload 824 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 1 -- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a080a5240 msgr2=0x7f1a08123160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1a080a5240 0x7f1a08123160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 1 -- 172.21.15.98:0/1798864950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a080a2530 msgr2=0x7f1a081cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a080a2530 0x7f1a081cdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 10 monclient: _finish_hunting 0 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 1 monclient: found mon.noname-c 2024-06-11T19:20:05.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 1 -- 172.21.15.98:0/1798864950 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a080a8da0 con 0x7f1a080a4920 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a07fff640 10 monclient: get_auth_request con 0x7f1a080a2530 auth_method 0 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a07fff640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a080a2530 0x7f1a081cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a077fe640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a4920 0x7f1a081ce1e0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f19f0000c00 tx=0x7f19f00046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a04ff9640 10 client.?.objecter ms_handle_connect 0x7f1a080a4920 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a04ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f19f0038050 con 0x7f1a080a4920 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a04ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a04ff9640 10 monclient: dump: 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a04ff9640 10 monclient: _finish_auth 0 2024-06-11T19:20:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a04ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19f0039040 con 0x7f1a080a4920 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 5 monclient: authenticate success, global_id 24277 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 10 monclient: _renew_subs 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.416+0000 7f1a0e55c640 1 -- 172.21.15.98:0/1798864950 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a08124b60 con 0x7f1a080a4920 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a04ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f19f0004450 con 0x7f1a080a4920 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a04ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a04ff9640 10 monclient: dump: 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a0e55c640 10 client.24277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a0e55c640 10 monclient: _renew_subs 2024-06-11T19:20:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a0e55c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:20:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a067fc640 4 set_mon_vals no callback set 2024-06-11T19:20:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a0e55c640 1 -- 172.21.15.98:0/1798864950 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a08126620 con 0x7f1a080a4920 2024-06-11T19:20:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a0e55c640 1 librados: init done 2024-06-11T19:20:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f19f0046030 con 0x7f1a080a4920 2024-06-11T19:20:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a0e55c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a0e55c640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a081236a0 0x7f1a08123a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a0e55c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.417+0000 7f1a0e55c640 1 -- 172.21.15.98:0/1798864950 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1a08050830 con 0x7f1a081236a0 2024-06-11T19:20:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f19e803fde0 0x7f19e80422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a07fff640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a081236a0 0x7f1a08123a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a07fff640 10 monclient: get_auth_request con 0x7f1a081236a0 auth_method 0 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a07fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a07fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(422..422 src has 1..422) ==== 6217+0+0 (secure 0 0 0) 0x7f19f0044020 con 0x7f1a080a4920 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 10 client.24277.objecter ms_dispatch 0x7f1a08000b90 osd_map(422..422 src has 1..422) 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 3 client.24277.objecter handle_osd_map got epochs [422,422] > 0 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 3 client.24277.objecter handle_osd_map decoding full epoch 422 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 20 client.24277.objecter dump_active .. 0 homeless 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a07fff640 10 monclient(hunting): handle_auth_done global_id 24286 payload 824 2024-06-11T19:20:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a07fff640 1 -- 172.21.15.98:0/1798864950 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a08126860 con 0x7f1a081236a0 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a0caf9640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f19e803fde0 0x7f19e80422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a0caf9640 10 monclient: get_auth_request con 0x7f19e803fde0 auth_method 0 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 10 client.24277.objecter ms_handle_connect 0x7f1a081236a0 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 10 client.24277.objecter resend_mon_ops 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a07fff640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a081236a0 0x7f1a08123a80 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f19f800c6f0 tx=0x7f19f800cbb0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f19f80039b0 con 0x7f1a081236a0 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 10 monclient: dump: 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 10 client.24277.objecter ms_handle_connect 0x7f19e803fde0 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a0caf9640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f19e803fde0 0x7f19e80422a0 secure :-1 s=READY pgs=4217 cs=0 l=1 rev1=1 crypto rx=0x7f1a081248c0 tx=0x7f1a00007780 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f19f8005a40 con 0x7f1a081236a0 2024-06-11T19:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a081236a0 msgr2=0x7f1a08123a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a081236a0 0x7f1a08123a80 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f19f800c6f0 tx=0x7f19f800cbb0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a081236a0 msgr2=0x7f1a08123a80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.418+0000 7f1a04ff9640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a081236a0 0x7f1a08123a80 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a0e55c640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:20:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a0e55c640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a08123fc0 0x7f1a0809e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a0e55c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a0e55c640 1 -- 172.21.15.98:0/1798864950 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1a080a66f0 con 0x7f1a08123fc0 2024-06-11T19:20:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a07fff640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a08123fc0 0x7f1a0809e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a07fff640 10 monclient: get_auth_request con 0x7f1a08123fc0 auth_method 0 2024-06-11T19:20:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a07fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a07fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a07fff640 10 monclient(hunting): handle_auth_done global_id 24295 payload 824 2024-06-11T19:20:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a07fff640 1 -- 172.21.15.98:0/1798864950 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f1a08126a70 con 0x7f1a08123fc0 2024-06-11T19:20:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a07fff640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a08123fc0 0x7f1a0809e9f0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f19f8000c00 tx=0x7f19f80059d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a04ff9640 10 client.24277.objecter ms_handle_connect 0x7f1a08123fc0 2024-06-11T19:20:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a04ff9640 10 client.24277.objecter resend_mon_ops 2024-06-11T19:20:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f19f801c580 con 0x7f1a08123fc0 2024-06-11T19:20:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a04ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:20:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.426+0000 7f1a04ff9640 10 monclient: dump: 2024-06-11T19:20:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f19f801cdb0 con 0x7f1a08123fc0 2024-06-11T19:20:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a04ff9640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:20:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a04ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a08123fc0 msgr2=0x7f1a0809e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a04ff9640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a08123fc0 0x7f1a0809e9f0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f19f8000c00 tx=0x7f19f80059d0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a04ff9640 1 -- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a08123fc0 msgr2=0x7f1a0809e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a04ff9640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a08123fc0 0x7f1a0809e9f0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.448 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:05.448 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-06-11T19:20:05.448 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-06-11T19:20:05.448 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 10 librados: watch_flush enter 2024-06-11T19:20:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 10 librados: watch_flush exit 2024-06-11T19:20:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 20 client.24277.objecter shutdown clearing up homeless session... 2024-06-11T19:20:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 10 client.24277.objecter successfully canceled tick 2024-06-11T19:20:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 1 -- 172.21.15.98:0/1798864950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f19e803fde0 msgr2=0x7f19e80422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f19e803fde0 0x7f19e80422a0 secure :-1 s=READY pgs=4217 cs=0 l=1 rev1=1 crypto rx=0x7f1a081248c0 tx=0x7f1a00007780 comp rx=0 tx=0).stop 2024-06-11T19:20:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 10 monclient: shutdown 2024-06-11T19:20:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 1 -- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a4920 msgr2=0x7f1a081ce1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0caf9640 1 -- 172.21.15.98:0/1798864950 reap_dead start 2024-06-11T19:20:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a4920 0x7f1a081ce1e0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f19f0000c00 tx=0x7f19f00046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 1 -- 172.21.15.98:0/1798864950 shutdown_connections 2024-06-11T19:20:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 1 --2- 172.21.15.98:0/1798864950 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1a080a4920 0x7f1a081ce1e0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 1 -- 172.21.15.98:0/1798864950 >> 172.21.15.98:0/1798864950 conn(0x7f1a080ab850 msgr2=0x7f1a081be740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 1 -- 172.21.15.98:0/1798864950 shutdown_connections 2024-06-11T19:20:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 1 -- 172.21.15.98:0/1798864950 wait complete. 2024-06-11T19:20:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.427+0000 7f1a0e55c640 1 librados: shutdown 2024-06-11T19:20:05.452 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-06-11T19:20:05.452 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-06-11T19:20:05.453 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:20:05.468 INFO:tasks.ceph.mon.c:Stopped 2024-06-11T19:20:05.468 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 6 2024-06-11T19:20:05.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:20:05.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.607+0000 7ff634e0a640 10 monclient: get_monmap_and_config 2024-06-11T19:20:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.607+0000 7ff634e0a640 10 monclient: build_initial_monmap 2024-06-11T19:20:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.607+0000 7ff634e0a640 10 monclient: monmap: 2024-06-11T19:20:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:05.608607+0000 2024-06-11T19:20:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:05.608607+0000 2024-06-11T19:20:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:05.607 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:05.608 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:05.608 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:05.608 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:05.608 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:05.608 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:05.608 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:05.608 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:05.608 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 1 Processor -- start 2024-06-11T19:20:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 1 -- start start 2024-06-11T19:20:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 10 monclient: init 2024-06-11T19:20:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 10 monclient: _add_conns ranks=[7,3,5,1,9,4,8,6,0,10,2] 2024-06-11T19:20:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6300a4920 0x7ff6300a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 10 monclient(hunting): picked mon.noname-i con 0x7ff6300a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6300a5240 0x7ff6300a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 10 monclient(hunting): picked mon.noname-g con 0x7ff6300a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff6300a2530 0x7ff6300a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 10 monclient(hunting): picked mon.noname-h con 0x7ff6300a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ff6300a7140 con 0x7ff6300a5240 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ff6300a2b50 con 0x7ff6300a2530 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff6300a2e30 con 0x7ff6300a4920 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff634e0a640 10 monclient(hunting): authenticate will time out at 1415.415829s 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff627fff640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6300a5240 msgr2=0x7ff6300a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.608+0000 7ff627fff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6300a5240 0x7ff6300a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62e59c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6300a4920 0x7ff6300a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff6300a2530 0x7ff6300a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62e59c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6300a4920 0x7ff6300a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:48440/0 (socket says 172.21.15.98:48440) 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff6300a2530 0x7ff6300a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:43358/0 (socket says 172.21.15.98:43358) 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62e59c640 1 -- 172.21.15.98:0/867892266 learned_addr learned my addr 172.21.15.98:0/867892266 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62e59c640 10 monclient(hunting): get_auth_request con 0x7ff6300a4920 auth_method 0 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 10 monclient(hunting): get_auth_request con 0x7ff6300a2530 auth_method 0 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 10 monclient(hunting): handle_auth_done global_id 23333 payload 274 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 1 -- 172.21.15.98:0/867892266 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6300a4920 msgr2=0x7ff6300a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 1 --2- 172.21.15.98:0/867892266 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6300a4920 0x7ff6300a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 1 -- 172.21.15.98:0/867892266 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6300a5240 msgr2=0x7ff6300a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 1 --2- 172.21.15.98:0/867892266 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6300a5240 0x7ff6300a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 10 monclient: _finish_hunting 0 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 1 monclient: found mon.noname-h 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62ed9d640 1 -- 172.21.15.98:0/867892266 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6300a3110 con 0x7ff6300a2530 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.609+0000 7ff62e59c640 1 --2- 172.21.15.98:0/867892266 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6300a4920 0x7ff6300a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62ed9d640 1 --2- 172.21.15.98:0/867892266 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff6300a2530 0x7ff6300a5ee0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7ff628009dc0 tx=0x7ff62802bce0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=7db591d840224bd server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 1 -- 172.21.15.98:0/867892266 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff628039070 con 0x7ff6300a2530 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 10 monclient: dump: 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 10 monclient: _finish_auth 0 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 10 monclient: _check_auth_tickets 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 1 -- 172.21.15.98:0/867892266 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff62802cd00 con 0x7ff6300a2530 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 5 monclient: authenticate success, global_id 23333 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 1 -- 172.21.15.98:0/867892266 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff6280327b0 con 0x7ff6300a2530 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 4 set_mon_vals no callback set 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff62dd9b640 10 monclient: dump: 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 10 monclient: shutdown 2024-06-11T19:20:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 1 -- 172.21.15.98:0/867892266 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff6300a2530 msgr2=0x7ff6300a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 1 --2- 172.21.15.98:0/867892266 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff6300a2530 0x7ff6300a5ee0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7ff628009dc0 tx=0x7ff62802bce0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 1 -- 172.21.15.98:0/867892266 shutdown_connections 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 1 --2- 172.21.15.98:0/867892266 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff6300a2530 0x7ff6300a5ee0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 1 --2- 172.21.15.98:0/867892266 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6300a5240 0x7ff6300a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 1 --2- 172.21.15.98:0/867892266 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6300a4920 0x7ff6300a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 1 -- 172.21.15.98:0/867892266 >> 172.21.15.98:0/867892266 conn(0x7ff6300ab850 msgr2=0x7ff6300a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 1 -- 172.21.15.98:0/867892266 shutdown_connections 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.610+0000 7ff634e0a640 1 -- 172.21.15.98:0/867892266 wait complete. 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.611+0000 7ff634e0a640 10 monclient: build_initial_monmap 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.611+0000 7ff634e0a640 10 monclient: monmap: 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:05.628 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.611+0000 7ff634e0a640 1 librados: starting msgr at 2024-06-11T19:20:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.611+0000 7ff634e0a640 1 librados: starting objecter 2024-06-11T19:20:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.611+0000 7ff634e0a640 1 Processor -- start 2024-06-11T19:20:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 1 -- start start 2024-06-11T19:20:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 1 librados: setting wanted keys 2024-06-11T19:20:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 1 librados: calling monclient init 2024-06-11T19:20:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 10 monclient: init 2024-06-11T19:20:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 10 monclient: _add_conns ranks=[0,7,1,5,8,2,4,6,3] 2024-06-11T19:20:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6300a2530 0x7ff6301cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 10 monclient(hunting): picked mon.noname-a con 0x7ff6300a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6300a4920 0x7ff6301ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 10 monclient(hunting): picked mon.noname-i con 0x7ff6300a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6300a5240 0x7ff630123f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 10 monclient(hunting): picked mon.noname-f con 0x7ff6300a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6300a6da0 con 0x7ff6300a2530 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff630138e50 con 0x7ff6300a5240 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff6300a6420 con 0x7ff6300a4920 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff634e0a640 10 monclient(hunting): authenticate will time out at 1415.419515s 2024-06-11T19:20:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6300a2530 0x7ff6301cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6300a2530 0x7ff6301cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54326/0 (socket says 172.21.15.98:54326) 2024-06-11T19:20:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 1 -- 172.21.15.98:0/2293014134 learned_addr learned my addr 172.21.15.98:0/2293014134 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 10 monclient(hunting): get_auth_request con 0x7ff6300a2530 auth_method 0 2024-06-11T19:20:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62ed9d640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6300a5240 0x7ff630123f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff627fff640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6300a4920 0x7ff6301ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62ed9d640 10 monclient(hunting): get_auth_request con 0x7ff6300a5240 auth_method 0 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62ed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff627fff640 10 monclient(hunting): get_auth_request con 0x7ff6300a4920 auth_method 0 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff627fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff627fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff627fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff627fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff627fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff627fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 10 monclient(hunting): handle_auth_done global_id 65421 payload 824 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 1 -- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6300a4920 msgr2=0x7ff6301ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6300a4920 0x7ff6301ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 1 -- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6300a5240 msgr2=0x7ff630123f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6300a5240 0x7ff630123f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 10 monclient: _finish_hunting 0 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 1 monclient: found mon.noname-a 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.612+0000 7ff62e59c640 1 -- 172.21.15.98:0/2293014134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff630126730 con 0x7ff6300a2530 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff62e59c640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6300a2530 0x7ff6301cdad0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff62000e9d0 tx=0x7ff62000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 10 client.?.objecter ms_handle_connect 0x7ff6300a2530 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 1 -- 172.21.15.98:0/2293014134 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff620017070 con 0x7ff6300a2530 2024-06-11T19:20:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 10 monclient: dump: 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 10 monclient: _finish_auth 0 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff634e0a640 5 monclient: authenticate success, global_id 65421 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff634e0a640 10 monclient: _renew_subs 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 1 -- 172.21.15.98:0/2293014134 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff620013470 con 0x7ff6300a2530 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff634e0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff634e0a640 1 -- 172.21.15.98:0/2293014134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff630127ec0 con 0x7ff6300a2530 2024-06-11T19:20:05.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 1 -- 172.21.15.98:0/2293014134 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff620013a90 con 0x7ff6300a2530 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff626ffd640 10 monclient: dump: 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff62d59a640 4 set_mon_vals no callback set 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff634e0a640 10 client.65421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff634e0a640 10 monclient: _renew_subs 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff634e0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff634e0a640 1 -- 172.21.15.98:0/2293014134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff630127900 con 0x7ff6300a2530 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.613+0000 7ff634e0a640 1 librados: init done 2024-06-11T19:20:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff626ffd640 1 -- 172.21.15.98:0/2293014134 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff62001a070 con 0x7ff6300a2530 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff626ffd640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff61003fce0 0x7ff6100421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff634e0a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff634e0a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff634e0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff626ffd640 1 -- 172.21.15.98:0/2293014134 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(422..422 src has 1..422) ==== 6217+0+0 (secure 0 0 0) 0x7ff62000f040 con 0x7ff6300a2530 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff626ffd640 10 client.65421.objecter ms_dispatch 0x7ff630000b90 osd_map(422..422 src has 1..422) 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff626ffd640 3 client.65421.objecter handle_osd_map got epochs [422,422] > 0 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff626ffd640 3 client.65421.objecter handle_osd_map decoding full epoch 422 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff634e0a640 1 -- 172.21.15.98:0/2293014134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6300a5060 con 0x7ff6300a2530 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff626ffd640 20 client.65421.objecter dump_active .. 0 homeless 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff627fff640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff61003fce0 0x7ff6100421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff627fff640 10 monclient: get_auth_request con 0x7ff61003fce0 auth_method 0 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff626ffd640 10 client.65421.objecter ms_handle_connect 0x7ff61003fce0 2024-06-11T19:20:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.614+0000 7ff627fff640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff61003fce0 0x7ff6100421a0 secure :-1 s=READY pgs=4218 cs=0 l=1 rev1=1 crypto rx=0x7ff618006b40 tx=0x7ff618006ad0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:05.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.618+0000 7ff626ffd640 1 -- 172.21.15.98:0/2293014134 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff6300a5060 con 0x7ff6300a2530 2024-06-11T19:20:05.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.618+0000 7ff626ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:05.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.618+0000 7ff626ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:05.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.875+0000 7ff634e0a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:20:05.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.875+0000 7ff634e0a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:20:05.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.875+0000 7ff634e0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:20:05.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.876+0000 7ff634e0a640 1 -- 172.21.15.98:0/2293014134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff6300a66f0 con 0x7ff6300a2530 2024-06-11T19:20:05.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.876+0000 7ff626ffd640 1 -- 172.21.15.98:0/2293014134 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7ff620021070 con 0x7ff6300a2530 2024-06-11T19:20:05.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.876+0000 7ff626ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:20:05.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.876+0000 7ff626ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:05.876 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":54,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:20:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 10 librados: watch_flush enter 2024-06-11T19:20:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 10 librados: watch_flush exit 2024-06-11T19:20:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 20 client.65421.objecter shutdown clearing up homeless session... 2024-06-11T19:20:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 10 client.65421.objecter successfully canceled tick 2024-06-11T19:20:05.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 -- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff61003fce0 msgr2=0x7ff6100421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff61003fce0 0x7ff6100421a0 secure :-1 s=READY pgs=4218 cs=0 l=1 rev1=1 crypto rx=0x7ff618006b40 tx=0x7ff618006ad0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 10 monclient: shutdown 2024-06-11T19:20:05.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:05.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 -- 172.21.15.98:0/2293014134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6300a2530 msgr2=0x7ff6301cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:05.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6300a2530 0x7ff6301cdad0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff62000e9d0 tx=0x7ff62000ee90 comp rx=0 tx=0).stop 2024-06-11T19:20:05.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 -- 172.21.15.98:0/2293014134 shutdown_connections 2024-06-11T19:20:05.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff61003fce0 0x7ff6100421a0 unknown :-1 s=CLOSED pgs=4218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6300a5240 0x7ff630123f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff6300a4920 0x7ff6301ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 --2- 172.21.15.98:0/2293014134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6300a2530 0x7ff6301cdad0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:05.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 -- 172.21.15.98:0/2293014134 >> 172.21.15.98:0/2293014134 conn(0x7ff6300ab850 msgr2=0x7ff630138060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:05.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 -- 172.21.15.98:0/2293014134 shutdown_connections 2024-06-11T19:20:05.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 -- 172.21.15.98:0/2293014134 wait complete. 2024-06-11T19:20:05.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:05.878+0000 7ff634e0a640 1 librados: shutdown 2024-06-11T19:20:08.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:20:08.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.000+0000 7f9360b5f640 10 monclient: get_monmap_and_config 2024-06-11T19:20:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.000+0000 7f9360b5f640 10 monclient: build_initial_monmap 2024-06-11T19:20:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.000+0000 7f9360b5f640 10 monclient: monmap: 2024-06-11T19:20:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:09.001861+0000 2024-06-11T19:20:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:09.001861+0000 2024-06-11T19:20:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:09.000 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:09.001 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:09.001 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.001+0000 7f9360b5f640 1 Processor -- start 2024-06-11T19:20:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.001+0000 7f9360b5f640 1 -- start start 2024-06-11T19:20:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.001+0000 7f9360b5f640 10 monclient: init 2024-06-11T19:20:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 10 monclient: _add_conns ranks=[7,10,1,8,6,0,2,9,4,5,3] 2024-06-11T19:20:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f935c0a4920 0x7f935c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 10 monclient(hunting): picked mon.noname-i con 0x7f935c0a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f935c0a5240 0x7f935c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 10 monclient(hunting): picked mon.noname-f con 0x7f935c0a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f935c0a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:20:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f935c0a7140 con 0x7f935c0a2530 2024-06-11T19:20:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f935c0a2b50 con 0x7f935c0a5240 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f935c0a2e30 con 0x7f935c0a4920 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9360b5f640 10 monclient(hunting): authenticate will time out at 1418.809319s 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f935c0a4920 0x7f935c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9359d9b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f935c0a5240 0x7f935c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f935c0a4920 0x7f935c0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:48456/0 (socket says 172.21.15.98:48456) 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 1 -- 172.21.15.98:0/599925455 learned_addr learned my addr 172.21.15.98:0/599925455 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 10 monclient(hunting): get_auth_request con 0x7f935c0a4920 auth_method 0 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9359d9b640 10 monclient(hunting): get_auth_request con 0x7f935c0a5240 auth_method 0 2024-06-11T19:20:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9359d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9359d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9359d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9359d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9359d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9359d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 10 monclient(hunting): handle_auth_done global_id 84580 payload 274 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 1 -- 172.21.15.98:0/599925455 >> v2:172.21.15.117:3300/0 conn(0x7f935c0a5240 msgr2=0x7f935c0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 1 --2- 172.21.15.98:0/599925455 >> v2:172.21.15.117:3300/0 conn(0x7f935c0a5240 0x7f935c0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 1 -- 172.21.15.98:0/599925455 >> v1:172.21.15.117:6789/0 conn(0x7f935c0a2530 legacy=0x7f935c0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 10 monclient: _finish_hunting 0 2024-06-11T19:20:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 1 monclient: found mon.noname-i 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935959a640 1 -- 172.21.15.98:0/599925455 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1238153306 0 0) 0x7f935c0a7140 con 0x7f935c0a2530 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935a59c640 1 -- 172.21.15.98:0/599925455 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f935c0a3110 con 0x7f935c0a4920 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f9359d9b640 1 --2- 172.21.15.98:0/599925455 >> v2:172.21.15.117:3300/0 conn(0x7f935c0a5240 0x7f935c0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.002+0000 7f935959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935a59c640 1 --2- 172.21.15.98:0/599925455 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f935c0a4920 0x7f935c0a4d00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f934c009870 tx=0x7f934c02b860 comp rx=0 tx=0).ready entity=mon.7 client_cookie=e8503af7b5617279 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 1 -- 172.21.15.98:0/599925455 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f934c031a30 con 0x7f935c0a4920 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 10 monclient: dump: 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 10 monclient: _finish_auth 0 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 10 monclient: _check_auth_tickets 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f9360b5f640 5 monclient: authenticate success, global_id 84580 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f9360b5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 1 -- 172.21.15.98:0/599925455 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f934c037070 con 0x7f935c0a4920 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f9360b5f640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 1 -- 172.21.15.98:0/599925455 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f934c031bd0 con 0x7f935c0a4920 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f9360b5f640 4 set_mon_vals no callback set 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.003+0000 7f935959a640 10 monclient: dump: 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.004+0000 7f9360b5f640 10 monclient: shutdown 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.004+0000 7f9360b5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.004+0000 7f9360b5f640 1 -- 172.21.15.98:0/599925455 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f935c0a4920 msgr2=0x7f935c0a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.004+0000 7f9360b5f640 1 --2- 172.21.15.98:0/599925455 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f935c0a4920 0x7f935c0a4d00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f934c009870 tx=0x7f934c02b860 comp rx=0 tx=0).stop 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.004+0000 7f9360b5f640 1 -- 172.21.15.98:0/599925455 shutdown_connections 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.004+0000 7f9360b5f640 1 --2- 172.21.15.98:0/599925455 >> v2:172.21.15.117:3300/0 conn(0x7f935c0a5240 0x7f935c0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.004+0000 7f9360b5f640 1 --2- 172.21.15.98:0/599925455 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f935c0a4920 0x7f935c0a4d00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.004+0000 7f9360b5f640 1 -- 172.21.15.98:0/599925455 >> 172.21.15.98:0/599925455 conn(0x7f935c0ab850 msgr2=0x7f935c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.004+0000 7f9360b5f640 1 -- 172.21.15.98:0/599925455 shutdown_connections 2024-06-11T19:20:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.004+0000 7f9360b5f640 1 -- 172.21.15.98:0/599925455 wait complete. 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.005+0000 7f9360b5f640 10 monclient: build_initial_monmap 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.005+0000 7f9360b5f640 10 monclient: monmap: 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.005+0000 7f9360b5f640 1 librados: starting msgr at 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.005+0000 7f9360b5f640 1 librados: starting objecter 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.005+0000 7f9360b5f640 1 Processor -- start 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.005+0000 7f9360b5f640 1 -- start start 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.005+0000 7f9360b5f640 1 librados: setting wanted keys 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.005+0000 7f9360b5f640 1 librados: calling monclient init 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.005+0000 7f9360b5f640 10 monclient: init 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient: _add_conns ranks=[2,3,4,1,6,5,0,8,7] 2024-06-11T19:20:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f935c0a2530 0x7f935c1cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient(hunting): picked mon.noname-b con 0x7f935c0a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f935c0a4920 0x7f935c1ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient(hunting): picked mon.noname-g con 0x7f935c0a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f935c0a5240 0x7f935c122f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient(hunting): picked mon.noname-c con 0x7f935c0a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f935c0a6da0 con 0x7f935c0a2530 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f935c138e50 con 0x7f935c0a4920 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f935c0a65f0 con 0x7f935c0a5240 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient(hunting): authenticate will time out at 1418.813098s 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935ad9d640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f935c0a5240 msgr2=0x7f935c122f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:20:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935ad9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f935c0a5240 0x7f935c122f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9359d9b640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f935c0a4920 msgr2=0x7f935c1ce2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f935c0a2530 0x7f935c1cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9359d9b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f935c0a4920 0x7f935c1ce2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f935c0a2530 0x7f935c1cdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40124/0 (socket says 172.21.15.98:40124) 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 1 -- 172.21.15.98:0/300223366 learned_addr learned my addr 172.21.15.98:0/300223366 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 10 monclient(hunting): get_auth_request con 0x7f935c0a2530 auth_method 0 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 10 monclient(hunting): handle_auth_done global_id 84746 payload 824 2024-06-11T19:20:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 1 -- 172.21.15.98:0/300223366 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f935c0a5240 msgr2=0x7f935c122f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 1 --2- 172.21.15.98:0/300223366 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f935c0a5240 0x7f935c122f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 1 -- 172.21.15.98:0/300223366 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f935c0a4920 msgr2=0x7f935c1ce2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 1 --2- 172.21.15.98:0/300223366 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f935c0a4920 0x7f935c1ce2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 10 monclient: _finish_hunting 0 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 1 monclient: found mon.noname-b 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 1 -- 172.21.15.98:0/300223366 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f935c124b60 con 0x7f935c0a2530 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f935a59c640 1 --2- 172.21.15.98:0/300223366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f935c0a2530 0x7f935c1cdd90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f934c000c00 tx=0x7f934c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 10 client.?.objecter ms_handle_connect 0x7f935c0a2530 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 1 -- 172.21.15.98:0/300223366 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f934c041070 con 0x7f935c0a2530 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 10 monclient: dump: 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 10 monclient: _finish_auth 0 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 10 monclient: _check_auth_tickets 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 1 -- 172.21.15.98:0/300223366 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f934c0315c0 con 0x7f935c0a2530 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 5 monclient: authenticate success, global_id 84746 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient: _renew_subs 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 1 -- 172.21.15.98:0/300223366 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f935c126e50 con 0x7f935c0a2530 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 1 -- 172.21.15.98:0/300223366 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f934c031be0 con 0x7f935c0a2530 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f934affd640 10 monclient: dump: 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9358d99640 4 set_mon_vals no callback set 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 client.84746.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient: _renew_subs 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 1 -- 172.21.15.98:0/300223366 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f935c126b70 con 0x7f935c0a2530 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.006+0000 7f9360b5f640 1 librados: init done 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.007+0000 7f934affd640 1 -- 172.21.15.98:0/300223366 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f934c04c060 con 0x7f935c0a2530 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.007+0000 7f934affd640 1 --2- 172.21.15.98:0/300223366 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f933c03fdb0 0x7f933c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.007+0000 7f934affd640 1 -- 172.21.15.98:0/300223366 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(422..422 src has 1..422) ==== 6217+0+0 (secure 0 0 0) 0x7f934c073230 con 0x7f935c0a2530 2024-06-11T19:20:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.007+0000 7f9360b5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.007+0000 7f934affd640 10 client.84746.objecter ms_dispatch 0x7f935c000b90 osd_map(422..422 src has 1..422) 2024-06-11T19:20:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.007+0000 7f934affd640 3 client.84746.objecter handle_osd_map got epochs [422,422] > 0 2024-06-11T19:20:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.007+0000 7f934affd640 3 client.84746.objecter handle_osd_map decoding full epoch 422 2024-06-11T19:20:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.007+0000 7f9360b5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.007+0000 7f9360b5f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:20:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.007+0000 7f9360b5f640 1 -- 172.21.15.98:0/300223366 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f935c074e30 con 0x7f935c0a2530 2024-06-11T19:20:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.008+0000 7f934affd640 20 client.84746.objecter dump_active .. 0 homeless 2024-06-11T19:20:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.008+0000 7f9359d9b640 1 --2- 172.21.15.98:0/300223366 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f933c03fdb0 0x7f933c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.008+0000 7f9359d9b640 10 monclient: get_auth_request con 0x7f933c03fdb0 auth_method 0 2024-06-11T19:20:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.008+0000 7f9359d9b640 1 --2- 172.21.15.98:0/300223366 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f933c03fdb0 0x7f933c042270 secure :-1 s=READY pgs=4219 cs=0 l=1 rev1=1 crypto rx=0x7f935c124690 tx=0x7f9350009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.008+0000 7f934affd640 10 client.84746.objecter ms_handle_connect 0x7f933c03fdb0 2024-06-11T19:20:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.012+0000 7f934affd640 1 -- 172.21.15.98:0/300223366 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f935c074e30 con 0x7f935c0a2530 2024-06-11T19:20:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.012+0000 7f934affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.012+0000 7f934affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.263+0000 7f9360b5f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:20:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.263+0000 7f9360b5f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:20:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.263+0000 7f9360b5f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:20:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.263+0000 7f9360b5f640 1 -- 172.21.15.98:0/300223366 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f935c0a4d00 con 0x7f935c0a2530 2024-06-11T19:20:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.264+0000 7f934affd640 1 -- 172.21.15.98:0/300223366 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f934c003da0 con 0x7f935c0a2530 2024-06-11T19:20:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.264+0000 7f934affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:20:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.264+0000 7f934affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:09.264 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":54,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"c","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:20:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.265+0000 7f9360b5f640 10 librados: watch_flush enter 2024-06-11T19:20:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.265+0000 7f9360b5f640 10 librados: watch_flush exit 2024-06-11T19:20:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.265+0000 7f9360b5f640 20 client.84746.objecter shutdown clearing up homeless session... 2024-06-11T19:20:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.265+0000 7f9360b5f640 10 client.84746.objecter successfully canceled tick 2024-06-11T19:20:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.265+0000 7f9360b5f640 1 -- 172.21.15.98:0/300223366 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f933c03fdb0 msgr2=0x7f933c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.265+0000 7f9360b5f640 1 --2- 172.21.15.98:0/300223366 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f933c03fdb0 0x7f933c042270 secure :-1 s=READY pgs=4219 cs=0 l=1 rev1=1 crypto rx=0x7f935c124690 tx=0x7f9350009190 comp rx=0 tx=0).stop 2024-06-11T19:20:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.265+0000 7f9360b5f640 10 monclient: shutdown 2024-06-11T19:20:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.265+0000 7f9360b5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.265+0000 7f9360b5f640 1 -- 172.21.15.98:0/300223366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f935c0a2530 msgr2=0x7f935c1cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.265+0000 7f9360b5f640 1 --2- 172.21.15.98:0/300223366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f935c0a2530 0x7f935c1cdd90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f934c000c00 tx=0x7f934c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.266+0000 7f9360b5f640 1 -- 172.21.15.98:0/300223366 shutdown_connections 2024-06-11T19:20:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.266+0000 7f9360b5f640 1 --2- 172.21.15.98:0/300223366 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f933c03fdb0 0x7f933c042270 unknown :-1 s=CLOSED pgs=4219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.266+0000 7f9360b5f640 1 --2- 172.21.15.98:0/300223366 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f935c0a5240 0x7f935c122f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.266+0000 7f9360b5f640 1 --2- 172.21.15.98:0/300223366 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f935c0a4920 0x7f935c1ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.266+0000 7f9360b5f640 1 --2- 172.21.15.98:0/300223366 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f935c0a2530 0x7f935c1cdd90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.266+0000 7f9360b5f640 1 -- 172.21.15.98:0/300223366 >> 172.21.15.98:0/300223366 conn(0x7f935c0ab850 msgr2=0x7f935c1bf370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.266+0000 7f9360b5f640 1 -- 172.21.15.98:0/300223366 shutdown_connections 2024-06-11T19:20:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.266+0000 7f9360b5f640 1 -- 172.21.15.98:0/300223366 wait complete. 2024-06-11T19:20:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:09.266+0000 7f9360b5f640 1 librados: shutdown 2024-06-11T19:20:12.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:20:12.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.386+0000 7f95b2b2a640 10 monclient: get_monmap_and_config 2024-06-11T19:20:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.386+0000 7f95b2b2a640 10 monclient: build_initial_monmap 2024-06-11T19:20:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.386+0000 7f95b2b2a640 10 monclient: monmap: 2024-06-11T19:20:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:12.387582+0000 2024-06-11T19:20:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:12.387582+0000 2024-06-11T19:20:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:12.387 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:12.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 1 Processor -- start 2024-06-11T19:20:12.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 1 -- start start 2024-06-11T19:20:12.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 10 monclient: init 2024-06-11T19:20:12.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:12.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 10 monclient: _add_conns ranks=[10,7,4,0,2,8,5,3,9,1,6] 2024-06-11T19:20:12.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f95ac0a4920 0x7f95ac0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:12.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 10 monclient(hunting): picked mon.noname-f con 0x7f95ac0a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:12.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f95ac0a5240 0x7f95ac0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:12.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 10 monclient(hunting): picked mon.noname-i con 0x7f95ac0a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:12.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f95ac0a2530 0x7f95ac0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:12.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 10 monclient(hunting): picked mon.noname-c con 0x7f95ac0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:12.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:12.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f95ac0a7140 con 0x7f95ac0a4920 2024-06-11T19:20:12.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:12.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f95ac0a2b50 con 0x7f95ac0a2530 2024-06-11T19:20:12.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:12.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f95ac0a2e30 con 0x7f95ac0a5240 2024-06-11T19:20:12.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:12.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b2b2a640 10 monclient(hunting): authenticate will time out at 1422.194917s 2024-06-11T19:20:12.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b10c7640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f95ac0a2530 msgr2=0x7f95ac0a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:20:12.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.387+0000 7f95b10c7640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f95ac0a2530 0x7f95ac0a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95a3fff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f95ac0a5240 0x7f95ac0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f95ac0a4920 0x7f95ac0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95a3fff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f95ac0a5240 0x7f95ac0a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:48468/0 (socket says 172.21.15.98:48468) 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f95ac0a4920 0x7f95ac0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:55274/0 (socket says 172.21.15.98:55274) 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95a3fff640 1 -- 172.21.15.98:0/2899119203 learned_addr learned my addr 172.21.15.98:0/2899119203 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95a3fff640 10 monclient(hunting): get_auth_request con 0x7f95ac0a5240 auth_method 0 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95a3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 10 monclient(hunting): get_auth_request con 0x7f95ac0a4920 auth_method 0 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:12.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 10 monclient(hunting): handle_auth_done global_id 32527 payload 274 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 1 -- 172.21.15.98:0/2899119203 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f95ac0a5240 msgr2=0x7f95ac0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 1 --2- 172.21.15.98:0/2899119203 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f95ac0a5240 0x7f95ac0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 1 -- 172.21.15.98:0/2899119203 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f95ac0a2530 msgr2=0x7f95ac0a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 1 --2- 172.21.15.98:0/2899119203 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f95ac0a2530 0x7f95ac0a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 10 monclient: _finish_hunting 0 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 1 monclient: found mon.noname-f 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:20:12.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95b08c6640 1 -- 172.21.15.98:0/2899119203 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ac0a3110 con 0x7f95ac0a4920 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.388+0000 7f95a3fff640 1 --2- 172.21.15.98:0/2899119203 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f95ac0a5240 0x7f95ac0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95b08c6640 1 --2- 172.21.15.98:0/2899119203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95ac0a4920 0x7f95ac0a4d00 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f95a80097a0 tx=0x7f95a802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=810c9edd42f352cc server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 1 -- 172.21.15.98:0/2899119203 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95a8037070 con 0x7f95ac0a4920 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 10 monclient: dump: 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:12.401 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 10 monclient: _finish_auth 0 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 10 monclient: _check_auth_tickets 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:12.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95b2b2a640 5 monclient: authenticate success, global_id 32527 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 1 -- 172.21.15.98:0/2899119203 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a8030980 con 0x7f95ac0a4920 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95b2b2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95b2b2a640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95b2b2a640 4 set_mon_vals no callback set 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 1 -- 172.21.15.98:0/2899119203 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95a80313a0 con 0x7f95ac0a4920 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.389+0000 7f95a37fe640 10 monclient: dump: 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:12.403 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 10 monclient: shutdown 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 1 -- 172.21.15.98:0/2899119203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95ac0a4920 msgr2=0x7f95ac0a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 1 --2- 172.21.15.98:0/2899119203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95ac0a4920 0x7f95ac0a4d00 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f95a80097a0 tx=0x7f95a802b710 comp rx=0 tx=0).stop 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 1 -- 172.21.15.98:0/2899119203 shutdown_connections 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 1 --2- 172.21.15.98:0/2899119203 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f95ac0a2530 0x7f95ac0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 1 --2- 172.21.15.98:0/2899119203 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f95ac0a5240 0x7f95ac0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 1 --2- 172.21.15.98:0/2899119203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95ac0a4920 0x7f95ac0a4d00 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 1 -- 172.21.15.98:0/2899119203 >> 172.21.15.98:0/2899119203 conn(0x7f95ac0ab850 msgr2=0x7f95ac0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 1 -- 172.21.15.98:0/2899119203 shutdown_connections 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 1 -- 172.21.15.98:0/2899119203 wait complete. 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 10 monclient: build_initial_monmap 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 10 monclient: monmap: 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:12.405 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:12.406 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 1 librados: starting msgr at 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.390+0000 7f95b2b2a640 1 librados: starting objecter 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 1 Processor -- start 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 1 -- start start 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 1 librados: setting wanted keys 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 1 librados: calling monclient init 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 10 monclient: init 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 10 monclient: _add_conns ranks=[6,4,2,7,8,0,5,3,1] 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95ac0a2530 0x7f95ac1378d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 10 monclient(hunting): picked mon.noname-d con 0x7f95ac0a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f95ac0a4920 0x7f95ac137e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 10 monclient(hunting): picked mon.noname-c con 0x7f95ac0a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:12.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95ac0a5240 0x7f95ac1232f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 10 monclient(hunting): picked mon.noname-b con 0x7f95ac0a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f95ac0a6da0 con 0x7f95ac0a5240 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f95ac138e50 con 0x7f95ac0a4920 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f95ac0a6420 con 0x7f95ac0a2530 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b2b2a640 10 monclient(hunting): authenticate will time out at 1422.198781s 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95a3fff640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f95ac0a4920 msgr2=0x7f95ac137e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95a3fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f95ac0a4920 0x7f95ac137e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b08c6640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95ac0a2530 0x7f95ac1378d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:12.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b10c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95ac0a5240 0x7f95ac1232f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b08c6640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95ac0a2530 0x7f95ac1378d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:37906/0 (socket says 172.21.15.98:37906) 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b10c7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95ac0a5240 0x7f95ac1232f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40130/0 (socket says 172.21.15.98:40130) 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b08c6640 1 -- 172.21.15.98:0/424787242 learned_addr learned my addr 172.21.15.98:0/424787242 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b08c6640 10 monclient(hunting): get_auth_request con 0x7f95ac0a2530 auth_method 0 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b08c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b08c6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b08c6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b10c7640 10 monclient(hunting): get_auth_request con 0x7f95ac0a5240 auth_method 0 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b10c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b10c7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.391+0000 7f95b10c7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:12.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b10c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b10c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b10c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 10 monclient(hunting): handle_auth_done global_id 55419 payload 824 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 1 -- 172.21.15.98:0/424787242 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f95ac0a4920 msgr2=0x7f95ac137e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f95ac0a4920 0x7f95ac137e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 1 -- 172.21.15.98:0/424787242 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95ac0a5240 msgr2=0x7f95ac1232f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95ac0a5240 0x7f95ac1232f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 10 monclient: _finish_hunting 0 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 1 monclient: found mon.noname-d 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:20:12.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 1 -- 172.21.15.98:0/424787242 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ac126fd0 con 0x7f95ac0a2530 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b10c7640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95ac0a5240 0x7f95ac1232f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b08c6640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95ac0a2530 0x7f95ac1378d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f95a80097a0 tx=0x7f95a80046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 10 client.?.objecter ms_handle_connect 0x7f95ac0a2530 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 1 -- 172.21.15.98:0/424787242 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95a8040020 con 0x7f95ac0a2530 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 10 monclient: dump: 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:12.411 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 10 monclient: _finish_auth 0 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 10 monclient: _check_auth_tickets 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b2b2a640 5 monclient: authenticate success, global_id 55419 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 1 -- 172.21.15.98:0/424787242 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a8038040 con 0x7f95ac0a2530 2024-06-11T19:20:12.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b2b2a640 10 monclient: _renew_subs 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b2b2a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b2b2a640 1 -- 172.21.15.98:0/424787242 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95ac125b20 con 0x7f95ac0a2530 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 1 -- 172.21.15.98:0/424787242 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95a8037070 con 0x7f95ac0a2530 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a17fa640 10 monclient: dump: 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:12.413 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b2b2a640 10 client.55419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95a2ffd640 4 set_mon_vals no callback set 2024-06-11T19:20:12.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b2b2a640 10 monclient: _renew_subs 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b2b2a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b2b2a640 1 -- 172.21.15.98:0/424787242 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f95ac126cf0 con 0x7f95ac0a2530 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.392+0000 7f95b2b2a640 1 librados: init done 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.393+0000 7f95a17fa640 1 -- 172.21.15.98:0/424787242 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f95a8043020 con 0x7f95ac0a2530 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.393+0000 7f95b2b2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.393+0000 7f95b2b2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.393+0000 7f95b2b2a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.393+0000 7f95b2b2a640 1 -- 172.21.15.98:0/424787242 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95ac0a7360 con 0x7f95ac0a2530 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.393+0000 7f95a17fa640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f959803fdb0 0x7f9598042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.393+0000 7f95a17fa640 1 -- 172.21.15.98:0/424787242 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(422..422 src has 1..422) ==== 6217+0+0 (secure 0 0 0) 0x7f95a8072ed0 con 0x7f95ac0a2530 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.393+0000 7f95a17fa640 10 client.55419.objecter ms_dispatch 0x7f95ac000b90 osd_map(422..422 src has 1..422) 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.393+0000 7f95a17fa640 3 client.55419.objecter handle_osd_map got epochs [422,422] > 0 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.393+0000 7f95a17fa640 3 client.55419.objecter handle_osd_map decoding full epoch 422 2024-06-11T19:20:12.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.394+0000 7f95a17fa640 20 client.55419.objecter dump_active .. 0 homeless 2024-06-11T19:20:12.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.394+0000 7f95a3fff640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f959803fdb0 0x7f9598042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:12.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.394+0000 7f95a3fff640 10 monclient: get_auth_request con 0x7f959803fdb0 auth_method 0 2024-06-11T19:20:12.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.394+0000 7f95a17fa640 10 client.55419.objecter ms_handle_connect 0x7f959803fdb0 2024-06-11T19:20:12.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.394+0000 7f95a3fff640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f959803fdb0 0x7f9598042270 secure :-1 s=READY pgs=4220 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0a5620 tx=0x7f959c009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:12.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.398+0000 7f95a17fa640 1 -- 172.21.15.98:0/424787242 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f95ac0a7360 con 0x7f95ac0a2530 2024-06-11T19:20:12.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.398+0000 7f95a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:12.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.398+0000 7f95a17fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:12.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.650+0000 7f95b2b2a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:20:12.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.650+0000 7f95b2b2a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:20:12.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.650+0000 7f95b2b2a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:12.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.650+0000 7f95b2b2a640 1 -- 172.21.15.98:0/424787242 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f95ac0a6720 con 0x7f95ac0a2530 2024-06-11T19:20:12.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.650+0000 7f95a17fa640 1 -- 172.21.15.98:0/424787242 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f95a804a030 con 0x7f95ac0a2530 2024-06-11T19:20:12.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.650+0000 7f95a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:20:12.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.650+0000 7f95a17fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:12.650 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":54,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"c","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:20:12.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 10 librados: watch_flush enter 2024-06-11T19:20:12.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 10 librados: watch_flush exit 2024-06-11T19:20:12.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 20 client.55419.objecter shutdown clearing up homeless session... 2024-06-11T19:20:12.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 10 client.55419.objecter successfully canceled tick 2024-06-11T19:20:12.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 1 -- 172.21.15.98:0/424787242 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f959803fdb0 msgr2=0x7f9598042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:12.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f959803fdb0 0x7f9598042270 secure :-1 s=READY pgs=4220 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0a5620 tx=0x7f959c009240 comp rx=0 tx=0).stop 2024-06-11T19:20:12.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 10 monclient: shutdown 2024-06-11T19:20:12.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:12.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 1 -- 172.21.15.98:0/424787242 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95ac0a2530 msgr2=0x7f95ac1378d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:12.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95ac0a2530 0x7f95ac1378d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f95a80097a0 tx=0x7f95a80046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:12.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 1 -- 172.21.15.98:0/424787242 shutdown_connections 2024-06-11T19:20:12.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f959803fdb0 0x7f9598042270 unknown :-1 s=CLOSED pgs=4220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:12.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95ac0a5240 0x7f95ac1232f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:12.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f95ac0a4920 0x7f95ac137e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:12.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.652+0000 7f95b2b2a640 1 --2- 172.21.15.98:0/424787242 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95ac0a2530 0x7f95ac1378d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:12.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.653+0000 7f95b2b2a640 1 -- 172.21.15.98:0/424787242 >> 172.21.15.98:0/424787242 conn(0x7f95ac0ab850 msgr2=0x7f95ac0726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:12.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.653+0000 7f95b2b2a640 1 -- 172.21.15.98:0/424787242 shutdown_connections 2024-06-11T19:20:12.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.653+0000 7f95b2b2a640 1 -- 172.21.15.98:0/424787242 wait complete. 2024-06-11T19:20:12.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:12.653+0000 7f95b2b2a640 1 librados: shutdown 2024-06-11T19:20:15.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:20:15.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.773+0000 7f2e55731640 10 monclient: get_monmap_and_config 2024-06-11T19:20:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.773+0000 7f2e55731640 10 monclient: build_initial_monmap 2024-06-11T19:20:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.773+0000 7f2e55731640 10 monclient: monmap: 2024-06-11T19:20:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:15.774878+0000 2024-06-11T19:20:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:15.774878+0000 2024-06-11T19:20:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:15.774 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:15.774 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:15.774 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:15.774 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:15.774 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:15.774 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:15.774 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:15.774 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:15.774 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:15.774 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:15.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.774+0000 7f2e55731640 1 Processor -- start 2024-06-11T19:20:15.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.774+0000 7f2e55731640 1 -- start start 2024-06-11T19:20:15.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.774+0000 7f2e55731640 10 monclient: init 2024-06-11T19:20:15.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.774+0000 7f2e55731640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:15.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.774+0000 7f2e55731640 10 monclient: _add_conns ranks=[4,5,1,8,0,7,3,10,6,9,2] 2024-06-11T19:20:15.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e55731640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2e500a4920 0x7f2e500a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:15.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e55731640 10 monclient(hunting): picked mon.noname-c con 0x7f2e500a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:15.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e55731640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e55731640 10 monclient(hunting): picked mon.noname-h con 0x7f2e500a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e55731640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f2e500a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e55731640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e500a7140 con 0x7f2e500a2530 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e55731640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e55731640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2e500a2b50 con 0x7f2e500a4920 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e55731640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e55731640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f2e500a2e30 con 0x7f2e500a5240 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e55731640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e55731640 10 monclient(hunting): authenticate will time out at 1425.582089s 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e4effd640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2e500a4920 msgr2=0x7f2e500a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e4effd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2e500a4920 0x7f2e500a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e467fc640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:15.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e467fc640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:37720/0 (socket says 172.21.15.98:37720) 2024-06-11T19:20:15.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 learned_addr learned my addr 172.21.15.98:0/340832490 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:15.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e467fc640 10 monclient(hunting): get_auth_request con 0x7f2e500a5240 auth_method 0 2024-06-11T19:20:15.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:15.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e467fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:15.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e467fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:15.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 msgr2=0x7f2e500a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:20:15.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 msgr2=0x7f2e500a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:20:15.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:20:15.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.775+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:15.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.976+0000 7f2e4effd640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2e500a4920 msgr2=0x7f2e500a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:20:16.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.976+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2e500a4920 0x7f2e500a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:20:16.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.976+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:16.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.976+0000 7f2e467fc640 10 monclient(hunting): get_auth_request con 0x7f2e500a5240 auth_method 0 2024-06-11T19:20:16.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.976+0000 7f2e467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:16.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.976+0000 7f2e467fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.976+0000 7f2e467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:20:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.976+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 msgr2=0x7f2e500a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:20:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.976+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 msgr2=0x7f2e500a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:20:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.976+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:20:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:15.976+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:20:16.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:16.377+0000 7f2e4effd640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2e500a4920 msgr2=0x7f2e500a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:20:16.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:16.377+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2e500a4920 0x7f2e500a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:20:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:16.377+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:16.377+0000 7f2e467fc640 10 monclient(hunting): get_auth_request con 0x7f2e500a5240 auth_method 0 2024-06-11T19:20:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:16.377+0000 7f2e467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:16.377+0000 7f2e467fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:16.377+0000 7f2e467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:20:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:16.377+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 msgr2=0x7f2e500a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:20:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:16.377+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 msgr2=0x7f2e500a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:20:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:16.377+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:20:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:16.377+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:20:17.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:17.178+0000 7f2e4effd640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2e500a4920 msgr2=0x7f2e500a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:20:17.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:17.178+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2e500a4920 0x7f2e500a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:20:17.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:17.178+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:17.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:17.178+0000 7f2e467fc640 10 monclient(hunting): get_auth_request con 0x7f2e500a5240 auth_method 0 2024-06-11T19:20:17.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:17.178+0000 7f2e467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:17.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:17.178+0000 7f2e467fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:17.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:17.178+0000 7f2e467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:20:17.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:17.178+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 msgr2=0x7f2e500a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:20:17.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:17.178+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 msgr2=0x7f2e500a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:20:17.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:17.178+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:20:17.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:17.178+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:20:18.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 10 monclient(hunting): tick 2024-06-11T19:20:18.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:20:18.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 1 monclient(hunting): continuing hunt 2024-06-11T19:20:18.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 msgr2=0x7f2e500a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2e500a5240 0x7f2e500a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2e500a4920 msgr2=0x7f2e500a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2e500a4920 0x7f2e500a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 1 -- 172.21.15.98:0/340832490 >> v1:172.21.15.117:6789/0 conn(0x7f2e500a2530 legacy=0x7f2e500a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 10 monclient: _add_conns ranks=[9,8,10,6,3,2,0,7] 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 0x7f2e30005c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2e30003820 addr v2:172.21.15.98:3300/0 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2e30006270 0x7f2e300086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 10 monclient(hunting): picked mon.noname-e con 0x7f2e30006270 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 10 monclient(hunting): picked mon.noname-f con 0x7f2e30008da0 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:18.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 1 -- 172.21.15.98:0/340832490 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2e500a2b50 con 0x7f2e30003820 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 1 -- 172.21.15.98:0/340832490 --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f2e500a2e30 con 0x7f2e30008da0 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 1 -- 172.21.15.98:0/340832490 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2e3000b9b0 con 0x7f2e30006270 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4d7fa640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2e30006270 msgr2=0x7f2e300086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2e30006270 0x7f2e300086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 0x7f2e30005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4effd640 10 monclient(hunting): get_auth_request con 0x7f2e30003820 auth_method 0 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4effd640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 msgr2=0x7f2e30005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4effd640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 msgr2=0x7f2e30005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 0x7f2e30005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 0x7f2e30005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4f7fe640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4f7fe640 10 monclient(hunting): get_auth_request con 0x7f2e30008da0 auth_method 0 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4f7fe640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 msgr2=0x7f2e3000b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4f7fe640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 msgr2=0x7f2e3000b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4f7fe640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.775+0000 7f2e4f7fe640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2e30006270 msgr2=0x7f2e300086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:20:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2e30006270 0x7f2e300086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:20:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 0x7f2e30005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4effd640 10 monclient(hunting): get_auth_request con 0x7f2e30003820 auth_method 0 2024-06-11T19:20:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:20:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4effd640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 msgr2=0x7f2e30005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:20:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4effd640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 msgr2=0x7f2e30005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:20:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 0x7f2e30005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:20:18.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 0x7f2e30005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:20:18.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4f7fe640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:18.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4f7fe640 10 monclient(hunting): get_auth_request con 0x7f2e30008da0 auth_method 0 2024-06-11T19:20:18.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:18.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:18.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:20:18.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4f7fe640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 msgr2=0x7f2e3000b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:20:18.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4f7fe640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 msgr2=0x7f2e3000b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:20:18.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4f7fe640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:20:18.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:18.976+0000 7f2e4f7fe640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:20:19.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2e30006270 msgr2=0x7f2e300086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:20:19.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2e30006270 0x7f2e300086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:20:19.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 0x7f2e30005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:19.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4effd640 10 monclient(hunting): get_auth_request con 0x7f2e30003820 auth_method 0 2024-06-11T19:20:19.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:19.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:19.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:20:19.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4effd640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 msgr2=0x7f2e30005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:20:19.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4effd640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 msgr2=0x7f2e30005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:20:19.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 0x7f2e30005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:20:19.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 0x7f2e30005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:20:19.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4f7fe640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:19.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4f7fe640 10 monclient(hunting): get_auth_request con 0x7f2e30008da0 auth_method 0 2024-06-11T19:20:19.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:19.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:19.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:20:19.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4f7fe640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 msgr2=0x7f2e3000b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:20:19.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4f7fe640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 msgr2=0x7f2e3000b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:20:19.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4f7fe640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:20:19.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:19.377+0000 7f2e4f7fe640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:20:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e467fc640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2e30006270 msgr2=0x7f2e300086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:20:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e467fc640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2e30006270 0x7f2e300086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:20:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.98:3300/0 conn(0x7f2e30003820 0x7f2e30005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 10 monclient(hunting): get_auth_request con 0x7f2e30003820 auth_method 0 2024-06-11T19:20:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 10 monclient(hunting): handle_auth_done global_id 65430 payload 274 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2e30006270 msgr2=0x7f2e300086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2e30006270 0x7f2e300086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 1 -- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 msgr2=0x7f2e3000b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4f7fe640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 10 monclient: _finish_hunting 0 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 1 monclient: found mon.noname-a 2024-06-11T19:20:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4effd640 1 -- 172.21.15.98:0/340832490 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e500a3110 con 0x7f2e30003820 2024-06-11T19:20:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4f7fe640 10 monclient: get_auth_request con 0x7f2e30008da0 auth_method 0 2024-06-11T19:20:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4f7fe640 1 --2- 172.21.15.98:0/340832490 >> v2:172.21.15.117:3300/0 conn(0x7f2e30008da0 0x7f2e3000b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:20:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.178+0000 7f2e4f7fe640 1 -- 172.21.15.98:0/340832490 reap_dead start 2024-06-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4effd640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e30003820 0x7f2e30005c70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2e30008d70 tx=0x7f2e4002c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b65e1902dea93c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 1 -- 172.21.15.98:0/340832490 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2e40036020 con 0x7f2e30003820 2024-06-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 10 monclient: dump: 2024-06-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.186 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.187 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 10 monclient: _finish_auth 0 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 1 -- 172.21.15.98:0/340832490 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e40032af0 con 0x7f2e30003820 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 1 -- 172.21.15.98:0/340832490 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2e40032ca0 con 0x7f2e30003820 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e55731640 5 monclient: authenticate success, global_id 65430 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e55731640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e55731640 4 set_mon_vals no callback set 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.179+0000 7f2e4e7fc640 10 monclient: dump: 2024-06-11T19:20:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.180+0000 7f2e55731640 10 monclient: shutdown 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.180+0000 7f2e55731640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.180+0000 7f2e55731640 1 -- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e30003820 msgr2=0x7f2e30005c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.180+0000 7f2e55731640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e30003820 0x7f2e30005c70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2e30008d70 tx=0x7f2e4002c920 comp rx=0 tx=0).stop 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.180+0000 7f2e55731640 1 -- 172.21.15.98:0/340832490 shutdown_connections 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.180+0000 7f2e55731640 1 --2- 172.21.15.98:0/340832490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e30003820 0x7f2e30005c70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.180+0000 7f2e55731640 1 -- 172.21.15.98:0/340832490 >> 172.21.15.98:0/340832490 conn(0x7f2e500ab850 msgr2=0x7f2e500a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.180+0000 7f2e55731640 1 -- 172.21.15.98:0/340832490 shutdown_connections 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.180+0000 7f2e55731640 1 -- 172.21.15.98:0/340832490 wait complete. 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.181+0000 7f2e55731640 10 monclient: build_initial_monmap 2024-06-11T19:20:20.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.181+0000 7f2e55731640 10 monclient: monmap: 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:20.191 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.181+0000 7f2e55731640 1 librados: starting msgr at 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.181+0000 7f2e55731640 1 librados: starting objecter 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.181+0000 7f2e55731640 1 Processor -- start 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 1 -- start start 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 1 librados: setting wanted keys 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 1 librados: calling monclient init 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 10 monclient: init 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 10 monclient: _add_conns ranks=[6,1,3,7,0,8,4,5,2] 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2e30003820 0x7f2e50137570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 10 monclient(hunting): picked mon.noname-d con 0x7f2e30003820 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e50137ab0 0x7f2e50122f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 10 monclient(hunting): picked mon.noname-f con 0x7f2e50137ab0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2e501234d0 0x7f2e50123950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 10 monclient(hunting): picked mon.noname-g con 0x7f2e501234d0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2e500a6da0 con 0x7f2e50137ab0 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f2e50138e50 con 0x7f2e501234d0 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f2e500a4d00 con 0x7f2e30003820 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e55731640 10 monclient(hunting): authenticate will time out at 1429.989822s 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e4effd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2e30003820 0x7f2e50137570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e4f7fe640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2e501234d0 msgr2=0x7f2e50123950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e4effd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2e30003820 0x7f2e50137570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:41002/0 (socket says 172.21.15.98:41002) 2024-06-11T19:20:20.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e4f7fe640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2e501234d0 0x7f2e50123950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e4effd640 1 -- 172.21.15.98:0/4004911453 learned_addr learned my addr 172.21.15.98:0/4004911453 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e4effd640 10 monclient(hunting): get_auth_request con 0x7f2e30003820 auth_method 0 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e4effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.182+0000 7f2e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e467fc640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e50137ab0 0x7f2e50122f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e467fc640 10 monclient(hunting): get_auth_request con 0x7f2e50137ab0 auth_method 0 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e467fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e467fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 10 monclient(hunting): handle_auth_done global_id 55428 payload 824 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 1 -- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2e501234d0 msgr2=0x7f2e50123950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2e501234d0 0x7f2e50123950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 1 -- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e50137ab0 msgr2=0x7f2e50122f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e50137ab0 0x7f2e50122f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 10 monclient: _finish_hunting 0 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 1 monclient: found mon.noname-d 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 1 -- 172.21.15.98:0/4004911453 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e500a8ea0 con 0x7f2e30003820 2024-06-11T19:20:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4effd640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2e30003820 0x7f2e50137570 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2e40035040 tx=0x7f2e40005db0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 10 client.?.objecter ms_handle_connect 0x7f2e30003820 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 1 -- 172.21.15.98:0/4004911453 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2e40036040 con 0x7f2e30003820 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 10 monclient: dump: 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 10 monclient: _finish_auth 0 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 10 monclient: _check_auth_tickets 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 1 -- 172.21.15.98:0/4004911453 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e40047480 con 0x7f2e30003820 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e55731640 5 monclient: authenticate success, global_id 55428 2024-06-11T19:20:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e55731640 10 monclient: _renew_subs 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e55731640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e55731640 1 -- 172.21.15.98:0/4004911453 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e50124c60 con 0x7f2e30003820 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 1 -- 172.21.15.98:0/4004911453 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2e40047aa0 con 0x7f2e30003820 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e47fff640 10 monclient: dump: 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.198 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e55731640 10 client.55428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e55731640 10 monclient: _renew_subs 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e4dffb640 4 set_mon_vals no callback set 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e55731640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e55731640 1 -- 172.21.15.98:0/4004911453 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e50126f50 con 0x7f2e30003820 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.183+0000 7f2e55731640 1 librados: init done 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.184+0000 7f2e47fff640 1 -- 172.21.15.98:0/4004911453 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f2e40039020 con 0x7f2e30003820 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.184+0000 7f2e55731640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.184+0000 7f2e55731640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.184+0000 7f2e55731640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.184+0000 7f2e55731640 1 -- 172.21.15.98:0/4004911453 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e500a5000 con 0x7f2e30003820 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.185+0000 7f2e47fff640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2e30040fd0 0x7f2e30043490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.185+0000 7f2e47fff640 1 -- 172.21.15.98:0/4004911453 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(423..423 src has 1..423) ==== 6230+0+0 (secure 0 0 0) 0x7f2e40074ea0 con 0x7f2e30003820 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.185+0000 7f2e47fff640 10 client.55428.objecter ms_dispatch 0x7f2e50000b90 osd_map(423..423 src has 1..423) 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.185+0000 7f2e47fff640 3 client.55428.objecter handle_osd_map got epochs [423,423] > 0 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.185+0000 7f2e47fff640 3 client.55428.objecter handle_osd_map decoding full epoch 423 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.185+0000 7f2e47fff640 20 client.55428.objecter dump_active .. 0 homeless 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.185+0000 7f2e467fc640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2e30040fd0 0x7f2e30043490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.185+0000 7f2e467fc640 10 monclient: get_auth_request con 0x7f2e30040fd0 auth_method 0 2024-06-11T19:20:20.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.185+0000 7f2e47fff640 10 client.55428.objecter ms_handle_connect 0x7f2e30040fd0 2024-06-11T19:20:20.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.185+0000 7f2e467fc640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2e30040fd0 0x7f2e30043490 secure :-1 s=READY pgs=4221 cs=0 l=1 rev1=1 crypto rx=0x7f2e500a5620 tx=0x7f2e38008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.189+0000 7f2e47fff640 1 -- 172.21.15.98:0/4004911453 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2e4003b430 con 0x7f2e30003820 2024-06-11T19:20:20.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.189+0000 7f2e47fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:20.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.189+0000 7f2e47fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:20.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.444+0000 7f2e55731640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:20:20.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.444+0000 7f2e55731640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:20:20.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.444+0000 7f2e55731640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:20.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.444+0000 7f2e55731640 1 -- 172.21.15.98:0/4004911453 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2e500a0100 con 0x7f2e30003820 2024-06-11T19:20:20.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.444+0000 7f2e47fff640 1 -- 172.21.15.98:0/4004911453 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3174 (secure 0 0 0) 0x7f2e40005e20 con 0x7f2e30003820 2024-06-11T19:20:20.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.444+0000 7f2e47fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:20:20.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.444+0000 7f2e47fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:20.444 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":58,"quorum":[0,1,2,5,6,7],"quorum_names":["a","f","b","h","d","i"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:20:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 10 librados: watch_flush enter 2024-06-11T19:20:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 10 librados: watch_flush exit 2024-06-11T19:20:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 20 client.55428.objecter shutdown clearing up homeless session... 2024-06-11T19:20:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 10 client.55428.objecter successfully canceled tick 2024-06-11T19:20:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 -- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2e30040fd0 msgr2=0x7f2e30043490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2e30040fd0 0x7f2e30043490 secure :-1 s=READY pgs=4221 cs=0 l=1 rev1=1 crypto rx=0x7f2e500a5620 tx=0x7f2e38008040 comp rx=0 tx=0).stop 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 10 monclient: shutdown 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 -- 172.21.15.98:0/4004911453 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2e30003820 msgr2=0x7f2e50137570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2e30003820 0x7f2e50137570 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2e40035040 tx=0x7f2e40005db0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 -- 172.21.15.98:0/4004911453 shutdown_connections 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2e30040fd0 0x7f2e30043490 unknown :-1 s=CLOSED pgs=4221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2e501234d0 0x7f2e50123950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e50137ab0 0x7f2e50122f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 --2- 172.21.15.98:0/4004911453 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2e30003820 0x7f2e50137570 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 -- 172.21.15.98:0/4004911453 >> 172.21.15.98:0/4004911453 conn(0x7f2e500ab850 msgr2=0x7f2e501bf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 -- 172.21.15.98:0/4004911453 shutdown_connections 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 -- 172.21.15.98:0/4004911453 wait complete. 2024-06-11T19:20:20.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.446+0000 7f2e55731640 1 librados: shutdown 2024-06-11T19:20:20.457 INFO:tasks.mon_thrash.ceph_manager:quorum is size 6 2024-06-11T19:20:20.457 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon a 2024-06-11T19:20:20.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:20:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.557+0000 7f4d1c099640 10 monclient: get_monmap_and_config 2024-06-11T19:20:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.557+0000 7f4d1c099640 10 monclient: build_initial_monmap 2024-06-11T19:20:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.557+0000 7f4d1c099640 10 monclient: monmap: 2024-06-11T19:20:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:20.558662+0000 2024-06-11T19:20:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:20.558662+0000 2024-06-11T19:20:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:20.558 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 1 Processor -- start 2024-06-11T19:20:20.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 1 -- start start 2024-06-11T19:20:20.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 10 monclient: init 2024-06-11T19:20:20.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:20.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 10 monclient: _add_conns ranks=[2,6,0,4,7,5,1,10,9,3,8] 2024-06-11T19:20:20.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d140a4920 0x7f4d140a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 10 monclient(hunting): picked mon.noname-b con 0x7f4d140a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4d140a5240 0x7f4d140a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 10 monclient(hunting): picked mon.noname-d con 0x7f4d140a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d140a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d140a7140 con 0x7f4d140a2530 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4d140a2b50 con 0x7f4d140a4920 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4d140a2e30 con 0x7f4d140a5240 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.558+0000 7f4d1c099640 10 monclient(hunting): authenticate will time out at 1430.365927s 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19634640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4d140a5240 0x7f4d140a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d140a4920 0x7f4d140a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19634640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4d140a5240 0x7f4d140a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:41016/0 (socket says 172.21.15.98:41016) 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19634640 1 -- 172.21.15.98:0/3949983187 learned_addr learned my addr 172.21.15.98:0/3949983187 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19634640 10 monclient(hunting): get_auth_request con 0x7f4d140a5240 auth_method 0 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19634640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19634640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19634640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d1a636640 1 --1- 172.21.15.98:0/3949983187 >> v1:172.21.15.98:6789/0 conn(0x7f4d140a2530 0x7f4d140a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46142/0 (socket says 172.21.15.98:46142) 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 10 monclient(hunting): get_auth_request con 0x7f4d140a4920 auth_method 0 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19634640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19634640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19634640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 10 monclient(hunting): handle_auth_done global_id 84764 payload 274 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 1 -- 172.21.15.98:0/3949983187 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4d140a5240 msgr2=0x7f4d140a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 1 --2- 172.21.15.98:0/3949983187 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4d140a5240 0x7f4d140a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 1 -- 172.21.15.98:0/3949983187 >> v1:172.21.15.98:6789/0 conn(0x7f4d140a2530 legacy=0x7f4d140a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 10 monclient: _finish_hunting 0 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 1 monclient: found mon.noname-b 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d18e33640 1 -- 172.21.15.98:0/3949983187 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4062795280 0 0) 0x7f4d140a7140 con 0x7f4d140a2530 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19e35640 1 -- 172.21.15.98:0/3949983187 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d140a3110 con 0x7f4d140a4920 2024-06-11T19:20:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d18e33640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.559+0000 7f4d19634640 1 --2- 172.21.15.98:0/3949983187 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4d140a5240 0x7f4d140a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d19e35640 1 --2- 172.21.15.98:0/3949983187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d140a4920 0x7f4d140a4d00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4d100097d0 tx=0x7f4d1002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=293ee0c9193482b5 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 1 -- 172.21.15.98:0/3949983187 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4d10039070 con 0x7f4d140a4920 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 10 monclient: dump: 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 10 monclient: _finish_auth 0 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 10 monclient: _check_auth_tickets 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 5 monclient: authenticate success, global_id 84764 2024-06-11T19:20:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 1 -- 172.21.15.98:0/3949983187 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d10031960 con 0x7f4d140a4920 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 1 -- 172.21.15.98:0/3949983187 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4d10032380 con 0x7f4d140a4920 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 4 set_mon_vals no callback set 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d18e33640 10 monclient: dump: 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.574 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 10 monclient: shutdown 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 1 -- 172.21.15.98:0/3949983187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d140a4920 msgr2=0x7f4d140a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 1 --2- 172.21.15.98:0/3949983187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d140a4920 0x7f4d140a4d00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4d100097d0 tx=0x7f4d1002b790 comp rx=0 tx=0).stop 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 1 -- 172.21.15.98:0/3949983187 shutdown_connections 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 1 --2- 172.21.15.98:0/3949983187 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4d140a5240 0x7f4d140a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 1 --2- 172.21.15.98:0/3949983187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4d140a4920 0x7f4d140a4d00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 1 -- 172.21.15.98:0/3949983187 >> 172.21.15.98:0/3949983187 conn(0x7f4d140ab850 msgr2=0x7f4d140a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 1 -- 172.21.15.98:0/3949983187 shutdown_connections 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.560+0000 7f4d1c099640 1 -- 172.21.15.98:0/3949983187 wait complete. 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.561+0000 7f4d1c099640 10 monclient: build_initial_monmap 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.561+0000 7f4d1c099640 10 monclient: monmap: 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.561+0000 7f4d1c099640 1 librados: starting msgr at 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.561+0000 7f4d1c099640 1 librados: starting objecter 2024-06-11T19:20:20.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.561+0000 7f4d1c099640 1 Processor -- start 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.561+0000 7f4d1c099640 1 -- start start 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.561+0000 7f4d1c099640 1 librados: setting wanted keys 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.561+0000 7f4d1c099640 1 librados: calling monclient init 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.561+0000 7f4d1c099640 10 monclient: init 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 10 monclient: _add_conns ranks=[5,8,0,7,2,4,1,3,6] 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4d140a2530 0x7f4d141cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 10 monclient(hunting): picked mon.noname-h con 0x7f4d140a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4d140a4920 0x7f4d141ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 10 monclient(hunting): picked mon.noname-e con 0x7f4d140a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d140a5240 0x7f4d14125030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 10 monclient(hunting): picked mon.noname-a con 0x7f4d140a5240 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:20:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d140a6da0 con 0x7f4d140a5240 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f4d14138e50 con 0x7f4d140a2530 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4d140a65f0 con 0x7f4d140a4920 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 10 monclient(hunting): authenticate will time out at 1430.369192s 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d19634640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4d140a4920 msgr2=0x7f4d141ce0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d19634640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4d140a4920 0x7f4d141ce0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d140a5240 0x7f4d14125030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d140a5240 0x7f4d14125030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36174/0 (socket says 172.21.15.98:36174) 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 1 -- 172.21.15.98:0/2886259580 learned_addr learned my addr 172.21.15.98:0/2886259580 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 10 monclient(hunting): get_auth_request con 0x7f4d140a5240 auth_method 0 2024-06-11T19:20:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 10 monclient(hunting): handle_auth_done global_id 65448 payload 824 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 1 -- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4d140a4920 msgr2=0x7f4d141ce0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d19e35640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4d140a2530 0x7f4d141cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4d140a4920 0x7f4d141ce0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 1 -- 172.21.15.98:0/2886259580 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4d140a2530 msgr2=0x7f4d141cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4d140a2530 0x7f4d141cdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 10 monclient: _finish_hunting 0 2024-06-11T19:20:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 1 monclient: found mon.noname-a 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 1 -- 172.21.15.98:0/2886259580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d140a8f10 con 0x7f4d140a5240 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1a636640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d140a5240 0x7f4d14125030 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00c770 tx=0x7f4d0c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 10 client.?.objecter ms_handle_connect 0x7f4d140a5240 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4d0c018070 con 0x7f4d140a5240 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 10 monclient: dump: 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:20:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:20:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 5 monclient: authenticate success, global_id 65448 2024-06-11T19:20:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d0c004d70 con 0x7f4d140a5240 2024-06-11T19:20:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 10 monclient: _renew_subs 2024-06-11T19:20:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4d0c0070d0 con 0x7f4d140a5240 2024-06-11T19:20:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:20:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d1c099640 1 -- 172.21.15.98:0/2886259580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d140aabe0 con 0x7f4d140a5240 2024-06-11T19:20:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0bfff640 4 set_mon_vals no callback set 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.562+0000 7f4d0a7fc640 10 monclient: dump: 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.584 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d1c099640 10 client.65448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d1c099640 10 monclient: _renew_subs 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d1c099640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d1c099640 1 -- 172.21.15.98:0/2886259580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d14128c40 con 0x7f4d140a5240 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d1c099640 1 librados: init done 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d1c099640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d1c099640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125570 0x7f4d14125950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d1c099640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d1c099640 1 -- 172.21.15.98:0/2886259580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d14050830 con 0x7f4d14125570 2024-06-11T19:20:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f4d0c007270 con 0x7f4d140a5240 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d19e35640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125570 0x7f4d14125950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d19e35640 10 monclient: get_auth_request con 0x7f4d14125570 auth_method 0 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d19e35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d19e35640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d19e35640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d19e35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d19e35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.563+0000 7f4d19e35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d19e35640 10 monclient(hunting): handle_auth_done global_id 65457 payload 824 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4cf803fdb0 0x7f4cf8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d19e35640 1 -- 172.21.15.98:0/2886259580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d14128680 con 0x7f4d14125570 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(423..423 src has 1..423) ==== 6230+0+0 (secure 0 0 0) 0x7f4d0c050ea0 con 0x7f4d140a5240 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 10 client.65448.objecter ms_dispatch 0x7f4d14000b90 osd_map(423..423 src has 1..423) 2024-06-11T19:20:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 3 client.65448.objecter handle_osd_map got epochs [423,423] > 0 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 3 client.65448.objecter handle_osd_map decoding full epoch 423 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d19e35640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125570 0x7f4d14125950 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4d140abf80 tx=0x7f4d100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 20 client.65448.objecter dump_active .. 0 homeless 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 10 client.65448.objecter ms_handle_connect 0x7f4d14125570 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 10 client.65448.objecter resend_mon_ops 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d19634640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4cf803fdb0 0x7f4cf8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d19634640 10 monclient: get_auth_request con 0x7f4cf803fdb0 auth_method 0 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4d10043070 con 0x7f4d14125570 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 10 monclient: dump: 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.587 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.589 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4d10003610 con 0x7f4d14125570 2024-06-11T19:20:20.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:20.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:20.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125570 msgr2=0x7f4d14125950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125570 0x7f4d14125950 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4d140abf80 tx=0x7f4d100046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125570 msgr2=0x7f4d14125950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:20.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125570 0x7f4d14125950 secure :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4d140abf80 tx=0x7f4d100046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d0a7fc640 10 client.65448.objecter ms_handle_connect 0x7f4cf803fdb0 2024-06-11T19:20:20.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.564+0000 7f4d19634640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4cf803fdb0 0x7f4cf8042270 secure :-1 s=READY pgs=4222 cs=0 l=1 rev1=1 crypto rx=0x7f4d14126790 tx=0x7f4d04009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.570+0000 7f4d1c099640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.570+0000 7f4d1c099640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125e90 0x7f4d141215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d1c099640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d1c099640 1 -- 172.21.15.98:0/2886259580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d14069740 con 0x7f4d14125e90 2024-06-11T19:20:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d19e35640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125e90 0x7f4d141215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d19e35640 10 monclient: get_auth_request con 0x7f4d14125e90 auth_method 0 2024-06-11T19:20:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d19e35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d19e35640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d19e35640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d19e35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d19e35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d19e35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d19e35640 10 monclient(hunting): handle_auth_done global_id 65466 payload 824 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d19e35640 1 -- 172.21.15.98:0/2886259580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4d140a3480 con 0x7f4d14125e90 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d19e35640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125e90 0x7f4d141215f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4d100096f0 tx=0x7f4d10009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 10 client.65448.objecter ms_handle_connect 0x7f4d14125e90 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 10 client.65448.objecter resend_mon_ops 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4d10039040 con 0x7f4d14125e90 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 10 monclient: dump: 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9410 (secure 0 0 0) 0x7f4d10034a60 con 0x7f4d14125e90 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125e90 msgr2=0x7f4d141215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125e90 0x7f4d141215f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4d100096f0 tx=0x7f4d10009190 comp rx=0 tx=0).stop 2024-06-11T19:20:20.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 1 -- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125e90 msgr2=0x7f4d141215f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:20.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.571+0000 7f4d0a7fc640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d14125e90 0x7f4d141215f0 secure :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4d100096f0 tx=0x7f4d10009190 comp rx=0 tx=0).stop 2024-06-11T19:20:20.595 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:20.595 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:20.595 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:20.595 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:20.595 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 58, 2024-06-11T19:20:20.595 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 412399, 2024-06-11T19:20:20.595 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:20.595 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:20.617 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:20.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 10 librados: watch_flush enter 2024-06-11T19:20:20.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 10 librados: watch_flush exit 2024-06-11T19:20:20.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 20 client.65448.objecter shutdown clearing up homeless session... 2024-06-11T19:20:20.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 10 client.65448.objecter successfully canceled tick 2024-06-11T19:20:20.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 1 -- 172.21.15.98:0/2886259580 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4cf803fdb0 msgr2=0x7f4cf8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4cf803fdb0 0x7f4cf8042270 secure :-1 s=READY pgs=4222 cs=0 l=1 rev1=1 crypto rx=0x7f4d14126790 tx=0x7f4d04009240 comp rx=0 tx=0).stop 2024-06-11T19:20:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 10 monclient: shutdown 2024-06-11T19:20:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 1 -- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d140a5240 msgr2=0x7f4d14125030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 1 --2- 172.21.15.98:0/2886259580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d140a5240 0x7f4d14125030 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00c770 tx=0x7f4d0c00cc30 comp rx=0 tx=0).stop 2024-06-11T19:20:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1a636640 1 -- 172.21.15.98:0/2886259580 reap_dead start 2024-06-11T19:20:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1a636640 1 -- 172.21.15.98:0/2886259580 reap_dead start 2024-06-11T19:20:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 1 -- 172.21.15.98:0/2886259580 shutdown_connections 2024-06-11T19:20:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 1 -- 172.21.15.98:0/2886259580 >> 172.21.15.98:0/2886259580 conn(0x7f4d140ab850 msgr2=0x7f4d140a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 1 -- 172.21.15.98:0/2886259580 shutdown_connections 2024-06-11T19:20:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 1 -- 172.21.15.98:0/2886259580 wait complete. 2024-06-11T19:20:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.572+0000 7f4d1c099640 1 librados: shutdown 2024-06-11T19:20:20.621 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-06-11T19:20:20.621 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon b 2024-06-11T19:20:20.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:20:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.721+0000 7f99fda17640 10 monclient: get_monmap_and_config 2024-06-11T19:20:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.721+0000 7f99fda17640 10 monclient: build_initial_monmap 2024-06-11T19:20:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.722+0000 7f99fda17640 10 monclient: monmap: 2024-06-11T19:20:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:20.722975+0000 2024-06-11T19:20:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:20.722975+0000 2024-06-11T19:20:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.721 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:20.722 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:20.722 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:20.722 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:20.722 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:20.722 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:20.722 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:20.722 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:20.722 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:20.722 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:20.722 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:20.722 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.722+0000 7f99fda17640 1 Processor -- start 2024-06-11T19:20:20.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.722+0000 7f99fda17640 1 -- start start 2024-06-11T19:20:20.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.722+0000 7f99fda17640 10 monclient: init 2024-06-11T19:20:20.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:20.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 10 monclient: _add_conns ranks=[9,10,4,1,2,0,7,6,8,3,5] 2024-06-11T19:20:20.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f99f80a4920 0x7f99f80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 10 monclient(hunting): picked mon.noname-a con 0x7f99f80a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:20:20.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f99f80a5240 0x7f99f80a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 10 monclient(hunting): picked mon.noname-f con 0x7f99f80a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f99f80a2530 0x7f99f80a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 10 monclient(hunting): picked mon.noname-c con 0x7f99f80a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f99f80a7140 con 0x7f99f80a4920 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f99f80a2b50 con 0x7f99f80a5240 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f99f80a2e30 con 0x7f99f80a2530 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99fda17640 10 monclient(hunting): authenticate will time out at 1430.530105s 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f77fe640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f99f80a2530 msgr2=0x7f99f80a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f77fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f99f80a2530 0x7f99f80a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:20.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f99f80a4920 0x7f99f80a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f99f80a4920 0x7f99f80a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36206/0 (socket says 172.21.15.98:36206) 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 1 -- 172.21.15.98:0/1993674254 learned_addr learned my addr 172.21.15.98:0/1993674254 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 10 monclient(hunting): get_auth_request con 0x7f99f80a4920 auth_method 0 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 10 monclient(hunting): handle_auth_done global_id 65475 payload 274 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 1 -- 172.21.15.98:0/1993674254 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f99f80a2530 msgr2=0x7f99f80a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 1 --2- 172.21.15.98:0/1993674254 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f99f80a2530 0x7f99f80a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 1 -- 172.21.15.98:0/1993674254 >> v2:172.21.15.117:3300/0 conn(0x7f99f80a5240 msgr2=0x7f99f80a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:20.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 1 --2- 172.21.15.98:0/1993674254 >> v2:172.21.15.117:3300/0 conn(0x7f99f80a5240 0x7f99f80a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 1 monclient: found mon.noname-a 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.723+0000 7f99f6ffd640 1 -- 172.21.15.98:0/1993674254 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f80a3110 con 0x7f99f80a4920 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f6ffd640 1 --2- 172.21.15.98:0/1993674254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99f80a4920 0x7f99f80a4d00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f99f8073fc0 tx=0x7f99e802bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93093158b4eb3118 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 1 -- 172.21.15.98:0/1993674254 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99e802cbf0 con 0x7f99f80a4920 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 10 monclient: dump: 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.735 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 10 monclient: _finish_auth 0 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:20:20.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 1 -- 172.21.15.98:0/1993674254 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99e802cdb0 con 0x7f99f80a4920 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 5 monclient: authenticate success, global_id 65475 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 1 -- 172.21.15.98:0/1993674254 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99e8034860 con 0x7f99f80a4920 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 4 set_mon_vals no callback set 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99f5ffb640 10 monclient: dump: 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.737 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 10 monclient: shutdown 2024-06-11T19:20:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 1 -- 172.21.15.98:0/1993674254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99f80a4920 msgr2=0x7f99f80a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 1 --2- 172.21.15.98:0/1993674254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99f80a4920 0x7f99f80a4d00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f99f8073fc0 tx=0x7f99e802bde0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 1 -- 172.21.15.98:0/1993674254 shutdown_connections 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 1 --2- 172.21.15.98:0/1993674254 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f99f80a2530 0x7f99f80a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 1 --2- 172.21.15.98:0/1993674254 >> v2:172.21.15.117:3300/0 conn(0x7f99f80a5240 0x7f99f80a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 1 --2- 172.21.15.98:0/1993674254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99f80a4920 0x7f99f80a4d00 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 1 -- 172.21.15.98:0/1993674254 >> 172.21.15.98:0/1993674254 conn(0x7f99f80ab850 msgr2=0x7f99f80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 1 -- 172.21.15.98:0/1993674254 shutdown_connections 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.724+0000 7f99fda17640 1 -- 172.21.15.98:0/1993674254 wait complete. 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.725+0000 7f99fda17640 10 monclient: build_initial_monmap 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.725+0000 7f99fda17640 10 monclient: monmap: 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.725+0000 7f99fda17640 1 librados: starting msgr at 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.725+0000 7f99fda17640 1 librados: starting objecter 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.725+0000 7f99fda17640 1 Processor -- start 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 1 -- start start 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 1 librados: setting wanted keys 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 1 librados: calling monclient init 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 10 monclient: init 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 10 monclient: _add_conns ranks=[3,8,1,2,6,7,4,5,0] 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99f80a2530 0x7f99f81376d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 10 monclient(hunting): picked mon.noname-g con 0x7f99f80a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f99f80a4920 0x7f99f8137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 10 monclient(hunting): picked mon.noname-e con 0x7f99f80a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99f80a5240 0x7f99f81231c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 10 monclient(hunting): picked mon.noname-f con 0x7f99f80a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:20:20.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f99f80a6da0 con 0x7f99f80a5240 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f99f8138e50 con 0x7f99f80a2530 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f99f80a6420 con 0x7f99f80a4920 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99fda17640 10 monclient(hunting): authenticate will time out at 1430.533445s 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f67fc640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f99f80a4920 msgr2=0x7f99f8137c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f67fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f99f80a4920 0x7f99f8137c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f6ffd640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99f80a2530 msgr2=0x7f99f81376d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f6ffd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99f80a2530 0x7f99f81376d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99f80a5240 0x7f99f81231c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99f80a5240 0x7f99f81231c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:53678/0 (socket says 172.21.15.98:53678) 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f77fe640 1 -- 172.21.15.98:0/1051320571 learned_addr learned my addr 172.21.15.98:0/1051320571 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f77fe640 10 monclient(hunting): get_auth_request con 0x7f99f80a5240 auth_method 0 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.726+0000 7f99f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99f77fe640 10 monclient(hunting): handle_auth_done global_id 32545 payload 824 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99f77fe640 1 -- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f99f80a4920 msgr2=0x7f99f8137c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99f77fe640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f99f80a4920 0x7f99f8137c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99f77fe640 1 -- 172.21.15.98:0/1051320571 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99f80a2530 msgr2=0x7f99f81376d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99f77fe640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99f80a2530 0x7f99f81376d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99f77fe640 10 monclient: _finish_hunting 0 2024-06-11T19:20:20.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99f77fe640 1 monclient: found mon.noname-f 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99f77fe640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99f77fe640 1 -- 172.21.15.98:0/1051320571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f81c0ce0 con 0x7f99f80a5240 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99f77fe640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99f80a5240 0x7f99f81231c0 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f99f000b410 tx=0x7f99f000b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 10 client.?.objecter ms_handle_connect 0x7f99f80a5240 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99f0017070 con 0x7f99f80a5240 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 10 monclient: dump: 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.744 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 10 monclient: _finish_auth 0 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 10 monclient: _check_auth_tickets 2024-06-11T19:20:20.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f0003bc0 con 0x7f99f80a5240 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99fda17640 5 monclient: authenticate success, global_id 32545 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99fda17640 10 monclient: _renew_subs 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99fda17640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99fda17640 1 -- 172.21.15.98:0/1051320571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99f8126af0 con 0x7f99f80a5240 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99f0003db0 con 0x7f99f80a5240 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99fda17640 10 client.32545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99effff640 10 monclient: dump: 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.746 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99f57fa640 4 set_mon_vals no callback set 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99fda17640 10 monclient: _renew_subs 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99fda17640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99fda17640 1 -- 172.21.15.98:0/1051320571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99f8124d50 con 0x7f99f80a5240 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.727+0000 7f99fda17640 1 librados: init done 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99fda17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99fda17640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f8126d30 0x7f99f8063180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99fda17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99fda17640 1 -- 172.21.15.98:0/1051320571 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f99f8050830 con 0x7f99f8126d30 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99f6ffd640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f8126d30 0x7f99f8063180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99f6ffd640 10 monclient: get_auth_request con 0x7f99f8126d30 auth_method 0 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99f6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99f6ffd640 10 monclient(hunting): handle_auth_done global_id 84773 payload 824 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99f6ffd640 1 -- 172.21.15.98:0/1051320571 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f99f80a6730 con 0x7f99f8126d30 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.728+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f99f00060d0 con 0x7f99f80a5240 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99f6ffd640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f8126d30 0x7f99f8063180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f99f8073de0 tx=0x7f99e8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f99d803fdb0 0x7f99d8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 10 client.32545.objecter ms_handle_connect 0x7f99f8126d30 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 10 client.32545.objecter resend_mon_ops 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(423..423 src has 1..423) ==== 6230+0+0 (secure 0 0 0) 0x7f99f0050a80 con 0x7f99f80a5240 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 10 client.32545.objecter ms_dispatch 0x7f99f8000b90 osd_map(423..423 src has 1..423) 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 3 client.32545.objecter handle_osd_map got epochs [423,423] > 0 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 3 client.32545.objecter handle_osd_map decoding full epoch 423 2024-06-11T19:20:20.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 20 client.32545.objecter dump_active .. 0 homeless 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99e8042070 con 0x7f99f8126d30 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 10 monclient: dump: 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f99e802c780 con 0x7f99f8126d30 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f8126d30 msgr2=0x7f99f8063180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f8126d30 0x7f99f8063180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f99f8073de0 tx=0x7f99e8004a40 comp rx=0 tx=0).stop 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f8126d30 msgr2=0x7f99f8063180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99effff640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f8126d30 0x7f99f8063180 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f99f8073de0 tx=0x7f99e8004a40 comp rx=0 tx=0).stop 2024-06-11T19:20:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99f67fc640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f99d803fdb0 0x7f99d8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.729+0000 7f99f67fc640 10 monclient: get_auth_request con 0x7f99d803fdb0 auth_method 0 2024-06-11T19:20:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.730+0000 7f99f67fc640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f99d803fdb0 0x7f99d8042270 secure :-1 s=READY pgs=4223 cs=0 l=1 rev1=1 crypto rx=0x7f99e0000f50 tx=0x7f99e0007a70 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.730+0000 7f99effff640 10 client.32545.objecter ms_handle_connect 0x7f99d803fdb0 2024-06-11T19:20:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.735+0000 7f99fda17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.735+0000 7f99fda17640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f80636c0 0x7f99f802cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.735+0000 7f99fda17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.735+0000 7f99fda17640 1 -- 172.21.15.98:0/1051320571 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f99f81258b0 con 0x7f99f80636c0 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.735+0000 7f99f6ffd640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f80636c0 0x7f99f802cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.735+0000 7f99f6ffd640 10 monclient: get_auth_request con 0x7f99f80636c0 auth_method 0 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.735+0000 7f99f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.735+0000 7f99f6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.735+0000 7f99f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99f6ffd640 10 monclient(hunting): handle_auth_done global_id 84782 payload 824 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99f6ffd640 1 -- 172.21.15.98:0/1051320571 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f99f80a0100 con 0x7f99f80636c0 2024-06-11T19:20:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99f6ffd640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f80636c0 0x7f99f802cb40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f99f80a4d00 tx=0x7f99e8009810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 10 client.32545.objecter ms_handle_connect 0x7f99f80636c0 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 10 client.32545.objecter resend_mon_ops 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99e80480d0 con 0x7f99f80636c0 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 10 monclient: dump: 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9410 (secure 0 0 0) 0x7f99e80480d0 con 0x7f99f80636c0 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f80636c0 msgr2=0x7f99f802cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f80636c0 0x7f99f802cb40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f99f80a4d00 tx=0x7f99e8009810 comp rx=0 tx=0).stop 2024-06-11T19:20:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 1 -- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f80636c0 msgr2=0x7f99f802cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99effff640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f99f80636c0 0x7f99f802cb40 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f99f80a4d00 tx=0x7f99e8009810 comp rx=0 tx=0).stop 2024-06-11T19:20:20.757 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 58, 2024-06-11T19:20:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 292510, 2024-06-11T19:20:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:20.779 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:20.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99fda17640 10 librados: watch_flush enter 2024-06-11T19:20:20.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99fda17640 10 librados: watch_flush exit 2024-06-11T19:20:20.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99fda17640 20 client.32545.objecter shutdown clearing up homeless session... 2024-06-11T19:20:20.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.736+0000 7f99fda17640 10 client.32545.objecter successfully canceled tick 2024-06-11T19:20:20.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99fda17640 1 -- 172.21.15.98:0/1051320571 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f99d803fdb0 msgr2=0x7f99d8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99fda17640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f99d803fdb0 0x7f99d8042270 secure :-1 s=READY pgs=4223 cs=0 l=1 rev1=1 crypto rx=0x7f99e0000f50 tx=0x7f99e0007a70 comp rx=0 tx=0).stop 2024-06-11T19:20:20.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99fda17640 10 monclient: shutdown 2024-06-11T19:20:20.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99fda17640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:20.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99fda17640 1 -- 172.21.15.98:0/1051320571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99f80a5240 msgr2=0x7f99f81231c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99fda17640 1 --2- 172.21.15.98:0/1051320571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99f80a5240 0x7f99f81231c0 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f99f000b410 tx=0x7f99f000b8d0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99f77fe640 1 -- 172.21.15.98:0/1051320571 reap_dead start 2024-06-11T19:20:20.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99f77fe640 1 -- 172.21.15.98:0/1051320571 reap_dead start 2024-06-11T19:20:20.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99fda17640 1 -- 172.21.15.98:0/1051320571 shutdown_connections 2024-06-11T19:20:20.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99fda17640 1 -- 172.21.15.98:0/1051320571 >> 172.21.15.98:0/1051320571 conn(0x7f99f80ab850 msgr2=0x7f99f81befd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:20.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99fda17640 1 -- 172.21.15.98:0/1051320571 shutdown_connections 2024-06-11T19:20:20.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99fda17640 1 -- 172.21.15.98:0/1051320571 wait complete. 2024-06-11T19:20:20.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.737+0000 7f99fda17640 1 librados: shutdown 2024-06-11T19:20:20.782 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-06-11T19:20:20.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon d 2024-06-11T19:20:20.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:20:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.883+0000 7f956746a640 10 monclient: get_monmap_and_config 2024-06-11T19:20:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.883+0000 7f956746a640 10 monclient: build_initial_monmap 2024-06-11T19:20:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.883+0000 7f956746a640 10 monclient: monmap: 2024-06-11T19:20:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:20.884673+0000 2024-06-11T19:20:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:20.884673+0000 2024-06-11T19:20:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:20.886 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:20.886 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:20.886 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:20.886 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.884+0000 7f956746a640 1 Processor -- start 2024-06-11T19:20:20.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.884+0000 7f956746a640 1 -- start start 2024-06-11T19:20:20.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.884+0000 7f956746a640 10 monclient: init 2024-06-11T19:20:20.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.884+0000 7f956746a640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:20.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.884+0000 7f956746a640 10 monclient: _add_conns ranks=[1,2,5,7,8,0,9,6,10,3,4] 2024-06-11T19:20:20.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.884+0000 7f956746a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f95600a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:20:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.884+0000 7f956746a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 0x7f956013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.884+0000 7f956746a640 10 monclient(hunting): picked mon.noname-b con 0x7f95600a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.884+0000 7f956746a640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f956013e1a0 0x7f9560140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.884+0000 7f956746a640 10 monclient(hunting): picked mon.noname-h con 0x7f956013e1a0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f956746a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95600a7140 con 0x7f95600a4920 2024-06-11T19:20:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f956746a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f956746a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f95600a2b50 con 0x7f95600a2530 2024-06-11T19:20:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f956746a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f956746a640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f95600a2e30 con 0x7f956013e1a0 2024-06-11T19:20:20.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f956746a640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f956746a640 10 monclient(hunting): authenticate will time out at 1430.692039s 2024-06-11T19:20:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 0x7f956013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 0x7f956013dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:36384/0 (socket says 172.21.15.98:36384) 2024-06-11T19:20:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 1 -- 172.21.15.98:0/4161661266 learned_addr learned my addr 172.21.15.98:0/4161661266 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 10 monclient(hunting): get_auth_request con 0x7f95600a2530 auth_method 0 2024-06-11T19:20:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 10 monclient(hunting): handle_auth_done global_id 84791 payload 274 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9565a07640 1 --2- 172.21.15.98:0/4161661266 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f956013e1a0 0x7f9560140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 1 -- 172.21.15.98:0/4161661266 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f956013e1a0 msgr2=0x7f9560140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 1 --2- 172.21.15.98:0/4161661266 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f956013e1a0 0x7f9560140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 1 -- 172.21.15.98:0/4161661266 >> v1:172.21.15.117:6789/0 conn(0x7f95600a4920 legacy=0x7f95600a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 10 monclient: _finish_hunting 0 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 1 monclient: found mon.noname-b 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.885+0000 7f9564a05640 1 -- 172.21.15.98:0/4161661266 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95600a3110 con 0x7f95600a2530 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9564a05640 1 --2- 172.21.15.98:0/4161661266 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 0x7f956013dc60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f95500098a0 tx=0x7f955002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a04220de2a0c773 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 1 -- 172.21.15.98:0/4161661266 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9550038070 con 0x7f95600a2530 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 10 monclient: dump: 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.905 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 10 monclient: _finish_auth 0 2024-06-11T19:20:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 10 monclient: _check_auth_tickets 2024-06-11T19:20:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 5 monclient: authenticate success, global_id 84791 2024-06-11T19:20:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 1 -- 172.21.15.98:0/4161661266 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9550031970 con 0x7f95600a2530 2024-06-11T19:20:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:20.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 4 set_mon_vals no callback set 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 1 -- 172.21.15.98:0/4161661266 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9550031ba0 con 0x7f95600a2530 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f9557fff640 10 monclient: dump: 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.906 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 10 monclient: shutdown 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 1 -- 172.21.15.98:0/4161661266 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 msgr2=0x7f956013dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 1 --2- 172.21.15.98:0/4161661266 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 0x7f956013dc60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f95500098a0 tx=0x7f955002b7a0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 1 -- 172.21.15.98:0/4161661266 shutdown_connections 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 1 --2- 172.21.15.98:0/4161661266 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f956013e1a0 0x7f9560140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 1 --2- 172.21.15.98:0/4161661266 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 0x7f956013dc60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 1 -- 172.21.15.98:0/4161661266 >> 172.21.15.98:0/4161661266 conn(0x7f95600ab850 msgr2=0x7f95600a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 1 -- 172.21.15.98:0/4161661266 shutdown_connections 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.886+0000 7f956746a640 1 -- 172.21.15.98:0/4161661266 wait complete. 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.887+0000 7f956746a640 10 monclient: build_initial_monmap 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.887+0000 7f956746a640 10 monclient: monmap: 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:20.908 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:20.909 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:20.909 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.887+0000 7f956746a640 1 librados: starting msgr at 2024-06-11T19:20:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.887+0000 7f956746a640 1 librados: starting objecter 2024-06-11T19:20:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.887+0000 7f956746a640 1 Processor -- start 2024-06-11T19:20:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.887+0000 7f956746a640 1 -- start start 2024-06-11T19:20:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.887+0000 7f956746a640 1 librados: setting wanted keys 2024-06-11T19:20:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.887+0000 7f956746a640 1 librados: calling monclient init 2024-06-11T19:20:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.887+0000 7f956746a640 10 monclient: init 2024-06-11T19:20:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 10 monclient: _add_conns ranks=[2,0,8,7,1,6,5,3,4] 2024-06-11T19:20:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 0x7f95601cd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 10 monclient(hunting): picked mon.noname-b con 0x7f95600a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95600a4920 0x7f9560135d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 10 monclient(hunting): picked mon.noname-a con 0x7f95600a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:20:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f956013e1a0 0x7f95601362a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 10 monclient(hunting): picked mon.noname-e con 0x7f956013e1a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9560138e50 con 0x7f95600a4920 2024-06-11T19:20:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f95600a84b0 con 0x7f95600a2530 2024-06-11T19:20:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f95600a6da0 con 0x7f956013e1a0 2024-06-11T19:20:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 10 monclient(hunting): authenticate will time out at 1430.695228s 2024-06-11T19:20:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565a07640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f956013e1a0 msgr2=0x7f95601362a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:20:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565a07640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f956013e1a0 0x7f95601362a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 0x7f95601cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 0x7f95601cd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:36390/0 (socket says 172.21.15.98:36390) 2024-06-11T19:20:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 1 -- 172.21.15.98:0/3640534576 learned_addr learned my addr 172.21.15.98:0/3640534576 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 10 monclient(hunting): get_auth_request con 0x7f95600a2530 auth_method 0 2024-06-11T19:20:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9564a05640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95600a4920 0x7f9560135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9564a05640 10 monclient(hunting): get_auth_request con 0x7f95600a4920 auth_method 0 2024-06-11T19:20:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9564a05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9564a05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9564a05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9564a05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9564a05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9564a05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 10 monclient(hunting): handle_auth_done global_id 84800 payload 824 2024-06-11T19:20:20.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 1 -- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f956013e1a0 msgr2=0x7f95601362a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:20:20.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f956013e1a0 0x7f95601362a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 1 -- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95600a4920 msgr2=0x7f9560135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95600a4920 0x7f9560135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 10 monclient: _finish_hunting 0 2024-06-11T19:20:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 1 monclient: found mon.noname-b 2024-06-11T19:20:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:20:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 1 -- 172.21.15.98:0/3640534576 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95600a0980 con 0x7f95600a2530 2024-06-11T19:20:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9565206640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 0x7f95601cd800 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f955c00c440 tx=0x7f955c00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 10 client.?.objecter ms_handle_connect 0x7f95600a2530 2024-06-11T19:20:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f955c017070 con 0x7f95600a2530 2024-06-11T19:20:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 10 monclient: dump: 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 10 monclient: _finish_auth 0 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f955c005280 con 0x7f95600a2530 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 5 monclient: authenticate success, global_id 84800 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 10 monclient: _renew_subs 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:20:20.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f956746a640 1 -- 172.21.15.98:0/3640534576 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9560121e70 con 0x7f95600a2530 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f955c005430 con 0x7f95600a2530 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f9555ffb640 10 monclient: dump: 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.920 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.889+0000 7f956746a640 10 client.84800.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.889+0000 7f956746a640 10 monclient: _renew_subs 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.889+0000 7f956746a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.888+0000 7f95577fe640 4 set_mon_vals no callback set 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.889+0000 7f956746a640 1 -- 172.21.15.98:0/3640534576 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f95600a14e0 con 0x7f95600a2530 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.889+0000 7f956746a640 1 librados: init done 2024-06-11T19:20:20.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.889+0000 7f956746a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.889+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f955c0035f0 con 0x7f95600a2530 2024-06-11T19:20:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.889+0000 7f956746a640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009eec0 0x7f956009f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f956746a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f956746a640 1 -- 172.21.15.98:0/3640534576 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f95600a17a0 con 0x7f956009eec0 2024-06-11T19:20:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9564a05640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009eec0 0x7f956009f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9564a05640 10 monclient: get_auth_request con 0x7f956009eec0 auth_method 0 2024-06-11T19:20:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9564a05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9564a05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9564a05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f954403fdb0 0x7f9544042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9564a05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9564a05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9564a05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(423..423 src has 1..423) ==== 6230+0+0 (secure 0 0 0) 0x7f955c050cb0 con 0x7f95600a2530 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 10 client.84800.objecter ms_dispatch 0x7f9560000b90 osd_map(423..423 src has 1..423) 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 3 client.84800.objecter handle_osd_map got epochs [423,423] > 0 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 3 client.84800.objecter handle_osd_map decoding full epoch 423 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 20 client.84800.objecter dump_active .. 0 homeless 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9564a05640 10 monclient(hunting): handle_auth_done global_id 55446 payload 824 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9564a05640 1 -- 172.21.15.98:0/3640534576 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95600a4d00 con 0x7f956009eec0 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9565a07640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f954403fdb0 0x7f9544042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9565a07640 10 monclient: get_auth_request con 0x7f954403fdb0 auth_method 0 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 10 client.84800.objecter ms_handle_connect 0x7f956009eec0 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 10 client.84800.objecter resend_mon_ops 2024-06-11T19:20:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9564a05640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009eec0 0x7f956009f2a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9560073de0 tx=0x7f95500046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9550038040 con 0x7f956009eec0 2024-06-11T19:20:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 10 monclient: dump: 2024-06-11T19:20:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9555ffb640 10 client.84800.objecter ms_handle_connect 0x7f954403fdb0 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.890+0000 7f9565a07640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f954403fdb0 0x7f9544042270 secure :-1 s=READY pgs=4224 cs=0 l=1 rev1=1 crypto rx=0x7f95600aaab0 tx=0x7f9558007250 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.891+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9550031ea0 con 0x7f956009eec0 2024-06-11T19:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.891+0000 7f9555ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.891+0000 7f9555ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.891+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009eec0 msgr2=0x7f956009f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.891+0000 7f9555ffb640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009eec0 0x7f956009f2a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9560073de0 tx=0x7f95500046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.891+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009eec0 msgr2=0x7f956009f2a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.891+0000 7f9555ffb640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009eec0 0x7f956009f2a0 secure :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9560073de0 tx=0x7f95500046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f956746a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f956746a640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009f7e0 0x7f956013bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f956746a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f956746a640 1 -- 172.21.15.98:0/3640534576 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f9560122040 con 0x7f956009f7e0 2024-06-11T19:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f9564a05640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009f7e0 0x7f956013bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f9564a05640 10 monclient: get_auth_request con 0x7f956009f7e0 auth_method 0 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f9564a05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f9564a05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f9564a05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f9564a05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f9564a05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f9564a05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f9564a05640 10 monclient(hunting): handle_auth_done global_id 55455 payload 824 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.897+0000 7f9564a05640 1 -- 172.21.15.98:0/3640534576 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f956006c300 con 0x7f956009f7e0 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 10 client.84800.objecter ms_handle_connect 0x7f956009f7e0 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 10 client.84800.objecter resend_mon_ops 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9564a05640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009f7e0 0x7f956013bee0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f95500097c0 tx=0x7f9550031000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9550038050 con 0x7f956009f7e0 2024-06-11T19:20:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 10 monclient: dump: 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:20.929 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9410 (secure 0 0 0) 0x7f955004f070 con 0x7f956009f7e0 2024-06-11T19:20:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009f7e0 msgr2=0x7f956013bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009f7e0 0x7f956013bee0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f95500097c0 tx=0x7f9550031000 comp rx=0 tx=0).stop 2024-06-11T19:20:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 1 -- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009f7e0 msgr2=0x7f956013bee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9555ffb640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f956009f7e0 0x7f956013bee0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:20.930 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 58, 2024-06-11T19:20:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 533655, 2024-06-11T19:20:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:20:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:20.953 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:20.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 10 librados: watch_flush enter 2024-06-11T19:20:20.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 10 librados: watch_flush exit 2024-06-11T19:20:20.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 20 client.84800.objecter shutdown clearing up homeless session... 2024-06-11T19:20:20.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 10 client.84800.objecter successfully canceled tick 2024-06-11T19:20:20.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 1 -- 172.21.15.98:0/3640534576 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f954403fdb0 msgr2=0x7f9544042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f954403fdb0 0x7f9544042270 secure :-1 s=READY pgs=4224 cs=0 l=1 rev1=1 crypto rx=0x7f95600aaab0 tx=0x7f9558007250 comp rx=0 tx=0).stop 2024-06-11T19:20:20.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 10 monclient: shutdown 2024-06-11T19:20:20.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:20.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 1 -- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 msgr2=0x7f95601cd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:20.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 1 --2- 172.21.15.98:0/3640534576 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95600a2530 0x7f95601cd800 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f955c00c440 tx=0x7f955c00c900 comp rx=0 tx=0).stop 2024-06-11T19:20:20.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9565a07640 1 -- 172.21.15.98:0/3640534576 reap_dead start 2024-06-11T19:20:20.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f9565a07640 1 -- 172.21.15.98:0/3640534576 reap_dead start 2024-06-11T19:20:20.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 1 -- 172.21.15.98:0/3640534576 shutdown_connections 2024-06-11T19:20:20.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 1 -- 172.21.15.98:0/3640534576 >> 172.21.15.98:0/3640534576 conn(0x7f95600ab850 msgr2=0x7f9560137b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:20.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 1 -- 172.21.15.98:0/3640534576 shutdown_connections 2024-06-11T19:20:20.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 1 -- 172.21.15.98:0/3640534576 wait complete. 2024-06-11T19:20:20.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:20.898+0000 7f956746a640 1 librados: shutdown 2024-06-11T19:20:20.957 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-06-11T19:20:20.957 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon f 2024-06-11T19:20:20.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:20:21.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.058+0000 7f7d0f4f1640 10 monclient: get_monmap_and_config 2024-06-11T19:20:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.058+0000 7f7d0f4f1640 10 monclient: build_initial_monmap 2024-06-11T19:20:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.058+0000 7f7d0f4f1640 10 monclient: monmap: 2024-06-11T19:20:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:21.059 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.060 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:21.059584+0000 2024-06-11T19:20:21.060 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:21.059584+0000 2024-06-11T19:20:21.060 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:21.060 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.060 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:21.061 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:21.061 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:21.061 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:21.061 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:21.062 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:21.062 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:21.062 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:21.062 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:21.063 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:21.063 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:21.063 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.059+0000 7f7d0f4f1640 1 Processor -- start 2024-06-11T19:20:21.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.059+0000 7f7d0f4f1640 1 -- start start 2024-06-11T19:20:21.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.059+0000 7f7d0f4f1640 10 monclient: init 2024-06-11T19:20:21.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.059+0000 7f7d0f4f1640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:21.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.059+0000 7f7d0f4f1640 10 monclient: _add_conns ranks=[10,5,9,4,3,0,2,7,1,6,8] 2024-06-11T19:20:21.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.059+0000 7f7d0f4f1640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7d080a4920 0x7f7d080a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 10 monclient(hunting): picked mon.noname-f con 0x7f7d080a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:21.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7d080a5240 0x7f7d080a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 10 monclient(hunting): picked mon.noname-h con 0x7f7d080a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:21.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d080a2530 0x7f7d080a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f7d080a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:20:21.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7d080a7140 con 0x7f7d080a2530 2024-06-11T19:20:21.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f7d080a2b50 con 0x7f7d080a4920 2024-06-11T19:20:21.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f7d080a2e30 con 0x7f7d080a5240 2024-06-11T19:20:21.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:21.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0f4f1640 10 monclient(hunting): authenticate will time out at 1430.867072s 2024-06-11T19:20:21.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d080a2530 0x7f7d080a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d080a2530 0x7f7d080a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36218/0 (socket says 172.21.15.98:36218) 2024-06-11T19:20:21.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 1 -- 172.21.15.98:0/84783509 learned_addr learned my addr 172.21.15.98:0/84783509 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:21.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 10 monclient(hunting): get_auth_request con 0x7f7d080a2530 auth_method 0 2024-06-11T19:20:21.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0ca8c640 1 --2- 172.21.15.98:0/84783509 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7d080a5240 0x7f7d080a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0ca8c640 10 monclient(hunting): get_auth_request con 0x7f7d080a5240 auth_method 0 2024-06-11T19:20:21.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0ca8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0ca8c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0ca8c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0d28d640 1 --2- 172.21.15.98:0/84783509 >> v2:172.21.15.117:3300/0 conn(0x7f7d080a4920 0x7f7d080a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0d28d640 10 monclient(hunting): get_auth_request con 0x7f7d080a4920 auth_method 0 2024-06-11T19:20:21.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0d28d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0d28d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0d28d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0ca8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0ca8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0ca8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0d28d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0d28d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0d28d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 10 monclient(hunting): handle_auth_done global_id 65493 payload 274 2024-06-11T19:20:21.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 1 -- 172.21.15.98:0/84783509 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7d080a5240 msgr2=0x7f7d080a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:21.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 1 --2- 172.21.15.98:0/84783509 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7d080a5240 0x7f7d080a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 1 -- 172.21.15.98:0/84783509 >> v2:172.21.15.117:3300/0 conn(0x7f7d080a4920 msgr2=0x7f7d080a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:21.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 1 --2- 172.21.15.98:0/84783509 >> v2:172.21.15.117:3300/0 conn(0x7f7d080a4920 0x7f7d080a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 10 monclient: _finish_hunting 0 2024-06-11T19:20:21.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 1 monclient: found mon.noname-a 2024-06-11T19:20:21.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:21.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:20:21.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.060+0000 7f7d0da8e640 1 -- 172.21.15.98:0/84783509 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d080a3110 con 0x7f7d080a2530 2024-06-11T19:20:21.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.061+0000 7f7d0da8e640 1 --2- 172.21.15.98:0/84783509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d080a2530 0x7f7d080a5ee0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7d08073de0 tx=0x7f7d0002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b096f03ec894718a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.061+0000 7f7cfffff640 1 -- 172.21.15.98:0/84783509 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d0002cbd0 con 0x7f7d080a2530 2024-06-11T19:20:21.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.061+0000 7f7cfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.061+0000 7f7cfffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:20:21.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.061+0000 7f7cfffff640 10 monclient: dump: 2024-06-11T19:20:21.097 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.097 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.097 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.097 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.098 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.098 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.098 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.098 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.098 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.099 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.099 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.099 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.099 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.099 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.099 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.099 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.061+0000 7f7cfffff640 10 monclient: _finish_auth 0 2024-06-11T19:20:21.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.061+0000 7f7cfffff640 10 monclient: _check_auth_tickets 2024-06-11T19:20:21.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.061+0000 7f7cfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:21.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 5 monclient: authenticate success, global_id 65493 2024-06-11T19:20:21.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:21.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7cfffff640 1 -- 172.21.15.98:0/84783509 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d0002cd90 con 0x7f7d080a2530 2024-06-11T19:20:21.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7cfffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:21.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:21.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7cfffff640 1 -- 172.21.15.98:0/84783509 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d00034860 con 0x7f7d080a2530 2024-06-11T19:20:21.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 4 set_mon_vals no callback set 2024-06-11T19:20:21.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7cfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7cfffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:20:21.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7cfffff640 10 monclient: dump: 2024-06-11T19:20:21.101 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.101 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.101 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.101 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.101 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.102 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.102 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.102 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.102 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.103 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.103 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.103 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.103 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.103 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.103 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.103 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 10 monclient: shutdown 2024-06-11T19:20:21.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:21.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/84783509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d080a2530 msgr2=0x7f7d080a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 1 --2- 172.21.15.98:0/84783509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d080a2530 0x7f7d080a5ee0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7d08073de0 tx=0x7f7d0002bdc0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/84783509 shutdown_connections 2024-06-11T19:20:21.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 1 --2- 172.21.15.98:0/84783509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d080a2530 0x7f7d080a5ee0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 1 --2- 172.21.15.98:0/84783509 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7d080a5240 0x7f7d080a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 1 --2- 172.21.15.98:0/84783509 >> v2:172.21.15.117:3300/0 conn(0x7f7d080a4920 0x7f7d080a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/84783509 >> 172.21.15.98:0/84783509 conn(0x7f7d080ab850 msgr2=0x7f7d080a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:21.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/84783509 shutdown_connections 2024-06-11T19:20:21.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.062+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/84783509 wait complete. 2024-06-11T19:20:21.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.063+0000 7f7d0f4f1640 10 monclient: build_initial_monmap 2024-06-11T19:20:21.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.063+0000 7f7d0f4f1640 10 monclient: monmap: 2024-06-11T19:20:21.105 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:21.105 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:21.105 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:21.106 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:21.106 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:21.106 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:21.106 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:21.106 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:21.107 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:21.107 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:21.107 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:21.107 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:21.107 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:21.107 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:21.107 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:21.108 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.063+0000 7f7d0f4f1640 1 librados: starting msgr at 2024-06-11T19:20:21.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.063+0000 7f7d0f4f1640 1 librados: starting objecter 2024-06-11T19:20:21.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.063+0000 7f7d0f4f1640 1 Processor -- start 2024-06-11T19:20:21.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.063+0000 7f7d0f4f1640 1 -- start start 2024-06-11T19:20:21.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.063+0000 7f7d0f4f1640 1 librados: setting wanted keys 2024-06-11T19:20:21.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.063+0000 7f7d0f4f1640 1 librados: calling monclient init 2024-06-11T19:20:21.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.063+0000 7f7d0f4f1640 10 monclient: init 2024-06-11T19:20:21.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:21.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 10 monclient: _add_conns ranks=[5,6,0,8,3,1,4,2,7] 2024-06-11T19:20:21.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7d080a2530 0x7f7d081cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 10 monclient(hunting): picked mon.noname-h con 0x7f7d080a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:20:21.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7d080a4920 0x7f7d081ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 10 monclient(hunting): picked mon.noname-d con 0x7f7d080a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:21.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d080a5240 0x7f7d08122030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f7d080a5240 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:20:21.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d080a6da0 con 0x7f7d080a5240 2024-06-11T19:20:21.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f7d08138e50 con 0x7f7d080a2530 2024-06-11T19:20:21.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f7d080a6420 con 0x7f7d080a4920 2024-06-11T19:20:21.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:21.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0f4f1640 10 monclient(hunting): authenticate will time out at 1430.871230s 2024-06-11T19:20:21.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0da8e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d080a5240 0x7f7d08122030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0ca8c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7d080a4920 0x7f7d081ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0ca8c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7d080a4920 0x7f7d081ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:41056/0 (socket says 172.21.15.98:41056) 2024-06-11T19:20:21.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0da8e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d080a5240 0x7f7d08122030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36224/0 (socket says 172.21.15.98:36224) 2024-06-11T19:20:21.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0ca8c640 1 -- 172.21.15.98:0/2364677067 learned_addr learned my addr 172.21.15.98:0/2364677067 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:21.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7d080a2530 0x7f7d081cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0ca8c640 10 monclient(hunting): get_auth_request con 0x7f7d080a4920 auth_method 0 2024-06-11T19:20:21.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0ca8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0ca8c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0ca8c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0da8e640 10 monclient(hunting): get_auth_request con 0x7f7d080a5240 auth_method 0 2024-06-11T19:20:21.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0da8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0da8e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0da8e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 10 monclient(hunting): get_auth_request con 0x7f7d080a2530 auth_method 0 2024-06-11T19:20:21.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0ca8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0ca8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0ca8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0da8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0da8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0da8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 10 monclient(hunting): handle_auth_done global_id 23351 payload 824 2024-06-11T19:20:21.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 1 -- 172.21.15.98:0/2364677067 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7d080a4920 msgr2=0x7f7d081ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7d080a4920 0x7f7d081ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 1 -- 172.21.15.98:0/2364677067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d080a5240 msgr2=0x7f7d08122030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d080a5240 0x7f7d08122030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 10 monclient: _finish_hunting 0 2024-06-11T19:20:21.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 1 monclient: found mon.noname-h 2024-06-11T19:20:21.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:21.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:20:21.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 1 -- 172.21.15.98:0/2364677067 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d08125a30 con 0x7f7d080a2530 2024-06-11T19:20:21.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0ca8c640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7d080a4920 0x7f7d081ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:21.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0da8e640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d080a5240 0x7f7d08122030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:21.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7d0d28d640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7d080a2530 0x7f7d081cdad0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400e850 tx=0x7f7d0400ed10 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7cfdffb640 10 client.?.objecter ms_handle_connect 0x7f7d080a2530 2024-06-11T19:20:21.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.064+0000 7f7cfdffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:21.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d04017070 con 0x7f7d080a2530 2024-06-11T19:20:21.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:20:21.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 10 monclient: dump: 2024-06-11T19:20:21.115 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.115 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.116 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.116 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.116 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.116 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.116 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.116 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.116 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.116 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.116 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.116 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.116 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.116 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.117 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.117 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 10 monclient: _finish_auth 0 2024-06-11T19:20:21.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 10 monclient: _check_auth_tickets 2024-06-11T19:20:21.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:21.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7d0f4f1640 5 monclient: authenticate success, global_id 23351 2024-06-11T19:20:21.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d0400cdb0 con 0x7f7d080a2530 2024-06-11T19:20:21.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:21.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d040138e0 con 0x7f7d080a2530 2024-06-11T19:20:21.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7d0f4f1640 10 monclient: _renew_subs 2024-06-11T19:20:21.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7d0f4f1640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:20:21.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/2364677067 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d08124860 con 0x7f7d080a2530 2024-06-11T19:20:21.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:20:21.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cff7fe640 4 set_mon_vals no callback set 2024-06-11T19:20:21.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7cfdffb640 10 monclient: dump: 2024-06-11T19:20:21.118 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.118 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.118 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.118 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.118 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.119 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.119 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.119 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.119 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.119 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.119 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.119 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.119 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.119 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.119 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.119 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7d0f4f1640 10 client.23351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:21.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7d0f4f1640 10 monclient: _renew_subs 2024-06-11T19:20:21.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7d0f4f1640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:20:21.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/2364677067 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d080a8e10 con 0x7f7d080a2530 2024-06-11T19:20:21.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.065+0000 7f7d0f4f1640 1 librados: init done 2024-06-11T19:20:21.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.066+0000 7f7d0f4f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:21.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.066+0000 7f7d0f4f1640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d08125c70 0x7f7d0809e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.066+0000 7f7d0f4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.066+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/2364677067 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7d08050830 con 0x7f7d08125c70 2024-06-11T19:20:21.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.066+0000 7f7d0ca8c640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d08125c70 0x7f7d0809e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.066+0000 7f7d0ca8c640 10 monclient: get_auth_request con 0x7f7d08125c70 auth_method 0 2024-06-11T19:20:21.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.066+0000 7f7d0ca8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.066+0000 7f7d0ca8c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.066+0000 7f7d0ca8c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.066+0000 7f7d0ca8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.066+0000 7f7d0ca8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.066+0000 7f7d0ca8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.067+0000 7f7d0ca8c640 10 monclient(hunting): handle_auth_done global_id 32563 payload 824 2024-06-11T19:20:21.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.067+0000 7f7d0ca8c640 1 -- 172.21.15.98:0/2364677067 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d08126050 con 0x7f7d08125c70 2024-06-11T19:20:21.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.067+0000 7f7d0ca8c640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d08125c70 0x7f7d0809e2f0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8011c40 tx=0x7f7cf800f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.068+0000 7f7cfdffb640 10 client.23351.objecter ms_handle_connect 0x7f7d08125c70 2024-06-11T19:20:21.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.068+0000 7f7cfdffb640 10 client.23351.objecter resend_mon_ops 2024-06-11T19:20:21.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.073+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7cf8019070 con 0x7f7d08125c70 2024-06-11T19:20:21.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.073+0000 7f7cfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.073+0000 7f7cfdffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:20:21.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.073+0000 7f7cfdffb640 10 monclient: dump: 2024-06-11T19:20:21.122 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.122 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.122 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.122 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.122 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.122 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.122 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.123 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.123 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.123 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.123 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.123 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.123 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.123 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.123 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.123 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.073+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7cf8004030 con 0x7f7d08125c70 2024-06-11T19:20:21.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.073+0000 7f7cfdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:21.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.073+0000 7f7cfdffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:21.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.073+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d08125c70 msgr2=0x7f7d0809e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.073+0000 7f7cfdffb640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d08125c70 0x7f7d0809e2f0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8011c40 tx=0x7f7cf800f610 comp rx=0 tx=0).stop 2024-06-11T19:20:21.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.073+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d08125c70 msgr2=0x7f7d0809e2f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:21.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.073+0000 7f7cfdffb640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d08125c70 0x7f7d0809e2f0 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.074+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 <== mon.5 v2:172.21.15.117:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f7d0401a070 con 0x7f7d080a2530 2024-06-11T19:20:21.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.074+0000 7f7cfdffb640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7cec03fbb0 0x7f7cec042070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.074+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 <== mon.5 v2:172.21.15.117:3302/0 5 ==== osd_map(424..424 src has 1..424) ==== 6230+0+0 (secure 0 0 0) 0x7f7d0400f040 con 0x7f7d080a2530 2024-06-11T19:20:21.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.074+0000 7f7cfdffb640 10 client.23351.objecter ms_dispatch 0x7f7d08000b90 osd_map(424..424 src has 1..424) 2024-06-11T19:20:21.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.074+0000 7f7cfdffb640 3 client.23351.objecter handle_osd_map got epochs [424,424] > 0 2024-06-11T19:20:21.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.074+0000 7f7cfdffb640 3 client.23351.objecter handle_osd_map decoding full epoch 424 2024-06-11T19:20:21.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.074+0000 7f7cfdffb640 20 client.23351.objecter dump_active .. 0 homeless 2024-06-11T19:20:21.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.074+0000 7f7d0ca8c640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7cec03fbb0 0x7f7cec042070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.074+0000 7f7d0ca8c640 10 monclient: get_auth_request con 0x7f7cec03fbb0 auth_method 0 2024-06-11T19:20:21.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.075+0000 7f7cfdffb640 10 client.23351.objecter ms_handle_connect 0x7f7cec03fbb0 2024-06-11T19:20:21.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.075+0000 7f7d0ca8c640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7cec03fbb0 0x7f7cec042070 secure :-1 s=READY pgs=4225 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8000c00 tx=0x7f7cf8008070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.081+0000 7f7d0f4f1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.081+0000 7f7d0f4f1640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d0809e830 0x7f7d0809ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.081+0000 7f7d0f4f1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.081+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/2364677067 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7d080a9040 con 0x7f7d0809e830 2024-06-11T19:20:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7d0da8e640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d0809e830 0x7f7d0809ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7d0da8e640 10 monclient: get_auth_request con 0x7f7d0809e830 auth_method 0 2024-06-11T19:20:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7d0da8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7d0da8e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7d0da8e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7d0da8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7d0da8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7d0da8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7d0da8e640 10 monclient(hunting): handle_auth_done global_id 32572 payload 824 2024-06-11T19:20:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7d0da8e640 1 -- 172.21.15.98:0/2364677067 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7d080a66f0 con 0x7f7d0809e830 2024-06-11T19:20:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7d0da8e640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d0809e830 0x7f7d0809ec10 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f7d080abfb0 tx=0x7f7d00004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7cfdffb640 10 client.23351.objecter ms_handle_connect 0x7f7d0809e830 2024-06-11T19:20:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7cfdffb640 10 client.23351.objecter resend_mon_ops 2024-06-11T19:20:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d00043030 con 0x7f7d0809e830 2024-06-11T19:20:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7cfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7cfdffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:20:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.082+0000 7f7cfdffb640 10 monclient: dump: 2024-06-11T19:20:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.127 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.128 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.129 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9412 (secure 0 0 0) 0x7f7d00004250 con 0x7f7d0809e830 2024-06-11T19:20:21.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7cfdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:21.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7cfdffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:21.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d0809e830 msgr2=0x7f7d0809ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7cfdffb640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d0809e830 0x7f7d0809ec10 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f7d080abfb0 tx=0x7f7d00004a40 comp rx=0 tx=0).stop 2024-06-11T19:20:21.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7cfdffb640 1 -- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d0809e830 msgr2=0x7f7d0809ec10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:21.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7cfdffb640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d0809e830 0x7f7d0809ec10 secure :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f7d080abfb0 tx=0x7f7d00004a40 comp rx=0 tx=0).stop 2024-06-11T19:20:21.130 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:21.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:21.130 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:21.130 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:20:21.130 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 58, 2024-06-11T19:20:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 671300, 2024-06-11T19:20:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:20:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:20:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:21.133 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:21.133 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:21.133 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:21.134 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:21.134 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.134 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:21.134 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:21.134 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:21.138 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:21.138 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:21.138 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:21.138 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:21.141 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:21.141 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:21.141 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:21.141 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:21.142 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:21.142 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:21.142 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.142 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:21.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.143 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:21.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.143 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:21.143 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:21.143 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:21.145 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.146 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:21.146 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:21.146 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.146 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.146 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.153 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:21.153 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:21.153 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.153 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.153 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.158 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:21.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.159 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.159 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:21.159 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:20:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:21.170 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:21.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 10 librados: watch_flush enter 2024-06-11T19:20:21.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 10 librados: watch_flush exit 2024-06-11T19:20:21.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 20 client.23351.objecter shutdown clearing up homeless session... 2024-06-11T19:20:21.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 10 client.23351.objecter successfully canceled tick 2024-06-11T19:20:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7cec03fbb0 msgr2=0x7f7cec042070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7cec03fbb0 0x7f7cec042070 secure :-1 s=READY pgs=4225 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8000c00 tx=0x7f7cf8008070 comp rx=0 tx=0).stop 2024-06-11T19:20:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 10 monclient: shutdown 2024-06-11T19:20:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7d080a2530 msgr2=0x7f7d081cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7d080a2530 0x7f7d081cdad0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400e850 tx=0x7f7d0400ed10 comp rx=0 tx=0).stop 2024-06-11T19:20:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0da8e640 1 -- 172.21.15.98:0/2364677067 reap_dead start 2024-06-11T19:20:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/2364677067 shutdown_connections 2024-06-11T19:20:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 1 --2- 172.21.15.98:0/2364677067 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7d080a2530 0x7f7d081cdad0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/2364677067 >> 172.21.15.98:0/2364677067 conn(0x7f7d080ab850 msgr2=0x7f7d080726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:21.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/2364677067 shutdown_connections 2024-06-11T19:20:21.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 1 -- 172.21.15.98:0/2364677067 wait complete. 2024-06-11T19:20:21.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.083+0000 7f7d0f4f1640 1 librados: shutdown 2024-06-11T19:20:21.174 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-06-11T19:20:21.174 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon h 2024-06-11T19:20:21.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:20:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.275+0000 7fcd61574640 10 monclient: get_monmap_and_config 2024-06-11T19:20:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.275+0000 7fcd61574640 10 monclient: build_initial_monmap 2024-06-11T19:20:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.275+0000 7fcd61574640 10 monclient: monmap: 2024-06-11T19:20:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:21.276958+0000 2024-06-11T19:20:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:21.276958+0000 2024-06-11T19:20:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:21.277 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.276+0000 7fcd61574640 1 Processor -- start 2024-06-11T19:20:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.276+0000 7fcd61574640 1 -- start start 2024-06-11T19:20:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.276+0000 7fcd61574640 10 monclient: init 2024-06-11T19:20:21.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.276+0000 7fcd61574640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:21.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.276+0000 7fcd61574640 10 monclient: _add_conns ranks=[7,3,4,5,9,2,10,0,1,8,6] 2024-06-11T19:20:21.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.276+0000 7fcd61574640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcd5c0a4920 0x7fcd5c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 10 monclient(hunting): picked mon.noname-i con 0x7fcd5c0a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:21.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd5c0a5240 0x7fcd5c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 10 monclient(hunting): picked mon.noname-g con 0x7fcd5c0a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:20:21.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd5c0a2530 0x7fcd5c0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 10 monclient(hunting): picked mon.noname-c con 0x7fcd5c0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:20:21.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fcd5c0a7140 con 0x7fcd5c0a5240 2024-06-11T19:20:21.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fcd5c0a2b50 con 0x7fcd5c0a2530 2024-06-11T19:20:21.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fcd5c0a2e30 con 0x7fcd5c0a4920 2024-06-11T19:20:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd61574640 10 monclient(hunting): authenticate will time out at 1431.084047s 2024-06-11T19:20:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5b7fe640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd5c0a2530 msgr2=0x7fcd5c0a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:20:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5b7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd5c0a2530 0x7fcd5c0a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5a7fc640 1 -- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd5c0a5240 msgr2=0x7fcd5c0a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3301/0 2024-06-11T19:20:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5a7fc640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd5c0a5240 0x7fcd5c0a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcd5c0a4920 0x7fcd5c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcd5c0a4920 0x7fcd5c0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:55762/0 (socket says 172.21.15.98:55762) 2024-06-11T19:20:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 1 -- 172.21.15.98:0/151447363 learned_addr learned my addr 172.21.15.98:0/151447363 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 10 monclient(hunting): get_auth_request con 0x7fcd5c0a4920 auth_method 0 2024-06-11T19:20:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 10 monclient(hunting): handle_auth_done global_id 84598 payload 274 2024-06-11T19:20:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 1 -- 172.21.15.98:0/151447363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd5c0a2530 msgr2=0x7fcd5c0a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:20:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 1 --2- 172.21.15.98:0/151447363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd5c0a2530 0x7fcd5c0a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 1 -- 172.21.15.98:0/151447363 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd5c0a5240 msgr2=0x7fcd5c0a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:20:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 1 --2- 172.21.15.98:0/151447363 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd5c0a5240 0x7fcd5c0a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 10 monclient: _finish_hunting 0 2024-06-11T19:20:21.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 1 monclient: found mon.noname-i 2024-06-11T19:20:21.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:21.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:20:21.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.277+0000 7fcd5affd640 1 -- 172.21.15.98:0/151447363 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5c0a3110 con 0x7fcd5c0a4920 2024-06-11T19:20:21.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd5affd640 1 --2- 172.21.15.98:0/151447363 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcd5c0a4920 0x7fcd5c0a4d00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c0098a0 tx=0x7fcd4c02b7e0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=59d63e3099c26fea server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 1 -- 172.21.15.98:0/151447363 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd4c038070 con 0x7fcd5c0a4920 2024-06-11T19:20:21.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:20:21.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 10 monclient: dump: 2024-06-11T19:20:21.297 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.298 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.298 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.298 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.298 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.298 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.298 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.298 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.298 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.298 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.298 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.298 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.299 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.299 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.299 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.299 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 10 monclient: _finish_auth 0 2024-06-11T19:20:21.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:20:21.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:21.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 1 -- 172.21.15.98:0/151447363 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd4c0319b0 con 0x7fcd5c0a4920 2024-06-11T19:20:21.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd61574640 5 monclient: authenticate success, global_id 84598 2024-06-11T19:20:21.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd61574640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:21.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:21.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd61574640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:21.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 1 -- 172.21.15.98:0/151447363 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd4c02f050 con 0x7fcd5c0a4920 2024-06-11T19:20:21.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd61574640 4 set_mon_vals no callback set 2024-06-11T19:20:21.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:20:21.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.278+0000 7fcd59ffb640 10 monclient: dump: 2024-06-11T19:20:21.300 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.300 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.300 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.300 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.300 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.300 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.301 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.301 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.301 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.301 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.301 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.301 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.301 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.301 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.301 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.301 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 10 monclient: shutdown 2024-06-11T19:20:21.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:21.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 1 -- 172.21.15.98:0/151447363 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcd5c0a4920 msgr2=0x7fcd5c0a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 1 --2- 172.21.15.98:0/151447363 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcd5c0a4920 0x7fcd5c0a4d00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c0098a0 tx=0x7fcd4c02b7e0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 1 -- 172.21.15.98:0/151447363 shutdown_connections 2024-06-11T19:20:21.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 1 --2- 172.21.15.98:0/151447363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd5c0a2530 0x7fcd5c0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 1 --2- 172.21.15.98:0/151447363 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd5c0a5240 0x7fcd5c0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 1 --2- 172.21.15.98:0/151447363 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcd5c0a4920 0x7fcd5c0a4d00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 1 -- 172.21.15.98:0/151447363 >> 172.21.15.98:0/151447363 conn(0x7fcd5c0ab850 msgr2=0x7fcd5c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:21.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 1 -- 172.21.15.98:0/151447363 shutdown_connections 2024-06-11T19:20:21.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 1 -- 172.21.15.98:0/151447363 wait complete. 2024-06-11T19:20:21.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 10 monclient: build_initial_monmap 2024-06-11T19:20:21.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 10 monclient: monmap: 2024-06-11T19:20:21.303 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:21.303 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:21.303 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:21.303 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:21.303 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:21.303 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:21.303 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:21.303 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:21.304 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:21.304 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:21.304 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:21.304 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:21.304 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:21.304 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:21.304 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:21.304 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 1 librados: starting msgr at 2024-06-11T19:20:21.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.279+0000 7fcd61574640 1 librados: starting objecter 2024-06-11T19:20:21.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 1 Processor -- start 2024-06-11T19:20:21.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 1 -- start start 2024-06-11T19:20:21.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 1 librados: setting wanted keys 2024-06-11T19:20:21.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 1 librados: calling monclient init 2024-06-11T19:20:21.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 10 monclient: init 2024-06-11T19:20:21.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:21.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 10 monclient: _add_conns ranks=[0,2,6,3,1,4,5,8,7] 2024-06-11T19:20:21.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd5c0a2530 0x7fcd5c1cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 10 monclient(hunting): picked mon.noname-a con 0x7fcd5c0a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:20:21.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd5c0a4920 0x7fcd5c1ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 10 monclient(hunting): picked mon.noname-b con 0x7fcd5c0a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:21.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd5c0a5240 0x7fcd5c123000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 10 monclient(hunting): picked mon.noname-d con 0x7fcd5c0a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:21.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c0a6da0 con 0x7fcd5c0a2530 2024-06-11T19:20:21.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcd5c138e50 con 0x7fcd5c0a4920 2024-06-11T19:20:21.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fcd5c0a6420 con 0x7fcd5c0a5240 2024-06-11T19:20:21.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:21.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd61574640 10 monclient(hunting): authenticate will time out at 1431.087671s 2024-06-11T19:20:21.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5a7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd5c0a4920 0x7fcd5c1ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd5c0a2530 0x7fcd5c1cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5b7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd5c0a5240 0x7fcd5c123000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5a7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd5c0a4920 0x7fcd5c1ce290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:36404/0 (socket says 172.21.15.98:36404) 2024-06-11T19:20:21.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5a7fc640 1 -- 172.21.15.98:0/3682408882 learned_addr learned my addr 172.21.15.98:0/3682408882 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:21.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5b7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd5c0a5240 0x7fcd5c123000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:41072/0 (socket says 172.21.15.98:41072) 2024-06-11T19:20:21.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5a7fc640 10 monclient(hunting): get_auth_request con 0x7fcd5c0a4920 auth_method 0 2024-06-11T19:20:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5b7fe640 10 monclient(hunting): get_auth_request con 0x7fcd5c0a5240 auth_method 0 2024-06-11T19:20:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5affd640 10 monclient(hunting): get_auth_request con 0x7fcd5c0a2530 auth_method 0 2024-06-11T19:20:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.280+0000 7fcd5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5a7fc640 10 monclient(hunting): handle_auth_done global_id 84809 payload 824 2024-06-11T19:20:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5a7fc640 1 -- 172.21.15.98:0/3682408882 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd5c0a5240 msgr2=0x7fcd5c123000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5a7fc640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd5c0a5240 0x7fcd5c123000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5a7fc640 1 -- 172.21.15.98:0/3682408882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd5c0a2530 msgr2=0x7fcd5c1cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5a7fc640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd5c0a2530 0x7fcd5c1cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5a7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:20:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5a7fc640 1 monclient: found mon.noname-b 2024-06-11T19:20:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:20:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5a7fc640 1 -- 172.21.15.98:0/3682408882 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5c127870 con 0x7fcd5c0a4920 2024-06-11T19:20:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5affd640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd5c0a2530 0x7fcd5c1cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5b7fe640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd5c0a5240 0x7fcd5c123000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:20:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd5a7fc640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd5c0a4920 0x7fcd5c1ce290 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcd5000b500 tx=0x7fcd5000b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 10 client.?.objecter ms_handle_connect 0x7fcd5c0a4920 2024-06-11T19:20:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd50003bc0 con 0x7fcd5c0a4920 2024-06-11T19:20:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:20:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 10 monclient: dump: 2024-06-11T19:20:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.312 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 10 monclient: _finish_auth 0 2024-06-11T19:20:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 10 monclient: _check_auth_tickets 2024-06-11T19:20:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd50015070 con 0x7fcd5c0a4920 2024-06-11T19:20:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd61574640 5 monclient: authenticate success, global_id 84809 2024-06-11T19:20:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd61574640 10 monclient: _renew_subs 2024-06-11T19:20:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd61574640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:20:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd61574640 1 -- 172.21.15.98:0/3682408882 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd5c126ea0 con 0x7fcd5c0a4920 2024-06-11T19:20:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd50003da0 con 0x7fcd5c0a4920 2024-06-11T19:20:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:20:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd61574640 10 client.84809.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd437fe640 10 monclient: dump: 2024-06-11T19:20:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.315 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd61574640 10 monclient: _renew_subs 2024-06-11T19:20:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd61574640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:20:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd597fa640 4 set_mon_vals no callback set 2024-06-11T19:20:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd61574640 1 -- 172.21.15.98:0/3682408882 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd5c126340 con 0x7fcd5c0a4920 2024-06-11T19:20:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.281+0000 7fcd61574640 1 librados: init done 2024-06-11T19:20:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd61574640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd61574640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c126840 0x7fcd5c063180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd61574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd61574640 1 -- 172.21.15.98:0/3682408882 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fcd5c050830 con 0x7fcd5c126840 2024-06-11T19:20:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fcd50005070 con 0x7fcd5c0a4920 2024-06-11T19:20:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd437fe640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcd4803fdb0 0x7fcd48042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(424..424 src has 1..424) ==== 6230+0+0 (secure 0 0 0) 0x7fcd50051750 con 0x7fcd5c0a4920 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd437fe640 10 client.84809.objecter ms_dispatch 0x7fcd5c000b90 osd_map(424..424 src has 1..424) 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd437fe640 3 client.84809.objecter handle_osd_map got epochs [424,424] > 0 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd437fe640 3 client.84809.objecter handle_osd_map decoding full epoch 424 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd5affd640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c126840 0x7fcd5c063180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd437fe640 20 client.84809.objecter dump_active .. 0 homeless 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd5affd640 10 monclient: get_auth_request con 0x7fcd5c126840 auth_method 0 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd5affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd5affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd5b7fe640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcd4803fdb0 0x7fcd48042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.282+0000 7fcd5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd5b7fe640 10 monclient: get_auth_request con 0x7fcd4803fdb0 auth_method 0 2024-06-11T19:20:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd5affd640 10 monclient(hunting): handle_auth_done global_id 23360 payload 824 2024-06-11T19:20:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd5affd640 1 -- 172.21.15.98:0/3682408882 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd5c0a66f0 con 0x7fcd5c126840 2024-06-11T19:20:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 10 client.84809.objecter ms_handle_connect 0x7fcd5c126840 2024-06-11T19:20:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 10 client.84809.objecter resend_mon_ops 2024-06-11T19:20:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd5affd640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c126840 0x7fcd5c063180 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c009870 tx=0x7fcd4c0046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 10 client.84809.objecter ms_handle_connect 0x7fcd4803fdb0 2024-06-11T19:20:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd5b7fe640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcd4803fdb0 0x7fcd48042270 secure :-1 s=READY pgs=4226 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c124bf0 tx=0x7fcd5400d040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd4c038040 con 0x7fcd5c126840 2024-06-11T19:20:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:20:21.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 10 monclient: dump: 2024-06-11T19:20:21.319 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.319 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.319 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.319 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.319 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.319 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.319 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.319 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.319 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.319 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.320 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.320 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.320 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.320 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.320 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.320 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcd4c003610 con 0x7fcd5c126840 2024-06-11T19:20:21.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:21.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:21.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c126840 msgr2=0x7fcd5c063180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c126840 0x7fcd5c063180 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c009870 tx=0x7fcd4c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c126840 msgr2=0x7fcd5c063180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:21.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.283+0000 7fcd437fe640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c126840 0x7fcd5c063180 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd61574640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:21.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd61574640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c0636c0 0x7fcd5c02cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd61574640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd61574640 1 -- 172.21.15.98:0/3682408882 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fcd5c0a0100 con 0x7fcd5c0636c0 2024-06-11T19:20:21.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd5affd640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c0636c0 0x7fcd5c02cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd5affd640 10 monclient: get_auth_request con 0x7fcd5c0636c0 auth_method 0 2024-06-11T19:20:21.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd5affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd5affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd5affd640 10 monclient(hunting): handle_auth_done global_id 23369 payload 824 2024-06-11T19:20:21.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd5affd640 1 -- 172.21.15.98:0/3682408882 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcd5c125420 con 0x7fcd5c0636c0 2024-06-11T19:20:21.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd5affd640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c0636c0 0x7fcd5c02cb40 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c000c00 tx=0x7fcd4c031000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd437fe640 10 client.84809.objecter ms_handle_connect 0x7fcd5c0636c0 2024-06-11T19:20:21.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.290+0000 7fcd437fe640 10 client.84809.objecter resend_mon_ops 2024-06-11T19:20:21.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd4c04f070 con 0x7fcd5c0636c0 2024-06-11T19:20:21.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd437fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:20:21.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd437fe640 10 monclient: dump: 2024-06-11T19:20:21.323 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.323 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.323 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.323 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.323 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.323 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.323 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.323 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.323 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.323 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.323 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.324 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.324 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.324 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.324 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.324 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9580 (secure 0 0 0) 0x7fcd4c0497f0 con 0x7fcd5c0636c0 2024-06-11T19:20:21.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd437fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:21.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd437fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:21.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c0636c0 msgr2=0x7fcd5c02cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd437fe640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c0636c0 0x7fcd5c02cb40 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c000c00 tx=0x7fcd4c031000 comp rx=0 tx=0).stop 2024-06-11T19:20:21.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd437fe640 1 -- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c0636c0 msgr2=0x7fcd5c02cb40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:21.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd437fe640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcd5c0636c0 0x7fcd5c02cb40 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.325 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:21.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:21.325 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:21.325 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 58, 2024-06-11T19:20:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 671485, 2024-06-11T19:20:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:20:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:20:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:21.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.339 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:21.339 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:21.339 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.342 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:21.342 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:21.342 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:20:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:20:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:21.356 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:21.356 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:21.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.356 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.356 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:21.356 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:21.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 10 librados: watch_flush enter 2024-06-11T19:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 10 librados: watch_flush exit 2024-06-11T19:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 20 client.84809.objecter shutdown clearing up homeless session... 2024-06-11T19:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 10 client.84809.objecter successfully canceled tick 2024-06-11T19:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 1 -- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcd4803fdb0 msgr2=0x7fcd48042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcd4803fdb0 0x7fcd48042270 secure :-1 s=READY pgs=4226 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c124bf0 tx=0x7fcd5400d040 comp rx=0 tx=0).stop 2024-06-11T19:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 10 monclient: shutdown 2024-06-11T19:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 1 -- 172.21.15.98:0/3682408882 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd5c0a4920 msgr2=0x7fcd5c1ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 1 --2- 172.21.15.98:0/3682408882 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd5c0a4920 0x7fcd5c1ce290 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcd5000b500 tx=0x7fcd5000b9c0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd5b7fe640 1 -- 172.21.15.98:0/3682408882 reap_dead start 2024-06-11T19:20:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd5b7fe640 1 -- 172.21.15.98:0/3682408882 reap_dead start 2024-06-11T19:20:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 1 -- 172.21.15.98:0/3682408882 shutdown_connections 2024-06-11T19:20:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 1 -- 172.21.15.98:0/3682408882 >> 172.21.15.98:0/3682408882 conn(0x7fcd5c0ab850 msgr2=0x7fcd5c0a40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 1 -- 172.21.15.98:0/3682408882 shutdown_connections 2024-06-11T19:20:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 1 -- 172.21.15.98:0/3682408882 wait complete. 2024-06-11T19:20:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.291+0000 7fcd61574640 1 librados: shutdown 2024-06-11T19:20:21.360 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-06-11T19:20:21.361 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon i 2024-06-11T19:20:21.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:20:21.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.462+0000 7ff4cb78f640 10 monclient: get_monmap_and_config 2024-06-11T19:20:21.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.462+0000 7ff4cb78f640 10 monclient: build_initial_monmap 2024-06-11T19:20:21.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.462+0000 7ff4cb78f640 10 monclient: monmap: 2024-06-11T19:20:21.462 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:21.463 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.463 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:20:21.463376+0000 2024-06-11T19:20:21.463 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:20:21.463376+0000 2024-06-11T19:20:21.463 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:21.463 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.463 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:20:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:20:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:20:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:20:21.465 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.462+0000 7ff4cb78f640 1 Processor -- start 2024-06-11T19:20:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 1 -- start start 2024-06-11T19:20:21.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 10 monclient: init 2024-06-11T19:20:21.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 10 monclient: _add_conns ranks=[8,6,10,1,5,3,4,2,7,0,9] 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff4c40a4920 0x7ff4c40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 10 monclient(hunting): picked mon.noname-e con 0x7ff4c40a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a5240 0x7ff4c40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 10 monclient(hunting): picked mon.noname-d con 0x7ff4c40a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff4c40a2530 0x7ff4c40a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 10 monclient(hunting): picked mon.noname-f con 0x7ff4c40a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c40a7140 con 0x7ff4c40a2530 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff4c40a2b50 con 0x7ff4c40a5240 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff4c40a2e30 con 0x7ff4c40a4920 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4c952b640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff4c40a4920 msgr2=0x7ff4c40a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4cb78f640 10 monclient(hunting): authenticate will time out at 1431.270715s 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4c952b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff4c40a4920 0x7ff4c40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4c8d2a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a5240 0x7ff4c40a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4c8d2a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a5240 0x7ff4c40a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:41088/0 (socket says 172.21.15.98:41088) 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4c8d2a640 1 -- 172.21.15.98:0/1527920083 learned_addr learned my addr 172.21.15.98:0/1527920083 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4c8d2a640 10 monclient(hunting): get_auth_request con 0x7ff4c40a5240 auth_method 0 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4c8d2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4c8d2a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4c8d2a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4c8d2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4c8d2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.463+0000 7ff4c8d2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4c8d2a640 10 monclient(hunting): handle_auth_done global_id 55482 payload 274 2024-06-11T19:20:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4c8d2a640 1 -- 172.21.15.98:0/1527920083 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff4c40a4920 msgr2=0x7ff4c40a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:20:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4c8d2a640 1 --2- 172.21.15.98:0/1527920083 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff4c40a4920 0x7ff4c40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4c8d2a640 1 -- 172.21.15.98:0/1527920083 >> v2:172.21.15.117:3300/0 conn(0x7ff4c40a2530 msgr2=0x7ff4c40a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:20:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4c9d2c640 1 --2- 172.21.15.98:0/1527920083 >> v2:172.21.15.117:3300/0 conn(0x7ff4c40a2530 0x7ff4c40a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4c8d2a640 1 --2- 172.21.15.98:0/1527920083 >> v2:172.21.15.117:3300/0 conn(0x7ff4c40a2530 0x7ff4c40a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4c8d2a640 10 monclient: _finish_hunting 0 2024-06-11T19:20:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4c8d2a640 1 monclient: found mon.noname-d 2024-06-11T19:20:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4c8d2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4c8d2a640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:20:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4c8d2a640 1 -- 172.21.15.98:0/1527920083 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c40a3110 con 0x7ff4c40a5240 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4c8d2a640 1 --2- 172.21.15.98:0/1527920083 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a5240 0x7ff4c40a59a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009870 tx=0x7ff4b402b7e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=d6d7b9c7fe7078ad server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4bbfff640 1 -- 172.21.15.98:0/1527920083 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4b4038070 con 0x7ff4c40a5240 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4bbfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.464+0000 7ff4bbfff640 10 monclient: dump: 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.483 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4bbfff640 10 monclient: _finish_auth 0 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4bbfff640 10 monclient: _check_auth_tickets 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4bbfff640 1 -- 172.21.15.98:0/1527920083 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b40319b0 con 0x7ff4c40a5240 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 5 monclient: authenticate success, global_id 55482 2024-06-11T19:20:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:20:21.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4bbfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:21.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4bbfff640 1 -- 172.21.15.98:0/1527920083 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4b402f050 con 0x7ff4c40a5240 2024-06-11T19:20:21.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 10 monclient: get_monmap_and_config success 2024-06-11T19:20:21.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 4 set_mon_vals no callback set 2024-06-11T19:20:21.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4bbfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:21.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4bbfff640 10 monclient: dump: 2024-06-11T19:20:21.485 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.485 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.485 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.485 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 10 monclient: shutdown 2024-06-11T19:20:21.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 1 -- 172.21.15.98:0/1527920083 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a5240 msgr2=0x7ff4c40a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 1 --2- 172.21.15.98:0/1527920083 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a5240 0x7ff4c40a59a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009870 tx=0x7ff4b402b7e0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 1 -- 172.21.15.98:0/1527920083 shutdown_connections 2024-06-11T19:20:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 1 --2- 172.21.15.98:0/1527920083 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a5240 0x7ff4c40a59a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 1 --2- 172.21.15.98:0/1527920083 >> v2:172.21.15.117:3300/0 conn(0x7ff4c40a2530 0x7ff4c40a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 1 --2- 172.21.15.98:0/1527920083 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff4c40a4920 0x7ff4c40a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 1 -- 172.21.15.98:0/1527920083 >> 172.21.15.98:0/1527920083 conn(0x7ff4c40ab850 msgr2=0x7ff4c40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 1 -- 172.21.15.98:0/1527920083 shutdown_connections 2024-06-11T19:20:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.465+0000 7ff4cb78f640 1 -- 172.21.15.98:0/1527920083 wait complete. 2024-06-11T19:20:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient: build_initial_monmap 2024-06-11T19:20:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient: monmap: 2024-06-11T19:20:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:20:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:20:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:20:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:20:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:20:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:20:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:20:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:20:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:20:21.489 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:20:21.489 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:20:21.489 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:20:21.489 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:20:21.489 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:20:21.489 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:20:21.489 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 1 librados: starting msgr at 2024-06-11T19:20:21.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 1 librados: starting objecter 2024-06-11T19:20:21.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 1 Processor -- start 2024-06-11T19:20:21.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 1 -- start start 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 1 librados: setting wanted keys 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 1 librados: calling monclient init 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient: init 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient: _reopen_session rank -1 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient: _add_conns ranks=[7,6,2,1,3,4,8,0,5] 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c40a2530 0x7ff4c41cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient(hunting): picked mon.noname-i con 0x7ff4c40a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a4920 0x7ff4c41ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient(hunting): picked mon.noname-d con 0x7ff4c40a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c40a5240 0x7ff4c4123070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient(hunting): picked mon.noname-b con 0x7ff4c40a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c40a6da0 con 0x7ff4c40a5240 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff4c4138e50 con 0x7ff4c40a4920 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff4c40a6420 con 0x7ff4c40a2530 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient(hunting): _renew_subs 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.466+0000 7ff4cb78f640 10 monclient(hunting): authenticate will time out at 1431.273978s 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a4920 0x7ff4c41ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a4920 0x7ff4c41ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:41096/0 (socket says 172.21.15.98:41096) 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 1 -- 172.21.15.98:0/2822308175 learned_addr learned my addr 172.21.15.98:0/2822308175 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c9d2c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c40a5240 0x7ff4c4123070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 10 monclient(hunting): get_auth_request con 0x7ff4c40a4920 auth_method 0 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c9d2c640 10 monclient(hunting): get_auth_request con 0x7ff4c40a5240 auth_method 0 2024-06-11T19:20:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c9d2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c9d2c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c9d2c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c952b640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c40a2530 0x7ff4c41cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c952b640 10 monclient(hunting): get_auth_request con 0x7ff4c40a2530 auth_method 0 2024-06-11T19:20:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c952b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c952b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c952b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c9d2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c9d2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c9d2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c952b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c952b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c952b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 10 monclient(hunting): handle_auth_done global_id 55491 payload 824 2024-06-11T19:20:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 1 -- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c40a2530 msgr2=0x7ff4c41cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c40a2530 0x7ff4c41cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 1 -- 172.21.15.98:0/2822308175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c40a5240 msgr2=0x7ff4c4123070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff4c40a5240 0x7ff4c4123070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 10 monclient: _finish_hunting 0 2024-06-11T19:20:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 1 monclient: found mon.noname-d 2024-06-11T19:20:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:20:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:20:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 1 -- 172.21.15.98:0/2822308175 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c4126bf0 con 0x7ff4c40a4920 2024-06-11T19:20:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4c8d2a640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a4920 0x7ff4c41ce0f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40097c0 tx=0x7ff4b40046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 10 client.?.objecter ms_handle_connect 0x7ff4c40a4920 2024-06-11T19:20:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:20:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4b4038050 con 0x7ff4c40a4920 2024-06-11T19:20:21.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:20:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 10 monclient: dump: 2024-06-11T19:20:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 10 monclient: _finish_auth 0 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b4044070 con 0x7ff4c40a4920 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4cb78f640 5 monclient: authenticate success, global_id 55491 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:20:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4b403f450 con 0x7ff4c40a4920 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4cb78f640 10 monclient: _renew_subs 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4cb78f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4cb78f640 1 -- 172.21.15.98:0/2822308175 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c4126ed0 con 0x7ff4c40a4920 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4b9ffb640 10 monclient: dump: 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4bb7fe640 4 set_mon_vals no callback set 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4cb78f640 10 client.55491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4cb78f640 10 monclient: _renew_subs 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4cb78f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:20:21.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4cb78f640 1 -- 172.21.15.98:0/2822308175 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c40a8da0 con 0x7ff4c40a4920 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.467+0000 7ff4cb78f640 1 librados: init done 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.468+0000 7ff4cb78f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.468+0000 7ff4cb78f640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c41267b0 0x7ff4c4123d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.468+0000 7ff4cb78f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.468+0000 7ff4cb78f640 1 -- 172.21.15.98:0/2822308175 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff4c4127900 con 0x7ff4c41267b0 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.468+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff4b4003610 con 0x7ff4c40a4920 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.468+0000 7ff4c952b640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c41267b0 0x7ff4c4123d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.468+0000 7ff4c952b640 10 monclient: get_auth_request con 0x7ff4c41267b0 auth_method 0 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.468+0000 7ff4c952b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.468+0000 7ff4c952b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.468+0000 7ff4c952b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.468+0000 7ff4b9ffb640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff4a803fdb0 0x7ff4a8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4c952b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4c952b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4c952b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(424..424 src has 1..424) ==== 6230+0+0 (secure 0 0 0) 0x7ff4b40735e0 con 0x7ff4c40a4920 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 10 client.55491.objecter ms_dispatch 0x7ff4c4000b90 osd_map(424..424 src has 1..424) 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 3 client.55491.objecter handle_osd_map got epochs [424,424] > 0 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 3 client.55491.objecter handle_osd_map decoding full epoch 424 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 20 client.55491.objecter dump_active .. 0 homeless 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4c952b640 10 monclient(hunting): handle_auth_done global_id 84616 payload 824 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4c952b640 1 -- 172.21.15.98:0/2822308175 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4c40a6750 con 0x7ff4c41267b0 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4c9d2c640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff4a803fdb0 0x7ff4a8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4c9d2c640 10 monclient: get_auth_request con 0x7ff4a803fdb0 auth_method 0 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 10 client.55491.objecter ms_handle_connect 0x7ff4c41267b0 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 10 client.55491.objecter resend_mon_ops 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4c952b640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c41267b0 0x7ff4c4123d70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0009610 tx=0x7ff4c0009ad0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4c0016070 con 0x7ff4c41267b0 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 10 monclient: dump: 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.501 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 10 client.55491.objecter ms_handle_connect 0x7ff4a803fdb0 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4c9d2c640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff4a803fdb0 0x7ff4a8042270 secure :-1 s=READY pgs=4227 cs=0 l=1 rev1=1 crypto rx=0x7ff4c41273c0 tx=0x7ff4bc0076f0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff4c0012600 con 0x7ff4c41267b0 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c41267b0 msgr2=0x7ff4c4123d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c41267b0 0x7ff4c4123d70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0009610 tx=0x7ff4c0009ad0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c41267b0 msgr2=0x7ff4c4123d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:21.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.469+0000 7ff4b9ffb640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c41267b0 0x7ff4c4123d70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4cb78f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:20:21.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4cb78f640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c402cb40 0x7ff4c402cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:20:21.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4cb78f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:20:21.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4cb78f640 1 -- 172.21.15.98:0/2822308175 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff4c4125500 con 0x7ff4c402cb40 2024-06-11T19:20:21.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4c952b640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c402cb40 0x7ff4c402cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:20:21.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4c952b640 10 monclient: get_auth_request con 0x7ff4c402cb40 auth_method 0 2024-06-11T19:20:21.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4c952b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:20:21.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4c952b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:20:21.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4c952b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:20:21.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4c952b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:20:21.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4c952b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:20:21.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4c952b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:20:21.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4c952b640 10 monclient(hunting): handle_auth_done global_id 84625 payload 824 2024-06-11T19:20:21.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4c952b640 1 -- 172.21.15.98:0/2822308175 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff4c40a0100 con 0x7ff4c402cb40 2024-06-11T19:20:21.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4c952b640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c402cb40 0x7ff4c402cf50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0000c00 tx=0x7ff4c00059d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:20:21.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4b9ffb640 10 client.55491.objecter ms_handle_connect 0x7ff4c402cb40 2024-06-11T19:20:21.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.476+0000 7ff4b9ffb640 10 client.55491.objecter resend_mon_ops 2024-06-11T19:20:21.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4c0016040 con 0x7ff4c402cb40 2024-06-11T19:20:21.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:20:21.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4b9ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:20:21.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4b9ffb640 10 monclient: dump: 2024-06-11T19:20:21.504 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:20:21.505 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:20:21.506 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:20:21.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9240 (secure 0 0 0) 0x7ff4c001c0d0 con 0x7ff4c402cb40 2024-06-11T19:20:21.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4b9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:20:21.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4b9ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:20:21.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c402cb40 msgr2=0x7ff4c402cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4b9ffb640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c402cb40 0x7ff4c402cf50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0000c00 tx=0x7ff4c00059d0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4b9ffb640 1 -- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c402cb40 msgr2=0x7ff4c402cf50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:20:21.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4b9ffb640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff4c402cb40 0x7ff4c402cf50 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0000c00 tx=0x7ff4c00059d0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.506 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:20:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:20:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 58, 2024-06-11T19:20:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 293247, 2024-06-11T19:20:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:20:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:20:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:20:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:20:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:20:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:20:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:20:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:20:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:20:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:20:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:20:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:20:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:20:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:20:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:20:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:20:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:20:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:20:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:20:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:20:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:20:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:20:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:20:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:20:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:20:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:20:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:20:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:20:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:20:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:20:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:20:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:20:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:20:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:20:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:20:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:20:21.530 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:20:21.530 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:20:21.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 10 librados: watch_flush enter 2024-06-11T19:20:21.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 10 librados: watch_flush exit 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 20 client.55491.objecter shutdown clearing up homeless session... 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 10 client.55491.objecter successfully canceled tick 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 1 -- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff4a803fdb0 msgr2=0x7ff4a8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff4a803fdb0 0x7ff4a8042270 secure :-1 s=READY pgs=4227 cs=0 l=1 rev1=1 crypto rx=0x7ff4c41273c0 tx=0x7ff4bc0076f0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 10 monclient: shutdown 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 1 -- 172.21.15.98:0/2822308175 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a4920 msgr2=0x7ff4c41ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 1 --2- 172.21.15.98:0/2822308175 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff4c40a4920 0x7ff4c41ce0f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40097c0 tx=0x7ff4b40046c0 comp rx=0 tx=0).stop 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4c9d2c640 1 -- 172.21.15.98:0/2822308175 reap_dead start 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4c9d2c640 1 -- 172.21.15.98:0/2822308175 reap_dead start 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 1 -- 172.21.15.98:0/2822308175 shutdown_connections 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 1 -- 172.21.15.98:0/2822308175 >> 172.21.15.98:0/2822308175 conn(0x7ff4c40ab850 msgr2=0x7ff4c41bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:20:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 1 -- 172.21.15.98:0/2822308175 shutdown_connections 2024-06-11T19:20:21.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 1 -- 172.21.15.98:0/2822308175 wait complete. 2024-06-11T19:20:21.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:20:21.477+0000 7ff4cb78f640 1 librados: shutdown 2024-06-11T19:20:21.533 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-06-11T19:20:21.533 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-06-11T19:20:31.090 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-06-11T19:20:31.089+0000 7fae012ec640 -1 snap_mapper.add_oid found existing snaps mapped on 104:cbfec0d4:test-rados-api-smithi117-36854-97::foo:21, removing 2024-06-11T19:20:31.091 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-06-11T19:20:31.090+0000 7f87b4388640 -1 snap_mapper.add_oid found existing snaps mapped on 104:cbfec0d4:test-rados-api-smithi117-36854-97::foo:21, removing 2024-06-11T19:21:51.534 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-06-11T19:21:51.535 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-06-11T19:21:51.535 INFO:tasks.ceph.mon.e:Restarting daemon 2024-06-11T19:21:51.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-06-11T19:21:51.539 INFO:tasks.ceph.mon.e:Started 2024-06-11T19:21:51.539 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-06-11T19:21:51.539 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-06-11T19:21:51.539 INFO:tasks.ceph.mon.g:Restarting daemon 2024-06-11T19:21:51.540 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-06-11T19:21:51.542 INFO:tasks.ceph.mon.g:Started 2024-06-11T19:21:51.542 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-06-11T19:21:51.543 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-06-11T19:21:51.543 INFO:tasks.ceph.mon.c:Restarting daemon 2024-06-11T19:21:51.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-06-11T19:21:51.545 INFO:tasks.ceph.mon.c:Started 2024-06-11T19:21:51.545 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:21:51.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:21:51.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.657+0000 7fd40f4b9640 10 monclient: get_monmap_and_config 2024-06-11T19:21:51.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.657+0000 7fd40f4b9640 10 monclient: build_initial_monmap 2024-06-11T19:21:51.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.657+0000 7fd40f4b9640 10 monclient: monmap: 2024-06-11T19:21:51.656 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:21:51.656 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:21:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:21:51.658229+0000 2024-06-11T19:21:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:21:51.658229+0000 2024-06-11T19:21:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:21:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:21:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:21:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:21:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:21:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:21:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:21:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:21:51.657 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:21:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:21:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:21:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:21:51.658 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:21:51.658 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.657+0000 7fd40f4b9640 1 Processor -- start 2024-06-11T19:21:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40f4b9640 1 -- start start 2024-06-11T19:21:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40f4b9640 10 monclient: init 2024-06-11T19:21:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40f4b9640 10 monclient: _reopen_session rank -1 2024-06-11T19:21:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40f4b9640 10 monclient: _add_conns ranks=[8,1,2,9,10,6,0,5,3,7,4] 2024-06-11T19:21:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40f4b9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd4080a4920 0x7fd4080a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40f4b9640 10 monclient(hunting): picked mon.noname-e con 0x7fd4080a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:21:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40f4b9640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd4080a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:21:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40f4b9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a2530 0x7fd4080a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40d255640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd4080a4920 msgr2=0x7fd4080a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:21:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40f4b9640 10 monclient(hunting): picked mon.noname-b con 0x7fd4080a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:21:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40f4b9640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4080a7140 con 0x7fd4080a5240 2024-06-11T19:21:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40f4b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.658+0000 7fd40d255640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd4080a4920 0x7fd4080a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:21:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a2530 0x7fd4080a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a2530 0x7fd4080a60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:37700/0 (socket says 172.21.15.98:37700) 2024-06-11T19:21:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 1 -- 172.21.15.98:0/2328563869 learned_addr learned my addr 172.21.15.98:0/2328563869 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:21:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40ca54640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd4080a5240 0x7fd4080a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:33262/0 (socket says 172.21.15.98:33262) 2024-06-11T19:21:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40f4b9640 1 -- 172.21.15.98:0/2328563869 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd4080a2b50 con 0x7fd4080a2530 2024-06-11T19:21:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40f4b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40f4b9640 1 -- 172.21.15.98:0/2328563869 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd4080a2e30 con 0x7fd4080a4920 2024-06-11T19:21:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40f4b9640 10 monclient(hunting): _renew_subs 2024-06-11T19:21:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40f4b9640 10 monclient(hunting): authenticate will time out at 1521.466113s 2024-06-11T19:21:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 10 monclient(hunting): get_auth_request con 0x7fd4080a2530 auth_method 0 2024-06-11T19:21:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 10 monclient(hunting): handle_auth_done global_id 84854 payload 274 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 1 -- 172.21.15.98:0/2328563869 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd4080a4920 msgr2=0x7fd4080a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 1 --2- 172.21.15.98:0/2328563869 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd4080a4920 0x7fd4080a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 1 -- 172.21.15.98:0/2328563869 >> v1:172.21.15.117:6789/0 conn(0x7fd4080a5240 legacy=0x7fd4080a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 10 monclient: _finish_hunting 0 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 1 monclient: found mon.noname-b 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:21:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd3fffff640 1 -- 172.21.15.98:0/2328563869 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2822677460 0 0) 0x7fd4080a7140 con 0x7fd4080a5240 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd40da56640 1 -- 172.21.15.98:0/2328563869 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4080a3110 con 0x7fd4080a2530 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.659+0000 7fd3fffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd40da56640 1 --2- 172.21.15.98:0/2328563869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a2530 0x7fd4080a60b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd400009e80 tx=0x7fd40002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31906a82ba408902 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 1 -- 172.21.15.98:0/2328563869 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd40002cc50 con 0x7fd4080a2530 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 10 monclient: dump: 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:21:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 10 monclient: _finish_auth 0 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 10 monclient: _check_auth_tickets 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:21:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 1 -- 172.21.15.98:0/2328563869 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd40002ce10 con 0x7fd4080a2530 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd40f4b9640 5 monclient: authenticate success, global_id 84854 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd40f4b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd40f4b9640 10 monclient: get_monmap_and_config success 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd40f4b9640 4 set_mon_vals no callback set 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 1 -- 172.21.15.98:0/2328563869 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd400034860 con 0x7fd4080a2530 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd3fffff640 10 monclient: dump: 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd40f4b9640 10 monclient: shutdown 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd40f4b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:21:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd40f4b9640 1 -- 172.21.15.98:0/2328563869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a2530 msgr2=0x7fd4080a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:21:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.660+0000 7fd40f4b9640 1 --2- 172.21.15.98:0/2328563869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a2530 0x7fd4080a60b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd400009e80 tx=0x7fd40002be40 comp rx=0 tx=0).stop 2024-06-11T19:21:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.661+0000 7fd40f4b9640 1 -- 172.21.15.98:0/2328563869 shutdown_connections 2024-06-11T19:21:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.661+0000 7fd40f4b9640 1 --2- 172.21.15.98:0/2328563869 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a2530 0x7fd4080a60b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.661+0000 7fd40f4b9640 1 --2- 172.21.15.98:0/2328563869 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd4080a4920 0x7fd4080a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.661+0000 7fd40f4b9640 1 -- 172.21.15.98:0/2328563869 >> 172.21.15.98:0/2328563869 conn(0x7fd4080ab850 msgr2=0x7fd4080a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:21:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.661+0000 7fd40f4b9640 1 -- 172.21.15.98:0/2328563869 shutdown_connections 2024-06-11T19:21:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.661+0000 7fd40f4b9640 1 -- 172.21.15.98:0/2328563869 wait complete. 2024-06-11T19:21:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.661+0000 7fd40f4b9640 10 monclient: build_initial_monmap 2024-06-11T19:21:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.661+0000 7fd40f4b9640 10 monclient: monmap: 2024-06-11T19:21:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:21:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:21:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:21:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:21:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:21:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:21:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:21:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:21:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:21:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:21:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:21:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:21:51.674 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:21:51.674 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:21:51.674 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:21:51.674 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:51.674 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-06-11T19:21:51.661+0000 7f9f12411b00 -1 mon.e@-1(???).log v0 log_external_backlog rewinding external_log_to from 453 -> 0 (sync_force? mon rebuild?) 2024-06-11T19:21:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 1 librados: starting msgr at 2024-06-11T19:21:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 1 librados: starting objecter 2024-06-11T19:21:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 1 Processor -- start 2024-06-11T19:21:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 1 -- start start 2024-06-11T19:21:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 1 librados: setting wanted keys 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 1 librados: calling monclient init 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 10 monclient: init 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 10 monclient: _reopen_session rank -1 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 10 monclient: _add_conns ranks=[0,2,8,7,4,3,1,5,6] 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4080a2530 0x7fd4081cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 10 monclient(hunting): picked mon.noname-a con 0x7fd4080a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a4920 0x7fd4081cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 10 monclient(hunting): picked mon.noname-b con 0x7fd4080a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd4080a5240 0x7fd408124f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 10 monclient(hunting): picked mon.noname-e con 0x7fd4080a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd4080a6da0 con 0x7fd4080a2530 2024-06-11T19:21:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd408138e50 con 0x7fd4080a4920 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd4080a65f0 con 0x7fd4080a5240 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 10 monclient(hunting): _renew_subs 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40da56640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd4080a5240 msgr2=0x7fd408124f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40f4b9640 10 monclient(hunting): authenticate will time out at 1521.469761s 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40da56640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd4080a5240 0x7fd408124f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40ca54640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a4920 0x7fd4081cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40ca54640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a4920 0x7fd4081cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:37704/0 (socket says 172.21.15.98:37704) 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40ca54640 1 -- 172.21.15.98:0/425676127 learned_addr learned my addr 172.21.15.98:0/425676127 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40ca54640 10 monclient(hunting): get_auth_request con 0x7fd4080a4920 auth_method 0 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40ca54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:51.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40ca54640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40ca54640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40d255640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4080a2530 0x7fd4081cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40d255640 10 monclient(hunting): get_auth_request con 0x7fd4080a2530 auth_method 0 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40d255640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40d255640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40d255640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40ca54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40ca54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.662+0000 7fd40ca54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40d255640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40d255640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40d255640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:21:51.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40ca54640 10 monclient(hunting): handle_auth_done global_id 84863 payload 824 2024-06-11T19:21:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40ca54640 1 -- 172.21.15.98:0/425676127 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd4080a5240 msgr2=0x7fd408124f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:21:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40ca54640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd4080a5240 0x7fd408124f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40ca54640 1 -- 172.21.15.98:0/425676127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4080a2530 msgr2=0x7fd4081cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:21:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40ca54640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4080a2530 0x7fd4081cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40ca54640 10 monclient: _finish_hunting 0 2024-06-11T19:21:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40ca54640 1 monclient: found mon.noname-b 2024-06-11T19:21:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40ca54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:21:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40ca54640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:21:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40ca54640 1 -- 172.21.15.98:0/425676127 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd408126990 con 0x7fd4080a4920 2024-06-11T19:21:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40ca54640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a4920 0x7fd4081cdfe0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd40400c340 tx=0x7fd40400c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:21:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 10 client.?.objecter ms_handle_connect 0x7fd4080a4920 2024-06-11T19:21:51.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 1 -- 172.21.15.98:0/425676127 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd4040047e0 con 0x7fd4080a4920 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 10 monclient: dump: 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:21:51.684 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 10 monclient: _finish_auth 0 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 10 monclient: _check_auth_tickets 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40f4b9640 5 monclient: authenticate success, global_id 84863 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 1 -- 172.21.15.98:0/425676127 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4040049a0 con 0x7fd4080a4920 2024-06-11T19:21:51.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40f4b9640 10 monclient: _renew_subs 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 1 -- 172.21.15.98:0/425676127 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd4040133c0 con 0x7fd4080a4920 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40f4b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40f4b9640 1 -- 172.21.15.98:0/425676127 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd408128cf0 con 0x7fd4080a4920 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3ff7fe640 4 set_mon_vals no callback set 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd3fdffb640 10 monclient: dump: 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:51.686 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40f4b9640 10 client.84863.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:21:51.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40f4b9640 10 monclient: _renew_subs 2024-06-11T19:21:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40f4b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:21:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40f4b9640 1 -- 172.21.15.98:0/425676127 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4080aac50 con 0x7fd4080a4920 2024-06-11T19:21:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.663+0000 7fd40f4b9640 1 librados: init done 2024-06-11T19:21:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.664+0000 7fd3fdffb640 1 -- 172.21.15.98:0/425676127 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd404011070 con 0x7fd4080a4920 2024-06-11T19:21:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd40f4b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:21:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd40f4b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:21:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd3fdffb640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3ec03fdb0 0x7fd3ec042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd40f4b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:21:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd40f4b9640 1 -- 172.21.15.98:0/425676127 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4080a8e80 con 0x7fd4080a4920 2024-06-11T19:21:51.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd3fdffb640 1 -- 172.21.15.98:0/425676127 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(505..505 src has 1..505) ==== 6690+0+0 (secure 0 0 0) 0x7fd404051620 con 0x7fd4080a4920 2024-06-11T19:21:51.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd3fdffb640 10 client.84863.objecter ms_dispatch 0x7fd408000b90 osd_map(505..505 src has 1..505) 2024-06-11T19:21:51.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd3fdffb640 3 client.84863.objecter handle_osd_map got epochs [505,505] > 0 2024-06-11T19:21:51.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd3fdffb640 3 client.84863.objecter handle_osd_map decoding full epoch 505 2024-06-11T19:21:51.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd40d255640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3ec03fdb0 0x7fd3ec042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:51.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd40d255640 10 monclient: get_auth_request con 0x7fd3ec03fdb0 auth_method 0 2024-06-11T19:21:51.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd3fdffb640 20 client.84863.objecter dump_active .. 0 homeless 2024-06-11T19:21:51.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd3fdffb640 10 client.84863.objecter ms_handle_connect 0x7fd3ec03fdb0 2024-06-11T19:21:51.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.665+0000 7fd40d255640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3ec03fdb0 0x7fd3ec042270 secure :-1 s=READY pgs=4234 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8003980 tx=0x7fd3f8009110 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:21:51.689 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:21:51.666+0000 7f0e7fc45b00 -1 mon.c@-1(???).log v0 log_external_backlog rewinding external_log_to from 453 -> 0 (sync_force? mon rebuild?) 2024-06-11T19:21:51.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.670+0000 7fd3fdffb640 1 -- 172.21.15.98:0/425676127 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd404011c60 con 0x7fd4080a4920 2024-06-11T19:21:51.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.670+0000 7fd3fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:21:51.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.670+0000 7fd3fdffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:21:51.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.940+0000 7fd40f4b9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:21:51.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.940+0000 7fd40f4b9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:21:51.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.940+0000 7fd40f4b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:21:51.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.940+0000 7fd40f4b9640 1 -- 172.21.15.98:0/425676127 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd4080a4d00 con 0x7fd4080a4920 2024-06-11T19:21:51.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.940+0000 7fd3fdffb640 1 -- 172.21.15.98:0/425676127 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3175 (secure 0 0 0) 0x7fd404004d30 con 0x7fd4080a4920 2024-06-11T19:21:51.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.940+0000 7fd3fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:21:51.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.940+0000 7fd3fdffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:21:51.941 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":58,"quorum":[0,1,2,5,6,7],"quorum_names":["a","f","b","h","d","i"],"quorum_leader_name":"f","quorum_age":91,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:21:51.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 10 librados: watch_flush enter 2024-06-11T19:21:51.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 10 librados: watch_flush exit 2024-06-11T19:21:51.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 20 client.84863.objecter shutdown clearing up homeless session... 2024-06-11T19:21:51.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 10 client.84863.objecter successfully canceled tick 2024-06-11T19:21:51.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 -- 172.21.15.98:0/425676127 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3ec03fdb0 msgr2=0x7fd3ec042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:21:51.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3ec03fdb0 0x7fd3ec042270 secure :-1 s=READY pgs=4234 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8003980 tx=0x7fd3f8009110 comp rx=0 tx=0).stop 2024-06-11T19:21:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 10 monclient: shutdown 2024-06-11T19:21:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:21:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 -- 172.21.15.98:0/425676127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a4920 msgr2=0x7fd4081cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:21:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a4920 0x7fd4081cdfe0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd40400c340 tx=0x7fd40400c740 comp rx=0 tx=0).stop 2024-06-11T19:21:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 -- 172.21.15.98:0/425676127 shutdown_connections 2024-06-11T19:21:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd3ec03fdb0 0x7fd3ec042270 unknown :-1 s=CLOSED pgs=4234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd4080a5240 0x7fd408124f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd4080a4920 0x7fd4081cdfe0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 --2- 172.21.15.98:0/425676127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4080a2530 0x7fd4081cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 -- 172.21.15.98:0/425676127 >> 172.21.15.98:0/425676127 conn(0x7fd4080ab850 msgr2=0x7fd40809c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:21:51.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 -- 172.21.15.98:0/425676127 shutdown_connections 2024-06-11T19:21:51.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 -- 172.21.15.98:0/425676127 wait complete. 2024-06-11T19:21:51.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:51.943+0000 7fd40f4b9640 1 librados: shutdown 2024-06-11T19:21:54.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:21:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.067+0000 7f8f76ad8640 10 monclient: get_monmap_and_config 2024-06-11T19:21:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.067+0000 7f8f76ad8640 10 monclient: build_initial_monmap 2024-06-11T19:21:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.067+0000 7f8f76ad8640 10 monclient: monmap: 2024-06-11T19:21:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:21:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:21:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:21:55.068681+0000 2024-06-11T19:21:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:21:55.068681+0000 2024-06-11T19:21:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:21:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:21:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:21:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:21:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:21:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:21:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:21:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:21:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:21:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:21:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:21:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:21:55.068 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:21:55.068 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:55.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.068+0000 7f8f76ad8640 1 Processor -- start 2024-06-11T19:21:55.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.068+0000 7f8f76ad8640 1 -- start start 2024-06-11T19:21:55.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.068+0000 7f8f76ad8640 10 monclient: init 2024-06-11T19:21:55.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.068+0000 7f8f76ad8640 10 monclient: _reopen_session rank -1 2024-06-11T19:21:55.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.068+0000 7f8f76ad8640 10 monclient: _add_conns ranks=[6,1,9,10,0,8,5,2,4,7,3] 2024-06-11T19:21:55.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.068+0000 7f8f76ad8640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:55.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.068+0000 7f8f76ad8640 10 monclient(hunting): picked mon.noname-d con 0x7f8f700a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f76ad8640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f8f700a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f76ad8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f76ad8640 10 monclient(hunting): picked mon.noname-a con 0x7f8f700a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f76ad8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f700a7140 con 0x7f8f700a5240 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f76ad8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f76ad8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8f700a2b50 con 0x7f8f700a2530 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f76ad8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f76ad8640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8f700a2e30 con 0x7f8f700a4920 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f76ad8640 10 monclient(hunting): _renew_subs 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f76ad8640 10 monclient(hunting): authenticate will time out at 1524.876076s 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f74874640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f75075640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f74874640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:59756/0 (socket says 172.21.15.98:59756) 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f74874640 1 -- 172.21.15.98:0/433230800 learned_addr learned my addr 172.21.15.98:0/433230800 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:21:55.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f74874640 10 monclient(hunting): get_auth_request con 0x7f8f700a4920 auth_method 0 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f74874640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f74874640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f74874640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f75075640 10 monclient(hunting): get_auth_request con 0x7f8f700a2530 auth_method 0 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f75075640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f75075640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f75075640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f74874640 1 -- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 msgr2=0x7f8f700a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f75075640 1 -- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 msgr2=0x7f8f700a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f74874640 1 -- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 msgr2=0x7f8f700a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f75075640 1 -- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 msgr2=0x7f8f700a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f75075640 1 --2- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:21:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f75075640 1 --2- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:21:55.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.069+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:21:55.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:55.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f75075640 1 --2- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:55.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f75075640 10 monclient(hunting): get_auth_request con 0x7f8f700a2530 auth_method 0 2024-06-11T19:21:55.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f75075640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:55.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f75075640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:55.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f75075640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:21:55.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f74874640 10 monclient(hunting): get_auth_request con 0x7f8f700a4920 auth_method 0 2024-06-11T19:21:55.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f74874640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:55.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f74874640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:55.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f74874640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:21:55.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f75075640 1 -- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 msgr2=0x7f8f700a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:21:55.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f75075640 1 -- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 msgr2=0x7f8f700a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:21:55.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f75075640 1 --2- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:21:55.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f75075640 1 --2- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:21:55.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f74874640 1 -- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 msgr2=0x7f8f700a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:21:55.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f74874640 1 -- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 msgr2=0x7f8f700a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:21:55.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:21:55.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.270+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:21:55.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f75075640 1 --2- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f75075640 10 monclient(hunting): get_auth_request con 0x7f8f700a2530 auth_method 0 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f75075640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f75075640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f75075640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f74874640 10 monclient(hunting): get_auth_request con 0x7f8f700a4920 auth_method 0 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f74874640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f74874640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f74874640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f75075640 1 -- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 msgr2=0x7f8f700a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f75075640 1 -- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 msgr2=0x7f8f700a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f75075640 1 --2- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:21:55.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f75075640 1 --2- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:21:55.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f74874640 1 -- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 msgr2=0x7f8f700a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:21:55.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f74874640 1 -- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 msgr2=0x7f8f700a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:21:55.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:21:55.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:55.671+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:21:56.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f75075640 1 --2- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:56.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f75075640 10 monclient(hunting): get_auth_request con 0x7f8f700a2530 auth_method 0 2024-06-11T19:21:56.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f75075640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:56.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f75075640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:56.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f75075640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:21:56.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f75075640 1 -- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 msgr2=0x7f8f700a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:21:56.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f75075640 1 -- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 msgr2=0x7f8f700a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:21:56.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f75075640 1 --2- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:21:56.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:56.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f75075640 1 --2- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:21:56.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f74874640 10 monclient(hunting): get_auth_request con 0x7f8f700a4920 auth_method 0 2024-06-11T19:21:56.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f74874640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:56.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f74874640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:56.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f74874640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:21:56.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f74874640 1 -- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 msgr2=0x7f8f700a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:21:56.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f74874640 1 -- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 msgr2=0x7f8f700a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:21:56.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:21:56.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:56.472+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:21:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.285+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4059131901 0 0) 0x7f8f700a7140 con 0x7f8f700a5240 2024-06-11T19:21:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.285+0000 7f8f6f7fe640 10 monclient(hunting): _negotiate 2024-06-11T19:21:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.285+0000 7f8f6f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.285+0000 7f8f6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:21:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.285+0000 7f8f6f7fe640 10 monclient(hunting): my global_id is 32608 2024-06-11T19:21:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.285+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f60003520 con 0x7f8f700a5240 2024-06-11T19:21:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3126071663 0 0) 0x7f8f60003520 con 0x7f8f700a5240 2024-06-11T19:21:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 msgr2=0x7f8f700a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:21:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f700a4920 0x7f8f700a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 msgr2=0x7f8f700a60b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:21:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 1 --2- 172.21.15.98:0/433230800 >> v2:172.21.15.98:3300/0 conn(0x7f8f700a2530 0x7f8f700a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:21:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 1 monclient: found mon.noname-f-legacy 2024-06-11T19:21:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:21:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.117:6789/0 2024-06-11T19:21:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f700a3110 con 0x7f8f700a5240 2024-06-11T19:21:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:21:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:21:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7f8f5c001960 con 0x7f8f700a5240 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 10 monclient: dump: 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:21:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 1 monclient: mon.f has (v2) addrs [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] but i'm connected to v1:172.21.15.117:6789/0, reconnecting 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 10 monclient: _reopen_session rank -1 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.286+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 >> v1:172.21.15.117:6789/0 conn(0x7f8f700a5240 legacy=0x7f8f700a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 10 monclient: _add_conns ranks=[0,2,5,3,1,4,7,8,6] 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f60004250 0x7f8f60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 10 monclient(hunting): picked mon.a con 0x7f8f60004250 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f60004e60 0x7f8f600099b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 10 monclient(hunting): picked mon.b con 0x7f8f60004e60 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f60009ef0 0x7f8f6000c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 10 monclient(hunting): picked mon.h con 0x7f8f60009ef0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f700a2b50 con 0x7f8f60004250 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8f700a2e30 con 0x7f8f60004e60 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f8f6000c9e0 con 0x7f8f60009ef0 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 10 monclient(hunting): _renew_subs 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6f7fe640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f60004250 0x7f8f60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f60004e60 0x7f8f600099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f74874640 10 monclient(hunting): get_auth_request con 0x7f8f60004250 auth_method 0 2024-06-11T19:21:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f74874640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f74874640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f74874640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 10 monclient(hunting): get_auth_request con 0x7f8f60004e60 auth_method 0 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f75075640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f60009ef0 0x7f8f6000c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f75075640 10 monclient(hunting): get_auth_request con 0x7f8f60009ef0 auth_method 0 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f75075640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f75075640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f75075640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f74874640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f74874640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f74874640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:21:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f75075640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f75075640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f75075640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 10 monclient(hunting): handle_auth_done global_id 32608 payload 293 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 1 -- 172.21.15.98:0/433230800 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f60009ef0 msgr2=0x7f8f6000c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f60009ef0 0x7f8f6000c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 1 -- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f60004250 msgr2=0x7f8f60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f60004250 0x7f8f60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 1 monclient: found mon.b 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:21:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f75075640 1 -- 172.21.15.98:0/433230800 reap_dead start 2024-06-11T19:21:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:21:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 1 -- 172.21.15.98:0/433230800 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f60003520 con 0x7f8f60004e60 2024-06-11T19:21:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 10 monclient: _finish_auth 0 2024-06-11T19:21:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:21:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:21:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f76ad8640 5 monclient: authenticate success, global_id 32608 2024-06-11T19:21:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f76ad8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:21:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.287+0000 7f8f74874640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f60004250 0x7f8f60004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:21:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.288+0000 7f8f6ffff640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f60004e60 0x7f8f600099b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8f70073fc0 tx=0x7f8f60005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce34843b7adc0fb4 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:21:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f5c0079e0 con 0x7f8f60004e60 2024-06-11T19:21:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:21:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:21:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f6f7fe640 10 monclient: dump: 2024-06-11T19:21:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:21:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:21:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:21:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:21:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:21:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:21:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:21:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:21:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:21:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:21:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:21:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:21:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:21:57.303 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f5c02d8e0 con 0x7f8f60004e60 2024-06-11T19:21:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f76ad8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:21:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f6f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:21:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f76ad8640 10 monclient: get_monmap_and_config success 2024-06-11T19:21:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f6f7fe640 1 -- 172.21.15.98:0/433230800 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f5c02e450 con 0x7f8f60004e60 2024-06-11T19:21:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f76ad8640 4 set_mon_vals no callback set 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.290+0000 7f8f6f7fe640 10 monclient: dump: 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:21:57.304 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.291+0000 7f8f76ad8640 10 monclient: shutdown 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.291+0000 7f8f76ad8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.291+0000 7f8f76ad8640 1 -- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f60004e60 msgr2=0x7f8f600099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.291+0000 7f8f76ad8640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f60004e60 0x7f8f600099b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8f70073fc0 tx=0x7f8f60005830 comp rx=0 tx=0).stop 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.291+0000 7f8f76ad8640 1 -- 172.21.15.98:0/433230800 shutdown_connections 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.291+0000 7f8f76ad8640 1 --2- 172.21.15.98:0/433230800 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f60004e60 0x7f8f600099b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.291+0000 7f8f76ad8640 1 -- 172.21.15.98:0/433230800 >> 172.21.15.98:0/433230800 conn(0x7f8f700ab850 msgr2=0x7f8f700a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:21:57.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.291+0000 7f8f76ad8640 1 -- 172.21.15.98:0/433230800 shutdown_connections 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.291+0000 7f8f76ad8640 1 -- 172.21.15.98:0/433230800 wait complete. 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.292+0000 7f8f76ad8640 10 monclient: build_initial_monmap 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.292+0000 7f8f76ad8640 10 monclient: monmap: 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:21:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.292+0000 7f8f76ad8640 1 librados: starting msgr at 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.292+0000 7f8f76ad8640 1 librados: starting objecter 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.292+0000 7f8f76ad8640 1 Processor -- start 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.292+0000 7f8f76ad8640 1 -- start start 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.292+0000 7f8f76ad8640 1 librados: setting wanted keys 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.292+0000 7f8f76ad8640 1 librados: calling monclient init 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.292+0000 7f8f76ad8640 10 monclient: init 2024-06-11T19:21:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 10 monclient: _reopen_session rank -1 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 10 monclient: _add_conns ranks=[1,0,6,4,2,3,7,5,8] 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f60004e60 0x7f8f701cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 10 monclient(hunting): picked mon.noname-f con 0x7f8f60004e60 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f70137c10 0x7f8f70137ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 10 monclient(hunting): picked mon.noname-a con 0x7f8f70137c10 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f70121670 0x7f8f70121ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 10 monclient(hunting): picked mon.noname-d con 0x7f8f70121670 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f700a6da0 con 0x7f8f70137c10 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8f70138e50 con 0x7f8f60004e60 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8f700a4d00 con 0x7f8f70121670 2024-06-11T19:21:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 10 monclient(hunting): _renew_subs 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f76ad8640 10 monclient(hunting): authenticate will time out at 1527.100722s 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f75075640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f70121670 0x7f8f70121ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f70137c10 0x7f8f70137ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f75075640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f70121670 0x7f8f70121ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:59782/0 (socket says 172.21.15.98:59782) 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f75075640 1 -- 172.21.15.98:0/3094063080 learned_addr learned my addr 172.21.15.98:0/3094063080 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f75075640 10 monclient(hunting): get_auth_request con 0x7f8f70121670 auth_method 0 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f75075640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f75075640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f75075640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f6ffff640 10 monclient(hunting): get_auth_request con 0x7f8f70137c10 auth_method 0 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f6ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f75075640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.293+0000 7f8f75075640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f75075640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f74874640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f60004e60 0x7f8f701cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f74874640 10 monclient(hunting): get_auth_request con 0x7f8f60004e60 auth_method 0 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f74874640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f74874640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f74874640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f74874640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:21:57.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f74874640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f74874640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f75075640 10 monclient(hunting): handle_auth_done global_id 55518 payload 824 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f75075640 1 -- 172.21.15.98:0/3094063080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f60004e60 msgr2=0x7f8f701cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f75075640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f60004e60 0x7f8f701cd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f75075640 1 -- 172.21.15.98:0/3094063080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f70137c10 msgr2=0x7f8f70137ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f75075640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f70137c10 0x7f8f70137ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f75075640 10 monclient: _finish_hunting 0 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f75075640 1 monclient: found mon.noname-d 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f75075640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f75075640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f75075640 1 -- 172.21.15.98:0/3094063080 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f70125010 con 0x7f8f70121670 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6ffff640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f70137c10 0x7f8f70137ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:21:57.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f75075640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f70121670 0x7f8f70121ad0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8f700a8e50 tx=0x7f8f68016710 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 10 client.?.objecter ms_handle_connect 0x7f8f70121670 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 1 -- 172.21.15.98:0/3094063080 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f6801c070 con 0x7f8f70121670 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 10 monclient: dump: 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:21:57.312 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 10 monclient: _finish_auth 0 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 1 -- 172.21.15.98:0/3094063080 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f68009ca0 con 0x7f8f70121670 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f76ad8640 5 monclient: authenticate success, global_id 55518 2024-06-11T19:21:57.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 1 -- 172.21.15.98:0/3094063080 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f68021d40 con 0x7f8f70121670 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f76ad8640 10 monclient: _renew_subs 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f76ad8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f76ad8640 1 -- 172.21.15.98:0/3094063080 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f701cd800 con 0x7f8f70121670 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6effd640 4 set_mon_vals no callback set 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f6d7fa640 10 monclient: dump: 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:21:57.314 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f76ad8640 10 client.55518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f76ad8640 10 monclient: _renew_subs 2024-06-11T19:21:57.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f76ad8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f76ad8640 1 -- 172.21.15.98:0/3094063080 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f700a4520 con 0x7f8f70121670 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.294+0000 7f8f76ad8640 1 librados: init done 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f6d7fa640 1 -- 172.21.15.98:0/3094063080 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8f68005d10 con 0x7f8f70121670 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f76ad8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f76ad8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f76ad8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f76ad8640 1 -- 172.21.15.98:0/3094063080 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f700a5000 con 0x7f8f70121670 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f6d7fa640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8f5403fdb0 0x7f8f54042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f6d7fa640 1 -- 172.21.15.98:0/3094063080 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(506..506 src has 1..506) ==== 6685+0+0 (secure 0 0 0) 0x7f8f68012040 con 0x7f8f70121670 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f6d7fa640 10 client.55518.objecter ms_dispatch 0x7f8f70000b90 osd_map(506..506 src has 1..506) 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f6d7fa640 3 client.55518.objecter handle_osd_map got epochs [506,506] > 0 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f6d7fa640 3 client.55518.objecter handle_osd_map decoding full epoch 506 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f6d7fa640 20 client.55518.objecter dump_active .. 0 homeless 2024-06-11T19:21:57.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f74874640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8f5403fdb0 0x7f8f54042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:21:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.296+0000 7f8f74874640 10 monclient: get_auth_request con 0x7f8f5403fdb0 auth_method 0 2024-06-11T19:21:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.297+0000 7f8f6d7fa640 10 client.55518.objecter ms_handle_connect 0x7f8f5403fdb0 2024-06-11T19:21:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.297+0000 7f8f74874640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8f5403fdb0 0x7f8f54042270 secure :-1 s=READY pgs=4237 cs=0 l=1 rev1=1 crypto rx=0x7f8f70051cc0 tx=0x7f8f64008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:21:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.301+0000 7f8f6d7fa640 1 -- 172.21.15.98:0/3094063080 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8f68026060 con 0x7f8f70121670 2024-06-11T19:21:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.301+0000 7f8f6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:21:57.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.301+0000 7f8f6d7fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:21:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.561+0000 7f8f76ad8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:21:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.561+0000 7f8f76ad8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:21:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.561+0000 7f8f76ad8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:21:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.561+0000 7f8f76ad8640 1 -- 172.21.15.98:0/3094063080 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8f7006c300 con 0x7f8f70121670 2024-06-11T19:21:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.561+0000 7f8f6d7fa640 1 -- 172.21.15.98:0/3094063080 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3180 (secure 0 0 0) 0x7f8f6801fc80 con 0x7f8f70121670 2024-06-11T19:21:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.561+0000 7f8f6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:21:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.561+0000 7f8f6d7fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:21:57.561 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":60,"quorum":[0,1,2,3,5,6,7],"quorum_names":["a","f","b","g","h","d","i"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:21:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 10 librados: watch_flush enter 2024-06-11T19:21:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 10 librados: watch_flush exit 2024-06-11T19:21:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 20 client.55518.objecter shutdown clearing up homeless session... 2024-06-11T19:21:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 10 client.55518.objecter successfully canceled tick 2024-06-11T19:21:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 -- 172.21.15.98:0/3094063080 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8f5403fdb0 msgr2=0x7f8f54042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:21:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8f5403fdb0 0x7f8f54042270 secure :-1 s=READY pgs=4237 cs=0 l=1 rev1=1 crypto rx=0x7f8f70051cc0 tx=0x7f8f64008040 comp rx=0 tx=0).stop 2024-06-11T19:21:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 10 monclient: shutdown 2024-06-11T19:21:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:21:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 -- 172.21.15.98:0/3094063080 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f70121670 msgr2=0x7f8f70121ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:21:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f70121670 0x7f8f70121ad0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8f700a8e50 tx=0x7f8f68016710 comp rx=0 tx=0).stop 2024-06-11T19:21:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 -- 172.21.15.98:0/3094063080 shutdown_connections 2024-06-11T19:21:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8f5403fdb0 0x7f8f54042270 unknown :-1 s=CLOSED pgs=4237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f70121670 0x7f8f70121ad0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f70137c10 0x7f8f70137ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 --2- 172.21.15.98:0/3094063080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f60004e60 0x7f8f701cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:21:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 -- 172.21.15.98:0/3094063080 >> 172.21.15.98:0/3094063080 conn(0x7f8f700ab850 msgr2=0x7f8f701c0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:21:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 -- 172.21.15.98:0/3094063080 shutdown_connections 2024-06-11T19:21:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 -- 172.21.15.98:0/3094063080 wait complete. 2024-06-11T19:21:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:21:57.563+0000 7f8f76ad8640 1 librados: shutdown 2024-06-11T19:22:00.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:22:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.684+0000 7f634667a640 10 monclient: get_monmap_and_config 2024-06-11T19:22:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.684+0000 7f634667a640 10 monclient: build_initial_monmap 2024-06-11T19:22:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.684+0000 7f634667a640 10 monclient: monmap: 2024-06-11T19:22:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:00.685401+0000 2024-06-11T19:22:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:00.685401+0000 2024-06-11T19:22:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:00.685 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:00.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 1 Processor -- start 2024-06-11T19:22:00.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 1 -- start start 2024-06-11T19:22:00.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 10 monclient: init 2024-06-11T19:22:00.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 10 monclient: _add_conns ranks=[8,3,10,2,6,7,9,1,0,4,5] 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63400a4920 0x7f63400a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 10 monclient(hunting): picked mon.noname-e con 0x7f63400a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a5240 0x7f63400a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 10 monclient(hunting): picked mon.noname-g con 0x7f63400a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f63400a2530 0x7f63400a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 10 monclient(hunting): picked mon.noname-f con 0x7f63400a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f63400a7140 con 0x7f63400a2530 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f63400a2b50 con 0x7f63400a5240 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f63400a2e30 con 0x7f63400a4920 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:00.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f634667a640 10 monclient(hunting): authenticate will time out at 1530.492861s 2024-06-11T19:22:00.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f633ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63400a4920 0x7f63400a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:00.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.685+0000 7f633ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63400a4920 0x7f63400a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:47892/0 (socket says 172.21.15.98:47892) 2024-06-11T19:22:00.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633ffff640 1 -- 172.21.15.98:0/3290659354 learned_addr learned my addr 172.21.15.98:0/3290659354 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:00.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633ffff640 10 monclient(hunting): get_auth_request con 0x7f63400a4920 auth_method 0 2024-06-11T19:22:00.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:00.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:00.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:00.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633f7fe640 1 --2- 172.21.15.98:0/3290659354 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a5240 0x7f63400a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:00.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 1 --2- 172.21.15.98:0/3290659354 >> v2:172.21.15.117:3300/0 conn(0x7f63400a2530 0x7f63400a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:00.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633ffff640 1 -- 172.21.15.98:0/3290659354 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63400a4920 msgr2=0x7f63400a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:22:00.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633ffff640 1 -- 172.21.15.98:0/3290659354 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63400a4920 msgr2=0x7f63400a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:00.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633ffff640 1 --2- 172.21.15.98:0/3290659354 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63400a4920 0x7f63400a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:00.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 10 monclient(hunting): get_auth_request con 0x7f63400a2530 auth_method 0 2024-06-11T19:22:00.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:00.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:00.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:00.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633f7fe640 10 monclient(hunting): get_auth_request con 0x7f63400a5240 auth_method 0 2024-06-11T19:22:00.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:00.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633ffff640 1 --2- 172.21.15.98:0/3290659354 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63400a4920 0x7f63400a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633f7fe640 1 -- 172.21.15.98:0/3290659354 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a5240 msgr2=0x7f63400a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633f7fe640 1 -- 172.21.15.98:0/3290659354 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a5240 msgr2=0x7f63400a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633f7fe640 1 --2- 172.21.15.98:0/3290659354 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a5240 0x7f63400a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f633f7fe640 1 --2- 172.21.15.98:0/3290659354 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a5240 0x7f63400a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 10 monclient(hunting): handle_auth_done global_id 32626 payload 274 2024-06-11T19:22:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 1 -- 172.21.15.98:0/3290659354 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63400a4920 msgr2=0x7f63400a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:00.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 1 --2- 172.21.15.98:0/3290659354 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63400a4920 0x7f63400a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 1 -- 172.21.15.98:0/3290659354 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a5240 msgr2=0x7f63400a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 1 --2- 172.21.15.98:0/3290659354 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a5240 0x7f63400a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 10 monclient: _finish_hunting 0 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 1 monclient: found mon.noname-f 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.686+0000 7f6344c17640 1 -- 172.21.15.98:0/3290659354 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63400a3110 con 0x7f63400a2530 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f6344c17640 1 --2- 172.21.15.98:0/3290659354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63400a2530 0x7f63400a5ee0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f6338009d90 tx=0x7f633802bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=232f7a517751b773 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 1 -- 172.21.15.98:0/3290659354 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6338039070 con 0x7f63400a2530 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 10 monclient: dump: 2024-06-11T19:22:00.700 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:00.701 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:00.702 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 10 monclient: _finish_auth 0 2024-06-11T19:22:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 10 monclient: _check_auth_tickets 2024-06-11T19:22:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 5 monclient: authenticate success, global_id 32626 2024-06-11T19:22:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 1 -- 172.21.15.98:0/3290659354 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f633802cd00 con 0x7f63400a2530 2024-06-11T19:22:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:00.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:00.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 4 set_mon_vals no callback set 2024-06-11T19:22:00.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 1 -- 172.21.15.98:0/3290659354 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63380327b0 con 0x7f63400a2530 2024-06-11T19:22:00.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:00.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:22:00.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f633effd640 10 monclient: dump: 2024-06-11T19:22:00.703 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:00.703 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:00.703 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:00.704 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 10 monclient: shutdown 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 1 -- 172.21.15.98:0/3290659354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63400a2530 msgr2=0x7f63400a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 1 --2- 172.21.15.98:0/3290659354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63400a2530 0x7f63400a5ee0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f6338009d90 tx=0x7f633802bce0 comp rx=0 tx=0).stop 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 1 -- 172.21.15.98:0/3290659354 shutdown_connections 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 1 --2- 172.21.15.98:0/3290659354 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a5240 0x7f63400a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 1 --2- 172.21.15.98:0/3290659354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63400a2530 0x7f63400a5ee0 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 1 --2- 172.21.15.98:0/3290659354 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63400a4920 0x7f63400a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.687+0000 7f634667a640 1 -- 172.21.15.98:0/3290659354 >> 172.21.15.98:0/3290659354 conn(0x7f63400ab850 msgr2=0x7f63400a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.688+0000 7f634667a640 1 -- 172.21.15.98:0/3290659354 shutdown_connections 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.688+0000 7f634667a640 1 -- 172.21.15.98:0/3290659354 wait complete. 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.688+0000 7f634667a640 10 monclient: build_initial_monmap 2024-06-11T19:22:00.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.688+0000 7f634667a640 10 monclient: monmap: 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:00.706 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.688+0000 7f634667a640 1 librados: starting msgr at 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.688+0000 7f634667a640 1 librados: starting objecter 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.688+0000 7f634667a640 1 Processor -- start 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 1 -- start start 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 1 librados: setting wanted keys 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 1 librados: calling monclient init 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 10 monclient: init 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 10 monclient: _add_conns ranks=[3,7,2,5,4,6,8,1,0] 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a2530 0x7f63401cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:00.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 10 monclient(hunting): picked mon.noname-g con 0x7f63400a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f63400a4920 0x7f63401ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 10 monclient(hunting): picked mon.noname-i con 0x7f63400a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63400a5240 0x7f6340123f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 10 monclient(hunting): picked mon.noname-b con 0x7f63400a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f63400a6da0 con 0x7f63400a5240 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f6340138e50 con 0x7f63400a2530 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f63400a6420 con 0x7f63400a4920 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f634667a640 10 monclient(hunting): authenticate will time out at 1530.496733s 2024-06-11T19:22:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f6344c17640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63400a5240 0x7f6340123f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f6344c17640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63400a5240 0x7f6340123f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:54254/0 (socket says 172.21.15.98:54254) 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f633ffff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a2530 0x7f63401cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f6344c17640 1 -- 172.21.15.98:0/1655318198 learned_addr learned my addr 172.21.15.98:0/1655318198 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f6344c17640 10 monclient(hunting): get_auth_request con 0x7f63400a5240 auth_method 0 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f6344c17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f6344c17640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f6344c17640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f633f7fe640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f63400a4920 0x7f63401ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f633ffff640 10 monclient(hunting): get_auth_request con 0x7f63400a2530 auth_method 0 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f633ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f633ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f633ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f633f7fe640 10 monclient(hunting): get_auth_request con 0x7f63400a4920 auth_method 0 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f633f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f633f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f633f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f6344c17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f6344c17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.689+0000 7f6344c17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633ffff640 1 -- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a2530 msgr2=0x7f63401cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633ffff640 1 -- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a2530 msgr2=0x7f63401cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633ffff640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a2530 0x7f63401cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633ffff640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a2530 0x7f63401cdad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f6344c17640 10 monclient(hunting): handle_auth_done global_id 84872 payload 824 2024-06-11T19:22:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f6344c17640 1 -- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f63400a4920 msgr2=0x7f63401ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f6344c17640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f63400a4920 0x7f63401ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f6344c17640 1 -- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a2530 msgr2=0x7f63401cdad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f6344c17640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a2530 0x7f63401cdad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f6344c17640 10 monclient: _finish_hunting 0 2024-06-11T19:22:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f6344c17640 1 monclient: found mon.noname-b 2024-06-11T19:22:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f6344c17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f6344c17640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:22:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f6344c17640 1 -- 172.21.15.98:0/1655318198 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6340127960 con 0x7f63400a5240 2024-06-11T19:22:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f6344c17640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63400a5240 0x7f6340123f60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f633802bd20 tx=0x7f6338004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 10 client.?.objecter ms_handle_connect 0x7f63400a5240 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 1 -- 172.21.15.98:0/1655318198 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6338039050 con 0x7f63400a5240 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 10 monclient: dump: 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:00.713 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 10 monclient: _finish_auth 0 2024-06-11T19:22:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 10 monclient: _check_auth_tickets 2024-06-11T19:22:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:00.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 1 -- 172.21.15.98:0/1655318198 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6338045070 con 0x7f63400a5240 2024-06-11T19:22:00.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f634667a640 5 monclient: authenticate success, global_id 84872 2024-06-11T19:22:00.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:00.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 1 -- 172.21.15.98:0/1655318198 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63380046c0 con 0x7f63400a5240 2024-06-11T19:22:00.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f634667a640 10 monclient: _renew_subs 2024-06-11T19:22:00.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f634667a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:00.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f634667a640 1 -- 172.21.15.98:0/1655318198 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6340127c40 con 0x7f63400a5240 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633e7fc640 4 set_mon_vals no callback set 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f633cff9640 10 monclient: dump: 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f634667a640 10 client.84872.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f634667a640 10 monclient: _renew_subs 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f634667a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f634667a640 1 -- 172.21.15.98:0/1655318198 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6340138760 con 0x7f63400a5240 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.690+0000 7f634667a640 1 librados: init done 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.691+0000 7f633cff9640 1 -- 172.21.15.98:0/1655318198 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f6338004250 con 0x7f63400a5240 2024-06-11T19:22:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.691+0000 7f634667a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.691+0000 7f634667a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.691+0000 7f634667a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.691+0000 7f634667a640 1 -- 172.21.15.98:0/1655318198 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63400a8f10 con 0x7f63400a5240 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.691+0000 7f633cff9640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f632c03fde0 0x7f632c0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.691+0000 7f633cff9640 1 -- 172.21.15.98:0/1655318198 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(510..510 src has 1..510) ==== 6677+0+0 (secure 0 0 0) 0x7f6338074540 con 0x7f63400a5240 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.691+0000 7f633cff9640 10 client.84872.objecter ms_dispatch 0x7f6340000b90 osd_map(510..510 src has 1..510) 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.691+0000 7f633cff9640 3 client.84872.objecter handle_osd_map got epochs [510,510] > 0 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.691+0000 7f633cff9640 3 client.84872.objecter handle_osd_map decoding full epoch 510 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.692+0000 7f633cff9640 20 client.84872.objecter dump_active .. 0 homeless 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.692+0000 7f633ffff640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f632c03fde0 0x7f632c0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.692+0000 7f633ffff640 10 monclient: get_auth_request con 0x7f632c03fde0 auth_method 0 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.692+0000 7f633cff9640 10 client.84872.objecter ms_handle_connect 0x7f632c03fde0 2024-06-11T19:22:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.692+0000 7f633ffff640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f632c03fde0 0x7f632c0422a0 secure :-1 s=READY pgs=4238 cs=0 l=1 rev1=1 crypto rx=0x7f6330009100 tx=0x7f6330008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:00.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.696+0000 7f633cff9640 1 -- 172.21.15.98:0/1655318198 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f633803e400 con 0x7f63400a5240 2024-06-11T19:22:00.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.696+0000 7f633cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:00.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.696+0000 7f633cff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:00.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.947+0000 7f634667a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:22:00.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.947+0000 7f634667a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:22:00.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.947+0000 7f634667a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:00.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.947+0000 7f634667a640 1 -- 172.21.15.98:0/1655318198 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f63400a6750 con 0x7f63400a5240 2024-06-11T19:22:00.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.948+0000 7f633cff9640 1 -- 172.21.15.98:0/1655318198 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3180 (secure 0 0 0) 0x7f633804a460 con 0x7f63400a5240 2024-06-11T19:22:00.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.948+0000 7f633cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:22:00.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.948+0000 7f633cff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:00.948 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":60,"quorum":[0,1,2,3,5,6,7],"quorum_names":["a","f","b","g","h","d","i"],"quorum_leader_name":"f","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:22:00.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.949+0000 7f634667a640 10 librados: watch_flush enter 2024-06-11T19:22:00.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.949+0000 7f634667a640 10 librados: watch_flush exit 2024-06-11T19:22:00.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.949+0000 7f634667a640 20 client.84872.objecter shutdown clearing up homeless session... 2024-06-11T19:22:00.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.949+0000 7f634667a640 10 client.84872.objecter successfully canceled tick 2024-06-11T19:22:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.949+0000 7f634667a640 1 -- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f632c03fde0 msgr2=0x7f632c0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.949+0000 7f634667a640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f632c03fde0 0x7f632c0422a0 secure :-1 s=READY pgs=4238 cs=0 l=1 rev1=1 crypto rx=0x7f6330009100 tx=0x7f6330008040 comp rx=0 tx=0).stop 2024-06-11T19:22:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.949+0000 7f634667a640 10 monclient: shutdown 2024-06-11T19:22:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.949+0000 7f634667a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.949+0000 7f634667a640 1 -- 172.21.15.98:0/1655318198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63400a5240 msgr2=0x7f6340123f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.949+0000 7f634667a640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63400a5240 0x7f6340123f60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f633802bd20 tx=0x7f6338004a40 comp rx=0 tx=0).stop 2024-06-11T19:22:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.950+0000 7f634667a640 1 -- 172.21.15.98:0/1655318198 shutdown_connections 2024-06-11T19:22:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.950+0000 7f634667a640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f632c03fde0 0x7f632c0422a0 unknown :-1 s=CLOSED pgs=4238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.950+0000 7f634667a640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63400a5240 0x7f6340123f60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.950+0000 7f634667a640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f63400a4920 0x7f63401ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.950+0000 7f634667a640 1 --2- 172.21.15.98:0/1655318198 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63400a2530 0x7f63401cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:00.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.950+0000 7f634667a640 1 -- 172.21.15.98:0/1655318198 >> 172.21.15.98:0/1655318198 conn(0x7f63400ab850 msgr2=0x7f63400a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:00.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.950+0000 7f634667a640 1 -- 172.21.15.98:0/1655318198 shutdown_connections 2024-06-11T19:22:00.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.950+0000 7f634667a640 1 -- 172.21.15.98:0/1655318198 wait complete. 2024-06-11T19:22:00.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:00.950+0000 7f634667a640 1 librados: shutdown 2024-06-11T19:22:03.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:22:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.072+0000 7f407cdda640 10 monclient: get_monmap_and_config 2024-06-11T19:22:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.072+0000 7f407cdda640 10 monclient: build_initial_monmap 2024-06-11T19:22:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.072+0000 7f407cdda640 10 monclient: monmap: 2024-06-11T19:22:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:04.073547+0000 2024-06-11T19:22:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:04.073547+0000 2024-06-11T19:22:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:04.073 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 1 Processor -- start 2024-06-11T19:22:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 1 -- start start 2024-06-11T19:22:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 10 monclient: init 2024-06-11T19:22:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 10 monclient: _add_conns ranks=[8,10,0,5,9,2,7,1,4,6,3] 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 10 monclient(hunting): picked mon.noname-e con 0x7f40780a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 10 monclient(hunting): picked mon.noname-f con 0x7f40780a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40780a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40780a7140 con 0x7f40780a2530 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f40780a2b50 con 0x7f40780a5240 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f40780a2e30 con 0x7f40780a4920 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407cdda640 10 monclient(hunting): authenticate will time out at 1533.880857s 2024-06-11T19:22:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f407659c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.073+0000 7f4076d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f40780a2530 0x7f40780a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56798/0 (socket says 172.21.15.98:56798) 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f4076d9d640 1 -- 172.21.15.98:0/4216347699 learned_addr learned my addr 172.21.15.98:0/4216347699 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f407659c640 10 monclient(hunting): get_auth_request con 0x7f40780a4920 auth_method 0 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f407659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f407659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f407659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f407659c640 1 -- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 msgr2=0x7f40780a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f407659c640 1 -- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 msgr2=0x7f40780a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f407659c640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f407659c640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f4075d9b640 10 monclient(hunting): get_auth_request con 0x7f40780a5240 auth_method 0 2024-06-11T19:22:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f4075d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f4075d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f4075d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f4075d9b640 1 -- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 msgr2=0x7f40780a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f4075d9b640 1 -- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 msgr2=0x7f40780a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:04.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.074+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f407659c640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f407659c640 10 monclient(hunting): get_auth_request con 0x7f40780a4920 auth_method 0 2024-06-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f407659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f407659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f407659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f407659c640 1 -- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 msgr2=0x7f40780a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f407659c640 1 -- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 msgr2=0x7f40780a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f407659c640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f4075d9b640 10 monclient(hunting): get_auth_request con 0x7f40780a5240 auth_method 0 2024-06-11T19:22:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f4075d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f4075d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f4075d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f407659c640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f4075d9b640 1 -- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 msgr2=0x7f40780a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f4075d9b640 1 -- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 msgr2=0x7f40780a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.275+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:04.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f407659c640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:04.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f407659c640 10 monclient(hunting): get_auth_request con 0x7f40780a4920 auth_method 0 2024-06-11T19:22:04.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f407659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:04.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f407659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f407659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f407659c640 1 -- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 msgr2=0x7f40780a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f407659c640 1 -- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 msgr2=0x7f40780a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f407659c640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f407659c640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f4075d9b640 10 monclient(hunting): get_auth_request con 0x7f40780a5240 auth_method 0 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f4075d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f4075d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f4075d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f4075d9b640 1 -- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 msgr2=0x7f40780a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f4075d9b640 1 -- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 msgr2=0x7f40780a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:04.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:04.676+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:05.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f407659c640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f407659c640 10 monclient(hunting): get_auth_request con 0x7f40780a4920 auth_method 0 2024-06-11T19:22:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f407659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f407659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f407659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f407659c640 1 -- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 msgr2=0x7f40780a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:22:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f407659c640 1 -- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 msgr2=0x7f40780a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f407659c640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f407659c640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f4075d9b640 10 monclient(hunting): get_auth_request con 0x7f40780a5240 auth_method 0 2024-06-11T19:22:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f4075d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f4075d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f4075d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f4075d9b640 1 -- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 msgr2=0x7f40780a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f4075d9b640 1 -- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 msgr2=0x7f40780a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:05.477+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:06.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:06.526+0000 7f407559a640 1 -- 172.21.15.98:0/4216347699 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 953941224 0 0) 0x7f40780a7140 con 0x7f40780a2530 2024-06-11T19:22:06.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:06.526+0000 7f407559a640 10 monclient(hunting): _negotiate 2024-06-11T19:22:06.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:06.526+0000 7f407559a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:06.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:06.526+0000 7f407559a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:06.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:06.526+0000 7f407559a640 10 monclient(hunting): my global_id is 65592 2024-06-11T19:22:06.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:06.526+0000 7f407559a640 1 -- 172.21.15.98:0/4216347699 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4060003520 con 0x7f40780a2530 2024-06-11T19:22:06.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:06.527+0000 7f4076d9d640 1 -- 172.21.15.98:0/4216347699 >> v1:172.21.15.98:6789/0 conn(0x7f40780a2530 legacy=0x7f40780a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-06-11T19:22:06.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:06.527+0000 7f4076d9d640 1 -- 172.21.15.98:0/4216347699 >> v1:172.21.15.98:6789/0 conn(0x7f40780a2530 legacy=0x7f40780a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:22:06.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:06.527+0000 7f4076d9d640 1 --1- 172.21.15.98:0/4216347699 >> v1:172.21.15.98:6789/0 conn(0x7f40780a2530 0x7f40780a5ee0 :-1 s=OPENED pgs=388 cs=1 l=1).handle_message read tag failed 2024-06-11T19:22:06.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:06.527+0000 7f4076d9d640 1 --1- 172.21.15.98:0/4216347699 >> v1:172.21.15.98:6789/0 conn(0x7f40780a2530 0x7f40780a5ee0 :-1 s=OPENED pgs=388 cs=1 l=1).fault on lossy channel, failing 2024-06-11T19:22:06.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:06.527+0000 7f407559a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.98:6789/0 2024-06-11T19:22:07.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.073+0000 7f4067fff640 10 monclient(hunting): tick 2024-06-11T19:22:07.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.073+0000 7f4067fff640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:22:07.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.073+0000 7f4067fff640 1 monclient(hunting): continuing hunt 2024-06-11T19:22:07.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.073+0000 7f4067fff640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:22:07.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.073+0000 7f4067fff640 1 -- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 msgr2=0x7f40780a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:07.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.073+0000 7f4067fff640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40780a4920 0x7f40780a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 1 -- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 msgr2=0x7f40780a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:07.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 1 --2- 172.21.15.98:0/4216347699 >> v2:172.21.15.117:3300/0 conn(0x7f40780a5240 0x7f40780a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 1 -- 172.21.15.98:0/4216347699 >> v1:172.21.15.98:6789/0 conn(0x7f40780a2530 legacy=0x7f40780a5ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:07.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 10 monclient: _add_conns ranks=[5,4,7,6,2,3,1,9] 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4058003740 0x7f4058005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 10 monclient(hunting): picked mon.noname-h con 0x7f4058003740 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f40580061f0 0x7f4058008680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 10 monclient(hunting): picked mon.noname-c con 0x7f40580061f0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4058008d70 0x7f405800b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 10 monclient(hunting): picked mon.noname-i con 0x7f4058008d70 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 1 -- 172.21.15.98:0/4216347699 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f40780a2b50 con 0x7f40580061f0 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 1 -- 172.21.15.98:0/4216347699 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f40780a2e30 con 0x7f4058003740 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 1 -- 172.21.15.98:0/4216347699 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f405800b9c0 con 0x7f4058008d70 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4067fff640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f40580061f0 0x7f4058008680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 10 monclient(hunting): get_auth_request con 0x7f40580061f0 auth_method 0 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4076d9d640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4058008d70 0x7f405800b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f407659c640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4058003740 0x7f4058005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4076d9d640 10 monclient(hunting): get_auth_request con 0x7f4058008d70 auth_method 0 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4076d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4076d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4076d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f407659c640 10 monclient(hunting): get_auth_request con 0x7f4058003740 auth_method 0 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f407659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f407659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f407659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4076d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4076d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4076d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f407659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f407659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f407659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 10 monclient(hunting): handle_auth_done global_id 114106 payload 274 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 1 -- 172.21.15.98:0/4216347699 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4058008d70 msgr2=0x7f405800b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4058008d70 0x7f405800b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 1 -- 172.21.15.98:0/4216347699 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4058003740 msgr2=0x7f4058005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4058003740 0x7f4058005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 1 monclient: found mon.noname-c 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4075d9b640 1 -- 172.21.15.98:0/4216347699 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40780a3110 con 0x7f40580061f0 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.074+0000 7f4076d9d640 1 -- 172.21.15.98:0/4216347699 reap_dead start 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f4075d9b640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f40580061f0 0x7f4058008680 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4078073fc0 tx=0x7f406c02c460 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1f648bd3869e4923 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 1 -- 172.21.15.98:0/4216347699 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f406c03a070 con 0x7f40580061f0 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 10 monclient: dump: 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 10 monclient: _finish_auth 0 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 10 monclient: _check_auth_tickets 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 1 -- 172.21.15.98:0/4216347699 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406c0347e0 con 0x7f40580061f0 2024-06-11T19:22:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407cdda640 5 monclient: authenticate success, global_id 114106 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407cdda640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407cdda640 4 set_mon_vals no callback set 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 1 -- 172.21.15.98:0/4216347699 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f406c034990 con 0x7f40580061f0 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.075+0000 7f407559a640 10 monclient: dump: 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.076+0000 7f407cdda640 10 monclient: shutdown 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.076+0000 7f407cdda640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.076+0000 7f407cdda640 1 -- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f40580061f0 msgr2=0x7f4058008680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.076+0000 7f407cdda640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f40580061f0 0x7f4058008680 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4078073fc0 tx=0x7f406c02c460 comp rx=0 tx=0).stop 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.076+0000 7f407cdda640 1 -- 172.21.15.98:0/4216347699 shutdown_connections 2024-06-11T19:22:07.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.076+0000 7f407cdda640 1 --2- 172.21.15.98:0/4216347699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f40580061f0 0x7f4058008680 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.076+0000 7f407cdda640 1 -- 172.21.15.98:0/4216347699 >> 172.21.15.98:0/4216347699 conn(0x7f40780ab850 msgr2=0x7f40780a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:07.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.076+0000 7f407cdda640 1 -- 172.21.15.98:0/4216347699 shutdown_connections 2024-06-11T19:22:07.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.076+0000 7f407cdda640 1 -- 172.21.15.98:0/4216347699 wait complete. 2024-06-11T19:22:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 10 monclient: build_initial_monmap 2024-06-11T19:22:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 10 monclient: monmap: 2024-06-11T19:22:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 1 librados: starting msgr at 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 1 librados: starting objecter 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 1 Processor -- start 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 1 -- start start 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 1 librados: setting wanted keys 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 1 librados: calling monclient init 2024-06-11T19:22:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 10 monclient: init 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 10 monclient: _add_conns ranks=[5,3,8,0,4,7,1,6,2] 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f40580061f0 0x7f40781cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 10 monclient(hunting): picked mon.noname-h con 0x7f40580061f0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40781cdfb0 0x7f4078124eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 10 monclient(hunting): picked mon.noname-g con 0x7f40781cdfb0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40781ce390 0x7f40781253f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 10 monclient(hunting): picked mon.noname-e con 0x7f40781ce390 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.078+0000 7f407cdda640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f40780a6da0 con 0x7f40781cdfb0 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407cdda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407cdda640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f4078138e50 con 0x7f40580061f0 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407cdda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407cdda640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f40780a4d00 con 0x7f40781ce390 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407cdda640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407cdda640 10 monclient(hunting): authenticate will time out at 1536.886028s 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4076d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40781ce390 0x7f40781253f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4076d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40781ce390 0x7f40781253f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:58020/0 (socket says 172.21.15.98:58020) 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4076d9d640 1 -- 172.21.15.98:0/3268294256 learned_addr learned my addr 172.21.15.98:0/3268294256 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4076d9d640 10 monclient(hunting): get_auth_request con 0x7f40781ce390 auth_method 0 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4076d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4076d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4076d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407659c640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f40580061f0 0x7f40781cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40781cdfb0 0x7f4078124eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407659c640 10 monclient(hunting): get_auth_request con 0x7f40580061f0 auth_method 0 2024-06-11T19:22:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 10 monclient(hunting): get_auth_request con 0x7f40781cdfb0 auth_method 0 2024-06-11T19:22:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4076d9d640 1 -- 172.21.15.98:0/3268294256 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40781ce390 msgr2=0x7f40781253f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:22:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4076d9d640 1 -- 172.21.15.98:0/3268294256 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40781ce390 msgr2=0x7f40781253f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:22:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4076d9d640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40781ce390 0x7f40781253f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4076d9d640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40781ce390 0x7f40781253f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 10 monclient(hunting): handle_auth_done global_id 114105 payload 824 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 1 -- 172.21.15.98:0/3268294256 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40781ce390 msgr2=0x7f40781253f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40781ce390 0x7f40781253f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 1 -- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f40580061f0 msgr2=0x7f40781cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f40580061f0 0x7f40781cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 1 monclient: found mon.noname-g 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:22:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 1 -- 172.21.15.98:0/3268294256 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4078128d60 con 0x7f40781cdfb0 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f407659c640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f40580061f0 0x7f40781cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.079+0000 7f4075d9b640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40781cdfb0 0x7f4078124eb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f406c034000 tx=0x7f406c0046c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 10 client.?.objecter ms_handle_connect 0x7f40781cdfb0 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 1 -- 172.21.15.98:0/3268294256 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f406c044070 con 0x7f40781cdfb0 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 10 monclient: dump: 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 10 monclient: _finish_auth 0 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f407cdda640 5 monclient: authenticate success, global_id 114105 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f407cdda640 10 monclient: _renew_subs 2024-06-11T19:22:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 1 -- 172.21.15.98:0/3268294256 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406c03f8d0 con 0x7f40781cdfb0 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f407cdda640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f407cdda640 1 -- 172.21.15.98:0/3268294256 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40781275d0 con 0x7f40781cdfb0 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 1 -- 172.21.15.98:0/3268294256 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f406c03a040 con 0x7f40781cdfb0 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4066ffd640 10 monclient: dump: 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f407cdda640 10 client.114105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f4074d99640 4 set_mon_vals no callback set 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f407cdda640 10 monclient: _renew_subs 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f407cdda640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f407cdda640 1 -- 172.21.15.98:0/3268294256 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40781277f0 con 0x7f40781cdfb0 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.080+0000 7f407cdda640 1 librados: init done 2024-06-11T19:22:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f4066ffd640 1 -- 172.21.15.98:0/3268294256 <== mon.3 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f406c047070 con 0x7f40781cdfb0 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f4066ffd640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4058041000 0x7f40580434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f4066ffd640 1 -- 172.21.15.98:0/3268294256 <== mon.3 v2:172.21.15.117:3301/0 5 ==== osd_map(512..512 src has 1..512) ==== 6677+0+0 (secure 0 0 0) 0x7f406c032070 con 0x7f40781cdfb0 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f4066ffd640 10 client.114105.objecter ms_dispatch 0x7f4078000b90 osd_map(512..512 src has 1..512) 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f4066ffd640 3 client.114105.objecter handle_osd_map got epochs [512,512] > 0 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f4066ffd640 3 client.114105.objecter handle_osd_map decoding full epoch 512 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f407cdda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f407cdda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f407cdda640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f407cdda640 1 -- 172.21.15.98:0/3268294256 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40781284c0 con 0x7f40781cdfb0 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f407659c640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4058041000 0x7f40580434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f4066ffd640 20 client.114105.objecter dump_active .. 0 homeless 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.081+0000 7f407659c640 10 monclient: get_auth_request con 0x7f4058041000 auth_method 0 2024-06-11T19:22:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.082+0000 7f4066ffd640 10 client.114105.objecter ms_handle_connect 0x7f4058041000 2024-06-11T19:22:07.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.082+0000 7f407659c640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4058041000 0x7f40580434c0 secure :-1 s=READY pgs=4239 cs=0 l=1 rev1=1 crypto rx=0x7f4068006fd0 tx=0x7f4068008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.087+0000 7f4066ffd640 1 -- 172.21.15.98:0/3268294256 <== mon.3 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f406c0347b0 con 0x7f40781cdfb0 2024-06-11T19:22:07.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.087+0000 7f4066ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:07.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.087+0000 7f4066ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:07.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.354+0000 7f407cdda640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:22:07.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.354+0000 7f407cdda640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:22:07.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.354+0000 7f407cdda640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:22:07.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.354+0000 7f407cdda640 1 -- 172.21.15.98:0/3268294256 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f40780a5000 con 0x7f40781cdfb0 2024-06-11T19:22:07.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.355+0000 7f4066ffd640 1 -- 172.21.15.98:0/3268294256 <== mon.3 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f406c03e3c0 con 0x7f40781cdfb0 2024-06-11T19:22:07.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.355+0000 7f4066ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:22:07.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.355+0000 7f4066ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:07.354 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":64,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:22:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.356+0000 7f407cdda640 10 librados: watch_flush enter 2024-06-11T19:22:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 10 librados: watch_flush exit 2024-06-11T19:22:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 20 client.114105.objecter shutdown clearing up homeless session... 2024-06-11T19:22:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 10 client.114105.objecter successfully canceled tick 2024-06-11T19:22:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 -- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4058041000 msgr2=0x7f40580434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4058041000 0x7f40580434c0 secure :-1 s=READY pgs=4239 cs=0 l=1 rev1=1 crypto rx=0x7f4068006fd0 tx=0x7f4068008040 comp rx=0 tx=0).stop 2024-06-11T19:22:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 10 monclient: shutdown 2024-06-11T19:22:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 -- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40781cdfb0 msgr2=0x7f4078124eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40781cdfb0 0x7f4078124eb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f406c034000 tx=0x7f406c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 -- 172.21.15.98:0/3268294256 shutdown_connections 2024-06-11T19:22:07.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4058041000 0x7f40580434c0 unknown :-1 s=CLOSED pgs=4239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40781cdfb0 0x7f4078124eb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40781ce390 0x7f40781253f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 --2- 172.21.15.98:0/3268294256 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f40580061f0 0x7f40781cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 -- 172.21.15.98:0/3268294256 >> 172.21.15.98:0/3268294256 conn(0x7f40780ab850 msgr2=0x7f407809c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:07.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 -- 172.21.15.98:0/3268294256 shutdown_connections 2024-06-11T19:22:07.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 -- 172.21.15.98:0/3268294256 wait complete. 2024-06-11T19:22:07.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.357+0000 7f407cdda640 1 librados: shutdown 2024-06-11T19:22:07.367 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:22:07.367 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:22:07.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:22:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.466+0000 7f5d2ab3f640 10 monclient: get_monmap_and_config 2024-06-11T19:22:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.466+0000 7f5d2ab3f640 10 monclient: build_initial_monmap 2024-06-11T19:22:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.466+0000 7f5d2ab3f640 10 monclient: monmap: 2024-06-11T19:22:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:07.467810+0000 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:07.467810+0000 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:07.468 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.467+0000 7f5d2ab3f640 1 Processor -- start 2024-06-11T19:22:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.467+0000 7f5d2ab3f640 1 -- start start 2024-06-11T19:22:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.467+0000 7f5d2ab3f640 10 monclient: init 2024-06-11T19:22:07.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.467+0000 7f5d2ab3f640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:07.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.467+0000 7f5d2ab3f640 10 monclient: _add_conns ranks=[4,9,5,0,7,3,1,2,8,6,10] 2024-06-11T19:22:07.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.467+0000 7f5d2ab3f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 0x7f5d24134de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.467+0000 7f5d2ab3f640 10 monclient(hunting): picked mon.noname-c con 0x7f5d24134a00 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d2ab3f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d24135630 0x7f5d24135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d2ab3f640 10 monclient(hunting): picked mon.noname-a con 0x7f5d24135630 addr v2:172.21.15.98:3300/0 2024-06-11T19:22:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d2ab3f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d24135f90 0x7f5d2413c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d2ab3f640 10 monclient(hunting): picked mon.noname-h con 0x7f5d24135f90 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d2ab3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d2ab3f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5d240a3560 con 0x7f5d24135630 2024-06-11T19:22:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d2ab3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d2ab3f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5d24136440 con 0x7f5d24134a00 2024-06-11T19:22:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d2ab3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d2ab3f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f5d2409f9b0 con 0x7f5d24135f90 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d2ab3f640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d2ab3f640 10 monclient(hunting): authenticate will time out at 1537.275039s 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d24135630 0x7f5d24135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d288db640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 0x7f5d24134de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d288db640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 0x7f5d24134de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:37964/0 (socket says 172.21.15.98:37964) 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d24135630 0x7f5d24135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52190/0 (socket says 172.21.15.98:52190) 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d288db640 1 -- 172.21.15.98:0/972055193 learned_addr learned my addr 172.21.15.98:0/972055193 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d288db640 10 monclient(hunting): get_auth_request con 0x7f5d24134a00 auth_method 0 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d288db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d288db640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d288db640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 10 monclient(hunting): get_auth_request con 0x7f5d24135630 auth_method 0 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d288db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d288db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d288db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 10 monclient(hunting): handle_auth_done global_id 65601 payload 274 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 1 -- 172.21.15.98:0/972055193 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d24135f90 msgr2=0x7f5d2413c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d290dc640 1 --2- 172.21.15.98:0/972055193 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d24135f90 0x7f5d2413c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 1 --2- 172.21.15.98:0/972055193 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d24135f90 0x7f5d2413c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 1 -- 172.21.15.98:0/972055193 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 msgr2=0x7f5d24134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 1 --2- 172.21.15.98:0/972055193 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 0x7f5d24134de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 1 monclient: found mon.noname-a 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d23fff640 1 -- 172.21.15.98:0/972055193 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2409fc90 con 0x7f5d24135630 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.468+0000 7f5d288db640 1 --2- 172.21.15.98:0/972055193 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 0x7f5d24134de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d23fff640 1 --2- 172.21.15.98:0/972055193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d24135630 0x7f5d24135a50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5d180097a0 tx=0x7f5d1802b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78b460053184db3 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 1 -- 172.21.15.98:0/972055193 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d18037070 con 0x7f5d24135630 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 10 monclient: dump: 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.482 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.482 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.482 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.482 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.482 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.482 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.482 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.483 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.483 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.483 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:07.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:07.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:07.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 1 -- 172.21.15.98:0/972055193 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d18030980 con 0x7f5d24135630 2024-06-11T19:22:07.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 5 monclient: authenticate success, global_id 65601 2024-06-11T19:22:07.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:07.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:07.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:07.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 1 -- 172.21.15.98:0/972055193 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d18030bb0 con 0x7f5d24135630 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 4 set_mon_vals no callback set 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d237fe640 10 monclient: dump: 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.485 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 10 monclient: shutdown 2024-06-11T19:22:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/972055193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d24135630 msgr2=0x7f5d24135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 1 --2- 172.21.15.98:0/972055193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d24135630 0x7f5d24135a50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5d180097a0 tx=0x7f5d1802b6f0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/972055193 shutdown_connections 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 1 --2- 172.21.15.98:0/972055193 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d24135f90 0x7f5d2413c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 1 --2- 172.21.15.98:0/972055193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d24135630 0x7f5d24135a50 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 1 --2- 172.21.15.98:0/972055193 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 0x7f5d24134de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/972055193 >> 172.21.15.98:0/972055193 conn(0x7f5d240a73b0 msgr2=0x7f5d240a3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/972055193 shutdown_connections 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.469+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/972055193 wait complete. 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.470+0000 7f5d2ab3f640 10 monclient: build_initial_monmap 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.470+0000 7f5d2ab3f640 10 monclient: monmap: 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.470+0000 7f5d2ab3f640 1 librados: starting msgr at 2024-06-11T19:22:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.470+0000 7f5d2ab3f640 1 librados: starting objecter 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.470+0000 7f5d2ab3f640 1 Processor -- start 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.470+0000 7f5d2ab3f640 1 -- start start 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.470+0000 7f5d2ab3f640 1 librados: setting wanted keys 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.470+0000 7f5d2ab3f640 1 librados: calling monclient init 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.470+0000 7f5d2ab3f640 10 monclient: init 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 10 monclient: _add_conns ranks=[4,3,5,2,0,8,1,6,7] 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 0x7f5d2411fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 10 monclient(hunting): picked mon.noname-c con 0x7f5d24134a00 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d24135630 0x7f5d24120330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 10 monclient(hunting): picked mon.noname-g con 0x7f5d24135630 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d24135f90 0x7f5d24120870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 10 monclient(hunting): picked mon.noname-h con 0x7f5d24135f90 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f5d24135320 con 0x7f5d24135630 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5d240a2ef0 con 0x7f5d24134a00 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f5d240a2b90 con 0x7f5d24135f90 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d2ab3f640 10 monclient(hunting): authenticate will time out at 1537.278544s 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 0x7f5d2411fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 0x7f5d2411fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:37976/0 (socket says 172.21.15.98:37976) 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 1 -- 172.21.15.98:0/1545427379 learned_addr learned my addr 172.21.15.98:0/1545427379 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 10 monclient(hunting): get_auth_request con 0x7f5d24134a00 auth_method 0 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 10 monclient(hunting): handle_auth_done global_id 114124 payload 824 2024-06-11T19:22:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 1 -- 172.21.15.98:0/1545427379 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d24135f90 msgr2=0x7f5d24120870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d24135f90 0x7f5d24120870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 1 -- 172.21.15.98:0/1545427379 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d24135630 msgr2=0x7f5d24120330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d23fff640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d24135630 0x7f5d24120330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d24135630 0x7f5d24120330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 10 monclient: _finish_hunting 0 2024-06-11T19:22:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 1 monclient: found mon.noname-c 2024-06-11T19:22:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:22:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.471+0000 7f5d288db640 1 -- 172.21.15.98:0/1545427379 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d24122740 con 0x7f5d24134a00 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d288db640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 0x7f5d2411fdf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009e80 tx=0x7f5d1c00c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 10 client.?.objecter ms_handle_connect 0x7f5d24134a00 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d1c00cc60 con 0x7f5d24134a00 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 10 monclient: dump: 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 10 monclient: _finish_auth 0 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 10 monclient: _check_auth_tickets 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d1c019070 con 0x7f5d24134a00 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d2ab3f640 5 monclient: authenticate success, global_id 114124 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d2ab3f640 10 monclient: _renew_subs 2024-06-11T19:22:07.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d1c00ce00 con 0x7f5d24134a00 2024-06-11T19:22:07.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d2ab3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:07.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/1545427379 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d24122a20 con 0x7f5d24134a00 2024-06-11T19:22:07.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:07.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d217fa640 10 monclient: dump: 2024-06-11T19:22:07.493 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.493 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d22ffd640 4 set_mon_vals no callback set 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d2ab3f640 10 client.114124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d2ab3f640 10 monclient: _renew_subs 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d2ab3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/1545427379 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d240a2440 con 0x7f5d24134a00 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d2ab3f640 1 librados: init done 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.472+0000 7f5d2ab3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d2ab3f640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cb890 0x7f5d241cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d2ab3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/1545427379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d24050830 con 0x7f5d241cb890 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f5d1c005d10 con 0x7f5d24134a00 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d23fff640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cb890 0x7f5d241cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d23fff640 10 monclient: get_auth_request con 0x7f5d241cb890 auth_method 0 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d23fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d1003fdb0 0x7f5d10042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d23fff640 10 monclient(hunting): handle_auth_done global_id 65610 payload 824 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d23fff640 1 -- 172.21.15.98:0/1545427379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d24068e20 con 0x7f5d241cb890 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(512..512 src has 1..512) ==== 6677+0+0 (secure 0 0 0) 0x7f5d1c051dc0 con 0x7f5d24134a00 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 10 client.114124.objecter ms_dispatch 0x7f5d24000b90 osd_map(512..512 src has 1..512) 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 3 client.114124.objecter handle_osd_map got epochs [512,512] > 0 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 3 client.114124.objecter handle_osd_map decoding full epoch 512 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d23fff640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cb890 0x7f5d241cbc70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5d2409dcc0 tx=0x7f5d180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 20 client.114124.objecter dump_active .. 0 homeless 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 10 client.114124.objecter ms_handle_connect 0x7f5d241cb890 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 10 client.114124.objecter resend_mon_ops 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d290dc640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d1003fdb0 0x7f5d10042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d290dc640 10 monclient: get_auth_request con 0x7f5d1003fdb0 auth_method 0 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d18042070 con 0x7f5d241cb890 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 10 monclient: dump: 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.497 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5d18030eb0 con 0x7f5d241cb890 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.473+0000 7f5d217fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.474+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cb890 msgr2=0x7f5d241cbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.474+0000 7f5d217fa640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cb890 0x7f5d241cbc70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5d2409dcc0 tx=0x7f5d180046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.474+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cb890 msgr2=0x7f5d241cbc70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:07.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.474+0000 7f5d217fa640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cb890 0x7f5d241cbc70 secure :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5d2409dcc0 tx=0x7f5d180046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.474+0000 7f5d290dc640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d1003fdb0 0x7f5d10042270 secure :-1 s=READY pgs=4240 cs=0 l=1 rev1=1 crypto rx=0x7f5d240a1ed0 tx=0x7f5d14008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.474+0000 7f5d217fa640 10 client.114124.objecter ms_handle_connect 0x7f5d1003fdb0 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d2ab3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d2ab3f640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cc1b0 0x7f5d240703c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d2ab3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/1545427379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d240a10e0 con 0x7f5d241cc1b0 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d23fff640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cc1b0 0x7f5d240703c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d23fff640 10 monclient: get_auth_request con 0x7f5d241cc1b0 auth_method 0 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d23fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d23fff640 10 monclient(hunting): handle_auth_done global_id 65619 payload 824 2024-06-11T19:22:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.480+0000 7f5d23fff640 1 -- 172.21.15.98:0/1545427379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5d24133410 con 0x7f5d241cc1b0 2024-06-11T19:22:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d23fff640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cc1b0 0x7f5d240703c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5d180097a0 tx=0x7f5d18030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 10 client.114124.objecter ms_handle_connect 0x7f5d241cc1b0 2024-06-11T19:22:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 10 client.114124.objecter resend_mon_ops 2024-06-11T19:22:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d18042040 con 0x7f5d241cc1b0 2024-06-11T19:22:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 10 monclient: dump: 2024-06-11T19:22:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.502 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f5d18003610 con 0x7f5d241cc1b0 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:07.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:07.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cc1b0 msgr2=0x7f5d240703c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cc1b0 0x7f5d240703c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5d180097a0 tx=0x7f5d18030000 comp rx=0 tx=0).stop 2024-06-11T19:22:07.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 1 -- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cc1b0 msgr2=0x7f5d240703c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:07.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d217fa640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d241cc1b0 0x7f5d240703c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.504 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 519309, 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:07.528 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:07.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 10 librados: watch_flush enter 2024-06-11T19:22:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 10 librados: watch_flush exit 2024-06-11T19:22:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 20 client.114124.objecter shutdown clearing up homeless session... 2024-06-11T19:22:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 10 client.114124.objecter successfully canceled tick 2024-06-11T19:22:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/1545427379 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d1003fdb0 msgr2=0x7f5d10042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d1003fdb0 0x7f5d10042270 secure :-1 s=READY pgs=4240 cs=0 l=1 rev1=1 crypto rx=0x7f5d240a1ed0 tx=0x7f5d14008040 comp rx=0 tx=0).stop 2024-06-11T19:22:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 10 monclient: shutdown 2024-06-11T19:22:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 msgr2=0x7f5d2411fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 1 --2- 172.21.15.98:0/1545427379 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d24134a00 0x7f5d2411fdf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009e80 tx=0x7f5d1c00c490 comp rx=0 tx=0).stop 2024-06-11T19:22:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d290dc640 1 -- 172.21.15.98:0/1545427379 reap_dead start 2024-06-11T19:22:07.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d290dc640 1 -- 172.21.15.98:0/1545427379 reap_dead start 2024-06-11T19:22:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/1545427379 shutdown_connections 2024-06-11T19:22:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/1545427379 >> 172.21.15.98:0/1545427379 conn(0x7f5d240a73b0 msgr2=0x7f5d241bb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/1545427379 shutdown_connections 2024-06-11T19:22:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 1 -- 172.21.15.98:0/1545427379 wait complete. 2024-06-11T19:22:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.481+0000 7f5d2ab3f640 1 librados: shutdown 2024-06-11T19:22:07.531 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:07.531 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:22:07.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:22:07.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.631+0000 7fcb687f3640 10 monclient: get_monmap_and_config 2024-06-11T19:22:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.631+0000 7fcb687f3640 10 monclient: build_initial_monmap 2024-06-11T19:22:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.631+0000 7fcb687f3640 10 monclient: monmap: 2024-06-11T19:22:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:07.632293+0000 2024-06-11T19:22:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:07.632293+0000 2024-06-11T19:22:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:07.632 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.631+0000 7fcb687f3640 1 Processor -- start 2024-06-11T19:22:07.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.631+0000 7fcb687f3640 1 -- start start 2024-06-11T19:22:07.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.631+0000 7fcb687f3640 10 monclient: init 2024-06-11T19:22:07.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:07.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 10 monclient: _add_conns ranks=[9,8,10,1,4,3,0,6,7,5,2] 2024-06-11T19:22:07.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb600a6260 0x7fcb600a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 10 monclient(hunting): picked mon.noname-a con 0x7fcb600a6260 addr v2:172.21.15.98:3300/0 2024-06-11T19:22:07.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 0x7fcb600a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 10 monclient(hunting): picked mon.noname-e con 0x7fcb6006c890 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:07.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fcb600a2d00 0x7fcb600a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 10 monclient(hunting): picked mon.noname-f con 0x7fcb600a2d00 addr v2:172.21.15.117:3300/0 2024-06-11T19:22:07.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcb600a05b0 con 0x7fcb600a6260 2024-06-11T19:22:07.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fcb600a6640 con 0x7fcb600a2d00 2024-06-11T19:22:07.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcb6006c650 con 0x7fcb6006c890 2024-06-11T19:22:07.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:07.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb687f3640 10 monclient(hunting): authenticate will time out at 1537.439557s 2024-06-11T19:22:07.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 0x7fcb600a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb6658f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb600a6260 0x7fcb600a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb6658f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb600a6260 0x7fcb600a1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52204/0 (socket says 172.21.15.98:52204) 2024-06-11T19:22:07.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb6658f640 1 -- 172.21.15.98:0/796830241 learned_addr learned my addr 172.21.15.98:0/796830241 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:07.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 0x7fcb600a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:58032/0 (socket says 172.21.15.98:58032) 2024-06-11T19:22:07.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb6658f640 10 monclient(hunting): get_auth_request con 0x7fcb600a6260 auth_method 0 2024-06-11T19:22:07.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb6658f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb6658f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb6658f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 10 monclient(hunting): get_auth_request con 0x7fcb6006c890 auth_method 0 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb6658f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb6658f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb6658f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb66d90640 1 --2- 172.21.15.98:0/796830241 >> v2:172.21.15.117:3300/0 conn(0x7fcb600a2d00 0x7fcb600a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb66d90640 10 monclient(hunting): get_auth_request con 0x7fcb600a2d00 auth_method 0 2024-06-11T19:22:07.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb66d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb66d90640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb66d90640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 10 monclient(hunting): handle_auth_done global_id 114110 payload 274 2024-06-11T19:22:07.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 1 -- 172.21.15.98:0/796830241 >> v2:172.21.15.117:3300/0 conn(0x7fcb600a2d00 msgr2=0x7fcb600a4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:07.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 1 --2- 172.21.15.98:0/796830241 >> v2:172.21.15.117:3300/0 conn(0x7fcb600a2d00 0x7fcb600a4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 1 -- 172.21.15.98:0/796830241 >> v2:172.21.15.98:3300/0 conn(0x7fcb600a6260 msgr2=0x7fcb600a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:07.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 1 --2- 172.21.15.98:0/796830241 >> v2:172.21.15.98:3300/0 conn(0x7fcb600a6260 0x7fcb600a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 10 monclient: _finish_hunting 0 2024-06-11T19:22:07.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 1 monclient: found mon.noname-e 2024-06-11T19:22:07.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:07.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.632+0000 7fcb65d8e640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb65d8e640 1 -- 172.21.15.98:0/796830241 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb60073b60 con 0x7fcb6006c890 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6658f640 1 --2- 172.21.15.98:0/796830241 >> v2:172.21.15.98:3300/0 conn(0x7fcb600a6260 0x7fcb600a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb66d90640 1 --2- 172.21.15.98:0/796830241 >> v2:172.21.15.117:3300/0 conn(0x7fcb600a2d00 0x7fcb600a4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb65d8e640 1 --2- 172.21.15.98:0/796830241 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 0x7fcb600a48b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcb600a3540 tx=0x7fcb5c02b710 comp rx=0 tx=0).ready entity=mon.8 client_cookie=33bb75f2618be7f6 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 1 -- 172.21.15.98:0/796830241 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcb5c037070 con 0x7fcb6006c890 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 10 monclient: dump: 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.647 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 10 monclient: _finish_auth 0 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 10 monclient: _check_auth_tickets 2024-06-11T19:22:07.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:07.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 1 -- 172.21.15.98:0/796830241 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5c030980 con 0x7fcb6006c890 2024-06-11T19:22:07.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb687f3640 5 monclient: authenticate success, global_id 114110 2024-06-11T19:22:07.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb687f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:07.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:07.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 1 -- 172.21.15.98:0/796830241 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcb5c0313a0 con 0x7fcb6006c890 2024-06-11T19:22:07.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb687f3640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:07.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb687f3640 4 set_mon_vals no callback set 2024-06-11T19:22:07.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:07.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.633+0000 7fcb6558d640 10 monclient: dump: 2024-06-11T19:22:07.649 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.650 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.650 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.650 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.650 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.650 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.650 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.650 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.650 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.650 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.650 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.634+0000 7fcb687f3640 10 monclient: shutdown 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.634+0000 7fcb687f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.634+0000 7fcb687f3640 1 -- 172.21.15.98:0/796830241 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 msgr2=0x7fcb600a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.634+0000 7fcb687f3640 1 --2- 172.21.15.98:0/796830241 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 0x7fcb600a48b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcb600a3540 tx=0x7fcb5c02b710 comp rx=0 tx=0).stop 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.634+0000 7fcb687f3640 1 -- 172.21.15.98:0/796830241 shutdown_connections 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.634+0000 7fcb687f3640 1 --2- 172.21.15.98:0/796830241 >> v2:172.21.15.117:3300/0 conn(0x7fcb600a2d00 0x7fcb600a4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.634+0000 7fcb687f3640 1 --2- 172.21.15.98:0/796830241 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 0x7fcb600a48b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.634+0000 7fcb687f3640 1 --2- 172.21.15.98:0/796830241 >> v2:172.21.15.98:3300/0 conn(0x7fcb600a6260 0x7fcb600a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.634+0000 7fcb687f3640 1 -- 172.21.15.98:0/796830241 >> 172.21.15.98:0/796830241 conn(0x7fcb600a95b0 msgr2=0x7fcb600a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.634+0000 7fcb687f3640 1 -- 172.21.15.98:0/796830241 shutdown_connections 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.634+0000 7fcb687f3640 1 -- 172.21.15.98:0/796830241 wait complete. 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.635+0000 7fcb687f3640 10 monclient: build_initial_monmap 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.635+0000 7fcb687f3640 10 monclient: monmap: 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:07.652 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.635+0000 7fcb687f3640 1 librados: starting msgr at 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.635+0000 7fcb687f3640 1 librados: starting objecter 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.635+0000 7fcb687f3640 1 Processor -- start 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.635+0000 7fcb687f3640 1 -- start start 2024-06-11T19:22:07.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.635+0000 7fcb687f3640 1 librados: setting wanted keys 2024-06-11T19:22:07.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.635+0000 7fcb687f3640 1 librados: calling monclient init 2024-06-11T19:22:07.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.635+0000 7fcb687f3640 10 monclient: init 2024-06-11T19:22:07.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.635+0000 7fcb687f3640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:07.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.635+0000 7fcb687f3640 10 monclient: _add_conns ranks=[8,5,4,1,2,3,7,0,6] 2024-06-11T19:22:07.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 0x7fcb601c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 monclient(hunting): picked mon.noname-e con 0x7fcb6006c890 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:07.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcb600a2d00 0x7fcb601c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 monclient(hunting): picked mon.noname-h con 0x7fcb600a2d00 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:07.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcb600a6260 0x7fcb601cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 monclient(hunting): picked mon.noname-c con 0x7fcb600a6260 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:07.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fcb600a2410 con 0x7fcb600a6260 2024-06-11T19:22:07.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fcb600a2290 con 0x7fcb600a2d00 2024-06-11T19:22:07.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcb600a5330 con 0x7fcb6006c890 2024-06-11T19:22:07.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:07.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 monclient(hunting): authenticate will time out at 1537.443080s 2024-06-11T19:22:07.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 0x7fcb601c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb66d90640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcb600a6260 0x7fcb601cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 0x7fcb601c9f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:58038/0 (socket says 172.21.15.98:58038) 2024-06-11T19:22:07.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 1 -- 172.21.15.98:0/2496840130 learned_addr learned my addr 172.21.15.98:0/2496840130 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:07.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 10 monclient(hunting): get_auth_request con 0x7fcb6006c890 auth_method 0 2024-06-11T19:22:07.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb66d90640 10 monclient(hunting): get_auth_request con 0x7fcb600a6260 auth_method 0 2024-06-11T19:22:07.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb66d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb66d90640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb66d90640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb66d90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb66d90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb66d90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb65d8e640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcb600a2d00 0x7fcb601c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb65d8e640 10 monclient(hunting): get_auth_request con 0x7fcb600a2d00 auth_method 0 2024-06-11T19:22:07.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb65d8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb65d8e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb65d8e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 10 monclient(hunting): handle_auth_done global_id 114119 payload 824 2024-06-11T19:22:07.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 1 -- 172.21.15.98:0/2496840130 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcb600a2d00 msgr2=0x7fcb601c74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcb600a2d00 0x7fcb601c74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 1 -- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcb600a6260 msgr2=0x7fcb601cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcb600a6260 0x7fcb601cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 10 monclient: _finish_hunting 0 2024-06-11T19:22:07.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 1 monclient: found mon.noname-e 2024-06-11T19:22:07.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:07.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:22:07.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 1 -- 172.21.15.98:0/2496840130 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb601ac350 con 0x7fcb6006c890 2024-06-11T19:22:07.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb6658f640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 0x7fcb601c9f20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400e9b0 tx=0x7fcb5400ee70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 10 client.?.objecter ms_handle_connect 0x7fcb6006c890 2024-06-11T19:22:07.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:07.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcb54017070 con 0x7fcb6006c890 2024-06-11T19:22:07.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:22:07.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 10 monclient: dump: 2024-06-11T19:22:07.658 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.659 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.659 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.659 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.659 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.659 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.659 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.659 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.659 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.659 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.659 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.659 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 10 monclient: _finish_auth 0 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5400ce10 con 0x7fcb6006c890 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 5 monclient: authenticate success, global_id 114119 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 monclient: _renew_subs 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 1 -- 172.21.15.98:0/2496840130 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb601aee10 con 0x7fcb6006c890 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcb540138e0 con 0x7fcb6006c890 2024-06-11T19:22:07.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb52ffd640 10 monclient: dump: 2024-06-11T19:22:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.661 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.662 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.662 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.662 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.662 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.662 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.662 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb64d8c640 4 set_mon_vals no callback set 2024-06-11T19:22:07.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 client.114119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:07.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 monclient: _renew_subs 2024-06-11T19:22:07.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:22:07.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 1 -- 172.21.15.98:0/2496840130 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb600a6da0 con 0x7fcb6006c890 2024-06-11T19:22:07.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.636+0000 7fcb687f3640 1 librados: init done 2024-06-11T19:22:07.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb687f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:07.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb687f3640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601cd9a0 0x7fcb601cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb687f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb687f3640 1 -- 172.21.15.98:0/2496840130 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcb60050830 con 0x7fcb601cd9a0 2024-06-11T19:22:07.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb65d8e640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601cd9a0 0x7fcb601cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fcb5401a070 con 0x7fcb6006c890 2024-06-11T19:22:07.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb65d8e640 10 monclient: get_auth_request con 0x7fcb601cd9a0 auth_method 0 2024-06-11T19:22:07.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb65d8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb65d8e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb65d8e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb65d8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb65d8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb65d8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb52ffd640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcb4403fdb0 0x7fcb44042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb65d8e640 10 monclient(hunting): handle_auth_done global_id 84881 payload 824 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb65d8e640 1 -- 172.21.15.98:0/2496840130 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb600a8ba0 con 0x7fcb601cd9a0 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.637+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(512..512 src has 1..512) ==== 6677+0+0 (secure 0 0 0) 0x7fcb5400f040 con 0x7fcb6006c890 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 10 client.114119.objecter ms_dispatch 0x7fcb60000b90 osd_map(512..512 src has 1..512) 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 3 client.114119.objecter handle_osd_map got epochs [512,512] > 0 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 3 client.114119.objecter handle_osd_map decoding full epoch 512 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb65d8e640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601cd9a0 0x7fcb601cdd80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb6006c400 tx=0x7fcb5c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 20 client.114119.objecter dump_active .. 0 homeless 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 10 client.114119.objecter ms_handle_connect 0x7fcb601cd9a0 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 10 client.114119.objecter resend_mon_ops 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb66d90640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcb4403fdb0 0x7fcb44042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb66d90640 10 monclient: get_auth_request con 0x7fcb4403fdb0 auth_method 0 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcb5c037040 con 0x7fcb601cd9a0 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 10 monclient: dump: 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcb5c003ba0 con 0x7fcb601cd9a0 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:07.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:07.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601cd9a0 msgr2=0x7fcb601cdd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601cd9a0 0x7fcb601cdd80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb6006c400 tx=0x7fcb5c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601cd9a0 msgr2=0x7fcb601cdd80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:07.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb601cd9a0 0x7fcb601cdd80 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb6006c400 tx=0x7fcb5c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb52ffd640 10 client.114119.objecter ms_handle_connect 0x7fcb4403fdb0 2024-06-11T19:22:07.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.638+0000 7fcb66d90640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcb4403fdb0 0x7fcb44042270 secure :-1 s=READY pgs=4241 cs=0 l=1 rev1=1 crypto rx=0x7fcb601ad430 tx=0x7fcb58009850 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.644+0000 7fcb687f3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:07.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.644+0000 7fcb687f3640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb600a37a0 0x7fcb600a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.644+0000 7fcb687f3640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.644+0000 7fcb687f3640 1 -- 172.21.15.98:0/2496840130 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcb601ae700 con 0x7fcb600a37a0 2024-06-11T19:22:07.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb65d8e640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb600a37a0 0x7fcb600a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb65d8e640 10 monclient: get_auth_request con 0x7fcb600a37a0 auth_method 0 2024-06-11T19:22:07.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb65d8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb65d8e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb65d8e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb65d8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb65d8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb65d8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb65d8e640 10 monclient(hunting): handle_auth_done global_id 84890 payload 824 2024-06-11T19:22:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb65d8e640 1 -- 172.21.15.98:0/2496840130 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb600a1d50 con 0x7fcb600a37a0 2024-06-11T19:22:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb65d8e640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb600a37a0 0x7fcb600a3bb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c02b750 tx=0x7fcb5c008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 10 client.114119.objecter ms_handle_connect 0x7fcb600a37a0 2024-06-11T19:22:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 10 client.114119.objecter resend_mon_ops 2024-06-11T19:22:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcb5c047c60 con 0x7fcb600a37a0 2024-06-11T19:22:07.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 10 monclient: dump: 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.671 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.672 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fcb5c047030 con 0x7fcb600a37a0 2024-06-11T19:22:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb600a37a0 msgr2=0x7fcb600a3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb600a37a0 0x7fcb600a3bb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c02b750 tx=0x7fcb5c008030 comp rx=0 tx=0).stop 2024-06-11T19:22:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 1 -- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb600a37a0 msgr2=0x7fcb600a3bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:07.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.645+0000 7fcb52ffd640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcb600a37a0 0x7fcb600a3bb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 399419, 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:07.673 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:07.674 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:07.674 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:07.674 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:07.674 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.674 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:22:07.674 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:07.674 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:07.674 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:07.674 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:07.674 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:07.674 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:07.677 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.678 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:07.679 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:07.681 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:07.682 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.682 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.682 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.682 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:07.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:07.682 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.684 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:07.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:07.684 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:07.684 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.685 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:07.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:07.686 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:07.686 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:07.687 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:07.690 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:07.690 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.690 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:07.690 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:07.691 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.692 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:07.694 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:07.694 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:07.694 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.694 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.694 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.695 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.696 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:07.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.696 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:07.696 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:07.696 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:07.696 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.696 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.696 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:07.696 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:07.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 10 librados: watch_flush enter 2024-06-11T19:22:07.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 10 librados: watch_flush exit 2024-06-11T19:22:07.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 20 client.114119.objecter shutdown clearing up homeless session... 2024-06-11T19:22:07.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 10 client.114119.objecter successfully canceled tick 2024-06-11T19:22:07.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 1 -- 172.21.15.98:0/2496840130 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcb4403fdb0 msgr2=0x7fcb44042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcb4403fdb0 0x7fcb44042270 secure :-1 s=READY pgs=4241 cs=0 l=1 rev1=1 crypto rx=0x7fcb601ad430 tx=0x7fcb58009850 comp rx=0 tx=0).stop 2024-06-11T19:22:07.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 10 monclient: shutdown 2024-06-11T19:22:07.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:07.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 1 -- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 msgr2=0x7fcb601c9f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 1 --2- 172.21.15.98:0/2496840130 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcb6006c890 0x7fcb601c9f20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400e9b0 tx=0x7fcb5400ee70 comp rx=0 tx=0).stop 2024-06-11T19:22:07.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb66d90640 1 -- 172.21.15.98:0/2496840130 reap_dead start 2024-06-11T19:22:07.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb66d90640 1 -- 172.21.15.98:0/2496840130 reap_dead start 2024-06-11T19:22:07.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 1 -- 172.21.15.98:0/2496840130 shutdown_connections 2024-06-11T19:22:07.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 1 -- 172.21.15.98:0/2496840130 >> 172.21.15.98:0/2496840130 conn(0x7fcb600a95b0 msgr2=0x7fcb600705e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:07.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 1 -- 172.21.15.98:0/2496840130 shutdown_connections 2024-06-11T19:22:07.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 1 -- 172.21.15.98:0/2496840130 wait complete. 2024-06-11T19:22:07.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.646+0000 7fcb687f3640 1 librados: shutdown 2024-06-11T19:22:07.699 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:07.699 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:22:07.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:22:07.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.800+0000 7ff8c1ffc640 10 monclient: get_monmap_and_config 2024-06-11T19:22:07.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.800+0000 7ff8c1ffc640 10 monclient: build_initial_monmap 2024-06-11T19:22:07.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.800+0000 7ff8c1ffc640 10 monclient: monmap: 2024-06-11T19:22:07.799 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:07.799 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.799 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:07.801117+0000 2024-06-11T19:22:07.799 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:07.801117+0000 2024-06-11T19:22:07.799 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:07.799 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.799 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:07.799 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:07.800 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:07.800 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:07.800 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:07.800 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:07.800 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:07.800 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:07.800 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:07.800 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:07.800 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:07.800 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.800+0000 7ff8c1ffc640 1 Processor -- start 2024-06-11T19:22:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.800+0000 7ff8c1ffc640 1 -- start start 2024-06-11T19:22:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.800+0000 7ff8c1ffc640 10 monclient: init 2024-06-11T19:22:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 10 monclient: _add_conns ranks=[10,7,2,1,9,5,6,4,3,8,0] 2024-06-11T19:22:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff8bc0a0d10 0x7ff8bc0a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 10 monclient(hunting): picked mon.noname-f con 0x7ff8bc0a0d10 addr v2:172.21.15.117:3300/0 2024-06-11T19:22:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff8bc138c80 0x7ff8bc139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 10 monclient(hunting): picked mon.noname-i con 0x7ff8bc138c80 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8bc0a22d0 0x7ff8bc0a26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 10 monclient(hunting): picked mon.noname-b con 0x7ff8bc0a22d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7ff8bc09f010 con 0x7ff8bc0a0d10 2024-06-11T19:22:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff8bc076a00 con 0x7ff8bc0a22d0 2024-06-11T19:22:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff8bc09ed30 con 0x7ff8bc138c80 2024-06-11T19:22:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8c1ffc640 10 monclient(hunting): authenticate will time out at 1537.608330s 2024-06-11T19:22:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8bc0a22d0 0x7ff8bc0a26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8bc0a22d0 0x7ff8bc0a26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57150/0 (socket says 172.21.15.98:57150) 2024-06-11T19:22:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 1 -- 172.21.15.98:0/413207145 learned_addr learned my addr 172.21.15.98:0/413207145 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 10 monclient(hunting): get_auth_request con 0x7ff8bc0a22d0 auth_method 0 2024-06-11T19:22:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8baffd640 1 --2- 172.21.15.98:0/413207145 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff8bc138c80 0x7ff8bc139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bb7fe640 1 --2- 172.21.15.98:0/413207145 >> v2:172.21.15.117:3300/0 conn(0x7ff8bc0a0d10 0x7ff8bc0a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_done global_id 84899 payload 274 2024-06-11T19:22:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 1 -- 172.21.15.98:0/413207145 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff8bc138c80 msgr2=0x7ff8bc139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 1 --2- 172.21.15.98:0/413207145 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff8bc138c80 0x7ff8bc139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 1 -- 172.21.15.98:0/413207145 >> v2:172.21.15.117:3300/0 conn(0x7ff8bc0a0d10 msgr2=0x7ff8bc0a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 1 --2- 172.21.15.98:0/413207145 >> v2:172.21.15.117:3300/0 conn(0x7ff8bc0a0d10 0x7ff8bc0a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 1 monclient: found mon.noname-b 2024-06-11T19:22:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bbfff640 1 -- 172.21.15.98:0/413207145 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8bc0a1630 con 0x7ff8bc0a22d0 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8baffd640 10 monclient: get_auth_request con 0x7ff8bc138c80 auth_method 0 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bb7fe640 10 monclient: get_auth_request con 0x7ff8bc0a0d10 auth_method 0 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8baffd640 1 --2- 172.21.15.98:0/413207145 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff8bc138c80 0x7ff8bc139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.801+0000 7ff8bb7fe640 1 --2- 172.21.15.98:0/413207145 >> v2:172.21.15.117:3300/0 conn(0x7ff8bc0a0d10 0x7ff8bc0a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8bbfff640 1 --2- 172.21.15.98:0/413207145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8bc0a22d0 0x7ff8bc0a26d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40098a0 tx=0x7ff8b402b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1131fc2c8c7dfda7 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 1 -- 172.21.15.98:0/413207145 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff8b4038070 con 0x7ff8bc0a22d0 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 10 monclient: dump: 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.820 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.822 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.822 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 10 monclient: _finish_auth 0 2024-06-11T19:22:07.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:22:07.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:07.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8c1ffc640 5 monclient: authenticate success, global_id 84899 2024-06-11T19:22:07.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 1 -- 172.21.15.98:0/413207145 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8b4031980 con 0x7ff8bc0a22d0 2024-06-11T19:22:07.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8c1ffc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:07.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:07.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 1 -- 172.21.15.98:0/413207145 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff8b4031bb0 con 0x7ff8bc0a22d0 2024-06-11T19:22:07.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8ba7fc640 10 monclient: dump: 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8c1ffc640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.802+0000 7ff8c1ffc640 4 set_mon_vals no callback set 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 10 monclient: shutdown 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/413207145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8bc0a22d0 msgr2=0x7ff8bc0a26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 1 --2- 172.21.15.98:0/413207145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8bc0a22d0 0x7ff8bc0a26d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40098a0 tx=0x7ff8b402b7b0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/413207145 shutdown_connections 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 1 --2- 172.21.15.98:0/413207145 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff8bc0a22d0 0x7ff8bc0a26d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 1 --2- 172.21.15.98:0/413207145 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff8bc138c80 0x7ff8bc139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 1 --2- 172.21.15.98:0/413207145 >> v2:172.21.15.117:3300/0 conn(0x7ff8bc0a0d10 0x7ff8bc0a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/413207145 >> 172.21.15.98:0/413207145 conn(0x7ff8bc0a7850 msgr2=0x7ff8bc0a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/413207145 shutdown_connections 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/413207145 wait complete. 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 10 monclient: build_initial_monmap 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.803+0000 7ff8c1ffc640 10 monclient: monmap: 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:07.825 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 1 librados: starting msgr at 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 1 librados: starting objecter 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 1 Processor -- start 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 1 -- start start 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 1 librados: setting wanted keys 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 1 librados: calling monclient init 2024-06-11T19:22:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 10 monclient: init 2024-06-11T19:22:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 10 monclient: _add_conns ranks=[7,5,6,8,4,3,2,1,0] 2024-06-11T19:22:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff8bc0a0d10 0x7ff8bc1cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 10 monclient(hunting): picked mon.noname-i con 0x7ff8bc0a0d10 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff8bc0a22d0 0x7ff8bc1ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 10 monclient(hunting): picked mon.noname-h con 0x7ff8bc0a22d0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff8bc138c80 0x7ff8bc12ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 10 monclient(hunting): picked mon.noname-d con 0x7ff8bc138c80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ff8bc0a2c10 con 0x7ff8bc0a22d0 2024-06-11T19:22:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff8bc0a3d70 con 0x7ff8bc138c80 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff8bc0a2d90 con 0x7ff8bc0a0d10 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8c1ffc640 10 monclient(hunting): authenticate will time out at 1537.611828s 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8bbfff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff8bc138c80 0x7ff8bc12ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8bbfff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff8bc138c80 0x7ff8bc12ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34228/0 (socket says 172.21.15.98:34228) 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8bbfff640 1 -- 172.21.15.98:0/2427873972 learned_addr learned my addr 172.21.15.98:0/2427873972 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8bbfff640 10 monclient(hunting): get_auth_request con 0x7ff8bc138c80 auth_method 0 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8bbfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.804+0000 7ff8bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bb7fe640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff8bc0a0d10 0x7ff8bc1cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bb7fe640 10 monclient(hunting): get_auth_request con 0x7ff8bc0a0d10 auth_method 0 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8baffd640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff8bc0a22d0 0x7ff8bc1ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8baffd640 10 monclient(hunting): get_auth_request con 0x7ff8bc0a22d0 auth_method 0 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8baffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_done global_id 55527 payload 824 2024-06-11T19:22:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff8bc0a0d10 msgr2=0x7ff8bc1cdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff8bc0a0d10 0x7ff8bc1cdd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff8bc0a22d0 msgr2=0x7ff8bc1ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff8bc0a22d0 0x7ff8bc1ce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 1 monclient: found mon.noname-d 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 1 -- 172.21.15.98:0/2427873972 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8bc0a47c0 con 0x7ff8bc138c80 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8baffd640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff8bc0a22d0 0x7ff8bc1ce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff8bc138c80 0x7ff8bc12ee80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40090b0 tx=0x7ff8b40046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 client.?.objecter ms_handle_connect 0x7ff8bc138c80 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:07.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff8b4042020 con 0x7ff8bc138c80 2024-06-11T19:22:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:22:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 monclient: dump: 2024-06-11T19:22:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 monclient: _finish_auth 0 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 monclient: _check_auth_tickets 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8b40043f0 con 0x7ff8bc138c80 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8c1ffc640 5 monclient: authenticate success, global_id 55527 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8c1ffc640 10 monclient: _renew_subs 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8c1ffc640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/2427873972 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8bc132e40 con 0x7ff8bc138c80 2024-06-11T19:22:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff8b4038040 con 0x7ff8bc138c80 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 monclient: dump: 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8b9ffb640 4 set_mon_vals no callback set 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8c1ffc640 10 client.55527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8c1ffc640 10 monclient: _renew_subs 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8c1ffc640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/2427873972 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8bc1316b0 con 0x7ff8bc138c80 2024-06-11T19:22:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8c1ffc640 1 librados: init done 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff8bc138c80 msgr2=0x7ff8bc12ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff8bc138c80 msgr2=0x7ff8bc12ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff8bc138c80 0x7ff8bc12ee80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40090b0 tx=0x7ff8b40046c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8bbfff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff8bc138c80 0x7ff8bc12ee80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40090b0 tx=0x7ff8b40046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff8bc138c80 msgr2=0x7ff8bc12ee80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.805+0000 7ff8a3fff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff8bc138c80 0x7ff8bc12ee80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 10 monclient: _add_conns ranks=[4,5,3,7,0,2,8,1,6] 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8a80066e0 0x7ff8a8008ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 10 monclient(hunting): picked mon.c con 0x7ff8a80066e0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff8bc0a22d0 0x7ff8a800b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 10 monclient(hunting): picked mon.h con 0x7ff8bc0a22d0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a800bc00 0x7ff8a800e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 10 monclient(hunting): picked mon.g con 0x7ff8a800bc00 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a800eb40 con 0x7ff8a800bc00 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff8a800ecc0 con 0x7ff8a80066e0 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ff8a800ee70 con 0x7ff8bc0a22d0 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8a80066e0 0x7ff8a8008ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): get_auth_request con 0x7ff8a80066e0 auth_method 0 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8baffd640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff8bc0a22d0 0x7ff8a800b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:22:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bbfff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a800bc00 0x7ff8a800e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8baffd640 10 monclient(hunting): get_auth_request con 0x7ff8bc0a22d0 auth_method 0 2024-06-11T19:22:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8baffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8baffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bbfff640 10 monclient(hunting): get_auth_request con 0x7ff8a800bc00 auth_method 0 2024-06-11T19:22:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bbfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8c1ffc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8c1ffc640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce2c0 0x7ff8bc12f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8c1ffc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/2427873972 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff8bc050830 con 0x7ff8bc1ce2c0 2024-06-11T19:22:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8baffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:22:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:22:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_done global_id 55527 payload 293 2024-06-11T19:22:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff8bc0a22d0 msgr2=0x7ff8a800b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff8bc0a22d0 0x7ff8a800b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a800bc00 msgr2=0x7ff8a800e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a800bc00 0x7ff8a800e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 1 monclient: found mon.c 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 1 -- 172.21.15.98:0/2427873972 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7ff8a800f1f0 con 0x7ff8a80066e0 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce2c0 0x7ff8bc12f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient: get_auth_request con 0x7ff8bc1ce2c0 auth_method 0 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 10 client.55527.objecter ms_handle_connect 0x7ff8a80066e0 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8a80066e0 0x7ff8a8008ad0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00d610 tx=0x7ff8b4008030 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8a3fff640 10 client.55527.objecter resend_mon_ops 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_done global_id 114142 payload 824 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.806+0000 7ff8bb7fe640 1 -- 172.21.15.98:0/2427873972 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8bc0a26d0 con 0x7ff8bc1ce2c0 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.807+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff8ac007dd0 con 0x7ff8a80066e0 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.807+0000 7ff8a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.807+0000 7ff8a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.807+0000 7ff8a3fff640 10 monclient: dump: 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.841 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.842 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.807+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8ac00a870 con 0x7ff8a80066e0 2024-06-11T19:22:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.807+0000 7ff8a3fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.807+0000 7ff8a3fff640 10 client.55527.objecter ms_handle_connect 0x7ff8bc1ce2c0 2024-06-11T19:22:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.807+0000 7ff8a3fff640 10 client.55527.objecter resend_mon_ops 2024-06-11T19:22:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.807+0000 7ff8bb7fe640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce2c0 0x7ff8bc12f3c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8005690 tx=0x7ff8ac00f050 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.807+0000 7ff8b97fa640 4 set_mon_vals no callback set 2024-06-11T19:22:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.807+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff8ac005d10 con 0x7ff8a80066e0 2024-06-11T19:22:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.807+0000 7ff8a3fff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff8a8049200 0x7ff8a804b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 <== mon.4 v2:172.21.15.98:3302/0 4 ==== osd_map(513..513 src has 1..513) ==== 6690+0+0 (secure 0 0 0) 0x7ff8ac054180 con 0x7ff8a80066e0 2024-06-11T19:22:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 10 client.55527.objecter ms_dispatch 0x7ff8bc000b90 osd_map(513..513 src has 1..513) 2024-06-11T19:22:07.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 3 client.55527.objecter handle_osd_map got epochs [513,513] > 0 2024-06-11T19:22:07.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 3 client.55527.objecter handle_osd_map decoding full epoch 513 2024-06-11T19:22:07.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 20 client.55527.objecter dump_active .. 0 homeless 2024-06-11T19:22:07.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff8ac054980 con 0x7ff8bc1ce2c0 2024-06-11T19:22:07.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:07.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8baffd640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff8a8049200 0x7ff8a804b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 10 monclient: dump: 2024-06-11T19:22:07.844 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.844 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.844 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.844 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff8ac057520 con 0x7ff8bc1ce2c0 2024-06-11T19:22:07.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8baffd640 10 monclient: get_auth_request con 0x7ff8a8049200 auth_method 0 2024-06-11T19:22:07.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:07.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:07.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce2c0 msgr2=0x7ff8bc12f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce2c0 0x7ff8bc12f3c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8005690 tx=0x7ff8ac00f050 comp rx=0 tx=0).stop 2024-06-11T19:22:07.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce2c0 msgr2=0x7ff8bc12f3c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:07.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8bbfff640 1 -- 172.21.15.98:0/2427873972 reap_dead start 2024-06-11T19:22:07.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce2c0 0x7ff8bc12f3c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8a3fff640 10 client.55527.objecter ms_handle_connect 0x7ff8a8049200 2024-06-11T19:22:07.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.808+0000 7ff8baffd640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff8a8049200 0x7ff8a804b6c0 secure :-1 s=READY pgs=4242 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8005a80 tx=0x7ff8b0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8c1ffc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:07.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8c1ffc640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce6a0 0x7ff8bc12ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8c1ffc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/2427873972 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff8bc0749e0 con 0x7ff8bc1ce6a0 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8bbfff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce6a0 0x7ff8bc12ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8bbfff640 10 monclient: get_auth_request con 0x7ff8bc1ce6a0 auth_method 0 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8bbfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_done global_id 114151 payload 824 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.814+0000 7ff8bbfff640 1 -- 172.21.15.98:0/2427873972 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff8bc130b50 con 0x7ff8bc1ce6a0 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 10 client.55527.objecter ms_handle_connect 0x7ff8bc1ce6a0 2024-06-11T19:22:07.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 10 client.55527.objecter resend_mon_ops 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8bbfff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce6a0 0x7ff8bc12ee80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc1312b0 tx=0x7ff8b4031000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff8b4031980 con 0x7ff8bc1ce6a0 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 10 monclient: dump: 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.848 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7ff8b403e420 con 0x7ff8bc1ce6a0 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce6a0 msgr2=0x7ff8bc12ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce6a0 0x7ff8bc12ee80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc1312b0 tx=0x7ff8b4031000 comp rx=0 tx=0).stop 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce6a0 msgr2=0x7ff8bc12ee80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8a3fff640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce6a0 0x7ff8bc12ee80 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc1312b0 tx=0x7ff8b4031000 comp rx=0 tx=0).stop 2024-06-11T19:22:07.850 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:07.850 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:07.850 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:07.850 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:07.850 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:07.850 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 16181, 2024-06-11T19:22:07.850 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:07.851 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:07.851 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:07.851 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:07.851 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:07.851 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:07.851 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:07.851 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:07.851 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:07.851 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:07.851 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.851 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:22:07.851 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.861 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:07.862 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.863 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.863 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:07.863 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:07.863 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.863 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.863 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:07.864 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:07.866 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:07.866 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.866 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.866 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:07.868 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:07.871 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.872 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:07.873 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 10 librados: watch_flush enter 2024-06-11T19:22:07.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 10 librados: watch_flush exit 2024-06-11T19:22:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 20 client.55527.objecter shutdown clearing up homeless session... 2024-06-11T19:22:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 10 client.55527.objecter successfully canceled tick 2024-06-11T19:22:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff8a8049200 msgr2=0x7ff8a804b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff8a8049200 0x7ff8a804b6c0 secure :-1 s=READY pgs=4242 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8005a80 tx=0x7ff8b0008040 comp rx=0 tx=0).stop 2024-06-11T19:22:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 10 monclient: shutdown 2024-06-11T19:22:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8a80066e0 msgr2=0x7ff8a8008ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8a80066e0 0x7ff8a8008ad0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00d610 tx=0x7ff8b4008030 comp rx=0 tx=0).stop 2024-06-11T19:22:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/2427873972 shutdown_connections 2024-06-11T19:22:07.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff8a8049200 0x7ff8a804b6c0 unknown :-1 s=CLOSED pgs=4242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8a80066e0 0x7ff8a8008ad0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 1 --2- 172.21.15.98:0/2427873972 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff8bc1ce6a0 0x7ff8bc12ee80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/2427873972 >> 172.21.15.98:0/2427873972 conn(0x7ff8bc0a7850 msgr2=0x7ff8bc09d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.815+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/2427873972 shutdown_connections 2024-06-11T19:22:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.816+0000 7ff8c1ffc640 1 -- 172.21.15.98:0/2427873972 wait complete. 2024-06-11T19:22:07.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.816+0000 7ff8c1ffc640 1 librados: shutdown 2024-06-11T19:22:07.877 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:07.877 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:22:07.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:22:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.978+0000 7f6af366a640 10 monclient: get_monmap_and_config 2024-06-11T19:22:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.978+0000 7f6af366a640 10 monclient: build_initial_monmap 2024-06-11T19:22:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.978+0000 7f6af366a640 10 monclient: monmap: 2024-06-11T19:22:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:07.979896+0000 2024-06-11T19:22:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:07.979896+0000 2024-06-11T19:22:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.979 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:07.979 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:07.979 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:07.979 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:07.979 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:07.979 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:07.979 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:07.979 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:07.979 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:07.979 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:07.979 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:07.979 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.979+0000 7f6af366a640 1 Processor -- start 2024-06-11T19:22:07.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.979+0000 7f6af366a640 1 -- start start 2024-06-11T19:22:07.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.979+0000 7f6af366a640 10 monclient: init 2024-06-11T19:22:07.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:07.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 10 monclient: _add_conns ranks=[1,7,4,5,2,9,10,8,0,6,3] 2024-06-11T19:22:07.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6aec0a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:22:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6aec0a2530 0x7f6aec13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 10 monclient(hunting): picked mon.noname-i con 0x7f6aec0a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6aec13e1a0 0x7f6aec140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 10 monclient(hunting): picked mon.noname-c con 0x7f6aec13e1a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aec0a7140 con 0x7f6aec0a4920 2024-06-11T19:22:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f6aec0a2b50 con 0x7f6aec13e1a0 2024-06-11T19:22:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f6aec0a2e30 con 0x7f6aec0a2530 2024-06-11T19:22:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:07.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af366a640 10 monclient(hunting): authenticate will time out at 1537.787230s 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6aec13e1a0 0x7f6aec140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6aec13e1a0 0x7f6aec140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38034/0 (socket says 172.21.15.98:38034) 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 1 -- 172.21.15.98:0/2283117024 learned_addr learned my addr 172.21.15.98:0/2283117024 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 10 monclient(hunting): get_auth_request con 0x7f6aec13e1a0 auth_method 0 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af0c05640 1 --2- 172.21.15.98:0/2283117024 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6aec0a2530 0x7f6aec13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af0c05640 10 monclient(hunting): get_auth_request con 0x7f6aec0a2530 auth_method 0 2024-06-11T19:22:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af0c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af0c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af0c05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af0c05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af0c05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af0c05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 10 monclient(hunting): handle_auth_done global_id 114160 payload 274 2024-06-11T19:22:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 1 -- 172.21.15.98:0/2283117024 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6aec0a2530 msgr2=0x7f6aec13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 1 --2- 172.21.15.98:0/2283117024 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6aec0a2530 0x7f6aec13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 1 -- 172.21.15.98:0/2283117024 >> v1:172.21.15.117:6789/0 conn(0x7f6aec0a4920 legacy=0x7f6aec0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 10 monclient: _finish_hunting 0 2024-06-11T19:22:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 1 monclient: found mon.noname-c 2024-06-11T19:22:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:22:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.980+0000 7f6af1c07640 1 -- 172.21.15.98:0/2283117024 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec0a3110 con 0x7f6aec13e1a0 2024-06-11T19:22:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6af1c07640 1 --2- 172.21.15.98:0/2283117024 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6aec13e1a0 0x7f6aec140590 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4009e80 tx=0x7f6ae402be10 comp rx=0 tx=0).ready entity=mon.4 client_cookie=57e1a297dd6e28ae server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 1 -- 172.21.15.98:0/2283117024 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ae402cc20 con 0x7f6aec13e1a0 2024-06-11T19:22:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:22:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 10 monclient: dump: 2024-06-11T19:22:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 10 monclient: _finish_auth 0 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 10 monclient: _check_auth_tickets 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6af366a640 5 monclient: authenticate success, global_id 114160 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 1 -- 172.21.15.98:0/2283117024 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ae402cde0 con 0x7f6aec13e1a0 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6af366a640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 1 -- 172.21.15.98:0/2283117024 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ae4034860 con 0x7f6aec13e1a0 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6af366a640 4 set_mon_vals no callback set 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.981+0000 7f6ae3fff640 10 monclient: dump: 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 10 monclient: shutdown 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 1 -- 172.21.15.98:0/2283117024 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6aec13e1a0 msgr2=0x7f6aec140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 1 --2- 172.21.15.98:0/2283117024 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6aec13e1a0 0x7f6aec140590 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4009e80 tx=0x7f6ae402be10 comp rx=0 tx=0).stop 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 1 -- 172.21.15.98:0/2283117024 shutdown_connections 2024-06-11T19:22:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 1 --2- 172.21.15.98:0/2283117024 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6aec13e1a0 0x7f6aec140590 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 1 --2- 172.21.15.98:0/2283117024 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6aec0a2530 0x7f6aec13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 1 -- 172.21.15.98:0/2283117024 >> 172.21.15.98:0/2283117024 conn(0x7f6aec0ab850 msgr2=0x7f6aec0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 1 -- 172.21.15.98:0/2283117024 shutdown_connections 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 1 -- 172.21.15.98:0/2283117024 wait complete. 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 10 monclient: build_initial_monmap 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 10 monclient: monmap: 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 1 librados: starting msgr at 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.982+0000 7f6af366a640 1 librados: starting objecter 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 1 Processor -- start 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 1 -- start start 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 1 librados: setting wanted keys 2024-06-11T19:22:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 1 librados: calling monclient init 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 10 monclient: init 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 10 monclient: _add_conns ranks=[7,0,3,8,5,6,4,2,1] 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6aec0a2530 0x7f6aec1cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 10 monclient(hunting): picked mon.noname-i con 0x7f6aec0a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec0a4920 0x7f6aec136c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 10 monclient(hunting): picked mon.noname-a con 0x7f6aec0a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6aec13e1a0 0x7f6aec137140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 10 monclient(hunting): picked mon.noname-g con 0x7f6aec13e1a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec138e50 con 0x7f6aec0a4920 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f6aec0a84b0 con 0x7f6aec13e1a0 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f6aec0a6da0 con 0x7f6aec0a2530 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af366a640 10 monclient(hunting): authenticate will time out at 1537.790638s 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec0a4920 0x7f6aec136c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec0a4920 0x7f6aec136c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52220/0 (socket says 172.21.15.98:52220) 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 1 -- 172.21.15.98:0/2661265148 learned_addr learned my addr 172.21.15.98:0/2661265148 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 10 monclient(hunting): get_auth_request con 0x7f6aec0a4920 auth_method 0 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 10 monclient(hunting): handle_auth_done global_id 65637 payload 824 2024-06-11T19:22:08.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af1406640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6aec0a2530 0x7f6aec1cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af1c07640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6aec13e1a0 0x7f6aec137140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 1 -- 172.21.15.98:0/2661265148 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6aec0a2530 msgr2=0x7f6aec1cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6aec0a2530 0x7f6aec1cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 1 -- 172.21.15.98:0/2661265148 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6aec13e1a0 msgr2=0x7f6aec137140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6aec13e1a0 0x7f6aec137140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.983+0000 7f6af0c05640 10 monclient: _finish_hunting 0 2024-06-11T19:22:08.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af0c05640 1 monclient: found mon.noname-a 2024-06-11T19:22:08.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af0c05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:08.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af0c05640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af0c05640 1 -- 172.21.15.98:0/2661265148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec1235b0 con 0x7f6aec0a4920 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af0c05640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec0a4920 0x7f6aec136c00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00e8b0 tx=0x7f6adc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 10 client.?.objecter ms_handle_connect 0x7f6aec0a4920 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6adc017070 con 0x7f6aec0a4920 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 10 monclient: dump: 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 10 monclient: _finish_auth 0 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6adc00cdb0 con 0x7f6aec0a4920 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af366a640 5 monclient: authenticate success, global_id 65637 2024-06-11T19:22:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6adc0138e0 con 0x7f6aec0a4920 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af366a640 10 monclient: _renew_subs 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af366a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af366a640 1 -- 172.21.15.98:0/2661265148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aec0aace0 con 0x7f6aec0a4920 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae37fe640 4 set_mon_vals no callback set 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6ae1ffb640 10 monclient: dump: 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.006 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.006 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.006 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.006 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.006 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.006 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.006 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.006 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.006 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af366a640 10 client.65637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:08.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af366a640 10 monclient: _renew_subs 2024-06-11T19:22:08.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af366a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:08.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af366a640 1 -- 172.21.15.98:0/2661265148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aec0a1890 con 0x7f6aec0a4920 2024-06-11T19:22:08.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.984+0000 7f6af366a640 1 librados: init done 2024-06-11T19:22:08.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f6adc01a070 con 0x7f6aec0a4920 2024-06-11T19:22:08.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af366a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:08.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af366a640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec09fbc0 0x7f6aec09ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af366a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af366a640 1 -- 172.21.15.98:0/2661265148 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f6aec13dc40 con 0x7f6aec09fbc0 2024-06-11T19:22:08.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6ae1ffb640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6ad003fdb0 0x7f6ad0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af1406640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec09fbc0 0x7f6aec09ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(513..513 src has 1..513) ==== 6690+0+0 (secure 0 0 0) 0x7f6adc00f040 con 0x7f6aec0a4920 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6ae1ffb640 10 client.65637.objecter ms_dispatch 0x7f6aec000b90 osd_map(513..513 src has 1..513) 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6ae1ffb640 3 client.65637.objecter handle_osd_map got epochs [513,513] > 0 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6ae1ffb640 3 client.65637.objecter handle_osd_map decoding full epoch 513 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af1406640 10 monclient: get_auth_request con 0x7f6aec09fbc0 auth_method 0 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af1406640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af1406640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af1406640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af1406640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af1406640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af1406640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6ae1ffb640 20 client.65637.objecter dump_active .. 0 homeless 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af1c07640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6ad003fdb0 0x7f6ad0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af1c07640 10 monclient: get_auth_request con 0x7f6ad003fdb0 auth_method 0 2024-06-11T19:22:08.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af1406640 10 monclient(hunting): handle_auth_done global_id 55536 payload 824 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.985+0000 7f6af1406640 1 -- 172.21.15.98:0/2661265148 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6aec0a4d00 con 0x7f6aec09fbc0 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 10 client.65637.objecter ms_handle_connect 0x7f6aec09fbc0 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 10 client.65637.objecter resend_mon_ops 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6af1406640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec09fbc0 0x7f6aec09ffa0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800d3b0 tx=0x7f6ae800d870 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ae8018070 con 0x7f6aec09fbc0 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 10 monclient: dump: 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.009 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 10 client.65637.objecter ms_handle_connect 0x7f6ad003fdb0 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6af1c07640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6ad003fdb0 0x7f6ad0042270 secure :-1 s=READY pgs=4243 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4035000 tx=0x7f6ae4030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6ae80052e0 con 0x7f6aec09fbc0 2024-06-11T19:22:08.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:08.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:08.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec09fbc0 msgr2=0x7f6aec09ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec09fbc0 0x7f6aec09ffa0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800d3b0 tx=0x7f6ae800d870 comp rx=0 tx=0).stop 2024-06-11T19:22:08.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec09fbc0 msgr2=0x7f6aec09ffa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:08.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.986+0000 7f6ae1ffb640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec09fbc0 0x7f6aec09ffa0 secure :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800d3b0 tx=0x7f6ae800d870 comp rx=0 tx=0).stop 2024-06-11T19:22:08.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.992+0000 7f6af366a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:08.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.992+0000 7f6af366a640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec0a04e0 0x7f6aec13bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.992+0000 7f6af366a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.992+0000 7f6af366a640 1 -- 172.21.15.98:0/2661265148 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f6aec050830 con 0x7f6aec0a04e0 2024-06-11T19:22:08.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af1406640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec0a04e0 0x7f6aec13bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af1406640 10 monclient: get_auth_request con 0x7f6aec0a04e0 auth_method 0 2024-06-11T19:22:08.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af1406640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af1406640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af1406640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af1406640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af1406640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af1406640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af1406640 10 monclient(hunting): handle_auth_done global_id 55545 payload 824 2024-06-11T19:22:08.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af1406640 1 -- 172.21.15.98:0/2661265148 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6aec122dd0 con 0x7f6aec0a04e0 2024-06-11T19:22:08.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af1406640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec0a04e0 0x7f6aec13bd50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8000fb0 tx=0x7f6ae8000ed0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 10 client.65637.objecter ms_handle_connect 0x7f6aec0a04e0 2024-06-11T19:22:08.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 10 client.65637.objecter resend_mon_ops 2024-06-11T19:22:08.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ae8018070 con 0x7f6aec0a04e0 2024-06-11T19:22:08.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:22:08.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 10 monclient: dump: 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.015 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.016 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f6ae8003800 con 0x7f6aec0a04e0 2024-06-11T19:22:08.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:08.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:08.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec0a04e0 msgr2=0x7f6aec13bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec0a04e0 0x7f6aec13bd50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8000fb0 tx=0x7f6ae8000ed0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 1 -- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec0a04e0 msgr2=0x7f6aec13bd50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:08.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6ae1ffb640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6aec0a04e0 0x7f6aec13bd50 secure :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8000fb0 tx=0x7f6ae8000ed0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.016 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 640749, 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:22:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:08.040 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af366a640 10 librados: watch_flush enter 2024-06-11T19:22:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af366a640 10 librados: watch_flush exit 2024-06-11T19:22:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af366a640 20 client.65637.objecter shutdown clearing up homeless session... 2024-06-11T19:22:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.993+0000 7f6af366a640 10 client.65637.objecter successfully canceled tick 2024-06-11T19:22:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af366a640 1 -- 172.21.15.98:0/2661265148 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6ad003fdb0 msgr2=0x7f6ad0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af366a640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6ad003fdb0 0x7f6ad0042270 secure :-1 s=READY pgs=4243 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4035000 tx=0x7f6ae4030000 comp rx=0 tx=0).stop 2024-06-11T19:22:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af366a640 10 monclient: shutdown 2024-06-11T19:22:08.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af366a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af366a640 1 -- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec0a4920 msgr2=0x7f6aec136c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af366a640 1 --2- 172.21.15.98:0/2661265148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6aec0a4920 0x7f6aec136c00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00e8b0 tx=0x7f6adc00ed70 comp rx=0 tx=0).stop 2024-06-11T19:22:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af1c07640 1 -- 172.21.15.98:0/2661265148 reap_dead start 2024-06-11T19:22:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af1c07640 1 -- 172.21.15.98:0/2661265148 reap_dead start 2024-06-11T19:22:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af366a640 1 -- 172.21.15.98:0/2661265148 shutdown_connections 2024-06-11T19:22:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af366a640 1 -- 172.21.15.98:0/2661265148 >> 172.21.15.98:0/2661265148 conn(0x7f6aec0ab850 msgr2=0x7f6aec072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af366a640 1 -- 172.21.15.98:0/2661265148 shutdown_connections 2024-06-11T19:22:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af366a640 1 -- 172.21.15.98:0/2661265148 wait complete. 2024-06-11T19:22:08.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:07.994+0000 7f6af366a640 1 librados: shutdown 2024-06-11T19:22:08.044 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:08.044 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:22:08.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:22:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.144+0000 7fd02ed6d640 10 monclient: get_monmap_and_config 2024-06-11T19:22:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.144+0000 7fd02ed6d640 10 monclient: build_initial_monmap 2024-06-11T19:22:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.144+0000 7fd02ed6d640 10 monclient: monmap: 2024-06-11T19:22:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:08.145659+0000 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:08.145659+0000 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:08.145 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:08.145 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:08.145 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:08.145 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 1 Processor -- start 2024-06-11T19:22:08.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 1 -- start start 2024-06-11T19:22:08.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 10 monclient: init 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 10 monclient: _add_conns ranks=[2,0,5,9,4,8,1,7,6,3,10] 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a4920 0x7fd0280a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 10 monclient(hunting): picked mon.noname-b con 0x7fd0280a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd0280a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd0280a2530 0x7fd0280a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 10 monclient(hunting): picked mon.noname-h con 0x7fd0280a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0280a7140 con 0x7fd0280a5240 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd0280a2b50 con 0x7fd0280a4920 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd0280a2e30 con 0x7fd0280a2530 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02ed6d640 10 monclient(hunting): authenticate will time out at 1537.952893s 2024-06-11T19:22:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.145+0000 7fd02cb09640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a4920 0x7fd0280a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd01ffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd0280a5240 0x7fd0280a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56806/0 (socket says 172.21.15.98:56806) 2024-06-11T19:22:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd01ffff640 1 -- 172.21.15.98:0/2077121203 learned_addr learned my addr 172.21.15.98:0/2077121203 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 10 monclient(hunting): get_auth_request con 0x7fd0280a4920 auth_method 0 2024-06-11T19:22:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02d30a640 1 --2- 172.21.15.98:0/2077121203 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd0280a2530 0x7fd0280a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02d30a640 10 monclient(hunting): get_auth_request con 0x7fd0280a2530 auth_method 0 2024-06-11T19:22:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02d30a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02d30a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02d30a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 10 monclient(hunting): handle_auth_done global_id 84908 payload 274 2024-06-11T19:22:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 1 -- 172.21.15.98:0/2077121203 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd0280a2530 msgr2=0x7fd0280a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd01f7fe640 1 -- 172.21.15.98:0/2077121203 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1143185669 0 0) 0x7fd0280a7140 con 0x7fd0280a5240 2024-06-11T19:22:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 1 --2- 172.21.15.98:0/2077121203 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd0280a2530 0x7fd0280a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 1 -- 172.21.15.98:0/2077121203 >> v1:172.21.15.98:6789/0 conn(0x7fd0280a5240 legacy=0x7fd0280a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 10 monclient: _finish_hunting 0 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 1 monclient: found mon.noname-b 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd02cb09640 1 -- 172.21.15.98:0/2077121203 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0280a3110 con 0x7fd0280a4920 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.146+0000 7fd01f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02cb09640 1 --2- 172.21.15.98:0/2077121203 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a4920 0x7fd0280a4d00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd0240097d0 tx=0x7fd02402b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33d074b2a5f9cb8b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 1 -- 172.21.15.98:0/2077121203 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd024039070 con 0x7fd0280a4920 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 10 monclient: dump: 2024-06-11T19:22:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 5 monclient: authenticate success, global_id 84908 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 1 -- 172.21.15.98:0/2077121203 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd024030070 con 0x7fd0280a4920 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 1 -- 172.21.15.98:0/2077121203 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd024031db0 con 0x7fd0280a4920 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 4 set_mon_vals no callback set 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd01f7fe640 10 monclient: dump: 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.162 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 10 monclient: shutdown 2024-06-11T19:22:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 1 -- 172.21.15.98:0/2077121203 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a4920 msgr2=0x7fd0280a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 1 --2- 172.21.15.98:0/2077121203 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a4920 0x7fd0280a4d00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd0240097d0 tx=0x7fd02402b770 comp rx=0 tx=0).stop 2024-06-11T19:22:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 1 -- 172.21.15.98:0/2077121203 shutdown_connections 2024-06-11T19:22:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 1 --2- 172.21.15.98:0/2077121203 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd0280a2530 0x7fd0280a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 1 --2- 172.21.15.98:0/2077121203 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a4920 0x7fd0280a4d00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 1 -- 172.21.15.98:0/2077121203 >> 172.21.15.98:0/2077121203 conn(0x7fd0280ab850 msgr2=0x7fd0280a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 1 -- 172.21.15.98:0/2077121203 shutdown_connections 2024-06-11T19:22:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.147+0000 7fd02ed6d640 1 -- 172.21.15.98:0/2077121203 wait complete. 2024-06-11T19:22:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.148+0000 7fd02ed6d640 10 monclient: build_initial_monmap 2024-06-11T19:22:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.148+0000 7fd02ed6d640 10 monclient: monmap: 2024-06-11T19:22:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.148+0000 7fd02ed6d640 1 librados: starting msgr at 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.148+0000 7fd02ed6d640 1 librados: starting objecter 2024-06-11T19:22:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.148+0000 7fd02ed6d640 1 Processor -- start 2024-06-11T19:22:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.148+0000 7fd02ed6d640 1 -- start start 2024-06-11T19:22:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.148+0000 7fd02ed6d640 1 librados: setting wanted keys 2024-06-11T19:22:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.148+0000 7fd02ed6d640 1 librados: calling monclient init 2024-06-11T19:22:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.148+0000 7fd02ed6d640 10 monclient: init 2024-06-11T19:22:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 10 monclient: _add_conns ranks=[0,4,2,7,5,3,1,8,6] 2024-06-11T19:22:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0280a2530 0x7fd0281cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 10 monclient(hunting): picked mon.noname-a con 0x7fd0280a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd0280a4920 0x7fd0281cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 10 monclient(hunting): picked mon.noname-c con 0x7fd0280a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a5240 0x7fd0281250c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 10 monclient(hunting): picked mon.noname-b con 0x7fd0280a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0280a6da0 con 0x7fd0280a2530 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd028138e50 con 0x7fd0280a5240 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd0280a65f0 con 0x7fd0280a4920 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02ed6d640 10 monclient(hunting): authenticate will time out at 1537.956369s 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02cb09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0280a2530 0x7fd0281cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a5240 0x7fd0281250c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd01ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd0280a4920 0x7fd0281cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02cb09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0280a2530 0x7fd0281cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52224/0 (socket says 172.21.15.98:52224) 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a5240 0x7fd0281250c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57158/0 (socket says 172.21.15.98:57158) 2024-06-11T19:22:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02cb09640 1 -- 172.21.15.98:0/590336375 learned_addr learned my addr 172.21.15.98:0/590336375 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02cb09640 10 monclient(hunting): get_auth_request con 0x7fd0280a2530 auth_method 0 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02cb09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02cb09640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02cb09640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 10 monclient(hunting): get_auth_request con 0x7fd0280a5240 auth_method 0 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd01ffff640 10 monclient(hunting): get_auth_request con 0x7fd0280a4920 auth_method 0 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd01ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd01ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd01ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02cb09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02cb09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02cb09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd01ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd01ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd01ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 10 monclient(hunting): handle_auth_done global_id 84917 payload 824 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 1 -- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd0280a4920 msgr2=0x7fd0281cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd0280a4920 0x7fd0281cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 1 -- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0280a2530 msgr2=0x7fd0281cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0280a2530 0x7fd0281cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 10 monclient: _finish_hunting 0 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 1 monclient: found mon.noname-b 2024-06-11T19:22:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 1 -- 172.21.15.98:0/590336375 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd028126d90 con 0x7fd0280a5240 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02cb09640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0280a2530 0x7fd0281cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd01ffff640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd0280a4920 0x7fd0281cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd02d30a640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a5240 0x7fd0281250c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd020007d90 tx=0x7fd02000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.149+0000 7fd01d7fa640 10 client.?.objecter ms_handle_connect 0x7fd0280a5240 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd02000edd0 con 0x7fd0280a5240 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 10 monclient: dump: 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 10 monclient: _finish_auth 0 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd02ed6d640 5 monclient: authenticate success, global_id 84917 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd020014890 con 0x7fd0280a5240 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd02ed6d640 10 monclient: _renew_subs 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd02ed6d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd02ed6d640 1 -- 172.21.15.98:0/590336375 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd028129080 con 0x7fd0280a5240 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd020013380 con 0x7fd0280a5240 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd02ed6d640 10 client.84917.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01effd640 4 set_mon_vals no callback set 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd01d7fa640 10 monclient: dump: 2024-06-11T19:22:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.174 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd02ed6d640 10 monclient: _renew_subs 2024-06-11T19:22:08.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd02ed6d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:08.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd02ed6d640 1 -- 172.21.15.98:0/590336375 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0280aaaa0 con 0x7fd0280a5240 2024-06-11T19:22:08.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.150+0000 7fd02ed6d640 1 librados: init done 2024-06-11T19:22:08.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02ed6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:08.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02ed6d640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125600 0x7fd0281259e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02ed6d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02ed6d640 1 -- 172.21.15.98:0/590336375 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd028050830 con 0x7fd028125600 2024-06-11T19:22:08.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd02001b030 con 0x7fd0280a5240 2024-06-11T19:22:08.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02cb09640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125600 0x7fd0281259e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02cb09640 10 monclient: get_auth_request con 0x7fd028125600 auth_method 0 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02cb09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02cb09640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02cb09640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02cb09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02cb09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02cb09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd01403fdb0 0x7fd014042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02cb09640 10 monclient(hunting): handle_auth_done global_id 114128 payload 824 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(513..513 src has 1..513) ==== 6690+0+0 (secure 0 0 0) 0x7fd020019020 con 0x7fd0280a5240 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 10 client.84917.objecter ms_dispatch 0x7fd028000b90 osd_map(513..513 src has 1..513) 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 3 client.84917.objecter handle_osd_map got epochs [513,513] > 0 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 3 client.84917.objecter handle_osd_map decoding full epoch 513 2024-06-11T19:22:08.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02cb09640 1 -- 172.21.15.98:0/590336375 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd0280a8f30 con 0x7fd028125600 2024-06-11T19:22:08.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 20 client.84917.objecter dump_active .. 0 homeless 2024-06-11T19:22:08.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 10 client.84917.objecter ms_handle_connect 0x7fd028125600 2024-06-11T19:22:08.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 10 client.84917.objecter resend_mon_ops 2024-06-11T19:22:08.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd02cb09640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125600 0x7fd0281259e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd024000c00 tx=0x7fd0240046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd024039040 con 0x7fd028125600 2024-06-11T19:22:08.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01ffff640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd01403fdb0 0x7fd014042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:08.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01d7fa640 10 monclient: dump: 2024-06-11T19:22:08.176 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.176 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.176 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.151+0000 7fd01ffff640 10 monclient: get_auth_request con 0x7fd01403fdb0 auth_method 0 2024-06-11T19:22:08.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.152+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd02403a040 con 0x7fd028125600 2024-06-11T19:22:08.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.152+0000 7fd01d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:08.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.152+0000 7fd01d7fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:08.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.152+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125600 msgr2=0x7fd0281259e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.152+0000 7fd01d7fa640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125600 0x7fd0281259e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd024000c00 tx=0x7fd0240046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.152+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125600 msgr2=0x7fd0281259e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:08.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.152+0000 7fd01d7fa640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125600 0x7fd0281259e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.152+0000 7fd01d7fa640 10 client.84917.objecter ms_handle_connect 0x7fd01403fdb0 2024-06-11T19:22:08.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.152+0000 7fd01ffff640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd01403fdb0 0x7fd014042270 secure :-1 s=READY pgs=4244 cs=0 l=1 rev1=1 crypto rx=0x7fd018000f60 tx=0x7fd018007e40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02ed6d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:08.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02ed6d640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125f20 0x7fd0281215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02ed6d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02ed6d640 1 -- 172.21.15.98:0/590336375 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd0280a4d00 con 0x7fd028125f20 2024-06-11T19:22:08.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02cb09640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125f20 0x7fd0281215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02cb09640 10 monclient: get_auth_request con 0x7fd028125f20 auth_method 0 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02cb09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02cb09640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02cb09640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02cb09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02cb09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02cb09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02cb09640 10 monclient(hunting): handle_auth_done global_id 114137 payload 824 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02cb09640 1 -- 172.21.15.98:0/590336375 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd028138720 con 0x7fd028125f20 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd01d7fa640 10 client.84917.objecter ms_handle_connect 0x7fd028125f20 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd01d7fa640 10 client.84917.objecter resend_mon_ops 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd02cb09640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125f20 0x7fd0281215f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd0280a9f80 tx=0x7fd024009190 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd024003e40 con 0x7fd028125f20 2024-06-11T19:22:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd01d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd01d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.158+0000 7fd01d7fa640 10 monclient: dump: 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.183 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fd024003e40 con 0x7fd028125f20 2024-06-11T19:22:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd01d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd01d7fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125f20 msgr2=0x7fd0281215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd01d7fa640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125f20 0x7fd0281215f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd0280a9f80 tx=0x7fd024009190 comp rx=0 tx=0).stop 2024-06-11T19:22:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd01d7fa640 1 -- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125f20 msgr2=0x7fd0281215f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd01d7fa640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd028125f20 0x7fd0281215f0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd0280a9f80 tx=0x7fd024009190 comp rx=0 tx=0).stop 2024-06-11T19:22:08.183 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:08.184 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:08.184 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:08.184 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:08.184 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:08.184 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 16528, 2024-06-11T19:22:08.184 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:08.184 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:08.184 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:08.184 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:08.184 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:08.184 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:08.184 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.185 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.186 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.187 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.187 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.187 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.187 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.187 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.187 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.187 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:08.187 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:08.187 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:08.187 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:08.187 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:08.188 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:08.188 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:08.188 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:08.188 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:08.188 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:08.188 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:08.188 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:08.188 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:08.188 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:08.188 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:08.188 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.189 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:08.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.193 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:08.194 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.194 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:08.194 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.194 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.194 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.194 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:08.194 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.196 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.197 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.198 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:08.198 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:08.198 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.198 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.198 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.198 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:08.198 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:08.198 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.198 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.199 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:08.199 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.199 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:08.199 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.199 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.199 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.200 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:08.200 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:08.200 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.200 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.200 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.200 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:08.200 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:08.200 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.202 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:08.203 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.205 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.206 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:08.207 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.207 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.207 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:08.207 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 10 librados: watch_flush enter 2024-06-11T19:22:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 10 librados: watch_flush exit 2024-06-11T19:22:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 20 client.84917.objecter shutdown clearing up homeless session... 2024-06-11T19:22:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 10 client.84917.objecter successfully canceled tick 2024-06-11T19:22:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 1 -- 172.21.15.98:0/590336375 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd01403fdb0 msgr2=0x7fd014042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd01403fdb0 0x7fd014042270 secure :-1 s=READY pgs=4244 cs=0 l=1 rev1=1 crypto rx=0x7fd018000f60 tx=0x7fd018007e40 comp rx=0 tx=0).stop 2024-06-11T19:22:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 10 monclient: shutdown 2024-06-11T19:22:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 1 -- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a5240 msgr2=0x7fd0281250c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02d30a640 1 -- 172.21.15.98:0/590336375 reap_dead start 2024-06-11T19:22:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a5240 0x7fd0281250c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd020007d90 tx=0x7fd02000e4b0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 1 -- 172.21.15.98:0/590336375 shutdown_connections 2024-06-11T19:22:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 1 --2- 172.21.15.98:0/590336375 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd0280a5240 0x7fd0281250c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 1 -- 172.21.15.98:0/590336375 >> 172.21.15.98:0/590336375 conn(0x7fd0280ab850 msgr2=0x7fd02809c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 1 -- 172.21.15.98:0/590336375 shutdown_connections 2024-06-11T19:22:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 1 -- 172.21.15.98:0/590336375 wait complete. 2024-06-11T19:22:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.159+0000 7fd02ed6d640 1 librados: shutdown 2024-06-11T19:22:08.211 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:08.211 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:22:08.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:22:08.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.311+0000 7fac3cc8b640 10 monclient: get_monmap_and_config 2024-06-11T19:22:08.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.311+0000 7fac3cc8b640 10 monclient: build_initial_monmap 2024-06-11T19:22:08.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.311+0000 7fac3cc8b640 10 monclient: monmap: 2024-06-11T19:22:08.310 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:08.310 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:08.312467+0000 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:08.312467+0000 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:08.311 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:08.312 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.311+0000 7fac3cc8b640 1 Processor -- start 2024-06-11T19:22:08.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 1 -- start start 2024-06-11T19:22:08.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 10 monclient: init 2024-06-11T19:22:08.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:08.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 10 monclient: _add_conns ranks=[0,5,9,10,2,8,7,4,1,3,6] 2024-06-11T19:22:08.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac380a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:08.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fac380a2530 0x7fac3813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 10 monclient(hunting): picked mon.noname-h con 0x7fac380a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:08.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fac3813e1a0 0x7fac38140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 10 monclient(hunting): picked mon.noname-a con 0x7fac3813e1a0 addr v2:172.21.15.98:3300/0 2024-06-11T19:22:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac380a7140 con 0x7fac380a4920 2024-06-11T19:22:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fac380a2b50 con 0x7fac3813e1a0 2024-06-11T19:22:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fac380a2e30 con 0x7fac380a2530 2024-06-11T19:22:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3cc8b640 10 monclient(hunting): authenticate will time out at 1538.119709s 2024-06-11T19:22:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3659c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fac380a4920 0x7fac380a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56818/0 (socket says 172.21.15.98:56818) 2024-06-11T19:22:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac3659c640 1 -- 172.21.15.98:0/258139914 learned_addr learned my addr 172.21.15.98:0/258139914 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac36d9d640 1 --2- 172.21.15.98:0/258139914 >> v2:172.21.15.98:3300/0 conn(0x7fac3813e1a0 0x7fac38140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac36d9d640 10 monclient(hunting): get_auth_request con 0x7fac3813e1a0 auth_method 0 2024-06-11T19:22:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac36d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac36d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac36d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac36d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac36d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac35d9b640 1 --2- 172.21.15.98:0/258139914 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fac380a2530 0x7fac3813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.312+0000 7fac35d9b640 10 monclient(hunting): get_auth_request con 0x7fac380a2530 auth_method 0 2024-06-11T19:22:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac35d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac35d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac36d9d640 10 monclient(hunting): handle_auth_done global_id 65664 payload 274 2024-06-11T19:22:08.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac36d9d640 1 -- 172.21.15.98:0/258139914 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fac380a2530 msgr2=0x7fac3813dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:08.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac36d9d640 1 --2- 172.21.15.98:0/258139914 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fac380a2530 0x7fac3813dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac36d9d640 1 -- 172.21.15.98:0/258139914 >> v1:172.21.15.98:6789/0 conn(0x7fac380a4920 legacy=0x7fac380a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac36d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:22:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac36d9d640 1 monclient: found mon.noname-a 2024-06-11T19:22:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac36d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac36d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac36d9d640 1 -- 172.21.15.98:0/258139914 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac380a3110 con 0x7fac3813e1a0 2024-06-11T19:22:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac35d9b640 1 --2- 172.21.15.98:0/258139914 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fac380a2530 0x7fac3813dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:22:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac36d9d640 1 --2- 172.21.15.98:0/258139914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3813e1a0 0x7fac38140590 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fac30009d90 tx=0x7fac3002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20185039b8154157 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac3559a640 1 -- 172.21.15.98:0/258139914 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac30039070 con 0x7fac3813e1a0 2024-06-11T19:22:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac3559a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac3559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac3559a640 10 monclient: dump: 2024-06-11T19:22:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac3559a640 10 monclient: _finish_auth 0 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.313+0000 7fac3559a640 10 monclient: _check_auth_tickets 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3559a640 1 -- 172.21.15.98:0/258139914 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac3002cda0 con 0x7fac3813e1a0 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 5 monclient: authenticate success, global_id 65664 2024-06-11T19:22:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3559a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3559a640 1 -- 172.21.15.98:0/258139914 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac300327b0 con 0x7fac3813e1a0 2024-06-11T19:22:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 4 set_mon_vals no callback set 2024-06-11T19:22:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3559a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3559a640 10 monclient: dump: 2024-06-11T19:22:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 10 monclient: shutdown 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 1 -- 172.21.15.98:0/258139914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3813e1a0 msgr2=0x7fac38140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 1 --2- 172.21.15.98:0/258139914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3813e1a0 0x7fac38140590 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fac30009d90 tx=0x7fac3002bd80 comp rx=0 tx=0).stop 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 1 -- 172.21.15.98:0/258139914 shutdown_connections 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 1 --2- 172.21.15.98:0/258139914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac3813e1a0 0x7fac38140590 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 1 --2- 172.21.15.98:0/258139914 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fac380a2530 0x7fac3813dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 1 -- 172.21.15.98:0/258139914 >> 172.21.15.98:0/258139914 conn(0x7fac380ab850 msgr2=0x7fac380a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 1 -- 172.21.15.98:0/258139914 shutdown_connections 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.314+0000 7fac3cc8b640 1 -- 172.21.15.98:0/258139914 wait complete. 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.315+0000 7fac3cc8b640 10 monclient: build_initial_monmap 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.315+0000 7fac3cc8b640 10 monclient: monmap: 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.315+0000 7fac3cc8b640 1 librados: starting msgr at 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.315+0000 7fac3cc8b640 1 librados: starting objecter 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.315+0000 7fac3cc8b640 1 Processor -- start 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.315+0000 7fac3cc8b640 1 -- start start 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.315+0000 7fac3cc8b640 1 librados: setting wanted keys 2024-06-11T19:22:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.315+0000 7fac3cc8b640 1 librados: calling monclient init 2024-06-11T19:22:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.315+0000 7fac3cc8b640 10 monclient: init 2024-06-11T19:22:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 10 monclient: _add_conns ranks=[7,5,3,2,1,4,6,8,0] 2024-06-11T19:22:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fac380a2530 0x7fac381cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 10 monclient(hunting): picked mon.noname-i con 0x7fac380a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fac380a4920 0x7fac381cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 10 monclient(hunting): picked mon.noname-h con 0x7fac380a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac3813e1a0 0x7fac381ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 10 monclient(hunting): picked mon.noname-g con 0x7fac3813e1a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fac38138e50 con 0x7fac3813e1a0 2024-06-11T19:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fac380a84b0 con 0x7fac380a4920 2024-06-11T19:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fac380a6da0 con 0x7fac380a2530 2024-06-11T19:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3cc8b640 10 monclient(hunting): authenticate will time out at 1538.123170s 2024-06-11T19:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac3813e1a0 0x7fac381ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac35d9b640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fac380a4920 0x7fac381cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3659c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fac380a2530 0x7fac381cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac35d9b640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fac380a4920 0x7fac381cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:50714/0 (socket says 172.21.15.98:50714) 2024-06-11T19:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac35d9b640 1 -- 172.21.15.98:0/142805474 learned_addr learned my addr 172.21.15.98:0/142805474 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 10 monclient(hunting): get_auth_request con 0x7fac3813e1a0 auth_method 0 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac35d9b640 10 monclient(hunting): get_auth_request con 0x7fac380a4920 auth_method 0 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac35d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac35d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3659c640 10 monclient(hunting): get_auth_request con 0x7fac380a2530 auth_method 0 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac35d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac35d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac35d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 10 monclient(hunting): handle_auth_done global_id 114114 payload 824 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 1 -- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fac380a2530 msgr2=0x7fac381cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fac380a2530 0x7fac381cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 1 -- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fac380a4920 msgr2=0x7fac381cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fac380a4920 0x7fac381cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 1 monclient: found mon.noname-g 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:22:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac36d9d640 1 -- 172.21.15.98:0/142805474 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac380a0ad0 con 0x7fac3813e1a0 2024-06-11T19:22:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.316+0000 7fac35d9b640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fac380a4920 0x7fac381cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac36d9d640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac3813e1a0 0x7fac381ce1a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fac30033000 tx=0x7fac30004a40 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 10 client.?.objecter ms_handle_connect 0x7fac3813e1a0 2024-06-11T19:22:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac30039050 con 0x7fac3813e1a0 2024-06-11T19:22:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:22:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 10 monclient: dump: 2024-06-11T19:22:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 10 monclient: _finish_auth 0 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac30045070 con 0x7fac3813e1a0 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac3cc8b640 5 monclient: authenticate success, global_id 114114 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac3cc8b640 10 monclient: _renew_subs 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac3cc8b640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac3cc8b640 1 -- 172.21.15.98:0/142805474 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac380a4290 con 0x7fac3813e1a0 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac300046c0 con 0x7fac3813e1a0 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:22:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac26ffd640 10 monclient: dump: 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac34d99640 4 set_mon_vals no callback set 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac3cc8b640 10 client.114114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac3cc8b640 10 monclient: _renew_subs 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac3cc8b640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac3cc8b640 1 -- 172.21.15.98:0/142805474 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac380a07f0 con 0x7fac3813e1a0 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac3cc8b640 1 librados: init done 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.317+0000 7fac3cc8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3cc8b640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38133710 0x7fac38133af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3cc8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3cc8b640 1 -- 172.21.15.98:0/142805474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fac380a0470 con 0x7fac38133710 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 <== mon.3 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fac30004250 con 0x7fac3813e1a0 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3659c640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38133710 0x7fac38133af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3659c640 10 monclient: get_auth_request con 0x7fac38133710 auth_method 0 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fac1803fde0 0x7fac180422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 <== mon.3 v2:172.21.15.117:3301/0 5 ==== osd_map(513..513 src has 1..513) ==== 6690+0+0 (secure 0 0 0) 0x7fac30074190 con 0x7fac3813e1a0 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 10 client.114114.objecter ms_dispatch 0x7fac38000b90 osd_map(513..513 src has 1..513) 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 3 client.114114.objecter handle_osd_map got epochs [513,513] > 0 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 3 client.114114.objecter handle_osd_map decoding full epoch 513 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3659c640 10 monclient(hunting): handle_auth_done global_id 32653 payload 824 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3659c640 1 -- 172.21.15.98:0/142805474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac380a4d00 con 0x7fac38133710 2024-06-11T19:22:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 20 client.114114.objecter dump_active .. 0 homeless 2024-06-11T19:22:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac35d9b640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fac1803fde0 0x7fac180422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac35d9b640 10 monclient: get_auth_request con 0x7fac1803fde0 auth_method 0 2024-06-11T19:22:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 10 client.114114.objecter ms_handle_connect 0x7fac38133710 2024-06-11T19:22:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 10 client.114114.objecter resend_mon_ops 2024-06-11T19:22:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac3659c640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38133710 0x7fac38133af0 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7fac28007fd0 tx=0x7fac2800fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac28005cc0 con 0x7fac38133710 2024-06-11T19:22:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:22:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.318+0000 7fac26ffd640 10 monclient: dump: 2024-06-11T19:22:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.345 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.319+0000 7fac26ffd640 10 client.114114.objecter ms_handle_connect 0x7fac1803fde0 2024-06-11T19:22:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.319+0000 7fac35d9b640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fac1803fde0 0x7fac180422a0 secure :-1 s=READY pgs=4245 cs=0 l=1 rev1=1 crypto rx=0x7fac2c006a90 tx=0x7fac2c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.319+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fac28018070 con 0x7fac38133710 2024-06-11T19:22:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.319+0000 7fac26ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.319+0000 7fac26ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.319+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38133710 msgr2=0x7fac38133af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.319+0000 7fac26ffd640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38133710 0x7fac38133af0 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7fac28007fd0 tx=0x7fac2800fc00 comp rx=0 tx=0).stop 2024-06-11T19:22:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.319+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38133710 msgr2=0x7fac38133af0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.319+0000 7fac26ffd640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38133710 0x7fac38133af0 secure :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7fac28007fd0 tx=0x7fac2800fc00 comp rx=0 tx=0).stop 2024-06-11T19:22:08.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.326+0000 7fac3cc8b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:08.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.326+0000 7fac3cc8b640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38134030 0x7fac38139df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.326+0000 7fac3cc8b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.326+0000 7fac3cc8b640 1 -- 172.21.15.98:0/142805474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fac3809e7e0 con 0x7fac38134030 2024-06-11T19:22:08.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.326+0000 7fac3659c640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38134030 0x7fac38139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.326+0000 7fac3659c640 10 monclient: get_auth_request con 0x7fac38134030 auth_method 0 2024-06-11T19:22:08.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.326+0000 7fac3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.326+0000 7fac3659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.326+0000 7fac3659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.326+0000 7fac3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.326+0000 7fac3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.326+0000 7fac3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac3659c640 10 monclient(hunting): handle_auth_done global_id 32662 payload 824 2024-06-11T19:22:08.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac3659c640 1 -- 172.21.15.98:0/142805474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac3813df00 con 0x7fac38134030 2024-06-11T19:22:08.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac3659c640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38134030 0x7fac38139df0 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7fac28007fd0 tx=0x7fac28000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 10 client.114114.objecter ms_handle_connect 0x7fac38134030 2024-06-11T19:22:08.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 10 client.114114.objecter resend_mon_ops 2024-06-11T19:22:08.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac280076c0 con 0x7fac38134030 2024-06-11T19:22:08.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:22:08.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 10 monclient: dump: 2024-06-11T19:22:08.350 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.350 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.352 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7fac280076c0 con 0x7fac38134030 2024-06-11T19:22:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38134030 msgr2=0x7fac38139df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38134030 0x7fac38139df0 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7fac28007fd0 tx=0x7fac28000ed0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 1 -- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38134030 msgr2=0x7fac38139df0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.327+0000 7fac26ffd640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac38134030 0x7fac38139df0 secure :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7fac28007fd0 tx=0x7fac28000ed0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.353 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:08.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:08.353 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:08.353 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:22:08.353 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:08.353 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 778545, 2024-06-11T19:22:08.353 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:08.353 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:08.353 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:08.353 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:08.353 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:08.354 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:08.354 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:08.354 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:08.354 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:08.354 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:08.354 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.354 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:22:08.354 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:08.354 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:08.354 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:08.354 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.354 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.355 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.355 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.355 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.355 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.355 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.355 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.355 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.355 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.355 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.355 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:08.355 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:08.357 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.358 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.359 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.359 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.359 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.359 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:08.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:08.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.359 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:08.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:08.359 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:08.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:08.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.360 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:08.362 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:08.362 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.362 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:08.363 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:08.363 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.363 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.363 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.363 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:08.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:08.363 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:08.364 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:08.364 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.364 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.368 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:08.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:08.370 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:08.370 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.370 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.371 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.371 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:08.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:08.371 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:08.372 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:08.372 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:08.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.374 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:08.374 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.374 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.374 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:08.374 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:08.374 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:08.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.378 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:08.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.378 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.378 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.378 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:22:08.378 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.378 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.378 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:08.378 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 10 librados: watch_flush enter 2024-06-11T19:22:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 10 librados: watch_flush exit 2024-06-11T19:22:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 20 client.114114.objecter shutdown clearing up homeless session... 2024-06-11T19:22:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 10 client.114114.objecter successfully canceled tick 2024-06-11T19:22:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 1 -- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fac1803fde0 msgr2=0x7fac180422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fac1803fde0 0x7fac180422a0 secure :-1 s=READY pgs=4245 cs=0 l=1 rev1=1 crypto rx=0x7fac2c006a90 tx=0x7fac2c008040 comp rx=0 tx=0).stop 2024-06-11T19:22:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 10 monclient: shutdown 2024-06-11T19:22:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 1 -- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac3813e1a0 msgr2=0x7fac381ce1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 1 --2- 172.21.15.98:0/142805474 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac3813e1a0 0x7fac381ce1a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fac30033000 tx=0x7fac30004a40 comp rx=0 tx=0).stop 2024-06-11T19:22:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac36d9d640 1 -- 172.21.15.98:0/142805474 reap_dead start 2024-06-11T19:22:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac36d9d640 1 -- 172.21.15.98:0/142805474 reap_dead start 2024-06-11T19:22:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 1 -- 172.21.15.98:0/142805474 shutdown_connections 2024-06-11T19:22:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 1 -- 172.21.15.98:0/142805474 >> 172.21.15.98:0/142805474 conn(0x7fac380ab850 msgr2=0x7fac380a61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 1 -- 172.21.15.98:0/142805474 shutdown_connections 2024-06-11T19:22:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 1 -- 172.21.15.98:0/142805474 wait complete. 2024-06-11T19:22:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.328+0000 7fac3cc8b640 1 librados: shutdown 2024-06-11T19:22:08.382 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:08.382 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:22:08.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:22:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.483+0000 7fbbcf4ca640 10 monclient: get_monmap_and_config 2024-06-11T19:22:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.483+0000 7fbbcf4ca640 10 monclient: build_initial_monmap 2024-06-11T19:22:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.483+0000 7fbbcf4ca640 10 monclient: monmap: 2024-06-11T19:22:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:08.484540+0000 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:08.484540+0000 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:08.484 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:08.484 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 1 Processor -- start 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 1 -- start start 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 10 monclient: init 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 10 monclient: _add_conns ranks=[4,10,9,7,1,2,3,6,5,8,0] 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbc80a4920 0x7fbbc80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 10 monclient(hunting): picked mon.noname-c con 0x7fbbc80a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fbbc80a5240 0x7fbbc80a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 10 monclient(hunting): picked mon.noname-f con 0x7fbbc80a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbbc80a2530 0x7fbbc80a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 10 monclient(hunting): picked mon.noname-a con 0x7fbbc80a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbbc80a7140 con 0x7fbbc80a2530 2024-06-11T19:22:08.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fbbc80a2b50 con 0x7fbbc80a5240 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fbbc80a2e30 con 0x7fbbc80a4920 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcf4ca640 10 monclient(hunting): authenticate will time out at 1538.291795s 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcd266640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbc80a4920 0x7fbbc80a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcda67640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbbc80a2530 0x7fbbc80a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcd266640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbc80a4920 0x7fbbc80a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38066/0 (socket says 172.21.15.98:38066) 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcd266640 1 -- 172.21.15.98:0/3828524412 learned_addr learned my addr 172.21.15.98:0/3828524412 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcda67640 10 monclient(hunting): get_auth_request con 0x7fbbc80a2530 auth_method 0 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcda67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcda67640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcda67640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcd266640 10 monclient(hunting): get_auth_request con 0x7fbbc80a4920 auth_method 0 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcd266640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcd266640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.484+0000 7fbbcd266640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcd266640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcd266640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcd266640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcca65640 1 --2- 172.21.15.98:0/3828524412 >> v2:172.21.15.117:3300/0 conn(0x7fbbc80a5240 0x7fbbc80a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcca65640 10 monclient(hunting): get_auth_request con 0x7fbbc80a5240 auth_method 0 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcca65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcca65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcca65640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 10 monclient(hunting): handle_auth_done global_id 65682 payload 274 2024-06-11T19:22:08.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 1 -- 172.21.15.98:0/3828524412 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbc80a4920 msgr2=0x7fbbc80a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 1 --2- 172.21.15.98:0/3828524412 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbc80a4920 0x7fbbc80a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 1 -- 172.21.15.98:0/3828524412 >> v2:172.21.15.117:3300/0 conn(0x7fbbc80a5240 msgr2=0x7fbbc80a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 1 --2- 172.21.15.98:0/3828524412 >> v2:172.21.15.117:3300/0 conn(0x7fbbc80a5240 0x7fbbc80a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 10 monclient: _finish_hunting 0 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 1 monclient: found mon.noname-a 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 1 -- 172.21.15.98:0/3828524412 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc80a3110 con 0x7fbbc80a2530 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcd266640 1 --2- 172.21.15.98:0/3828524412 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbc80a4920 0x7fbbc80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcca65640 1 --2- 172.21.15.98:0/3828524412 >> v2:172.21.15.117:3300/0 conn(0x7fbbc80a5240 0x7fbbc80a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.485+0000 7fbbcda67640 1 --2- 172.21.15.98:0/3828524412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc80a2530 0x7fbbc80a5ee0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80a8420 tx=0x7fbbc002bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c0244fe4233108 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 1 -- 172.21.15.98:0/3828524412 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbbc0039070 con 0x7fbbc80a2530 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 10 monclient: dump: 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.499 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 10 monclient: _finish_auth 0 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 10 monclient: _check_auth_tickets 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 1 -- 172.21.15.98:0/3828524412 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbc002cd20 con 0x7fbbc80a2530 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 5 monclient: authenticate success, global_id 65682 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 1 -- 172.21.15.98:0/3828524412 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbbc00327b0 con 0x7fbbc80a2530 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 4 set_mon_vals no callback set 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:08.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbbffff640 10 monclient: dump: 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.501 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 10 monclient: shutdown 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/3828524412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc80a2530 msgr2=0x7fbbc80a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 1 --2- 172.21.15.98:0/3828524412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc80a2530 0x7fbbc80a5ee0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80a8420 tx=0x7fbbc002bd00 comp rx=0 tx=0).stop 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/3828524412 shutdown_connections 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 1 --2- 172.21.15.98:0/3828524412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbc80a2530 0x7fbbc80a5ee0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 1 --2- 172.21.15.98:0/3828524412 >> v2:172.21.15.117:3300/0 conn(0x7fbbc80a5240 0x7fbbc80a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 1 --2- 172.21.15.98:0/3828524412 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbbc80a4920 0x7fbbc80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/3828524412 >> 172.21.15.98:0/3828524412 conn(0x7fbbc80ab850 msgr2=0x7fbbc80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/3828524412 shutdown_connections 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.486+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/3828524412 wait complete. 2024-06-11T19:22:08.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.487+0000 7fbbcf4ca640 10 monclient: build_initial_monmap 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.487+0000 7fbbcf4ca640 10 monclient: monmap: 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:08.503 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.487+0000 7fbbcf4ca640 1 librados: starting msgr at 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.487+0000 7fbbcf4ca640 1 librados: starting objecter 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.487+0000 7fbbcf4ca640 1 Processor -- start 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.487+0000 7fbbcf4ca640 1 -- start start 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.487+0000 7fbbcf4ca640 1 librados: setting wanted keys 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.487+0000 7fbbcf4ca640 1 librados: calling monclient init 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.487+0000 7fbbcf4ca640 10 monclient: init 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 10 monclient: _add_conns ranks=[6,3,2,1,0,8,7,5,4] 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbbc80a2530 0x7fbbc81cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 10 monclient(hunting): picked mon.noname-d con 0x7fbbc80a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc80a4920 0x7fbbc81cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 10 monclient(hunting): picked mon.noname-g con 0x7fbbc80a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbc80a5240 0x7fbbc8122ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 10 monclient(hunting): picked mon.noname-b con 0x7fbbc80a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbbc80a6da0 con 0x7fbbc80a5240 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fbbc8138e50 con 0x7fbbc80a4920 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fbbc80a6420 con 0x7fbbc80a2530 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 10 monclient(hunting): authenticate will time out at 1538.295140s 2024-06-11T19:22:08.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbbc80a2530 0x7fbbc81cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcda67640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbc80a5240 0x7fbbc8122ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbbc80a2530 0x7fbbc81cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34250/0 (socket says 172.21.15.98:34250) 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 1 -- 172.21.15.98:0/1131144713 learned_addr learned my addr 172.21.15.98:0/1131144713 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 10 monclient(hunting): get_auth_request con 0x7fbbc80a2530 auth_method 0 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcda67640 10 monclient(hunting): get_auth_request con 0x7fbbc80a5240 auth_method 0 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcda67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcda67640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcda67640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcda67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcda67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcda67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcca65640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc80a4920 0x7fbbc81cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcca65640 10 monclient(hunting): get_auth_request con 0x7fbbc80a4920 auth_method 0 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcca65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcca65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcca65640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 10 monclient(hunting): handle_auth_done global_id 55554 payload 824 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 1 -- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc80a4920 msgr2=0x7fbbc81cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc80a4920 0x7fbbc81cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 1 -- 172.21.15.98:0/1131144713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbc80a5240 msgr2=0x7fbbc8122ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbc80a5240 0x7fbbc8122ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 10 monclient: _finish_hunting 0 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 1 monclient: found mon.noname-d 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 1 -- 172.21.15.98:0/1131144713 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc8126bb0 con 0x7fbbc80a2530 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcca65640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc80a4920 0x7fbbc81cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcda67640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbbc80a5240 0x7fbbc8122ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcd266640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbbc80a2530 0x7fbbc81cda70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbbc400e9d0 tx=0x7fbbc400ee90 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 10 client.?.objecter ms_handle_connect 0x7fbbc80a2530 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbbc4017070 con 0x7fbbc80a2530 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 10 monclient: dump: 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.508 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 10 monclient: _finish_auth 0 2024-06-11T19:22:08.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 10 monclient: _check_auth_tickets 2024-06-11T19:22:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbc4013470 con 0x7fbbc80a2530 2024-06-11T19:22:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 5 monclient: authenticate success, global_id 55554 2024-06-11T19:22:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbbc4013a90 con 0x7fbbc80a2530 2024-06-11T19:22:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 10 monclient: _renew_subs 2024-06-11T19:22:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:22:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/1131144713 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc8126e90 con 0x7fbbc80a2530 2024-06-11T19:22:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:22:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbf7fe640 4 set_mon_vals no callback set 2024-06-11T19:22:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.488+0000 7fbbbdffb640 10 monclient: dump: 2024-06-11T19:22:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.489+0000 7fbbcf4ca640 10 client.55554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.489+0000 7fbbcf4ca640 10 monclient: _renew_subs 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.489+0000 7fbbcf4ca640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.489+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/1131144713 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc80a92b0 con 0x7fbbc80a2530 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.489+0000 7fbbcf4ca640 1 librados: init done 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fbbc401a070 con 0x7fbbc80a2530 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcf4ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcf4ca640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123410 0x7fbbc81237f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcf4ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/1131144713 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fbbc8050830 con 0x7fbbc8123410 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbbdffb640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbbac03fdb0 0x7fbbac042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(513..513 src has 1..513) ==== 6690+0+0 (secure 0 0 0) 0x7fbbc400f040 con 0x7fbbc80a2530 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbbdffb640 10 client.55554.objecter ms_dispatch 0x7fbbc8000b90 osd_map(513..513 src has 1..513) 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbbdffb640 3 client.55554.objecter handle_osd_map got epochs [513,513] > 0 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbbdffb640 3 client.55554.objecter handle_osd_map decoding full epoch 513 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcca65640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123410 0x7fbbc81237f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbbdffb640 20 client.55554.objecter dump_active .. 0 homeless 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcca65640 10 monclient: get_auth_request con 0x7fbbc8123410 auth_method 0 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcca65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcca65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcca65640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcca65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcca65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcca65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcda67640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbbac03fdb0 0x7fbbac042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcda67640 10 monclient: get_auth_request con 0x7fbbac03fdb0 auth_method 0 2024-06-11T19:22:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcca65640 10 monclient(hunting): handle_auth_done global_id 114132 payload 824 2024-06-11T19:22:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.490+0000 7fbbcca65640 1 -- 172.21.15.98:0/1131144713 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbbc80a6750 con 0x7fbbc8123410 2024-06-11T19:22:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 10 client.55554.objecter ms_handle_connect 0x7fbbc8123410 2024-06-11T19:22:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbcca65640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123410 0x7fbbc81237f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80a5620 tx=0x7fbbb800f8a0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 10 client.55554.objecter resend_mon_ops 2024-06-11T19:22:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbbb8018030 con 0x7fbbc8123410 2024-06-11T19:22:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 10 monclient: dump: 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.516 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbcda67640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbbac03fdb0 0x7fbbac042270 secure :-1 s=READY pgs=4246 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0036000 tx=0x7fbbc0030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 10 client.55554.objecter ms_handle_connect 0x7fbbac03fdb0 2024-06-11T19:22:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbbb8014670 con 0x7fbbc8123410 2024-06-11T19:22:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123410 msgr2=0x7fbbc81237f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123410 0x7fbbc81237f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80a5620 tx=0x7fbbb800f8a0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123410 msgr2=0x7fbbc81237f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.491+0000 7fbbbdffb640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123410 0x7fbbc81237f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.497+0000 7fbbcf4ca640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.497+0000 7fbbcf4ca640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123d30 0x7fbbc809e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.497+0000 7fbbcf4ca640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.497+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/1131144713 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fbbc8125450 con 0x7fbbc8123d30 2024-06-11T19:22:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbcca65640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123d30 0x7fbbc809e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbcca65640 10 monclient: get_auth_request con 0x7fbbc8123d30 auth_method 0 2024-06-11T19:22:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbcca65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbcca65640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbcca65640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbcca65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbcca65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbcca65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbcca65640 10 monclient(hunting): handle_auth_done global_id 114141 payload 824 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbcca65640 1 -- 172.21.15.98:0/1131144713 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbbc806c300 con 0x7fbbc8123d30 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbcca65640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123d30 0x7fbbc809e9f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8000c00 tx=0x7fbbb80046c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbbdffb640 10 client.55554.objecter ms_handle_connect 0x7fbbc8123d30 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.498+0000 7fbbbdffb640 10 client.55554.objecter resend_mon_ops 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbbb8003610 con 0x7fbbc8123d30 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbbdffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbbdffb640 10 monclient: dump: 2024-06-11T19:22:08.519 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.520 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.521 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fbbb8014a30 con 0x7fbbc8123d30 2024-06-11T19:22:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbbdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbbdffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123d30 msgr2=0x7fbbc809e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbbdffb640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123d30 0x7fbbc809e9f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8000c00 tx=0x7fbbb80046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbbdffb640 1 -- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123d30 msgr2=0x7fbbc809e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:08.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbbdffb640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbbc8123d30 0x7fbbc809e9f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.521 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:08.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:08.521 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:08.522 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:08.522 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:08.522 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 16867, 2024-06-11T19:22:08.522 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:08.522 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:08.522 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:08.522 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:08.522 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:08.522 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:08.522 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:08.522 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:08.522 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:08.523 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:08.523 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.523 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:22:08.523 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:08.523 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:08.523 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:08.523 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.523 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.523 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.523 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.523 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:08.525 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:08.526 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:08.526 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:08.526 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:08.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:08.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:08.526 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:08.526 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:08.526 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:08.526 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:08.526 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:08.526 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:08.526 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:08.527 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.527 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.527 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.527 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.527 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.527 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.527 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.528 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.528 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.528 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.528 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.528 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.528 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:08.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.528 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:08.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.528 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:08.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:08.528 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:08.530 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:08.534 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.535 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.535 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.535 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:08.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:08.535 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.535 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.535 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.536 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:08.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:08.539 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:08.539 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:08.543 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.544 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:08.545 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:08.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.545 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.545 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.545 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:08.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.545 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.545 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:08.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.546 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.546 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.546 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:08.546 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.546 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.547 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:08.547 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 10 librados: watch_flush enter 2024-06-11T19:22:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 10 librados: watch_flush exit 2024-06-11T19:22:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 20 client.55554.objecter shutdown clearing up homeless session... 2024-06-11T19:22:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 10 client.55554.objecter successfully canceled tick 2024-06-11T19:22:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbbac03fdb0 msgr2=0x7fbbac042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbbac03fdb0 0x7fbbac042270 secure :-1 s=READY pgs=4246 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0036000 tx=0x7fbbc0030000 comp rx=0 tx=0).stop 2024-06-11T19:22:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 10 monclient: shutdown 2024-06-11T19:22:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/1131144713 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbbc80a2530 msgr2=0x7fbbc81cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 1 --2- 172.21.15.98:0/1131144713 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbbc80a2530 0x7fbbc81cda70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbbc400e9d0 tx=0x7fbbc400ee90 comp rx=0 tx=0).stop 2024-06-11T19:22:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcda67640 1 -- 172.21.15.98:0/1131144713 reap_dead start 2024-06-11T19:22:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcda67640 1 -- 172.21.15.98:0/1131144713 reap_dead start 2024-06-11T19:22:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/1131144713 shutdown_connections 2024-06-11T19:22:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/1131144713 >> 172.21.15.98:0/1131144713 conn(0x7fbbc80ab850 msgr2=0x7fbbc81bff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/1131144713 shutdown_connections 2024-06-11T19:22:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 1 -- 172.21.15.98:0/1131144713 wait complete. 2024-06-11T19:22:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.499+0000 7fbbcf4ca640 1 librados: shutdown 2024-06-11T19:22:08.550 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:08.550 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:22:08.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:22:08.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.653+0000 7f284afcd640 10 monclient: get_monmap_and_config 2024-06-11T19:22:08.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.653+0000 7f284afcd640 10 monclient: build_initial_monmap 2024-06-11T19:22:08.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.653+0000 7f284afcd640 10 monclient: monmap: 2024-06-11T19:22:08.652 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:08.652 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:08.654268+0000 2024-06-11T19:22:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:08.654268+0000 2024-06-11T19:22:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:08.654 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:08.654 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:08.654 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:08.654 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:08.654 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.653+0000 7f284afcd640 1 Processor -- start 2024-06-11T19:22:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 1 -- start start 2024-06-11T19:22:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 10 monclient: init 2024-06-11T19:22:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 10 monclient: _add_conns ranks=[10,2,1,6,9,5,8,4,3,0,7] 2024-06-11T19:22:08.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f28441347f0 0x7f284409e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 10 monclient(hunting): picked mon.noname-f con 0x7f28441347f0 addr v2:172.21.15.117:3300/0 2024-06-11T19:22:08.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844134bd0 0x7f28440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 10 monclient(hunting): picked mon.noname-b con 0x7f2844134bd0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:08.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f2844135010 addr v1:172.21.15.117:6789/0 2024-06-11T19:22:08.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f284409d410 con 0x7f2844135010 2024-06-11T19:22:08.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f28440687f0 con 0x7f28441347f0 2024-06-11T19:22:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f284409d6f0 con 0x7f2844134bd0 2024-06-11T19:22:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f284afcd640 10 monclient(hunting): authenticate will time out at 1538.461713s 2024-06-11T19:22:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2843fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844134bd0 0x7f28440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2843fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844134bd0 0x7f28440a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57176/0 (socket says 172.21.15.98:57176) 2024-06-11T19:22:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2843fff640 1 -- 172.21.15.98:0/4218496874 learned_addr learned my addr 172.21.15.98:0/4218496874 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2843fff640 10 monclient(hunting): get_auth_request con 0x7f2844134bd0 auth_method 0 2024-06-11T19:22:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2843fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2843fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2843fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2848d69640 1 --2- 172.21.15.98:0/4218496874 >> v2:172.21.15.117:3300/0 conn(0x7f28441347f0 0x7f284409e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2848d69640 10 monclient(hunting): get_auth_request con 0x7f28441347f0 auth_method 0 2024-06-11T19:22:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2848d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2848d69640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2848d69640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2843fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2843fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2843fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2848d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.654+0000 7f2848d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f2848d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f2843fff640 10 monclient(hunting): handle_auth_done global_id 84935 payload 274 2024-06-11T19:22:08.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f2843fff640 1 -- 172.21.15.98:0/4218496874 >> v2:172.21.15.117:3300/0 conn(0x7f28441347f0 msgr2=0x7f284409e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:08.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f2843fff640 1 --2- 172.21.15.98:0/4218496874 >> v2:172.21.15.117:3300/0 conn(0x7f28441347f0 0x7f284409e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f2843fff640 1 -- 172.21.15.98:0/4218496874 >> v1:172.21.15.117:6789/0 conn(0x7f2844135010 legacy=0x7f28440a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f2843fff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:08.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f2843fff640 1 monclient: found mon.noname-b 2024-06-11T19:22:08.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f2843fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:08.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f2843fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:22:08.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f2843fff640 1 -- 172.21.15.98:0/4218496874 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f284409d9d0 con 0x7f2844134bd0 2024-06-11T19:22:08.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f2843fff640 1 --2- 172.21.15.98:0/4218496874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844134bd0 0x7f28440a3690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2830009870 tx=0x7f283002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a4dd18a4cba894c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f28437fe640 1 -- 172.21.15.98:0/4218496874 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2830035020 con 0x7f2844134bd0 2024-06-11T19:22:08.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f28437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f28437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:22:08.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.655+0000 7f28437fe640 10 monclient: dump: 2024-06-11T19:22:08.671 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.671 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.671 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.671 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.671 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.671 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f28437fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f28437fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:08.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f28437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:08.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 5 monclient: authenticate success, global_id 84935 2024-06-11T19:22:08.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f28437fe640 1 -- 172.21.15.98:0/4218496874 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28300318b0 con 0x7f2844134bd0 2024-06-11T19:22:08.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:08.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f28437fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:08.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:08.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 4 set_mon_vals no callback set 2024-06-11T19:22:08.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f28437fe640 1 -- 172.21.15.98:0/4218496874 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f283002d040 con 0x7f2844134bd0 2024-06-11T19:22:08.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f28437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f28437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:08.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f28437fe640 10 monclient: dump: 2024-06-11T19:22:08.673 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.674 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.675 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 10 monclient: shutdown 2024-06-11T19:22:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 1 -- 172.21.15.98:0/4218496874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844134bd0 msgr2=0x7f28440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 1 --2- 172.21.15.98:0/4218496874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844134bd0 0x7f28440a3690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2830009870 tx=0x7f283002b8f0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 1 -- 172.21.15.98:0/4218496874 shutdown_connections 2024-06-11T19:22:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 1 --2- 172.21.15.98:0/4218496874 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844134bd0 0x7f28440a3690 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 1 --2- 172.21.15.98:0/4218496874 >> v2:172.21.15.117:3300/0 conn(0x7f28441347f0 0x7f284409e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 1 -- 172.21.15.98:0/4218496874 >> 172.21.15.98:0/4218496874 conn(0x7f28440a4780 msgr2=0x7f28440a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 1 -- 172.21.15.98:0/4218496874 shutdown_connections 2024-06-11T19:22:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.656+0000 7f284afcd640 1 -- 172.21.15.98:0/4218496874 wait complete. 2024-06-11T19:22:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.657+0000 7f284afcd640 10 monclient: build_initial_monmap 2024-06-11T19:22:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.657+0000 7f284afcd640 10 monclient: monmap: 2024-06-11T19:22:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:08.676 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:08.677 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:08.678 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.657+0000 7f284afcd640 1 librados: starting msgr at 2024-06-11T19:22:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.657+0000 7f284afcd640 1 librados: starting objecter 2024-06-11T19:22:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.657+0000 7f284afcd640 1 Processor -- start 2024-06-11T19:22:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.657+0000 7f284afcd640 1 -- start start 2024-06-11T19:22:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.657+0000 7f284afcd640 1 librados: setting wanted keys 2024-06-11T19:22:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.657+0000 7f284afcd640 1 librados: calling monclient init 2024-06-11T19:22:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.657+0000 7f284afcd640 10 monclient: init 2024-06-11T19:22:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 10 monclient: _add_conns ranks=[3,1,2,4,6,7,8,5,0] 2024-06-11T19:22:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28441347f0 0x7f28441cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 10 monclient(hunting): picked mon.noname-g con 0x7f28441347f0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2844134bd0 0x7f2844133950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 10 monclient(hunting): picked mon.noname-f con 0x7f2844134bd0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:22:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844135010 0x7f284411f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 10 monclient(hunting): picked mon.noname-b con 0x7f2844135010 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:08.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2844068c40 con 0x7f2844134bd0 2024-06-11T19:22:08.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2844068ac0 con 0x7f2844135010 2024-06-11T19:22:08.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f28440a42e0 con 0x7f28441347f0 2024-06-11T19:22:08.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:08.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284afcd640 10 monclient(hunting): authenticate will time out at 1538.465269s 2024-06-11T19:22:08.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844135010 0x7f284411f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844135010 0x7f284411f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57182/0 (socket says 172.21.15.98:57182) 2024-06-11T19:22:08.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 1 -- 172.21.15.98:0/2938344398 learned_addr learned my addr 172.21.15.98:0/2938344398 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:08.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 10 monclient(hunting): get_auth_request con 0x7f2844135010 auth_method 0 2024-06-11T19:22:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2843fff640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2844134bd0 0x7f2844133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2848d69640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28441347f0 0x7f28441cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2843fff640 10 monclient(hunting): get_auth_request con 0x7f2844134bd0 auth_method 0 2024-06-11T19:22:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2843fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2843fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2843fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2848d69640 10 monclient(hunting): get_auth_request con 0x7f28441347f0 auth_method 0 2024-06-11T19:22:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2848d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2848d69640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2848d69640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2843fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2843fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2843fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2848d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2848d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f2848d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 10 monclient(hunting): handle_auth_done global_id 84944 payload 824 2024-06-11T19:22:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 1 -- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28441347f0 msgr2=0x7f28441cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28441347f0 0x7f28441cdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 1 -- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2844134bd0 msgr2=0x7f2844133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2844134bd0 0x7f2844133950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 10 monclient: _finish_hunting 0 2024-06-11T19:22:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 1 monclient: found mon.noname-b 2024-06-11T19:22:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:22:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 1 -- 172.21.15.98:0/2938344398 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28440a36c0 con 0x7f2844135010 2024-06-11T19:22:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f284956a640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844135010 0x7f284411f370 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f283c00c6b0 tx=0x7f283c00cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f28417fa640 10 client.?.objecter ms_handle_connect 0x7f2844135010 2024-06-11T19:22:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.658+0000 7f28417fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f283c001770 con 0x7f2844135010 2024-06-11T19:22:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:22:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 10 monclient: dump: 2024-06-11T19:22:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.686 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 10 monclient: _finish_auth 0 2024-06-11T19:22:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 10 monclient: _check_auth_tickets 2024-06-11T19:22:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f283c003e00 con 0x7f2844135010 2024-06-11T19:22:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f283c0055c0 con 0x7f2844135010 2024-06-11T19:22:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f28417fa640 10 monclient: dump: 2024-06-11T19:22:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.687 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.687 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.687 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.687 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.687 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.687 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.687 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.687 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.687 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.687 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.688 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.688 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.688 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.688 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f2842ffd640 4 set_mon_vals no callback set 2024-06-11T19:22:08.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f284afcd640 5 monclient: authenticate success, global_id 84944 2024-06-11T19:22:08.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f284afcd640 10 monclient: _renew_subs 2024-06-11T19:22:08.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f284afcd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:08.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.662+0000 7f284afcd640 1 -- 172.21.15.98:0/2938344398 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2844121920 con 0x7f2844135010 2024-06-11T19:22:08.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.663+0000 7f284afcd640 10 client.84944.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.663+0000 7f284afcd640 10 monclient: _renew_subs 2024-06-11T19:22:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.663+0000 7f284afcd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.663+0000 7f284afcd640 1 -- 172.21.15.98:0/2938344398 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2844121040 con 0x7f2844135010 2024-06-11T19:22:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.663+0000 7f284afcd640 1 librados: init done 2024-06-11T19:22:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f284afcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f284afcd640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2844133e90 0x7f2844122f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f284afcd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f284afcd640 1 -- 172.21.15.98:0/2938344398 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f2844122cd0 con 0x7f2844133e90 2024-06-11T19:22:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f283c005110 con 0x7f2844135010 2024-06-11T19:22:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f2848d69640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2844133e90 0x7f2844122f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f28417fa640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f282803fdb0 0x7f2828042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f2848d69640 10 monclient: get_auth_request con 0x7f2844133e90 auth_method 0 2024-06-11T19:22:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f2848d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f2848d69640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f2848d69640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(514..514 src has 1..514) ==== 6690+0+0 (secure 0 0 0) 0x7f283c0513a0 con 0x7f2844135010 2024-06-11T19:22:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f28417fa640 10 client.84944.objecter ms_dispatch 0x7f2844000b90 osd_map(514..514 src has 1..514) 2024-06-11T19:22:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f28417fa640 3 client.84944.objecter handle_osd_map got epochs [514,514] > 0 2024-06-11T19:22:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f28417fa640 3 client.84944.objecter handle_osd_map decoding full epoch 514 2024-06-11T19:22:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f2848d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f2848d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f2848d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f28417fa640 20 client.84944.objecter dump_active .. 0 homeless 2024-06-11T19:22:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f2843fff640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f282803fdb0 0x7f2828042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f2843fff640 10 monclient: get_auth_request con 0x7f282803fdb0 auth_method 0 2024-06-11T19:22:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.664+0000 7f2848d69640 10 monclient(hunting): handle_auth_done global_id 23459 payload 824 2024-06-11T19:22:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f2848d69640 1 -- 172.21.15.98:0/2938344398 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f28441cee60 con 0x7f2844133e90 2024-06-11T19:22:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f2848d69640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2844133e90 0x7f2844122f90 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f28380118e0 tx=0x7f2838011da0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 10 client.84944.objecter ms_handle_connect 0x7f2844133e90 2024-06-11T19:22:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 10 client.84944.objecter resend_mon_ops 2024-06-11T19:22:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 10 client.84944.objecter ms_handle_connect 0x7f282803fdb0 2024-06-11T19:22:08.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f2843fff640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f282803fdb0 0x7f2828042270 secure :-1 s=READY pgs=4247 cs=0 l=1 rev1=1 crypto rx=0x7f2830036000 tx=0x7f283003c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2838019070 con 0x7f2844133e90 2024-06-11T19:22:08.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:22:08.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 10 monclient: dump: 2024-06-11T19:22:08.693 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.693 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.693 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.693 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.693 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.693 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.693 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.694 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.694 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.694 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.694 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.694 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.694 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.694 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.694 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.694 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2838004300 con 0x7f2844133e90 2024-06-11T19:22:08.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:08.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:08.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2844133e90 msgr2=0x7f2844122f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2844133e90 0x7f2844122f90 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f28380118e0 tx=0x7f2838011da0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2844133e90 msgr2=0x7f2844122f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:08.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.665+0000 7f28417fa640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f2844133e90 0x7f2844122f90 secure :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f28380118e0 tx=0x7f2838011da0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f284afcd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:08.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f284afcd640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f284411f8b0 0x7f284411fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f284afcd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f284afcd640 1 -- 172.21.15.98:0/2938344398 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f2844050830 con 0x7f284411f8b0 2024-06-11T19:22:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f2848d69640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f284411f8b0 0x7f284411fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f2848d69640 10 monclient: get_auth_request con 0x7f284411f8b0 auth_method 0 2024-06-11T19:22:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f2848d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f2848d69640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f2848d69640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f2848d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f2848d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f2848d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f2848d69640 10 monclient(hunting): handle_auth_done global_id 23468 payload 824 2024-06-11T19:22:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f2848d69640 1 -- 172.21.15.98:0/2938344398 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2844120200 con 0x7f284411f8b0 2024-06-11T19:22:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f2848d69640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f284411f8b0 0x7f284411fcc0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f28380118e0 tx=0x7f28380046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f28417fa640 10 client.84944.objecter ms_handle_connect 0x7f284411f8b0 2024-06-11T19:22:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f28417fa640 10 client.84944.objecter resend_mon_ops 2024-06-11T19:22:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2838019040 con 0x7f284411f8b0 2024-06-11T19:22:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.674+0000 7f28417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f28417fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:22:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f28417fa640 10 monclient: dump: 2024-06-11T19:22:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.701 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7f283801e450 con 0x7f284411f8b0 2024-06-11T19:22:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f28417fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f28417fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f284411f8b0 msgr2=0x7f284411fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f28417fa640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f284411f8b0 0x7f284411fcc0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f28380118e0 tx=0x7f28380046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f28417fa640 1 -- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f284411f8b0 msgr2=0x7f284411fcc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f28417fa640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f284411f8b0 0x7f284411fcc0 secure :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f28380118e0 tx=0x7f28380046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.702 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:08.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:08.702 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:08.702 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:08.702 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:08.702 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 778869, 2024-06-11T19:22:08.702 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:08.702 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:08.702 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:08.702 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.703 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:08.704 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.705 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.705 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.705 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.705 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.705 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.705 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.705 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.705 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.705 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.705 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.706 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:08.706 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:08.706 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:08.706 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:08.706 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:08.706 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:08.706 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:08.706 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:08.706 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:08.706 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:08.706 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.707 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:08.712 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:08.712 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.712 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.712 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.712 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:08.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:08.712 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:08.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:08.714 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.714 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.714 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.714 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:08.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:08.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:08.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:08.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.718 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:08.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:08.718 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:08.718 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:08.719 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:08.720 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:08.722 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.722 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:08.722 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:08.722 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.722 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.722 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:08.723 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:08.724 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.725 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.726 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.727 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:08.727 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 10 librados: watch_flush enter 2024-06-11T19:22:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 10 librados: watch_flush exit 2024-06-11T19:22:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 20 client.84944.objecter shutdown clearing up homeless session... 2024-06-11T19:22:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 10 client.84944.objecter successfully canceled tick 2024-06-11T19:22:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 1 -- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f282803fdb0 msgr2=0x7f2828042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f282803fdb0 0x7f2828042270 secure :-1 s=READY pgs=4247 cs=0 l=1 rev1=1 crypto rx=0x7f2830036000 tx=0x7f283003c040 comp rx=0 tx=0).stop 2024-06-11T19:22:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 10 monclient: shutdown 2024-06-11T19:22:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 1 -- 172.21.15.98:0/2938344398 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844135010 msgr2=0x7f284411f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 1 --2- 172.21.15.98:0/2938344398 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2844135010 0x7f284411f370 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f283c00c6b0 tx=0x7f283c00cb70 comp rx=0 tx=0).stop 2024-06-11T19:22:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284956a640 1 -- 172.21.15.98:0/2938344398 reap_dead start 2024-06-11T19:22:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284956a640 1 -- 172.21.15.98:0/2938344398 reap_dead start 2024-06-11T19:22:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 1 -- 172.21.15.98:0/2938344398 shutdown_connections 2024-06-11T19:22:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 1 -- 172.21.15.98:0/2938344398 >> 172.21.15.98:0/2938344398 conn(0x7f28440a4780 msgr2=0x7f28441bf5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 1 -- 172.21.15.98:0/2938344398 shutdown_connections 2024-06-11T19:22:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 1 -- 172.21.15.98:0/2938344398 wait complete. 2024-06-11T19:22:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.675+0000 7f284afcd640 1 librados: shutdown 2024-06-11T19:22:08.730 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:08.730 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:22:08.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:22:08.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.830+0000 7f8f792e1640 10 monclient: get_monmap_and_config 2024-06-11T19:22:08.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.830+0000 7f8f792e1640 10 monclient: build_initial_monmap 2024-06-11T19:22:08.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.830+0000 7f8f792e1640 10 monclient: monmap: 2024-06-11T19:22:08.830 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:08.830 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.830 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:08.831591+0000 2024-06-11T19:22:08.830 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:08.831591+0000 2024-06-11T19:22:08.830 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:08.830 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.830 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:08.830 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:08.830 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:08.830 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:08.830 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:08.830 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:08.831 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:08.831 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:08.831 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:08.831 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:08.831 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:08.831 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 1 Processor -- start 2024-06-11T19:22:08.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 1 -- start start 2024-06-11T19:22:08.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 10 monclient: init 2024-06-11T19:22:08.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:08.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 10 monclient: _add_conns ranks=[7,5,4,1,0,3,10,6,2,9,8] 2024-06-11T19:22:08.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f741347f0 0x7f8f7409e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 10 monclient(hunting): picked mon.noname-i con 0x7f8f741347f0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:08.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f74134bd0 0x7f8f740a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 10 monclient(hunting): picked mon.noname-h con 0x7f8f74134bd0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:08.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f74135010 0x7f8f740a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 10 monclient(hunting): picked mon.noname-c con 0x7f8f74135010 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f8f740687f0 con 0x7f8f74135010 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f8f7409d410 con 0x7f8f74134bd0 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8f7409d6f0 con 0x7f8f741347f0 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f792e1640 10 monclient(hunting): authenticate will time out at 1538.638690s 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f737fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f74135010 0x7f8f740a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f737fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f74135010 0x7f8f740a3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38082/0 (socket says 172.21.15.98:38082) 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f737fe640 1 -- 172.21.15.98:0/328196695 learned_addr learned my addr 172.21.15.98:0/328196695 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f737fe640 10 monclient(hunting): get_auth_request con 0x7f8f74135010 auth_method 0 2024-06-11T19:22:08.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f737fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.831+0000 7f8f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f727fc640 1 --2- 172.21.15.98:0/328196695 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f74134bd0 0x7f8f740a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f737fe640 10 monclient(hunting): handle_auth_done global_id 114187 payload 274 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f72ffd640 1 --2- 172.21.15.98:0/328196695 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f741347f0 0x7f8f7409e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f737fe640 1 -- 172.21.15.98:0/328196695 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f741347f0 msgr2=0x7f8f7409e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f737fe640 1 --2- 172.21.15.98:0/328196695 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f741347f0 0x7f8f7409e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f737fe640 1 -- 172.21.15.98:0/328196695 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f74134bd0 msgr2=0x7f8f740a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f737fe640 1 --2- 172.21.15.98:0/328196695 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f74134bd0 0x7f8f740a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f737fe640 10 monclient: _finish_hunting 0 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f737fe640 1 monclient: found mon.noname-c 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f737fe640 1 -- 172.21.15.98:0/328196695 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f7409d9d0 con 0x7f8f74135010 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f727fc640 10 monclient: get_auth_request con 0x7f8f74134bd0 auth_method 0 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f72ffd640 10 monclient: get_auth_request con 0x7f8f741347f0 auth_method 0 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f727fc640 1 --2- 172.21.15.98:0/328196695 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f74134bd0 0x7f8f740a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f72ffd640 1 --2- 172.21.15.98:0/328196695 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f741347f0 0x7f8f7409e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.832+0000 7f8f737fe640 1 --2- 172.21.15.98:0/328196695 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f74135010 0x7f8f740a3bd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c009e80 tx=0x7f8f6c02be00 comp rx=0 tx=0).ready entity=mon.4 client_cookie=2fb385dba764b82d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 1 -- 172.21.15.98:0/328196695 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f6c02cc10 con 0x7f8f74135010 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 10 monclient: dump: 2024-06-11T19:22:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.846 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.846 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.846 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.846 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.846 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.846 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.846 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.846 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.846 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.846 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.846 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 10 monclient: _finish_auth 0 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 1 -- 172.21.15.98:0/328196695 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f6c02cdd0 con 0x7f8f74135010 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 5 monclient: authenticate success, global_id 114187 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:08.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 1 -- 172.21.15.98:0/328196695 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f6c034860 con 0x7f8f74135010 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 4 set_mon_vals no callback set 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f71ffb640 10 monclient: dump: 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.848 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 10 monclient: shutdown 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 1 -- 172.21.15.98:0/328196695 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f74135010 msgr2=0x7f8f740a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 1 --2- 172.21.15.98:0/328196695 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f74135010 0x7f8f740a3bd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c009e80 tx=0x7f8f6c02be00 comp rx=0 tx=0).stop 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 1 -- 172.21.15.98:0/328196695 shutdown_connections 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 1 --2- 172.21.15.98:0/328196695 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f74135010 0x7f8f740a3bd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 1 --2- 172.21.15.98:0/328196695 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f8f74134bd0 0x7f8f740a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 1 --2- 172.21.15.98:0/328196695 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f741347f0 0x7f8f7409e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 1 -- 172.21.15.98:0/328196695 >> 172.21.15.98:0/328196695 conn(0x7f8f740a4780 msgr2=0x7f8f740a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 1 -- 172.21.15.98:0/328196695 shutdown_connections 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.833+0000 7f8f792e1640 1 -- 172.21.15.98:0/328196695 wait complete. 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.834+0000 7f8f792e1640 10 monclient: build_initial_monmap 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.834+0000 7f8f792e1640 10 monclient: monmap: 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:08.850 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.834+0000 7f8f792e1640 1 librados: starting msgr at 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.834+0000 7f8f792e1640 1 librados: starting objecter 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.834+0000 7f8f792e1640 1 Processor -- start 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.834+0000 7f8f792e1640 1 -- start start 2024-06-11T19:22:08.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.834+0000 7f8f792e1640 1 librados: setting wanted keys 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.834+0000 7f8f792e1640 1 librados: calling monclient init 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.834+0000 7f8f792e1640 10 monclient: init 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 10 monclient: _add_conns ranks=[3,0,8,6,5,4,1,7,2] 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f741347f0 0x7f8f741cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 10 monclient(hunting): picked mon.noname-g con 0x7f8f741347f0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f74134bd0 0x7f8f741ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 10 monclient(hunting): picked mon.noname-a con 0x7f8f74134bd0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f74135010 0x7f8f7411eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 10 monclient(hunting): picked mon.noname-e con 0x7f8f74135010 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f74068c40 con 0x7f8f74134bd0 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f8f74068ac0 con 0x7f8f741347f0 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8f740a4110 con 0x7f8f74135010 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 10 monclient(hunting): authenticate will time out at 1538.642113s 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f727fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f74134bd0 0x7f8f741ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f74135010 0x7f8f7411eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f74135010 0x7f8f7411eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:58064/0 (socket says 172.21.15.98:58064) 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f727fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f74134bd0 0x7f8f741ce0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52256/0 (socket says 172.21.15.98:52256) 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 1 -- 172.21.15.98:0/1387168923 learned_addr learned my addr 172.21.15.98:0/1387168923 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 10 monclient(hunting): get_auth_request con 0x7f8f74135010 auth_method 0 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f727fc640 10 monclient(hunting): get_auth_request con 0x7f8f74134bd0 auth_method 0 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f727fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f727fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f72ffd640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f741347f0 0x7f8f741cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f72ffd640 10 monclient(hunting): get_auth_request con 0x7f8f741347f0 auth_method 0 2024-06-11T19:22:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f72ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 10 monclient(hunting): handle_auth_done global_id 114146 payload 824 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 1 -- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f741347f0 msgr2=0x7f8f741cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f741347f0 0x7f8f741cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 1 -- 172.21.15.98:0/1387168923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f74134bd0 msgr2=0x7f8f741ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f74134bd0 0x7f8f741ce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 10 monclient: _finish_hunting 0 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 1 monclient: found mon.noname-e 2024-06-11T19:22:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 1 -- 172.21.15.98:0/1387168923 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f74122720 con 0x7f8f74135010 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f727fc640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f74134bd0 0x7f8f741ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f737fe640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f74135010 0x7f8f7411eb10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c036040 tx=0x7f8f6c004a40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 10 client.?.objecter ms_handle_connect 0x7f8f74135010 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f6c044030 con 0x7f8f74135010 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 10 monclient: dump: 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f6c0046c0 con 0x7f8f74135010 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 5 monclient: authenticate success, global_id 114146 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 10 monclient: _renew_subs 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f792e1640 1 -- 172.21.15.98:0/1387168923 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f74121270 con 0x7f8f74135010 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f6c009510 con 0x7f8f74135010 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f5b7fe640 10 monclient: dump: 2024-06-11T19:22:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.836+0000 7f8f792e1640 10 client.114146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.836+0000 7f8f792e1640 10 monclient: _renew_subs 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.836+0000 7f8f792e1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.836+0000 7f8f792e1640 1 -- 172.21.15.98:0/1387168923 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f74136000 con 0x7f8f74135010 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.836+0000 7f8f792e1640 1 librados: init done 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.835+0000 7f8f717fa640 4 set_mon_vals no callback set 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.836+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8f6c047070 con 0x7f8f74135010 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.836+0000 7f8f792e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.836+0000 7f8f792e1640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f050 0x7f8f7411f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.836+0000 7f8f792e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.836+0000 7f8f792e1640 1 -- 172.21.15.98:0/1387168923 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8f74120ca0 con 0x7f8f7411f050 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8f6003fdb0 0x7f8f60042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f72ffd640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f050 0x7f8f7411f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f72ffd640 10 monclient: get_auth_request con 0x7f8f7411f050 auth_method 0 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(514..514 src has 1..514) ==== 6690+0+0 (secure 0 0 0) 0x7f8f6c045070 con 0x7f8f74135010 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f72ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 10 client.114146.objecter ms_dispatch 0x7f8f74000b90 osd_map(514..514 src has 1..514) 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 3 client.114146.objecter handle_osd_map got epochs [514,514] > 0 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 3 client.114146.objecter handle_osd_map decoding full epoch 514 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 20 client.114146.objecter dump_active .. 0 homeless 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f727fc640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8f6003fdb0 0x7f8f60042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f727fc640 10 monclient: get_auth_request con 0x7f8f6003fdb0 auth_method 0 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f72ffd640 10 monclient(hunting): handle_auth_done global_id 84697 payload 824 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f72ffd640 1 -- 172.21.15.98:0/1387168923 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f740a35a0 con 0x7f8f7411f050 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 10 client.114146.objecter ms_handle_connect 0x7f8f7411f050 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 10 client.114146.objecter resend_mon_ops 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f72ffd640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f050 0x7f8f7411f430 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400f880 tx=0x7f8f6400fd40 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 10 client.114146.objecter ms_handle_connect 0x7f8f6003fdb0 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f727fc640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8f6003fdb0 0x7f8f60042270 secure :-1 s=READY pgs=4248 cs=0 l=1 rev1=1 crypto rx=0x7f8f741201f0 tx=0x7f8f68008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f6400de90 con 0x7f8f7411f050 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.837+0000 7f8f5b7fe640 10 monclient: dump: 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.864 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.838+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8f6400da20 con 0x7f8f7411f050 2024-06-11T19:22:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.838+0000 7f8f5b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.838+0000 7f8f5b7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.838+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f050 msgr2=0x7f8f7411f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.838+0000 7f8f5b7fe640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f050 0x7f8f7411f430 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400f880 tx=0x7f8f6400fd40 comp rx=0 tx=0).stop 2024-06-11T19:22:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.838+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f050 msgr2=0x7f8f7411f430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.838+0000 7f8f5b7fe640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f050 0x7f8f7411f430 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400f880 tx=0x7f8f6400fd40 comp rx=0 tx=0).stop 2024-06-11T19:22:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f792e1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f792e1640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f970 0x7f8f7409f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f792e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f792e1640 1 -- 172.21.15.98:0/1387168923 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8f74050830 con 0x7f8f7411f970 2024-06-11T19:22:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f72ffd640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f970 0x7f8f7409f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f72ffd640 10 monclient: get_auth_request con 0x7f8f7411f970 auth_method 0 2024-06-11T19:22:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f72ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.844+0000 7f8f72ffd640 10 monclient(hunting): handle_auth_done global_id 84706 payload 824 2024-06-11T19:22:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f72ffd640 1 -- 172.21.15.98:0/1387168923 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8f7409e6b0 con 0x7f8f7411f970 2024-06-11T19:22:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f72ffd640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f970 0x7f8f7409f080 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400f880 tx=0x7f8f640046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 10 client.114146.objecter ms_handle_connect 0x7f8f7411f970 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 10 client.114146.objecter resend_mon_ops 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f64006730 con 0x7f8f7411f970 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 10 monclient: dump: 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:08.869 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f8f64006730 con 0x7f8f7411f970 2024-06-11T19:22:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f970 msgr2=0x7f8f7409f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f970 0x7f8f7409f080 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400f880 tx=0x7f8f640046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 1 -- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f970 msgr2=0x7f8f7409f080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f5b7fe640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8f7411f970 0x7f8f7409f080 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400f880 tx=0x7f8f640046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:08.870 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:08.870 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:08.870 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:08.870 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:08.870 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 400614, 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.871 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.872 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.873 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.874 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.874 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.874 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:08.874 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:08.874 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:08.874 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:08.874 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:08.874 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:08.874 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:08.874 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:08.874 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:08.875 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:08.876 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.881 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.882 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:08.884 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.887 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:08.889 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.890 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.890 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.890 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:08.890 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:08.890 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:08.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:08.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.890 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:08.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:08.890 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.891 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:08.893 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:08.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:08.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:08.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:08.893 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:08.893 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f792e1640 10 librados: watch_flush enter 2024-06-11T19:22:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f792e1640 10 librados: watch_flush exit 2024-06-11T19:22:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f792e1640 20 client.114146.objecter shutdown clearing up homeless session... 2024-06-11T19:22:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.845+0000 7f8f792e1640 10 client.114146.objecter successfully canceled tick 2024-06-11T19:22:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f792e1640 1 -- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8f6003fdb0 msgr2=0x7f8f60042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f792e1640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8f6003fdb0 0x7f8f60042270 secure :-1 s=READY pgs=4248 cs=0 l=1 rev1=1 crypto rx=0x7f8f741201f0 tx=0x7f8f68008040 comp rx=0 tx=0).stop 2024-06-11T19:22:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f792e1640 10 monclient: shutdown 2024-06-11T19:22:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f792e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f792e1640 1 -- 172.21.15.98:0/1387168923 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f74135010 msgr2=0x7f8f7411eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f792e1640 1 --2- 172.21.15.98:0/1387168923 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f74135010 0x7f8f7411eb10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c036040 tx=0x7f8f6c004a40 comp rx=0 tx=0).stop 2024-06-11T19:22:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f737fe640 1 -- 172.21.15.98:0/1387168923 reap_dead start 2024-06-11T19:22:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f737fe640 1 -- 172.21.15.98:0/1387168923 reap_dead start 2024-06-11T19:22:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f792e1640 1 -- 172.21.15.98:0/1387168923 shutdown_connections 2024-06-11T19:22:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f792e1640 1 -- 172.21.15.98:0/1387168923 >> 172.21.15.98:0/1387168923 conn(0x7f8f740a4780 msgr2=0x7f8f741bf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f792e1640 1 -- 172.21.15.98:0/1387168923 shutdown_connections 2024-06-11T19:22:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f792e1640 1 -- 172.21.15.98:0/1387168923 wait complete. 2024-06-11T19:22:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.846+0000 7f8f792e1640 1 librados: shutdown 2024-06-11T19:22:08.896 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:08.896 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-06-11T19:22:08.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-06-11T19:22:08.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.996+0000 7f699ed3f640 10 monclient: get_monmap_and_config 2024-06-11T19:22:08.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.996+0000 7f699ed3f640 10 monclient: build_initial_monmap 2024-06-11T19:22:08.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.996+0000 7f699ed3f640 10 monclient: monmap: 2024-06-11T19:22:08.995 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:08.995 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:08.995 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:08.997085+0000 2024-06-11T19:22:08.995 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:08.997085+0000 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:08.996 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:09.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.996+0000 7f699ed3f640 1 Processor -- start 2024-06-11T19:22:09.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.996+0000 7f699ed3f640 1 -- start start 2024-06-11T19:22:09.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.996+0000 7f699ed3f640 10 monclient: init 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 10 monclient: _add_conns ranks=[10,5,9,8,4,1,0,2,6,3,7] 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f69980a2f40 0x7f69980a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 10 monclient(hunting): picked mon.noname-f con 0x7f69980a2f40 addr v2:172.21.15.117:3300/0 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f69980a4df0 0x7f69980a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 10 monclient(hunting): picked mon.noname-h con 0x7f69980a4df0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f69981366d0 0x7f6998136ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 10 monclient(hunting): picked mon.noname-a con 0x7f69981366d0 addr v2:172.21.15.98:3300/0 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f69980a5710 con 0x7f69981366d0 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f69980a3320 con 0x7f69980a2f40 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f69980a2460 con 0x7f69980a4df0 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699ed3f640 10 monclient(hunting): authenticate will time out at 1538.804371s 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f69981366d0 0x7f6998136ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f69981366d0 0x7f6998136ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52258/0 (socket says 172.21.15.98:52258) 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 1 -- 172.21.15.98:0/4051659441 learned_addr learned my addr 172.21.15.98:0/4051659441 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 10 monclient(hunting): get_auth_request con 0x7f69981366d0 auth_method 0 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699cadb640 1 --2- 172.21.15.98:0/4051659441 >> v2:172.21.15.117:3300/0 conn(0x7f69980a2f40 0x7f69980a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699cadb640 10 monclient(hunting): get_auth_request con 0x7f69980a2f40 auth_method 0 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699cadb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699cadb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699cadb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f6997fff640 1 --2- 172.21.15.98:0/4051659441 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f69980a4df0 0x7f69980a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f6997fff640 10 monclient(hunting): get_auth_request con 0x7f69980a4df0 auth_method 0 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f6997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f6997fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f6997fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 10 monclient(hunting): handle_auth_done global_id 65700 payload 274 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 1 -- 172.21.15.98:0/4051659441 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f69980a4df0 msgr2=0x7f69980a51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 1 --2- 172.21.15.98:0/4051659441 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f69980a4df0 0x7f69980a51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 1 -- 172.21.15.98:0/4051659441 >> v2:172.21.15.117:3300/0 conn(0x7f69980a2f40 msgr2=0x7f69980a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 1 --2- 172.21.15.98:0/4051659441 >> v2:172.21.15.117:3300/0 conn(0x7f69980a2f40 0x7f69980a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 10 monclient: _finish_hunting 0 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 1 monclient: found mon.noname-a 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.997+0000 7f699d2dc640 1 -- 172.21.15.98:0/4051659441 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69980a0ed0 con 0x7f69981366d0 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f699d2dc640 1 --2- 172.21.15.98:0/4051659441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69981366d0 0x7f6998136ae0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f69980a3560 tx=0x7f699002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb65f5497800e4df server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 1 -- 172.21.15.98:0/4051659441 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6990038070 con 0x7f69981366d0 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 10 monclient: dump: 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 1 -- 172.21.15.98:0/4051659441 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6990031990 con 0x7f69981366d0 2024-06-11T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f699ed3f640 5 monclient: authenticate success, global_id 65700 2024-06-11T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f699ed3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 1 -- 172.21.15.98:0/4051659441 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f69900323b0 con 0x7f69981366d0 2024-06-11T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f699ed3f640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f699ed3f640 4 set_mon_vals no callback set 2024-06-11T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.998+0000 7f69977fe640 10 monclient: dump: 2024-06-11T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 10 monclient: shutdown 2024-06-11T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 1 -- 172.21.15.98:0/4051659441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69981366d0 msgr2=0x7f6998136ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 1 --2- 172.21.15.98:0/4051659441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69981366d0 0x7f6998136ae0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f69980a3560 tx=0x7f699002b7c0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 1 -- 172.21.15.98:0/4051659441 shutdown_connections 2024-06-11T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 1 --2- 172.21.15.98:0/4051659441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69981366d0 0x7f6998136ae0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 1 --2- 172.21.15.98:0/4051659441 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f69980a4df0 0x7f69980a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 1 --2- 172.21.15.98:0/4051659441 >> v2:172.21.15.117:3300/0 conn(0x7f69980a2f40 0x7f69980a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 1 -- 172.21.15.98:0/4051659441 >> 172.21.15.98:0/4051659441 conn(0x7f69980a95b0 msgr2=0x7f69980a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 1 -- 172.21.15.98:0/4051659441 shutdown_connections 2024-06-11T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 1 -- 172.21.15.98:0/4051659441 wait complete. 2024-06-11T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 10 monclient: build_initial_monmap 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:08.999+0000 7f699ed3f640 10 monclient: monmap: 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 1 librados: starting msgr at 2024-06-11T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 1 librados: starting objecter 2024-06-11T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 1 Processor -- start 2024-06-11T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 1 -- start start 2024-06-11T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 1 librados: setting wanted keys 2024-06-11T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 1 librados: calling monclient init 2024-06-11T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 10 monclient: init 2024-06-11T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 10 monclient: _add_conns ranks=[5,2,4,8,1,3,6,7,0] 2024-06-11T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f69980a2f40 0x7f699806bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 10 monclient(hunting): picked mon.noname-h con 0x7f69980a2f40 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69980a4df0 0x7f699806c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 10 monclient(hunting): picked mon.noname-b con 0x7f69980a4df0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f69981366d0 0x7f699806c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 10 monclient(hunting): picked mon.noname-c con 0x7f69981366d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f69980a1250 con 0x7f69980a4df0 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f6998137020 con 0x7f69981366d0 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f69980a7fd0 con 0x7f69980a2f40 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699ed3f640 10 monclient(hunting): authenticate will time out at 1538.807878s 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699d2dc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f69981366d0 0x7f699806c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f6997fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69980a4df0 0x7f699806c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699d2dc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f69981366d0 0x7f699806c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38098/0 (socket says 172.21.15.98:38098) 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.000+0000 7f699d2dc640 1 -- 172.21.15.98:0/2687936699 learned_addr learned my addr 172.21.15.98:0/2687936699 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 10 monclient(hunting): get_auth_request con 0x7f69981366d0 auth_method 0 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f6997fff640 10 monclient(hunting): get_auth_request con 0x7f69980a4df0 auth_method 0 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f6997fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f6997fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f6997fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699cadb640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f69980a2f40 0x7f699806bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699cadb640 10 monclient(hunting): get_auth_request con 0x7f69980a2f40 auth_method 0 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699cadb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699cadb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699cadb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f6997fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f6997fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f6997fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699cadb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699cadb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699cadb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 10 monclient(hunting): handle_auth_done global_id 114196 payload 824 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 1 -- 172.21.15.98:0/2687936699 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f69980a2f40 msgr2=0x7f699806bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f69980a2f40 0x7f699806bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 1 -- 172.21.15.98:0/2687936699 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69980a4df0 msgr2=0x7f699806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69980a4df0 0x7f699806c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 10 monclient: _finish_hunting 0 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 1 monclient: found mon.noname-c 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 1 -- 172.21.15.98:0/2687936699 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699806dbf0 con 0x7f69981366d0 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f6997fff640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69980a4df0 0x7f699806c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699d2dc640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f69981366d0 0x7f699806c9e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f69900090b0 tx=0x7f69900046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 10 client.?.objecter ms_handle_connect 0x7f69981366d0 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 1 -- 172.21.15.98:0/2687936699 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6990042020 con 0x7f69981366d0 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 10 monclient: dump: 2024-06-11T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:09.019 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 10 monclient: _finish_auth 0 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 10 monclient: _check_auth_tickets 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699ed3f640 5 monclient: authenticate success, global_id 114196 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 1 -- 172.21.15.98:0/2687936699 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6990004280 con 0x7f69981366d0 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 1 -- 172.21.15.98:0/2687936699 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6990038040 con 0x7f69981366d0 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f69957fa640 10 monclient: dump: 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699ed3f640 10 monclient: _renew_subs 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f6996ffd640 4 set_mon_vals no callback set 2024-06-11T19:22:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699ed3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699ed3f640 1 -- 172.21.15.98:0/2687936699 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69980a53d0 con 0x7f69981366d0 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699ed3f640 10 client.114196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699ed3f640 10 monclient: _renew_subs 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699ed3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699ed3f640 1 -- 172.21.15.98:0/2687936699 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f6998135ff0 con 0x7f69981366d0 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.001+0000 7f699ed3f640 1 librados: init done 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f69957fa640 1 -- 172.21.15.98:0/2687936699 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f6990045020 con 0x7f69981366d0 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f699ed3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f699ed3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f699ed3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f699ed3f640 1 -- 172.21.15.98:0/2687936699 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69980a51d0 con 0x7f69981366d0 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f69957fa640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f698403fdb0 0x7f6984042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f69957fa640 1 -- 172.21.15.98:0/2687936699 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(514..514 src has 1..514) ==== 6690+0+0 (secure 0 0 0) 0x7f6990030070 con 0x7f69981366d0 2024-06-11T19:22:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f69957fa640 10 client.114196.objecter ms_dispatch 0x7f6998000b90 osd_map(514..514 src has 1..514) 2024-06-11T19:22:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f69957fa640 3 client.114196.objecter handle_osd_map got epochs [514,514] > 0 2024-06-11T19:22:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f69957fa640 3 client.114196.objecter handle_osd_map decoding full epoch 514 2024-06-11T19:22:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f699cadb640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f698403fdb0 0x7f6984042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f69957fa640 20 client.114196.objecter dump_active .. 0 homeless 2024-06-11T19:22:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.002+0000 7f699cadb640 10 monclient: get_auth_request con 0x7f698403fdb0 auth_method 0 2024-06-11T19:22:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.003+0000 7f699cadb640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f698403fdb0 0x7f6984042270 secure :-1 s=READY pgs=4249 cs=0 l=1 rev1=1 crypto rx=0x7f6988006fd0 tx=0x7f6988008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.003+0000 7f69957fa640 10 client.114196.objecter ms_handle_connect 0x7f698403fdb0 2024-06-11T19:22:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.007+0000 7f69957fa640 1 -- 172.21.15.98:0/2687936699 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f69980a51d0 con 0x7f69981366d0 2024-06-11T19:22:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.007+0000 7f69957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.007+0000 7f69957fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.262+0000 7f699ed3f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-06-11T19:22:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.262+0000 7f699ed3f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-06-11T19:22:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.262+0000 7f699ed3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.262+0000 7f699ed3f640 1 -- 172.21.15.98:0/2687936699 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f69981373b0 con 0x7f69981366d0 2024-06-11T19:22:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.265+0000 7f69957fa640 1 -- 172.21.15.98:0/2687936699 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f699004c070 con 0x7f69981366d0 2024-06-11T19:22:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.265+0000 7f69957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-06-11T19:22:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.265+0000 7f69957fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:09.266 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.266 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=397803579,config=692619927,health=3233892354,kv=2000936593,logm=392942143}) 2024-06-11T19:22:09.266 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=1,health=10,kv=10,logm=60} crc {auth=3950510274,config=3678701417,health=3869265827,kv=2000936593,logm=3215040852}) 2024-06-11T19:22:09.266 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.266 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=397803579,config=692619927,health=3233892354,kv=2000936593,logm=392942143}) 2024-06-11T19:22:09.266 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=11,kv=8,logm=62} crc {auth=1829957908,config=692619927,health=3345235253,kv=863197348,logm=1425041697}) 2024-06-11T19:22:09.266 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.266 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=397803579,config=692619927,health=3233892354,kv=2000936593,logm=392942143}) 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=19,config=2,health=10,kv=9,logm=60} crc {auth=4137556905,config=692619927,health=3790403369,kv=1217681957,logm=1844486003}) 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=397803579,config=692619927,health=3233892354,kv=2000936593,logm=392942143}) 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=19,config=2,health=12,kv=10,logm=57} crc {auth=594663285,config=692619927,health=287983214,kv=2000936593,logm=1540475894}) 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=397803579,config=692619927,health=3233892354,kv=2000936593,logm=392942143}) 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=18,config=2,health=10,kv=9,logm=61} crc {auth=2165867059,config=692619927,health=1422203095,kv=3604117662,logm=119381265}) 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=397803579,config=692619927,health=3233892354,kv=2000936593,logm=392942143}) 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=19,config=1,health=12,kv=9,logm=59} crc {auth=3781646966,config=1228700967,health=287983214,kv=1471832156,logm=1134132571}) 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=397803579,config=692619927,health=3233892354,kv=2000936593,logm=392942143}) 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=1278509752,config=692619927,health=3488154960,kv=2000936593,logm=1927492732}) 2024-06-11T19:22:09.267 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.268 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=397803579,config=692619927,health=3233892354,kv=2000936593,logm=392942143}) 2024-06-11T19:22:09.268 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.264+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=20,config=1,health=12,kv=10,logm=57} crc {auth=1278509752,config=3678701417,health=287983214,kv=2000936593,logm=3704954097}) 2024-06-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 10 librados: watch_flush enter 2024-06-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 10 librados: watch_flush exit 2024-06-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 20 client.114196.objecter shutdown clearing up homeless session... 2024-06-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 10 client.114196.objecter successfully canceled tick 2024-06-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 1 -- 172.21.15.98:0/2687936699 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f698403fdb0 msgr2=0x7f6984042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f698403fdb0 0x7f6984042270 secure :-1 s=READY pgs=4249 cs=0 l=1 rev1=1 crypto rx=0x7f6988006fd0 tx=0x7f6988008040 comp rx=0 tx=0).stop 2024-06-11T19:22:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 10 monclient: shutdown 2024-06-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 1 -- 172.21.15.98:0/2687936699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f69981366d0 msgr2=0x7f699806c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f69981366d0 0x7f699806c9e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f69900090b0 tx=0x7f69900046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 1 -- 172.21.15.98:0/2687936699 shutdown_connections 2024-06-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f698403fdb0 0x7f6984042270 unknown :-1 s=CLOSED pgs=4249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f69981366d0 0x7f699806c9e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f69980a4df0 0x7f699806c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 1 --2- 172.21.15.98:0/2687936699 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f69980a2f40 0x7f699806bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.267+0000 7f699ed3f640 1 -- 172.21.15.98:0/2687936699 >> 172.21.15.98:0/2687936699 conn(0x7f69980a95b0 msgr2=0x7f69981bd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.268+0000 7f699ed3f640 1 -- 172.21.15.98:0/2687936699 shutdown_connections 2024-06-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.268+0000 7f699ed3f640 1 -- 172.21.15.98:0/2687936699 wait complete. 2024-06-11T19:22:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:09.268+0000 7f699ed3f640 1 librados: shutdown 2024-06-11T19:22:09.288 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-06-11T19:22:09.491 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.266+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.491 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.266+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694827765}) 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.266+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095594549}) 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.266+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.266+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694827765}) 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.266+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3278245678}) 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.266+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.266+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694827765}) 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1829344409}) 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694827765}) 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1880292502}) 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694827765}) 2024-06-11T19:22:09.492 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3180795702}) 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694827765}) 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=429823096}) 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694827765}) 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3611937651}) 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694827765}) 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.267+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2597745962}) 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960410506}) 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061388371}) 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.493 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960410506}) 2024-06-11T19:22:09.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2017894562}) 2024-06-11T19:22:09.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960410506}) 2024-06-11T19:22:09.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2091345042}) 2024-06-11T19:22:09.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960410506}) 2024-06-11T19:22:09.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3790318246}) 2024-06-11T19:22:09.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960410506}) 2024-06-11T19:22:09.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=425297720}) 2024-06-11T19:22:09.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.494 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960410506}) 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3940643227}) 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960410506}) 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1735945070}) 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960410506}) 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.269+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3759976094}) 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.271+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.271+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441310351}) 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.271+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2658687071}) 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.271+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.271+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441310351}) 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.271+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1296246735}) 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.271+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.271+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441310351}) 2024-06-11T19:22:09.495 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.271+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1633897524}) 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441310351}) 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2487038829}) 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441310351}) 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1002515782}) 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441310351}) 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2815433690}) 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441310351}) 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2340030215}) 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.496 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441310351}) 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.272+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1235344467}) 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.273+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.273+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016529253}) 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.273+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449686999}) 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.273+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016529253}) 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=901005323}) 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016529253}) 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4118844509}) 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016529253}) 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3567394148}) 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.497 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016529253}) 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1986387206}) 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016529253}) 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=220831169}) 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016529253}) 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=108883248}) 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016529253}) 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.274+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=342772154}) 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=302341403}) 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004101472}) 2024-06-11T19:22:09.498 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=302341403}) 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=776764282}) 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=302341403}) 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4079910944}) 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=302341403}) 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4098191654}) 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=302341403}) 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3165170470}) 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.499 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=302341403}) 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3461967993}) 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.275+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=302341403}) 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.276+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3793774346}) 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.276+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.276+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=302341403}) 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.276+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=619446819}) 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1266622721}) 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3593706433}) 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1266622721}) 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290935141}) 2024-06-11T19:22:09.500 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.501 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1266622721}) 2024-06-11T19:22:09.501 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3245545364}) 2024-06-11T19:22:09.501 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.501 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1266622721}) 2024-06-11T19:22:09.501 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3881612697}) 2024-06-11T19:22:09.501 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.501 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1266622721}) 2024-06-11T19:22:09.501 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2415273814}) 2024-06-11T19:22:09.501 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.501 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1266622721}) 2024-06-11T19:22:09.501 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2675503194}) 2024-06-11T19:22:09.501 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1266622721}) 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=766118031}) 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1266622721}) 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.277+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1518019723}) 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195129320}) 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=140948305}) 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195129320}) 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269860482}) 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.502 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195129320}) 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=565206074}) 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195129320}) 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3350020090}) 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195129320}) 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=650929051}) 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195129320}) 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1513306689}) 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195129320}) 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=750893115}) 2024-06-11T19:22:09.503 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195129320}) 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.279+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2860389529}) 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=151875698}) 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2330189351}) 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=151875698}) 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2750945083}) 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=151875698}) 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4027022754}) 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=151875698}) 2024-06-11T19:22:09.504 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=692728786}) 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=151875698}) 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1076778673}) 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=151875698}) 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3606939188}) 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=151875698}) 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2044460055}) 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=151875698}) 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.280+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3962053277}) 2024-06-11T19:22:09.505 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.281+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.281+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270259863}) 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.281+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=670654711}) 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.281+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.281+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270259863}) 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2840380034}) 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270259863}) 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2662808176}) 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270259863}) 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1033009770}) 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.506 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270259863}) 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1300077635}) 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270259863}) 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=879148020}) 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270259863}) 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4137915037}) 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270259863}) 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.282+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1455052601}) 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10760530}) 2024-06-11T19:22:09.507 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=282893689}) 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10760530}) 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018965635}) 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10760530}) 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3037955528}) 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10760530}) 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2773442252}) 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10760530}) 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2851104479}) 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.508 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10760530}) 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4221090616}) 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10760530}) 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3435192625}) 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10760530}) 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.283+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3105173282}) 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.284+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.284+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584280323}) 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.284+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192850816}) 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.284+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.284+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584280323}) 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.284+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186090675}) 2024-06-11T19:22:09.509 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584280323}) 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1416861358}) 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584280323}) 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1662075381}) 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584280323}) 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1005068297}) 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584280323}) 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4198876463}) 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584280323}) 2024-06-11T19:22:09.510 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2987830686}) 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584280323}) 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.285+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1736323001}) 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077612580}) 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991744558}) 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077612580}) 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=600590215}) 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077612580}) 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2125954534}) 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.511 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077612580}) 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2735346740}) 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077612580}) 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2287243860}) 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077612580}) 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1128866406}) 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077612580}) 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=652750273}) 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077612580}) 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.286+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1173315383}) 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.512 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091123669}) 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2150671513}) 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091123669}) 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3305606767}) 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091123669}) 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=318644253}) 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091123669}) 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1879189187}) 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091123669}) 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1925902478}) 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.287+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.513 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.288+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091123669}) 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.288+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3377218479}) 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.288+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.288+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091123669}) 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.288+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1957139979}) 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.288+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.288+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091123669}) 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.288+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=951899606}) 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572521061}) 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2647912776}) 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572521061}) 2024-06-11T19:22:09.514 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2230559271}) 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572521061}) 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1429115759}) 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572521061}) 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3793735809}) 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572521061}) 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1318791715}) 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572521061}) 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3135428572}) 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.515 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572521061}) 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=94789535}) 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572521061}) 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.289+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1036042504}) 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996493454}) 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3607640331}) 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996493454}) 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=902212978}) 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996493454}) 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3863270078}) 2024-06-11T19:22:09.516 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.517 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996493454}) 2024-06-11T19:22:09.517 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2445488055}) 2024-06-11T19:22:09.517 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.517 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996493454}) 2024-06-11T19:22:09.517 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2285706032}) 2024-06-11T19:22:09.517 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.517 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996493454}) 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3589520246}) 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996493454}) 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1526225563}) 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996493454}) 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.290+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1369877870}) 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4202165816}) 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=183620150}) 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4202165816}) 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3146601635}) 2024-06-11T19:22:09.518 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4202165816}) 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2130580086}) 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4202165816}) 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1508816140}) 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4202165816}) 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3194942383}) 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4202165816}) 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=49915606}) 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4202165816}) 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1490973923}) 2024-06-11T19:22:09.519 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4202165816}) 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.292+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1104696316}) 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936170090}) 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8324649}) 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936170090}) 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2283566329}) 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936170090}) 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3806517721}) 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936170090}) 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4115187251}) 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.520 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936170090}) 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3056862794}) 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936170090}) 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3389286056}) 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936170090}) 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3280508555}) 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936170090}) 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.293+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1543014576}) 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279455810}) 2024-06-11T19:22:09.521 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3238437443}) 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279455810}) 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=379605506}) 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279455810}) 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1243512472}) 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279455810}) 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1338101096}) 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279455810}) 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1044078163}) 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.522 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279455810}) 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=557831870}) 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279455810}) 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4223040422}) 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279455810}) 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.295+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2385896329}) 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225570533}) 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3882549098}) 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225570533}) 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2458555517}) 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.523 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225570533}) 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=228915106}) 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225570533}) 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2815951047}) 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225570533}) 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4058730001}) 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225570533}) 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2470999526}) 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225570533}) 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2895146482}) 2024-06-11T19:22:09.524 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225570533}) 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.296+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1881957570}) 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.297+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.297+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838838918}) 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.297+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4212700310}) 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.297+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.297+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838838918}) 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.297+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3994914339}) 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.297+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.297+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838838918}) 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.297+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3639215859}) 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.297+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.297+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838838918}) 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=869187256}) 2024-06-11T19:22:09.525 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838838918}) 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3610338547}) 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838838918}) 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2037995510}) 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838838918}) 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3126233753}) 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838838918}) 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.298+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4040304507}) 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.526 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40195783}) 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4245914481}) 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40195783}) 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284926081}) 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40195783}) 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2074016979}) 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40195783}) 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=182883431}) 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40195783}) 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1422351838}) 2024-06-11T19:22:09.527 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40195783}) 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=214930260}) 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40195783}) 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2752790547}) 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40195783}) 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.299+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=394557520}) 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787637555}) 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656295622}) 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787637555}) 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2260812387}) 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.528 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787637555}) 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2355978947}) 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787637555}) 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1684418867}) 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787637555}) 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3116590583}) 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787637555}) 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1535712831}) 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787637555}) 2024-06-11T19:22:09.529 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1930255322}) 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787637555}) 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.300+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3544776021}) 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521418558}) 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1441281173}) 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521418558}) 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4228516457}) 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521418558}) 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=903727641}) 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.530 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521418558}) 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=119067213}) 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521418558}) 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2892647975}) 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521418558}) 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1923297572}) 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521418558}) 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2570962611}) 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521418558}) 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.302+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3866273326}) 2024-06-11T19:22:09.531 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968158606}) 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304871608}) 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968158606}) 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2438652942}) 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968158606}) 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=133218668}) 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968158606}) 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=353123166}) 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.532 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968158606}) 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2104593180}) 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968158606}) 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4022124437}) 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968158606}) 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=452027065}) 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968158606}) 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.303+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3284335710}) 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.304+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.304+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788582096}) 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.304+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1964510740}) 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.304+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.533 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788582096}) 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232411216}) 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788582096}) 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2767057217}) 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788582096}) 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=382170506}) 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788582096}) 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3640711849}) 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788582096}) 2024-06-11T19:22:09.534 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3761354899}) 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788582096}) 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2981973710}) 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788582096}) 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.305+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3682535184}) 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094746576}) 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2026894870}) 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094746576}) 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1192318851}) 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.535 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094746576}) 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=579944927}) 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094746576}) 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2571454804}) 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094746576}) 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=639467745}) 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094746576}) 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2626350138}) 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094746576}) 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2824930353}) 2024-06-11T19:22:09.536 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094746576}) 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.306+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1676945702}) 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.307+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.307+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627236576}) 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.307+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1632387326}) 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.307+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.307+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627236576}) 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.307+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3133341735}) 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.307+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.307+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627236576}) 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.307+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1049082822}) 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627236576}) 2024-06-11T19:22:09.537 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=898021921}) 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627236576}) 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2160523081}) 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627236576}) 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=27109354}) 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627236576}) 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3515540979}) 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627236576}) 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.308+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3570220651}) 2024-06-11T19:22:09.538 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753542238}) 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2362308182}) 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753542238}) 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483304428}) 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753542238}) 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1953118649}) 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753542238}) 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2943189627}) 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753542238}) 2024-06-11T19:22:09.539 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3489344883}) 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753542238}) 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3382315603}) 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753542238}) 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2575212531}) 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753542238}) 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.309+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2988340669}) 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417911761}) 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4101881953}) 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.540 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417911761}) 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2694347844}) 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417911761}) 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1403515970}) 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417911761}) 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4245831706}) 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.310+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.311+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417911761}) 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.311+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=58209271}) 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.311+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.311+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417911761}) 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.311+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=386331066}) 2024-06-11T19:22:09.541 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.311+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.311+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417911761}) 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.311+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=19684553}) 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.311+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.311+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417911761}) 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.311+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=851779121}) 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2037462465}) 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421372784}) 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2037462465}) 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3278453183}) 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.542 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2037462465}) 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2567313322}) 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2037462465}) 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1035182159}) 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2037462465}) 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3441939934}) 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2037462465}) 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1782754731}) 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2037462465}) 2024-06-11T19:22:09.543 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=259385816}) 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2037462465}) 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.312+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2993304640}) 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156069279}) 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030573597}) 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156069279}) 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=958221851}) 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156069279}) 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3510504751}) 2024-06-11T19:22:09.544 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156069279}) 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2204801521}) 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156069279}) 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.313+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2816477369}) 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.314+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.314+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156069279}) 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.314+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2038706058}) 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.314+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.314+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156069279}) 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.314+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=409030062}) 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.314+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.545 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.314+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156069279}) 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.314+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1197880048}) 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833383523}) 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3921319460}) 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833383523}) 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3526180241}) 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833383523}) 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1538032365}) 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833383523}) 2024-06-11T19:22:09.546 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=837731764}) 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833383523}) 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1787760548}) 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833383523}) 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4084615139}) 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833383523}) 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2586399167}) 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833383523}) 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.315+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1395619515}) 2024-06-11T19:22:09.547 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901534699}) 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662594415}) 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901534699}) 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3202734914}) 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901534699}) 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1911652327}) 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901534699}) 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3764318405}) 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.548 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901534699}) 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2695163705}) 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901534699}) 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1973377012}) 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.316+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901534699}) 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.317+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2824327377}) 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.317+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.317+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901534699}) 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.317+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3056281480}) 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140371088}) 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2287451739}) 2024-06-11T19:22:09.549 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.550 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140371088}) 2024-06-11T19:22:09.550 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1310164077}) 2024-06-11T19:22:09.550 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.550 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140371088}) 2024-06-11T19:22:09.550 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4001146191}) 2024-06-11T19:22:09.550 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.550 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140371088}) 2024-06-11T19:22:09.550 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4261723299}) 2024-06-11T19:22:09.550 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.550 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140371088}) 2024-06-11T19:22:09.550 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2071791465}) 2024-06-11T19:22:09.550 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.551 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140371088}) 2024-06-11T19:22:09.551 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3340855255}) 2024-06-11T19:22:09.551 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.551 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140371088}) 2024-06-11T19:22:09.551 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3570853143}) 2024-06-11T19:22:09.551 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.551 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140371088}) 2024-06-11T19:22:09.551 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.318+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=310454306}) 2024-06-11T19:22:09.551 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.551 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4027524911}) 2024-06-11T19:22:09.551 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=351272263}) 2024-06-11T19:22:09.551 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4027524911}) 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1137236148}) 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4027524911}) 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=376740884}) 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4027524911}) 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2340574921}) 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4027524911}) 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2214495196}) 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4027524911}) 2024-06-11T19:22:09.552 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=724186998}) 2024-06-11T19:22:09.553 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.553 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4027524911}) 2024-06-11T19:22:09.553 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3122477604}) 2024-06-11T19:22:09.553 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.553 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4027524911}) 2024-06-11T19:22:09.553 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.319+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4220998282}) 2024-06-11T19:22:09.553 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.320+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.553 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.320+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050987678}) 2024-06-11T19:22:09.553 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.320+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744601386}) 2024-06-11T19:22:09.553 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.320+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.553 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.320+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050987678}) 2024-06-11T19:22:09.553 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2873933212}) 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050987678}) 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=243871392}) 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050987678}) 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4166246544}) 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050987678}) 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=222424766}) 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050987678}) 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2223459319}) 2024-06-11T19:22:09.554 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050987678}) 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=481979801}) 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050987678}) 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.321+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=824649477}) 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517876003}) 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2108798367}) 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517876003}) 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2160582846}) 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.555 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517876003}) 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3099371100}) 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517876003}) 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=647933360}) 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517876003}) 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2470379333}) 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517876003}) 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1662696636}) 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517876003}) 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3940605223}) 2024-06-11T19:22:09.556 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517876003}) 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.322+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3344936239}) 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=955725822}) 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444354733}) 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=955725822}) 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2744688012}) 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=955725822}) 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1058181153}) 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=955725822}) 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1206987389}) 2024-06-11T19:22:09.557 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=955725822}) 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2970326135}) 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=955725822}) 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3772323743}) 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=955725822}) 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3420003668}) 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=955725822}) 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.323+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3734418101}) 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976053214}) 2024-06-11T19:22:09.558 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1466178739}) 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976053214}) 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281457239}) 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976053214}) 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4132755224}) 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976053214}) 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3313045075}) 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976053214}) 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3616772119}) 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976053214}) 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2304883558}) 2024-06-11T19:22:09.559 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976053214}) 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4122691997}) 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976053214}) 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.325+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2992457422}) 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=513376703}) 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334337406}) 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=513376703}) 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1359962216}) 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=513376703}) 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2640276434}) 2024-06-11T19:22:09.560 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=513376703}) 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=110450489}) 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=513376703}) 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1774810773}) 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=513376703}) 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=804508018}) 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=513376703}) 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=138116633}) 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=513376703}) 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.326+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1131490234}) 2024-06-11T19:22:09.561 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.327+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.327+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205287692}) 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.327+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2055192361}) 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.327+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.327+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205287692}) 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.327+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316610283}) 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.327+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.327+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205287692}) 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.327+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=141117305}) 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.327+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.327+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205287692}) 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2228635443}) 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205287692}) 2024-06-11T19:22:09.562 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=612137271}) 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205287692}) 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2986302236}) 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205287692}) 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2505593335}) 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205287692}) 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.328+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3580323838}) 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1031629333}) 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2763770494}) 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1031629333}) 2024-06-11T19:22:09.563 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2629733378}) 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1031629333}) 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1941997825}) 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1031629333}) 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3728403796}) 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1031629333}) 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1933951967}) 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1031629333}) 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4105366381}) 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.564 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1031629333}) 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2438452145}) 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1031629333}) 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.329+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2304236163}) 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494250143}) 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=985677015}) 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494250143}) 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317910656}) 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494250143}) 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2790411284}) 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494250143}) 2024-06-11T19:22:09.565 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3007186369}) 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494250143}) 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2016931003}) 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494250143}) 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1638890360}) 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494250143}) 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3668886341}) 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494250143}) 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.330+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3454115552}) 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.566 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=628732854}) 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1809858883}) 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=628732854}) 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3025544073}) 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=628732854}) 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4185683866}) 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=628732854}) 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=28677057}) 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=628732854}) 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3930003533}) 2024-06-11T19:22:09.567 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=628732854}) 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2584130616}) 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=628732854}) 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3718360083}) 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=628732854}) 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.332+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=634647079}) 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157180022}) 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838048187}) 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157180022}) 2024-06-11T19:22:09.568 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2276150574}) 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157180022}) 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4274699491}) 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157180022}) 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3411813508}) 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157180022}) 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3521236609}) 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157180022}) 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2900839202}) 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157180022}) 2024-06-11T19:22:09.569 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1788082161}) 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157180022}) 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.333+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1348578846}) 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345207170}) 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893213878}) 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345207170}) 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1622682196}) 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345207170}) 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4004320192}) 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.570 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345207170}) 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3151110275}) 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345207170}) 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=264076504}) 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345207170}) 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3470368233}) 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345207170}) 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2545480374}) 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345207170}) 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.335+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=576067639}) 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.571 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866603245}) 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3970527574}) 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866603245}) 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1798306912}) 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866603245}) 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3180927823}) 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866603245}) 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2378191755}) 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866603245}) 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3897368614}) 2024-06-11T19:22:09.572 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866603245}) 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3467568660}) 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866603245}) 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1888718713}) 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866603245}) 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.336+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1242871253}) 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.337+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.337+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2540929084}) 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.337+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106874537}) 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.337+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2540929084}) 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1601174714}) 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.573 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2540929084}) 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4099104104}) 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2540929084}) 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1101998996}) 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2540929084}) 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=338843926}) 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2540929084}) 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3416266656}) 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2540929084}) 2024-06-11T19:22:09.574 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4195744521}) 2024-06-11T19:22:09.575 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.575 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2540929084}) 2024-06-11T19:22:09.575 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.338+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2108968835}) 2024-06-11T19:22:09.575 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.575 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3821171306}) 2024-06-11T19:22:09.575 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165990936}) 2024-06-11T19:22:09.575 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.575 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3821171306}) 2024-06-11T19:22:09.575 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1519920822}) 2024-06-11T19:22:09.575 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.575 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3821171306}) 2024-06-11T19:22:09.575 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3231406054}) 2024-06-11T19:22:09.576 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.576 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3821171306}) 2024-06-11T19:22:09.576 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2128991544}) 2024-06-11T19:22:09.576 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.576 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3821171306}) 2024-06-11T19:22:09.576 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1167759498}) 2024-06-11T19:22:09.576 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.576 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3821171306}) 2024-06-11T19:22:09.576 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=391606780}) 2024-06-11T19:22:09.576 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.576 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3821171306}) 2024-06-11T19:22:09.576 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4015414095}) 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3821171306}) 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.339+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1929399061}) 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.340+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.340+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=156998048}) 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.340+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106191766}) 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.340+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.340+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=156998048}) 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.340+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4283409884}) 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.340+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.340+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=156998048}) 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.340+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3818147360}) 2024-06-11T19:22:09.577 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=156998048}) 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3768503031}) 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=156998048}) 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=32119663}) 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=156998048}) 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=692861304}) 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=156998048}) 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3761275812}) 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.578 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=156998048}) 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.341+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=948862361}) 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916959382}) 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=102206274}) 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916959382}) 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352163835}) 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916959382}) 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3321351836}) 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916959382}) 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2386243616}) 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.579 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916959382}) 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1145552088}) 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916959382}) 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1517934644}) 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916959382}) 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2376672059}) 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.342+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916959382}) 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.343+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1644258614}) 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284856990}) 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463143918}) 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.580 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284856990}) 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2013107364}) 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284856990}) 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4107628500}) 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284856990}) 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2008187404}) 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284856990}) 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1501775624}) 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284856990}) 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=492687519}) 2024-06-11T19:22:09.581 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284856990}) 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=756146575}) 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284856990}) 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.344+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3144734026}) 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228989033}) 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=504510201}) 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228989033}) 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2175593276}) 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.582 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228989033}) 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=680780520}) 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228989033}) 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3883346127}) 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228989033}) 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=683085763}) 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228989033}) 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=595811810}) 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228989033}) 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2162150813}) 2024-06-11T19:22:09.583 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228989033}) 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.345+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=366663343}) 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.346+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.346+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151543597}) 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.346+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047940748}) 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.346+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.346+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151543597}) 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.346+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=689227243}) 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.346+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.346+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151543597}) 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3775188595}) 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.584 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151543597}) 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4095494769}) 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151543597}) 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=939870746}) 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151543597}) 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1833227187}) 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151543597}) 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=584407434}) 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151543597}) 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.347+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4164507795}) 2024-06-11T19:22:09.585 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.586 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3293809225}) 2024-06-11T19:22:09.586 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1444959532}) 2024-06-11T19:22:09.586 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.586 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3293809225}) 2024-06-11T19:22:09.586 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3419749898}) 2024-06-11T19:22:09.586 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.586 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3293809225}) 2024-06-11T19:22:09.586 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2329728747}) 2024-06-11T19:22:09.586 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.586 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3293809225}) 2024-06-11T19:22:09.586 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3707291001}) 2024-06-11T19:22:09.586 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3293809225}) 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2333099956}) 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3293809225}) 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=426902837}) 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3293809225}) 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1319651918}) 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3293809225}) 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.348+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2666382982}) 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.349+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.349+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961502051}) 2024-06-11T19:22:09.587 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.349+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333105058}) 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.349+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.349+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961502051}) 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.349+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3509964307}) 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961502051}) 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=953531106}) 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961502051}) 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2015131614}) 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961502051}) 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2534290425}) 2024-06-11T19:22:09.588 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.589 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961502051}) 2024-06-11T19:22:09.589 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=117875945}) 2024-06-11T19:22:09.589 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.589 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961502051}) 2024-06-11T19:22:09.589 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3579495390}) 2024-06-11T19:22:09.589 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.589 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961502051}) 2024-06-11T19:22:09.589 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.350+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1074098739}) 2024-06-11T19:22:09.589 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.589 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565115989}) 2024-06-11T19:22:09.589 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306384698}) 2024-06-11T19:22:09.589 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.590 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565115989}) 2024-06-11T19:22:09.590 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256407134}) 2024-06-11T19:22:09.590 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.590 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565115989}) 2024-06-11T19:22:09.590 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3267966026}) 2024-06-11T19:22:09.590 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.590 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565115989}) 2024-06-11T19:22:09.590 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2466904859}) 2024-06-11T19:22:09.590 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.590 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565115989}) 2024-06-11T19:22:09.590 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=205046581}) 2024-06-11T19:22:09.590 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565115989}) 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2822525102}) 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565115989}) 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4058214559}) 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565115989}) 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.351+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1621296492}) 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533318149}) 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136116303}) 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.591 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533318149}) 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3176601954}) 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533318149}) 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1142573363}) 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533318149}) 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3026827049}) 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533318149}) 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2650768121}) 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.352+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.353+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533318149}) 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.353+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2985668565}) 2024-06-11T19:22:09.592 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.353+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.353+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533318149}) 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.353+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4164099018}) 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.353+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.353+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533318149}) 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.353+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2896288498}) 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178989791}) 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3375647873}) 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178989791}) 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1803801700}) 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178989791}) 2024-06-11T19:22:09.593 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2879342724}) 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178989791}) 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3869772891}) 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178989791}) 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1703958686}) 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178989791}) 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4230016912}) 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178989791}) 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=815270520}) 2024-06-11T19:22:09.594 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178989791}) 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.354+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=903525005}) 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2273649187}) 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394916997}) 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2273649187}) 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3940835020}) 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2273649187}) 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=860224195}) 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.595 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2273649187}) 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=420036878}) 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2273649187}) 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1246674572}) 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2273649187}) 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=176875185}) 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2273649187}) 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4263612218}) 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2273649187}) 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.355+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1434725435}) 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.356+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.596 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.356+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047782921}) 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.356+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3281305807}) 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.356+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.356+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047782921}) 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2618925863}) 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047782921}) 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2705904590}) 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047782921}) 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2668520227}) 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047782921}) 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=679199045}) 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.597 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047782921}) 2024-06-11T19:22:09.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3653937838}) 2024-06-11T19:22:09.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047782921}) 2024-06-11T19:22:09.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1801516509}) 2024-06-11T19:22:09.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047782921}) 2024-06-11T19:22:09.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.357+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1017868932}) 2024-06-11T19:22:09.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105036776}) 2024-06-11T19:22:09.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3698591826}) 2024-06-11T19:22:09.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.598 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105036776}) 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2320447221}) 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105036776}) 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1819013334}) 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105036776}) 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3468284502}) 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105036776}) 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2281331294}) 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105036776}) 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=775912165}) 2024-06-11T19:22:09.599 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105036776}) 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2619488952}) 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105036776}) 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.358+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4291780192}) 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3774056462}) 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597905592}) 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3774056462}) 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332463705}) 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3774056462}) 2024-06-11T19:22:09.600 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1991530042}) 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3774056462}) 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2757611965}) 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.359+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3774056462}) 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.360+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1668038689}) 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.360+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.360+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3774056462}) 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.360+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=896039568}) 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.360+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.360+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3774056462}) 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.360+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1814472860}) 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.360+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.360+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3774056462}) 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.360+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3528359214}) 2024-06-11T19:22:09.601 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872097241}) 2024-06-11T19:22:09.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158583453}) 2024-06-11T19:22:09.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872097241}) 2024-06-11T19:22:09.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1421104365}) 2024-06-11T19:22:09.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872097241}) 2024-06-11T19:22:09.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=531248167}) 2024-06-11T19:22:09.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872097241}) 2024-06-11T19:22:09.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2993247300}) 2024-06-11T19:22:09.602 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872097241}) 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3736289998}) 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872097241}) 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=707836948}) 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872097241}) 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4190024662}) 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872097241}) 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.361+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3766302539}) 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987856299}) 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4185223271}) 2024-06-11T19:22:09.603 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987856299}) 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=533547261}) 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987856299}) 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3406844119}) 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987856299}) 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2452274023}) 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987856299}) 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2632203686}) 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.604 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987856299}) 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2340924664}) 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987856299}) 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2236040938}) 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.362+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987856299}) 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.363+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2952193383}) 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607724428}) 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=586904364}) 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607724428}) 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3562114179}) 2024-06-11T19:22:09.605 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607724428}) 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=497037071}) 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607724428}) 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3579786596}) 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607724428}) 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2831807721}) 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607724428}) 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1506250652}) 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607724428}) 2024-06-11T19:22:09.606 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1345161451}) 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607724428}) 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.364+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=313640074}) 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=471902971}) 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2752881688}) 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=471902971}) 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1647201860}) 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=471902971}) 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1185116143}) 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=471902971}) 2024-06-11T19:22:09.607 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=956077284}) 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=471902971}) 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2496147081}) 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=471902971}) 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=605005048}) 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=471902971}) 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2592120308}) 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=471902971}) 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.366+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1402140896}) 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.367+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.367+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071243121}) 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1682151677}) 2024-06-11T19:22:09.608 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071243121}) 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2097433616}) 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071243121}) 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=856188190}) 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071243121}) 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=229716452}) 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071243121}) 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3710119504}) 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071243121}) 2024-06-11T19:22:09.609 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3924361390}) 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071243121}) 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3587087844}) 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071243121}) 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.368+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=101157104}) 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079217946}) 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=53158609}) 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079217946}) 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3730104113}) 2024-06-11T19:22:09.610 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079217946}) 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2787491219}) 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079217946}) 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=844941311}) 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079217946}) 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3966390991}) 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079217946}) 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3493547072}) 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079217946}) 2024-06-11T19:22:09.611 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3558612105}) 2024-06-11T19:22:09.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079217946}) 2024-06-11T19:22:09.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.369+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=47217675}) 2024-06-11T19:22:09.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3138122133}) 2024-06-11T19:22:09.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1058704198}) 2024-06-11T19:22:09.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3138122133}) 2024-06-11T19:22:09.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=427354535}) 2024-06-11T19:22:09.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3138122133}) 2024-06-11T19:22:09.612 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3473537157}) 2024-06-11T19:22:09.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3138122133}) 2024-06-11T19:22:09.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2416871592}) 2024-06-11T19:22:09.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3138122133}) 2024-06-11T19:22:09.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3259140344}) 2024-06-11T19:22:09.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3138122133}) 2024-06-11T19:22:09.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1155500261}) 2024-06-11T19:22:09.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3138122133}) 2024-06-11T19:22:09.613 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2823559052}) 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3138122133}) 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.371+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2002684517}) 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343608054}) 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374231400}) 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343608054}) 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=490853919}) 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343608054}) 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2428899643}) 2024-06-11T19:22:09.614 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343608054}) 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937586124}) 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343608054}) 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.372+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1069525547}) 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.373+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.373+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343608054}) 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.373+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=214072882}) 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.373+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.373+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343608054}) 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.373+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1124486647}) 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.373+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.373+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343608054}) 2024-06-11T19:22:09.615 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.373+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1202823956}) 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532120134}) 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1186133993}) 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532120134}) 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2526467738}) 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532120134}) 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2810909118}) 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532120134}) 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4072061421}) 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532120134}) 2024-06-11T19:22:09.616 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3678339288}) 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532120134}) 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2337807}) 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532120134}) 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=37784320}) 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532120134}) 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.374+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3299584875}) 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984674257}) 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=995744920}) 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984674257}) 2024-06-11T19:22:09.617 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3121053246}) 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984674257}) 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=234521968}) 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984674257}) 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2197388596}) 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984674257}) 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.375+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2273966014}) 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.376+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.376+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984674257}) 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.376+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2782757078}) 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.376+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.618 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.376+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984674257}) 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.376+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2395487294}) 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.376+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.376+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984674257}) 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.376+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2182235219}) 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713912394}) 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3757619816}) 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713912394}) 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=189032104}) 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713912394}) 2024-06-11T19:22:09.619 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4027593143}) 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713912394}) 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=447239891}) 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713912394}) 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3968480400}) 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713912394}) 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=231983379}) 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713912394}) 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2603349174}) 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713912394}) 2024-06-11T19:22:09.620 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.377+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=409268950}) 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4050815678}) 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2451779256}) 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4050815678}) 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2197472217}) 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4050815678}) 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1525246461}) 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4050815678}) 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2251479095}) 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.621 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.378+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4050815678}) 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.379+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2940447491}) 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.379+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.379+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4050815678}) 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.379+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2029272353}) 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.379+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.379+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4050815678}) 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.379+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1259938227}) 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.379+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.379+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4050815678}) 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.379+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2850176711}) 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=164005008}) 2024-06-11T19:22:09.622 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170255326}) 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=164005008}) 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3658415654}) 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=164005008}) 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4226919193}) 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=164005008}) 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3176118239}) 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=164005008}) 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3291770886}) 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.623 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=164005008}) 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2123004171}) 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=164005008}) 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1620587379}) 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=164005008}) 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.380+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=131066082}) 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.381+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.381+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4107530504}) 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.381+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175843594}) 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.381+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.381+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4107530504}) 2024-06-11T19:22:09.624 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.381+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=558864510}) 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.381+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.381+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4107530504}) 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.381+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2061030625}) 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.381+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.381+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4107530504}) 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1439125608}) 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4107530504}) 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3251839926}) 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4107530504}) 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=123621928}) 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.625 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4107530504}) 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4092833940}) 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4107530504}) 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.382+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3063135730}) 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187143003}) 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4081654419}) 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187143003}) 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2365331839}) 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187143003}) 2024-06-11T19:22:09.626 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4131884030}) 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187143003}) 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3263365490}) 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187143003}) 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=147562372}) 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187143003}) 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2512187820}) 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187143003}) 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3530936708}) 2024-06-11T19:22:09.627 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187143003}) 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.383+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3491033987}) 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.384+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.384+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926771926}) 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.384+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3579696951}) 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.384+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.384+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926771926}) 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.384+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1935524158}) 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.384+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.384+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926771926}) 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.384+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1339708045}) 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.384+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.628 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.384+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926771926}) 2024-06-11T19:22:09.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.384+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4059397816}) 2024-06-11T19:22:09.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.385+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.385+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926771926}) 2024-06-11T19:22:09.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.385+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2494279017}) 2024-06-11T19:22:09.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.385+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.385+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926771926}) 2024-06-11T19:22:09.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.385+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1550898102}) 2024-06-11T19:22:09.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.385+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.385+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926771926}) 2024-06-11T19:22:09.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.385+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3484751984}) 2024-06-11T19:22:09.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.385+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.629 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.385+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926771926}) 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.385+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2349710614}) 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445021375}) 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2723209485}) 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445021375}) 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=22641164}) 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445021375}) 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2078186981}) 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445021375}) 2024-06-11T19:22:09.630 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3113887994}) 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445021375}) 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3916539124}) 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445021375}) 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=379658075}) 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445021375}) 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4215019780}) 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445021375}) 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.386+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=838145940}) 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.631 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619032264}) 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562153770}) 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619032264}) 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3925005445}) 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619032264}) 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2567486125}) 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619032264}) 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1877876269}) 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619032264}) 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=711078479}) 2024-06-11T19:22:09.632 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619032264}) 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2705344425}) 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619032264}) 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=64618614}) 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619032264}) 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.387+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2252987124}) 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.388+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759498732}) 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1855145292}) 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759498732}) 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1043899070}) 2024-06-11T19:22:09.633 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759498732}) 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1301548454}) 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759498732}) 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2412629732}) 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759498732}) 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3981238249}) 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759498732}) 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2241436734}) 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.634 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759498732}) 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3407718629}) 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759498732}) 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.389+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=433406380}) 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832747042}) 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2109254642}) 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832747042}) 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2609960311}) 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832747042}) 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3414409580}) 2024-06-11T19:22:09.635 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832747042}) 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1692458441}) 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832747042}) 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2781013306}) 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832747042}) 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1544916975}) 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832747042}) 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=302377403}) 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832747042}) 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.390+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2124774570}) 2024-06-11T19:22:09.636 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416539729}) 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077963339}) 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416539729}) 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079173441}) 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416539729}) 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1634940}) 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416539729}) 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1599452084}) 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416539729}) 2024-06-11T19:22:09.637 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=575732735}) 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416539729}) 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1652226626}) 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416539729}) 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=736397917}) 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416539729}) 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.392+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=167914845}) 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675644994}) 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063745724}) 2024-06-11T19:22:09.638 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675644994}) 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=920510049}) 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675644994}) 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3202081268}) 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675644994}) 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=176490824}) 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675644994}) 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1971835216}) 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675644994}) 2024-06-11T19:22:09.639 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=393681051}) 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675644994}) 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1969648768}) 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675644994}) 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.393+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1476356652}) 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.394+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.394+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612011358}) 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.394+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1737442256}) 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.394+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.394+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612011358}) 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.394+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323637310}) 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.394+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.640 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.394+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612011358}) 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.394+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2881190113}) 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.394+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612011358}) 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2274247440}) 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612011358}) 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1264182313}) 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612011358}) 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3335458074}) 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612011358}) 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2619215480}) 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.641 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612011358}) 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.395+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3920295069}) 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390038725}) 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263022508}) 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390038725}) 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4136575235}) 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390038725}) 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2872088679}) 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390038725}) 2024-06-11T19:22:09.642 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3029253528}) 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390038725}) 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=489134994}) 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390038725}) 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1895246822}) 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390038725}) 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3530525603}) 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390038725}) 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.396+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2768718261}) 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.643 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461747413}) 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=697829851}) 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461747413}) 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715087685}) 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461747413}) 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1918724335}) 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461747413}) 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1641605739}) 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.397+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461747413}) 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.398+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3922129843}) 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.398+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.644 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.398+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461747413}) 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.398+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4105794085}) 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.398+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.398+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461747413}) 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.398+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3865966508}) 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.398+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.398+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461747413}) 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.398+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1331265855}) 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24933266}) 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390937488}) 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24933266}) 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2357986991}) 2024-06-11T19:22:09.645 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24933266}) 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2809097955}) 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24933266}) 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4013521731}) 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24933266}) 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=691857568}) 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24933266}) 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3180922210}) 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.646 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24933266}) 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3161451226}) 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24933266}) 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.399+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2998881739}) 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.400+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.400+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531207976}) 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.400+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308844397}) 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.400+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.400+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531207976}) 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.400+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1672416357}) 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.400+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.400+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531207976}) 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.400+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2815905478}) 2024-06-11T19:22:09.647 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.400+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.400+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531207976}) 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.400+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3230645035}) 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.401+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.401+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531207976}) 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.401+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1489368732}) 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.401+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.401+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531207976}) 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.401+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2220181824}) 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.401+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.401+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531207976}) 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.401+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2155844818}) 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.401+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.648 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.401+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531207976}) 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.401+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=45385303}) 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443766606}) 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=35840626}) 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443766606}) 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1656213113}) 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443766606}) 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4214229017}) 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443766606}) 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2743874250}) 2024-06-11T19:22:09.649 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443766606}) 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1218849274}) 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443766606}) 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=890843402}) 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443766606}) 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2912084055}) 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443766606}) 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.402+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2089611828}) 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.403+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.650 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.403+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1337382057}) 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.403+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2066464860}) 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.403+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.403+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1337382057}) 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.403+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950538468}) 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.403+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.403+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1337382057}) 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.403+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2860428645}) 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1337382057}) 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1129254049}) 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1337382057}) 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3028495727}) 2024-06-11T19:22:09.651 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1337382057}) 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4187568147}) 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1337382057}) 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3019935403}) 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1337382057}) 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.404+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2604675096}) 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3927158130}) 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013642227}) 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.652 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3927158130}) 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=960427045}) 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3927158130}) 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1239367849}) 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3927158130}) 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2650716550}) 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3927158130}) 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2790181245}) 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3927158130}) 2024-06-11T19:22:09.653 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2666815662}) 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3927158130}) 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=532763251}) 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3927158130}) 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.405+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2527365547}) 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.406+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579529996}) 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3825016511}) 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579529996}) 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=497569892}) 2024-06-11T19:22:09.654 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579529996}) 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2379798779}) 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579529996}) 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1801625395}) 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579529996}) 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2447337482}) 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579529996}) 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=926756396}) 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579529996}) 2024-06-11T19:22:09.655 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3259030315}) 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579529996}) 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.407+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3446592261}) 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272672685}) 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548472645}) 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272672685}) 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3633065265}) 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272672685}) 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=715731369}) 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272672685}) 2024-06-11T19:22:09.656 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3443821274}) 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272672685}) 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3193315555}) 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272672685}) 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1250107573}) 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272672685}) 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2796784403}) 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272672685}) 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.408+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3405514481}) 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307007363}) 2024-06-11T19:22:09.657 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=234070733}) 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307007363}) 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1477938281}) 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307007363}) 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1390796582}) 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307007363}) 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1406660845}) 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307007363}) 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.409+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2302896789}) 2024-06-11T19:22:09.658 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.410+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.410+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307007363}) 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.410+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2775465014}) 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.410+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.410+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307007363}) 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.410+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=311964388}) 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.410+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.410+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307007363}) 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.410+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=772664990}) 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372318488}) 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293990669}) 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372318488}) 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3653288627}) 2024-06-11T19:22:09.659 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372318488}) 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1014935939}) 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372318488}) 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2651776703}) 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372318488}) 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3540261630}) 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372318488}) 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1261080300}) 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372318488}) 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1217646019}) 2024-06-11T19:22:09.660 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372318488}) 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.411+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=613142848}) 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234124}) 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1802871909}) 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234124}) 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2278859785}) 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234124}) 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=923903610}) 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234124}) 2024-06-11T19:22:09.661 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4213433753}) 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234124}) 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2504477829}) 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234124}) 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1916406530}) 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234124}) 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=862032892}) 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234124}) 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.412+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2053662322}) 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.413+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.413+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116908234}) 2024-06-11T19:22:09.662 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.413+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072673811}) 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.413+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.413+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116908234}) 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.413+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2212842948}) 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.413+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.413+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116908234}) 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.413+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2714954268}) 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.413+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116908234}) 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1219150974}) 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116908234}) 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=593040875}) 2024-06-11T19:22:09.663 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116908234}) 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2926464002}) 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116908234}) 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2924368332}) 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116908234}) 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.414+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3488059857}) 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1142977654}) 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921713010}) 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1142977654}) 2024-06-11T19:22:09.664 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1841659441}) 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1142977654}) 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1929444979}) 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1142977654}) 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1172971155}) 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1142977654}) 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3674369957}) 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1142977654}) 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3770927804}) 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.665 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1142977654}) 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2743129646}) 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1142977654}) 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.415+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2489911596}) 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316658941}) 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2358217509}) 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316658941}) 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3678852735}) 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316658941}) 2024-06-11T19:22:09.666 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1861522373}) 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316658941}) 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1802703616}) 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.416+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316658941}) 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.417+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1757001876}) 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.417+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.417+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316658941}) 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.417+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3432094805}) 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.417+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.417+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316658941}) 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.417+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1930709841}) 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.417+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.667 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.417+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316658941}) 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.417+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2807786571}) 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1182966428}) 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701931960}) 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1182966428}) 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2164675240}) 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1182966428}) 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3733965470}) 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1182966428}) 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1034277927}) 2024-06-11T19:22:09.668 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1182966428}) 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3335343700}) 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1182966428}) 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2880430016}) 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1182966428}) 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1423429653}) 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1182966428}) 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.418+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1687666010}) 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.419+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.669 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.419+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1183819731}) 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.419+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3946526365}) 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.419+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.419+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1183819731}) 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.419+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1463394232}) 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.419+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.419+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1183819731}) 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.419+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2917122485}) 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.419+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1183819731}) 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1255276438}) 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1183819731}) 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4236947651}) 2024-06-11T19:22:09.670 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1183819731}) 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3865622182}) 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1183819731}) 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2262139967}) 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1183819731}) 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.420+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=920461099}) 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2921864849}) 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=899772201}) 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2921864849}) 2024-06-11T19:22:09.671 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715123845}) 2024-06-11T19:22:09.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2921864849}) 2024-06-11T19:22:09.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1502377358}) 2024-06-11T19:22:09.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2921864849}) 2024-06-11T19:22:09.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2731322385}) 2024-06-11T19:22:09.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2921864849}) 2024-06-11T19:22:09.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1123854882}) 2024-06-11T19:22:09.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2921864849}) 2024-06-11T19:22:09.672 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=181728410}) 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2921864849}) 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1783830076}) 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2921864849}) 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.421+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1440163269}) 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769288629}) 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917620430}) 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769288629}) 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=626661881}) 2024-06-11T19:22:09.673 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769288629}) 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=872486788}) 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769288629}) 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=84286234}) 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769288629}) 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2993660132}) 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.422+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769288629}) 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.423+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3381102005}) 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.423+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.674 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.423+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769288629}) 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.423+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1500325441}) 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.423+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.423+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769288629}) 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.423+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2028342717}) 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.423+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.423+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335432272}) 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=989661777}) 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335432272}) 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1787383094}) 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335432272}) 2024-06-11T19:22:09.675 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=386523374}) 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335432272}) 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1552531242}) 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335432272}) 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=563707799}) 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335432272}) 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1665757617}) 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335432272}) 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=709322994}) 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.676 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335432272}) 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.424+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3011914381}) 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=15} crc {logm=2240611856,mdsmap=2840952586,mgr=3166249245,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2178057952,mon_config_key=124724172,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=2910676205,osdmap=1324979343}) 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=9,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=32,osdmap=15} crc {logm=2240611856,mdsmap=2840952586,mgr=2241844087,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=1917550078,mon_config_key=1597087497,monmap=506305176,osd_metadata=728168835,osd_pg_creating=676785445,osd_snap=3137034786,osdmap=3894734674}) 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=15} crc {logm=2240611856,mdsmap=2840952586,mgr=3166249245,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2178057952,mon_config_key=124724172,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=2910676205,osdmap=1324979343}) 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=30,osdmap=16} crc {logm=2240611856,mdsmap=2840952586,mgr=2803191070,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2298873653,mon_config_key=2314039359,monmap=506305176,osd_metadata=1052760870,osd_pg_creating=676785445,osd_snap=2042510534,osdmap=3840491650}) 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=15} crc {logm=2240611856,mdsmap=2840952586,mgr=3166249245,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2178057952,mon_config_key=124724172,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=2910676205,osdmap=1324979343}) 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=15} crc {logm=2240611856,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3068685930,mon_config_key=124724172,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=1630370760,osdmap=1243547184}) 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=15} crc {logm=2240611856,mdsmap=2840952586,mgr=3166249245,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2178057952,mon_config_key=124724172,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=2910676205,osdmap=1324979343}) 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=9,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=18} crc {logm=2240611857,mdsmap=1401145748,mgr=235511605,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=545170953,mon_config_key=1624270173,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=2909453534,osdmap=2720102721}) 2024-06-11T19:22:09.677 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=15} crc {logm=2240611856,mdsmap=2840952586,mgr=3166249245,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2178057952,mon_config_key=124724172,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=2910676205,osdmap=1324979343}) 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=16} crc {logm=2240611856,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3526329308,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=1591253667,osdmap=1955397382}) 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=15} crc {logm=2240611856,mdsmap=2840952586,mgr=3166249245,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2178057952,mon_config_key=124724172,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=2910676205,osdmap=1324979343}) 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=32,osdmap=13} crc {logm=2240611856,mdsmap=2840952586,mgr=235511605,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2178057952,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=3137034786,osdmap=364541961}) 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=15} crc {logm=2240611856,mdsmap=2840952586,mgr=3166249245,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2178057952,mon_config_key=124724172,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=2910676205,osdmap=1324979343}) 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=14} crc {logm=2240611856,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3669378479,mon_config_key=2314039359,monmap=1460791232,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=3320741213,osdmap=2548294943}) 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=15} crc {logm=2240611856,mdsmap=2840952586,mgr=3166249245,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2178057952,mon_config_key=124724172,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=676785445,osd_snap=2910676205,osdmap=1324979343}) 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.426+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=9,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=8,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=29,osdmap=22} crc {logm=2240611856,mdsmap=2840952586,mgr=3452237538,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3669378479,mon_config_key=1784220926,monmap=506305176,osd_metadata=3351267617,osd_pg_creating=676785445,osd_snap=3622903105,osdmap=798692337}) 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.678 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1834274053}) 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4227089798}) 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1834274053}) 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1541368587}) 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1834274053}) 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=877740646}) 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1834274053}) 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=883600113}) 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1834274053}) 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1187713638}) 2024-06-11T19:22:09.679 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1834274053}) 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1525795654}) 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1834274053}) 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=690508420}) 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1834274053}) 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.428+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1953877025}) 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.429+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.429+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3807934040}) 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.429+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1462711706}) 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.429+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.429+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3807934040}) 2024-06-11T19:22:09.680 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.429+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3146615602}) 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.429+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.429+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3807934040}) 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.429+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=4080457604}) 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3807934040}) 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=485361398}) 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3807934040}) 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1297704751}) 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3807934040}) 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2347890159}) 2024-06-11T19:22:09.681 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3807934040}) 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=180086746}) 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3807934040}) 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.430+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1967528241}) 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3248371746}) 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=874916408}) 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3248371746}) 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3348688026}) 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3248371746}) 2024-06-11T19:22:09.682 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2573090144}) 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3248371746}) 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3587115272}) 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3248371746}) 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4083762567}) 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3248371746}) 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=4247236698}) 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3248371746}) 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2508993206}) 2024-06-11T19:22:09.683 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.684 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3248371746}) 2024-06-11T19:22:09.684 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.431+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1131952335}) 2024-06-11T19:22:09.684 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.432+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.684 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.432+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=9343695}) 2024-06-11T19:22:09.684 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.432+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2898073076}) 2024-06-11T19:22:09.684 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.684 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=9343695}) 2024-06-11T19:22:09.684 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3871826350}) 2024-06-11T19:22:09.684 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.684 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=9343695}) 2024-06-11T19:22:09.684 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=982148648}) 2024-06-11T19:22:09.684 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=9343695}) 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=145202539}) 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=9343695}) 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3380238425}) 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=9343695}) 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=603098794}) 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=9343695}) 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2147887450}) 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=9343695}) 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.433+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3734982395}) 2024-06-11T19:22:09.685 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3140679051}) 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2412113864}) 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3140679051}) 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2788907341}) 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3140679051}) 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1544418150}) 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3140679051}) 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3063390892}) 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3140679051}) 2024-06-11T19:22:09.686 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3839961679}) 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3140679051}) 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=4127944536}) 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3140679051}) 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2625068385}) 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3140679051}) 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.435+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2854415846}) 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4157176086}) 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2555312808}) 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4157176086}) 2024-06-11T19:22:09.687 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1446387378}) 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4157176086}) 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=501095687}) 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4157176086}) 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2684650658}) 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4157176086}) 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.437+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1967570923}) 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.438+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.438+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4157176086}) 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.438+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1888820074}) 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.438+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.438+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4157176086}) 2024-06-11T19:22:09.688 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.438+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1241202805}) 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.438+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.438+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4157176086}) 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.438+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3773175682}) 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2364659584}) 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2790633618}) 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2364659584}) 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2077210166}) 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2364659584}) 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=182283891}) 2024-06-11T19:22:09.689 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2364659584}) 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2203332983}) 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2364659584}) 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=275976962}) 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2364659584}) 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=4250614721}) 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2364659584}) 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3096509989}) 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2364659584}) 2024-06-11T19:22:09.690 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.440+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3123712177}) 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=3675963498}) 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=38} crc {osdmap=363330200}) 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=3675963498}) 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=39} crc {osdmap=2982918279}) 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=3675963498}) 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=38} crc {osdmap=1315342652}) 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=3675963498}) 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=39} crc {osdmap=198296744}) 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=3675963498}) 2024-06-11T19:22:09.691 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=36} crc {osdmap=1132985034}) 2024-06-11T19:22:09.692 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.692 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=3675963498}) 2024-06-11T19:22:09.692 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=39} crc {osdmap=1216493772}) 2024-06-11T19:22:09.692 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.692 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=3675963498}) 2024-06-11T19:22:09.692 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=39} crc {osdmap=3141341754}) 2024-06-11T19:22:09.692 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:22:09.692 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=3675963498}) 2024-06-11T19:22:09.692 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:22:09.448+0000 7fab17ca1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=34} crc {osdmap=594840077}) 2024-06-11T19:22:10.290 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:22:10.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:22:10.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.389+0000 7f86e9635640 10 monclient: get_monmap_and_config 2024-06-11T19:22:10.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.389+0000 7f86e9635640 10 monclient: build_initial_monmap 2024-06-11T19:22:10.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.389+0000 7f86e9635640 10 monclient: monmap: 2024-06-11T19:22:10.398 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:10.390479+0000 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:10.390479+0000 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 1 Processor -- start 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 1 -- start start 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 10 monclient: init 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 10 monclient: _add_conns ranks=[6,2,5,3,4,0,9,10,7,8,1] 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f86e40a4920 0x7f86e40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 10 monclient(hunting): picked mon.noname-d con 0x7f86e40a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a5240 0x7f86e40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 10 monclient(hunting): picked mon.noname-b con 0x7f86e40a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f86e40a2530 0x7f86e40a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 10 monclient(hunting): picked mon.noname-h con 0x7f86e40a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f86e40a7140 con 0x7f86e40a5240 2024-06-11T19:22:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f86e40a2b50 con 0x7f86e40a2530 2024-06-11T19:22:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f86e40a2e30 con 0x7f86e40a4920 2024-06-11T19:22:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e9635640 10 monclient(hunting): authenticate will time out at 1540.197589s 2024-06-11T19:22:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a5240 0x7f86e40a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e2ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f86e40a4920 0x7f86e40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a5240 0x7f86e40a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57200/0 (socket says 172.21.15.98:57200) 2024-06-11T19:22:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 1 -- 172.21.15.98:0/3577708128 learned_addr learned my addr 172.21.15.98:0/3577708128 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 10 monclient(hunting): get_auth_request con 0x7f86e40a5240 auth_method 0 2024-06-11T19:22:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e2ffd640 10 monclient(hunting): get_auth_request con 0x7f86e40a4920 auth_method 0 2024-06-11T19:22:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e37fe640 1 --2- 172.21.15.98:0/3577708128 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f86e40a2530 0x7f86e40a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e37fe640 10 monclient(hunting): get_auth_request con 0x7f86e40a2530 auth_method 0 2024-06-11T19:22:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 10 monclient(hunting): handle_auth_done global_id 84962 payload 274 2024-06-11T19:22:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 1 -- 172.21.15.98:0/3577708128 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f86e40a4920 msgr2=0x7f86e40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 1 --2- 172.21.15.98:0/3577708128 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f86e40a4920 0x7f86e40a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 1 -- 172.21.15.98:0/3577708128 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f86e40a2530 msgr2=0x7f86e40a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.390+0000 7f86e27fc640 1 --2- 172.21.15.98:0/3577708128 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f86e40a2530 0x7f86e40a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e27fc640 10 monclient: _finish_hunting 0 2024-06-11T19:22:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e27fc640 1 monclient: found mon.noname-b 2024-06-11T19:22:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:22:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e27fc640 1 -- 172.21.15.98:0/3577708128 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e40a3110 con 0x7f86e40a5240 2024-06-11T19:22:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e37fe640 1 --2- 172.21.15.98:0/3577708128 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f86e40a2530 0x7f86e40a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:22:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e27fc640 1 --2- 172.21.15.98:0/3577708128 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a5240 0x7f86e40a59a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0098a0 tx=0x7f86cc02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c0ea5b03b06abf5 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 1 -- 172.21.15.98:0/3577708128 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86cc038070 con 0x7f86e40a5240 2024-06-11T19:22:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:22:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 10 monclient: dump: 2024-06-11T19:22:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 10 monclient: _finish_auth 0 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e9635640 5 monclient: authenticate success, global_id 84962 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e9635640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 1 -- 172.21.15.98:0/3577708128 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86cc031970 con 0x7f86e40a5240 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e9635640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e9635640 4 set_mon_vals no callback set 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 1 -- 172.21.15.98:0/3577708128 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86cc031ba0 con 0x7f86e40a5240 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.391+0000 7f86e1ffb640 10 monclient: dump: 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.392+0000 7f86e9635640 10 monclient: shutdown 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.392+0000 7f86e9635640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.392+0000 7f86e9635640 1 -- 172.21.15.98:0/3577708128 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a5240 msgr2=0x7f86e40a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.392+0000 7f86e9635640 1 --2- 172.21.15.98:0/3577708128 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a5240 0x7f86e40a59a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0098a0 tx=0x7f86cc02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.392+0000 7f86e9635640 1 -- 172.21.15.98:0/3577708128 shutdown_connections 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.392+0000 7f86e9635640 1 --2- 172.21.15.98:0/3577708128 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f86e40a2530 0x7f86e40a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.392+0000 7f86e9635640 1 --2- 172.21.15.98:0/3577708128 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a5240 0x7f86e40a59a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.392+0000 7f86e9635640 1 --2- 172.21.15.98:0/3577708128 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f86e40a4920 0x7f86e40a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.392+0000 7f86e9635640 1 -- 172.21.15.98:0/3577708128 >> 172.21.15.98:0/3577708128 conn(0x7f86e40ab850 msgr2=0x7f86e40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.392+0000 7f86e9635640 1 -- 172.21.15.98:0/3577708128 shutdown_connections 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.392+0000 7f86e9635640 1 -- 172.21.15.98:0/3577708128 wait complete. 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.392+0000 7f86e9635640 10 monclient: build_initial_monmap 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 10 monclient: monmap: 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 1 librados: starting msgr at 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 1 librados: starting objecter 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 1 Processor -- start 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 1 -- start start 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 1 librados: setting wanted keys 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 1 librados: calling monclient init 2024-06-11T19:22:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 10 monclient: init 2024-06-11T19:22:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 10 monclient: _add_conns ranks=[3,2,4,0,7,1,8,6,5] 2024-06-11T19:22:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e40a2530 0x7f86e41cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 10 monclient(hunting): picked mon.noname-g con 0x7f86e40a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a4920 0x7f86e41ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 10 monclient(hunting): picked mon.noname-b con 0x7f86e40a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f86e40a5240 0x7f86e4121f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 10 monclient(hunting): picked mon.noname-c con 0x7f86e40a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f86e40a6da0 con 0x7f86e40a4920 2024-06-11T19:22:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f86e4138e50 con 0x7f86e40a2530 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f86e40a6420 con 0x7f86e40a5240 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.393+0000 7f86e9635640 10 monclient(hunting): authenticate will time out at 1540.200990s 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a4920 0x7f86e41ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a4920 0x7f86e41ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57202/0 (socket says 172.21.15.98:57202) 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 1 -- 172.21.15.98:0/3099139626 learned_addr learned my addr 172.21.15.98:0/3099139626 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e37fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f86e40a5240 0x7f86e4121f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 10 monclient(hunting): get_auth_request con 0x7f86e40a4920 auth_method 0 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e2ffd640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e40a2530 0x7f86e41cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e37fe640 10 monclient(hunting): get_auth_request con 0x7f86e40a5240 auth_method 0 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e2ffd640 10 monclient(hunting): get_auth_request con 0x7f86e40a2530 auth_method 0 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 10 monclient(hunting): handle_auth_done global_id 84971 payload 824 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 1 -- 172.21.15.98:0/3099139626 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f86e40a5240 msgr2=0x7f86e4121f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f86e40a5240 0x7f86e4121f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 1 -- 172.21.15.98:0/3099139626 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e40a2530 msgr2=0x7f86e41cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e40a2530 0x7f86e41cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 10 monclient: _finish_hunting 0 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 1 monclient: found mon.noname-b 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 1 -- 172.21.15.98:0/3099139626 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e4125c30 con 0x7f86e40a4920 2024-06-11T19:22:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e27fc640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a4920 0x7f86e41ce010 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0097c0 tx=0x7f86cc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 10 client.?.objecter ms_handle_connect 0x7f86e40a4920 2024-06-11T19:22:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 1 -- 172.21.15.98:0/3099139626 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86cc038050 con 0x7f86e40a4920 2024-06-11T19:22:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:22:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 10 monclient: dump: 2024-06-11T19:22:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 1 -- 172.21.15.98:0/3099139626 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86cc044070 con 0x7f86e40a4920 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e9635640 5 monclient: authenticate success, global_id 84971 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e9635640 10 monclient: _renew_subs 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e9635640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e9635640 1 -- 172.21.15.98:0/3099139626 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e4124780 con 0x7f86e40a4920 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 1 -- 172.21.15.98:0/3099139626 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86cc0043d0 con 0x7f86e40a4920 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e17fa640 4 set_mon_vals no callback set 2024-06-11T19:22:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86db7fe640 10 monclient: dump: 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.421 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e9635640 10 client.84971.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e9635640 10 monclient: _renew_subs 2024-06-11T19:22:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e9635640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e9635640 1 -- 172.21.15.98:0/3099139626 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e40a9010 con 0x7f86e40a4920 2024-06-11T19:22:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.394+0000 7f86e9635640 1 librados: init done 2024-06-11T19:22:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.395+0000 7f86db7fe640 1 -- 172.21.15.98:0/3099139626 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f86cc003610 con 0x7f86e40a4920 2024-06-11T19:22:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.395+0000 7f86e9635640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.395+0000 7f86e9635640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.395+0000 7f86e9635640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.395+0000 7f86e9635640 1 -- 172.21.15.98:0/3099139626 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86e4138760 con 0x7f86e40a4920 2024-06-11T19:22:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.395+0000 7f86db7fe640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f86c403fdb0 0x7f86c4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.396+0000 7f86db7fe640 1 -- 172.21.15.98:0/3099139626 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(515..515 src has 1..515) ==== 6685+0+0 (secure 0 0 0) 0x7f86cc073990 con 0x7f86e40a4920 2024-06-11T19:22:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.396+0000 7f86db7fe640 10 client.84971.objecter ms_dispatch 0x7f86e4000b90 osd_map(515..515 src has 1..515) 2024-06-11T19:22:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.396+0000 7f86db7fe640 3 client.84971.objecter handle_osd_map got epochs [515,515] > 0 2024-06-11T19:22:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.396+0000 7f86db7fe640 3 client.84971.objecter handle_osd_map decoding full epoch 515 2024-06-11T19:22:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.396+0000 7f86db7fe640 20 client.84971.objecter dump_active .. 0 homeless 2024-06-11T19:22:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.396+0000 7f86e2ffd640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f86c403fdb0 0x7f86c4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.396+0000 7f86e2ffd640 10 monclient: get_auth_request con 0x7f86c403fdb0 auth_method 0 2024-06-11T19:22:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.396+0000 7f86db7fe640 10 client.84971.objecter ms_handle_connect 0x7f86c403fdb0 2024-06-11T19:22:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.396+0000 7f86e2ffd640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f86c403fdb0 0x7f86c4042270 secure :-1 s=READY pgs=4250 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0113e0 tx=0x7f86dc00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.400+0000 7f86db7fe640 1 -- 172.21.15.98:0/3099139626 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f86cc030070 con 0x7f86e40a4920 2024-06-11T19:22:10.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.400+0000 7f86db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:10.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.400+0000 7f86db7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:10.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.650+0000 7f86e9635640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:22:10.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.650+0000 7f86e9635640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:22:10.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.650+0000 7f86e9635640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:10.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.650+0000 7f86e9635640 1 -- 172.21.15.98:0/3099139626 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f86e40aace0 con 0x7f86e40a4920 2024-06-11T19:22:10.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.651+0000 7f86db7fe640 1 -- 172.21.15.98:0/3099139626 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f86cc003900 con 0x7f86e40a4920 2024-06-11T19:22:10.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.651+0000 7f86db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:22:10.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.651+0000 7f86db7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:10.652 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":64,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:22:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 10 librados: watch_flush enter 2024-06-11T19:22:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 10 librados: watch_flush exit 2024-06-11T19:22:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 20 client.84971.objecter shutdown clearing up homeless session... 2024-06-11T19:22:10.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 10 client.84971.objecter successfully canceled tick 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 -- 172.21.15.98:0/3099139626 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f86c403fdb0 msgr2=0x7f86c4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f86c403fdb0 0x7f86c4042270 secure :-1 s=READY pgs=4250 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0113e0 tx=0x7f86dc00b040 comp rx=0 tx=0).stop 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 10 monclient: shutdown 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 -- 172.21.15.98:0/3099139626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a4920 msgr2=0x7f86e41ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a4920 0x7f86e41ce010 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0097c0 tx=0x7f86cc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 -- 172.21.15.98:0/3099139626 shutdown_connections 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f86c403fdb0 0x7f86c4042270 unknown :-1 s=CLOSED pgs=4250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f86e40a5240 0x7f86e4121f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f86e40a4920 0x7f86e41ce010 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 --2- 172.21.15.98:0/3099139626 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e40a2530 0x7f86e41cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 -- 172.21.15.98:0/3099139626 >> 172.21.15.98:0/3099139626 conn(0x7f86e40ab850 msgr2=0x7f86e40a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 -- 172.21.15.98:0/3099139626 shutdown_connections 2024-06-11T19:22:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 -- 172.21.15.98:0/3099139626 wait complete. 2024-06-11T19:22:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.653+0000 7f86e9635640 1 librados: shutdown 2024-06-11T19:22:10.663 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:22:10.663 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-06-11T19:22:10.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:22:10.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:22:10.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.766+0000 7f055dde2640 10 monclient: get_monmap_and_config 2024-06-11T19:22:10.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.766+0000 7f055dde2640 10 monclient: build_initial_monmap 2024-06-11T19:22:10.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.767+0000 7f055dde2640 10 monclient: monmap: 2024-06-11T19:22:10.766 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:10.766 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.766 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:10.768024+0000 2024-06-11T19:22:10.766 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:10.768024+0000 2024-06-11T19:22:10.766 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:10.767 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.767+0000 7f055dde2640 1 Processor -- start 2024-06-11T19:22:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.767+0000 7f055dde2640 1 -- start start 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.767+0000 7f055dde2640 10 monclient: init 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 10 monclient: _add_conns ranks=[5,2,6,9,4,8,3,1,10,0,7] 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f05580a4920 0x7f05580a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 10 monclient(hunting): picked mon.noname-h con 0x7f05580a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05580a5240 0x7f05580a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 10 monclient(hunting): picked mon.noname-b con 0x7f05580a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a2530 0x7f05580a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 10 monclient(hunting): picked mon.noname-d con 0x7f05580a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f05580a7140 con 0x7f05580a5240 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f05580a2b50 con 0x7f05580a4920 2024-06-11T19:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f05580a2e30 con 0x7f05580a2530 2024-06-11T19:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f055dde2640 10 monclient(hunting): authenticate will time out at 1540.575456s 2024-06-11T19:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05580a5240 0x7f05580a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f0557fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a2530 0x7f05580a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05580a5240 0x7f05580a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57218/0 (socket says 172.21.15.98:57218) 2024-06-11T19:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f0557fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a2530 0x7f05580a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34256/0 (socket says 172.21.15.98:34256) 2024-06-11T19:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 1 -- 172.21.15.98:0/3331388006 learned_addr learned my addr 172.21.15.98:0/3331388006 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 10 monclient(hunting): get_auth_request con 0x7f05580a5240 auth_method 0 2024-06-11T19:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f0557fff640 10 monclient(hunting): get_auth_request con 0x7f05580a2530 auth_method 0 2024-06-11T19:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f0557fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f0557fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f0557fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f05577fe640 1 --2- 172.21.15.98:0/3331388006 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f05580a4920 0x7f05580a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f05577fe640 10 monclient(hunting): get_auth_request con 0x7f05580a4920 auth_method 0 2024-06-11T19:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f05577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f05577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f05577fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f0557fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f0557fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f0557fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f05577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f05577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f05577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 10 monclient(hunting): handle_auth_done global_id 84980 payload 274 2024-06-11T19:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 1 -- 172.21.15.98:0/3331388006 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a2530 msgr2=0x7f05580a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 1 --2- 172.21.15.98:0/3331388006 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a2530 0x7f05580a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 1 -- 172.21.15.98:0/3331388006 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f05580a4920 msgr2=0x7f05580a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 1 --2- 172.21.15.98:0/3331388006 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f05580a4920 0x7f05580a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 1 monclient: found mon.noname-b 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f054ffff640 1 -- 172.21.15.98:0/3331388006 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05580a3110 con 0x7f05580a5240 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.768+0000 7f0557fff640 1 --2- 172.21.15.98:0/3331388006 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a2530 0x7f05580a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f054ffff640 1 --2- 172.21.15.98:0/3331388006 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05580a5240 0x7f05580a59a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f05400098a0 tx=0x7f054002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ed7c03f31c2bdd server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 1 -- 172.21.15.98:0/3331388006 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0540038070 con 0x7f05580a5240 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 10 monclient: dump: 2024-06-11T19:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 10 monclient: _finish_auth 0 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 1 -- 172.21.15.98:0/3331388006 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0540031970 con 0x7f05580a5240 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f055dde2640 5 monclient: authenticate success, global_id 84980 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f055dde2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f055dde2640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 1 -- 172.21.15.98:0/3331388006 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0540031ba0 con 0x7f05580a5240 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f055dde2640 4 set_mon_vals no callback set 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.769+0000 7f0556ffd640 10 monclient: dump: 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 10 monclient: shutdown 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 1 -- 172.21.15.98:0/3331388006 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05580a5240 msgr2=0x7f05580a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 1 --2- 172.21.15.98:0/3331388006 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05580a5240 0x7f05580a59a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f05400098a0 tx=0x7f054002b7a0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 1 -- 172.21.15.98:0/3331388006 shutdown_connections 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 1 --2- 172.21.15.98:0/3331388006 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a2530 0x7f05580a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 1 --2- 172.21.15.98:0/3331388006 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05580a5240 0x7f05580a59a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 1 --2- 172.21.15.98:0/3331388006 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f05580a4920 0x7f05580a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 1 -- 172.21.15.98:0/3331388006 >> 172.21.15.98:0/3331388006 conn(0x7f05580ab850 msgr2=0x7f05580a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 1 -- 172.21.15.98:0/3331388006 shutdown_connections 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 1 -- 172.21.15.98:0/3331388006 wait complete. 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 10 monclient: build_initial_monmap 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.770+0000 7f055dde2640 10 monclient: monmap: 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:10.788 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 1 librados: starting msgr at 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 1 librados: starting objecter 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 1 Processor -- start 2024-06-11T19:22:10.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 1 -- start start 2024-06-11T19:22:10.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 1 librados: setting wanted keys 2024-06-11T19:22:10.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 1 librados: calling monclient init 2024-06-11T19:22:10.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 10 monclient: init 2024-06-11T19:22:10.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:10.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 10 monclient: _add_conns ranks=[1,6,2,7,4,3,0,5,8] 2024-06-11T19:22:10.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05580a2530 0x7f0558137620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 10 monclient(hunting): picked mon.noname-f con 0x7f05580a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:22:10.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a4920 0x7f0558137b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 10 monclient(hunting): picked mon.noname-d con 0x7f05580a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:10.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05580a5240 0x7f05581231a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 10 monclient(hunting): picked mon.noname-b con 0x7f05580a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f05580a6da0 con 0x7f05580a2530 2024-06-11T19:22:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0558138e50 con 0x7f05580a5240 2024-06-11T19:22:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f05580a6420 con 0x7f05580a4920 2024-06-11T19:22:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f055dde2640 10 monclient(hunting): authenticate will time out at 1540.578878s 2024-06-11T19:22:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f054ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a4920 0x7f0558137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f054ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a4920 0x7f0558137b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34258/0 (socket says 172.21.15.98:34258) 2024-06-11T19:22:10.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f0557fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05580a5240 0x7f05581231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f054ffff640 1 -- 172.21.15.98:0/1257258718 learned_addr learned my addr 172.21.15.98:0/1257258718 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f054ffff640 10 monclient(hunting): get_auth_request con 0x7f05580a4920 auth_method 0 2024-06-11T19:22:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f054ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f054ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f054ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f0557fff640 10 monclient(hunting): get_auth_request con 0x7f05580a5240 auth_method 0 2024-06-11T19:22:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f0557fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f0557fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.771+0000 7f0557fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f05577fe640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05580a2530 0x7f0558137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0557fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0557fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0557fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f05577fe640 10 monclient(hunting): get_auth_request con 0x7f05580a2530 auth_method 0 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f05577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f05577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f05577fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f05577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f05577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f05577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 10 monclient(hunting): handle_auth_done global_id 55581 payload 824 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 1 -- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05580a5240 msgr2=0x7f05581231a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05580a5240 0x7f05581231a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 1 -- 172.21.15.98:0/1257258718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05580a2530 msgr2=0x7f0558137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05580a2530 0x7f0558137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:10.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 1 monclient: found mon.noname-d 2024-06-11T19:22:10.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:10.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:22:10.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 1 -- 172.21.15.98:0/1257258718 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05581268c0 con 0x7f05580a4920 2024-06-11T19:22:10.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0557fff640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05580a5240 0x7f05581231a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:10.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f054ffff640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a4920 0x7f0558137b60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0540000c00 tx=0x7f05400046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 10 client.?.objecter ms_handle_connect 0x7f05580a4920 2024-06-11T19:22:10.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:10.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0540038050 con 0x7f05580a4920 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 10 monclient: dump: 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.796 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 10 monclient: _finish_auth 0 2024-06-11T19:22:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:22:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0540039040 con 0x7f05580a4920 2024-06-11T19:22:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f055dde2640 5 monclient: authenticate success, global_id 55581 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f055dde2640 10 monclient: _renew_subs 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f055dde2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f055dde2640 1 -- 172.21.15.98:0/1257258718 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05580a90c0 con 0x7f05580a4920 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0540004350 con 0x7f05580a4920 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f05567fc640 4 set_mon_vals no callback set 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f0554ff9640 10 monclient: dump: 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f055dde2640 10 client.55581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f055dde2640 10 monclient: _renew_subs 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f055dde2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f055dde2640 1 -- 172.21.15.98:0/1257258718 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f0558126e80 con 0x7f05580a4920 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.772+0000 7f055dde2640 1 librados: init done 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f055dde2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f055dde2640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05581236e0 0x7f0558123ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f055dde2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f055dde2640 1 -- 172.21.15.98:0/1257258718 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0558069750 con 0x7f05581236e0 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f0540046030 con 0x7f05580a4920 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f05577fe640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05581236e0 0x7f0558123ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f05577fe640 10 monclient: get_auth_request con 0x7f05581236e0 auth_method 0 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f05577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f05577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f05577fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f05577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f05577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f05577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f0554ff9640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f053803fdb0 0x7f0538042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(516..516 src has 1..516) ==== 6685+0+0 (secure 0 0 0) 0x7f0540044020 con 0x7f05580a4920 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f0554ff9640 10 client.55581.objecter ms_dispatch 0x7f0558000b90 osd_map(516..516 src has 1..516) 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f0554ff9640 3 client.55581.objecter handle_osd_map got epochs [516,516] > 0 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f0554ff9640 3 client.55581.objecter handle_osd_map decoding full epoch 516 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f05577fe640 10 monclient(hunting): handle_auth_done global_id 65709 payload 824 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f05577fe640 1 -- 172.21.15.98:0/1257258718 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05580a92f0 con 0x7f05581236e0 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f0554ff9640 20 client.55581.objecter dump_active .. 0 homeless 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f0557fff640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f053803fdb0 0x7f0538042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f0557fff640 10 monclient: get_auth_request con 0x7f053803fdb0 auth_method 0 2024-06-11T19:22:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f0554ff9640 10 client.55581.objecter ms_handle_connect 0x7f05581236e0 2024-06-11T19:22:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f0554ff9640 10 client.55581.objecter resend_mon_ops 2024-06-11T19:22:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.773+0000 7f05577fe640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05581236e0 0x7f0558123ac0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0548011860 tx=0x7f0548011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0548019070 con 0x7f05581236e0 2024-06-11T19:22:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0554ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0554ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0554ff9640 10 monclient: dump: 2024-06-11T19:22:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.803 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0548004240 con 0x7f05581236e0 2024-06-11T19:22:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0554ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0554ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05581236e0 msgr2=0x7f0558123ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0557fff640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f053803fdb0 0x7f0538042270 secure :-1 s=READY pgs=4251 cs=0 l=1 rev1=1 crypto rx=0x7f0558124ca0 tx=0x7f055000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0554ff9640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05581236e0 0x7f0558123ac0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0548011860 tx=0x7f0548011d20 comp rx=0 tx=0).stop 2024-06-11T19:22:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05581236e0 msgr2=0x7f0558123ac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0554ff9640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05581236e0 0x7f0558123ac0 secure :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0548011860 tx=0x7f0548011d20 comp rx=0 tx=0).stop 2024-06-11T19:22:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.774+0000 7f0554ff9640 10 client.55581.objecter ms_handle_connect 0x7f053803fdb0 2024-06-11T19:22:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.780+0000 7f055dde2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.780+0000 7f055dde2640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0558124000 0x7f055809e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.780+0000 7f055dde2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.780+0000 7f055dde2640 1 -- 172.21.15.98:0/1257258718 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0558050830 con 0x7f0558124000 2024-06-11T19:22:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.780+0000 7f05577fe640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0558124000 0x7f055809e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.780+0000 7f05577fe640 10 monclient: get_auth_request con 0x7f0558124000 auth_method 0 2024-06-11T19:22:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.780+0000 7f05577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.780+0000 7f05577fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.780+0000 7f05577fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.780+0000 7f05577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.780+0000 7f05577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.780+0000 7f05577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f05577fe640 10 monclient(hunting): handle_auth_done global_id 65718 payload 824 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f05577fe640 1 -- 172.21.15.98:0/1257258718 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05580a66f0 con 0x7f0558124000 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f05577fe640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0558124000 0x7f055809e9f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0548000c00 tx=0x7f05480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 10 client.55581.objecter ms_handle_connect 0x7f0558124000 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 10 client.55581.objecter resend_mon_ops 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0548019040 con 0x7f0558124000 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 10 monclient: dump: 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f054801e450 con 0x7f0558124000 2024-06-11T19:22:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0558124000 msgr2=0x7f055809e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0558124000 0x7f055809e9f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0548000c00 tx=0x7f05480046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 1 -- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0558124000 msgr2=0x7f055809e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0554ff9640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0558124000 0x7f055809e9f0 secure :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0548000c00 tx=0x7f05480046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.809 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:10.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:10.809 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:10.810 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:10.810 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:10.810 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 522609, 2024-06-11T19:22:10.810 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:10.810 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:10.810 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:10.810 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:10.810 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:10.810 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:10.810 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:10.810 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:10.810 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:10.811 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:10.811 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:10.811 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-06-11T19:22:10.811 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:10.811 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:10.811 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:10.811 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:10.811 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:10.811 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:10.811 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:10.812 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:10.813 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:10.814 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:10.814 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:10.814 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:10.814 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:10.814 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:10.814 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:10.814 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:10.818 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:10.819 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:10.819 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:10.819 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:10.819 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:10.819 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:10.819 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:10.819 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:10.819 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:10.819 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:10.819 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:10.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:10.820 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:10.820 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:10.820 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:10.820 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:10.820 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:10.820 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:10.820 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:10.820 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:10.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.820 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:10.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.821 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:10.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:10.821 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:10.821 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:10.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.822 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:10.822 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:10.822 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:10.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.824 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.826 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.827 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.827 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:10.827 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:10.827 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:10.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:10.830 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.831 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.832 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:10.832 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:10.832 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.832 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.832 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.832 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:10.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:10.833 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:10.833 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:10.833 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.834 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.834 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:10.834 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:10.834 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.834 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.834 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.835 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:10.837 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:10.838 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.840 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:10.840 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:10.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f055dde2640 10 librados: watch_flush enter 2024-06-11T19:22:10.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f055dde2640 10 librados: watch_flush exit 2024-06-11T19:22:10.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f055dde2640 20 client.55581.objecter shutdown clearing up homeless session... 2024-06-11T19:22:10.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f055dde2640 10 client.55581.objecter successfully canceled tick 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f055dde2640 1 -- 172.21.15.98:0/1257258718 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f053803fdb0 msgr2=0x7f0538042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f055dde2640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f053803fdb0 0x7f0538042270 secure :-1 s=READY pgs=4251 cs=0 l=1 rev1=1 crypto rx=0x7f0558124ca0 tx=0x7f055000b040 comp rx=0 tx=0).stop 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f055dde2640 10 monclient: shutdown 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f055dde2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f055dde2640 1 -- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a4920 msgr2=0x7f0558137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f0557fff640 1 -- 172.21.15.98:0/1257258718 reap_dead start 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.781+0000 7f055dde2640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a4920 0x7f0558137b60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0540000c00 tx=0x7f05400046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.782+0000 7f055dde2640 1 -- 172.21.15.98:0/1257258718 shutdown_connections 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.782+0000 7f055dde2640 1 --2- 172.21.15.98:0/1257258718 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05580a4920 0x7f0558137b60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.782+0000 7f055dde2640 1 -- 172.21.15.98:0/1257258718 >> 172.21.15.98:0/1257258718 conn(0x7f05580ab850 msgr2=0x7f05581bffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.782+0000 7f055dde2640 1 -- 172.21.15.98:0/1257258718 shutdown_connections 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.782+0000 7f055dde2640 1 -- 172.21.15.98:0/1257258718 wait complete. 2024-06-11T19:22:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.782+0000 7f055dde2640 1 librados: shutdown 2024-06-11T19:22:10.843 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:10.843 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:22:10.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:22:10.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.943+0000 7fbcd617d640 10 monclient: get_monmap_and_config 2024-06-11T19:22:10.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.943+0000 7fbcd617d640 10 monclient: build_initial_monmap 2024-06-11T19:22:10.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.943+0000 7fbcd617d640 10 monclient: monmap: 2024-06-11T19:22:10.942 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:10.942 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.942 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:10.944202+0000 2024-06-11T19:22:10.942 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:10.944202+0000 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:10.943 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.943+0000 7fbcd617d640 1 Processor -- start 2024-06-11T19:22:10.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 1 -- start start 2024-06-11T19:22:10.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 10 monclient: init 2024-06-11T19:22:10.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:10.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 10 monclient: _add_conns ranks=[8,0,4,10,7,6,9,2,3,1,5] 2024-06-11T19:22:10.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcd00a4920 0x7fbcd00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 10 monclient(hunting): picked mon.noname-e con 0x7fbcd00a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:10.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbcd00a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:10.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbcd00a2530 0x7fbcd00a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd00a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:10.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd00a7140 con 0x7fbcd00a5240 2024-06-11T19:22:10.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fbcd00a2b50 con 0x7fbcd00a2530 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbcd00a2e30 con 0x7fbcd00a4920 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcd617d640 10 monclient(hunting): authenticate will time out at 1540.751532s 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccf7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcd00a4920 0x7fbcd00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcceffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbcd00a5240 0x7fbcd00a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56830/0 (socket says 172.21.15.98:56830) 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcceffd640 1 -- 172.21.15.98:0/1367821788 learned_addr learned my addr 172.21.15.98:0/1367821788 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 1 --2- 172.21.15.98:0/1367821788 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbcd00a2530 0x7fbcd00a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccf7fe640 10 monclient(hunting): get_auth_request con 0x7fbcd00a4920 auth_method 0 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccf7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 10 monclient(hunting): get_auth_request con 0x7fbcd00a2530 auth_method 0 2024-06-11T19:22:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 10 monclient(hunting): handle_auth_done global_id 114214 payload 274 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcce7fc640 1 -- 172.21.15.98:0/1367821788 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 967102584 0 0) 0x7fbcd00a7140 con 0x7fbcd00a5240 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 1 -- 172.21.15.98:0/1367821788 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcd00a4920 msgr2=0x7fbcd00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 1 --2- 172.21.15.98:0/1367821788 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcd00a4920 0x7fbcd00a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 1 -- 172.21.15.98:0/1367821788 >> v1:172.21.15.98:6789/0 conn(0x7fbcd00a5240 legacy=0x7fbcd00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 1 monclient: found mon.noname-c 2024-06-11T19:22:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:22:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccffff640 1 -- 172.21.15.98:0/1367821788 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd00a3110 con 0x7fbcd00a2530 2024-06-11T19:22:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbcce7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:22:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.944+0000 7fbccf7fe640 1 --2- 172.21.15.98:0/1367821788 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcd00a4920 0x7fbcd00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbccffff640 1 --2- 172.21.15.98:0/1367821788 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbcd00a2530 0x7fbcd00a60b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8009dc0 tx=0x7fbcc802bd50 comp rx=0 tx=0).ready entity=mon.4 client_cookie=fa40f10792205580 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 1 -- 172.21.15.98:0/1367821788 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbcc8039070 con 0x7fbcd00a2530 2024-06-11T19:22:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:22:10.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 10 monclient: dump: 2024-06-11T19:22:10.958 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.958 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.958 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.958 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.958 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.958 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.958 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.958 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.958 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 10 monclient: _finish_auth 0 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 1 -- 172.21.15.98:0/1367821788 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc802cd70 con 0x7fbcd00a2530 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcd617d640 5 monclient: authenticate success, global_id 114214 2024-06-11T19:22:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcd617d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcd617d640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcd617d640 4 set_mon_vals no callback set 2024-06-11T19:22:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 1 -- 172.21.15.98:0/1367821788 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbcc80327b0 con 0x7fbcd00a2530 2024-06-11T19:22:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.945+0000 7fbcce7fc640 10 monclient: dump: 2024-06-11T19:22:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.961 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.961 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.961 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.961 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.961 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.961 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.961 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.961 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.961 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.961 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.962 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.962 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.962 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.946+0000 7fbcd617d640 10 monclient: shutdown 2024-06-11T19:22:10.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.946+0000 7fbcd617d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:10.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.946+0000 7fbcd617d640 1 -- 172.21.15.98:0/1367821788 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbcd00a2530 msgr2=0x7fbcd00a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.946+0000 7fbcd617d640 1 --2- 172.21.15.98:0/1367821788 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbcd00a2530 0x7fbcd00a60b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8009dc0 tx=0x7fbcc802bd50 comp rx=0 tx=0).stop 2024-06-11T19:22:10.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.946+0000 7fbcd617d640 1 -- 172.21.15.98:0/1367821788 shutdown_connections 2024-06-11T19:22:10.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.946+0000 7fbcd617d640 1 --2- 172.21.15.98:0/1367821788 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbcd00a2530 0x7fbcd00a60b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.946+0000 7fbcd617d640 1 --2- 172.21.15.98:0/1367821788 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbcd00a4920 0x7fbcd00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.946+0000 7fbcd617d640 1 -- 172.21.15.98:0/1367821788 >> 172.21.15.98:0/1367821788 conn(0x7fbcd00ab850 msgr2=0x7fbcd00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.946+0000 7fbcd617d640 1 -- 172.21.15.98:0/1367821788 shutdown_connections 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.946+0000 7fbcd617d640 1 -- 172.21.15.98:0/1367821788 wait complete. 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.946+0000 7fbcd617d640 10 monclient: build_initial_monmap 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.946+0000 7fbcd617d640 10 monclient: monmap: 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 1 librados: starting msgr at 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 1 librados: starting objecter 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 1 Processor -- start 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 1 -- start start 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 1 librados: setting wanted keys 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 1 librados: calling monclient init 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 10 monclient: init 2024-06-11T19:22:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 10 monclient: _add_conns ranks=[5,1,3,0,7,8,2,4,6] 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbcd00a2530 0x7fbcd01cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 10 monclient(hunting): picked mon.noname-h con 0x7fbcd00a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbcd00a4920 0x7fbcd01ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 10 monclient(hunting): picked mon.noname-f con 0x7fbcd00a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbcd00a5240 0x7fbcd0122fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 10 monclient(hunting): picked mon.noname-g con 0x7fbcd00a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd00a6da0 con 0x7fbcd00a4920 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd0138e50 con 0x7fbcd00a5240 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fbcd00a65f0 con 0x7fbcd00a2530 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.947+0000 7fbcd617d640 10 monclient(hunting): authenticate will time out at 1540.754993s 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccf7fe640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbcd00a2530 0x7fbcd01cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbcd00a4920 0x7fbcd01ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccffff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbcd00a5240 0x7fbcd0122fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbcd00a4920 0x7fbcd01ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:53318/0 (socket says 172.21.15.98:53318) 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 1 -- 172.21.15.98:0/1106958865 learned_addr learned my addr 172.21.15.98:0/1106958865 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 10 monclient(hunting): get_auth_request con 0x7fbcd00a4920 auth_method 0 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccffff640 10 monclient(hunting): get_auth_request con 0x7fbcd00a5240 auth_method 0 2024-06-11T19:22:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccf7fe640 10 monclient(hunting): get_auth_request con 0x7fbcd00a2530 auth_method 0 2024-06-11T19:22:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccf7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 10 monclient(hunting): handle_auth_done global_id 32716 payload 824 2024-06-11T19:22:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 1 -- 172.21.15.98:0/1106958865 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbcd00a2530 msgr2=0x7fbcd01cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fbcd00a2530 0x7fbcd01cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 1 -- 172.21.15.98:0/1106958865 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbcd00a5240 msgr2=0x7fbcd0122fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbcd00a5240 0x7fbcd0122fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 10 monclient: _finish_hunting 0 2024-06-11T19:22:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 1 monclient: found mon.noname-f 2024-06-11T19:22:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:22:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 1 -- 172.21.15.98:0/1106958865 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd0127840 con 0x7fbcd00a4920 2024-06-11T19:22:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcceffd640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbcd00a4920 0x7fbcd01ce190 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400c370 tx=0x7fbcc400c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 10 client.?.objecter ms_handle_connect 0x7fbcd00a4920 2024-06-11T19:22:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbcc40047e0 con 0x7fbcd00a4920 2024-06-11T19:22:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:22:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 10 monclient: dump: 2024-06-11T19:22:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.971 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 10 monclient: _finish_auth 0 2024-06-11T19:22:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:22:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc40049a0 con 0x7fbcd00a4920 2024-06-11T19:22:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcd617d640 5 monclient: authenticate success, global_id 32716 2024-06-11T19:22:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcd617d640 10 monclient: _renew_subs 2024-06-11T19:22:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcd617d640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:22:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcd617d640 1 -- 172.21.15.98:0/1106958865 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd0126e60 con 0x7fbcd00a4920 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbcc40133c0 con 0x7fbcd00a4920 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbcb7fff640 10 monclient: dump: 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.972 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.948+0000 7fbccdffb640 4 set_mon_vals no callback set 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbcd617d640 10 client.32716.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbcd617d640 10 monclient: _renew_subs 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbcd617d640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbcd617d640 1 -- 172.21.15.98:0/1106958865 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd00a8fa0 con 0x7fbcd00a4920 2024-06-11T19:22:10.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbcd617d640 1 librados: init done 2024-06-11T19:22:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbcd617d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbcd617d640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0123ca0 0x7fbcd002cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbcd617d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbcd617d640 1 -- 172.21.15.98:0/1106958865 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd0050830 con 0x7fbcd0123ca0 2024-06-11T19:22:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbccf7fe640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0123ca0 0x7fbcd002cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbccf7fe640 10 monclient: get_auth_request con 0x7fbcd0123ca0 auth_method 0 2024-06-11T19:22:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbccf7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.949+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fbcc4011070 con 0x7fbcd00a4920 2024-06-11T19:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_done global_id 84998 payload 824 2024-06-11T19:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbccf7fe640 1 -- 172.21.15.98:0/1106958865 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbcd00a7360 con 0x7fbcd0123ca0 2024-06-11T19:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbcbc03fdb0 0x7fbcbc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(516..516 src has 1..516) ==== 6685+0+0 (secure 0 0 0) 0x7fbcc4051760 con 0x7fbcd00a4920 2024-06-11T19:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 10 client.32716.objecter ms_dispatch 0x7fbcd0000b90 osd_map(516..516 src has 1..516) 2024-06-11T19:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 3 client.32716.objecter handle_osd_map got epochs [516,516] > 0 2024-06-11T19:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 3 client.32716.objecter handle_osd_map decoding full epoch 516 2024-06-11T19:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbccf7fe640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0123ca0 0x7fbcd002cb40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00118e0 tx=0x7fbcc0011da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 20 client.32716.objecter dump_active .. 0 homeless 2024-06-11T19:22:10.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 10 client.32716.objecter ms_handle_connect 0x7fbcd0123ca0 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 10 client.32716.objecter resend_mon_ops 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbcc000fc20 con 0x7fbcd0123ca0 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbccffff640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbcbc03fdb0 0x7fbcbc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 10 monclient: dump: 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.976 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.978 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbccffff640 10 monclient: get_auth_request con 0x7fbcbc03fdb0 auth_method 0 2024-06-11T19:22:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbcc000fc20 con 0x7fbcd0123ca0 2024-06-11T19:22:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0123ca0 msgr2=0x7fbcd002cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0123ca0 0x7fbcd002cb40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00118e0 tx=0x7fbcc0011da0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0123ca0 msgr2=0x7fbcd002cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:10.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0123ca0 0x7fbcd002cb40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbcb7fff640 10 client.32716.objecter ms_handle_connect 0x7fbcbc03fdb0 2024-06-11T19:22:10.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.950+0000 7fbccffff640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbcbc03fdb0 0x7fbcbc042270 secure :-1 s=READY pgs=4252 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8036000 tx=0x7fbcc8030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcd617d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcd617d640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0124080 0x7fbcd002d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcd617d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcd617d640 1 -- 172.21.15.98:0/1106958865 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd0124b00 con 0x7fbcd0124080 2024-06-11T19:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbccf7fe640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0124080 0x7fbcd002d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbccf7fe640 10 monclient: get_auth_request con 0x7fbcd0124080 auth_method 0 2024-06-11T19:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbccf7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbccf7fe640 10 monclient(hunting): handle_auth_done global_id 85007 payload 824 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbccf7fe640 1 -- 172.21.15.98:0/1106958865 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbcd00a0100 con 0x7fbcd0124080 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 10 client.32716.objecter ms_handle_connect 0x7fbcd0124080 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 10 client.32716.objecter resend_mon_ops 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbccf7fe640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0124080 0x7fbcd002d080 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00118e0 tx=0x7fbcc00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbcc0022bf0 con 0x7fbcd0124080 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 10 monclient: dump: 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:10.983 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:10.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fbcc002c420 con 0x7fbcd0124080 2024-06-11T19:22:10.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:10.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:10.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0124080 msgr2=0x7fbcd002d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:10.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0124080 0x7fbcd002d080 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00118e0 tx=0x7fbcc00046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 1 -- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0124080 msgr2=0x7fbcd002d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:10.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.957+0000 7fbcb7fff640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbcd0124080 0x7fbcd002d080 secure :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00118e0 tx=0x7fbcc00046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:10.984 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:10.984 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:10.984 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:10.984 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 402731, 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:10.986 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:10.988 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:10.988 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:10.988 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:10.988 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.988 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:10.988 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:10.988 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:10.989 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:10.990 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.992 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:10.993 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:10.993 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:10.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:10.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:10.994 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:10.997 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:10.997 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.997 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:10.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:10.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:10.999 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:10.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:10.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:10.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:10.999 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:10.999 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:10.999 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:11.001 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:11.002 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:11.003 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.003 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.003 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:11.003 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:11.003 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.003 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.007 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.008 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:11.009 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.010 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.010 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:11.010 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 10 librados: watch_flush enter 2024-06-11T19:22:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 10 librados: watch_flush exit 2024-06-11T19:22:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 20 client.32716.objecter shutdown clearing up homeless session... 2024-06-11T19:22:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 10 client.32716.objecter successfully canceled tick 2024-06-11T19:22:11.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 1 -- 172.21.15.98:0/1106958865 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbcbc03fdb0 msgr2=0x7fbcbc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fbcbc03fdb0 0x7fbcbc042270 secure :-1 s=READY pgs=4252 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8036000 tx=0x7fbcc8030000 comp rx=0 tx=0).stop 2024-06-11T19:22:11.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 10 monclient: shutdown 2024-06-11T19:22:11.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:11.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 1 -- 172.21.15.98:0/1106958865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbcd00a4920 msgr2=0x7fbcd01ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 1 --2- 172.21.15.98:0/1106958865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbcd00a4920 0x7fbcd01ce190 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400c370 tx=0x7fbcc400c830 comp rx=0 tx=0).stop 2024-06-11T19:22:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbccffff640 1 -- 172.21.15.98:0/1106958865 reap_dead start 2024-06-11T19:22:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbccffff640 1 -- 172.21.15.98:0/1106958865 reap_dead start 2024-06-11T19:22:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 1 -- 172.21.15.98:0/1106958865 shutdown_connections 2024-06-11T19:22:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 1 -- 172.21.15.98:0/1106958865 >> 172.21.15.98:0/1106958865 conn(0x7fbcd00ab850 msgr2=0x7fbcd01beed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 1 -- 172.21.15.98:0/1106958865 shutdown_connections 2024-06-11T19:22:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 1 -- 172.21.15.98:0/1106958865 wait complete. 2024-06-11T19:22:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:10.958+0000 7fbcd617d640 1 librados: shutdown 2024-06-11T19:22:11.017 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:11.018 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:22:11.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:22:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.118+0000 7f5d83415640 10 monclient: get_monmap_and_config 2024-06-11T19:22:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.118+0000 7f5d83415640 10 monclient: build_initial_monmap 2024-06-11T19:22:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.118+0000 7f5d83415640 10 monclient: monmap: 2024-06-11T19:22:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:11.119790+0000 2024-06-11T19:22:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:11.119790+0000 2024-06-11T19:22:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.118 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:11.119 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.119+0000 7f5d83415640 1 Processor -- start 2024-06-11T19:22:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.119+0000 7f5d83415640 1 -- start start 2024-06-11T19:22:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.119+0000 7f5d83415640 10 monclient: init 2024-06-11T19:22:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.119+0000 7f5d83415640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 10 monclient: _add_conns ranks=[7,0,5,9,6,1,2,10,8,3,4] 2024-06-11T19:22:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d7c0a4920 0x7f5d7c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 10 monclient(hunting): picked mon.noname-i con 0x7f5d7c0a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d7c0a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d7c0a2530 0x7f5d7c0a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 10 monclient(hunting): picked mon.noname-h con 0x7f5d7c0a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d7c0a7140 con 0x7f5d7c0a5240 2024-06-11T19:22:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f5d7c0a2b50 con 0x7f5d7c0a2530 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f5d7c0a2e30 con 0x7f5d7c0a4920 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d83415640 10 monclient(hunting): authenticate will time out at 1540.927083s 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d809b0640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5d7c0a5240 0x7f5d7c0a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56840/0 (socket says 172.21.15.98:56840) 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d809b0640 1 -- 172.21.15.98:0/1645377404 learned_addr learned my addr 172.21.15.98:0/1645377404 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 1 --2- 172.21.15.98:0/1645377404 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d7c0a2530 0x7f5d7c0a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d811b1640 1 --2- 172.21.15.98:0/1645377404 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d7c0a4920 0x7f5d7c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 10 monclient(hunting): get_auth_request con 0x7f5d7c0a2530 auth_method 0 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d811b1640 10 monclient(hunting): get_auth_request con 0x7f5d7c0a4920 auth_method 0 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d811b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d811b1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d811b1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d73fff640 1 -- 172.21.15.98:0/1645377404 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4036040375 0 0) 0x7f5d7c0a7140 con 0x7f5d7c0a5240 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d73fff640 10 monclient(hunting): _negotiate 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d73fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d73fff640 10 monclient(hunting): my global_id is 65736 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d73fff640 1 -- 172.21.15.98:0/1645377404 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d68003520 con 0x7f5d7c0a5240 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d811b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d811b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d811b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 10 monclient(hunting): handle_auth_done global_id 23522 payload 274 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 1 -- 172.21.15.98:0/1645377404 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d7c0a4920 msgr2=0x7f5d7c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 1 --2- 172.21.15.98:0/1645377404 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d7c0a4920 0x7f5d7c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 1 -- 172.21.15.98:0/1645377404 >> v1:172.21.15.98:6789/0 conn(0x7f5d7c0a5240 legacy=0x7f5d7c0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d73fff640 1 -- 172.21.15.98:0/1645377404 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2339325292 0 0) 0x7f5d68003520 con 0x7f5d7c0a5240 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 1 monclient: found mon.noname-h 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d819b2640 1 -- 172.21.15.98:0/1645377404 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7c0a3110 con 0x7f5d7c0a2530 2024-06-11T19:22:11.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.120+0000 7f5d73fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d819b2640 1 --2- 172.21.15.98:0/1645377404 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d7c0a2530 0x7f5d7c0a60b0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009dc0 tx=0x7f5d7402bc30 comp rx=0 tx=0).ready entity=mon.5 client_cookie=ec280ed943311a19 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 1 -- 172.21.15.98:0/1645377404 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d74039070 con 0x7f5d7c0a2530 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 10 monclient: dump: 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.133 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d83415640 5 monclient: authenticate success, global_id 23522 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d83415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 1 -- 172.21.15.98:0/1645377404 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d7402cbd0 con 0x7f5d7c0a2530 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:11.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d83415640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d83415640 4 set_mon_vals no callback set 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 1 -- 172.21.15.98:0/1645377404 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d6c001960 con 0x7f5d7c0a2530 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.121+0000 7f5d73fff640 10 monclient: dump: 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.136 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.122+0000 7f5d83415640 10 monclient: shutdown 2024-06-11T19:22:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.122+0000 7f5d83415640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.122+0000 7f5d83415640 1 -- 172.21.15.98:0/1645377404 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d7c0a2530 msgr2=0x7f5d7c0a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.122+0000 7f5d83415640 1 --2- 172.21.15.98:0/1645377404 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d7c0a2530 0x7f5d7c0a60b0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009dc0 tx=0x7f5d7402bc30 comp rx=0 tx=0).stop 2024-06-11T19:22:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.122+0000 7f5d83415640 1 -- 172.21.15.98:0/1645377404 shutdown_connections 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.122+0000 7f5d83415640 1 --2- 172.21.15.98:0/1645377404 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f5d7c0a2530 0x7f5d7c0a60b0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.122+0000 7f5d83415640 1 --2- 172.21.15.98:0/1645377404 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f5d7c0a4920 0x7f5d7c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.122+0000 7f5d83415640 1 -- 172.21.15.98:0/1645377404 >> 172.21.15.98:0/1645377404 conn(0x7f5d7c0ab850 msgr2=0x7f5d7c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.122+0000 7f5d83415640 1 -- 172.21.15.98:0/1645377404 shutdown_connections 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.122+0000 7f5d83415640 1 -- 172.21.15.98:0/1645377404 wait complete. 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.122+0000 7f5d83415640 10 monclient: build_initial_monmap 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.122+0000 7f5d83415640 10 monclient: monmap: 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 1 librados: starting msgr at 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 1 librados: starting objecter 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 1 Processor -- start 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 1 -- start start 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 1 librados: setting wanted keys 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 1 librados: calling monclient init 2024-06-11T19:22:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 10 monclient: init 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 10 monclient: _add_conns ranks=[8,6,1,2,5,7,0,4,3] 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d7c0a2530 0x7f5d7c1cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 10 monclient(hunting): picked mon.noname-e con 0x7f5d7c0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d7c0a4920 0x7f5d7c1cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 10 monclient(hunting): picked mon.noname-d con 0x7f5d7c0a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d7c0a5240 0x7f5d7c124ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 10 monclient(hunting): picked mon.noname-f con 0x7f5d7c0a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c0a6da0 con 0x7f5d7c0a5240 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5d7c138e50 con 0x7f5d7c0a4920 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5d7c0a65f0 con 0x7f5d7c0a2530 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d83415640 10 monclient(hunting): authenticate will time out at 1540.930644s 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d811b1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d7c0a2530 0x7f5d7c1cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d809b0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d7c0a4920 0x7f5d7c1cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d811b1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d7c0a2530 0x7f5d7c1cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:58086/0 (socket says 172.21.15.98:58086) 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d811b1640 1 -- 172.21.15.98:0/1225666305 learned_addr learned my addr 172.21.15.98:0/1225666305 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d811b1640 10 monclient(hunting): get_auth_request con 0x7f5d7c0a2530 auth_method 0 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d811b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d811b1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d811b1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d809b0640 10 monclient(hunting): get_auth_request con 0x7f5d7c0a4920 auth_method 0 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d809b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d809b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d809b0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d811b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d811b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d811b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d809b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d809b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d809b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d819b2640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d7c0a5240 0x7f5d7c124ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d819b2640 10 monclient(hunting): get_auth_request con 0x7f5d7c0a5240 auth_method 0 2024-06-11T19:22:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d819b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d819b2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.123+0000 7f5d819b2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d811b1640 10 monclient(hunting): handle_auth_done global_id 114164 payload 824 2024-06-11T19:22:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d811b1640 1 -- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d7c0a4920 msgr2=0x7f5d7c1cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d811b1640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d7c0a4920 0x7f5d7c1cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d811b1640 1 -- 172.21.15.98:0/1225666305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d7c0a5240 msgr2=0x7f5d7c124ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d811b1640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d7c0a5240 0x7f5d7c124ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d811b1640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d811b1640 1 monclient: found mon.noname-e 2024-06-11T19:22:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d811b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d811b1640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d811b1640 1 -- 172.21.15.98:0/1225666305 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7c1289a0 con 0x7f5d7c0a2530 2024-06-11T19:22:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d809b0640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d7c0a4920 0x7f5d7c1cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d819b2640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d7c0a5240 0x7f5d7c124ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:22:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d811b1640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d7c0a2530 0x7f5d7c1cdaa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800e9e0 tx=0x7f5d7800eea0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 10 client.?.objecter ms_handle_connect 0x7f5d7c0a2530 2024-06-11T19:22:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d78017070 con 0x7f5d7c0a2530 2024-06-11T19:22:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:22:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 10 monclient: dump: 2024-06-11T19:22:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d83415640 5 monclient: authenticate success, global_id 114164 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d7800ce40 con 0x7f5d7c0a2530 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d780138e0 con 0x7f5d7c0a2530 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d83415640 10 monclient: _renew_subs 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d83415640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d83415640 1 -- 172.21.15.98:0/1225666305 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d7c1277d0 con 0x7f5d7c0a2530 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d737fe640 4 set_mon_vals no callback set 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d71ffb640 10 monclient: dump: 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.146 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.146 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.146 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.146 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.146 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.146 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.146 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.146 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.146 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.146 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.147 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d83415640 10 client.114164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d83415640 10 monclient: _renew_subs 2024-06-11T19:22:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d83415640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d83415640 1 -- 172.21.15.98:0/1225666305 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d7c1286c0 con 0x7f5d7c0a2530 2024-06-11T19:22:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.124+0000 7f5d83415640 1 librados: init done 2024-06-11T19:22:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f5d7801a070 con 0x7f5d7c0a2530 2024-06-11T19:22:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d83415640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d83415640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125530 0x7f5d7c125910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d83415640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d83415640 1 -- 172.21.15.98:0/1225666305 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5d7c050830 con 0x7f5d7c125530 2024-06-11T19:22:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d71ffb640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d6003fdb0 0x7f5d60042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(516..516 src has 1..516) ==== 6685+0+0 (secure 0 0 0) 0x7f5d7800f040 con 0x7f5d7c0a2530 2024-06-11T19:22:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d71ffb640 10 client.114164.objecter ms_dispatch 0x7f5d7c000b90 osd_map(516..516 src has 1..516) 2024-06-11T19:22:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d71ffb640 3 client.114164.objecter handle_osd_map got epochs [516,516] > 0 2024-06-11T19:22:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d71ffb640 3 client.114164.objecter handle_osd_map decoding full epoch 516 2024-06-11T19:22:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d809b0640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125530 0x7f5d7c125910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d809b0640 10 monclient: get_auth_request con 0x7f5d7c125530 auth_method 0 2024-06-11T19:22:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d809b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d809b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d809b0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d809b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d809b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d809b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d71ffb640 20 client.114164.objecter dump_active .. 0 homeless 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d819b2640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d6003fdb0 0x7f5d60042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d819b2640 10 monclient: get_auth_request con 0x7f5d6003fdb0 auth_method 0 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d809b0640 10 monclient(hunting): handle_auth_done global_id 114223 payload 824 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d809b0640 1 -- 172.21.15.98:0/1225666305 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d7c1387f0 con 0x7f5d7c125530 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d71ffb640 10 client.114164.objecter ms_handle_connect 0x7f5d7c125530 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d71ffb640 10 client.114164.objecter resend_mon_ops 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.125+0000 7f5d809b0640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125530 0x7f5d7c125910 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c073fc0 tx=0x7f5d6c00ff70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d6c00d880 con 0x7f5d7c125530 2024-06-11T19:22:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d71ffb640 10 monclient: dump: 2024-06-11T19:22:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.151 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.151 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.151 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.151 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.151 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.151 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.151 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d71ffb640 10 client.114164.objecter ms_handle_connect 0x7f5d6003fdb0 2024-06-11T19:22:11.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d819b2640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d6003fdb0 0x7f5d60042270 secure :-1 s=READY pgs=4253 cs=0 l=1 rev1=1 crypto rx=0x7f5d74036000 tx=0x7f5d74030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5d6c018070 con 0x7f5d7c125530 2024-06-11T19:22:11.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d71ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:11.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d71ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:11.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125530 msgr2=0x7f5d7c125910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d71ffb640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125530 0x7f5d7c125910 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c073fc0 tx=0x7f5d6c00ff70 comp rx=0 tx=0).stop 2024-06-11T19:22:11.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125530 msgr2=0x7f5d7c125910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:11.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.126+0000 7f5d71ffb640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125530 0x7f5d7c125910 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.132+0000 7f5d83415640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:11.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.132+0000 7f5d83415640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125e50 0x7f5d7c1215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.132+0000 7f5d83415640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.132+0000 7f5d83415640 1 -- 172.21.15.98:0/1225666305 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5d7c069740 con 0x7f5d7c125e50 2024-06-11T19:22:11.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.132+0000 7f5d809b0640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125e50 0x7f5d7c1215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.132+0000 7f5d809b0640 10 monclient: get_auth_request con 0x7f5d7c125e50 auth_method 0 2024-06-11T19:22:11.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.132+0000 7f5d809b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.132+0000 7f5d809b0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.132+0000 7f5d809b0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.132+0000 7f5d809b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.132+0000 7f5d809b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.132+0000 7f5d809b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d809b0640 10 monclient(hunting): handle_auth_done global_id 114232 payload 824 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d809b0640 1 -- 172.21.15.98:0/1225666305 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5d7c0a63f0 con 0x7f5d7c125e50 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 10 client.114164.objecter ms_handle_connect 0x7f5d7c125e50 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 10 client.114164.objecter resend_mon_ops 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d809b0640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125e50 0x7f5d7c1215f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c017000 tx=0x7f5d6c000ed0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d6c007630 con 0x7f5d7c125e50 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:11.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 10 monclient: dump: 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.156 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.157 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.157 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f5d6c007630 con 0x7f5d7c125e50 2024-06-11T19:22:11.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:11.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:11.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125e50 msgr2=0x7f5d7c1215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125e50 0x7f5d7c1215f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c017000 tx=0x7f5d6c000ed0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 1 -- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125e50 msgr2=0x7f5d7c1215f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:11.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d71ffb640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d7c125e50 0x7f5d7c1215f0 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c017000 tx=0x7f5d6c000ed0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.157 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:11.157 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 19499, 2024-06-11T19:22:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:11.158 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-06-11T19:22:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.159 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.161 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:11.162 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:11.164 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:11.167 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:11.167 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.167 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:11.167 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:11.168 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.168 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.168 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.168 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:11.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:11.168 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.171 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:11.174 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:11.174 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.174 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.174 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:11.175 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:11.175 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.175 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.175 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.175 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:11.175 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:11.175 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:11.176 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:11.178 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.178 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.178 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:11.178 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:11.178 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.180 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:11.182 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.182 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.182 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:11.182 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d83415640 10 librados: watch_flush enter 2024-06-11T19:22:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d83415640 10 librados: watch_flush exit 2024-06-11T19:22:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d83415640 20 client.114164.objecter shutdown clearing up homeless session... 2024-06-11T19:22:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d83415640 10 client.114164.objecter successfully canceled tick 2024-06-11T19:22:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d83415640 1 -- 172.21.15.98:0/1225666305 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d6003fdb0 msgr2=0x7f5d60042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d83415640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f5d6003fdb0 0x7f5d60042270 secure :-1 s=READY pgs=4253 cs=0 l=1 rev1=1 crypto rx=0x7f5d74036000 tx=0x7f5d74030000 comp rx=0 tx=0).stop 2024-06-11T19:22:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d83415640 10 monclient: shutdown 2024-06-11T19:22:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d83415640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d83415640 1 -- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d7c0a2530 msgr2=0x7f5d7c1cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d83415640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d7c0a2530 0x7f5d7c1cdaa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800e9e0 tx=0x7f5d7800eea0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.133+0000 7f5d819b2640 1 -- 172.21.15.98:0/1225666305 reap_dead start 2024-06-11T19:22:11.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.134+0000 7f5d83415640 1 -- 172.21.15.98:0/1225666305 shutdown_connections 2024-06-11T19:22:11.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.134+0000 7f5d83415640 1 --2- 172.21.15.98:0/1225666305 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d7c0a2530 0x7f5d7c1cdaa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.134+0000 7f5d83415640 1 -- 172.21.15.98:0/1225666305 >> 172.21.15.98:0/1225666305 conn(0x7f5d7c0ab850 msgr2=0x7f5d7c09c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:11.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.134+0000 7f5d83415640 1 -- 172.21.15.98:0/1225666305 shutdown_connections 2024-06-11T19:22:11.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.134+0000 7f5d83415640 1 -- 172.21.15.98:0/1225666305 wait complete. 2024-06-11T19:22:11.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.134+0000 7f5d83415640 1 librados: shutdown 2024-06-11T19:22:11.185 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:11.185 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:22:11.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:22:11.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.286+0000 7f7212699640 10 monclient: get_monmap_and_config 2024-06-11T19:22:11.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.286+0000 7f7212699640 10 monclient: build_initial_monmap 2024-06-11T19:22:11.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.286+0000 7f7212699640 10 monclient: monmap: 2024-06-11T19:22:11.286 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:11.286 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.286 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:11.287723+0000 2024-06-11T19:22:11.286 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:11.287723+0000 2024-06-11T19:22:11.286 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:11.286 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.286 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:11.286 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:11.287 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:11.287 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:11.287 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:11.287 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:11.287 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:11.287 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:11.287 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:11.287 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:11.287 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:11.287 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 1 Processor -- start 2024-06-11T19:22:11.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 1 -- start start 2024-06-11T19:22:11.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 10 monclient: init 2024-06-11T19:22:11.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 10 monclient: _add_conns ranks=[8,0,7,4,3,5,6,1,9,10,2] 2024-06-11T19:22:11.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f720c0a4920 0x7f720c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 10 monclient(hunting): picked mon.noname-e con 0x7f720c0a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f720c0a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f720c0a2530 0x7f720c0a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 10 monclient(hunting): picked mon.noname-i con 0x7f720c0a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f720c0a7140 con 0x7f720c0a5240 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f720c0a2b50 con 0x7f720c0a2530 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f720c0a2e30 con 0x7f720c0a4920 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.287+0000 7f7212699640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f7212699640 10 monclient(hunting): authenticate will time out at 1541.095011s 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720b7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f720c0a5240 0x7f720c0a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56856/0 (socket says 172.21.15.98:56856) 2024-06-11T19:22:11.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f720c0a4920 0x7f720c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720b7fe640 1 -- 172.21.15.98:0/1042963412 learned_addr learned my addr 172.21.15.98:0/1042963412 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 10 monclient(hunting): get_auth_request con 0x7f720c0a4920 auth_method 0 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f7210c36640 1 --2- 172.21.15.98:0/1042963412 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f720c0a2530 0x7f720c0a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 10 monclient(hunting): handle_auth_done global_id 114173 payload 274 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 1 -- 172.21.15.98:0/1042963412 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f720c0a2530 msgr2=0x7f720c0a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 1 --2- 172.21.15.98:0/1042963412 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f720c0a2530 0x7f720c0a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720affd640 1 -- 172.21.15.98:0/1042963412 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 648571269 0 0) 0x7f720c0a7140 con 0x7f720c0a5240 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 1 -- 172.21.15.98:0/1042963412 >> v1:172.21.15.98:6789/0 conn(0x7f720c0a5240 legacy=0x7f720c0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 1 monclient: found mon.noname-e 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720bfff640 1 -- 172.21.15.98:0/1042963412 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f720c0a3110 con 0x7f720c0a4920 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f7210c36640 10 monclient: get_auth_request con 0x7f720c0a2530 auth_method 0 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f7210c36640 1 --2- 172.21.15.98:0/1042963412 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f720c0a2530 0x7f720c0a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.288+0000 7f720affd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720bfff640 1 --2- 172.21.15.98:0/1042963412 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f720c0a4920 0x7f720c0a4d00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0097d0 tx=0x7f71fc02b770 comp rx=0 tx=0).ready entity=mon.8 client_cookie=9446102cb68a17a5 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 1 -- 172.21.15.98:0/1042963412 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71fc039070 con 0x7f720c0a4920 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 10 monclient: dump: 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 5 monclient: authenticate success, global_id 114173 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 1 -- 172.21.15.98:0/1042963412 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71fc030070 con 0x7f720c0a4920 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 4 set_mon_vals no callback set 2024-06-11T19:22:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 1 -- 172.21.15.98:0/1042963412 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71fc031db0 con 0x7f720c0a4920 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f720affd640 10 monclient: dump: 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 10 monclient: shutdown 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 1 -- 172.21.15.98:0/1042963412 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f720c0a4920 msgr2=0x7f720c0a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 1 --2- 172.21.15.98:0/1042963412 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f720c0a4920 0x7f720c0a4d00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0097d0 tx=0x7f71fc02b770 comp rx=0 tx=0).stop 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 1 -- 172.21.15.98:0/1042963412 shutdown_connections 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 1 --2- 172.21.15.98:0/1042963412 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f720c0a2530 0x7f720c0a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 1 --2- 172.21.15.98:0/1042963412 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f720c0a4920 0x7f720c0a4d00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 1 -- 172.21.15.98:0/1042963412 >> 172.21.15.98:0/1042963412 conn(0x7f720c0ab850 msgr2=0x7f720c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 1 -- 172.21.15.98:0/1042963412 shutdown_connections 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.289+0000 7f7212699640 1 -- 172.21.15.98:0/1042963412 wait complete. 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.290+0000 7f7212699640 10 monclient: build_initial_monmap 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.290+0000 7f7212699640 10 monclient: monmap: 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.290+0000 7f7212699640 1 librados: starting msgr at 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.290+0000 7f7212699640 1 librados: starting objecter 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.290+0000 7f7212699640 1 Processor -- start 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 1 -- start start 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 1 librados: setting wanted keys 2024-06-11T19:22:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 1 librados: calling monclient init 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 10 monclient: init 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 10 monclient: _add_conns ranks=[2,0,6,3,8,5,4,1,7] 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f720c0a2530 0x7f720c1215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 10 monclient(hunting): picked mon.noname-b con 0x7f720c0a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f720c0a4920 0x7f720c121b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 10 monclient(hunting): picked mon.noname-a con 0x7f720c0a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c0a5240 0x7f720c122070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 10 monclient(hunting): picked mon.noname-d con 0x7f720c0a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f720c0a6da0 con 0x7f720c0a4920 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f720c138e50 con 0x7f720c0a2530 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f720c0a65f0 con 0x7f720c0a5240 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7212699640 10 monclient(hunting): authenticate will time out at 1541.098545s 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f720c0a2530 0x7f720c1215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f720c0a4920 0x7f720c121b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f720c0a4920 0x7f720c121b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52288/0 (socket says 172.21.15.98:52288) 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 1 -- 172.21.15.98:0/823381038 learned_addr learned my addr 172.21.15.98:0/823381038 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720bfff640 10 monclient(hunting): get_auth_request con 0x7f720c0a2530 auth_method 0 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 10 monclient(hunting): get_auth_request con 0x7f720c0a4920 auth_method 0 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7210c36640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c0a5240 0x7f720c122070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7210c36640 10 monclient(hunting): get_auth_request con 0x7f720c0a5240 auth_method 0 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7210c36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7210c36640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7210c36640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720bfff640 1 -- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f720c0a2530 msgr2=0x7f720c1215f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720bfff640 1 -- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f720c0a2530 msgr2=0x7f720c1215f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720bfff640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f720c0a2530 0x7f720c1215f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720bfff640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f720c0a2530 0x7f720c1215f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7210c36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7210c36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7210c36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 10 monclient(hunting): handle_auth_done global_id 65754 payload 824 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 1 -- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c0a5240 msgr2=0x7f720c122070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c0a5240 0x7f720c122070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 1 -- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f720c0a2530 msgr2=0x7f720c1215f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f720c0a2530 0x7f720c1215f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 1 monclient: found mon.noname-a 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f720b7fe640 1 -- 172.21.15.98:0/823381038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f720c124a60 con 0x7f720c0a4920 2024-06-11T19:22:11.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.291+0000 7f7210c36640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c0a5240 0x7f720c122070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f720b7fe640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f720c0a4920 0x7f720c121b30 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f720000c340 tx=0x7f720000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 10 client.?.objecter ms_handle_connect 0x7f720c0a4920 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72000047e0 con 0x7f720c0a4920 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 10 monclient: dump: 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.311 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72000049a0 con 0x7f720c0a4920 2024-06-11T19:22:11.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7212699640 5 monclient: authenticate success, global_id 65754 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7212699640 10 monclient: _renew_subs 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7212699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7212699640 1 -- 172.21.15.98:0/823381038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f720c124d40 con 0x7f720c0a4920 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72000133c0 con 0x7f720c0a4920 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7208ff9640 10 monclient: dump: 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.313 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7212699640 10 client.65754.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7212699640 10 monclient: _renew_subs 2024-06-11T19:22:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f720a7fc640 4 set_mon_vals no callback set 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7212699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7212699640 1 -- 172.21.15.98:0/823381038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f720c124780 con 0x7f720c0a4920 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.292+0000 7f7212699640 1 librados: init done 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7212699640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7212699640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1cda80 0x7f720c1cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7212699640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7212699640 1 -- 172.21.15.98:0/823381038 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f720c123540 con 0x7f720c1cda80 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f7200011070 con 0x7f720c0a4920 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f720bfff640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1cda80 0x7f720c1cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f720bfff640 10 monclient: get_auth_request con 0x7f720c1cda80 auth_method 0 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f720bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f720bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f720bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f720bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f720bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f720bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f720bfff640 10 monclient(hunting): handle_auth_done global_id 55608 payload 824 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f720bfff640 1 -- 172.21.15.98:0/823381038 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f720c0a4d00 con 0x7f720c1cda80 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f71f803fdb0 0x7f71f8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(516..516 src has 1..516) ==== 6685+0+0 (secure 0 0 0) 0x7f7200051670 con 0x7f720c0a4920 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 10 client.65754.objecter ms_dispatch 0x7f720c000b90 osd_map(516..516 src has 1..516) 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 3 client.65754.objecter handle_osd_map got epochs [516,516] > 0 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 3 client.65754.objecter handle_osd_map decoding full epoch 516 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f720bfff640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1cda80 0x7f720c1cde60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f720c0a8420 tx=0x7f71fc0046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 20 client.65754.objecter dump_active .. 0 homeless 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 10 client.65754.objecter ms_handle_connect 0x7f720c1cda80 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 10 client.65754.objecter resend_mon_ops 2024-06-11T19:22:11.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7210c36640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f71f803fdb0 0x7f71f8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7210c36640 10 monclient: get_auth_request con 0x7f71f803fdb0 auth_method 0 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71fc044030 con 0x7f720c1cda80 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.293+0000 7f7208ff9640 10 monclient: dump: 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.294+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f71fc003c90 con 0x7f720c1cda80 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.294+0000 7f7208ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.294+0000 7f7208ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.294+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1cda80 msgr2=0x7f720c1cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.294+0000 7f7208ff9640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1cda80 0x7f720c1cde60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f720c0a8420 tx=0x7f71fc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.294+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1cda80 msgr2=0x7f720c1cde60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.294+0000 7f7208ff9640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1cda80 0x7f720c1cde60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.294+0000 7f7208ff9640 10 client.65754.objecter ms_handle_connect 0x7f71f803fdb0 2024-06-11T19:22:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.294+0000 7f7210c36640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f71f803fdb0 0x7f71f8042270 secure :-1 s=READY pgs=4254 cs=0 l=1 rev1=1 crypto rx=0x7f720400d8a0 tx=0x7f7204007780 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f7212699640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f7212699640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1ce3a0 0x7f720c1bed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f7212699640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f7212699640 1 -- 172.21.15.98:0/823381038 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f720c122cb0 con 0x7f720c1ce3a0 2024-06-11T19:22:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f720bfff640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1ce3a0 0x7f720c1bed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f720bfff640 10 monclient: get_auth_request con 0x7f720c1ce3a0 auth_method 0 2024-06-11T19:22:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f720bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f720bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f720bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f720bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f720bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f720bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f720bfff640 10 monclient(hunting): handle_auth_done global_id 55617 payload 824 2024-06-11T19:22:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.300+0000 7f720bfff640 1 -- 172.21.15.98:0/823381038 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f720c124110 con 0x7f720c1ce3a0 2024-06-11T19:22:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 10 client.65754.objecter ms_handle_connect 0x7f720c1ce3a0 2024-06-11T19:22:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 10 client.65754.objecter resend_mon_ops 2024-06-11T19:22:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f720bfff640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1ce3a0 0x7f720c1bed40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f71fc02b7b0 tx=0x7f71fc031000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71fc044040 con 0x7f720c1ce3a0 2024-06-11T19:22:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:22:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 10 monclient: dump: 2024-06-11T19:22:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f71fc031840 con 0x7f720c1ce3a0 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1ce3a0 msgr2=0x7f720c1bed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1ce3a0 0x7f720c1bed40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f71fc02b7b0 tx=0x7f71fc031000 comp rx=0 tx=0).stop 2024-06-11T19:22:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 1 -- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1ce3a0 msgr2=0x7f720c1bed40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7208ff9640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f720c1ce3a0 0x7f720c1bed40 secure :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f71fc02b7b0 tx=0x7f71fc031000 comp rx=0 tx=0).stop 2024-06-11T19:22:11.324 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:11.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:11.324 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:11.324 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:11.324 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:11.324 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 644057, 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-06-11T19:22:11.325 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.326 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.327 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:11.328 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.329 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:11.331 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.333 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:11.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:11.334 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:11.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:11.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:22:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:11.347 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7212699640 10 librados: watch_flush enter 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7212699640 10 librados: watch_flush exit 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7212699640 20 client.65754.objecter shutdown clearing up homeless session... 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7212699640 10 client.65754.objecter successfully canceled tick 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7212699640 1 -- 172.21.15.98:0/823381038 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f71f803fdb0 msgr2=0x7f71f8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7212699640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f71f803fdb0 0x7f71f8042270 secure :-1 s=READY pgs=4254 cs=0 l=1 rev1=1 crypto rx=0x7f720400d8a0 tx=0x7f7204007780 comp rx=0 tx=0).stop 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7212699640 10 monclient: shutdown 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7212699640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7212699640 1 -- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f720c0a4920 msgr2=0x7f720c121b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7212699640 1 --2- 172.21.15.98:0/823381038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f720c0a4920 0x7f720c121b30 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f720000c340 tx=0x7f720000c800 comp rx=0 tx=0).stop 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7210c36640 1 -- 172.21.15.98:0/823381038 reap_dead start 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7210c36640 1 -- 172.21.15.98:0/823381038 reap_dead start 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7212699640 1 -- 172.21.15.98:0/823381038 shutdown_connections 2024-06-11T19:22:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.301+0000 7f7212699640 1 -- 172.21.15.98:0/823381038 >> 172.21.15.98:0/823381038 conn(0x7f720c0ab850 msgr2=0x7f720c137f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:11.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.302+0000 7f7212699640 1 -- 172.21.15.98:0/823381038 shutdown_connections 2024-06-11T19:22:11.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.302+0000 7f7212699640 1 -- 172.21.15.98:0/823381038 wait complete. 2024-06-11T19:22:11.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.302+0000 7f7212699640 1 librados: shutdown 2024-06-11T19:22:11.350 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:11.350 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:22:11.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:22:11.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.449+0000 7f2073aa4640 10 monclient: get_monmap_and_config 2024-06-11T19:22:11.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.449+0000 7f2073aa4640 10 monclient: build_initial_monmap 2024-06-11T19:22:11.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.449+0000 7f2073aa4640 10 monclient: monmap: 2024-06-11T19:22:11.448 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:11.448 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:11.450468+0000 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:11.450468+0000 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:11.449 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:11.450 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:11.450 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:11.450 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.449+0000 7f2073aa4640 1 Processor -- start 2024-06-11T19:22:11.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 1 -- start start 2024-06-11T19:22:11.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 10 monclient: init 2024-06-11T19:22:11.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 10 monclient: _add_conns ranks=[5,1,0,4,2,8,9,10,3,6,7] 2024-06-11T19:22:11.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f206c0a4920 0x7f206c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 10 monclient(hunting): picked mon.noname-h con 0x7f206c0a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f206c0a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f206c0a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f206c0a7140 con 0x7f206c0a2530 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f206c0a67c0 con 0x7f206c0a5240 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f206c0a2b50 con 0x7f206c0a4920 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2073aa4640 10 monclient(hunting): authenticate will time out at 1541.257730s 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2072041640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f206c0a2530 0x7f206c0a60b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56872/0 (socket says 172.21.15.98:56872) 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.450+0000 7f2072041640 1 -- 172.21.15.98:0/812024617 learned_addr learned my addr 172.21.15.98:0/812024617 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f206c0a4920 0x7f206c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 10 monclient(hunting): get_auth_request con 0x7f206c0a4920 auth_method 0 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3964055870 0 0) 0x7f206c0a7140 con 0x7f206c0a2530 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient(hunting): _negotiate 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient(hunting): my global_id is 65763 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2054003520 con 0x7f206c0a2530 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 1 -- 172.21.15.98:0/812024617 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f206c0a4920 msgr2=0x7f206c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 1 -- 172.21.15.98:0/812024617 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f206c0a4920 msgr2=0x7f206c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f206c0a4920 0x7f206c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f206c0a4920 0x7f206c0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3461166190 0 0) 0x7f2054003520 con 0x7f206c0a2530 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f206c0a4920 msgr2=0x7f206c0a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f206c0a4920 0x7f206c0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 >> v1:172.21.15.117:6789/0 conn(0x7f206c0a5240 legacy=0x7f206c0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206c0a2e30 con 0x7f206c0a2530 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2073aa4640 5 monclient: authenticate success, global_id 65763 2024-06-11T19:22:11.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7f205c001960 con 0x7f206c0a2530 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2073aa4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient: dump: 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.466 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 >> v1:172.21.15.98:6789/0 conn(0x7f206c0a2530 legacy=0x7f206c0a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient: _add_conns ranks=[6,4,1,7,8,2,0,5,3] 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2054004230 0x7f2054004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient(hunting): picked mon.d con 0x7f2054004230 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2054004e60 0x7f20540099b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient(hunting): picked mon.c con 0x7f2054004e60 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2054009ef0 0x7f205400c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient(hunting): picked mon.f con 0x7f2054009ef0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:22:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f206c0a2b50 con 0x7f2054009ef0 2024-06-11T19:22:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f205400ca30 con 0x7f2054004e60 2024-06-11T19:22:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f205400cbe0 con 0x7f2054004230 2024-06-11T19:22:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207083e640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:22:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2073aa4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2054004230 0x7f2054004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2054004e60 0x7f20540099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 10 monclient(hunting): get_auth_request con 0x7f2054004230 auth_method 0 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 10 monclient(hunting): get_auth_request con 0x7f2054004e60 auth_method 0 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2072041640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2054009ef0 0x7f205400c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2072041640 10 monclient(hunting): get_auth_request con 0x7f2054009ef0 auth_method 0 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2072041640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2072041640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2072041640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2072041640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2072041640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2072041640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 10 monclient(hunting): handle_auth_done global_id 65763 payload 293 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 1 -- 172.21.15.98:0/812024617 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2054004230 msgr2=0x7f2054004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2054004230 0x7f2054004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 1 -- 172.21.15.98:0/812024617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2054009ef0 msgr2=0x7f205400c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2054009ef0 0x7f205400c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 1 monclient: found mon.c 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2072041640 1 -- 172.21.15.98:0/812024617 reap_dead start 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 1 -- 172.21.15.98:0/812024617 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2054003520 con 0x7f2054004e60 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f207103f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.451+0000 7f2071840640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2054004230 0x7f2054004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f207103f640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2054004e60 0x7f20540099b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f206c073fc0 tx=0x7f2054005830 comp rx=0 tx=0).ready entity=mon.4 client_cookie=c689086e82f05794 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2060001b30 con 0x7f2054004e60 2024-06-11T19:22:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f207083e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f207083e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f207083e640 10 monclient: dump: 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f206002d890 con 0x7f2054004e60 2024-06-11T19:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f2073aa4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f207083e640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f207083e640 1 -- 172.21.15.98:0/812024617 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2060037400 con 0x7f2054004e60 2024-06-11T19:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f2073aa4640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f2073aa4640 4 set_mon_vals no callback set 2024-06-11T19:22:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f207083e640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f207083e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.452+0000 7f207083e640 10 monclient: dump: 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.475 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.453+0000 7f2073aa4640 10 monclient: shutdown 2024-06-11T19:22:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.453+0000 7f2073aa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.453+0000 7f2073aa4640 1 -- 172.21.15.98:0/812024617 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2054004e60 msgr2=0x7f20540099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.453+0000 7f2073aa4640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2054004e60 0x7f20540099b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f206c073fc0 tx=0x7f2054005830 comp rx=0 tx=0).stop 2024-06-11T19:22:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.453+0000 7f2073aa4640 1 -- 172.21.15.98:0/812024617 shutdown_connections 2024-06-11T19:22:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.453+0000 7f2073aa4640 1 --2- 172.21.15.98:0/812024617 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2054004e60 0x7f20540099b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.453+0000 7f2073aa4640 1 -- 172.21.15.98:0/812024617 >> 172.21.15.98:0/812024617 conn(0x7f206c0ab850 msgr2=0x7f206c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.453+0000 7f2073aa4640 1 -- 172.21.15.98:0/812024617 shutdown_connections 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.453+0000 7f2073aa4640 1 -- 172.21.15.98:0/812024617 wait complete. 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.453+0000 7f2073aa4640 10 monclient: build_initial_monmap 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 10 monclient: monmap: 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 1 librados: starting msgr at 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 1 librados: starting objecter 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 1 Processor -- start 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 1 -- start start 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 1 librados: setting wanted keys 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 1 librados: calling monclient init 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 10 monclient: init 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 10 monclient: _add_conns ranks=[8,0,4,6,3,2,7,1,5] 2024-06-11T19:22:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2054004e60 0x7f206c137bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 10 monclient(hunting): picked mon.noname-e con 0x7f2054004e60 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f206c1cd270 0x7f206c1380f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 10 monclient(hunting): picked mon.noname-a con 0x7f206c1cd270 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f206c121800 0x7f206c121c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 10 monclient(hunting): picked mon.noname-c con 0x7f206c121800 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f206c138e50 con 0x7f206c1cd270 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f206c0a84b0 con 0x7f206c121800 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f206c0a6da0 con 0x7f2054004e60 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2073aa4640 10 monclient(hunting): authenticate will time out at 1541.261812s 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f207103f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f206c1cd270 0x7f206c1380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2071840640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2054004e60 0x7f206c137bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2072041640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f206c121800 0x7f206c121c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f207103f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f206c1cd270 0x7f206c1380f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52298/0 (socket says 172.21.15.98:52298) 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f207103f640 1 -- 172.21.15.98:0/1723420566 learned_addr learned my addr 172.21.15.98:0/1723420566 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2071840640 10 monclient(hunting): get_auth_request con 0x7f2054004e60 auth_method 0 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2071840640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2071840640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2071840640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f207103f640 10 monclient(hunting): get_auth_request con 0x7f206c1cd270 auth_method 0 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f207103f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f207103f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f207103f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2072041640 10 monclient(hunting): get_auth_request con 0x7f206c121800 auth_method 0 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2072041640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2072041640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2072041640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.454+0000 7f2071840640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f207103f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f207103f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f207103f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2072041640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2072041640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2072041640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 10 monclient(hunting): handle_auth_done global_id 114182 payload 824 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 1 -- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f206c121800 msgr2=0x7f206c121c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f206c121800 0x7f206c121c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 1 -- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f206c1cd270 msgr2=0x7f206c1380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f206c1cd270 0x7f206c1380f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 1 monclient: found mon.noname-e 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 1 -- 172.21.15.98:0/1723420566 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206c123280 con 0x7f2054004e60 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f207103f640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f206c1cd270 0x7f206c1380f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2071840640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2054004e60 0x7f206c137bb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f206c0698d0 tx=0x7f2064010da0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 10 client.?.objecter ms_handle_connect 0x7f2054004e60 2024-06-11T19:22:11.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f206401a020 con 0x7f2054004e60 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 10 monclient: dump: 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.482 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2064016480 con 0x7f2054004e60 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2073aa4640 5 monclient: authenticate success, global_id 114182 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2073aa4640 10 monclient: _renew_subs 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2073aa4640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2073aa4640 1 -- 172.21.15.98:0/1723420566 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f206c125570 con 0x7f2054004e60 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f206400b5e0 con 0x7f2054004e60 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206a7fc640 10 monclient: dump: 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.484 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2073aa4640 10 client.114182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2073aa4640 10 monclient: _renew_subs 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f206bfff640 4 set_mon_vals no callback set 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2073aa4640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2073aa4640 1 -- 172.21.15.98:0/1723420566 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f206c125290 con 0x7f2054004e60 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.455+0000 7f2073aa4640 1 librados: init done 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f2073aa4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f2073aa4640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c1221a0 0x7f206c09cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f2073aa4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f2073aa4640 1 -- 172.21.15.98:0/1723420566 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f206c050830 con 0x7f206c1221a0 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f206401d020 con 0x7f2054004e60 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f207103f640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c1221a0 0x7f206c09cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f207103f640 10 monclient: get_auth_request con 0x7f206c1221a0 auth_method 0 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f207103f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f207103f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f207103f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f207103f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f207103f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f207103f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f206a7fc640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f205803fde0 0x7f20580422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f207103f640 10 monclient(hunting): handle_auth_done global_id 114191 payload 824 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f207103f640 1 -- 172.21.15.98:0/1723420566 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f206c0a4d00 con 0x7f206c1221a0 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(516..516 src has 1..516) ==== 6685+0+0 (secure 0 0 0) 0x7f206400f040 con 0x7f2054004e60 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f206a7fc640 10 client.114182.objecter ms_dispatch 0x7f206c000b90 osd_map(516..516 src has 1..516) 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f206a7fc640 3 client.114182.objecter handle_osd_map got epochs [516,516] > 0 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.456+0000 7f206a7fc640 3 client.114182.objecter handle_osd_map decoding full epoch 516 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 20 client.114182.objecter dump_active .. 0 homeless 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f207103f640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c1221a0 0x7f206c09cf60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2060001fd0 tx=0x7f2060001490 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 10 client.114182.objecter ms_handle_connect 0x7f206c1221a0 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 10 client.114182.objecter resend_mon_ops 2024-06-11T19:22:11.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f2072041640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f205803fde0 0x7f20580422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f2072041640 10 monclient: get_auth_request con 0x7f205803fde0 auth_method 0 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2060047020 con 0x7f206c1221a0 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 10 monclient: dump: 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.488 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2060043410 con 0x7f206c1221a0 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c1221a0 msgr2=0x7f206c09cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c1221a0 0x7f206c09cf60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2060001fd0 tx=0x7f2060001490 comp rx=0 tx=0).stop 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c1221a0 msgr2=0x7f206c09cf60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:11.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c1221a0 0x7f206c09cf60 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2060001fd0 tx=0x7f2060001490 comp rx=0 tx=0).stop 2024-06-11T19:22:11.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f206a7fc640 10 client.114182.objecter ms_handle_connect 0x7f205803fde0 2024-06-11T19:22:11.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.457+0000 7f2072041640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f205803fde0 0x7f20580422a0 secure :-1 s=READY pgs=4255 cs=0 l=1 rev1=1 crypto rx=0x7f20540039b0 tx=0x7f205c001740 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f2073aa4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:11.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f2073aa4640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c09d4a0 0x7f206c09d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f2073aa4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f2073aa4640 1 -- 172.21.15.98:0/1723420566 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f206c138630 con 0x7f206c09d4a0 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f207103f640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c09d4a0 0x7f206c09d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f207103f640 10 monclient: get_auth_request con 0x7f206c09d4a0 auth_method 0 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f207103f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f207103f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f207103f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f207103f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f207103f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f207103f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f207103f640 10 monclient(hunting): handle_auth_done global_id 114200 payload 824 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f207103f640 1 -- 172.21.15.98:0/1723420566 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f206c0aa960 con 0x7f206c09d4a0 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f207103f640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c09d4a0 0x7f206c09d8b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2060045000 tx=0x7f20600016f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 10 client.114182.objecter ms_handle_connect 0x7f206c09d4a0 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 10 client.114182.objecter resend_mon_ops 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2060047030 con 0x7f206c09d4a0 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 10 monclient: dump: 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.493 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f206004b710 con 0x7f206c09d4a0 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c09d4a0 msgr2=0x7f206c09d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c09d4a0 0x7f206c09d8b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2060045000 tx=0x7f20600016f0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 1 -- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c09d4a0 msgr2=0x7f206c09d8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:11.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.464+0000 7f206a7fc640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f206c09d4a0 0x7f206c09d8b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.495 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:11.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:11.495 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:11.495 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:11.495 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:11.495 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 19833, 2024-06-11T19:22:11.495 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:11.496 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.497 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.498 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:11.499 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.500 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:11.502 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:11.503 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.505 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.507 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:11.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.513 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:11.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 10 librados: watch_flush enter 2024-06-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 10 librados: watch_flush exit 2024-06-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 20 client.114182.objecter shutdown clearing up homeless session... 2024-06-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 10 client.114182.objecter successfully canceled tick 2024-06-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 1 -- 172.21.15.98:0/1723420566 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f205803fde0 msgr2=0x7f20580422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f205803fde0 0x7f20580422a0 secure :-1 s=READY pgs=4255 cs=0 l=1 rev1=1 crypto rx=0x7f20540039b0 tx=0x7f205c001740 comp rx=0 tx=0).stop 2024-06-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 10 monclient: shutdown 2024-06-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 1 -- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2054004e60 msgr2=0x7f206c137bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 1 --2- 172.21.15.98:0/1723420566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2054004e60 0x7f206c137bb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f206c0698d0 tx=0x7f2064010da0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2072041640 1 -- 172.21.15.98:0/1723420566 reap_dead start 2024-06-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2072041640 1 -- 172.21.15.98:0/1723420566 reap_dead start 2024-06-11T19:22:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 1 -- 172.21.15.98:0/1723420566 shutdown_connections 2024-06-11T19:22:11.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 1 -- 172.21.15.98:0/1723420566 >> 172.21.15.98:0/1723420566 conn(0x7f206c0ab850 msgr2=0x7f206c0726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:11.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 1 -- 172.21.15.98:0/1723420566 shutdown_connections 2024-06-11T19:22:11.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 1 -- 172.21.15.98:0/1723420566 wait complete. 2024-06-11T19:22:11.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.465+0000 7f2073aa4640 1 librados: shutdown 2024-06-11T19:22:11.520 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:11.520 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:22:11.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:22:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.620+0000 7f905f174640 10 monclient: get_monmap_and_config 2024-06-11T19:22:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.620+0000 7f905f174640 10 monclient: build_initial_monmap 2024-06-11T19:22:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.620+0000 7f905f174640 10 monclient: monmap: 2024-06-11T19:22:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:11.621491+0000 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:11.621491+0000 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:11.620 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:11.621 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 1 Processor -- start 2024-06-11T19:22:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 1 -- start start 2024-06-11T19:22:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 10 monclient: init 2024-06-11T19:22:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 10 monclient: _add_conns ranks=[8,7,1,3,9,2,0,10,6,5,4] 2024-06-11T19:22:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f90580a4920 0x7f90580a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 10 monclient(hunting): picked mon.noname-e con 0x7f90580a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f90580a5240 0x7f90580a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 10 monclient(hunting): picked mon.noname-i con 0x7f90580a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f90580a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:22:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90580a7140 con 0x7f90580a2530 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f90580a2b50 con 0x7f90580a5240 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f90580a2e30 con 0x7f90580a4920 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905f174640 10 monclient(hunting): authenticate will time out at 1541.428778s 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905cf10640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f90580a4920 0x7f90580a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905cf10640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f90580a4920 0x7f90580a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:58152/0 (socket says 172.21.15.98:58152) 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905cf10640 1 -- 172.21.15.98:0/2576710076 learned_addr learned my addr 172.21.15.98:0/2576710076 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905cf10640 10 monclient(hunting): get_auth_request con 0x7f90580a4920 auth_method 0 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905cf10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905cf10640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.621+0000 7f905cf10640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f9057fff640 1 --2- 172.21.15.98:0/2576710076 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f90580a5240 0x7f90580a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 10 monclient(hunting): handle_auth_done global_id 114209 payload 274 2024-06-11T19:22:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 1 -- 172.21.15.98:0/2576710076 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f90580a5240 msgr2=0x7f90580a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 1 --2- 172.21.15.98:0/2576710076 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f90580a5240 0x7f90580a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 1 -- 172.21.15.98:0/2576710076 >> v1:172.21.15.117:6789/0 conn(0x7f90580a2530 legacy=0x7f90580a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 1 monclient: found mon.noname-e 2024-06-11T19:22:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 1 -- 172.21.15.98:0/2576710076 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90580a3110 con 0x7f90580a4920 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f905cf10640 1 --2- 172.21.15.98:0/2576710076 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f90580a4920 0x7f90580a4d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f904c0098a0 tx=0x7f904c02b7a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=719534592eec697a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.622+0000 7f90577fe640 1 -- 172.21.15.98:0/2576710076 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f904c038070 con 0x7f90580a4920 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f90577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f90577fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f90577fe640 10 monclient: dump: 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f90577fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f90577fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f90577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f90577fe640 1 -- 172.21.15.98:0/2576710076 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f904c031970 con 0x7f90580a4920 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 5 monclient: authenticate success, global_id 114209 2024-06-11T19:22:11.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f90577fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 4 set_mon_vals no callback set 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f90577fe640 1 -- 172.21.15.98:0/2576710076 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f904c031ba0 con 0x7f90580a4920 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f90577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f90577fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f90577fe640 10 monclient: dump: 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.638 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 10 monclient: shutdown 2024-06-11T19:22:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 1 -- 172.21.15.98:0/2576710076 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f90580a4920 msgr2=0x7f90580a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 1 --2- 172.21.15.98:0/2576710076 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f90580a4920 0x7f90580a4d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f904c0098a0 tx=0x7f904c02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 1 -- 172.21.15.98:0/2576710076 shutdown_connections 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 1 --2- 172.21.15.98:0/2576710076 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f90580a5240 0x7f90580a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 1 --2- 172.21.15.98:0/2576710076 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f90580a4920 0x7f90580a4d00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 1 -- 172.21.15.98:0/2576710076 >> 172.21.15.98:0/2576710076 conn(0x7f90580ab850 msgr2=0x7f90580a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 1 -- 172.21.15.98:0/2576710076 shutdown_connections 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.623+0000 7f905f174640 1 -- 172.21.15.98:0/2576710076 wait complete. 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 10 monclient: build_initial_monmap 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 10 monclient: monmap: 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:11.641 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 1 librados: starting msgr at 2024-06-11T19:22:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 1 librados: starting objecter 2024-06-11T19:22:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 1 Processor -- start 2024-06-11T19:22:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 1 -- start start 2024-06-11T19:22:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 1 librados: setting wanted keys 2024-06-11T19:22:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 1 librados: calling monclient init 2024-06-11T19:22:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 10 monclient: init 2024-06-11T19:22:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 10 monclient: _add_conns ranks=[1,0,5,6,2,8,3,4,7] 2024-06-11T19:22:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90580a2530 0x7f90581cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.624+0000 7f905f174640 10 monclient(hunting): picked mon.noname-f con 0x7f90580a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:22:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90580a4920 0x7f90581ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 10 monclient(hunting): picked mon.noname-a con 0x7f90580a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f90580a5240 0x7f9058121fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 10 monclient(hunting): picked mon.noname-h con 0x7f90580a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90580a6da0 con 0x7f90580a4920 2024-06-11T19:22:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9058138e50 con 0x7f90580a2530 2024-06-11T19:22:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f90580a65f0 con 0x7f90580a5240 2024-06-11T19:22:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 10 monclient(hunting): authenticate will time out at 1541.432055s 2024-06-11T19:22:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90580a4920 0x7f90581ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90580a4920 0x7f90581ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52314/0 (socket says 172.21.15.98:52314) 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 1 -- 172.21.15.98:0/3485613639 learned_addr learned my addr 172.21.15.98:0/3485613639 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 10 monclient(hunting): get_auth_request con 0x7f90580a4920 auth_method 0 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905cf10640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90580a2530 0x7f90581cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905cf10640 10 monclient(hunting): get_auth_request con 0x7f90580a2530 auth_method 0 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905cf10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905cf10640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905cf10640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905d711640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f90580a5240 0x7f9058121fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905d711640 10 monclient(hunting): get_auth_request con 0x7f90580a5240 auth_method 0 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905d711640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905d711640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905d711640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905cf10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905cf10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905cf10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905d711640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905d711640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905d711640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 10 monclient(hunting): handle_auth_done global_id 65781 payload 824 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 1 -- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f90580a5240 msgr2=0x7f9058121fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f90580a5240 0x7f9058121fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 1 -- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90580a2530 msgr2=0x7f90581cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90580a2530 0x7f90581cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 1 monclient: found mon.noname-a 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 1 -- 172.21.15.98:0/3485613639 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9058137720 con 0x7f90580a4920 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905cf10640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90580a2530 0x7f90581cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9057fff640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90580a4920 0x7f90581ce130 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f904400c5a0 tx=0x7f904400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 10 client.?.objecter ms_handle_connect 0x7f90580a4920 2024-06-11T19:22:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9044015070 con 0x7f90580a4920 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 10 monclient: dump: 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.647 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9044011a70 con 0x7f90580a4920 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 5 monclient: authenticate success, global_id 65781 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 10 monclient: _renew_subs 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 1 -- 172.21.15.98:0/3485613639 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9058125160 con 0x7f90580a4920 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:11.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9044011c60 con 0x7f90580a4920 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f90557fa640 10 monclient: dump: 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.649 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 10 client.65781.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f9056ffd640 4 set_mon_vals no callback set 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 10 monclient: _renew_subs 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 1 -- 172.21.15.98:0/3485613639 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90581ce670 con 0x7f90580a4920 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.625+0000 7f905f174640 1 librados: init done 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.626+0000 7f905f174640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.626+0000 7f905f174640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90581224f0 0x7f90581228d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.626+0000 7f905f174640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.626+0000 7f905f174640 1 -- 172.21.15.98:0/3485613639 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9058050830 con 0x7f90581224f0 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.626+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f9044011600 con 0x7f90580a4920 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.626+0000 7f905cf10640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90581224f0 0x7f90581228d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.626+0000 7f905cf10640 10 monclient: get_auth_request con 0x7f90581224f0 auth_method 0 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.626+0000 7f905cf10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.626+0000 7f905cf10640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.626+0000 7f905cf10640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f905cf10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f905cf10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f905cf10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f903c03fde0 0x7f903c0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(516..516 src has 1..516) ==== 6685+0+0 (secure 0 0 0) 0x7f9044051830 con 0x7f90580a4920 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 10 client.65781.objecter ms_dispatch 0x7f9058000b90 osd_map(516..516 src has 1..516) 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 3 client.65781.objecter handle_osd_map got epochs [516,516] > 0 2024-06-11T19:22:11.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 3 client.65781.objecter handle_osd_map decoding full epoch 516 2024-06-11T19:22:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 20 client.65781.objecter dump_active .. 0 homeless 2024-06-11T19:22:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f905cf10640 10 monclient(hunting): handle_auth_done global_id 32752 payload 824 2024-06-11T19:22:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f905cf10640 1 -- 172.21.15.98:0/3485613639 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90581239d0 con 0x7f90581224f0 2024-06-11T19:22:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f905d711640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f903c03fde0 0x7f903c0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f905d711640 10 monclient: get_auth_request con 0x7f903c03fde0 auth_method 0 2024-06-11T19:22:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 10 client.65781.objecter ms_handle_connect 0x7f90581224f0 2024-06-11T19:22:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 10 client.65781.objecter resend_mon_ops 2024-06-11T19:22:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f905cf10640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90581224f0 0x7f90581228d0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7f90580abf80 tx=0x7f904c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f904c038040 con 0x7f90581224f0 2024-06-11T19:22:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:22:11.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 10 monclient: dump: 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.653 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.654 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 10 client.65781.objecter ms_handle_connect 0x7f903c03fde0 2024-06-11T19:22:11.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f905d711640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f903c03fde0 0x7f903c0422a0 secure :-1 s=READY pgs=4256 cs=0 l=1 rev1=1 crypto rx=0x7f90580a2950 tx=0x7f9050003d10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f904c003610 con 0x7f90581224f0 2024-06-11T19:22:11.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:11.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:11.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90581224f0 msgr2=0x7f90581228d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90581224f0 0x7f90581228d0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7f90580abf80 tx=0x7f904c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90581224f0 msgr2=0x7f90581228d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:11.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.627+0000 7f90557fa640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90581224f0 0x7f90581228d0 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905f174640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905f174640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9058122e10 0x7f905809e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905f174640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905f174640 1 -- 172.21.15.98:0/3485613639 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f90580a8f30 con 0x7f9058122e10 2024-06-11T19:22:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905cf10640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9058122e10 0x7f905809e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905cf10640 10 monclient: get_auth_request con 0x7f9058122e10 auth_method 0 2024-06-11T19:22:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905cf10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905cf10640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905cf10640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905cf10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905cf10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905cf10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905cf10640 10 monclient(hunting): handle_auth_done global_id 32761 payload 824 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.634+0000 7f905cf10640 1 -- 172.21.15.98:0/3485613639 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f90580a3480 con 0x7f9058122e10 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 10 client.65781.objecter ms_handle_connect 0x7f9058122e10 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f905cf10640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9058122e10 0x7f905809e9f0 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f904c0097c0 tx=0x7f904c031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 10 client.65781.objecter resend_mon_ops 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f904c038040 con 0x7f9058122e10 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 10 monclient: dump: 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.658 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7f904c04e070 con 0x7f9058122e10 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:11.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:11.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9058122e10 msgr2=0x7f905809e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9058122e10 0x7f905809e9f0 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f904c0097c0 tx=0x7f904c031000 comp rx=0 tx=0).stop 2024-06-11T19:22:11.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 1 -- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9058122e10 msgr2=0x7f905809e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:11.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f90557fa640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9058122e10 0x7f905809e9f0 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.660 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:11.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:11.660 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:11.660 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:22:11.660 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:11.660 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 781854, 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:22:11.661 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.662 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.663 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:11.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.665 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:11.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.670 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:11.672 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:11.675 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.677 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.678 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.680 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.681 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:11.682 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:11.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f905f174640 10 librados: watch_flush enter 2024-06-11T19:22:11.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f905f174640 10 librados: watch_flush exit 2024-06-11T19:22:11.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f905f174640 20 client.65781.objecter shutdown clearing up homeless session... 2024-06-11T19:22:11.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.635+0000 7f905f174640 10 client.65781.objecter successfully canceled tick 2024-06-11T19:22:11.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905f174640 1 -- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f903c03fde0 msgr2=0x7f903c0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905f174640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f903c03fde0 0x7f903c0422a0 secure :-1 s=READY pgs=4256 cs=0 l=1 rev1=1 crypto rx=0x7f90580a2950 tx=0x7f9050003d10 comp rx=0 tx=0).stop 2024-06-11T19:22:11.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905f174640 10 monclient: shutdown 2024-06-11T19:22:11.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905f174640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:11.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905f174640 1 -- 172.21.15.98:0/3485613639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90580a4920 msgr2=0x7f90581ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905f174640 1 --2- 172.21.15.98:0/3485613639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90580a4920 0x7f90581ce130 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f904400c5a0 tx=0x7f904400ca60 comp rx=0 tx=0).stop 2024-06-11T19:22:11.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905d711640 1 -- 172.21.15.98:0/3485613639 reap_dead start 2024-06-11T19:22:11.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905d711640 1 -- 172.21.15.98:0/3485613639 reap_dead start 2024-06-11T19:22:11.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905f174640 1 -- 172.21.15.98:0/3485613639 shutdown_connections 2024-06-11T19:22:11.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905f174640 1 -- 172.21.15.98:0/3485613639 >> 172.21.15.98:0/3485613639 conn(0x7f90580ab850 msgr2=0x7f9058137960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:11.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905f174640 1 -- 172.21.15.98:0/3485613639 shutdown_connections 2024-06-11T19:22:11.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905f174640 1 -- 172.21.15.98:0/3485613639 wait complete. 2024-06-11T19:22:11.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.636+0000 7f905f174640 1 librados: shutdown 2024-06-11T19:22:11.685 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:11.686 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:22:11.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:22:11.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.791+0000 7f36cbd33640 10 monclient: get_monmap_and_config 2024-06-11T19:22:11.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.791+0000 7f36cbd33640 10 monclient: build_initial_monmap 2024-06-11T19:22:11.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.791+0000 7f36cbd33640 10 monclient: monmap: 2024-06-11T19:22:11.790 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:11.790 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.790 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:11.792291+0000 2024-06-11T19:22:11.790 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:11.792291+0000 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:11.791 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.791+0000 7f36cbd33640 1 Processor -- start 2024-06-11T19:22:11.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 1 -- start start 2024-06-11T19:22:11.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 10 monclient: init 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 10 monclient: _add_conns ranks=[9,7,0,10,8,1,4,5,2,6,3] 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36c41347f0 0x7f36c409e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 10 monclient(hunting): picked mon.noname-a con 0x7f36c41347f0 addr v2:172.21.15.98:3300/0 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f36c4134bd0 0x7f36c40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 10 monclient(hunting): picked mon.noname-i con 0x7f36c4134bd0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36c4135010 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36c409d410 con 0x7f36c4135010 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f36c40687f0 con 0x7f36c41347f0 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f36c409d6f0 con 0x7f36c4134bd0 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36cbd33640 10 monclient(hunting): authenticate will time out at 1541.599582s 2024-06-11T19:22:11.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36c41347f0 0x7f36c409e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36ca2d0640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f36c4135010 0x7f36c40a3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56882/0 (socket says 172.21.15.98:56882) 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36ca2d0640 1 -- 172.21.15.98:0/47100360 learned_addr learned my addr 172.21.15.98:0/47100360 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 10 monclient(hunting): get_auth_request con 0x7f36c41347f0 auth_method 0 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 10 monclient(hunting): handle_auth_done global_id 65790 payload 274 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 1 -- 172.21.15.98:0/47100360 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f36c4134bd0 msgr2=0x7f36c40a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 1 --2- 172.21.15.98:0/47100360 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f36c4134bd0 0x7f36c40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 1 -- 172.21.15.98:0/47100360 >> v1:172.21.15.98:6789/0 conn(0x7f36c4135010 legacy=0x7f36c40a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 1 monclient: found mon.noname-a 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c9acf640 1 -- 172.21.15.98:0/47100360 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36c409d9d0 con 0x7f36c41347f0 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c8acd640 1 -- 172.21.15.98:0/47100360 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 891235593 0 0) 0x7f36c409d410 con 0x7f36c4135010 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.792+0000 7f36c8acd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c9acf640 1 --2- 172.21.15.98:0/47100360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c41347f0 0x7f36c409e6b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f36b40097a0 tx=0x7f36b402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4362b376fc93e4c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 1 -- 172.21.15.98:0/47100360 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f36b4031a10 con 0x7f36c41347f0 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 10 monclient: dump: 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.804 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 1 -- 172.21.15.98:0/47100360 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36b4037070 con 0x7f36c41347f0 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36cbd33640 5 monclient: authenticate success, global_id 65790 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36cbd33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36cbd33640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 1 -- 172.21.15.98:0/47100360 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f36b4031bd0 con 0x7f36c41347f0 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36cbd33640 4 set_mon_vals no callback set 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.793+0000 7f36c8acd640 10 monclient: dump: 2024-06-11T19:22:11.805 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.806 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.794+0000 7f36cbd33640 10 monclient: shutdown 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.794+0000 7f36cbd33640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.794+0000 7f36cbd33640 1 -- 172.21.15.98:0/47100360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c41347f0 msgr2=0x7f36c409e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.794+0000 7f36cbd33640 1 --2- 172.21.15.98:0/47100360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c41347f0 0x7f36c409e6b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f36b40097a0 tx=0x7f36b402b840 comp rx=0 tx=0).stop 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.794+0000 7f36cbd33640 1 -- 172.21.15.98:0/47100360 shutdown_connections 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.794+0000 7f36cbd33640 1 --2- 172.21.15.98:0/47100360 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f36c4134bd0 0x7f36c40a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.794+0000 7f36cbd33640 1 --2- 172.21.15.98:0/47100360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36c41347f0 0x7f36c409e6b0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.794+0000 7f36cbd33640 1 -- 172.21.15.98:0/47100360 >> 172.21.15.98:0/47100360 conn(0x7f36c40a4780 msgr2=0x7f36c40a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.794+0000 7f36cbd33640 1 -- 172.21.15.98:0/47100360 shutdown_connections 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.794+0000 7f36cbd33640 1 -- 172.21.15.98:0/47100360 wait complete. 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient: build_initial_monmap 2024-06-11T19:22:11.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient: monmap: 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:11.808 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 1 librados: starting msgr at 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 1 librados: starting objecter 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 1 Processor -- start 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 1 -- start start 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 1 librados: setting wanted keys 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 1 librados: calling monclient init 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient: init 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient: _add_conns ranks=[8,6,4,0,2,7,3,1,5] 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f36c41347f0 0x7f36c41cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient(hunting): picked mon.noname-e con 0x7f36c41347f0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f36c4134bd0 0x7f36c41ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient(hunting): picked mon.noname-d con 0x7f36c4134bd0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f36c4135010 0x7f36c411ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient(hunting): picked mon.noname-c con 0x7f36c4135010 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f36c4068c40 con 0x7f36c4135010 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f36c4068ac0 con 0x7f36c4134bd0 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f36c40a42e0 con 0x7f36c41347f0 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.795+0000 7f36cbd33640 10 monclient(hunting): authenticate will time out at 1541.602955s 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f36c41347f0 0x7f36c41cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f36c41347f0 0x7f36c41cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:58164/0 (socket says 172.21.15.98:58164) 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 1 -- 172.21.15.98:0/4113022712 learned_addr learned my addr 172.21.15.98:0/4113022712 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:11.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 10 monclient(hunting): get_auth_request con 0x7f36c41347f0 auth_method 0 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c92ce640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f36c4134bd0 0x7f36c41ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ca2d0640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f36c4135010 0x7f36c411ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c92ce640 10 monclient(hunting): get_auth_request con 0x7f36c4134bd0 auth_method 0 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c92ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c92ce640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c92ce640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ca2d0640 10 monclient(hunting): get_auth_request con 0x7f36c4135010 auth_method 0 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ca2d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ca2d0640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ca2d0640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c92ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c92ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c92ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ca2d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ca2d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ca2d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 10 monclient(hunting): handle_auth_done global_id 114218 payload 824 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 1 -- 172.21.15.98:0/4113022712 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f36c4134bd0 msgr2=0x7f36c41ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f36c4134bd0 0x7f36c41ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 1 -- 172.21.15.98:0/4113022712 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f36c4135010 msgr2=0x7f36c411ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f36c4135010 0x7f36c411ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 1 monclient: found mon.noname-e 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 1 -- 172.21.15.98:0/4113022712 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36c4135f60 con 0x7f36c41347f0 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c92ce640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f36c4134bd0 0x7f36c41ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36c9acf640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f36c41347f0 0x7f36c41cdb40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f36b4000c00 tx=0x7f36b40046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 10 client.?.objecter ms_handle_connect 0x7f36c41347f0 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f36b4031a10 con 0x7f36c41347f0 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 10 monclient: dump: 2024-06-11T19:22:11.813 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.814 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36cbd33640 5 monclient: authenticate success, global_id 114218 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36b4031bb0 con 0x7f36c41347f0 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f36b4031d60 con 0x7f36c41347f0 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36cbd33640 10 monclient: _renew_subs 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36cbd33640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36cbd33640 1 -- 172.21.15.98:0/4113022712 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36c40a3510 con 0x7f36c41347f0 2024-06-11T19:22:11.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36bbfff640 4 set_mon_vals no callback set 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36ba7fc640 10 monclient: dump: 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.816 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36cbd33640 10 client.114218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36cbd33640 10 monclient: _renew_subs 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36cbd33640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36cbd33640 1 -- 172.21.15.98:0/4113022712 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f36c4122a70 con 0x7f36c41347f0 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.796+0000 7f36cbd33640 1 librados: init done 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36cbd33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36cbd33640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411f1b0 0x7f36c411f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36cbd33640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36cbd33640 1 -- 172.21.15.98:0/4113022712 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f36c4122540 con 0x7f36c411f1b0 2024-06-11T19:22:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f36b40315a0 con 0x7f36c41347f0 2024-06-11T19:22:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36ba7fc640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f36a803fde0 0x7f36a80422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36c92ce640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411f1b0 0x7f36c411f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(517..517 src has 1..517) ==== 6685+0+0 (secure 0 0 0) 0x7f36b402f050 con 0x7f36c41347f0 2024-06-11T19:22:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36ba7fc640 10 client.114218.objecter ms_dispatch 0x7f36c4000b90 osd_map(517..517 src has 1..517) 2024-06-11T19:22:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36ba7fc640 3 client.114218.objecter handle_osd_map got epochs [517,517] > 0 2024-06-11T19:22:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36c92ce640 10 monclient: get_auth_request con 0x7f36c411f1b0 auth_method 0 2024-06-11T19:22:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36ba7fc640 3 client.114218.objecter handle_osd_map decoding full epoch 517 2024-06-11T19:22:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36c92ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36c92ce640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36c92ce640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36c92ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36c92ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36ba7fc640 20 client.114218.objecter dump_active .. 0 homeless 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.797+0000 7f36c92ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ca2d0640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f36a803fde0 0x7f36a80422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ca2d0640 10 monclient: get_auth_request con 0x7f36a803fde0 auth_method 0 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36c92ce640 10 monclient(hunting): handle_auth_done global_id 114186 payload 824 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36c92ce640 1 -- 172.21.15.98:0/4113022712 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f36c409e6b0 con 0x7f36c411f1b0 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 10 client.114218.objecter ms_handle_connect 0x7f36c411f1b0 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 10 client.114218.objecter resend_mon_ops 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36c92ce640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411f1b0 0x7f36c411f590 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f36c000eae0 tx=0x7f36c0009f20 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 10 client.114218.objecter ms_handle_connect 0x7f36a803fde0 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ca2d0640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f36a803fde0 0x7f36a80422a0 secure :-1 s=READY pgs=4257 cs=0 l=1 rev1=1 crypto rx=0x7f36c41203d0 tx=0x7f36bc004d40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f36c0017070 con 0x7f36c411f1b0 2024-06-11T19:22:11.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 10 monclient: dump: 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.820 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.821 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f36c000cad0 con 0x7f36c411f1b0 2024-06-11T19:22:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411f1b0 msgr2=0x7f36c411f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411f1b0 0x7f36c411f590 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f36c000eae0 tx=0x7f36c0009f20 comp rx=0 tx=0).stop 2024-06-11T19:22:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411f1b0 msgr2=0x7f36c411f590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:11.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.798+0000 7f36ba7fc640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411f1b0 0x7f36c411f590 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36cbd33640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:11.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36cbd33640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411fad0 0x7f36c40a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36cbd33640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36cbd33640 1 -- 172.21.15.98:0/4113022712 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f36c4050830 con 0x7f36c411fad0 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36c92ce640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411fad0 0x7f36c40a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36c92ce640 10 monclient: get_auth_request con 0x7f36c411fad0 auth_method 0 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36c92ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36c92ce640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36c92ce640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36c92ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36c92ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36c92ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36c92ce640 10 monclient(hunting): handle_auth_done global_id 114195 payload 824 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.805+0000 7f36c92ce640 1 -- 172.21.15.98:0/4113022712 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f36c41222f0 con 0x7f36c411fad0 2024-06-11T19:22:11.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36c92ce640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411fad0 0x7f36c40a5a80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f36c0000c00 tx=0x7f36c00047d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 10 client.114218.objecter ms_handle_connect 0x7f36c411fad0 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 10 client.114218.objecter resend_mon_ops 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f36c0017070 con 0x7f36c411fad0 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 10 monclient: dump: 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f36c0013680 con 0x7f36c411fad0 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411fad0 msgr2=0x7f36c40a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411fad0 0x7f36c40a5a80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f36c0000c00 tx=0x7f36c00047d0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 1 -- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411fad0 msgr2=0x7f36c40a5a80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:11.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ba7fc640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36c411fad0 0x7f36c40a5a80 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f36c0000c00 tx=0x7f36c00047d0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.827 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:11.827 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:11.827 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:11.827 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:11.827 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:11.827 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 20175, 2024-06-11T19:22:11.827 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:11.827 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:11.827 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:11.827 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:11.827 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:11.827 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.828 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.831 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.835 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:11.836 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:11.838 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:11.839 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.840 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.844 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.846 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:11.847 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:11.848 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:11.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.848 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.848 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:11.848 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:11.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 10 librados: watch_flush enter 2024-06-11T19:22:11.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 10 librados: watch_flush exit 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 20 client.114218.objecter shutdown clearing up homeless session... 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 10 client.114218.objecter successfully canceled tick 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 1 -- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f36a803fde0 msgr2=0x7f36a80422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f36a803fde0 0x7f36a80422a0 secure :-1 s=READY pgs=4257 cs=0 l=1 rev1=1 crypto rx=0x7f36c41203d0 tx=0x7f36bc004d40 comp rx=0 tx=0).stop 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 10 monclient: shutdown 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 1 -- 172.21.15.98:0/4113022712 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f36c41347f0 msgr2=0x7f36c41cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 1 --2- 172.21.15.98:0/4113022712 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f36c41347f0 0x7f36c41cdb40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f36b4000c00 tx=0x7f36b40046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ca2d0640 1 -- 172.21.15.98:0/4113022712 reap_dead start 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36ca2d0640 1 -- 172.21.15.98:0/4113022712 reap_dead start 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 1 -- 172.21.15.98:0/4113022712 shutdown_connections 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 1 -- 172.21.15.98:0/4113022712 >> 172.21.15.98:0/4113022712 conn(0x7f36c40a4780 msgr2=0x7f36c41be7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:11.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 1 -- 172.21.15.98:0/4113022712 shutdown_connections 2024-06-11T19:22:11.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.806+0000 7f36cbd33640 1 -- 172.21.15.98:0/4113022712 wait complete. 2024-06-11T19:22:11.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.807+0000 7f36cbd33640 1 librados: shutdown 2024-06-11T19:22:11.851 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:11.851 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:22:11.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:22:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.952+0000 7fdda4dfa640 10 monclient: get_monmap_and_config 2024-06-11T19:22:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.952+0000 7fdda4dfa640 10 monclient: build_initial_monmap 2024-06-11T19:22:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.952+0000 7fdda4dfa640 10 monclient: monmap: 2024-06-11T19:22:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:11.953476+0000 2024-06-11T19:22:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:11.953476+0000 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:11.952 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 1 Processor -- start 2024-06-11T19:22:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 1 -- start start 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 10 monclient: init 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 10 monclient: _add_conns ranks=[10,3,4,5,9,1,7,6,0,8,2] 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdda00a4920 0x7fdda00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 10 monclient(hunting): picked mon.noname-f con 0x7fdda00a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdda00a5240 0x7fdda00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 10 monclient(hunting): picked mon.noname-g con 0x7fdda00a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdda00a2530 0x7fdda00a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 10 monclient(hunting): picked mon.noname-c con 0x7fdda00a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fdda00a7140 con 0x7fdda00a4920 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fdda00a2b50 con 0x7fdda00a5240 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fdda00a2e30 con 0x7fdda00a2530 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdda4dfa640 10 monclient(hunting): authenticate will time out at 1541.760725s 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9ed9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdda00a2530 0x7fdda00a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9ed9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdda00a2530 0x7fdda00a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38170/0 (socket says 172.21.15.98:38170) 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9ed9d640 1 -- 172.21.15.98:0/3806664011 learned_addr learned my addr 172.21.15.98:0/3806664011 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9ed9d640 10 monclient(hunting): get_auth_request con 0x7fdda00a2530 auth_method 0 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9ed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9dd9b640 1 --2- 172.21.15.98:0/3806664011 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdda00a5240 0x7fdda00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9dd9b640 10 monclient(hunting): get_auth_request con 0x7fdda00a5240 auth_method 0 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9dd9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.953+0000 7fdd9e59c640 1 --2- 172.21.15.98:0/3806664011 >> v2:172.21.15.117:3300/0 conn(0x7fdda00a4920 0x7fdda00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9e59c640 10 monclient(hunting): get_auth_request con 0x7fdda00a4920 auth_method 0 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9ed9d640 10 monclient(hunting): handle_auth_done global_id 114259 payload 274 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9ed9d640 1 -- 172.21.15.98:0/3806664011 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdda00a5240 msgr2=0x7fdda00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9ed9d640 1 --2- 172.21.15.98:0/3806664011 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdda00a5240 0x7fdda00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9ed9d640 1 -- 172.21.15.98:0/3806664011 >> v2:172.21.15.117:3300/0 conn(0x7fdda00a4920 msgr2=0x7fdda00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9ed9d640 1 --2- 172.21.15.98:0/3806664011 >> v2:172.21.15.117:3300/0 conn(0x7fdda00a4920 0x7fdda00a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9ed9d640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9ed9d640 1 monclient: found mon.noname-c 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9ed9d640 1 -- 172.21.15.98:0/3806664011 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda00a3110 con 0x7fdda00a2530 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9e59c640 1 --2- 172.21.15.98:0/3806664011 >> v2:172.21.15.117:3300/0 conn(0x7fdda00a4920 0x7fdda00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.954+0000 7fdd9ed9d640 1 --2- 172.21.15.98:0/3806664011 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdda00a2530 0x7fdda00a5ee0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdd98009e50 tx=0x7fdd9802bdb0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=58ba7fcd95507b83 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 1 -- 172.21.15.98:0/3806664011 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd9802cbc0 con 0x7fdda00a2530 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 10 monclient: dump: 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.967 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 1 -- 172.21.15.98:0/3806664011 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd9802cd80 con 0x7fdda00a2530 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 5 monclient: authenticate success, global_id 114259 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 1 -- 172.21.15.98:0/3806664011 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd98034860 con 0x7fdda00a2530 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 4 set_mon_vals no callback set 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdd9d59a640 10 monclient: dump: 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.969 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 10 monclient: shutdown 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 1 -- 172.21.15.98:0/3806664011 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdda00a2530 msgr2=0x7fdda00a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 1 --2- 172.21.15.98:0/3806664011 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdda00a2530 0x7fdda00a5ee0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdd98009e50 tx=0x7fdd9802bdb0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 1 -- 172.21.15.98:0/3806664011 shutdown_connections 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 1 --2- 172.21.15.98:0/3806664011 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdda00a2530 0x7fdda00a5ee0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 1 --2- 172.21.15.98:0/3806664011 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdda00a5240 0x7fdda00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 1 --2- 172.21.15.98:0/3806664011 >> v2:172.21.15.117:3300/0 conn(0x7fdda00a4920 0x7fdda00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 1 -- 172.21.15.98:0/3806664011 >> 172.21.15.98:0/3806664011 conn(0x7fdda00ab850 msgr2=0x7fdda00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 1 -- 172.21.15.98:0/3806664011 shutdown_connections 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.955+0000 7fdda4dfa640 1 -- 172.21.15.98:0/3806664011 wait complete. 2024-06-11T19:22:11.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.956+0000 7fdda4dfa640 10 monclient: build_initial_monmap 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.956+0000 7fdda4dfa640 10 monclient: monmap: 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:11.971 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.956+0000 7fdda4dfa640 1 librados: starting msgr at 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.956+0000 7fdda4dfa640 1 librados: starting objecter 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.956+0000 7fdda4dfa640 1 Processor -- start 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.956+0000 7fdda4dfa640 1 -- start start 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.956+0000 7fdda4dfa640 1 librados: setting wanted keys 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.956+0000 7fdda4dfa640 1 librados: calling monclient init 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.956+0000 7fdda4dfa640 10 monclient: init 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 monclient: _add_conns ranks=[8,3,1,4,7,0,5,6,2] 2024-06-11T19:22:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdda00a2530 0x7fdda01cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 monclient(hunting): picked mon.noname-e con 0x7fdda00a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdda00a4920 0x7fdda01ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 monclient(hunting): picked mon.noname-g con 0x7fdda00a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdda00a5240 0x7fdda0122d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 monclient(hunting): picked mon.noname-f con 0x7fdda00a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdda00a6da0 con 0x7fdda00a5240 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fdda0138e50 con 0x7fdda00a4920 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fdda00a6420 con 0x7fdda00a2530 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:11.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 monclient(hunting): authenticate will time out at 1541.764171s 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdda00a2530 0x7fdda01cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdda00a2530 0x7fdda01cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:58176/0 (socket says 172.21.15.98:58176) 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 1 -- 172.21.15.98:0/399029685 learned_addr learned my addr 172.21.15.98:0/399029685 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 10 monclient(hunting): get_auth_request con 0x7fdda00a2530 auth_method 0 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9dd9b640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdda00a4920 0x7fdda01ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9ed9d640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdda00a5240 0x7fdda0122d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9dd9b640 10 monclient(hunting): get_auth_request con 0x7fdda00a4920 auth_method 0 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9dd9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 10 monclient(hunting): handle_auth_done global_id 114227 payload 824 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 1 -- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdda00a4920 msgr2=0x7fdda01ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdda00a4920 0x7fdda01ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 1 -- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdda00a5240 msgr2=0x7fdda0122d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdda00a5240 0x7fdda0122d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 10 monclient: _finish_hunting 0 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 1 monclient: found mon.noname-e 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 1 -- 172.21.15.98:0/399029685 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda0126ce0 con 0x7fdda00a2530 2024-06-11T19:22:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9ed9d640 10 monclient: get_auth_request con 0x7fdda00a5240 auth_method 0 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9ed9d640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdda00a5240 0x7fdda0122d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9e59c640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdda00a2530 0x7fdda01cdc50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000e790 tx=0x7fdd9000ec50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 10 client.?.objecter ms_handle_connect 0x7fdda00a2530 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd9000f040 con 0x7fdda00a2530 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 10 monclient: dump: 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 10 monclient: _finish_auth 0 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 10 monclient: _check_auth_tickets 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 5 monclient: authenticate success, global_id 114227 2024-06-11T19:22:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd900044a0 con 0x7fdda00a2530 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 monclient: _renew_subs 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd9000eea0 con 0x7fdda00a2530 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 1 -- 172.21.15.98:0/399029685 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda00aac70 con 0x7fdda00a2530 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd9cd99640 4 set_mon_vals no callback set 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdd8effd640 10 monclient: dump: 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.957+0000 7fdda4dfa640 10 client.114227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.958+0000 7fdda4dfa640 10 monclient: _renew_subs 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.958+0000 7fdda4dfa640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.958+0000 7fdda4dfa640 1 -- 172.21.15.98:0/399029685 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda0126720 con 0x7fdda00a2530 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.958+0000 7fdda4dfa640 1 librados: init done 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.958+0000 7fdda4dfa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.958+0000 7fdda4dfa640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda01232b0 0x7fdda0123690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.958+0000 7fdda4dfa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.958+0000 7fdda4dfa640 1 -- 172.21.15.98:0/399029685 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fdda0125550 con 0x7fdda01232b0 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.958+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fdd90004030 con 0x7fdda00a2530 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9dd9b640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda01232b0 0x7fdda0123690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9dd9b640 10 monclient: get_auth_request con 0x7fdda01232b0 auth_method 0 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9dd9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdd8003fdb0 0x7fdd80042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9dd9b640 10 monclient(hunting): handle_auth_done global_id 23540 payload 824 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(517..517 src has 1..517) ==== 6685+0+0 (secure 0 0 0) 0x7fdd900518d0 con 0x7fdda00a2530 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9dd9b640 1 -- 172.21.15.98:0/399029685 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdda0126520 con 0x7fdda01232b0 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 10 client.114227.objecter ms_dispatch 0x7fdda0000b90 osd_map(517..517 src has 1..517) 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 3 client.114227.objecter handle_osd_map got epochs [517,517] > 0 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 3 client.114227.objecter handle_osd_map decoding full epoch 517 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 20 client.114227.objecter dump_active .. 0 homeless 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9ed9d640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdd8003fdb0 0x7fdd80042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9ed9d640 10 monclient: get_auth_request con 0x7fdd8003fdb0 auth_method 0 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 10 client.114227.objecter ms_handle_connect 0x7fdda01232b0 2024-06-11T19:22:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 10 client.114227.objecter resend_mon_ops 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9dd9b640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda01232b0 0x7fdda0123690 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fdda0124820 tx=0x7fdd94011920 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd940044a0 con 0x7fdda01232b0 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 10 monclient: dump: 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd8effd640 10 client.114227.objecter ms_handle_connect 0x7fdd8003fdb0 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.959+0000 7fdd9ed9d640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdd8003fdb0 0x7fdd80042270 secure :-1 s=READY pgs=4258 cs=0 l=1 rev1=1 crypto rx=0x7fdd98035000 tx=0x7fdd98030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.960+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdd94004030 con 0x7fdda01232b0 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.960+0000 7fdd8effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.960+0000 7fdd8effd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.960+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda01232b0 msgr2=0x7fdda0123690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.960+0000 7fdd8effd640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda01232b0 0x7fdda0123690 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fdda0124820 tx=0x7fdd94011920 comp rx=0 tx=0).stop 2024-06-11T19:22:11.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.960+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda01232b0 msgr2=0x7fdda0123690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.960+0000 7fdd8effd640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda01232b0 0x7fdda0123690 secure :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fdda0124820 tx=0x7fdd94011920 comp rx=0 tx=0).stop 2024-06-11T19:22:11.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.966+0000 7fdda4dfa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:11.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.966+0000 7fdda4dfa640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda0123bd0 0x7fdda009e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:11.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.966+0000 7fdda4dfa640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.966+0000 7fdda4dfa640 1 -- 172.21.15.98:0/399029685 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fdda0050830 con 0x7fdda0123bd0 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.966+0000 7fdd9dd9b640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda0123bd0 0x7fdda009e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.966+0000 7fdd9dd9b640 10 monclient: get_auth_request con 0x7fdda0123bd0 auth_method 0 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.966+0000 7fdd9dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.966+0000 7fdd9dd9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.966+0000 7fdd9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.966+0000 7fdd9dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.966+0000 7fdd9dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.966+0000 7fdd9dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd9dd9b640 10 monclient(hunting): handle_auth_done global_id 23549 payload 824 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd9dd9b640 1 -- 172.21.15.98:0/399029685 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdda00a4d00 con 0x7fdda0123bd0 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 10 client.114227.objecter ms_handle_connect 0x7fdda0123bd0 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 10 client.114227.objecter resend_mon_ops 2024-06-11T19:22:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd9dd9b640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda0123bd0 0x7fdda009e9f0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fdda0124820 tx=0x7fdd940046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd94005c60 con 0x7fdda0123bd0 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 10 monclient: dump: 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7fdd94005c60 con 0x7fdda0123bd0 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda0123bd0 msgr2=0x7fdda009e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda0123bd0 0x7fdda009e9f0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fdda0124820 tx=0x7fdd940046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 1 -- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda0123bd0 msgr2=0x7fdda009e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.967+0000 7fdd8effd640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdda0123bd0 0x7fdda009e9f0 secure :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fdda0124820 tx=0x7fdd940046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:11.989 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:11.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:11.989 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:11.989 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:11.989 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:11.989 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 782162, 2024-06-11T19:22:11.989 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:11.989 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:11.989 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:11.989 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:11.989 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:11.989 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.990 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.991 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:11.992 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:11.993 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.995 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:11.996 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:11.998 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:11.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:12.001 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:12.003 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:12.005 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.006 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:12.008 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.009 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:12.010 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:22:12.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.010 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:12.010 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:12.010 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:12.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.010 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.010 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:12.010 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 10 librados: watch_flush enter 2024-06-11T19:22:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 10 librados: watch_flush exit 2024-06-11T19:22:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 20 client.114227.objecter shutdown clearing up homeless session... 2024-06-11T19:22:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 10 client.114227.objecter successfully canceled tick 2024-06-11T19:22:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 1 -- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdd8003fdb0 msgr2=0x7fdd80042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdd8003fdb0 0x7fdd80042270 secure :-1 s=READY pgs=4258 cs=0 l=1 rev1=1 crypto rx=0x7fdd98035000 tx=0x7fdd98030000 comp rx=0 tx=0).stop 2024-06-11T19:22:12.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 10 monclient: shutdown 2024-06-11T19:22:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 1 -- 172.21.15.98:0/399029685 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdda00a2530 msgr2=0x7fdda01cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdda00a2530 0x7fdda01cdc50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000e790 tx=0x7fdd9000ec50 comp rx=0 tx=0).stop 2024-06-11T19:22:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdd9ed9d640 1 -- 172.21.15.98:0/399029685 reap_dead start 2024-06-11T19:22:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 1 -- 172.21.15.98:0/399029685 shutdown_connections 2024-06-11T19:22:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 1 --2- 172.21.15.98:0/399029685 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdda00a2530 0x7fdda01cdc50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 1 -- 172.21.15.98:0/399029685 >> 172.21.15.98:0/399029685 conn(0x7fdda00ab850 msgr2=0x7fdda01bef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 1 -- 172.21.15.98:0/399029685 shutdown_connections 2024-06-11T19:22:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 1 -- 172.21.15.98:0/399029685 wait complete. 2024-06-11T19:22:12.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:11.968+0000 7fdda4dfa640 1 librados: shutdown 2024-06-11T19:22:12.013 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:12.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:22:12.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:22:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.114+0000 7fd9366e0640 10 monclient: get_monmap_and_config 2024-06-11T19:22:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.114+0000 7fd9366e0640 10 monclient: build_initial_monmap 2024-06-11T19:22:12.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.114+0000 7fd9366e0640 10 monclient: monmap: 2024-06-11T19:22:12.114 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:12.114 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.114 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:12.115642+0000 2024-06-11T19:22:12.114 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:12.115642+0000 2024-06-11T19:22:12.114 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:12.114 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.114 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:12.114 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:12.114 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:12.114 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:12.114 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:12.115 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:12.115 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:12.115 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:12.115 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:12.115 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:12.115 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:12.115 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 1 Processor -- start 2024-06-11T19:22:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 1 -- start start 2024-06-11T19:22:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 10 monclient: init 2024-06-11T19:22:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 10 monclient: _add_conns ranks=[4,9,7,10,2,5,8,1,6,0,3] 2024-06-11T19:22:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd9300a4920 0x7fd9300a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 10 monclient(hunting): picked mon.noname-c con 0x7fd9300a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9300a5240 0x7fd9300a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 10 monclient(hunting): picked mon.noname-a con 0x7fd9300a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:22:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd9300a2530 0x7fd9300a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 10 monclient(hunting): picked mon.noname-i con 0x7fd9300a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd9300a7140 con 0x7fd9300a5240 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd9300a2b50 con 0x7fd9300a4920 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd9300a2e30 con 0x7fd9300a2530 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.115+0000 7fd9366e0640 10 monclient(hunting): authenticate will time out at 1541.922942s 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92f7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9300a5240 0x7fd9300a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd9300a4920 0x7fd9300a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92f7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9300a5240 0x7fd9300a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52330/0 (socket says 172.21.15.98:52330) 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92f7fe640 1 -- 172.21.15.98:0/2705897789 learned_addr learned my addr 172.21.15.98:0/2705897789 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92f7fe640 10 monclient(hunting): get_auth_request con 0x7fd9300a5240 auth_method 0 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 10 monclient(hunting): get_auth_request con 0x7fd9300a4920 auth_method 0 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92f7fe640 1 -- 172.21.15.98:0/2705897789 >> v2:172.21.15.98:3300/0 conn(0x7fd9300a5240 msgr2=0x7fd9300a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92f7fe640 1 -- 172.21.15.98:0/2705897789 >> v2:172.21.15.98:3300/0 conn(0x7fd9300a5240 msgr2=0x7fd9300a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92f7fe640 1 --2- 172.21.15.98:0/2705897789 >> v2:172.21.15.98:3300/0 conn(0x7fd9300a5240 0x7fd9300a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92f7fe640 1 --2- 172.21.15.98:0/2705897789 >> v2:172.21.15.98:3300/0 conn(0x7fd9300a5240 0x7fd9300a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd934c7d640 1 --2- 172.21.15.98:0/2705897789 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd9300a2530 0x7fd9300a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd934c7d640 10 monclient(hunting): get_auth_request con 0x7fd9300a2530 auth_method 0 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd934c7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd934c7d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd934c7d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd934c7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd934c7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd934c7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 10 monclient(hunting): handle_auth_done global_id 114268 payload 274 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 1 -- 172.21.15.98:0/2705897789 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd9300a2530 msgr2=0x7fd9300a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 1 --2- 172.21.15.98:0/2705897789 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd9300a2530 0x7fd9300a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 1 -- 172.21.15.98:0/2705897789 >> v2:172.21.15.98:3300/0 conn(0x7fd9300a5240 msgr2=0x7fd9300a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 1 --2- 172.21.15.98:0/2705897789 >> v2:172.21.15.98:3300/0 conn(0x7fd9300a5240 0x7fd9300a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 1 monclient: found mon.noname-c 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.116+0000 7fd92ffff640 1 -- 172.21.15.98:0/2705897789 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9300a3110 con 0x7fd9300a4920 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92ffff640 1 --2- 172.21.15.98:0/2705897789 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd9300a4920 0x7fd9300a4d00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd9200097a0 tx=0x7fd92002b6f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1959ac33b70abeb0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 1 -- 172.21.15.98:0/2705897789 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd920037070 con 0x7fd9300a4920 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 10 monclient: dump: 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 10 monclient: _finish_auth 0 2024-06-11T19:22:12.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 10 monclient: _check_auth_tickets 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 1 -- 172.21.15.98:0/2705897789 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd920030980 con 0x7fd9300a4920 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 5 monclient: authenticate success, global_id 114268 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 1 -- 172.21.15.98:0/2705897789 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd920030bb0 con 0x7fd9300a4920 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 4 set_mon_vals no callback set 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd92effd640 10 monclient: dump: 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.132 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.133 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 10 monclient: shutdown 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 1 -- 172.21.15.98:0/2705897789 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd9300a4920 msgr2=0x7fd9300a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 1 --2- 172.21.15.98:0/2705897789 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd9300a4920 0x7fd9300a4d00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd9200097a0 tx=0x7fd92002b6f0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 1 -- 172.21.15.98:0/2705897789 shutdown_connections 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 1 --2- 172.21.15.98:0/2705897789 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd9300a2530 0x7fd9300a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 1 --2- 172.21.15.98:0/2705897789 >> v2:172.21.15.98:3300/0 conn(0x7fd9300a5240 0x7fd9300a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 1 --2- 172.21.15.98:0/2705897789 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd9300a4920 0x7fd9300a4d00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 1 -- 172.21.15.98:0/2705897789 >> 172.21.15.98:0/2705897789 conn(0x7fd9300ab850 msgr2=0x7fd9300a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 1 -- 172.21.15.98:0/2705897789 shutdown_connections 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.117+0000 7fd9366e0640 1 -- 172.21.15.98:0/2705897789 wait complete. 2024-06-11T19:22:12.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.118+0000 7fd9366e0640 10 monclient: build_initial_monmap 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.118+0000 7fd9366e0640 10 monclient: monmap: 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:12.135 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.118+0000 7fd9366e0640 1 librados: starting msgr at 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.118+0000 7fd9366e0640 1 librados: starting objecter 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.118+0000 7fd9366e0640 1 Processor -- start 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 1 -- start start 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 1 librados: setting wanted keys 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 1 librados: calling monclient init 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 10 monclient: init 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 10 monclient: _add_conns ranks=[6,5,2,1,0,4,3,7,8] 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd9300a2530 0x7fd9301cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 10 monclient(hunting): picked mon.noname-d con 0x7fd9300a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd9300a4920 0x7fd9301cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 10 monclient(hunting): picked mon.noname-h con 0x7fd9300a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9300a5240 0x7fd930122df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 10 monclient(hunting): picked mon.noname-b con 0x7fd9300a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd9300a6da0 con 0x7fd9300a5240 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd930138e50 con 0x7fd9300a4920 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd9300a6420 con 0x7fd9300a2530 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd9366e0640 10 monclient(hunting): authenticate will time out at 1541.926489s 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd9300a2530 0x7fd9301cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd9300a2530 0x7fd9301cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34346/0 (socket says 172.21.15.98:34346) 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 1 -- 172.21.15.98:0/403057563 learned_addr learned my addr 172.21.15.98:0/403057563 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 10 monclient(hunting): get_auth_request con 0x7fd9300a2530 auth_method 0 2024-06-11T19:22:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd934c7d640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9300a5240 0x7fd930122df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd934c7d640 10 monclient(hunting): get_auth_request con 0x7fd9300a5240 auth_method 0 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd934c7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd934c7d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd934c7d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd934c7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd934c7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd934c7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92f7fe640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd9300a4920 0x7fd9301cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92f7fe640 10 monclient(hunting): get_auth_request con 0x7fd9300a4920 auth_method 0 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 10 monclient(hunting): handle_auth_done global_id 55635 payload 824 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 1 -- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd9300a4920 msgr2=0x7fd9301cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd9300a4920 0x7fd9301cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 1 -- 172.21.15.98:0/403057563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9300a5240 msgr2=0x7fd930122df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9300a5240 0x7fd930122df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 1 monclient: found mon.noname-d 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:22:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd92ffff640 1 -- 172.21.15.98:0/403057563 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9300a8ea0 con 0x7fd9300a2530 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.119+0000 7fd934c7d640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd9300a5240 0x7fd930122df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92ffff640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd9300a2530 0x7fd9301cdaa0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd9200097a0 tx=0x7fd9200046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 10 client.?.objecter ms_handle_connect 0x7fd9300a2530 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd920040020 con 0x7fd9300a2530 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 10 monclient: dump: 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 10 monclient: _finish_auth 0 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 10 monclient: _check_auth_tickets 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd9366e0640 5 monclient: authenticate success, global_id 55635 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd920038040 con 0x7fd9300a2530 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd9366e0640 10 monclient: _renew_subs 2024-06-11T19:22:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd9366e0640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd9366e0640 1 -- 172.21.15.98:0/403057563 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd930126930 con 0x7fd9300a2530 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd920037070 con 0x7fd9300a2530 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92cff9640 10 monclient: dump: 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd9366e0640 10 client.55635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd9366e0640 10 monclient: _renew_subs 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd9366e0640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd92e7fc640 4 set_mon_vals no callback set 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd9366e0640 1 -- 172.21.15.98:0/403057563 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9301251d0 con 0x7fd9300a2530 2024-06-11T19:22:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.120+0000 7fd9366e0640 1 librados: init done 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd920043020 con 0x7fd9300a2530 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd9366e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd9366e0640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123330 0x7fd930123710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd9366e0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd9366e0640 1 -- 172.21.15.98:0/403057563 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd930050830 con 0x7fd930123330 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92cff9640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd91c03fdb0 0x7fd91c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(517..517 src has 1..517) ==== 6685+0+0 (secure 0 0 0) 0x7fd9200730c0 con 0x7fd9300a2530 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92cff9640 10 client.55635.objecter ms_dispatch 0x7fd930000b90 osd_map(517..517 src has 1..517) 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92cff9640 3 client.55635.objecter handle_osd_map got epochs [517,517] > 0 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92cff9640 3 client.55635.objecter handle_osd_map decoding full epoch 517 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92f7fe640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123330 0x7fd930123710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92f7fe640 10 monclient: get_auth_request con 0x7fd930123330 auth_method 0 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd934c7d640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd91c03fdb0 0x7fd91c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92cff9640 20 client.55635.objecter dump_active .. 0 homeless 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd934c7d640 10 monclient: get_auth_request con 0x7fd91c03fdb0 auth_method 0 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92f7fe640 10 monclient(hunting): handle_auth_done global_id 84733 payload 824 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92f7fe640 1 -- 172.21.15.98:0/403057563 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9300a92d0 con 0x7fd930123330 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92cff9640 10 client.55635.objecter ms_handle_connect 0x7fd91c03fdb0 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd934c7d640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd91c03fdb0 0x7fd91c042270 secure :-1 s=READY pgs=4259 cs=0 l=1 rev1=1 crypto rx=0x7fd930125ea0 tx=0x7fd92800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 10 client.55635.objecter ms_handle_connect 0x7fd930123330 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 10 client.55635.objecter resend_mon_ops 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.121+0000 7fd92f7fe640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123330 0x7fd930123710 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd9300a5620 tx=0x7fd924010b60 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd92401a020 con 0x7fd930123330 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 10 monclient: dump: 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.147 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd92400e790 con 0x7fd930123330 2024-06-11T19:22:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123330 msgr2=0x7fd930123710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123330 0x7fd930123710 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd9300a5620 tx=0x7fd924010b60 comp rx=0 tx=0).stop 2024-06-11T19:22:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123330 msgr2=0x7fd930123710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.122+0000 7fd92cff9640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123330 0x7fd930123710 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.128+0000 7fd9366e0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.128+0000 7fd9366e0640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123c50 0x7fd93009e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.128+0000 7fd9366e0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.128+0000 7fd9366e0640 1 -- 172.21.15.98:0/403057563 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd9300a6750 con 0x7fd930123c50 2024-06-11T19:22:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92f7fe640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123c50 0x7fd93009e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92f7fe640 10 monclient: get_auth_request con 0x7fd930123c50 auth_method 0 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92f7fe640 10 monclient(hunting): handle_auth_done global_id 84742 payload 824 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92f7fe640 1 -- 172.21.15.98:0/403057563 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd930138500 con 0x7fd930123c50 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92f7fe640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123c50 0x7fd93009e9f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd92401b000 tx=0x7fd9240046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92cff9640 10 client.55635.objecter ms_handle_connect 0x7fd930123c50 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92cff9640 10 client.55635.objecter resend_mon_ops 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd92401a040 con 0x7fd930123c50 2024-06-11T19:22:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92cff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.129+0000 7fd92cff9640 10 monclient: dump: 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.152 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fd924015ac0 con 0x7fd930123c50 2024-06-11T19:22:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd92cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd92cff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123c50 msgr2=0x7fd93009e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd92cff9640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123c50 0x7fd93009e9f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd92401b000 tx=0x7fd9240046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd92cff9640 1 -- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123c50 msgr2=0x7fd93009e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd92cff9640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd930123c50 0x7fd93009e9f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.153 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:12.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:12.153 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:12.153 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:12.153 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2024-06-11T19:22:12.153 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 403899, 2024-06-11T19:22:12.153 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:12.153 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:12.153 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:22:12.153 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:12.153 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:12.153 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:12.154 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:12.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:12.156 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:12.157 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.159 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:12.174 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:12.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 10 librados: watch_flush enter 2024-06-11T19:22:12.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 10 librados: watch_flush exit 2024-06-11T19:22:12.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 20 client.55635.objecter shutdown clearing up homeless session... 2024-06-11T19:22:12.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 10 client.55635.objecter successfully canceled tick 2024-06-11T19:22:12.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 1 -- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd91c03fdb0 msgr2=0x7fd91c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd91c03fdb0 0x7fd91c042270 secure :-1 s=READY pgs=4259 cs=0 l=1 rev1=1 crypto rx=0x7fd930125ea0 tx=0x7fd92800b040 comp rx=0 tx=0).stop 2024-06-11T19:22:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 10 monclient: shutdown 2024-06-11T19:22:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 1 -- 172.21.15.98:0/403057563 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd9300a2530 msgr2=0x7fd9301cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 1 --2- 172.21.15.98:0/403057563 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd9300a2530 0x7fd9301cdaa0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd9200097a0 tx=0x7fd9200046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd934c7d640 1 -- 172.21.15.98:0/403057563 reap_dead start 2024-06-11T19:22:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd934c7d640 1 -- 172.21.15.98:0/403057563 reap_dead start 2024-06-11T19:22:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 1 -- 172.21.15.98:0/403057563 shutdown_connections 2024-06-11T19:22:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 1 -- 172.21.15.98:0/403057563 >> 172.21.15.98:0/403057563 conn(0x7fd9300ab850 msgr2=0x7fd9301bffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 1 -- 172.21.15.98:0/403057563 shutdown_connections 2024-06-11T19:22:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 1 -- 172.21.15.98:0/403057563 wait complete. 2024-06-11T19:22:12.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.130+0000 7fd9366e0640 1 librados: shutdown 2024-06-11T19:22:12.178 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:22:12.178 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['e', 'h', 'f', 'd'] 2024-06-11T19:22:12.178 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-06-11T19:22:12.178 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-06-11T19:22:12.178 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-06-11T19:22:12.178 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-06-11T19:22:12.179 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:22:12.190 INFO:tasks.ceph.mon.e:Stopped 2024-06-11T19:22:12.190 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h 2024-06-11T19:22:12.190 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h store 2024-06-11T19:22:12.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h sync_force --yes-i-really-mean-it 2024-06-11T19:22:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.333+0000 7fdb29620640 10 monclient: get_monmap_and_config 2024-06-11T19:22:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.333+0000 7fdb29620640 10 monclient: build_initial_monmap 2024-06-11T19:22:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.333+0000 7fdb29620640 10 monclient: monmap: 2024-06-11T19:22:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:12.334810+0000 2024-06-11T19:22:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:12.334810+0000 2024-06-11T19:22:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:12.335 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 1 Processor -- start 2024-06-11T19:22:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 1 -- start start 2024-06-11T19:22:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 10 monclient: init 2024-06-11T19:22:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 10 monclient: _add_conns ranks=[9,8,7,1,10,5,0,4,3,6,2] 2024-06-11T19:22:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdb240a4920 0x7fdb240a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 10 monclient(hunting): picked mon.noname-a con 0x7fdb240a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:22:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdb240a5240 0x7fdb240a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 10 monclient(hunting): picked mon.noname-e con 0x7fdb240a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdb240a2530 0x7fdb240a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 10 monclient(hunting): picked mon.noname-i con 0x7fdb240a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdb240a7140 con 0x7fdb240a4920 2024-06-11T19:22:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fdb240a2b50 con 0x7fdb240a2530 2024-06-11T19:22:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fdb240a2e30 con 0x7fdb240a5240 2024-06-11T19:22:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb29620640 10 monclient(hunting): authenticate will time out at 1542.141916s 2024-06-11T19:22:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb227fc640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdb240a5240 msgr2=0x7fdb240a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.334+0000 7fdb227fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdb240a5240 0x7fdb240a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdb240a4920 0x7fdb240a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdb240a4920 0x7fdb240a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52340/0 (socket says 172.21.15.98:52340) 2024-06-11T19:22:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 1 -- 172.21.15.98:0/1718023973 learned_addr learned my addr 172.21.15.98:0/1718023973 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 10 monclient(hunting): get_auth_request con 0x7fdb240a4920 auth_method 0 2024-06-11T19:22:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb237fe640 1 --2- 172.21.15.98:0/1718023973 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdb240a2530 0x7fdb240a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb237fe640 10 monclient(hunting): get_auth_request con 0x7fdb240a2530 auth_method 0 2024-06-11T19:22:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb237fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb237fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 10 monclient(hunting): handle_auth_done global_id 65817 payload 274 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 1 -- 172.21.15.98:0/1718023973 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdb240a5240 msgr2=0x7fdb240a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 1 --2- 172.21.15.98:0/1718023973 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdb240a5240 0x7fdb240a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 1 -- 172.21.15.98:0/1718023973 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdb240a2530 msgr2=0x7fdb240a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 1 --2- 172.21.15.98:0/1718023973 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdb240a2530 0x7fdb240a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 1 monclient: found mon.noname-a 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 1 -- 172.21.15.98:0/1718023973 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb240a3110 con 0x7fdb240a4920 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.335+0000 7fdb22ffd640 1 --2- 172.21.15.98:0/1718023973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb240a4920 0x7fdb240a4d00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009e50 tx=0x7fdb1c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5363f1623f6a8a1c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 1 -- 172.21.15.98:0/1718023973 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb1c02cbd0 con 0x7fdb240a4920 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 10 monclient: dump: 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 10 monclient: _finish_auth 0 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 1 -- 172.21.15.98:0/1718023973 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb1c02cd90 con 0x7fdb240a4920 2024-06-11T19:22:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 5 monclient: authenticate success, global_id 65817 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 1 -- 172.21.15.98:0/1718023973 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb1c034860 con 0x7fdb240a4920 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 4 set_mon_vals no callback set 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb21ffb640 10 monclient: dump: 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.351 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 10 monclient: shutdown 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 1 -- 172.21.15.98:0/1718023973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb240a4920 msgr2=0x7fdb240a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 1 --2- 172.21.15.98:0/1718023973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb240a4920 0x7fdb240a4d00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009e50 tx=0x7fdb1c02bdc0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 1 -- 172.21.15.98:0/1718023973 shutdown_connections 2024-06-11T19:22:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 1 --2- 172.21.15.98:0/1718023973 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdb240a2530 0x7fdb240a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 1 --2- 172.21.15.98:0/1718023973 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdb240a5240 0x7fdb240a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 1 --2- 172.21.15.98:0/1718023973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb240a4920 0x7fdb240a4d00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 1 -- 172.21.15.98:0/1718023973 >> 172.21.15.98:0/1718023973 conn(0x7fdb240ab850 msgr2=0x7fdb240a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 1 -- 172.21.15.98:0/1718023973 shutdown_connections 2024-06-11T19:22:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.336+0000 7fdb29620640 1 -- 172.21.15.98:0/1718023973 wait complete. 2024-06-11T19:22:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.337+0000 7fdb29620640 10 monclient: build_initial_monmap 2024-06-11T19:22:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.337+0000 7fdb29620640 10 monclient: monmap: 2024-06-11T19:22:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.337+0000 7fdb29620640 1 librados: starting msgr at 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.337+0000 7fdb29620640 1 librados: starting objecter 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.337+0000 7fdb29620640 1 Processor -- start 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.337+0000 7fdb29620640 1 -- start start 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.337+0000 7fdb29620640 1 librados: setting wanted keys 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.337+0000 7fdb29620640 1 librados: calling monclient init 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.337+0000 7fdb29620640 10 monclient: init 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 10 monclient: _add_conns ranks=[2,7,6,8,3,4,1,5,0] 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb240a2530 0x7fdb241cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 10 monclient(hunting): picked mon.noname-b con 0x7fdb240a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdb240a4920 0x7fdb241ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 10 monclient(hunting): picked mon.noname-i con 0x7fdb240a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdb240a5240 0x7fdb24122d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 10 monclient(hunting): picked mon.noname-d con 0x7fdb240a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdb240a6da0 con 0x7fdb240a2530 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fdb24138e50 con 0x7fdb240a5240 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fdb240a6420 con 0x7fdb240a4920 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb29620640 10 monclient(hunting): authenticate will time out at 1542.145332s 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb240a2530 0x7fdb241cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb240a2530 0x7fdb241cdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57274/0 (socket says 172.21.15.98:57274) 2024-06-11T19:22:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 1 -- 172.21.15.98:0/1235698240 learned_addr learned my addr 172.21.15.98:0/1235698240 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 10 monclient(hunting): get_auth_request con 0x7fdb240a2530 auth_method 0 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb237fe640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdb240a5240 0x7fdb24122d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb237fe640 10 monclient(hunting): get_auth_request con 0x7fdb240a5240 auth_method 0 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb237fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb237fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb227fc640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdb240a4920 0x7fdb241ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 10 monclient(hunting): handle_auth_done global_id 85034 payload 824 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 1 -- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdb240a4920 msgr2=0x7fdb241ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdb240a4920 0x7fdb241ce1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 1 -- 172.21.15.98:0/1235698240 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdb240a5240 msgr2=0x7fdb24122d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdb240a5240 0x7fdb24122d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 1 monclient: found mon.noname-b 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 1 -- 172.21.15.98:0/1235698240 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb24125500 con 0x7fdb240a2530 2024-06-11T19:22:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb22ffd640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb240a2530 0x7fdb241cdc80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c000f80 tx=0x7fdb1c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb0b7fe640 10 client.?.objecter ms_handle_connect 0x7fdb240a2530 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.338+0000 7fdb0b7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb1c043020 con 0x7fdb240a2530 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 10 monclient: dump: 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb1c02cbe0 con 0x7fdb240a2530 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb29620640 5 monclient: authenticate success, global_id 85034 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb29620640 10 monclient: _renew_subs 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb29620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb29620640 1 -- 172.21.15.98:0/1235698240 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb24126c90 con 0x7fdb240a2530 2024-06-11T19:22:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb1c0046c0 con 0x7fdb240a2530 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb0b7fe640 10 monclient: dump: 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb217fa640 4 set_mon_vals no callback set 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb29620640 10 client.85034.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb29620640 10 monclient: _renew_subs 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb29620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb29620640 1 -- 172.21.15.98:0/1235698240 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb24125250 con 0x7fdb240a2530 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.339+0000 7fdb29620640 1 librados: init done 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb29620640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb29620640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241265f0 0x7fdb24123290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb29620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb29620640 1 -- 172.21.15.98:0/1235698240 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fdb24125720 con 0x7fdb241265f0 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fdb1c046020 con 0x7fdb240a2530 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb227fc640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241265f0 0x7fdb24123290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb227fc640 10 monclient: get_auth_request con 0x7fdb241265f0 auth_method 0 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb227fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb227fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb0b7fe640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdb1003fde0 0x7fdb100422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(517..517 src has 1..517) ==== 6685+0+0 (secure 0 0 0) 0x7fdb1c03a040 con 0x7fdb240a2530 2024-06-11T19:22:12.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb0b7fe640 10 client.85034.objecter ms_dispatch 0x7fdb24000b90 osd_map(517..517 src has 1..517) 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb0b7fe640 3 client.85034.objecter handle_osd_map got epochs [517,517] > 0 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb0b7fe640 3 client.85034.objecter handle_osd_map decoding full epoch 517 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb227fc640 10 monclient(hunting): handle_auth_done global_id 23567 payload 824 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb0b7fe640 20 client.85034.objecter dump_active .. 0 homeless 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb237fe640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdb1003fde0 0x7fdb100422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb227fc640 1 -- 172.21.15.98:0/1235698240 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb240a4d00 con 0x7fdb241265f0 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb237fe640 10 monclient: get_auth_request con 0x7fdb1003fde0 auth_method 0 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb0b7fe640 10 client.85034.objecter ms_handle_connect 0x7fdb241265f0 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb0b7fe640 10 client.85034.objecter resend_mon_ops 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.340+0000 7fdb227fc640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241265f0 0x7fdb24123290 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fdb241262a0 tx=0x7fdb1800c950 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb18003bc0 con 0x7fdb241265f0 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb0b7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:22:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb0b7fe640 10 monclient: dump: 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.366 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb0b7fe640 10 client.85034.objecter ms_handle_connect 0x7fdb1003fde0 2024-06-11T19:22:12.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb237fe640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdb1003fde0 0x7fdb100422a0 secure :-1 s=READY pgs=4260 cs=0 l=1 rev1=1 crypto rx=0x7fdb240aaf40 tx=0x7fdb14009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdb18016070 con 0x7fdb241265f0 2024-06-11T19:22:12.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb0b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:12.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb0b7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:12.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241265f0 msgr2=0x7fdb24123290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb0b7fe640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241265f0 0x7fdb24123290 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fdb241262a0 tx=0x7fdb1800c950 comp rx=0 tx=0).stop 2024-06-11T19:22:12.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241265f0 msgr2=0x7fdb24123290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:12.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.341+0000 7fdb0b7fe640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241265f0 0x7fdb24123290 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.348+0000 7fdb29620640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:22:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.348+0000 7fdb29620640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241237d0 0x7fdb24123bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.348+0000 7fdb29620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.348+0000 7fdb29620640 1 -- 172.21.15.98:0/1235698240 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fdb24050830 con 0x7fdb241237d0 2024-06-11T19:22:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb227fc640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241237d0 0x7fdb24123bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb227fc640 10 monclient: get_auth_request con 0x7fdb241237d0 auth_method 0 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb227fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb227fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb227fc640 10 monclient(hunting): handle_auth_done global_id 23576 payload 824 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb227fc640 1 -- 172.21.15.98:0/1235698240 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7fdb241240f0 con 0x7fdb241237d0 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb227fc640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241237d0 0x7fdb24123bb0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fdb24124430 tx=0x7fdb180059d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb0b7fe640 10 client.85034.objecter ms_handle_connect 0x7fdb241237d0 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb0b7fe640 10 client.85034.objecter resend_mon_ops 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb1802a080 con 0x7fdb241237d0 2024-06-11T19:22:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb0b7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.349+0000 7fdb0b7fe640 10 monclient: dump: 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.370 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.371 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.371 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.371 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.371 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7fdb18005690 con 0x7fdb241237d0 2024-06-11T19:22:12.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb0b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:22:12.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb0b7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:12.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241237d0 msgr2=0x7fdb24123bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb0b7fe640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241237d0 0x7fdb24123bb0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fdb24124430 tx=0x7fdb180059d0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb0b7fe640 1 -- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241237d0 msgr2=0x7fdb24123bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:12.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb0b7fe640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdb241237d0 0x7fdb24123bb0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.371 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-06-11T19:22:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-06-11T19:22:12.372 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:12.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 10 librados: watch_flush enter 2024-06-11T19:22:12.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 10 librados: watch_flush exit 2024-06-11T19:22:12.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 20 client.85034.objecter shutdown clearing up homeless session... 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 10 client.85034.objecter successfully canceled tick 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 1 -- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdb1003fde0 msgr2=0x7fdb100422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdb1003fde0 0x7fdb100422a0 secure :-1 s=READY pgs=4260 cs=0 l=1 rev1=1 crypto rx=0x7fdb240aaf40 tx=0x7fdb14009190 comp rx=0 tx=0).stop 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 10 monclient: shutdown 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 1 -- 172.21.15.98:0/1235698240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb240a2530 msgr2=0x7fdb241cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb240a2530 0x7fdb241cdc80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c000f80 tx=0x7fdb1c004a40 comp rx=0 tx=0).stop 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb237fe640 1 -- 172.21.15.98:0/1235698240 reap_dead start 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 1 -- 172.21.15.98:0/1235698240 shutdown_connections 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 1 --2- 172.21.15.98:0/1235698240 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb240a2530 0x7fdb241cdc80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 1 -- 172.21.15.98:0/1235698240 >> 172.21.15.98:0/1235698240 conn(0x7fdb240ab850 msgr2=0x7fdb241be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 1 -- 172.21.15.98:0/1235698240 shutdown_connections 2024-06-11T19:22:12.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 1 -- 172.21.15.98:0/1235698240 wait complete. 2024-06-11T19:22:12.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.350+0000 7fdb29620640 1 librados: shutdown 2024-06-11T19:22:12.375 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-06-11T19:22:12.375 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-06-11T19:22:12.375 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:22:12.399 INFO:tasks.ceph.mon.h:Stopped 2024-06-11T19:22:12.400 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f 2024-06-11T19:22:12.400 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f store 2024-06-11T19:22:12.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f sync_force --yes-i-really-mean-it 2024-06-11T19:22:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.502+0000 7f46e2cb9640 10 monclient: get_monmap_and_config 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.502+0000 7f46e2cb9640 10 monclient: build_initial_monmap 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.502+0000 7f46e2cb9640 10 monclient: monmap: 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:12.503757+0000 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:12.503757+0000 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:12.503 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 1 Processor -- start 2024-06-11T19:22:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 1 -- start start 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 10 monclient: init 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 10 monclient: _add_conns ranks=[8,3,6,7,5,2,0,1,9,10,4] 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f46dc0a4920 0x7f46dc0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 10 monclient(hunting): picked mon.noname-e con 0x7f46dc0a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46dc0a5240 0x7f46dc0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 10 monclient(hunting): picked mon.noname-g con 0x7f46dc0a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46dc0a2530 0x7f46dc0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 10 monclient(hunting): picked mon.noname-d con 0x7f46dc0a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f46dc0a7140 con 0x7f46dc0a5240 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f46dc0a2b50 con 0x7f46dc0a2530 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f46dc0a2e30 con 0x7f46dc0a4920 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e2cb9640 10 monclient(hunting): authenticate will time out at 1542.310990s 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.503+0000 7f46e0a55640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f46dc0a4920 msgr2=0x7f46dc0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e0a55640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f46dc0a4920 0x7f46dc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46dc0a2530 0x7f46dc0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46dc0a2530 0x7f46dc0a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34368/0 (socket says 172.21.15.98:34368) 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 1 -- 172.21.15.98:0/346931329 learned_addr learned my addr 172.21.15.98:0/346931329 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 10 monclient(hunting): get_auth_request con 0x7f46dc0a2530 auth_method 0 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46dbfff640 1 --2- 172.21.15.98:0/346931329 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46dc0a5240 0x7f46dc0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46dbfff640 10 monclient(hunting): get_auth_request con 0x7f46dc0a5240 auth_method 0 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46dbfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 10 monclient(hunting): handle_auth_done global_id 55653 payload 274 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 1 -- 172.21.15.98:0/346931329 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f46dc0a4920 msgr2=0x7f46dc0a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 1 --2- 172.21.15.98:0/346931329 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f46dc0a4920 0x7f46dc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 1 -- 172.21.15.98:0/346931329 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46dc0a5240 msgr2=0x7f46dc0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 1 --2- 172.21.15.98:0/346931329 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46dc0a5240 0x7f46dc0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 10 monclient: _finish_hunting 0 2024-06-11T19:22:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 1 monclient: found mon.noname-d 2024-06-11T19:22:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:22:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.504+0000 7f46e1256640 1 -- 172.21.15.98:0/346931329 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46dc0a3110 con 0x7f46dc0a2530 2024-06-11T19:22:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e1256640 1 --2- 172.21.15.98:0/346931329 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46dc0a2530 0x7f46dc0a5ee0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f46d4009e80 tx=0x7f46d402bd90 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c5ac56ef5450c956 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 1 -- 172.21.15.98:0/346931329 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46d402cba0 con 0x7f46dc0a2530 2024-06-11T19:22:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:22:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 10 monclient: dump: 2024-06-11T19:22:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 1 -- 172.21.15.98:0/346931329 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46d402cd60 con 0x7f46dc0a2530 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 5 monclient: authenticate success, global_id 55653 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 1 -- 172.21.15.98:0/346931329 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46d4034860 con 0x7f46dc0a2530 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 4 set_mon_vals no callback set 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46db7fe640 10 monclient: dump: 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 10 monclient: shutdown 2024-06-11T19:22:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 1 -- 172.21.15.98:0/346931329 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46dc0a2530 msgr2=0x7f46dc0a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 1 --2- 172.21.15.98:0/346931329 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46dc0a2530 0x7f46dc0a5ee0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f46d4009e80 tx=0x7f46d402bd90 comp rx=0 tx=0).stop 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 1 -- 172.21.15.98:0/346931329 shutdown_connections 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 1 --2- 172.21.15.98:0/346931329 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46dc0a2530 0x7f46dc0a5ee0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 1 --2- 172.21.15.98:0/346931329 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46dc0a5240 0x7f46dc0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 1 --2- 172.21.15.98:0/346931329 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f46dc0a4920 0x7f46dc0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 1 -- 172.21.15.98:0/346931329 >> 172.21.15.98:0/346931329 conn(0x7f46dc0ab850 msgr2=0x7f46dc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 1 -- 172.21.15.98:0/346931329 shutdown_connections 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.505+0000 7f46e2cb9640 1 -- 172.21.15.98:0/346931329 wait complete. 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.506+0000 7f46e2cb9640 10 monclient: build_initial_monmap 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.506+0000 7f46e2cb9640 10 monclient: monmap: 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:12.523 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.506+0000 7f46e2cb9640 1 librados: starting msgr at 2024-06-11T19:22:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.506+0000 7f46e2cb9640 1 librados: starting objecter 2024-06-11T19:22:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.506+0000 7f46e2cb9640 1 Processor -- start 2024-06-11T19:22:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.506+0000 7f46e2cb9640 1 -- start start 2024-06-11T19:22:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.506+0000 7f46e2cb9640 1 librados: setting wanted keys 2024-06-11T19:22:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.506+0000 7f46e2cb9640 1 librados: calling monclient init 2024-06-11T19:22:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.506+0000 7f46e2cb9640 10 monclient: init 2024-06-11T19:22:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 10 monclient: _add_conns ranks=[3,0,5,8,4,7,1,6,2] 2024-06-11T19:22:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46dc0a2530 0x7f46dc1cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 10 monclient(hunting): picked mon.noname-g con 0x7f46dc0a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46dc0a4920 0x7f46dc1ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 10 monclient(hunting): picked mon.noname-a con 0x7f46dc0a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f46dc0a5240 0x7f46dc123fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 10 monclient(hunting): picked mon.noname-h con 0x7f46dc0a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc0a6da0 con 0x7f46dc0a4920 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f46dc138e50 con 0x7f46dc0a2530 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f46dc0a6420 con 0x7f46dc0a5240 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e2cb9640 10 monclient(hunting): authenticate will time out at 1542.314305s 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46dc0a4920 0x7f46dc1ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e1256640 1 -- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f46dc0a5240 msgr2=0x7f46dc123fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46dc0a4920 0x7f46dc1ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52348/0 (socket says 172.21.15.98:52348) 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e1256640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f46dc0a5240 0x7f46dc123fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 1 -- 172.21.15.98:0/2399566950 learned_addr learned my addr 172.21.15.98:0/2399566950 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 10 monclient(hunting): get_auth_request con 0x7f46dc0a4920 auth_method 0 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e0a55640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46dc0a2530 0x7f46dc1cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e0a55640 10 monclient(hunting): get_auth_request con 0x7f46dc0a2530 auth_method 0 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e0a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e0a55640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e0a55640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e0a55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e0a55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46e0a55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 10 monclient(hunting): handle_auth_done global_id 65826 payload 824 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 1 -- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f46dc0a5240 msgr2=0x7f46dc123fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f46dc0a5240 0x7f46dc123fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 1 -- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46dc0a2530 msgr2=0x7f46dc1cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46dc0a2530 0x7f46dc1cdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 1 monclient: found mon.noname-a 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 1 -- 172.21.15.98:0/2399566950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46dc126770 con 0x7f46dc0a4920 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46dbfff640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46dc0a4920 0x7f46dc1ce040 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f46c800d600 tx=0x7f46c800da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46d97fa640 10 client.?.objecter ms_handle_connect 0x7f46dc0a4920 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.507+0000 7f46d97fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46c8012a60 con 0x7f46dc0a4920 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 10 monclient: dump: 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 10 monclient: _finish_auth 0 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 10 monclient: _check_auth_tickets 2024-06-11T19:22:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c8019070 con 0x7f46dc0a4920 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 5 monclient: authenticate success, global_id 65826 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 10 monclient: _renew_subs 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 1 -- 172.21.15.98:0/2399566950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46dc127f00 con 0x7f46dc0a4920 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46c8012c40 con 0x7f46dc0a4920 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46daffd640 4 set_mon_vals no callback set 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 10 monclient: dump: 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.531 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 10 client.65826.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 10 monclient: _renew_subs 2024-06-11T19:22:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 1 -- 172.21.15.98:0/2399566950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46dc0aab10 con 0x7f46dc0a4920 2024-06-11T19:22:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 1 librados: init done 2024-06-11T19:22:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc1278a0 0x7f46dc124cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46e2cb9640 1 -- 172.21.15.98:0/2399566950 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc127350 con 0x7f46dc1278a0 2024-06-11T19:22:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.508+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f46c80125f0 con 0x7f46dc0a4920 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e0a55640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc1278a0 0x7f46dc124cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e0a55640 10 monclient: get_auth_request con 0x7f46dc1278a0 auth_method 0 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e0a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e0a55640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e0a55640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f46c003fde0 0x7f46c00422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e0a55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e0a55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e0a55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(517..517 src has 1..517) ==== 6685+0+0 (secure 0 0 0) 0x7f46c800f040 con 0x7f46dc0a4920 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 10 client.65826.objecter ms_dispatch 0x7f46dc000b90 osd_map(517..517 src has 1..517) 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 3 client.65826.objecter handle_osd_map got epochs [517,517] > 0 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 3 client.65826.objecter handle_osd_map decoding full epoch 517 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 20 client.65826.objecter dump_active .. 0 homeless 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e1256640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f46c003fde0 0x7f46c00422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e0a55640 10 monclient(hunting): handle_auth_done global_id 32779 payload 824 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e0a55640 1 -- 172.21.15.98:0/2399566950 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46dc138760 con 0x7f46dc1278a0 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e1256640 10 monclient: get_auth_request con 0x7f46c003fde0 auth_method 0 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 10 client.65826.objecter ms_handle_connect 0x7f46dc1278a0 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 10 client.65826.objecter resend_mon_ops 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e0a55640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc1278a0 0x7f46dc124cd0 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009610 tx=0x7f46d0009ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46d0016070 con 0x7f46dc1278a0 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 10 monclient: dump: 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46d97fa640 10 client.65826.objecter ms_handle_connect 0x7f46c003fde0 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.509+0000 7f46e1256640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f46c003fde0 0x7f46c00422a0 secure :-1 s=READY pgs=4261 cs=0 l=1 rev1=1 crypto rx=0x7f46d4035000 tx=0x7f46d4030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.510+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f46d0012600 con 0x7f46dc1278a0 2024-06-11T19:22:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.510+0000 7f46d97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.510+0000 7f46d97fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.510+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc1278a0 msgr2=0x7f46dc124cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.510+0000 7f46d97fa640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc1278a0 0x7f46dc124cd0 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009610 tx=0x7f46d0009ad0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.510+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc1278a0 msgr2=0x7f46dc124cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.510+0000 7f46d97fa640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc1278a0 0x7f46dc124cd0 secure :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009610 tx=0x7f46d0009ad0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.517+0000 7f46e2cb9640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:22:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.517+0000 7f46e2cb9640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc02cb40 0x7f46dc02cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.517+0000 7f46e2cb9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.517+0000 7f46e2cb9640 1 -- 172.21.15.98:0/2399566950 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc050830 con 0x7f46dc02cb40 2024-06-11T19:22:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.517+0000 7f46e0a55640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc02cb40 0x7f46dc02cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46e0a55640 10 monclient: get_auth_request con 0x7f46dc02cb40 auth_method 0 2024-06-11T19:22:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46e0a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46e0a55640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46e0a55640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46e0a55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46e0a55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46e0a55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46e0a55640 10 monclient(hunting): handle_auth_done global_id 32788 payload 824 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46e0a55640 1 -- 172.21.15.98:0/2399566950 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f46dc0a6720 con 0x7f46dc02cb40 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46e0a55640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc02cb40 0x7f46dc02cf20 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f46d0000c00 tx=0x7f46d00059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46d97fa640 10 client.65826.objecter ms_handle_connect 0x7f46dc02cb40 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46d97fa640 10 client.65826.objecter resend_mon_ops 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46d0016040 con 0x7f46dc02cb40 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46d97fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.518+0000 7f46d97fa640 10 monclient: dump: 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f46d001ca50 con 0x7f46dc02cb40 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46d97fa640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46d97fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc02cb40 msgr2=0x7f46dc02cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46d97fa640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc02cb40 0x7f46dc02cf20 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f46d0000c00 tx=0x7f46d00059d0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46d97fa640 1 -- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc02cb40 msgr2=0x7f46dc02cf20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46d97fa640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46dc02cb40 0x7f46dc02cf20 secure :-1 s=CLOSED pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f46d0000c00 tx=0x7f46d00059d0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.540 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-06-11T19:22:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-06-11T19:22:12.540 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 10 librados: watch_flush enter 2024-06-11T19:22:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 10 librados: watch_flush exit 2024-06-11T19:22:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 20 client.65826.objecter shutdown clearing up homeless session... 2024-06-11T19:22:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 10 client.65826.objecter successfully canceled tick 2024-06-11T19:22:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 1 -- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f46c003fde0 msgr2=0x7f46c00422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f46c003fde0 0x7f46c00422a0 secure :-1 s=READY pgs=4261 cs=0 l=1 rev1=1 crypto rx=0x7f46d4035000 tx=0x7f46d4030000 comp rx=0 tx=0).stop 2024-06-11T19:22:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 10 monclient: shutdown 2024-06-11T19:22:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 1 -- 172.21.15.98:0/2399566950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46dc0a4920 msgr2=0x7f46dc1ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 1 --2- 172.21.15.98:0/2399566950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46dc0a4920 0x7f46dc1ce040 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f46c800d600 tx=0x7f46c800da00 comp rx=0 tx=0).stop 2024-06-11T19:22:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e1256640 1 -- 172.21.15.98:0/2399566950 reap_dead start 2024-06-11T19:22:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e1256640 1 -- 172.21.15.98:0/2399566950 reap_dead start 2024-06-11T19:22:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 1 -- 172.21.15.98:0/2399566950 shutdown_connections 2024-06-11T19:22:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 1 -- 172.21.15.98:0/2399566950 >> 172.21.15.98:0/2399566950 conn(0x7f46dc0ab850 msgr2=0x7f46dc0a93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 1 -- 172.21.15.98:0/2399566950 shutdown_connections 2024-06-11T19:22:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 1 -- 172.21.15.98:0/2399566950 wait complete. 2024-06-11T19:22:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.519+0000 7f46e2cb9640 1 librados: shutdown 2024-06-11T19:22:12.543 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-06-11T19:22:12.543 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-06-11T19:22:12.543 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:22:12.564 INFO:tasks.ceph.mon.f:Stopped 2024-06-11T19:22:12.564 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-06-11T19:22:12.564 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d store 2024-06-11T19:22:12.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d sync_force --yes-i-really-mean-it 2024-06-11T19:22:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.665+0000 7f46923a1640 10 monclient: get_monmap_and_config 2024-06-11T19:22:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.665+0000 7f46923a1640 10 monclient: build_initial_monmap 2024-06-11T19:22:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.665+0000 7f46923a1640 10 monclient: monmap: 2024-06-11T19:22:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:12.666783+0000 2024-06-11T19:22:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:12.666783+0000 2024-06-11T19:22:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:12.665 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:12.666 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.666+0000 7f46923a1640 1 Processor -- start 2024-06-11T19:22:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.666+0000 7f46923a1640 1 -- start start 2024-06-11T19:22:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.666+0000 7f46923a1640 10 monclient: init 2024-06-11T19:22:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.666+0000 7f46923a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.666+0000 7f46923a1640 10 monclient: _add_conns ranks=[0,8,1,3,6,5,2,7,9,10,4] 2024-06-11T19:22:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.666+0000 7f46923a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f468c0a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.666+0000 7f46923a1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f468c0a2530 0x7f468c13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f46923a1640 10 monclient(hunting): picked mon.noname-e con 0x7f468c0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f46923a1640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f468c13e1a0 addr v1:172.21.15.117:6789/0 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f46923a1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f468c0a7140 con 0x7f468c0a4920 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f46923a1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f468c0a2b50 con 0x7f468c13e1a0 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f46923a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f46923a1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f468c0a2e30 con 0x7f468c0a2530 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f46923a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f46923a1640 10 monclient(hunting): authenticate will time out at 1542.474053s 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468b7fe640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f468c0a2530 msgr2=0x7f468c13dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468b7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f468c0a2530 0x7f468c13dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468bfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f468c0a4920 0x7f468c0a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56888/0 (socket says 172.21.15.98:56888) 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468bfff640 1 -- 172.21.15.98:0/2701868117 learned_addr learned my addr 172.21.15.98:0/2701868117 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f469093e640 1 -- 172.21.15.98:0/2701868117 >> v1:172.21.15.117:6789/0 conn(0x7f468c13e1a0 legacy=0x7f468c140590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 925854543 0 0) 0x7f468c0a7140 con 0x7f468c0a4920 2024-06-11T19:22:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient(hunting): _negotiate 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient(hunting): my global_id is 65835 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4674003520 con 0x7f468c0a4920 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 284586954 0 0) 0x7f4674003520 con 0x7f468c0a4920 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f468c0a2530 msgr2=0x7f468c13dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 --2- 172.21.15.98:0/2701868117 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f468c0a2530 0x7f468c13dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 >> v1:172.21.15.117:6789/0 conn(0x7f468c13e1a0 legacy=0x7f468c140590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient: _finish_hunting 0 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:22:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468c0a3110 con 0x7f468c0a4920 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient: _finish_auth 0 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient: _check_auth_tickets 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f46923a1640 5 monclient: authenticate success, global_id 65835 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7f467c001960 con 0x7f468c0a4920 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f46923a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient: dump: 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 >> v1:172.21.15.98:6789/0 conn(0x7f468c0a4920 legacy=0x7f468c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient: _add_conns ranks=[0,4,3,8,1,6,5,7,2] 2024-06-11T19:22:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 --2- 172.21.15.98:0/2701868117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4674004230 0x7f4674004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient(hunting): picked mon.a con 0x7f4674004230 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 --2- 172.21.15.98:0/2701868117 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4674004f40 0x7f46740099b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient(hunting): picked mon.c con 0x7f4674004f40 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 --2- 172.21.15.98:0/2701868117 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4674009ef0 0x7f467400c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient(hunting): picked mon.g con 0x7f4674009ef0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f468c0a2e30 con 0x7f4674004230 2024-06-11T19:22:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f467400cb50 con 0x7f4674009ef0 2024-06-11T19:22:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f467400cd00 con 0x7f4674004f40 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468affd640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f46923a1640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468bfff640 1 --2- 172.21.15.98:0/2701868117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4674004230 0x7f4674004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468bfff640 10 monclient(hunting): get_auth_request con 0x7f4674004230 auth_method 0 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468b7fe640 1 --2- 172.21.15.98:0/2701868117 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4674004f40 0x7f46740099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468b7fe640 10 monclient(hunting): get_auth_request con 0x7f4674004f40 auth_method 0 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.667+0000 7f468b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 10 monclient(hunting): handle_auth_done global_id 65835 payload 293 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 1 -- 172.21.15.98:0/2701868117 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4674004f40 msgr2=0x7f46740099b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 1 --2- 172.21.15.98:0/2701868117 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4674004f40 0x7f46740099b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 1 -- 172.21.15.98:0/2701868117 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4674009ef0 msgr2=0x7f467400c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 1 --2- 172.21.15.98:0/2701868117 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4674009ef0 0x7f467400c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 1 monclient: found mon.a 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f469093e640 1 -- 172.21.15.98:0/2701868117 reap_dead start 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 1 -- 172.21.15.98:0/2701868117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4674003520 con 0x7f4674004230 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 10 monclient: _finish_auth 0 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468bfff640 1 --2- 172.21.15.98:0/2701868117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4674004230 0x7f4674004800 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f467c007ed0 tx=0x7f4674005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b04f74959fa751d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f467c007af0 con 0x7f4674004230 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.668+0000 7f468affd640 10 monclient: dump: 2024-06-11T19:22:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f467c030070 con 0x7f4674004230 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f46923a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f468affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f468affd640 1 -- 172.21.15.98:0/2701868117 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f467c005d40 con 0x7f4674004230 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f46923a1640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f46923a1640 4 set_mon_vals no callback set 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f468affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f468affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f468affd640 10 monclient: dump: 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f46923a1640 10 monclient: shutdown 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f46923a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f46923a1640 1 -- 172.21.15.98:0/2701868117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4674004230 msgr2=0x7f4674004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f46923a1640 1 --2- 172.21.15.98:0/2701868117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4674004230 0x7f4674004800 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f467c007ed0 tx=0x7f4674005830 comp rx=0 tx=0).stop 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f46923a1640 1 -- 172.21.15.98:0/2701868117 shutdown_connections 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f46923a1640 1 --2- 172.21.15.98:0/2701868117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4674004230 0x7f4674004800 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f46923a1640 1 -- 172.21.15.98:0/2701868117 >> 172.21.15.98:0/2701868117 conn(0x7f468c0ab850 msgr2=0x7f468c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f46923a1640 1 -- 172.21.15.98:0/2701868117 shutdown_connections 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.669+0000 7f46923a1640 1 -- 172.21.15.98:0/2701868117 wait complete. 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.670+0000 7f46923a1640 10 monclient: build_initial_monmap 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.670+0000 7f46923a1640 10 monclient: monmap: 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.670+0000 7f46923a1640 1 librados: starting msgr at 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.670+0000 7f46923a1640 1 librados: starting objecter 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.670+0000 7f46923a1640 1 Processor -- start 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.670+0000 7f46923a1640 1 -- start start 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.670+0000 7f46923a1640 1 librados: setting wanted keys 2024-06-11T19:22:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.670+0000 7f46923a1640 1 librados: calling monclient init 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.670+0000 7f46923a1640 10 monclient: init 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient: _add_conns ranks=[3,0,4,5,8,1,6,2,7] 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4674004230 0x7f468c0a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient(hunting): picked mon.noname-g con 0x7f4674004230 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f468c137bc0 0x7f468c137fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient(hunting): picked mon.noname-a con 0x7f468c137bc0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f468c1215f0 0x7f468c121a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient(hunting): picked mon.noname-c con 0x7f468c1215f0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f468c0a6da0 con 0x7f468c137bc0 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f468c138e50 con 0x7f4674004230 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f468c0a4d00 con 0x7f468c1215f0 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient(hunting): authenticate will time out at 1542.478130s 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f468c137bc0 0x7f468c137fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f469093e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f468c1215f0 0x7f468c121a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f468c137bc0 0x7f468c137fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52366/0 (socket says 172.21.15.98:52366) 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 1 -- 172.21.15.98:0/111350129 learned_addr learned my addr 172.21.15.98:0/111350129 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 10 monclient(hunting): get_auth_request con 0x7f468c137bc0 auth_method 0 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f469093e640 10 monclient(hunting): get_auth_request con 0x7f468c1215f0 auth_method 0 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f469093e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f469093e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f469093e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f469093e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f469093e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f469093e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468bfff640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4674004230 0x7f468c0a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 10 monclient(hunting): handle_auth_done global_id 65844 payload 824 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 1 -- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f468c1215f0 msgr2=0x7f468c121a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f468c1215f0 0x7f468c121a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 1 -- 172.21.15.98:0/111350129 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4674004230 msgr2=0x7f468c0a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4674004230 0x7f468c0a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 1 monclient: found mon.noname-a 2024-06-11T19:22:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 1 -- 172.21.15.98:0/111350129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468c124570 con 0x7f468c137bc0 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468bfff640 10 monclient: get_auth_request con 0x7f4674004230 auth_method 0 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468bfff640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4674004230 0x7f468c0a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f469093e640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f468c1215f0 0x7f468c121a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468b7fe640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f468c137bc0 0x7f468c137fc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f46800096e0 tx=0x7f4680009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 10 client.?.objecter ms_handle_connect 0x7f468c137bc0 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4680016070 con 0x7f468c137bc0 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 10 monclient: dump: 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 10 monclient: _finish_auth 0 2024-06-11T19:22:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 5 monclient: authenticate success, global_id 65844 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f468000ebc0 con 0x7f468c137bc0 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f468000ed70 con 0x7f468c137bc0 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f468a7fc640 4 set_mon_vals no callback set 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f4688ff9640 10 monclient: dump: 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient: _renew_subs 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 1 -- 172.21.15.98:0/111350129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f468c124290 con 0x7f468c137bc0 2024-06-11T19:22:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 client.65844.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient: _renew_subs 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 1 -- 172.21.15.98:0/111350129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f468c122190 con 0x7f468c137bc0 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.671+0000 7f46923a1640 1 librados: init done 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.672+0000 7f46923a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.672+0000 7f46923a1640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09c9e0 0x7f468c09cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.672+0000 7f46923a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.672+0000 7f46923a1640 1 -- 172.21.15.98:0/111350129 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f468c0aa490 con 0x7f468c09c9e0 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.672+0000 7f468bfff640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09c9e0 0x7f468c09cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.672+0000 7f468bfff640 10 monclient: get_auth_request con 0x7f468c09c9e0 auth_method 0 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.672+0000 7f468bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.672+0000 7f468bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.672+0000 7f468bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.672+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f4680019070 con 0x7f468c137bc0 2024-06-11T19:22:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.672+0000 7f468bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.672+0000 7f468bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f468bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f468bfff640 10 monclient(hunting): handle_auth_done global_id 55662 payload 824 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f468bfff640 1 -- 172.21.15.98:0/111350129 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f468c13ded0 con 0x7f468c09c9e0 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f467803fdb0 0x7f4678042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(517..517 src has 1..517) ==== 6685+0+0 (secure 0 0 0) 0x7f468000f040 con 0x7f468c137bc0 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 10 client.65844.objecter ms_dispatch 0x7f468c000b90 osd_map(517..517 src has 1..517) 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 3 client.65844.objecter handle_osd_map got epochs [517,517] > 0 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 3 client.65844.objecter handle_osd_map decoding full epoch 517 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f468bfff640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09c9e0 0x7f468c09cdc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f467c031000 tx=0x7f467c001c50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 20 client.65844.objecter dump_active .. 0 homeless 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 10 client.65844.objecter ms_handle_connect 0x7f468c09c9e0 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 10 client.65844.objecter resend_mon_ops 2024-06-11T19:22:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f469093e640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f467803fdb0 0x7f4678042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f469093e640 10 monclient: get_auth_request con 0x7f467803fdb0 auth_method 0 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f467c046070 con 0x7f468c09c9e0 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 10 monclient: dump: 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f467c030040 con 0x7f468c09c9e0 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09c9e0 msgr2=0x7f468c09cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09c9e0 0x7f468c09cdc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f467c031000 tx=0x7f467c001c50 comp rx=0 tx=0).stop 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09c9e0 msgr2=0x7f468c09cdc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09c9e0 0x7f468c09cdc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f4688ff9640 10 client.65844.objecter ms_handle_connect 0x7f467803fdb0 2024-06-11T19:22:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.673+0000 7f469093e640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f467803fdb0 0x7f4678042270 secure :-1 s=READY pgs=4262 cs=0 l=1 rev1=1 crypto rx=0x7f468c0a2980 tx=0x7f4684009470 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f46923a1640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:22:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f46923a1640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09d300 0x7f468c1d8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f46923a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f46923a1640 1 -- 172.21.15.98:0/111350129 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f468c050830 con 0x7f468c09d300 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f468bfff640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09d300 0x7f468c1d8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f468bfff640 10 monclient: get_auth_request con 0x7f468c09d300 auth_method 0 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f468bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f468bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f468bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f468bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f468bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f468bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f468bfff640 10 monclient(hunting): handle_auth_done global_id 55671 payload 824 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f468bfff640 1 -- 172.21.15.98:0/111350129 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f468c138500 con 0x7f468c09d300 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f4688ff9640 10 client.65844.objecter ms_handle_connect 0x7f468c09d300 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f4688ff9640 10 client.65844.objecter resend_mon_ops 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f468bfff640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09d300 0x7f468c1d8610 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f467c040000 tx=0x7f467c001be0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f467c046040 con 0x7f468c09d300 2024-06-11T19:22:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f4688ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f4688ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.681+0000 7f4688ff9640 10 monclient: dump: 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:12.707 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f467c04c070 con 0x7f468c09d300 2024-06-11T19:22:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f4688ff9640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:22:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f4688ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09d300 msgr2=0x7f468c1d8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f4688ff9640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09d300 0x7f468c1d8610 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f467c040000 tx=0x7f467c001be0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f4688ff9640 1 -- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09d300 msgr2=0x7f468c1d8610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f4688ff9640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f468c09d300 0x7f468c1d8610 secure :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f467c040000 tx=0x7f467c001be0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.708 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-06-11T19:22:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-06-11T19:22:12.708 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 10 librados: watch_flush enter 2024-06-11T19:22:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 10 librados: watch_flush exit 2024-06-11T19:22:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 20 client.65844.objecter shutdown clearing up homeless session... 2024-06-11T19:22:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 10 client.65844.objecter successfully canceled tick 2024-06-11T19:22:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 1 -- 172.21.15.98:0/111350129 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f467803fdb0 msgr2=0x7f4678042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f467803fdb0 0x7f4678042270 secure :-1 s=READY pgs=4262 cs=0 l=1 rev1=1 crypto rx=0x7f468c0a2980 tx=0x7f4684009470 comp rx=0 tx=0).stop 2024-06-11T19:22:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 10 monclient: shutdown 2024-06-11T19:22:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 1 -- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f468c137bc0 msgr2=0x7f468c137fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f468c137bc0 0x7f468c137fc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f46800096e0 tx=0x7f4680009ba0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f469093e640 1 -- 172.21.15.98:0/111350129 reap_dead start 2024-06-11T19:22:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 1 -- 172.21.15.98:0/111350129 shutdown_connections 2024-06-11T19:22:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 1 --2- 172.21.15.98:0/111350129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f468c137bc0 0x7f468c137fc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 1 -- 172.21.15.98:0/111350129 >> 172.21.15.98:0/111350129 conn(0x7f468c0ab850 msgr2=0x7f468c0a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 1 -- 172.21.15.98:0/111350129 shutdown_connections 2024-06-11T19:22:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 1 -- 172.21.15.98:0/111350129 wait complete. 2024-06-11T19:22:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:12.682+0000 7f46923a1640 1 librados: shutdown 2024-06-11T19:22:12.711 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-06-11T19:22:12.711 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-06-11T19:22:12.711 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:22:12.727 INFO:tasks.ceph.mon.d:Stopped 2024-06-11T19:22:12.727 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-06-11T19:22:12.727 INFO:tasks.ceph.mon.a:Sent signal 19 2024-06-11T19:22:12.727 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-06-11T19:22:13.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.d is failed for ~0s 2024-06-11T19:22:19.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.d has been restored 2024-06-11T19:22:27.728 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-06-11T19:22:27.728 INFO:tasks.ceph.mon.a:Sent signal 18 2024-06-11T19:22:27.729 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-06-11T19:22:27.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:22:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.838+0000 7fa8f5620640 10 monclient: get_monmap_and_config 2024-06-11T19:22:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.838+0000 7fa8f5620640 10 monclient: build_initial_monmap 2024-06-11T19:22:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.838+0000 7fa8f5620640 10 monclient: monmap: 2024-06-11T19:22:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:27.839362+0000 2024-06-11T19:22:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:27.839362+0000 2024-06-11T19:22:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:27.841 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 1 Processor -- start 2024-06-11T19:22:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 1 -- start start 2024-06-11T19:22:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 10 monclient: init 2024-06-11T19:22:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 10 monclient: _add_conns ranks=[0,9,5,6,1,7,10,4,3,2,8] 2024-06-11T19:22:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8f00a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 10 monclient(hunting): picked mon.noname-a con 0x7fa8f00a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:22:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8f013e1a0 0x7fa8f0140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 10 monclient(hunting): picked mon.noname-h con 0x7fa8f013e1a0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8f00a7140 con 0x7fa8f00a4920 2024-06-11T19:22:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:27.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa8f00a2b50 con 0x7fa8f00a2530 2024-06-11T19:22:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fa8f00a2e30 con 0x7fa8f013e1a0 2024-06-11T19:22:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8f5620640 10 monclient(hunting): authenticate will time out at 1557.646587s 2024-06-11T19:22:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8ee7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8ef7fe640 1 -- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8f013e1a0 msgr2=0x7fa8f0140590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8eeffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa8f00a4920 0x7fa8f00a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43118/0 (socket says 172.21.15.98:43118) 2024-06-11T19:22:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 learned_addr learned my addr 172.21.15.98:0/3217503350 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8ee7fc640 10 monclient(hunting): get_auth_request con 0x7fa8f00a2530 auth_method 0 2024-06-11T19:22:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8ee7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8f013e1a0 0x7fa8f0140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 msgr2=0x7fa8f013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 msgr2=0x7fa8f013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:27.839+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.040+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8f013e1a0 msgr2=0x7fa8f0140590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.040+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8f013e1a0 0x7fa8f0140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.040+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.040+0000 7fa8ee7fc640 10 monclient(hunting): get_auth_request con 0x7fa8f00a2530 auth_method 0 2024-06-11T19:22:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.040+0000 7fa8ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.040+0000 7fa8ee7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.040+0000 7fa8ee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.040+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 msgr2=0x7fa8f013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:22:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.040+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 msgr2=0x7fa8f013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.040+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:28.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.040+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:28.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.441+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8f013e1a0 msgr2=0x7fa8f0140590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:28.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.441+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8f013e1a0 0x7fa8f0140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.441+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.441+0000 7fa8ee7fc640 10 monclient(hunting): get_auth_request con 0x7fa8f00a2530 auth_method 0 2024-06-11T19:22:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.441+0000 7fa8ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.441+0000 7fa8ee7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.441+0000 7fa8ee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.441+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 msgr2=0x7fa8f013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.441+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 msgr2=0x7fa8f013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.441+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:28.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:28.441+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:29.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:29.242+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8f013e1a0 msgr2=0x7fa8f0140590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:29.242+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8f013e1a0 0x7fa8f0140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:29.242+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:29.242+0000 7fa8ee7fc640 10 monclient(hunting): get_auth_request con 0x7fa8f00a2530 auth_method 0 2024-06-11T19:22:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:29.242+0000 7fa8ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:29.242+0000 7fa8ee7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:29.242+0000 7fa8ee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:29.242+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 msgr2=0x7fa8f013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:29.242+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 msgr2=0x7fa8f013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:29.242+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:29.242+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:30.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 10 monclient(hunting): tick 2024-06-11T19:22:30.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:22:30.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 1 monclient(hunting): continuing hunt 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8f013e1a0 msgr2=0x7fa8f0140590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8f013e1a0 0x7fa8f0140590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 msgr2=0x7fa8f013dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8f00a2530 0x7fa8f013dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 >> v1:172.21.15.98:6789/0 conn(0x7fa8f00a4920 legacy=0x7fa8f00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 10 monclient: _add_conns ranks=[2,6,7,10,1,4,8,3] 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 10 monclient(hunting): picked mon.noname-b con 0x7fa8dc003810 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc0062e0 0x7fa8dc008770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 10 monclient(hunting): picked mon.noname-d con 0x7fa8dc0062e0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 10 monclient(hunting): picked mon.noname-i con 0x7fa8dc008e40 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:30.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f00a2b50 con 0x7fa8dc003810 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa8f00a2e30 con 0x7fa8dc0062e0 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa8dc00ba90 con 0x7fa8dc008e40 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ecff9640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.839+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc0062e0 msgr2=0x7fa8dc008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc0062e0 0x7fa8dc008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request con 0x7fa8dc003810 auth_method 0 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:30.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 msgr2=0x7fa8dc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 msgr2=0x7fa8dc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa8dc008e40 auth_method 0 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 msgr2=0x7fa8dc00b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 msgr2=0x7fa8dc00b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:30.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:30.840+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc0062e0 msgr2=0x7fa8dc008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc0062e0 0x7fa8dc008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request con 0x7fa8dc003810 auth_method 0 2024-06-11T19:22:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8eeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa8dc008e40 auth_method 0 2024-06-11T19:22:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:31.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:31.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 msgr2=0x7fa8dc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:22:31.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 msgr2=0x7fa8dc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:31.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:31.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:31.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 msgr2=0x7fa8dc00b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:31.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 msgr2=0x7fa8dc00b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:31.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:31.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.041+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:31.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc0062e0 msgr2=0x7fa8dc008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:31.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc0062e0 0x7fa8dc008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:31.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:31.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request con 0x7fa8dc003810 auth_method 0 2024-06-11T19:22:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8eeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 msgr2=0x7fa8dc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:22:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 msgr2=0x7fa8dc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa8dc008e40 auth_method 0 2024-06-11T19:22:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:31.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:31.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 msgr2=0x7fa8dc00b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:31.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 msgr2=0x7fa8dc00b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:31.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:31.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:31.442+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:32.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc0062e0 msgr2=0x7fa8dc008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc0062e0 0x7fa8dc008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request con 0x7fa8dc003810 auth_method 0 2024-06-11T19:22:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8eeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 msgr2=0x7fa8dc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:22:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 msgr2=0x7fa8dc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa8dc008e40 auth_method 0 2024-06-11T19:22:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 msgr2=0x7fa8dc00b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 msgr2=0x7fa8dc00b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:32.243+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:33.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 10 monclient(hunting): tick 2024-06-11T19:22:33.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:22:33.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 1 monclient(hunting): continuing hunt 2024-06-11T19:22:33.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:22:33.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 msgr2=0x7fa8dc00b2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:33.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa8dc008e40 0x7fa8dc00b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:33.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc0062e0 msgr2=0x7fa8dc008770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc0062e0 0x7fa8dc008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 msgr2=0x7fa8dc005c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 reap_dead start 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa8dc003810 0x7fa8dc005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 10 monclient: _add_conns ranks=[3,1,4,10,8] 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 10 monclient(hunting): picked mon.noname-g con 0x7fa8dc006760 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa8dc009800 addr v1:172.21.15.117:6789/0 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 10 monclient(hunting): picked mon.noname-c con 0x7fa8dc00b3b0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8f00a3110 con 0x7fa8dc009800 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f00a2b50 con 0x7fa8dc006760 2024-06-11T19:22:33.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa8f00a2e30 con 0x7fa8dc00b3b0 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ecff9640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> v1:172.21.15.117:6789/0 conn(0x7fa8dc009800 legacy=0x7fa8dc009c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa8dc00b3b0 auth_method 0 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 msgr2=0x7fa8dc00ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 msgr2=0x7fa8dc00ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:33.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request con 0x7fa8dc006760 auth_method 0 2024-06-11T19:22:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 msgr2=0x7fa8dc0092c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:22:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 msgr2=0x7fa8dc0092c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:33.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:33.840+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:34.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> v1:172.21.15.117:6789/0 conn(0x7fa8dc009800 legacy=0x7fa8dc009c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-06-11T19:22:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa8dc00b3b0 auth_method 0 2024-06-11T19:22:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 msgr2=0x7fa8dc00ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 msgr2=0x7fa8dc00ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:34.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request con 0x7fa8dc006760 auth_method 0 2024-06-11T19:22:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8eeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 msgr2=0x7fa8dc0092c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:22:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 msgr2=0x7fa8dc0092c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.041+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:34.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> v1:172.21.15.117:6789/0 conn(0x7fa8dc009800 legacy=0x7fa8dc009c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-06-11T19:22:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa8dc00b3b0 auth_method 0 2024-06-11T19:22:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 msgr2=0x7fa8dc00ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:22:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 msgr2=0x7fa8dc00ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request con 0x7fa8dc006760 auth_method 0 2024-06-11T19:22:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:34.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:34.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8eeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:34.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:34.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 msgr2=0x7fa8dc0092c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:22:34.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 msgr2=0x7fa8dc0092c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:34.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:34.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:34.442+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> v1:172.21.15.117:6789/0 conn(0x7fa8dc009800 legacy=0x7fa8dc009c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-06-11T19:22:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa8dc00b3b0 auth_method 0 2024-06-11T19:22:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 msgr2=0x7fa8dc00ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:22:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 msgr2=0x7fa8dc00ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:35.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:35.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:35.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request con 0x7fa8dc006760 auth_method 0 2024-06-11T19:22:35.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:35.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:35.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8eeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 msgr2=0x7fa8dc0092c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:22:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 msgr2=0x7fa8dc0092c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:22:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:22:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:35.243+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 10 monclient(hunting): tick 2024-06-11T19:22:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:22:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 1 monclient(hunting): continuing hunt 2024-06-11T19:22:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:22:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 msgr2=0x7fa8dc00ff90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa8dc00b3b0 0x7fa8dc00ff90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 msgr2=0x7fa8dc0092c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8dc006760 0x7fa8dc0092c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 >> v1:172.21.15.117:6789/0 conn(0x7fa8dc009800 legacy=0x7fa8dc009c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 10 monclient: _add_conns ranks=[10,8] 2024-06-11T19:22:36.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.117:3300/0 conn(0x7fa8dc010660 0x7fa8dc010a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 10 monclient(hunting): picked mon.noname-f con 0x7fa8dc010660 addr v2:172.21.15.117:3300/0 2024-06-11T19:22:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa8dc0110e0 0x7fa8dc015b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 10 monclient(hunting): picked mon.noname-e con 0x7fa8dc0110e0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fa8f00a2b50 con 0x7fa8dc010660 2024-06-11T19:22:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa8f00a2e30 con 0x7fa8dc0110e0 2024-06-11T19:22:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ecff9640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa8dc0110e0 msgr2=0x7fa8dc015b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.117:3300/0 conn(0x7fa8dc010660 msgr2=0x7fa8dc010a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-06-11T19:22:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa8dc0110e0 0x7fa8dc015b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:36.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:36.840+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.117:3300/0 conn(0x7fa8dc010660 0x7fa8dc010a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:37.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:37.041+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa8dc0110e0 msgr2=0x7fa8dc015b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:37.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:37.041+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa8dc0110e0 0x7fa8dc015b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:37.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:37.041+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.117:3300/0 conn(0x7fa8dc010660 msgr2=0x7fa8dc010a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-06-11T19:22:37.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:37.041+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.117:3300/0 conn(0x7fa8dc010660 0x7fa8dc010a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:37.442+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa8dc0110e0 msgr2=0x7fa8dc015b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:37.442+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa8dc0110e0 0x7fa8dc015b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:37.442+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.117:3300/0 conn(0x7fa8dc010660 msgr2=0x7fa8dc010a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-06-11T19:22:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:37.442+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.117:3300/0 conn(0x7fa8dc010660 0x7fa8dc010a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:38.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:38.243+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa8dc0110e0 msgr2=0x7fa8dc015b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:38.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:38.243+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa8dc0110e0 0x7fa8dc015b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:38.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:38.243+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.117:3300/0 conn(0x7fa8dc010660 msgr2=0x7fa8dc010a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-06-11T19:22:38.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:38.243+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.117:3300/0 conn(0x7fa8dc010660 0x7fa8dc010a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:39.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 10 monclient(hunting): tick 2024-06-11T19:22:39.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:22:39.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 1 monclient(hunting): continuing hunt 2024-06-11T19:22:39.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:22:39.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa8dc0110e0 msgr2=0x7fa8dc015b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:39.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa8dc0110e0 0x7fa8dc015b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:39.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 >> v2:172.21.15.117:3300/0 conn(0x7fa8dc010660 msgr2=0x7fa8dc010a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:39.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.117:3300/0 conn(0x7fa8dc010660 0x7fa8dc010a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:39.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 reap_dead start 2024-06-11T19:22:39.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 10 monclient: _add_conns ranks=[1,6,9,7,8,0,10,5,4,3,2] 2024-06-11T19:22:39.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa8dc003810 addr v1:172.21.15.117:6789/0 2024-06-11T19:22:39.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc010660 0x7fa8dc010a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:39.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 10 monclient(hunting): picked mon.noname-d con 0x7fa8dc010660 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:39.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8dc008730 0x7fa8dc005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:39.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 10 monclient(hunting): picked mon.noname-a con 0x7fa8dc008730 addr v2:172.21.15.98:3300/0 2024-06-11T19:22:39.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8dc011780 con 0x7fa8dc003810 2024-06-11T19:22:39.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:39.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa8f00a2b50 con 0x7fa8dc008730 2024-06-11T19:22:39.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:39.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 1 -- 172.21.15.98:0/3217503350 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa8f00a2e30 con 0x7fa8dc010660 2024-06-11T19:22:39.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ecff9640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:39.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ee7fc640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc010660 msgr2=0x7fa8dc010a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:39.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8ee7fc640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc010660 0x7fa8dc010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:39.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.840+0000 7fa8eeffd640 1 -- 172.21.15.98:0/3217503350 >> v1:172.21.15.117:6789/0 conn(0x7fa8dc003810 legacy=0x7fa8dc009220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-06-11T19:22:39.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> v2:172.21.15.98:3300/0 conn(0x7fa8dc008730 0x7fa8dc005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:39.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa8dc008730 auth_method 0 2024-06-11T19:22:39.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:39.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:39.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:39.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:39.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:39.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:39.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 10 monclient(hunting): handle_auth_done global_id 65853 payload 274 2024-06-11T19:22:39.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc010660 msgr2=0x7fa8dc010a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:39.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc010660 0x7fa8dc010a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:39.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 >> v1:172.21.15.117:6789/0 conn(0x7fa8dc003810 legacy=0x7fa8dc009220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:39.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:22:39.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 1 monclient: found mon.noname-a 2024-06-11T19:22:39.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:39.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:39.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.841+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/3217503350 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f00a3110 con 0x7fa8dc008730 2024-06-11T19:22:39.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8dc008730 0x7fa8dc005c00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa8f00a2fb0 tx=0x7fa8e802d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91f96fe8d689bbc server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:39.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 1 -- 172.21.15.98:0/3217503350 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa8e8032a70 con 0x7fa8dc008730 2024-06-11T19:22:39.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:39.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:39.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 10 monclient: dump: 2024-06-11T19:22:39.853 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:39.853 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:39.853 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:39.853 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:39.853 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:39.853 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:39.853 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:39.853 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:39.853 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:39.853 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:39.854 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:39.854 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:39.854 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:39.854 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:39.854 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:39.854 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:39.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 10 monclient: _finish_auth 0 2024-06-11T19:22:39.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 10 monclient: _check_auth_tickets 2024-06-11T19:22:39.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:39.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8f5620640 5 monclient: authenticate success, global_id 65853 2024-06-11T19:22:39.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8f5620640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:39.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 1 -- 172.21.15.98:0/3217503350 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e8039070 con 0x7fa8dc008730 2024-06-11T19:22:39.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:39.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8f5620640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:39.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8f5620640 4 set_mon_vals no callback set 2024-06-11T19:22:39.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 1 -- 172.21.15.98:0/3217503350 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa8e8032c50 con 0x7fa8dc008730 2024-06-11T19:22:39.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:39.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:39.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.842+0000 7fa8edffb640 10 monclient: dump: 2024-06-11T19:22:39.855 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:39.855 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:39.855 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:39.855 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:39.856 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.843+0000 7fa8f5620640 10 monclient: shutdown 2024-06-11T19:22:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.843+0000 7fa8f5620640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.843+0000 7fa8f5620640 1 -- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8dc008730 msgr2=0x7fa8dc005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.843+0000 7fa8f5620640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8dc008730 0x7fa8dc005c00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa8f00a2fb0 tx=0x7fa8e802d9c0 comp rx=0 tx=0).stop 2024-06-11T19:22:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.843+0000 7fa8f5620640 1 -- 172.21.15.98:0/3217503350 shutdown_connections 2024-06-11T19:22:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.843+0000 7fa8f5620640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8dc008730 0x7fa8dc005c00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.843+0000 7fa8f5620640 1 --2- 172.21.15.98:0/3217503350 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa8dc010660 0x7fa8dc010a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.843+0000 7fa8f5620640 1 -- 172.21.15.98:0/3217503350 >> 172.21.15.98:0/3217503350 conn(0x7fa8f00ab850 msgr2=0x7fa8f00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.843+0000 7fa8f5620640 1 -- 172.21.15.98:0/3217503350 shutdown_connections 2024-06-11T19:22:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.843+0000 7fa8f5620640 1 -- 172.21.15.98:0/3217503350 wait complete. 2024-06-11T19:22:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.843+0000 7fa8f5620640 10 monclient: build_initial_monmap 2024-06-11T19:22:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.844+0000 7fa8f5620640 10 monclient: monmap: 2024-06-11T19:22:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:39.859 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.844+0000 7fa8f5620640 1 librados: starting msgr at 2024-06-11T19:22:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.844+0000 7fa8f5620640 1 librados: starting objecter 2024-06-11T19:22:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.844+0000 7fa8f5620640 1 Processor -- start 2024-06-11T19:22:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.844+0000 7fa8f5620640 1 -- start start 2024-06-11T19:22:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.844+0000 7fa8f5620640 1 librados: setting wanted keys 2024-06-11T19:22:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.844+0000 7fa8f5620640 1 librados: calling monclient init 2024-06-11T19:22:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.844+0000 7fa8f5620640 10 monclient: init 2024-06-11T19:22:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 10 monclient: _add_conns ranks=[1,5,0,4,2,6,8,3,7] 2024-06-11T19:22:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8dc003810 0x7fa8f0137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 10 monclient(hunting): picked mon.noname-f con 0x7fa8dc003810 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:22:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8dc008730 0x7fa8f0138150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 10 monclient(hunting): picked mon.noname-h con 0x7fa8dc008730 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8dc010660 0x7fa8f01215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 10 monclient(hunting): picked mon.noname-a con 0x7fa8dc010660 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f00a6da0 con 0x7fa8dc010660 2024-06-11T19:22:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f0138e50 con 0x7fa8dc003810 2024-06-11T19:22:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fa8f00a4d00 con 0x7fa8dc008730 2024-06-11T19:22:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8f5620640 10 monclient(hunting): authenticate will time out at 1569.652163s 2024-06-11T19:22:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8eeffd640 1 -- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8dc003810 msgr2=0x7fa8f0137c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3300/0 2024-06-11T19:22:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8eeffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8dc003810 0x7fa8f0137c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ee7fc640 1 -- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8dc008730 msgr2=0x7fa8f0138150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ee7fc640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8dc008730 0x7fa8f0138150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8dc010660 0x7fa8f01215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8dc010660 0x7fa8f01215f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51474/0 (socket says 172.21.15.98:51474) 2024-06-11T19:22:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/1947159572 learned_addr learned my addr 172.21.15.98:0/1947159572 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa8dc010660 auth_method 0 2024-06-11T19:22:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 10 monclient(hunting): handle_auth_done global_id 65862 payload 824 2024-06-11T19:22:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/1947159572 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8dc008730 msgr2=0x7fa8f0138150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/1947159572 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8dc008730 0x7fa8f0138150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/1947159572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8dc003810 msgr2=0x7fa8f0137c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/1947159572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8dc003810 0x7fa8f0137c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:22:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 1 monclient: found mon.noname-a 2024-06-11T19:22:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 1 -- 172.21.15.98:0/1947159572 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f01ce1e0 con 0x7fa8dc010660 2024-06-11T19:22:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8ef7fe640 1 --2- 172.21.15.98:0/1947159572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8dc010660 0x7fa8f01215f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa8e802da00 tx=0x7fa8e80060c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8d77fe640 10 client.?.objecter ms_handle_connect 0x7fa8dc010660 2024-06-11T19:22:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8d77fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8d77fe640 1 -- 172.21.15.98:0/1947159572 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa8e8031d80 con 0x7fa8dc010660 2024-06-11T19:22:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.845+0000 7fa8d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8d77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8d77fe640 10 monclient: dump: 2024-06-11T19:22:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:39.865 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8d77fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8d77fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8d77fe640 1 -- 172.21.15.98:0/1947159572 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e8048410 con 0x7fa8dc010660 2024-06-11T19:22:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8f5620640 5 monclient: authenticate success, global_id 65862 2024-06-11T19:22:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8f5620640 10 monclient: _renew_subs 2024-06-11T19:22:39.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8f5620640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:39.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8f5620640 1 -- 172.21.15.98:0/1947159572 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f01c3750 con 0x7fa8dc010660 2024-06-11T19:22:39.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8d77fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:39.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8d77fe640 1 -- 172.21.15.98:0/1947159572 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa8e8048a30 con 0x7fa8dc010660 2024-06-11T19:22:39.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:39.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:39.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8d77fe640 10 monclient: dump: 2024-06-11T19:22:39.866 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:39.866 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:39.866 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:39.866 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:39.867 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:39.867 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:39.867 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:39.867 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:39.867 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:39.867 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:39.867 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:39.867 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:39.867 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:39.867 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:39.868 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:39.868 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:39.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8f5620640 10 client.65862.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:39.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8f5620640 10 monclient: _renew_subs 2024-06-11T19:22:39.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8f5620640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:39.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8f5620640 1 -- 172.21.15.98:0/1947159572 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f0122e30 con 0x7fa8dc010660 2024-06-11T19:22:39.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8f5620640 1 librados: init done 2024-06-11T19:22:39.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.846+0000 7fa8ed7fa640 4 set_mon_vals no callback set 2024-06-11T19:22:39.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8d77fe640 1 -- 172.21.15.98:0/1947159572 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fa8e8046020 con 0x7fa8dc010660 2024-06-11T19:22:39.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8f5620640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:39.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8f5620640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:39.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8f5620640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:39.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8f5620640 1 -- 172.21.15.98:0/1947159572 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8f00aa3c0 con 0x7fa8dc010660 2024-06-11T19:22:39.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8d77fe640 1 --2- 172.21.15.98:0/1947159572 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa8dc042120 0x7fa8dc0445e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:39.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8d77fe640 1 -- 172.21.15.98:0/1947159572 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(521..521 src has 1..521) ==== 6360+0+0 (secure 0 0 0) 0x7fa8e8075740 con 0x7fa8dc010660 2024-06-11T19:22:39.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8d77fe640 10 client.65862.objecter ms_dispatch 0x7fa8f0000b90 osd_map(521..521 src has 1..521) 2024-06-11T19:22:39.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8d77fe640 3 client.65862.objecter handle_osd_map got epochs [521,521] > 0 2024-06-11T19:22:39.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8d77fe640 3 client.65862.objecter handle_osd_map decoding full epoch 521 2024-06-11T19:22:39.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8d77fe640 20 client.65862.objecter dump_active .. 0 homeless 2024-06-11T19:22:39.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/1947159572 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa8dc042120 0x7fa8dc0445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:39.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.847+0000 7fa8eeffd640 10 monclient: get_auth_request con 0x7fa8dc042120 auth_method 0 2024-06-11T19:22:39.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.848+0000 7fa8d77fe640 10 client.65862.objecter ms_handle_connect 0x7fa8dc042120 2024-06-11T19:22:39.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.848+0000 7fa8eeffd640 1 --2- 172.21.15.98:0/1947159572 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa8dc042120 0x7fa8dc0445e0 secure :-1 s=READY pgs=4263 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0007fd0 tx=0x7fa8e0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:39.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.852+0000 7fa8d77fe640 1 -- 172.21.15.98:0/1947159572 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa8f01c3750 con 0x7fa8dc010660 2024-06-11T19:22:39.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.852+0000 7fa8d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:39.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:39.852+0000 7fa8d77fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.109+0000 7fa8f5620640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:22:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.109+0000 7fa8f5620640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:22:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.109+0000 7fa8f5620640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.109+0000 7fa8f5620640 1 -- 172.21.15.98:0/1947159572 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa8f00a5000 con 0x7fa8dc010660 2024-06-11T19:22:40.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.109+0000 7fa8d77fe640 1 -- 172.21.15.98:0/1947159572 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3168 (secure 0 0 0) 0x7fa8e8039e20 con 0x7fa8dc010660 2024-06-11T19:22:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.109+0000 7fa8d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:22:40.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.109+0000 7fa8d77fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:40.109 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":76,"quorum":[0,2,3,4,7],"quorum_names":["a","b","g","c","i"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:22:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 10 librados: watch_flush enter 2024-06-11T19:22:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 10 librados: watch_flush exit 2024-06-11T19:22:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 20 client.65862.objecter shutdown clearing up homeless session... 2024-06-11T19:22:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 10 client.65862.objecter successfully canceled tick 2024-06-11T19:22:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 -- 172.21.15.98:0/1947159572 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa8dc042120 msgr2=0x7fa8dc0445e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 --2- 172.21.15.98:0/1947159572 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa8dc042120 0x7fa8dc0445e0 secure :-1 s=READY pgs=4263 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0007fd0 tx=0x7fa8e0009040 comp rx=0 tx=0).stop 2024-06-11T19:22:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 10 monclient: shutdown 2024-06-11T19:22:40.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 -- 172.21.15.98:0/1947159572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8dc010660 msgr2=0x7fa8f01215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 --2- 172.21.15.98:0/1947159572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8dc010660 0x7fa8f01215f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa8e802da00 tx=0x7fa8e80060c0 comp rx=0 tx=0).stop 2024-06-11T19:22:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 -- 172.21.15.98:0/1947159572 shutdown_connections 2024-06-11T19:22:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 --2- 172.21.15.98:0/1947159572 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa8dc042120 0x7fa8dc0445e0 unknown :-1 s=CLOSED pgs=4263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 --2- 172.21.15.98:0/1947159572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8dc010660 0x7fa8f01215f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 --2- 172.21.15.98:0/1947159572 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa8dc008730 0x7fa8f0138150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 --2- 172.21.15.98:0/1947159572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8dc003810 0x7fa8f0137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 -- 172.21.15.98:0/1947159572 >> 172.21.15.98:0/1947159572 conn(0x7fa8f00ab850 msgr2=0x7fa8f00a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 -- 172.21.15.98:0/1947159572 shutdown_connections 2024-06-11T19:22:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 -- 172.21.15.98:0/1947159572 wait complete. 2024-06-11T19:22:40.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.111+0000 7fa8f5620640 1 librados: shutdown 2024-06-11T19:22:40.122 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-06-11T19:22:40.122 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon a 2024-06-11T19:22:40.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:22:40.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.223+0000 7fa7f1af1640 10 monclient: get_monmap_and_config 2024-06-11T19:22:40.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.223+0000 7fa7f1af1640 10 monclient: build_initial_monmap 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.223+0000 7fa7f1af1640 10 monclient: monmap: 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:40.224486+0000 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:40.224486+0000 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:40.223 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:40.224 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:40.224 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:40.224 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:40.224 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:40.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 1 Processor -- start 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 1 -- start start 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 10 monclient: init 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 10 monclient: _add_conns ranks=[8,0,1,3,10,6,4,2,5,7,9] 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec0a4920 0x7fa7ec0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 10 monclient(hunting): picked mon.noname-e con 0x7fa7ec0a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7ec0a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa7ec0a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7ec0a7140 con 0x7fa7ec0a5240 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7ec0a67c0 con 0x7fa7ec0a2530 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa7ec0a2b50 con 0x7fa7ec0a4920 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7f1af1640 10 monclient(hunting): authenticate will time out at 1570.031781s 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7eb7fe640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec0a4920 msgr2=0x7fa7ec0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7eb7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec0a4920 0x7fa7ec0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7ebfff640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7fa7ec0a2530 legacy=0x7fa7ec0a60b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7eaffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa7ec0a5240 0x7fa7ec0a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40846/0 (socket says 172.21.15.98:40846) 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.224+0000 7fa7eaffd640 1 -- 172.21.15.98:0/3638438078 learned_addr learned my addr 172.21.15.98:0/3638438078 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1821215883 0 0) 0x7fa7ec0a7140 con 0x7fa7ec0a5240 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient(hunting): _negotiate 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient(hunting): my global_id is 65871 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7d4003520 con 0x7fa7ec0a5240 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3915627802 0 0) 0x7fa7d4003520 con 0x7fa7ec0a5240 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec0a4920 msgr2=0x7fa7ec0a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec0a4920 0x7fa7ec0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 >> v1:172.21.15.117:6789/0 conn(0x7fa7ec0a2530 legacy=0x7fa7ec0a60b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec0a2e30 con 0x7fa7ec0a5240 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient: _finish_auth 0 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7fa7e0001960 con 0x7fa7ec0a5240 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7f1af1640 5 monclient: authenticate success, global_id 65871 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7f1af1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient: dump: 2024-06-11T19:22:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 >> v1:172.21.15.98:6789/0 conn(0x7fa7ec0a5240 legacy=0x7fa7ec0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient: _add_conns ranks=[7,2,1,6,5,8,3,0,4] 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7d4004230 0x7fa7d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient(hunting): picked mon.i con 0x7fa7d4004230 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d4004f40 0x7fa7d40099b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient(hunting): picked mon.b con 0x7fa7d4004f40 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d4009ef0 0x7fa7d400c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient(hunting): picked mon.f con 0x7fa7d4009ef0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec0a2b50 con 0x7fa7d4009ef0 2024-06-11T19:22:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d400cb50 con 0x7fa7d4004f40 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa7d400cd00 con 0x7fa7d4004230 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ea7fc640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7f1af1640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d4004f40 0x7fa7d40099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ebfff640 1 -- 172.21.15.98:0/3638438078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d4009ef0 msgr2=0x7fa7d400c2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ebfff640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d4009ef0 0x7fa7d400c2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 10 monclient(hunting): get_auth_request con 0x7fa7d4004f40 auth_method 0 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eb7fe640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7d4004230 0x7fa7d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 10 monclient(hunting): handle_auth_done global_id 65871 payload 293 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 1 -- 172.21.15.98:0/3638438078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7d4004230 msgr2=0x7fa7d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7d4004230 0x7fa7d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 1 -- 172.21.15.98:0/3638438078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d4009ef0 msgr2=0x7fa7d400c2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d4009ef0 0x7fa7d400c2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 10 monclient: _finish_hunting 0 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 1 monclient: found mon.b 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7ebfff640 1 -- 172.21.15.98:0/3638438078 reap_dead start 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 1 -- 172.21.15.98:0/3638438078 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d4003520 con 0x7fa7d4004f40 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 10 monclient: _finish_auth 0 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 10 monclient: _check_auth_tickets 2024-06-11T19:22:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eb7fe640 10 monclient: get_auth_request con 0x7fa7d4004230 auth_method 0 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.225+0000 7fa7eb7fe640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7d4004230 0x7fa7d4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7eaffd640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d4004f40 0x7fa7d40099b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec073fc0 tx=0x7fa7d4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1f26b63dd09a891 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7e0007b10 con 0x7fa7d4004f40 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7ea7fc640 10 monclient: dump: 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7f1af1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7e0030070 con 0x7fa7d4004f40 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7ea7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7f1af1640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7f1af1640 4 set_mon_vals no callback set 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7ea7fc640 1 -- 172.21.15.98:0/3638438078 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7e0005d40 con 0x7fa7d4004f40 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.226+0000 7fa7ea7fc640 10 monclient: dump: 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.227+0000 7fa7f1af1640 10 monclient: shutdown 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.227+0000 7fa7f1af1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.227+0000 7fa7f1af1640 1 -- 172.21.15.98:0/3638438078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d4004f40 msgr2=0x7fa7d40099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.227+0000 7fa7f1af1640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d4004f40 0x7fa7d40099b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec073fc0 tx=0x7fa7d4005830 comp rx=0 tx=0).stop 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.227+0000 7fa7f1af1640 1 -- 172.21.15.98:0/3638438078 shutdown_connections 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.227+0000 7fa7f1af1640 1 --2- 172.21.15.98:0/3638438078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d4004f40 0x7fa7d40099b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.227+0000 7fa7f1af1640 1 -- 172.21.15.98:0/3638438078 >> 172.21.15.98:0/3638438078 conn(0x7fa7ec0ab850 msgr2=0x7fa7ec0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.227+0000 7fa7f1af1640 1 -- 172.21.15.98:0/3638438078 shutdown_connections 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.227+0000 7fa7f1af1640 1 -- 172.21.15.98:0/3638438078 wait complete. 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.227+0000 7fa7f1af1640 10 monclient: build_initial_monmap 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.227+0000 7fa7f1af1640 10 monclient: monmap: 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:40.242 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 1 librados: starting msgr at 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 1 librados: starting objecter 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 1 Processor -- start 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 1 -- start start 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 1 librados: setting wanted keys 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 1 librados: calling monclient init 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 10 monclient: init 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 10 monclient: _add_conns ranks=[6,5,8,7,4,1,0,2,3] 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7d4004f40 0x7fa7ec1cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 10 monclient(hunting): picked mon.noname-d con 0x7fa7d4004f40 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa7ec1cdfe0 0x7fa7ec124fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 10 monclient(hunting): picked mon.noname-h con 0x7fa7ec1cdfe0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec1ce3c0 0x7fa7ec1254e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:40.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 10 monclient(hunting): picked mon.noname-e con 0x7fa7ec1ce3c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fa7ec138e50 con 0x7fa7ec1cdfe0 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa7ec0a84b0 con 0x7fa7d4004f40 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa7ec0a6da0 con 0x7fa7ec1ce3c0 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7f1af1640 10 monclient(hunting): authenticate will time out at 1570.035921s 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7eb7fe640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7d4004f40 msgr2=0x7fa7ec1cdaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7eb7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7d4004f40 0x7fa7ec1cdaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7ebfff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec1ce3c0 msgr2=0x7fa7ec1254e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.228+0000 7fa7ebfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec1ce3c0 0x7fa7ec1254e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.229+0000 7fa7eaffd640 1 -- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa7ec1cdfe0 msgr2=0x7fa7ec124fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.229+0000 7fa7eaffd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa7ec1cdfe0 0x7fa7ec124fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.429+0000 7fa7ebfff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec1ce3c0 msgr2=0x7fa7ec1254e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.429+0000 7fa7ebfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec1ce3c0 0x7fa7ec1254e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.429+0000 7fa7eb7fe640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7d4004f40 msgr2=0x7fa7ec1cdaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.429+0000 7fa7eb7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7d4004f40 0x7fa7ec1cdaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.430+0000 7fa7eaffd640 1 -- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa7ec1cdfe0 msgr2=0x7fa7ec124fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.430+0000 7fa7eaffd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa7ec1cdfe0 0x7fa7ec124fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.830+0000 7fa7eb7fe640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7d4004f40 msgr2=0x7fa7ec1cdaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.830+0000 7fa7ebfff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec1ce3c0 msgr2=0x7fa7ec1254e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.830+0000 7fa7ebfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec1ce3c0 0x7fa7ec1254e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.830+0000 7fa7eb7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7d4004f40 0x7fa7ec1cdaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.831+0000 7fa7eaffd640 1 -- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa7ec1cdfe0 msgr2=0x7fa7ec124fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:40.831+0000 7fa7eaffd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa7ec1cdfe0 0x7fa7ec124fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:41.631+0000 7fa7eb7fe640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7d4004f40 msgr2=0x7fa7ec1cdaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:41.631+0000 7fa7ebfff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec1ce3c0 msgr2=0x7fa7ec1254e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:41.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:41.631+0000 7fa7eb7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7d4004f40 0x7fa7ec1cdaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:41.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:41.631+0000 7fa7ebfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec1ce3c0 0x7fa7ec1254e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:41.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:41.632+0000 7fa7eaffd640 1 -- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa7ec1cdfe0 msgr2=0x7fa7ec124fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:41.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:41.632+0000 7fa7eaffd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa7ec1cdfe0 0x7fa7ec124fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 10 monclient(hunting): tick 2024-06-11T19:22:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:22:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 monclient(hunting): continuing hunt 2024-06-11T19:22:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:22:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec1ce3c0 msgr2=0x7fa7ec1254e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7ec1ce3c0 0x7fa7ec1254e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7d4004f40 msgr2=0x7fa7ec1cdaa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7d4004f40 0x7fa7ec1cdaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 -- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa7ec1cdfe0 msgr2=0x7fa7ec124fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa7ec1cdfe0 0x7fa7ec124fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 10 monclient: _add_conns ranks=[4,7,1,2,0,3] 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc003820 0x7fa7cc005c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fa7cc003820 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7cc006230 0x7fa7cc0086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 10 monclient(hunting): picked mon.noname-i con 0x7fa7cc006230 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7cc008d60 0x7fa7cc00b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 10 monclient(hunting): picked mon.noname-f con 0x7fa7cc008d60 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec138e50 con 0x7fa7cc008d60 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa7ec0a84b0 con 0x7fa7cc003820 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa7ec0a6da0 con 0x7fa7cc006230 2024-06-11T19:22:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d2ffd640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc003820 0x7fa7cc005c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc003820 0x7fa7cc005c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45688/0 (socket says 172.21.15.98:45688) 2024-06-11T19:22:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7ebfff640 1 -- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7cc008d60 msgr2=0x7fa7cc00b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3300/0 2024-06-11T19:22:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 1 -- 172.21.15.98:0/1889329493 learned_addr learned my addr 172.21.15.98:0/1889329493 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 10 monclient(hunting): get_auth_request con 0x7fa7cc003820 auth_method 0 2024-06-11T19:22:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7ebfff640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7cc008d60 0x7fa7cc00b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:43.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eaffd640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7cc006230 0x7fa7cc0086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eaffd640 10 monclient(hunting): get_auth_request con 0x7fa7cc006230 auth_method 0 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eaffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 10 monclient(hunting): handle_auth_done global_id 114286 payload 824 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 1 -- 172.21.15.98:0/1889329493 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7cc006230 msgr2=0x7fa7cc0086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7cc006230 0x7fa7cc0086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 1 -- 172.21.15.98:0/1889329493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7cc008d60 msgr2=0x7fa7cc00b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7cc008d60 0x7fa7cc00b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:22:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 1 monclient: found mon.noname-c 2024-06-11T19:22:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7ebfff640 1 -- 172.21.15.98:0/1889329493 reap_dead start 2024-06-11T19:22:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:22:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 1 -- 172.21.15.98:0/1889329493 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec128c80 con 0x7fa7cc003820 2024-06-11T19:22:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7eb7fe640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc003820 0x7fa7cc005c30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0038c0 tx=0x7fa7dc0038f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d3fff640 10 client.?.objecter ms_handle_connect 0x7fa7cc003820 2024-06-11T19:22:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.229+0000 7fa7d3fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7dc003e20 con 0x7fa7cc003820 2024-06-11T19:22:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 10 monclient: dump: 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.239 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 10 monclient: _finish_auth 0 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 10 monclient: _check_auth_tickets 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7dc016070 con 0x7fa7cc003820 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7dc005540 con 0x7fa7cc003820 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7f1af1640 5 monclient: authenticate success, global_id 114286 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7f1af1640 10 monclient: _renew_subs 2024-06-11T19:22:43.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7f1af1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7f1af1640 1 -- 172.21.15.98:0/1889329493 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7ec1277d0 con 0x7fa7cc003820 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7e9ffb640 4 set_mon_vals no callback set 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7d3fff640 10 monclient: dump: 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.242 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7f1af1640 10 client.114286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7f1af1640 10 monclient: _renew_subs 2024-06-11T19:22:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7f1af1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7f1af1640 1 -- 172.21.15.98:0/1889329493 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7ec0aaac0 con 0x7fa7cc003820 2024-06-11T19:22:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.230+0000 7fa7f1af1640 1 librados: init done 2024-06-11T19:22:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fa7dc0056e0 con 0x7fa7cc003820 2024-06-11T19:22:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7f1af1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7f1af1640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec1cdaa0 0x7fa7ec09f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7f1af1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7f1af1640 1 -- 172.21.15.98:0/1889329493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec0a4480 con 0x7fa7ec1cdaa0 2024-06-11T19:22:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7eaffd640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec1cdaa0 0x7fa7ec09f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7eaffd640 10 monclient: get_auth_request con 0x7fa7ec1cdaa0 auth_method 0 2024-06-11T19:22:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7eaffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7d3fff640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa7d803fd50 0x7fa7d8042210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(524..524 src has 1..524) ==== 6746+0+0 (secure 0 0 0) 0x7fa7dc051610 con 0x7fa7cc003820 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7d3fff640 10 client.114286.objecter ms_dispatch 0x7fa7ec000b90 osd_map(524..524 src has 1..524) 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7d3fff640 3 client.114286.objecter handle_osd_map got epochs [524,524] > 0 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7d3fff640 3 client.114286.objecter handle_osd_map decoding full epoch 524 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.231+0000 7fa7d3fff640 20 client.114286.objecter dump_active .. 0 homeless 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7eaffd640 10 monclient(hunting): handle_auth_done global_id 65880 payload 824 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7eaffd640 1 -- 172.21.15.98:0/1889329493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7ec0a4d00 con 0x7fa7ec1cdaa0 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7ebfff640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa7d803fd50 0x7fa7d8042210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7ebfff640 10 monclient: get_auth_request con 0x7fa7d803fd50 auth_method 0 2024-06-11T19:22:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 10 client.114286.objecter ms_handle_connect 0x7fa7ec1cdaa0 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 10 client.114286.objecter resend_mon_ops 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7eaffd640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec1cdaa0 0x7fa7ec09f340 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0031000 tx=0x7fa7e0001be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7e00014b0 con 0x7fa7ec1cdaa0 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 10 monclient: dump: 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.246 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 10 client.114286.objecter ms_handle_connect 0x7fa7d803fd50 2024-06-11T19:22:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7ebfff640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa7d803fd50 0x7fa7d8042210 secure :-1 s=READY pgs=4264 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec1389b0 tx=0x7fa7e4009210 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa7e0030070 con 0x7fa7ec1cdaa0 2024-06-11T19:22:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec1cdaa0 msgr2=0x7fa7ec09f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec1cdaa0 0x7fa7ec09f340 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0031000 tx=0x7fa7e0001be0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec1cdaa0 msgr2=0x7fa7ec09f340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.232+0000 7fa7d3fff640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec1cdaa0 0x7fa7ec09f340 secure :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0031000 tx=0x7fa7e0001be0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.239+0000 7fa7f1af1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.239+0000 7fa7f1af1640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec09f880 0x7fa7ec09fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.239+0000 7fa7f1af1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.239+0000 7fa7f1af1640 1 -- 172.21.15.98:0/1889329493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec050830 con 0x7fa7ec09f880 2024-06-11T19:22:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.239+0000 7fa7eaffd640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec09f880 0x7fa7ec09fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.239+0000 7fa7eaffd640 10 monclient: get_auth_request con 0x7fa7ec09f880 auth_method 0 2024-06-11T19:22:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.239+0000 7fa7eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.239+0000 7fa7eaffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.239+0000 7fa7eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7eaffd640 10 monclient(hunting): handle_auth_done global_id 65889 payload 824 2024-06-11T19:22:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7eaffd640 1 -- 172.21.15.98:0/1889329493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7ec1270c0 con 0x7fa7ec09f880 2024-06-11T19:22:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7eaffd640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec09f880 0x7fa7ec09fc60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0000c00 tx=0x7fa7e0009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 10 client.114286.objecter ms_handle_connect 0x7fa7ec09f880 2024-06-11T19:22:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 10 client.114286.objecter resend_mon_ops 2024-06-11T19:22:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7e0001940 con 0x7fa7ec09f880 2024-06-11T19:22:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 10 monclient: dump: 2024-06-11T19:22:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.254 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9399 (secure 0 0 0) 0x7fa7e00014b0 con 0x7fa7ec09f880 2024-06-11T19:22:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec09f880 msgr2=0x7fa7ec09fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec09f880 0x7fa7ec09fc60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0000c00 tx=0x7fa7e0009030 comp rx=0 tx=0).stop 2024-06-11T19:22:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 1 -- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec09f880 msgr2=0x7fa7ec09fc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.240+0000 7fa7d3fff640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec09f880 0x7fa7ec09fc60 secure :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0000c00 tx=0x7fa7e0009030 comp rx=0 tx=0).stop 2024-06-11T19:22:43.255 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:43.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:43.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:43.255 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:43.255 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-06-11T19:22:43.255 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 555067, 2024-06-11T19:22:43.255 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:43.256 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:43.256 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:43.256 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:43.256 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:43.256 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:22:43.256 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.256 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-06-11T19:22:43.256 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:43.256 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:43.256 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:43.257 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:43.257 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:43.257 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:43.257 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:43.257 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:43.257 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:43.257 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:43.257 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:43.257 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:43.257 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:43.258 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:43.258 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.258 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:43.258 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:43.258 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:43.258 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:43.258 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:43.258 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:43.258 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:43.258 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:43.258 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:43.259 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:43.259 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:43.259 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:43.259 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:43.259 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.259 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:43.259 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:43.259 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:43.259 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:43.259 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:43.260 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:43.260 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:43.260 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:43.260 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:43.260 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:43.260 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:43.260 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:43.260 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:43.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:43.261 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:43.261 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:43.261 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:43.261 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:43.261 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:43.261 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:43.261 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:43.261 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:43.261 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:43.261 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:43.261 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:43.262 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:43.262 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:43.262 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:43.262 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.262 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:43.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.262 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:43.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:43.263 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:43.263 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.263 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.263 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.263 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:43.263 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.263 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.263 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:43.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.264 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.264 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:43.264 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:43.264 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.264 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.264 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:22:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:43.280 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 10 librados: watch_flush enter 2024-06-11T19:22:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 10 librados: watch_flush exit 2024-06-11T19:22:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 20 client.114286.objecter shutdown clearing up homeless session... 2024-06-11T19:22:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 10 client.114286.objecter successfully canceled tick 2024-06-11T19:22:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 -- 172.21.15.98:0/1889329493 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa7d803fd50 msgr2=0x7fa7d8042210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa7d803fd50 0x7fa7d8042210 secure :-1 s=READY pgs=4264 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec1389b0 tx=0x7fa7e4009210 comp rx=0 tx=0).stop 2024-06-11T19:22:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 10 monclient: shutdown 2024-06-11T19:22:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 -- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc003820 msgr2=0x7fa7cc005c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc003820 0x7fa7cc005c30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0038c0 tx=0x7fa7dc0038f0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 -- 172.21.15.98:0/1889329493 shutdown_connections 2024-06-11T19:22:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa7d803fd50 0x7fa7d8042210 unknown :-1 s=CLOSED pgs=4264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc003820 0x7fa7cc005c30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec09f880 0x7fa7ec09fc60 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 --2- 172.21.15.98:0/1889329493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7ec1cdaa0 0x7fa7ec09f340 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 -- 172.21.15.98:0/1889329493 >> 172.21.15.98:0/1889329493 conn(0x7fa7ec0ab850 msgr2=0x7fa7ec09c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:43.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 -- 172.21.15.98:0/1889329493 shutdown_connections 2024-06-11T19:22:43.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 -- 172.21.15.98:0/1889329493 wait complete. 2024-06-11T19:22:43.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.241+0000 7fa7f1af1640 1 librados: shutdown 2024-06-11T19:22:43.284 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-06-11T19:22:43.284 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon b 2024-06-11T19:22:43.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:22:43.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.398+0000 7f3fc80a1640 10 monclient: get_monmap_and_config 2024-06-11T19:22:43.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.398+0000 7f3fc80a1640 10 monclient: build_initial_monmap 2024-06-11T19:22:43.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.399+0000 7f3fc80a1640 10 monclient: monmap: 2024-06-11T19:22:43.398 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:43.398 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.398 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:43.399990+0000 2024-06-11T19:22:43.398 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:43.399990+0000 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:43.399 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.399+0000 7f3fc80a1640 1 Processor -- start 2024-06-11T19:22:43.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.399+0000 7f3fc80a1640 1 -- start start 2024-06-11T19:22:43.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.399+0000 7f3fc80a1640 10 monclient: init 2024-06-11T19:22:43.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:43.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 10 monclient: _add_conns ranks=[0,4,9,5,2,10,6,3,8,7,1] 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3fc00a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc00a2530 0x7f3fc013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 10 monclient(hunting): picked mon.noname-c con 0x7f3fc00a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3fc013e1a0 0x7f3fc0140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 10 monclient(hunting): picked mon.noname-a con 0x7f3fc013e1a0 addr v2:172.21.15.98:3300/0 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fc00a7140 con 0x7f3fc00a4920 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3fc00a2b50 con 0x7f3fc013e1a0 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3fc00a2e30 con 0x7f3fc00a2530 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc80a1640 10 monclient(hunting): authenticate will time out at 1573.207353s 2024-06-11T19:22:43.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3fc013e1a0 0x7f3fc0140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc563c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc00a2530 0x7f3fc013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3fc013e1a0 0x7f3fc0140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43350/0 (socket says 172.21.15.98:43350) 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc563c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc00a2530 0x7f3fc013dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45694/0 (socket says 172.21.15.98:45694) 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 1 -- 172.21.15.98:0/3216361701 learned_addr learned my addr 172.21.15.98:0/3216361701 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 10 monclient(hunting): get_auth_request con 0x7f3fc013e1a0 auth_method 0 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc563c640 10 monclient(hunting): get_auth_request con 0x7f3fc00a2530 auth_method 0 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc563c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc563c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc563c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc563c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc563c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc563c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 10 monclient(hunting): handle_auth_done global_id 65898 payload 274 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 1 -- 172.21.15.98:0/3216361701 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc00a2530 msgr2=0x7f3fc013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 1 --2- 172.21.15.98:0/3216361701 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc00a2530 0x7f3fc013dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 1 -- 172.21.15.98:0/3216361701 >> v1:172.21.15.98:6789/0 conn(0x7f3fc00a4920 legacy=0x7f3fc00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 10 monclient: _finish_hunting 0 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 1 monclient: found mon.noname-a 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc663e640 1 -- 172.21.15.98:0/3216361701 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc00a3110 con 0x7f3fc013e1a0 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc4e3b640 1 -- 172.21.15.98:0/3216361701 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 327955000 0 0) 0x7f3fc00a7140 con 0x7f3fc00a4920 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc4e3b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.400+0000 7f3fc563c640 1 --2- 172.21.15.98:0/3216361701 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc00a2530 0x7f3fc013dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc663e640 1 --2- 172.21.15.98:0/3216361701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc013e1a0 0x7f3fc0140590 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009e50 tx=0x7f3fb802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c2ff8795228557 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 1 -- 172.21.15.98:0/3216361701 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3fb802cc50 con 0x7f3fc013e1a0 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 10 monclient: dump: 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 10 monclient: _finish_auth 0 2024-06-11T19:22:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 10 monclient: _check_auth_tickets 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc80a1640 5 monclient: authenticate success, global_id 65898 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc80a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 1 -- 172.21.15.98:0/3216361701 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fb802ce10 con 0x7f3fc013e1a0 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc80a1640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 1 -- 172.21.15.98:0/3216361701 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3fb8034860 con 0x7f3fc013e1a0 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc80a1640 4 set_mon_vals no callback set 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.401+0000 7f3fc4e3b640 10 monclient: dump: 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.415 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.402+0000 7f3fc80a1640 10 monclient: shutdown 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.402+0000 7f3fc80a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.402+0000 7f3fc80a1640 1 -- 172.21.15.98:0/3216361701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc013e1a0 msgr2=0x7f3fc0140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.402+0000 7f3fc80a1640 1 --2- 172.21.15.98:0/3216361701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc013e1a0 0x7f3fc0140590 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009e50 tx=0x7f3fb802be40 comp rx=0 tx=0).stop 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.402+0000 7f3fc80a1640 1 -- 172.21.15.98:0/3216361701 shutdown_connections 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.402+0000 7f3fc80a1640 1 --2- 172.21.15.98:0/3216361701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fc013e1a0 0x7f3fc0140590 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.402+0000 7f3fc80a1640 1 --2- 172.21.15.98:0/3216361701 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc00a2530 0x7f3fc013dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.402+0000 7f3fc80a1640 1 -- 172.21.15.98:0/3216361701 >> 172.21.15.98:0/3216361701 conn(0x7f3fc00ab850 msgr2=0x7f3fc00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.402+0000 7f3fc80a1640 1 -- 172.21.15.98:0/3216361701 shutdown_connections 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.402+0000 7f3fc80a1640 1 -- 172.21.15.98:0/3216361701 wait complete. 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.402+0000 7f3fc80a1640 10 monclient: build_initial_monmap 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.402+0000 7f3fc80a1640 10 monclient: monmap: 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 1 librados: starting msgr at 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 1 librados: starting objecter 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 1 Processor -- start 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 1 -- start start 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 1 librados: setting wanted keys 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 1 librados: calling monclient init 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 10 monclient: init 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 10 monclient: _add_conns ranks=[7,1,4,0,6,5,2,3,8] 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3fc00a2530 0x7f3fc01cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 10 monclient(hunting): picked mon.noname-i con 0x7f3fc00a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fc00a4920 0x7f3fc0136ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 10 monclient(hunting): picked mon.noname-f con 0x7f3fc00a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:22:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc013e1a0 0x7f3fc0137010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 10 monclient(hunting): picked mon.noname-c con 0x7f3fc013e1a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc0138e50 con 0x7f3fc00a4920 2024-06-11T19:22:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3fc00a84b0 con 0x7f3fc013e1a0 2024-06-11T19:22:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f3fc00a6da0 con 0x7f3fc00a2530 2024-06-11T19:22:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc80a1640 10 monclient(hunting): authenticate will time out at 1573.210835s 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc663e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc013e1a0 0x7f3fc0137010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc663e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc013e1a0 0x7f3fc0137010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45706/0 (socket says 172.21.15.98:45706) 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc663e640 1 -- 172.21.15.98:0/2866949827 learned_addr learned my addr 172.21.15.98:0/2866949827 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc663e640 10 monclient(hunting): get_auth_request con 0x7f3fc013e1a0 auth_method 0 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc663e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc663e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc663e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc563c640 1 -- 172.21.15.98:0/2866949827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fc00a4920 msgr2=0x7f3fc0136ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3300/0 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc663e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc663e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.403+0000 7f3fc563c640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fc00a4920 0x7f3fc0136ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc663e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc663e640 10 monclient(hunting): handle_auth_done global_id 114304 payload 824 2024-06-11T19:22:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc663e640 1 -- 172.21.15.98:0/2866949827 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3fc00a2530 msgr2=0x7f3fc01cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc663e640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3fc00a2530 0x7f3fc01cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc663e640 1 -- 172.21.15.98:0/2866949827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fc00a4920 msgr2=0x7f3fc0136ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc663e640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fc00a4920 0x7f3fc0136ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc663e640 10 monclient: _finish_hunting 0 2024-06-11T19:22:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc663e640 1 monclient: found mon.noname-c 2024-06-11T19:22:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc663e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc663e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:22:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc663e640 1 -- 172.21.15.98:0/2866949827 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc00aa5d0 con 0x7f3fc013e1a0 2024-06-11T19:22:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc663e640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc013e1a0 0x7f3fc0137010 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8036040 tx=0x7f3fb8004a40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 10 client.?.objecter ms_handle_connect 0x7f3fc013e1a0 2024-06-11T19:22:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3fb8043020 con 0x7f3fc013e1a0 2024-06-11T19:22:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:22:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 10 monclient: dump: 2024-06-11T19:22:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.426 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.426 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.426 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 10 monclient: _finish_auth 0 2024-06-11T19:22:43.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 10 monclient: _check_auth_tickets 2024-06-11T19:22:43.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:43.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc80a1640 5 monclient: authenticate success, global_id 114304 2024-06-11T19:22:43.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fb80046c0 con 0x7f3fc013e1a0 2024-06-11T19:22:43.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:43.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3fb8009510 con 0x7f3fc013e1a0 2024-06-11T19:22:43.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc80a1640 10 monclient: _renew_subs 2024-06-11T19:22:43.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc80a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:43.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc80a1640 1 -- 172.21.15.98:0/2866949827 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fc01231a0 con 0x7f3fc013e1a0 2024-06-11T19:22:43.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb7fff640 4 set_mon_vals no callback set 2024-06-11T19:22:43.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:43.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc80a1640 10 client.114304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:43.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fb67fc640 10 monclient: dump: 2024-06-11T19:22:43.427 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.428 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc80a1640 10 monclient: _renew_subs 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc80a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc80a1640 1 -- 172.21.15.98:0/2866949827 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fc0121cf0 con 0x7f3fc013e1a0 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.404+0000 7f3fc80a1640 1 librados: init done 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3fb8046020 con 0x7f3fc013e1a0 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc80a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc80a1640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc009fc60 0x7f3fc00a0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc80a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc80a1640 1 -- 172.21.15.98:0/2866949827 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3fc00a4220 con 0x7f3fc009fc60 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fb67fc640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3fa403fdb0 0x7f3fa4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc5e3d640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc009fc60 0x7f3fc00a0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc5e3d640 10 monclient: get_auth_request con 0x7f3fc009fc60 auth_method 0 2024-06-11T19:22:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc5e3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc5e3d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc5e3d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(525..525 src has 1..525) ==== 6741+0+0 (secure 0 0 0) 0x7f3fb803a040 con 0x7f3fc013e1a0 2024-06-11T19:22:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fb67fc640 10 client.114304.objecter ms_dispatch 0x7f3fc0000b90 osd_map(525..525 src has 1..525) 2024-06-11T19:22:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fb67fc640 3 client.114304.objecter handle_osd_map got epochs [525,525] > 0 2024-06-11T19:22:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fb67fc640 3 client.114304.objecter handle_osd_map decoding full epoch 525 2024-06-11T19:22:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc5e3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc5e3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc5e3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fb67fc640 20 client.114304.objecter dump_active .. 0 homeless 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc5e3d640 10 monclient(hunting): handle_auth_done global_id 85043 payload 824 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc5e3d640 1 -- 172.21.15.98:0/2866949827 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3fc00a4d00 con 0x7f3fc009fc60 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc563c640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3fa403fdb0 0x7f3fa4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fb67fc640 10 client.114304.objecter ms_handle_connect 0x7f3fc009fc60 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc5e3d640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc009fc60 0x7f3fc00a0040 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc007fd0 tx=0x7f3fbc00c780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fb67fc640 10 client.114304.objecter resend_mon_ops 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.405+0000 7f3fc563c640 10 monclient: get_auth_request con 0x7f3fa403fdb0 auth_method 0 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3fbc017070 con 0x7f3fc009fc60 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fb67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fb67fc640 10 monclient: dump: 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3fbc0052e0 con 0x7f3fc009fc60 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fb67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fb67fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc009fc60 msgr2=0x7f3fc00a0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fb67fc640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc009fc60 0x7f3fc00a0040 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc007fd0 tx=0x7f3fbc00c780 comp rx=0 tx=0).stop 2024-06-11T19:22:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc009fc60 msgr2=0x7f3fc00a0040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fb67fc640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc009fc60 0x7f3fc00a0040 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fb67fc640 10 client.114304.objecter ms_handle_connect 0x7f3fa403fdb0 2024-06-11T19:22:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.406+0000 7f3fc563c640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3fa403fdb0 0x7f3fa4042270 secure :-1 s=READY pgs=4265 cs=0 l=1 rev1=1 crypto rx=0x7f3fc00a1640 tx=0x7f3fb0009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.412+0000 7f3fc80a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.412+0000 7f3fc80a1640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc00a0580 0x7f3fc013bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.412+0000 7f3fc80a1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.412+0000 7f3fc80a1640 1 -- 172.21.15.98:0/2866949827 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3fc0050830 con 0x7f3fc00a0580 2024-06-11T19:22:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.412+0000 7f3fc5e3d640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc00a0580 0x7f3fc013bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.412+0000 7f3fc5e3d640 10 monclient: get_auth_request con 0x7f3fc00a0580 auth_method 0 2024-06-11T19:22:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.412+0000 7f3fc5e3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.412+0000 7f3fc5e3d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.412+0000 7f3fc5e3d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.412+0000 7f3fc5e3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.412+0000 7f3fc5e3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.412+0000 7f3fc5e3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc5e3d640 10 monclient(hunting): handle_auth_done global_id 85052 payload 824 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc5e3d640 1 -- 172.21.15.98:0/2866949827 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3fc013de60 con 0x7f3fc00a0580 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc5e3d640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc00a0580 0x7f3fc013bc30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc000fb0 tx=0x7f3fbc000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 10 client.114304.objecter ms_handle_connect 0x7f3fc00a0580 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 10 client.114304.objecter resend_mon_ops 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3fbc017040 con 0x7f3fc00a0580 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 10 monclient: dump: 2024-06-11T19:22:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.437 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9401 (secure 0 0 0) 0x7f3fbc0038f0 con 0x7f3fc00a0580 2024-06-11T19:22:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc00a0580 msgr2=0x7f3fc013bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc00a0580 0x7f3fc013bc30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc000fb0 tx=0x7f3fbc000ed0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 1 -- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc00a0580 msgr2=0x7f3fc013bc30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fb67fc640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3fc00a0580 0x7f3fc013bc30 secure :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc000fb0 tx=0x7f3fbc000ed0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.438 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:43.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:43.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 435187, 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:43.439 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:43.440 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:43.441 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:43.442 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:43.443 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:22:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:43.461 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:43.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc80a1640 10 librados: watch_flush enter 2024-06-11T19:22:43.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc80a1640 10 librados: watch_flush exit 2024-06-11T19:22:43.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc80a1640 20 client.114304.objecter shutdown clearing up homeless session... 2024-06-11T19:22:43.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc80a1640 10 client.114304.objecter successfully canceled tick 2024-06-11T19:22:43.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc80a1640 1 -- 172.21.15.98:0/2866949827 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3fa403fdb0 msgr2=0x7f3fa4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc80a1640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3fa403fdb0 0x7f3fa4042270 secure :-1 s=READY pgs=4265 cs=0 l=1 rev1=1 crypto rx=0x7f3fc00a1640 tx=0x7f3fb0009190 comp rx=0 tx=0).stop 2024-06-11T19:22:43.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc80a1640 10 monclient: shutdown 2024-06-11T19:22:43.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc80a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:43.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc80a1640 1 -- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc013e1a0 msgr2=0x7f3fc0137010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc80a1640 1 --2- 172.21.15.98:0/2866949827 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3fc013e1a0 0x7f3fc0137010 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8036040 tx=0x7f3fb8004a40 comp rx=0 tx=0).stop 2024-06-11T19:22:43.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc663e640 1 -- 172.21.15.98:0/2866949827 reap_dead start 2024-06-11T19:22:43.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.413+0000 7f3fc663e640 1 -- 172.21.15.98:0/2866949827 reap_dead start 2024-06-11T19:22:43.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.414+0000 7f3fc80a1640 1 -- 172.21.15.98:0/2866949827 shutdown_connections 2024-06-11T19:22:43.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.414+0000 7f3fc80a1640 1 -- 172.21.15.98:0/2866949827 >> 172.21.15.98:0/2866949827 conn(0x7f3fc00ab850 msgr2=0x7f3fc00a9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:43.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.414+0000 7f3fc80a1640 1 -- 172.21.15.98:0/2866949827 shutdown_connections 2024-06-11T19:22:43.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.414+0000 7f3fc80a1640 1 -- 172.21.15.98:0/2866949827 wait complete. 2024-06-11T19:22:43.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.414+0000 7f3fc80a1640 1 librados: shutdown 2024-06-11T19:22:43.465 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-06-11T19:22:43.465 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon c 2024-06-11T19:22:43.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:22:43.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.565+0000 7f47eec1e640 10 monclient: get_monmap_and_config 2024-06-11T19:22:43.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.565+0000 7f47eec1e640 10 monclient: build_initial_monmap 2024-06-11T19:22:43.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.565+0000 7f47eec1e640 10 monclient: monmap: 2024-06-11T19:22:43.565 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:43.565 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.565 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:43.566720+0000 2024-06-11T19:22:43.565 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:43.566720+0000 2024-06-11T19:22:43.565 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:43.565 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.565 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:43.565 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:43.565 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:43.566 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:43.566 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:43.566 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:43.566 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:43.566 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:43.566 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:43.566 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:43.566 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:43.566 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 1 Processor -- start 2024-06-11T19:22:43.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 1 -- start start 2024-06-11T19:22:43.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 10 monclient: init 2024-06-11T19:22:43.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:43.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 10 monclient: _add_conns ranks=[10,7,0,8,6,2,9,1,5,4,3] 2024-06-11T19:22:43.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f47e80a4920 0x7f47e80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 10 monclient(hunting): picked mon.noname-f con 0x7f47e80a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:22:43.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f47e80a5240 0x7f47e80a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 10 monclient(hunting): picked mon.noname-i con 0x7f47e80a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47e80a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e80a7140 con 0x7f47e80a2530 2024-06-11T19:22:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f47e80a2b50 con 0x7f47e80a4920 2024-06-11T19:22:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f47e80a2e30 con 0x7f47e80a5240 2024-06-11T19:22:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.566+0000 7f47eec1e640 10 monclient(hunting): authenticate will time out at 1573.373933s 2024-06-11T19:22:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47ec9ba640 1 -- >> v2:172.21.15.117:3300/0 conn(0x7f47e80a4920 msgr2=0x7f47e80a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-06-11T19:22:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47ed1bb640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f47e80a2530 0x7f47e80a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44110/0 (socket says 172.21.15.98:44110) 2024-06-11T19:22:43.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47ed1bb640 1 -- 172.21.15.98:0/671992483 learned_addr learned my addr 172.21.15.98:0/671992483 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47ec9ba640 1 --2- 172.21.15.98:0/671992483 >> v2:172.21.15.117:3300/0 conn(0x7f47e80a4920 0x7f47e80a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f47e80a5240 0x7f47e80a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): get_auth_request con 0x7f47e80a5240 auth_method 0 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1279701805 0 0) 0x7f47e80a7140 con 0x7f47e80a2530 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient(hunting): _negotiate 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient(hunting): my global_id is 65916 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47d8003520 con 0x7f47e80a2530 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 91657074 0 0) 0x7f47d8003520 con 0x7f47e80a2530 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f47e80a5240 msgr2=0x7f47e80a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f47e80a5240 0x7f47e80a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 >> v2:172.21.15.117:3300/0 conn(0x7f47e80a4920 msgr2=0x7f47e80a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 --2- 172.21.15.98:0/671992483 >> v2:172.21.15.117:3300/0 conn(0x7f47e80a4920 0x7f47e80a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient: _finish_hunting 0 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e80a3110 con 0x7f47e80a2530 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:43.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7f47e0001ce0 con 0x7f47e80a2530 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47eec1e640 5 monclient: authenticate success, global_id 65916 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47eec1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient: dump: 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.582 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 >> v1:172.21.15.98:6789/0 conn(0x7f47e80a2530 legacy=0x7f47e80a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient: _add_conns ranks=[5,0,3,1,6,8,2,7,4] 2024-06-11T19:22:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d8004200 0x7f47d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient(hunting): picked mon.h con 0x7f47d8004200 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d8004d90 0x7f47d8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient(hunting): picked mon.a con 0x7f47d8004d90 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47d8009c30 0x7f47d800c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient(hunting): picked mon.g con 0x7f47d8009c30 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47e80a2b50 con 0x7f47d8004d90 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f47e80a2e30 con 0x7f47d8009c30 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f47d800c9d0 con 0x7f47d8004200 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e77fe640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47ec9ba640 1 -- 172.21.15.98:0/671992483 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d8004200 msgr2=0x7f47d8004800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47eec1e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47ec9ba640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d8004200 0x7f47d8004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d8004d90 0x7f47d8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): get_auth_request con 0x7f47d8004d90 auth_method 0 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47ed1bb640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47d8009c30 0x7f47d800c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47ed1bb640 10 monclient(hunting): get_auth_request con 0x7f47d8009c30 auth_method 0 2024-06-11T19:22:43.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47ed1bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47ed1bb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47ed1bb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient(hunting): handle_auth_done global_id 65916 payload 293 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 1 -- 172.21.15.98:0/671992483 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d8004200 msgr2=0x7f47d8004800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d8004200 0x7f47d8004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 1 -- 172.21.15.98:0/671992483 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47d8009c30 msgr2=0x7f47d800c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47d8009c30 0x7f47d800c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 1 monclient: found mon.a 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47ed1bb640 1 -- 172.21.15.98:0/671992483 reap_dead start 2024-06-11T19:22:43.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 1 -- 172.21.15.98:0/671992483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d8003520 con 0x7f47d8004d90 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient: _finish_auth 0 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.567+0000 7f47e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47e7fff640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d8004d90 0x7f47d8005210 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f47e80a5620 tx=0x7f47d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f621e3c1fd5beaaf server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47d4039070 con 0x7f47d8004d90 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47e77fe640 10 monclient: dump: 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.587 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d402caa0 con 0x7f47d8004d90 2024-06-11T19:22:43.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47eec1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47e77fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47e77fe640 1 -- 172.21.15.98:0/671992483 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47d4009690 con 0x7f47d8004d90 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47eec1e640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47eec1e640 4 set_mon_vals no callback set 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.568+0000 7f47e77fe640 10 monclient: dump: 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.589 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.569+0000 7f47eec1e640 10 monclient: shutdown 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.569+0000 7f47eec1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.569+0000 7f47eec1e640 1 -- 172.21.15.98:0/671992483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d8004d90 msgr2=0x7f47d8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.569+0000 7f47eec1e640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d8004d90 0x7f47d8005210 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f47e80a5620 tx=0x7f47d8005830 comp rx=0 tx=0).stop 2024-06-11T19:22:43.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.569+0000 7f47eec1e640 1 -- 172.21.15.98:0/671992483 shutdown_connections 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.569+0000 7f47eec1e640 1 --2- 172.21.15.98:0/671992483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47d8004d90 0x7f47d8005210 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.569+0000 7f47eec1e640 1 -- 172.21.15.98:0/671992483 >> 172.21.15.98:0/671992483 conn(0x7f47e80ab850 msgr2=0x7f47e80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.569+0000 7f47eec1e640 1 -- 172.21.15.98:0/671992483 shutdown_connections 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.569+0000 7f47eec1e640 1 -- 172.21.15.98:0/671992483 wait complete. 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.569+0000 7f47eec1e640 10 monclient: build_initial_monmap 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.569+0000 7f47eec1e640 10 monclient: monmap: 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:43.591 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 1 librados: starting msgr at 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 1 librados: starting objecter 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 1 Processor -- start 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 1 -- start start 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 1 librados: setting wanted keys 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 1 librados: calling monclient init 2024-06-11T19:22:43.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 10 monclient: init 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 10 monclient: _add_conns ranks=[6,0,7,5,2,8,4,3,1] 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f47d8004d90 0x7f47e81cde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 10 monclient(hunting): picked mon.noname-d con 0x7f47d8004d90 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81ce3d0 0x7f47e8124100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 10 monclient(hunting): picked mon.noname-a con 0x7f47e81ce3d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.570+0000 7f47eec1e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f47e8124640 0x7f47e8124ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 10 monclient(hunting): picked mon.noname-i con 0x7f47e8124640 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47e80a6da0 con 0x7f47e81ce3d0 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f47e8138e50 con 0x7f47d8004d90 2024-06-11T19:22:43.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f47e80a4d00 con 0x7f47e8124640 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47ec9ba640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f47d8004d90 msgr2=0x7f47e81cde90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 10 monclient(hunting): authenticate will time out at 1573.378027s 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47ec9ba640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f47d8004d90 0x7f47e81cde90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81ce3d0 0x7f47e8124100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81ce3d0 0x7f47e8124100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43374/0 (socket says 172.21.15.98:43374) 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 1 -- 172.21.15.98:0/3880473325 learned_addr learned my addr 172.21.15.98:0/3880473325 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 10 monclient(hunting): get_auth_request con 0x7f47e81ce3d0 auth_method 0 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47ed1bb640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f47e8124640 0x7f47e8124ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47ed1bb640 10 monclient(hunting): get_auth_request con 0x7f47e8124640 auth_method 0 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47ed1bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47ed1bb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47ed1bb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47ed1bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47ed1bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47ed1bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 10 monclient(hunting): handle_auth_done global_id 65925 payload 824 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 1 -- 172.21.15.98:0/3880473325 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f47e8124640 msgr2=0x7f47e8124ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f47e8124640 0x7f47e8124ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 1 -- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f47d8004d90 msgr2=0x7f47e81cde90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:43.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f47d8004d90 0x7f47e81cde90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 10 monclient: _finish_hunting 0 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 1 monclient: found mon.noname-a 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 1 -- 172.21.15.98:0/3880473325 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e8127b00 con 0x7f47e81ce3d0 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e7fff640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81ce3d0 0x7f47e8124100 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f47d4037040 tx=0x7f47d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 10 client.?.objecter ms_handle_connect 0x7f47e81ce3d0 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47d4042020 con 0x7f47e81ce3d0 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 10 monclient: dump: 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.596 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.597 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 10 monclient: _finish_auth 0 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 10 monclient: _check_auth_tickets 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d403a040 con 0x7f47e81ce3d0 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 5 monclient: authenticate success, global_id 65925 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 10 monclient: _renew_subs 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 1 -- 172.21.15.98:0/3880473325 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e8126270 con 0x7f47e81ce3d0 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47d4039070 con 0x7f47e81ce3d0 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:43.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e57fa640 10 monclient: dump: 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.599 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 10 client.65925.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47e6ffd640 4 set_mon_vals no callback set 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 10 monclient: _renew_subs 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 1 -- 172.21.15.98:0/3880473325 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e8127de0 con 0x7f47e81ce3d0 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.571+0000 7f47eec1e640 1 librados: init done 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47eec1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47eec1e640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e802dcc0 0x7f47e8125000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47eec1e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47eec1e640 1 -- 172.21.15.98:0/3880473325 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f47e8128af0 con 0x7f47e802dcc0 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f47d4045020 con 0x7f47e81ce3d0 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47ec9ba640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e802dcc0 0x7f47e8125000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47ec9ba640 10 monclient: get_auth_request con 0x7f47e802dcc0 auth_method 0 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47ec9ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47ec9ba640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47ec9ba640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47ec9ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47ec9ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47ec9ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47e57fa640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f47cc03fdb0 0x7f47cc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47ec9ba640 10 monclient(hunting): handle_auth_done global_id 114313 payload 824 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(525..525 src has 1..525) ==== 6741+0+0 (secure 0 0 0) 0x7f47d400f040 con 0x7f47e81ce3d0 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47e57fa640 10 client.65925.objecter ms_dispatch 0x7f47e8000b90 osd_map(525..525 src has 1..525) 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47e57fa640 3 client.65925.objecter handle_osd_map got epochs [525,525] > 0 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47ec9ba640 1 -- 172.21.15.98:0/3880473325 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f47e802e0a0 con 0x7f47e802dcc0 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.572+0000 7f47e57fa640 3 client.65925.objecter handle_osd_map decoding full epoch 525 2024-06-11T19:22:43.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47ed1bb640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f47cc03fdb0 0x7f47cc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 20 client.65925.objecter dump_active .. 0 homeless 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47ed1bb640 10 monclient: get_auth_request con 0x7f47cc03fdb0 auth_method 0 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 10 client.65925.objecter ms_handle_connect 0x7f47e802dcc0 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47ec9ba640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e802dcc0 0x7f47e8125000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00b450 tx=0x7f47dc00b910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 10 client.65925.objecter resend_mon_ops 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47dc003c00 con 0x7f47e802dcc0 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 10 monclient: dump: 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.602 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f47dc015070 con 0x7f47e802dcc0 2024-06-11T19:22:43.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e802dcc0 msgr2=0x7f47e8125000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e802dcc0 0x7f47e8125000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00b450 tx=0x7f47dc00b910 comp rx=0 tx=0).stop 2024-06-11T19:22:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e802dcc0 msgr2=0x7f47e8125000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e802dcc0 0x7f47e8125000 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00b450 tx=0x7f47dc00b910 comp rx=0 tx=0).stop 2024-06-11T19:22:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47e57fa640 10 client.65925.objecter ms_handle_connect 0x7f47cc03fdb0 2024-06-11T19:22:43.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.573+0000 7f47ed1bb640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f47cc03fdb0 0x7f47cc042270 secure :-1 s=READY pgs=4266 cs=0 l=1 rev1=1 crypto rx=0x7f47e80a8420 tx=0x7f47e0001460 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.579+0000 7f47eec1e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:43.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.579+0000 7f47eec1e640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e8063180 0x7f47e8063590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.579+0000 7f47eec1e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.579+0000 7f47eec1e640 1 -- 172.21.15.98:0/3880473325 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f47e8050830 con 0x7f47e8063180 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.579+0000 7f47ec9ba640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e8063180 0x7f47e8063590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47ec9ba640 10 monclient: get_auth_request con 0x7f47e8063180 auth_method 0 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47ec9ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47ec9ba640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47ec9ba640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47ec9ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47ec9ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47ec9ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47ec9ba640 10 monclient(hunting): handle_auth_done global_id 114322 payload 824 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47ec9ba640 1 -- 172.21.15.98:0/3880473325 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f47e80a1f40 con 0x7f47e8063180 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 10 client.65925.objecter ms_handle_connect 0x7f47e8063180 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 10 client.65925.objecter resend_mon_ops 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47ec9ba640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e8063180 0x7f47e8063590 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00bfd0 tx=0x7f47dc005a10 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47dc0057b0 con 0x7f47e8063180 2024-06-11T19:22:43.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 10 monclient: dump: 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.608 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.609 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.609 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.609 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.609 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9228 (secure 0 0 0) 0x7f47dc005340 con 0x7f47e8063180 2024-06-11T19:22:43.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:43.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:43.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e8063180 msgr2=0x7f47e8063590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e8063180 0x7f47e8063590 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00bfd0 tx=0x7f47dc005a10 comp rx=0 tx=0).stop 2024-06-11T19:22:43.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 1 -- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e8063180 msgr2=0x7f47e8063590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:43.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.580+0000 7f47e57fa640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47e8063180 0x7f47e8063590 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.610 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:43.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:43.610 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:43.610 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:43.610 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-06-11T19:22:43.610 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 51946, 2024-06-11T19:22:43.610 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:43.610 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:43.610 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:43.610 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:43.611 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:43.612 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:43.612 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:43.612 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:43.612 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:43.612 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.612 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:43.612 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:43.612 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:43.612 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:43.613 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:43.614 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:43.615 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:43.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:43.619 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:43.620 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:43.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.622 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:43.624 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:43.628 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:43.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:43.631 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:43.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.632 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.632 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:43.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.632 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:43.632 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:43.632 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:43.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.632 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.632 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:43.632 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 10 librados: watch_flush enter 2024-06-11T19:22:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 10 librados: watch_flush exit 2024-06-11T19:22:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 20 client.65925.objecter shutdown clearing up homeless session... 2024-06-11T19:22:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 10 client.65925.objecter successfully canceled tick 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 1 -- 172.21.15.98:0/3880473325 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f47cc03fdb0 msgr2=0x7f47cc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f47cc03fdb0 0x7f47cc042270 secure :-1 s=READY pgs=4266 cs=0 l=1 rev1=1 crypto rx=0x7f47e80a8420 tx=0x7f47e0001460 comp rx=0 tx=0).stop 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 10 monclient: shutdown 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 1 -- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81ce3d0 msgr2=0x7f47e8124100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 1 --2- 172.21.15.98:0/3880473325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81ce3d0 0x7f47e8124100 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f47d4037040 tx=0x7f47d4004730 comp rx=0 tx=0).stop 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47ed1bb640 1 -- 172.21.15.98:0/3880473325 reap_dead start 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47ed1bb640 1 -- 172.21.15.98:0/3880473325 reap_dead start 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 1 -- 172.21.15.98:0/3880473325 shutdown_connections 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 1 -- 172.21.15.98:0/3880473325 >> 172.21.15.98:0/3880473325 conn(0x7f47e80ab850 msgr2=0x7f47e8137fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 1 -- 172.21.15.98:0/3880473325 shutdown_connections 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 1 -- 172.21.15.98:0/3880473325 wait complete. 2024-06-11T19:22:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.581+0000 7f47eec1e640 1 librados: shutdown 2024-06-11T19:22:43.636 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-06-11T19:22:43.636 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon g 2024-06-11T19:22:43.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:22:43.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.736+0000 7f1658d8c640 10 monclient: get_monmap_and_config 2024-06-11T19:22:43.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.736+0000 7f1658d8c640 10 monclient: build_initial_monmap 2024-06-11T19:22:43.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.736+0000 7f1658d8c640 10 monclient: monmap: 2024-06-11T19:22:43.736 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:43.736 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.736 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:43.737793+0000 2024-06-11T19:22:43.736 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:43.737793+0000 2024-06-11T19:22:43.736 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:43.736 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.736 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:43.736 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:43.737 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:43.737 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:43.737 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:43.737 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:43.737 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:43.737 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:43.737 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:43.737 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:43.737 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:43.737 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.737+0000 7f1658d8c640 1 Processor -- start 2024-06-11T19:22:43.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.737+0000 7f1658d8c640 1 -- start start 2024-06-11T19:22:43.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.737+0000 7f1658d8c640 10 monclient: init 2024-06-11T19:22:43.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.737+0000 7f1658d8c640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:43.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.737+0000 7f1658d8c640 10 monclient: _add_conns ranks=[2,5,3,6,10,7,4,9,0,8,1] 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.737+0000 7f1658d8c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16541347f0 0x7f165409e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.737+0000 7f1658d8c640 10 monclient(hunting): picked mon.noname-b con 0x7f16541347f0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.737+0000 7f1658d8c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1654134bd0 0x7f16540a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.737+0000 7f1658d8c640 10 monclient(hunting): picked mon.noname-h con 0x7f1654134bd0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1658d8c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1654135010 0x7f16540a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1658d8c640 10 monclient(hunting): picked mon.noname-g con 0x7f1654135010 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1658d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1658d8c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f16540687f0 con 0x7f16541347f0 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1658d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1658d8c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f165409d410 con 0x7f1654135010 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1658d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1658d8c640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f165409d6f0 con 0x7f1654134bd0 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1658d8c640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:43.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1658d8c640 10 monclient(hunting): authenticate will time out at 1573.545042s 2024-06-11T19:22:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f164bfff640 1 -- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1654134bd0 msgr2=0x7f16540a3690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16541347f0 0x7f165409e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f164bfff640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1654134bd0 0x7f16540a3690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16541347f0 0x7f165409e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:54210/0 (socket says 172.21.15.98:54210) 2024-06-11T19:22:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 1 -- 172.21.15.98:0/3020444547 learned_addr learned my addr 172.21.15.98:0/3020444547 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 10 monclient(hunting): get_auth_request con 0x7f16541347f0 auth_method 0 2024-06-11T19:22:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1652d9d640 1 --2- 172.21.15.98:0/3020444547 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1654135010 0x7f16540a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1652d9d640 10 monclient(hunting): get_auth_request con 0x7f1654135010 auth_method 0 2024-06-11T19:22:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1652d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1652d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f1652d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 10 monclient(hunting): handle_auth_done global_id 85061 payload 274 2024-06-11T19:22:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 1 -- 172.21.15.98:0/3020444547 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1654134bd0 msgr2=0x7f16540a3690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 1 --2- 172.21.15.98:0/3020444547 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1654134bd0 0x7f16540a3690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 1 -- 172.21.15.98:0/3020444547 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1654135010 msgr2=0x7f16540a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:22:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 1 --2- 172.21.15.98:0/3020444547 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1654135010 0x7f16540a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 10 monclient: _finish_hunting 0 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 1 monclient: found mon.noname-b 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.738+0000 7f165259c640 1 -- 172.21.15.98:0/3020444547 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165409d9d0 con 0x7f16541347f0 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f165259c640 1 --2- 172.21.15.98:0/3020444547 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16541347f0 0x7f165409e6b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f16440098a0 tx=0x7f164402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fe5e474dc5d51c9 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 1 -- 172.21.15.98:0/3020444547 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1644031a30 con 0x7f16541347f0 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 10 monclient: dump: 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.751 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 10 monclient: _finish_auth 0 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:22:43.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 1 -- 172.21.15.98:0/3020444547 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1644037070 con 0x7f16541347f0 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 5 monclient: authenticate success, global_id 85061 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 1 -- 172.21.15.98:0/3020444547 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1644031c30 con 0x7f16541347f0 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 4 set_mon_vals no callback set 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1651d9b640 10 monclient: dump: 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.753 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 10 monclient: shutdown 2024-06-11T19:22:43.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 1 -- 172.21.15.98:0/3020444547 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16541347f0 msgr2=0x7f165409e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 1 --2- 172.21.15.98:0/3020444547 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16541347f0 0x7f165409e6b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f16440098a0 tx=0x7f164402b860 comp rx=0 tx=0).stop 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 1 -- 172.21.15.98:0/3020444547 shutdown_connections 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 1 --2- 172.21.15.98:0/3020444547 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1654135010 0x7f16540a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 1 --2- 172.21.15.98:0/3020444547 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1654134bd0 0x7f16540a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 1 --2- 172.21.15.98:0/3020444547 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16541347f0 0x7f165409e6b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 1 -- 172.21.15.98:0/3020444547 >> 172.21.15.98:0/3020444547 conn(0x7f16540a4780 msgr2=0x7f16540a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 1 -- 172.21.15.98:0/3020444547 shutdown_connections 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.739+0000 7f1658d8c640 1 -- 172.21.15.98:0/3020444547 wait complete. 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.740+0000 7f1658d8c640 10 monclient: build_initial_monmap 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.740+0000 7f1658d8c640 10 monclient: monmap: 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:43.755 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:43.756 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.740+0000 7f1658d8c640 1 librados: starting msgr at 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.740+0000 7f1658d8c640 1 librados: starting objecter 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.740+0000 7f1658d8c640 1 Processor -- start 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 1 -- start start 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 1 librados: setting wanted keys 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 1 librados: calling monclient init 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 10 monclient: init 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 10 monclient: _add_conns ranks=[4,0,3,7,6,5,8,1,2] 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16541347f0 0x7f1654133270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 10 monclient(hunting): picked mon.noname-c con 0x7f16541347f0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654134bd0 0x7f16541337b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 10 monclient(hunting): picked mon.noname-a con 0x7f1654134bd0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:22:43.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1654135010 0x7f165411ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 10 monclient(hunting): picked mon.noname-g con 0x7f1654135010 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1654068c40 con 0x7f1654134bd0 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f1654068ac0 con 0x7f1654135010 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f16540a4110 con 0x7f16541347f0 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1658d8c640 10 monclient(hunting): authenticate will time out at 1573.548530s 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16541347f0 0x7f1654133270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16541347f0 0x7f1654133270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45730/0 (socket says 172.21.15.98:45730) 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f164bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654134bd0 0x7f16541337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 1 -- 172.21.15.98:0/1281952264 learned_addr learned my addr 172.21.15.98:0/1281952264 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 10 monclient(hunting): get_auth_request con 0x7f16541347f0 auth_method 0 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f164bfff640 10 monclient(hunting): get_auth_request con 0x7f1654134bd0 auth_method 0 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f164bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f164bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f164bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1652d9d640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1654135010 0x7f165411ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f164bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f164bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f164bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1652d9d640 10 monclient(hunting): get_auth_request con 0x7f1654135010 auth_method 0 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1652d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1652d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1652d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1652d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1652d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f1652d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 10 monclient(hunting): handle_auth_done global_id 114331 payload 824 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 1 -- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1654135010 msgr2=0x7f165411ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1654135010 0x7f165411ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 1 -- 172.21.15.98:0/1281952264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654134bd0 msgr2=0x7f16541337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654134bd0 0x7f16541337b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 10 monclient: _finish_hunting 0 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 1 monclient: found mon.noname-c 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f165259c640 1 -- 172.21.15.98:0/1281952264 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16540a3630 con 0x7f16541347f0 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.741+0000 7f164bfff640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654134bd0 0x7f16541337b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f165259c640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16541347f0 0x7f1654133270 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f16440097c0 tx=0x7f16440046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 10 client.?.objecter ms_handle_connect 0x7f16541347f0 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1644041070 con 0x7f16541347f0 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 10 monclient: dump: 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.762 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 10 monclient: _finish_auth 0 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 10 monclient: _check_auth_tickets 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f1658d8c640 5 monclient: authenticate success, global_id 114331 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1644003c50 con 0x7f16541347f0 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f1658d8c640 10 monclient: _renew_subs 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f1658d8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f1658d8c640 1 -- 172.21.15.98:0/1281952264 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16541360e0 con 0x7f16541347f0 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16440044e0 con 0x7f16541347f0 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f1658d8c640 10 client.114331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:43.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f165159a640 4 set_mon_vals no callback set 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f164affd640 10 monclient: dump: 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f1658d8c640 10 monclient: _renew_subs 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f1658d8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f1658d8c640 1 -- 172.21.15.98:0/1281952264 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f1654122c50 con 0x7f16541347f0 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.742+0000 7f1658d8c640 1 librados: init done 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f1658d8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f1658d8c640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411f1d0 0x7f165411f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f1658d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f1658d8c640 1 -- 172.21.15.98:0/1281952264 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f16541225f0 con 0x7f165411f1d0 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f164404c060 con 0x7f16541347f0 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164bfff640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411f1d0 0x7f165411f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164bfff640 10 monclient: get_auth_request con 0x7f165411f1d0 auth_method 0 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f163403fdb0 0x7f1634042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(525..525 src has 1..525) ==== 6741+0+0 (secure 0 0 0) 0x7f1644072f70 con 0x7f16541347f0 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 10 client.114331.objecter ms_dispatch 0x7f1654000b90 osd_map(525..525 src has 1..525) 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 3 client.114331.objecter handle_osd_map got epochs [525,525] > 0 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 3 client.114331.objecter handle_osd_map decoding full epoch 525 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164bfff640 10 monclient(hunting): handle_auth_done global_id 114258 payload 824 2024-06-11T19:22:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164bfff640 1 -- 172.21.15.98:0/1281952264 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f165409e6b0 con 0x7f165411f1d0 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 20 client.114331.objecter dump_active .. 0 homeless 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f1652d9d640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f163403fdb0 0x7f1634042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f1652d9d640 10 monclient: get_auth_request con 0x7f163403fdb0 auth_method 0 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164bfff640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411f1d0 0x7f165411f5b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f163c005310 tx=0x7f163c0057d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 10 client.114331.objecter ms_handle_connect 0x7f165411f1d0 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 10 client.114331.objecter resend_mon_ops 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f163c016070 con 0x7f165411f1d0 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.743+0000 7f164affd640 10 monclient: dump: 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.744+0000 7f164affd640 10 client.114331.objecter ms_handle_connect 0x7f163403fdb0 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.744+0000 7f1652d9d640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f163403fdb0 0x7f1634042270 secure :-1 s=READY pgs=4267 cs=0 l=1 rev1=1 crypto rx=0x7f16541203f0 tx=0x7f164c009780 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.744+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f163c00ea70 con 0x7f165411f1d0 2024-06-11T19:22:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.744+0000 7f164affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.744+0000 7f164affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.744+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411f1d0 msgr2=0x7f165411f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.744+0000 7f164affd640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411f1d0 0x7f165411f5b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f163c005310 tx=0x7f163c0057d0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.744+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411f1d0 msgr2=0x7f165411f5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.744+0000 7f164affd640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411f1d0 0x7f165411f5b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.750+0000 7f1658d8c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:43.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.750+0000 7f1658d8c640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411faf0 0x7f165409f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.750+0000 7f1658d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.750+0000 7f1658d8c640 1 -- 172.21.15.98:0/1281952264 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f1654075170 con 0x7f165411faf0 2024-06-11T19:22:43.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.750+0000 7f164bfff640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411faf0 0x7f165409f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:43.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.750+0000 7f164bfff640 10 monclient: get_auth_request con 0x7f165411faf0 auth_method 0 2024-06-11T19:22:43.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.750+0000 7f164bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:43.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.750+0000 7f164bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:43.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.750+0000 7f164bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164bfff640 10 monclient(hunting): handle_auth_done global_id 114267 payload 824 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164bfff640 1 -- 172.21.15.98:0/1281952264 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1654121210 con 0x7f165411faf0 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164bfff640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411faf0 0x7f165409f080 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f163c005310 tx=0x7f163c0059c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 10 client.114331.objecter ms_handle_connect 0x7f165411faf0 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 10 client.114331.objecter resend_mon_ops 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f163c016040 con 0x7f165411faf0 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 10 monclient: dump: 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9228 (secure 0 0 0) 0x7f163c005a30 con 0x7f165411faf0 2024-06-11T19:22:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411faf0 msgr2=0x7f165409f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411faf0 0x7f165409f080 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f163c005310 tx=0x7f163c0059c0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 1 -- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411faf0 msgr2=0x7f165409f080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.751+0000 7f164affd640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f165411faf0 0x7f165409f080 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.774 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:43.774 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:43.774 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:43.774 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:43.774 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 52119, 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:43.775 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:43.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:43.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:43.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:43.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:43.797 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:43.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:43.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:43.799 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:43.799 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:43.799 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:43.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:43.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:43.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:43.799 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:43.799 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 10 librados: watch_flush enter 2024-06-11T19:22:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 10 librados: watch_flush exit 2024-06-11T19:22:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 20 client.114331.objecter shutdown clearing up homeless session... 2024-06-11T19:22:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 10 client.114331.objecter successfully canceled tick 2024-06-11T19:22:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 1 -- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f163403fdb0 msgr2=0x7f1634042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f163403fdb0 0x7f1634042270 secure :-1 s=READY pgs=4267 cs=0 l=1 rev1=1 crypto rx=0x7f16541203f0 tx=0x7f164c009780 comp rx=0 tx=0).stop 2024-06-11T19:22:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 10 monclient: shutdown 2024-06-11T19:22:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 1 -- 172.21.15.98:0/1281952264 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16541347f0 msgr2=0x7f1654133270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16541347f0 0x7f1654133270 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f16440097c0 tx=0x7f16440046c0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1652d9d640 1 -- 172.21.15.98:0/1281952264 reap_dead start 2024-06-11T19:22:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 1 -- 172.21.15.98:0/1281952264 shutdown_connections 2024-06-11T19:22:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 1 --2- 172.21.15.98:0/1281952264 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16541347f0 0x7f1654133270 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 1 -- 172.21.15.98:0/1281952264 >> 172.21.15.98:0/1281952264 conn(0x7f16540a4780 msgr2=0x7f16541c00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 1 -- 172.21.15.98:0/1281952264 shutdown_connections 2024-06-11T19:22:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 1 -- 172.21.15.98:0/1281952264 wait complete. 2024-06-11T19:22:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.752+0000 7f1658d8c640 1 librados: shutdown 2024-06-11T19:22:43.803 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-06-11T19:22:43.803 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon i 2024-06-11T19:22:43.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:22:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.903+0000 7f9bdb035640 10 monclient: get_monmap_and_config 2024-06-11T19:22:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.903+0000 7f9bdb035640 10 monclient: build_initial_monmap 2024-06-11T19:22:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.903+0000 7f9bdb035640 10 monclient: monmap: 2024-06-11T19:22:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:22:43.904773+0000 2024-06-11T19:22:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:22:43.904773+0000 2024-06-11T19:22:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:22:43.903 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:22:43.904 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:43.904 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:43.904 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:43.904 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:43.904 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:43.904 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:43.904 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:43.904 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:22:43.904 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:22:43.904 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:43.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.904+0000 7f9bdb035640 1 Processor -- start 2024-06-11T19:22:43.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.904+0000 7f9bdb035640 1 -- start start 2024-06-11T19:22:43.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.904+0000 7f9bdb035640 10 monclient: init 2024-06-11T19:22:43.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.904+0000 7f9bdb035640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:43.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.904+0000 7f9bdb035640 10 monclient: _add_conns ranks=[8,6,1,4,5,3,10,0,2,7,9] 2024-06-11T19:22:43.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.904+0000 7f9bdb035640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd40a4920 0x7f9bd40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.904+0000 7f9bdb035640 10 monclient(hunting): picked mon.noname-e con 0x7f9bd40a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:43.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bdb035640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9bd40a5240 0x7f9bd40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:43.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bdb035640 10 monclient(hunting): picked mon.noname-d con 0x7f9bd40a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:22:43.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bdb035640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f9bd40a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:22:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bdb035640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bd40a7140 con 0x7f9bd40a2530 2024-06-11T19:22:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bdb035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bdb035640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f9bd40a2b50 con 0x7f9bd40a5240 2024-06-11T19:22:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bdb035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bdb035640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9bd40a2e30 con 0x7f9bd40a4920 2024-06-11T19:22:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bdb035640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bd3fff640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9bd40a5240 msgr2=0x7f9bd40a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bd8dd1640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd40a4920 msgr2=0x7f9bd40a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bdb035640 10 monclient(hunting): authenticate will time out at 1573.712082s 2024-06-11T19:22:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bd3fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9bd40a5240 0x7f9bd40a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bd8dd1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd40a4920 0x7f9bd40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:43.905+0000 7f9bd95d2640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f9bd40a2530 legacy=0x7f9bd40a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-06-11T19:22:44.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:44.106+0000 7f9bd8dd1640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd40a4920 msgr2=0x7f9bd40a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:44.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:44.106+0000 7f9bd3fff640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9bd40a5240 msgr2=0x7f9bd40a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:44.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:44.106+0000 7f9bd3fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9bd40a5240 0x7f9bd40a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:44.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:44.106+0000 7f9bd8dd1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd40a4920 0x7f9bd40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:22:44.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:44.106+0000 7f9bd95d2640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f9bd40a2530 legacy=0x7f9bd40a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-06-11T19:22:44.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:44.507+0000 7f9bd8dd1640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd40a4920 msgr2=0x7f9bd40a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:44.507+0000 7f9bd3fff640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9bd40a5240 msgr2=0x7f9bd40a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:44.507+0000 7f9bd3fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9bd40a5240 0x7f9bd40a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:44.507+0000 7f9bd8dd1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd40a4920 0x7f9bd40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:22:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:44.507+0000 7f9bd95d2640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f9bd40a2530 legacy=0x7f9bd40a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-06-11T19:22:45.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:45.308+0000 7f9bd8dd1640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd40a4920 msgr2=0x7f9bd40a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:45.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:45.308+0000 7f9bd8dd1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd40a4920 0x7f9bd40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:45.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:45.308+0000 7f9bd3fff640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9bd40a5240 msgr2=0x7f9bd40a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-06-11T19:22:45.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:45.308+0000 7f9bd3fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9bd40a5240 0x7f9bd40a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:22:45.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:45.308+0000 7f9bd95d2640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f9bd40a2530 legacy=0x7f9bd40a5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-06-11T19:22:46.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 10 monclient(hunting): tick 2024-06-11T19:22:46.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:22:46.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 1 monclient(hunting): continuing hunt 2024-06-11T19:22:46.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:22:46.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd40a4920 msgr2=0x7f9bd40a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:46.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd40a4920 0x7f9bd40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:46.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9bd40a5240 msgr2=0x7f9bd40a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9bd40a5240 0x7f9bd40a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f9bd40a2530 legacy=0x7f9bd40a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 10 monclient: _add_conns ranks=[0,10,9,5,7,4,2,3] 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9bb8003820 addr v1:172.21.15.98:6789/0 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9bb8007480 0x7f9bb8009940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 10 monclient(hunting): picked mon.noname-f con 0x7f9bb8007480 addr v2:172.21.15.117:3300/0 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9bb800a030 0x7f9bb800c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 10 monclient(hunting): picked mon.noname-a con 0x7f9bb800a030 addr v2:172.21.15.98:3300/0 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bd40a3110 con 0x7f9bb8003820 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9bd40a2b50 con 0x7f9bb800a030 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:46.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f9bd40a2e30 con 0x7f9bb8007480 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd27fc640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd3fff640 1 -- >> v2:172.21.15.117:3300/0 conn(0x7f9bb8007480 msgr2=0x7f9bb8009940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3300/0 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd3fff640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9bb8007480 0x7f9bb8009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9bb800a030 0x7f9bb800c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd8dd1640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9bb8003820 0x7f9bb8005c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44122/0 (socket says 172.21.15.98:44122) 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd8dd1640 1 -- 172.21.15.98:0/2203246023 learned_addr learned my addr 172.21.15.98:0/2203246023 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 10 monclient(hunting): get_auth_request con 0x7f9bb800a030 auth_method 0 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:46.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 10 monclient(hunting): handle_auth_done global_id 65943 payload 274 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 1 -- 172.21.15.98:0/2203246023 >> v2:172.21.15.117:3300/0 conn(0x7f9bb8007480 msgr2=0x7f9bb8009940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 1 --2- 172.21.15.98:0/2203246023 >> v2:172.21.15.117:3300/0 conn(0x7f9bb8007480 0x7f9bb8009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 1 -- 172.21.15.98:0/2203246023 >> v1:172.21.15.98:6789/0 conn(0x7f9bb8003820 legacy=0x7f9bb8005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 10 monclient: _finish_hunting 0 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 1 monclient: found mon.noname-a 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 1 -- 172.21.15.98:0/2203246023 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd40a7140 con 0x7f9bb800a030 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 1 -- 172.21.15.98:0/2203246023 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3620975219 0 0) 0x7f9bd40a3110 con 0x7f9bb8003820 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 1 -- 172.21.15.98:0/2203246023 reap_dead start 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd95d2640 1 --2- 172.21.15.98:0/2203246023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bb800a030 0x7f9bb800c4f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8007450 tx=0x7f9bcc02c0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90361cfd607bf60 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:46.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 1 -- 172.21.15.98:0/2203246023 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9bcc035470 con 0x7f9bb800a030 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 10 monclient: dump: 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:46.912 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 10 monclient: _finish_auth 0 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 10 monclient: _check_auth_tickets 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 1 -- 172.21.15.98:0/2203246023 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bcc030070 con 0x7f9bb800a030 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 1 -- 172.21.15.98:0/2203246023 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9bcc035aa0 con 0x7f9bb800a030 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bdb035640 5 monclient: authenticate success, global_id 65943 2024-06-11T19:22:46.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bdb035640 10 monclient: get_monmap_and_config success 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bdb035640 4 set_mon_vals no callback set 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.905+0000 7f9bd37fe640 10 monclient: dump: 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:46.914 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.906+0000 7f9bdb035640 10 monclient: shutdown 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.906+0000 7f9bdb035640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.906+0000 7f9bdb035640 1 -- 172.21.15.98:0/2203246023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bb800a030 msgr2=0x7f9bb800c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.906+0000 7f9bdb035640 1 --2- 172.21.15.98:0/2203246023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bb800a030 0x7f9bb800c4f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8007450 tx=0x7f9bcc02c0d0 comp rx=0 tx=0).stop 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.906+0000 7f9bdb035640 1 -- 172.21.15.98:0/2203246023 shutdown_connections 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.906+0000 7f9bdb035640 1 --2- 172.21.15.98:0/2203246023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bb800a030 0x7f9bb800c4f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:46.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.907+0000 7f9bdb035640 1 -- 172.21.15.98:0/2203246023 >> 172.21.15.98:0/2203246023 conn(0x7f9bd40ab850 msgr2=0x7f9bd40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.907+0000 7f9bdb035640 1 -- 172.21.15.98:0/2203246023 shutdown_connections 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.907+0000 7f9bdb035640 1 -- 172.21.15.98:0/2203246023 wait complete. 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.907+0000 7f9bdb035640 10 monclient: build_initial_monmap 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.907+0000 7f9bdb035640 10 monclient: monmap: 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:22:46.916 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 1 librados: starting msgr at 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 1 librados: starting objecter 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 1 Processor -- start 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 1 -- start start 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 1 librados: setting wanted keys 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 1 librados: calling monclient init 2024-06-11T19:22:46.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 10 monclient: init 2024-06-11T19:22:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 10 monclient: _reopen_session rank -1 2024-06-11T19:22:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 10 monclient: _add_conns ranks=[4,8,5,7,0,1,3,2,6] 2024-06-11T19:22:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9bb800a030 0x7f9bd4137af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 10 monclient(hunting): picked mon.noname-c con 0x7f9bb800a030 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:22:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd41cd3c0 0x7f9bd4138030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.908+0000 7f9bdb035640 10 monclient(hunting): picked mon.noname-e con 0x7f9bd41cd3c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:22:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bdb035640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9bd41215f0 0x7f9bd4121a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bdb035640 10 monclient(hunting): picked mon.noname-h con 0x7f9bd41215f0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:22:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bdb035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bdb035640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f9bd40a6da0 con 0x7f9bb800a030 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bdb035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bdb035640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f9bd4138e50 con 0x7f9bd41215f0 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bdb035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bdb035640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9bd40a4d00 con 0x7f9bd41cd3c0 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bdb035640 10 monclient(hunting): _renew_subs 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bdb035640 10 monclient(hunting): authenticate will time out at 1576.717050s 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd3fff640 1 -- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd41cd3c0 msgr2=0x7f9bd4138030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3304/0 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd41cd3c0 0x7f9bd4138030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9bb800a030 0x7f9bd4137af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9bb800a030 0x7f9bd4137af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45738/0 (socket says 172.21.15.98:45738) 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 1 -- 172.21.15.98:0/1706397673 learned_addr learned my addr 172.21.15.98:0/1706397673 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 10 monclient(hunting): get_auth_request con 0x7f9bb800a030 auth_method 0 2024-06-11T19:22:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd95d2640 1 -- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9bd41215f0 msgr2=0x7f9bd4121a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd95d2640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9bd41215f0 0x7f9bd4121a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 10 monclient(hunting): handle_auth_done global_id 114340 payload 824 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 1 -- 172.21.15.98:0/1706397673 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd41cd3c0 msgr2=0x7f9bd4138030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9bd41cd3c0 0x7f9bd4138030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 1 -- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9bd41215f0 msgr2=0x7f9bd4121a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9bd41215f0 0x7f9bd4121a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 10 monclient: _finish_hunting 0 2024-06-11T19:22:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 1 monclient: found mon.noname-c 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 1 -- 172.21.15.98:0/1706397673 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd41251d0 con 0x7f9bb800a030 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd8dd1640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9bb800a030 0x7f9bd4137af0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800bcb0 tx=0x7f9bc8009e00 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd17fa640 10 client.?.objecter ms_handle_connect 0x7f9bb800a030 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd17fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9bc800cda0 con 0x7f9bb800a030 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.909+0000 7f9bd17fa640 10 monclient: dump: 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bd17fa640 10 monclient: _finish_auth 0 2024-06-11T19:22:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bd17fa640 10 monclient: _check_auth_tickets 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bd17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bdb035640 5 monclient: authenticate success, global_id 114340 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc8010030 con 0x7f9bb800a030 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bdb035640 10 monclient: _renew_subs 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bdb035640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bdb035640 1 -- 172.21.15.98:0/1706397673 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd41254b0 con 0x7f9bb800a030 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bd17fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9bc80069b0 con 0x7f9bb800a030 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bd17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bd17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bd17fa640 10 monclient: dump: 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:46.932 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bdb035640 10 client.114340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bd2ffd640 4 set_mon_vals no callback set 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bdb035640 10 monclient: _renew_subs 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bdb035640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bdb035640 1 -- 172.21.15.98:0/1706397673 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd4124ef0 con 0x7f9bb800a030 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bdb035640 1 librados: init done 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.910+0000 7f9bdb035640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bdb035640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd4121fb0 0x7f9bd409d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bdb035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bdb035640 1 -- 172.21.15.98:0/1706397673 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f9bd4138760 con 0x7f9bd4121fb0 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f9bc800c040 con 0x7f9bb800a030 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd3fff640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd4121fb0 0x7f9bd409d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd3fff640 10 monclient: get_auth_request con 0x7f9bd4121fb0 auth_method 0 2024-06-11T19:22:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:46.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:46.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:46.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd17fa640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9bb8040d70 0x7f9bb8043230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:46.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:46.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:46.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:46.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(528..528 src has 1..528) ==== 6825+0+0 (secure 0 0 0) 0x7f9bc8051d10 con 0x7f9bb800a030 2024-06-11T19:22:46.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd17fa640 10 client.114340.objecter ms_dispatch 0x7f9bd4000b90 osd_map(528..528 src has 1..528) 2024-06-11T19:22:46.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd17fa640 3 client.114340.objecter handle_osd_map got epochs [528,528] > 0 2024-06-11T19:22:46.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd17fa640 3 client.114340.objecter handle_osd_map decoding full epoch 528 2024-06-11T19:22:46.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd17fa640 20 client.114340.objecter dump_active .. 0 homeless 2024-06-11T19:22:46.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd95d2640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9bb8040d70 0x7f9bb8043230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd95d2640 10 monclient: get_auth_request con 0x7f9bb8040d70 auth_method 0 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd3fff640 10 monclient(hunting): handle_auth_done global_id 84787 payload 824 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.911+0000 7f9bd3fff640 1 -- 172.21.15.98:0/1706397673 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9bd40a5000 con 0x7f9bd4121fb0 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 10 client.114340.objecter ms_handle_connect 0x7f9bd4121fb0 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 10 client.114340.objecter resend_mon_ops 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd3fff640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd4121fb0 0x7f9bd409d180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40a1be0 tx=0x7f9bc000b8c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9bc0003c40 con 0x7f9bd4121fb0 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 10 monclient: dump: 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:46.935 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 10 client.114340.objecter ms_handle_connect 0x7f9bb8040d70 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd95d2640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9bb8040d70 0x7f9bb8043230 secure :-1 s=READY pgs=4268 cs=0 l=1 rev1=1 crypto rx=0x7f9bb80037f0 tx=0x7f9bcc02f000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9bc0015070 con 0x7f9bd4121fb0 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd4121fb0 msgr2=0x7f9bd409d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd4121fb0 0x7f9bd409d180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40a1be0 tx=0x7f9bc000b8c0 comp rx=0 tx=0).stop 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd4121fb0 msgr2=0x7f9bd409d180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.912+0000 7f9bd17fa640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd4121fb0 0x7f9bd409d180 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bdb035640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bdb035640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd409d6c0 0x7f9bd409dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bdb035640 10 monclient(hunting): start opening mon connection 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bdb035640 1 -- 172.21.15.98:0/1706397673 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f9bd4050830 con 0x7f9bd409d6c0 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd3fff640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd409d6c0 0x7f9bd409dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd3fff640 10 monclient: get_auth_request con 0x7f9bd409d6c0 auth_method 0 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:22:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd3fff640 10 monclient(hunting): handle_auth_done global_id 84796 payload 824 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd3fff640 1 -- 172.21.15.98:0/1706397673 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9bd40aa9c0 con 0x7f9bd409d6c0 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd3fff640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd409d6c0 0x7f9bd409dad0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0014000 tx=0x7f9bc0005a50 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd17fa640 10 client.114340.objecter ms_handle_connect 0x7f9bd409d6c0 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd17fa640 10 client.114340.objecter resend_mon_ops 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9bc0005690 con 0x7f9bd409d6c0 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd17fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:22:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.920+0000 7f9bd17fa640 10 monclient: dump: 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:22:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:22:46.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:22:46.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9570 (secure 0 0 0) 0x7f9bc0005690 con 0x7f9bd409d6c0 2024-06-11T19:22:46.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bd17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:22:46.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bd17fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:22:46.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd409d6c0 msgr2=0x7f9bd409dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:46.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bd17fa640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd409d6c0 0x7f9bd409dad0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0014000 tx=0x7f9bc0005a50 comp rx=0 tx=0).stop 2024-06-11T19:22:46.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bd17fa640 1 -- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd409d6c0 msgr2=0x7f9bd409dad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:22:46.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bd17fa640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9bd409d6c0 0x7f9bd409dad0 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0014000 tx=0x7f9bc0005a50 comp rx=0 tx=0).stop 2024-06-11T19:22:46.940 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:22:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 438690, 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: 7 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 10, 2024-06-11T19:22:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:22:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:22:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:22:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:22:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:22:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:22:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:22:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:22:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:22:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:22:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:22:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:22:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:22:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:22:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:22:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:22:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:22:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:22:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:22:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:22:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:22:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:22:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:22:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:22:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:22:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:22:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:22:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:22:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:22:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:22:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:22:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:22:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:22:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:22:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:22:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:22:46.964 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:22:46.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 10 librados: watch_flush enter 2024-06-11T19:22:46.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 10 librados: watch_flush exit 2024-06-11T19:22:46.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 20 client.114340.objecter shutdown clearing up homeless session... 2024-06-11T19:22:46.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 10 client.114340.objecter successfully canceled tick 2024-06-11T19:22:46.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 1 -- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9bb8040d70 msgr2=0x7f9bb8043230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:46.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9bb8040d70 0x7f9bb8043230 secure :-1 s=READY pgs=4268 cs=0 l=1 rev1=1 crypto rx=0x7f9bb80037f0 tx=0x7f9bcc02f000 comp rx=0 tx=0).stop 2024-06-11T19:22:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 10 monclient: shutdown 2024-06-11T19:22:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:22:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 1 -- 172.21.15.98:0/1706397673 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9bb800a030 msgr2=0x7f9bd4137af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:22:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 1 --2- 172.21.15.98:0/1706397673 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9bb800a030 0x7f9bd4137af0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800bcb0 tx=0x7f9bc8009e00 comp rx=0 tx=0).stop 2024-06-11T19:22:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bd95d2640 1 -- 172.21.15.98:0/1706397673 reap_dead start 2024-06-11T19:22:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bd95d2640 1 -- 172.21.15.98:0/1706397673 reap_dead start 2024-06-11T19:22:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 1 -- 172.21.15.98:0/1706397673 shutdown_connections 2024-06-11T19:22:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 1 -- 172.21.15.98:0/1706397673 >> 172.21.15.98:0/1706397673 conn(0x7f9bd40ab850 msgr2=0x7f9bd41c0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:22:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 1 -- 172.21.15.98:0/1706397673 shutdown_connections 2024-06-11T19:22:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 1 -- 172.21.15.98:0/1706397673 wait complete. 2024-06-11T19:22:46.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:22:46.921+0000 7f9bdb035640 1 librados: shutdown 2024-06-11T19:22:46.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-06-11T19:22:46.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-06-11T19:23:23.297 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:23:23.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8991 ms) 2024-06-11T19:23:23.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-06-11T19:23:23.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:23:23.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6101 ms) 2024-06-11T19:23:23.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-06-11T19:23:23.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:23:23.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (4904 ms) 2024-06-11T19:23:23.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-06-11T19:23:23.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:23:23.298 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (28051 ms) 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (28192 ms) 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11860 ms) 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 00000010 69 73 |is| 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 00000012 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (11967 ms) 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: require_osd_release = squid 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6087 ms) 2024-06-11T19:23:23.299 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (612229 ms total) 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1023 ms) 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10982 ms) 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18001 ms) 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (3 ms) 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30009 ms total) 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 2024-06-11T19:23:23.300 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6952 ms) 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (12408 ms) 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: waiting for scrub... 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: done waiting 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (38446 ms) 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9537 ms) 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7952 ms) 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-06-11T19:23:23.301 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (7982 ms) 2024-06-11T19:23:23.302 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-06-11T19:23:23.302 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10005 ms) 2024-06-11T19:23:23.302 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-06-11T19:24:16.969 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-06-11T19:24:16.970 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-06-11T19:24:16.970 INFO:tasks.ceph.mon.e:Restarting daemon 2024-06-11T19:24:16.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-06-11T19:24:16.974 INFO:tasks.ceph.mon.e:Started 2024-06-11T19:24:16.974 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-06-11T19:24:16.974 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.h 2024-06-11T19:24:16.974 INFO:tasks.ceph.mon.h:Restarting daemon 2024-06-11T19:24:16.974 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-06-11T19:24:16.977 INFO:tasks.ceph.mon.h:Started 2024-06-11T19:24:16.977 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-06-11T19:24:16.977 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.f 2024-06-11T19:24:16.978 INFO:tasks.ceph.mon.f:Restarting daemon 2024-06-11T19:24:16.978 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-06-11T19:24:16.980 INFO:tasks.ceph.mon.f:Started 2024-06-11T19:24:16.980 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-06-11T19:24:16.980 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-06-11T19:24:16.980 INFO:tasks.ceph.mon.d:Restarting daemon 2024-06-11T19:24:16.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-06-11T19:24:16.982 INFO:tasks.ceph.mon.d:Started 2024-06-11T19:24:16.982 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-06-11T19:24:16.983 INFO:tasks.ceph.mon.a:Sent signal 19 2024-06-11T19:24:16.983 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-06-11T19:24:17.089 INFO:tasks.ceph.mon.f.smithi117.stderr:2024-06-11T19:24:17.090+0000 7f9b41c0db00 -1 mon.f@-1(???).log v0 log_external_backlog rewinding external_log_to from 544 -> 0 (sync_force? mon rebuild?) 2024-06-11T19:24:17.089 INFO:tasks.ceph.mon.h.smithi117.stderr:2024-06-11T19:24:17.090+0000 7fd6d3436b00 -1 mon.h@-1(???).log v0 log_external_backlog rewinding external_log_to from 544 -> 0 (sync_force? mon rebuild?) 2024-06-11T19:24:17.093 INFO:tasks.ceph.mon.d.smithi098.stderr:2024-06-11T19:24:17.094+0000 7f15d5352b00 -1 mon.d@-1(???).log v0 log_external_backlog rewinding external_log_to from 544 -> 0 (sync_force? mon rebuild?) 2024-06-11T19:24:31.984 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-06-11T19:24:31.984 INFO:tasks.ceph.mon.a:Sent signal 18 2024-06-11T19:24:31.985 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:24:31.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:24:32.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.095+0000 7fb26f9e2640 10 monclient: get_monmap_and_config 2024-06-11T19:24:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.095+0000 7fb26f9e2640 10 monclient: build_initial_monmap 2024-06-11T19:24:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.095+0000 7fb26f9e2640 10 monclient: monmap: 2024-06-11T19:24:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:32.096846+0000 2024-06-11T19:24:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:32.096846+0000 2024-06-11T19:24:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:32.095 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:32.096 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:32.096 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.096+0000 7fb26f9e2640 1 Processor -- start 2024-06-11T19:24:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.096+0000 7fb26f9e2640 1 -- start start 2024-06-11T19:24:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.096+0000 7fb26f9e2640 10 monclient: init 2024-06-11T19:24:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 10 monclient: _add_conns ranks=[6,2,8,3,10,4,5,1,9,7,0] 2024-06-11T19:24:32.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 10 monclient(hunting): picked mon.noname-d con 0x7fb2680a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 10 monclient(hunting): picked mon.noname-b con 0x7fb2680a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 10 monclient(hunting): picked mon.noname-e con 0x7fb2680a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb2680a7140 con 0x7fb2680a5240 2024-06-11T19:24:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb2680a2b50 con 0x7fb2680a4920 2024-06-11T19:24:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb2680a2e30 con 0x7fb2680a2530 2024-06-11T19:24:32.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26f9e2640 10 monclient(hunting): authenticate will time out at 1681.904139s 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26df7f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26d77e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26d77e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:36824/0 (socket says 172.21.15.98:36824) 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26d77e640 1 -- 172.21.15.98:0/2352103894 learned_addr learned my addr 172.21.15.98:0/2352103894 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26cf7d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26d77e640 10 monclient(hunting): get_auth_request con 0x7fb2680a4920 auth_method 0 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26d77e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26d77e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26d77e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26df7f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:42578/0 (socket says 172.21.15.98:42578) 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26df7f640 10 monclient(hunting): get_auth_request con 0x7fb2680a2530 auth_method 0 2024-06-11T19:24:32.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26df7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26df7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26df7f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26cf7d640 10 monclient(hunting): get_auth_request con 0x7fb2680a5240 auth_method 0 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26cf7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26cf7d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26cf7d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26d77e640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 msgr2=0x7fb2680a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26d77e640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 msgr2=0x7fb2680a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26d77e640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26cf7d640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 msgr2=0x7fb2680a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26cf7d640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 msgr2=0x7fb2680a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26cf7d640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:32.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26df7f640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 msgr2=0x7fb2680a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:24:32.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26df7f640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 msgr2=0x7fb2680a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:32.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26df7f640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:32.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26d77e640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:32.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26cf7d640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:32.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.097+0000 7fb26df7f640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:32.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26df7f640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:32.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26d77e640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:32.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26cf7d640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26df7f640 10 monclient(hunting): get_auth_request con 0x7fb2680a2530 auth_method 0 2024-06-11T19:24:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26df7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26df7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26df7f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26cf7d640 10 monclient(hunting): get_auth_request con 0x7fb2680a5240 auth_method 0 2024-06-11T19:24:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26cf7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26cf7d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26cf7d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26d77e640 10 monclient(hunting): get_auth_request con 0x7fb2680a4920 auth_method 0 2024-06-11T19:24:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26d77e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26d77e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26d77e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26df7f640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 msgr2=0x7fb2680a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:24:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26df7f640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 msgr2=0x7fb2680a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26df7f640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26d77e640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 msgr2=0x7fb2680a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:24:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26d77e640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 msgr2=0x7fb2680a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26d77e640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26cf7d640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 msgr2=0x7fb2680a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:24:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26cf7d640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 msgr2=0x7fb2680a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26cf7d640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26df7f640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:24:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26d77e640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:24:32.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.298+0000 7fb26cf7d640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:24:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26df7f640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26d77e640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26cf7d640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26df7f640 10 monclient(hunting): get_auth_request con 0x7fb2680a2530 auth_method 0 2024-06-11T19:24:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26df7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26df7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26df7f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26cf7d640 10 monclient(hunting): get_auth_request con 0x7fb2680a5240 auth_method 0 2024-06-11T19:24:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26cf7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:32.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26cf7d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26cf7d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26d77e640 10 monclient(hunting): get_auth_request con 0x7fb2680a4920 auth_method 0 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26d77e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26d77e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26d77e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26d77e640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 msgr2=0x7fb2680a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26d77e640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 msgr2=0x7fb2680a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26d77e640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26df7f640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 msgr2=0x7fb2680a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26df7f640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 msgr2=0x7fb2680a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26df7f640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26cf7d640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 msgr2=0x7fb2680a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26cf7d640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 msgr2=0x7fb2680a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26cf7d640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:32.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26d77e640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:24:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26df7f640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:24:32.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:32.699+0000 7fb26cf7d640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:24:33.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:33.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 10 monclient(hunting): get_auth_request con 0x7fb2680a4920 auth_method 0 2024-06-11T19:24:33.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:33.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:33.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:33.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26df7f640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26cf7d640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26df7f640 10 monclient(hunting): get_auth_request con 0x7fb2680a2530 auth_method 0 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26df7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26df7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26df7f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26cf7d640 10 monclient(hunting): get_auth_request con 0x7fb2680a5240 auth_method 0 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26cf7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26cf7d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26cf7d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:33.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26cf7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26cf7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26cf7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 10 monclient(hunting): handle_auth_done global_id 124107 payload 274 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 msgr2=0x7fb2680a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 msgr2=0x7fb2680a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 10 monclient: _finish_hunting 0 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 1 monclient: found mon.noname-d 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26d77e640 1 -- 172.21.15.98:0/2352103894 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2680a3110 con 0x7fb2680a4920 2024-06-11T19:24:33.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.500+0000 7fb26df7f640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.501+0000 7fb26d77e640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb264010920 tx=0x7fb26402c250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c66e7caae5bea4d0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.501+0000 7fb25bfff640 1 -- 172.21.15.98:0/2352103894 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb264032070 con 0x7fb2680a4920 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.501+0000 7fb25bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb25bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb25bfff640 10 monclient: dump: 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:33.504 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb25bfff640 10 monclient: _finish_auth 0 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb25bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb25bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb25bfff640 1 -- 172.21.15.98:0/2352103894 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2640044a0 con 0x7fb2680a4920 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb26f9e2640 5 monclient: authenticate success, global_id 124107 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb26f9e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb25bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:33.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb26f9e2640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb25bfff640 1 -- 172.21.15.98:0/2352103894 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb26400f040 con 0x7fb2680a4920 2024-06-11T19:24:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb26f9e2640 4 set_mon_vals no callback set 2024-06-11T19:24:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb25bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb25bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb25bfff640 10 monclient: dump: 2024-06-11T19:24:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:33.506 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:33.507 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:33.507 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:33.507 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:33.507 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:33.507 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:33.507 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:33.507 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:33.507 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:33.507 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:33.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb26f9e2640 10 monclient: shutdown 2024-06-11T19:24:33.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb26f9e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:33.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb26f9e2640 1 -- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 msgr2=0x7fb2680a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:33.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.502+0000 7fb26f9e2640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb264010920 tx=0x7fb26402c250 comp rx=0 tx=0).stop 2024-06-11T19:24:33.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.503+0000 7fb26f9e2640 1 -- 172.21.15.98:0/2352103894 shutdown_connections 2024-06-11T19:24:33.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.503+0000 7fb26f9e2640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2680a2530 0x7fb2680a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:33.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.503+0000 7fb26f9e2640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb2680a5240 0x7fb2680a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:33.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.503+0000 7fb26f9e2640 1 --2- 172.21.15.98:0/2352103894 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb2680a4920 0x7fb2680a4d00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:33.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.503+0000 7fb26f9e2640 1 -- 172.21.15.98:0/2352103894 >> 172.21.15.98:0/2352103894 conn(0x7fb2680ab850 msgr2=0x7fb2680a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:33.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.503+0000 7fb26f9e2640 1 -- 172.21.15.98:0/2352103894 shutdown_connections 2024-06-11T19:24:33.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.503+0000 7fb26f9e2640 1 -- 172.21.15.98:0/2352103894 wait complete. 2024-06-11T19:24:33.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.503+0000 7fb26f9e2640 10 monclient: build_initial_monmap 2024-06-11T19:24:33.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.503+0000 7fb26f9e2640 10 monclient: monmap: 2024-06-11T19:24:33.514 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:33.514 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:33.515 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:33.516 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:33.516 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:33.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 1 librados: starting msgr at 2024-06-11T19:24:33.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 1 librados: starting objecter 2024-06-11T19:24:33.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 1 Processor -- start 2024-06-11T19:24:33.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 1 -- start start 2024-06-11T19:24:33.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 1 librados: setting wanted keys 2024-06-11T19:24:33.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 1 librados: calling monclient init 2024-06-11T19:24:33.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 10 monclient: init 2024-06-11T19:24:33.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:33.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 10 monclient: _add_conns ranks=[1,3,5,0,6,7,2,8,4] 2024-06-11T19:24:33.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2680a2530 0x7fb2681cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:33.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 10 monclient(hunting): picked mon.noname-f con 0x7fb2680a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:24:33.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb2680a4920 0x7fb2681ce070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:33.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 10 monclient(hunting): picked mon.noname-g con 0x7fb2680a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:33.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb2680a5240 0x7fb268122ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:33.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 10 monclient(hunting): picked mon.noname-h con 0x7fb2680a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:33.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb2680a6da0 con 0x7fb2680a2530 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.504+0000 7fb26f9e2640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fb268138e50 con 0x7fb2680a4920 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26f9e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26f9e2640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fb2680a6420 con 0x7fb2680a5240 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26f9e2640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26f9e2640 10 monclient(hunting): authenticate will time out at 1683.312022s 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb2680a5240 0x7fb268122ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26d77e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2680a2530 0x7fb2681cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26cf7d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb2680a4920 0x7fb2681ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26cf7d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb2680a4920 0x7fb2681ce070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:56148/0 (socket says 172.21.15.98:56148) 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26d77e640 10 monclient(hunting): get_auth_request con 0x7fb2680a2530 auth_method 0 2024-06-11T19:24:33.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26cf7d640 1 -- 172.21.15.98:0/3696414533 learned_addr learned my addr 172.21.15.98:0/3696414533 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb2680a5240 0x7fb268122ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3302/0 says I am v2:172.21.15.98:37018/0 (socket says 172.21.15.98:37018) 2024-06-11T19:24:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26d77e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26d77e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26d77e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 10 monclient(hunting): get_auth_request con 0x7fb2680a5240 auth_method 0 2024-06-11T19:24:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26cf7d640 10 monclient(hunting): get_auth_request con 0x7fb2680a4920 auth_method 0 2024-06-11T19:24:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26cf7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26cf7d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:33.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26cf7d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26d77e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26d77e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26d77e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26cf7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26cf7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26cf7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 10 monclient(hunting): handle_auth_done global_id 124106 payload 824 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 1 -- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb2680a4920 msgr2=0x7fb2681ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb2680a4920 0x7fb2681ce070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:33.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 1 -- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2680a2530 msgr2=0x7fb2681cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2680a2530 0x7fb2681cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 10 monclient: _finish_hunting 0 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 1 monclient: found mon.noname-h 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 1 -- 172.21.15.98:0/3696414533 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2681268e0 con 0x7fb2680a5240 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26d77e640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2680a2530 0x7fb2681cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26cf7d640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb2680a4920 0x7fb2681ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb26df7f640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb2680a5240 0x7fb268122ee0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb2680a2950 tx=0x7fb260013620 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb259ffb640 10 client.?.objecter ms_handle_connect 0x7fb2680a5240 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.505+0000 7fb259ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 1 -- 172.21.15.98:0/3696414533 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb26001d030 con 0x7fb2680a5240 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:24:33.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 10 monclient: dump: 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:33.522 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 10 monclient: _finish_auth 0 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 1 -- 172.21.15.98:0/3696414533 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb260007e90 con 0x7fb2680a5240 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb26f9e2640 5 monclient: authenticate success, global_id 124106 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb26f9e2640 10 monclient: _renew_subs 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb26f9e2640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb26f9e2640 1 -- 172.21.15.98:0/3696414533 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb268126bc0 con 0x7fb2680a5240 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 1 -- 172.21.15.98:0/3696414533 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb260019d60 con 0x7fb2680a5240 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:24:33.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb259ffb640 10 monclient: dump: 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:33.524 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb26f9e2640 10 client.124106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb26f9e2640 10 monclient: _renew_subs 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb26f9e2640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb26f9e2640 1 -- 172.21.15.98:0/3696414533 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2680a9260 con 0x7fb2680a5240 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb26f9e2640 1 librados: init done 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.506+0000 7fb25b7fe640 4 set_mon_vals no callback set 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.507+0000 7fb26f9e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.507+0000 7fb26f9e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.507+0000 7fb26f9e2640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.507+0000 7fb259ffb640 1 -- 172.21.15.98:0/3696414533 <== mon.5 v2:172.21.15.117:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fb260020070 con 0x7fb2680a5240 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.507+0000 7fb26f9e2640 1 -- 172.21.15.98:0/3696414533 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb268126f90 con 0x7fb2680a5240 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.508+0000 7fb259ffb640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb24c03fdb0 0x7fb24c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:33.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.508+0000 7fb259ffb640 1 -- 172.21.15.98:0/3696414533 <== mon.5 v2:172.21.15.117:3302/0 5 ==== osd_map(619..619 src has 1..619) ==== 6797+0+0 (secure 0 0 0) 0x7fb26001e070 con 0x7fb2680a5240 2024-06-11T19:24:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.508+0000 7fb259ffb640 10 client.124106.objecter ms_dispatch 0x7fb268000b90 osd_map(619..619 src has 1..619) 2024-06-11T19:24:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.508+0000 7fb259ffb640 3 client.124106.objecter handle_osd_map got epochs [619,619] > 0 2024-06-11T19:24:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.508+0000 7fb259ffb640 3 client.124106.objecter handle_osd_map decoding full epoch 619 2024-06-11T19:24:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.508+0000 7fb259ffb640 20 client.124106.objecter dump_active .. 0 homeless 2024-06-11T19:24:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.508+0000 7fb26d77e640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb24c03fdb0 0x7fb24c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.508+0000 7fb26d77e640 10 monclient: get_auth_request con 0x7fb24c03fdb0 auth_method 0 2024-06-11T19:24:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.508+0000 7fb259ffb640 10 client.124106.objecter ms_handle_connect 0x7fb24c03fdb0 2024-06-11T19:24:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.508+0000 7fb26d77e640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb24c03fdb0 0x7fb24c042270 secure :-1 s=READY pgs=4279 cs=0 l=1 rev1=1 crypto rx=0x7fb264036000 tx=0x7fb264010040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.513+0000 7fb259ffb640 1 -- 172.21.15.98:0/3696414533 <== mon.5 v2:172.21.15.117:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb260027020 con 0x7fb2680a5240 2024-06-11T19:24:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.513+0000 7fb259ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:33.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.513+0000 7fb259ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.770+0000 7fb26f9e2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:24:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.770+0000 7fb26f9e2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.770+0000 7fb26f9e2640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.117:3302/0 2024-06-11T19:24:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.770+0000 7fb26f9e2640 1 -- 172.21.15.98:0/3696414533 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb2680a8da0 con 0x7fb2680a5240 2024-06-11T19:24:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.771+0000 7fb259ffb640 1 -- 172.21.15.98:0/3696414533 <== mon.5 v2:172.21.15.117:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7fb260019660 con 0x7fb2680a5240 2024-06-11T19:24:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.771+0000 7fb259ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.771+0000 7fb259ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:33.770 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":80,"quorum":[1,2,3,4,5,6,7,8],"quorum_names":["f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:24:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 10 librados: watch_flush enter 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 10 librados: watch_flush exit 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 20 client.124106.objecter shutdown clearing up homeless session... 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 10 client.124106.objecter successfully canceled tick 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 -- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb24c03fdb0 msgr2=0x7fb24c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb24c03fdb0 0x7fb24c042270 secure :-1 s=READY pgs=4279 cs=0 l=1 rev1=1 crypto rx=0x7fb264036000 tx=0x7fb264010040 comp rx=0 tx=0).stop 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 10 monclient: shutdown 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 -- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb2680a5240 msgr2=0x7fb268122ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb2680a5240 0x7fb268122ee0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb2680a2950 tx=0x7fb260013620 comp rx=0 tx=0).stop 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 -- 172.21.15.98:0/3696414533 shutdown_connections 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb24c03fdb0 0x7fb24c042270 unknown :-1 s=CLOSED pgs=4279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fb2680a5240 0x7fb268122ee0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb2680a4920 0x7fb2681ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 --2- 172.21.15.98:0/3696414533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2680a2530 0x7fb2681cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 -- 172.21.15.98:0/3696414533 >> 172.21.15.98:0/3696414533 conn(0x7fb2680ab850 msgr2=0x7fb2681be740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 -- 172.21.15.98:0/3696414533 shutdown_connections 2024-06-11T19:24:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 -- 172.21.15.98:0/3696414533 wait complete. 2024-06-11T19:24:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:33.773+0000 7fb26f9e2640 1 librados: shutdown 2024-06-11T19:24:36.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:24:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.892+0000 7f399b4ba640 10 monclient: get_monmap_and_config 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.892+0000 7f399b4ba640 10 monclient: build_initial_monmap 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.892+0000 7f399b4ba640 10 monclient: monmap: 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:36.893638+0000 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:36.893638+0000 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:36.893 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 1 Processor -- start 2024-06-11T19:24:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 1 -- start start 2024-06-11T19:24:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 10 monclient: init 2024-06-11T19:24:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 10 monclient: _add_conns ranks=[0,8,4,1,3,7,5,6,9,10,2] 2024-06-11T19:24:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39940a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:24:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f39940a2530 0x7f399413dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 10 monclient(hunting): picked mon.noname-e con 0x7f39940a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f399413e1a0 0x7f3994140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 10 monclient(hunting): picked mon.noname-c con 0x7f399413e1a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39940a7140 con 0x7f39940a4920 2024-06-11T19:24:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f39940a2b50 con 0x7f399413e1a0 2024-06-11T19:24:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f39940a2e30 con 0x7f39940a2530 2024-06-11T19:24:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.893+0000 7f399b4ba640 10 monclient(hunting): authenticate will time out at 1686.700958s 2024-06-11T19:24:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f39940a2530 0x7f399413dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3999256640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f39940a4920 0x7f39940a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53764/0 (socket says 172.21.15.98:53764) 2024-06-11T19:24:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3999256640 1 -- 172.21.15.98:0/3677364130 learned_addr learned my addr 172.21.15.98:0/3677364130 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3999a57640 1 --2- 172.21.15.98:0/3677364130 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f399413e1a0 0x7f3994140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 10 monclient(hunting): get_auth_request con 0x7f39940a2530 auth_method 0 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3999a57640 10 monclient(hunting): get_auth_request con 0x7f399413e1a0 auth_method 0 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3999a57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3999a57640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3999a57640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3999a57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3999a57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3999a57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 10 monclient(hunting): handle_auth_done global_id 124118 payload 274 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 1 -- 172.21.15.98:0/3677364130 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f399413e1a0 msgr2=0x7f3994140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 1 --2- 172.21.15.98:0/3677364130 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f399413e1a0 0x7f3994140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 1 -- 172.21.15.98:0/3677364130 >> v1:172.21.15.98:6789/0 conn(0x7f39940a4920 legacy=0x7f39940a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 10 monclient: _finish_hunting 0 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 1 monclient: found mon.noname-e 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3998a55640 1 -- 172.21.15.98:0/3677364130 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39940a3110 con 0x7f39940a2530 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.894+0000 7f3999a57640 1 --2- 172.21.15.98:0/3677364130 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f399413e1a0 0x7f3994140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3998a55640 1 --2- 172.21.15.98:0/3677364130 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f39940a2530 0x7f399413dc60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f39880097d0 tx=0x7f398802b790 comp rx=0 tx=0).ready entity=mon.8 client_cookie=4ab1aa5601869213 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 1 -- 172.21.15.98:0/3677364130 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3988039070 con 0x7f39940a2530 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 10 monclient: dump: 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:36.906 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 10 monclient: _finish_auth 0 2024-06-11T19:24:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 1 -- 172.21.15.98:0/3677364130 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3988031960 con 0x7f39940a2530 2024-06-11T19:24:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f399b4ba640 5 monclient: authenticate success, global_id 124118 2024-06-11T19:24:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f399b4ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f399b4ba640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 1 -- 172.21.15.98:0/3677364130 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3988032380 con 0x7f39940a2530 2024-06-11T19:24:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f399b4ba640 4 set_mon_vals no callback set 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f3987fff640 10 monclient: dump: 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f399b4ba640 10 monclient: shutdown 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f399b4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f399b4ba640 1 -- 172.21.15.98:0/3677364130 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f39940a2530 msgr2=0x7f399413dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f399b4ba640 1 --2- 172.21.15.98:0/3677364130 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f39940a2530 0x7f399413dc60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f39880097d0 tx=0x7f398802b790 comp rx=0 tx=0).stop 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f399b4ba640 1 -- 172.21.15.98:0/3677364130 shutdown_connections 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f399b4ba640 1 --2- 172.21.15.98:0/3677364130 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f399413e1a0 0x7f3994140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f399b4ba640 1 --2- 172.21.15.98:0/3677364130 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f39940a2530 0x7f399413dc60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.895+0000 7f399b4ba640 1 -- 172.21.15.98:0/3677364130 >> 172.21.15.98:0/3677364130 conn(0x7f39940ab850 msgr2=0x7f39940a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.896+0000 7f399b4ba640 1 -- 172.21.15.98:0/3677364130 shutdown_connections 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.896+0000 7f399b4ba640 1 -- 172.21.15.98:0/3677364130 wait complete. 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.896+0000 7f399b4ba640 10 monclient: build_initial_monmap 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.896+0000 7f399b4ba640 10 monclient: monmap: 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.896+0000 7f399b4ba640 1 librados: starting msgr at 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.896+0000 7f399b4ba640 1 librados: starting objecter 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.896+0000 7f399b4ba640 1 Processor -- start 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.896+0000 7f399b4ba640 1 -- start start 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.896+0000 7f399b4ba640 1 librados: setting wanted keys 2024-06-11T19:24:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.896+0000 7f399b4ba640 1 librados: calling monclient init 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.896+0000 7f399b4ba640 10 monclient: init 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 10 monclient: _add_conns ranks=[3,1,0,6,4,7,5,2,8] 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f39940a2530 0x7f39941cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 10 monclient(hunting): picked mon.noname-g con 0x7f39940a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39940a4920 0x7f3994135bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 10 monclient(hunting): picked mon.noname-f con 0x7f39940a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f399413e1a0 0x7f3994136110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 10 monclient(hunting): picked mon.noname-a con 0x7f399413e1a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3994138e50 con 0x7f399413e1a0 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f39940a84b0 con 0x7f39940a4920 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f39940a6da0 con 0x7f39940a2530 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f399b4ba640 10 monclient(hunting): authenticate will time out at 1686.704386s 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999a57640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f399413e1a0 0x7f3994136110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999a57640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f399413e1a0 0x7f3994136110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53570/0 (socket says 172.21.15.98:53570) 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999a57640 1 -- 172.21.15.98:0/3518148394 learned_addr learned my addr 172.21.15.98:0/3518148394 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999a57640 10 monclient(hunting): get_auth_request con 0x7f399413e1a0 auth_method 0 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999a57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999a57640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999a57640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999a57640 1 -- 172.21.15.98:0/3518148394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f399413e1a0 msgr2=0x7f3994136110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999a57640 1 -- 172.21.15.98:0/3518148394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f399413e1a0 msgr2=0x7f3994136110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:24:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999a57640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f399413e1a0 0x7f3994136110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999a57640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f399413e1a0 0x7f3994136110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3998a55640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39940a4920 0x7f3994135bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999256640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f39940a2530 0x7f39941cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3998a55640 10 monclient(hunting): get_auth_request con 0x7f39940a4920 auth_method 0 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3998a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3998a55640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3998a55640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999256640 10 monclient(hunting): get_auth_request con 0x7f39940a2530 auth_method 0 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999256640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999256640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999256640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3998a55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3998a55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3998a55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999256640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999256640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.897+0000 7f3999256640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3998a55640 10 monclient(hunting): handle_auth_done global_id 124111 payload 824 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3998a55640 1 -- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f39940a2530 msgr2=0x7f39941cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3998a55640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f39940a2530 0x7f39941cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3998a55640 1 -- 172.21.15.98:0/3518148394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f399413e1a0 msgr2=0x7f3994136110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3998a55640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f399413e1a0 0x7f3994136110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3998a55640 10 monclient: _finish_hunting 0 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3998a55640 1 monclient: found mon.noname-f 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3998a55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3998a55640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:24:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3998a55640 1 -- 172.21.15.98:0/3518148394 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3994122580 con 0x7f39940a4920 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3998a55640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39940a4920 0x7f3994135bd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3988033000 tx=0x7f39880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 10 client.?.objecter ms_handle_connect 0x7f39940a4920 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 1 -- 172.21.15.98:0/3518148394 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3988043030 con 0x7f39940a4920 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 10 monclient: dump: 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:36.916 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 10 monclient: _finish_auth 0 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f399b4ba640 5 monclient: authenticate success, global_id 124111 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 1 -- 172.21.15.98:0/3518148394 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39880044b0 con 0x7f39940a4920 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 1 -- 172.21.15.98:0/3518148394 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3988039040 con 0x7f39940a4920 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f39877fe640 4 set_mon_vals no callback set 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f3985ffb640 10 monclient: dump: 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f399b4ba640 10 monclient: _renew_subs 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f399b4ba640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f399b4ba640 1 -- 172.21.15.98:0/3518148394 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39940a4180 con 0x7f39940a4920 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f399b4ba640 10 client.124111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f399b4ba640 10 monclient: _renew_subs 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f399b4ba640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f399b4ba640 1 -- 172.21.15.98:0/3518148394 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39940a08b0 con 0x7f39940a4920 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.898+0000 7f399b4ba640 1 librados: init done 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.899+0000 7f399b4ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.899+0000 7f3985ffb640 1 -- 172.21.15.98:0/3518148394 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3988046070 con 0x7f39940a4920 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.899+0000 7f399b4ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.899+0000 7f399b4ba640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:24:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.899+0000 7f399b4ba640 1 -- 172.21.15.98:0/3518148394 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39940008d0 con 0x7f39940a4920 2024-06-11T19:24:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.899+0000 7f3985ffb640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f397803fdb0 0x7f3978042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.899+0000 7f3985ffb640 1 -- 172.21.15.98:0/3518148394 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(621..621 src has 1..621) ==== 6416+0+0 (secure 0 0 0) 0x7f3988044070 con 0x7f39940a4920 2024-06-11T19:24:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.899+0000 7f3985ffb640 10 client.124111.objecter ms_dispatch 0x7f3994000b90 osd_map(621..621 src has 1..621) 2024-06-11T19:24:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.900+0000 7f3985ffb640 3 client.124111.objecter handle_osd_map got epochs [621,621] > 0 2024-06-11T19:24:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.900+0000 7f3985ffb640 3 client.124111.objecter handle_osd_map decoding full epoch 621 2024-06-11T19:24:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.900+0000 7f3985ffb640 20 client.124111.objecter dump_active .. 0 homeless 2024-06-11T19:24:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.900+0000 7f3999256640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f397803fdb0 0x7f3978042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.900+0000 7f3999256640 10 monclient: get_auth_request con 0x7f397803fdb0 auth_method 0 2024-06-11T19:24:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.900+0000 7f3985ffb640 10 client.124111.objecter ms_handle_connect 0x7f397803fdb0 2024-06-11T19:24:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.900+0000 7f3999256640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f397803fdb0 0x7f3978042270 secure :-1 s=READY pgs=4280 cs=0 l=1 rev1=1 crypto rx=0x7f3990000f60 tx=0x7f3990009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.904+0000 7f3985ffb640 1 -- 172.21.15.98:0/3518148394 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f398804d020 con 0x7f39940a4920 2024-06-11T19:24:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.904+0000 7f3985ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:36.904+0000 7f3985ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.166+0000 7f399b4ba640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:24:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.166+0000 7f399b4ba640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.166+0000 7f399b4ba640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:24:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.166+0000 7f399b4ba640 1 -- 172.21.15.98:0/3518148394 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f39940a4d00 con 0x7f39940a4920 2024-06-11T19:24:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.167+0000 7f3985ffb640 1 -- 172.21.15.98:0/3518148394 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f398803de60 con 0x7f39940a4920 2024-06-11T19:24:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.167+0000 7f3985ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.167+0000 7f3985ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:37.167 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":82,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:24:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 10 librados: watch_flush enter 2024-06-11T19:24:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 10 librados: watch_flush exit 2024-06-11T19:24:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 20 client.124111.objecter shutdown clearing up homeless session... 2024-06-11T19:24:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 10 client.124111.objecter successfully canceled tick 2024-06-11T19:24:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 1 -- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f397803fdb0 msgr2=0x7f3978042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f397803fdb0 0x7f3978042270 secure :-1 s=READY pgs=4280 cs=0 l=1 rev1=1 crypto rx=0x7f3990000f60 tx=0x7f3990009040 comp rx=0 tx=0).stop 2024-06-11T19:24:37.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 10 monclient: shutdown 2024-06-11T19:24:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 1 -- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39940a4920 msgr2=0x7f3994135bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39940a4920 0x7f3994135bd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3988033000 tx=0x7f39880046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 1 -- 172.21.15.98:0/3518148394 shutdown_connections 2024-06-11T19:24:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f397803fdb0 0x7f3978042270 unknown :-1 s=CLOSED pgs=4280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f399413e1a0 0x7f3994136110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39940a4920 0x7f3994135bd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 1 --2- 172.21.15.98:0/3518148394 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f39940a2530 0x7f39941cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 1 -- 172.21.15.98:0/3518148394 >> 172.21.15.98:0/3518148394 conn(0x7f39940ab850 msgr2=0x7f39940a61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 1 -- 172.21.15.98:0/3518148394 shutdown_connections 2024-06-11T19:24:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.169+0000 7f399b4ba640 1 -- 172.21.15.98:0/3518148394 wait complete. 2024-06-11T19:24:37.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.170+0000 7f399b4ba640 1 librados: shutdown 2024-06-11T19:24:37.179 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:24:37.179 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:24:37.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:24:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.279+0000 7f9855f19640 10 monclient: get_monmap_and_config 2024-06-11T19:24:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.279+0000 7f9855f19640 10 monclient: build_initial_monmap 2024-06-11T19:24:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.279+0000 7f9855f19640 10 monclient: monmap: 2024-06-11T19:24:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:37.280935+0000 2024-06-11T19:24:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:37.280935+0000 2024-06-11T19:24:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:37.280 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.280+0000 7f9855f19640 1 Processor -- start 2024-06-11T19:24:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.280+0000 7f9855f19640 1 -- start start 2024-06-11T19:24:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.280+0000 7f9855f19640 10 monclient: init 2024-06-11T19:24:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 10 monclient: _add_conns ranks=[5,0,3,4,8,7,6,1,2,10,9] 2024-06-11T19:24:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98500a4920 0x7f98500a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 10 monclient(hunting): picked mon.noname-h con 0x7f98500a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98500a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:24:37.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f98500a2530 0x7f98500a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 10 monclient(hunting): picked mon.noname-g con 0x7f98500a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:37.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98500a7140 con 0x7f98500a5240 2024-06-11T19:24:37.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f98500a2b50 con 0x7f98500a2530 2024-06-11T19:24:37.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f98500a2e30 con 0x7f98500a4920 2024-06-11T19:24:37.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:37.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 10 monclient(hunting): authenticate will time out at 1687.088212s 2024-06-11T19:24:37.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984effd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f98500a5240 0x7f98500a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53770/0 (socket says 172.21.15.98:53770) 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984effd640 1 -- 172.21.15.98:0/1028915277 learned_addr learned my addr 172.21.15.98:0/1028915277 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 909257702 0 0) 0x7f98500a7140 con 0x7f98500a5240 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient(hunting): _negotiate 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient(hunting): my global_id is 65961 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f983c003520 con 0x7f98500a5240 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984f7fe640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98500a4920 0x7f98500a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984f7fe640 10 monclient(hunting): get_auth_request con 0x7f98500a4920 auth_method 0 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984ffff640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f98500a2530 0x7f98500a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984ffff640 10 monclient(hunting): get_auth_request con 0x7f98500a2530 auth_method 0 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1881307968 0 0) 0x7f983c003520 con 0x7f98500a5240 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98500a4920 msgr2=0x7f98500a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98500a4920 0x7f98500a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f98500a2530 msgr2=0x7f98500a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f98500a2530 0x7f98500a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98500a3110 con 0x7f98500a5240 2024-06-11T19:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient: _finish_auth 0 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7f9838001960 con 0x7f98500a5240 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 5 monclient: authenticate success, global_id 65961 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984f7fe640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98500a4920 0x7f98500a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient: dump: 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 >> v1:172.21.15.98:6789/0 conn(0x7f98500a5240 legacy=0x7f98500a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient: _add_conns ranks=[7,5,4,1,2,0,8,3,6] 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f983c004200 0x7f983c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient(hunting): picked mon.i con 0x7f983c004200 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98500a4920 0x7f983c004e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient(hunting): picked mon.h con 0x7f98500a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f983c0099b0 0x7f983c00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient(hunting): picked mon.c con 0x7f983c0099b0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f98500a2b50 con 0x7f983c0099b0 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f98500a2e30 con 0x7f98500a4920 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f983c00c390 con 0x7f983c004200 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f984e7fc640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.281+0000 7f9855f19640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f983c0099b0 0x7f983c00bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 10 monclient(hunting): get_auth_request con 0x7f983c0099b0 auth_method 0 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984f7fe640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f983c004200 0x7f983c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984effd640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98500a4920 0x7f983c004e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984f7fe640 10 monclient(hunting): get_auth_request con 0x7f983c004200 auth_method 0 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984effd640 10 monclient(hunting): get_auth_request con 0x7f98500a4920 auth_method 0 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 10 monclient(hunting): handle_auth_done global_id 65961 payload 293 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 1 -- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f983c004200 msgr2=0x7f983c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f983c004200 0x7f983c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 1 -- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98500a4920 msgr2=0x7f983c004e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98500a4920 0x7f983c004e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 1 monclient: found mon.c 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 1 -- 172.21.15.98:0/1028915277 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f983c003520 con 0x7f983c0099b0 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 10 monclient: _finish_auth 0 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.282+0000 7f984ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:37.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f984ffff640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f983c0099b0 0x7f983c00bda0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9848007360 tx=0x7f983c005830 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ed0811c3256fb0a3 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9848039070 con 0x7f983c0099b0 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f984e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f984e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f984e7fc640 10 monclient: dump: 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f9855f19640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9848032340 con 0x7f983c0099b0 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f984e7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f9855f19640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f9855f19640 4 set_mon_vals no callback set 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f984e7fc640 1 -- 172.21.15.98:0/1028915277 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9848009be0 con 0x7f983c0099b0 2024-06-11T19:24:37.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f984e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f984e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f984e7fc640 10 monclient: dump: 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.306 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f9855f19640 10 monclient: shutdown 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f9855f19640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f9855f19640 1 -- 172.21.15.98:0/1028915277 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f983c0099b0 msgr2=0x7f983c00bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f9855f19640 1 --2- 172.21.15.98:0/1028915277 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f983c0099b0 0x7f983c00bda0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9848007360 tx=0x7f983c005830 comp rx=0 tx=0).stop 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f984ffff640 1 -- 172.21.15.98:0/1028915277 reap_dead start 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f9855f19640 1 -- 172.21.15.98:0/1028915277 shutdown_connections 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f9855f19640 1 -- 172.21.15.98:0/1028915277 >> 172.21.15.98:0/1028915277 conn(0x7f98500ab850 msgr2=0x7f98500a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:37.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f9855f19640 1 -- 172.21.15.98:0/1028915277 shutdown_connections 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.283+0000 7f9855f19640 1 -- 172.21.15.98:0/1028915277 wait complete. 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.284+0000 7f9855f19640 10 monclient: build_initial_monmap 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.284+0000 7f9855f19640 10 monclient: monmap: 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:37.308 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.284+0000 7f9855f19640 1 librados: starting msgr at 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.284+0000 7f9855f19640 1 librados: starting objecter 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.284+0000 7f9855f19640 1 Processor -- start 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.284+0000 7f9855f19640 1 -- start start 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.284+0000 7f9855f19640 1 librados: setting wanted keys 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.284+0000 7f9855f19640 1 librados: calling monclient init 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.284+0000 7f9855f19640 10 monclient: init 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:37.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 10 monclient: _add_conns ranks=[6,4,5,2,8,7,0,3,1] 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f983c0099b0 0x7f98501cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 10 monclient(hunting): picked mon.noname-d con 0x7f983c0099b0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9850137a80 0x7f9850137e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 10 monclient(hunting): picked mon.noname-c con 0x7f9850137a80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98501215f0 0x7f9850121a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 10 monclient(hunting): picked mon.noname-h con 0x7f98501215f0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f98500a6da0 con 0x7f9850137a80 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f9850138e50 con 0x7f98501215f0 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f98500a65f0 con 0x7f983c0099b0 2024-06-11T19:24:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9855f19640 10 monclient(hunting): authenticate will time out at 1687.092362s 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9850137a80 0x7f9850137e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984f7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f983c0099b0 0x7f98501cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9850137a80 0x7f9850137e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46804/0 (socket says 172.21.15.98:46804) 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 1 -- 172.21.15.98:0/1740540544 learned_addr learned my addr 172.21.15.98:0/1740540544 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 10 monclient(hunting): get_auth_request con 0x7f9850137a80 auth_method 0 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984f7fe640 10 monclient(hunting): get_auth_request con 0x7f983c0099b0 auth_method 0 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984ffff640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98501215f0 0x7f9850121a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984ffff640 10 monclient(hunting): get_auth_request con 0x7f98501215f0 auth_method 0 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 10 monclient(hunting): handle_auth_done global_id 114358 payload 824 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 1 -- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f983c0099b0 msgr2=0x7f98501cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f983c0099b0 0x7f98501cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 1 -- 172.21.15.98:0/1740540544 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98501215f0 msgr2=0x7f9850121a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f98501215f0 0x7f9850121a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 10 monclient: _finish_hunting 0 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 1 monclient: found mon.noname-c 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 1 -- 172.21.15.98:0/1740540544 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98500a60b0 con 0x7f9850137a80 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984f7fe640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f983c0099b0 0x7f98501cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:37.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f984effd640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9850137a80 0x7f9850137e60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9838007310 tx=0x7f98380077d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9847fff640 10 client.?.objecter ms_handle_connect 0x7f9850137a80 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.285+0000 7f9847fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f983800fdb0 con 0x7f9850137a80 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 10 monclient: dump: 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.314 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 10 monclient: _finish_auth 0 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9838018070 con 0x7f9850137a80 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9855f19640 5 monclient: authenticate success, global_id 114358 2024-06-11T19:24:37.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9855f19640 10 monclient: _renew_subs 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9855f19640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9855f19640 1 -- 172.21.15.98:0/1740540544 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9850125160 con 0x7f9850137a80 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9838013860 con 0x7f9850137a80 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f984dffb640 4 set_mon_vals no callback set 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 10 monclient: dump: 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.316 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9855f19640 10 client.114358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9855f19640 10 monclient: _renew_subs 2024-06-11T19:24:37.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9855f19640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9855f19640 1 -- 172.21.15.98:0/1740540544 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f98500aac20 con 0x7f9850137a80 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9855f19640 1 librados: init done 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.286+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f98380133f0 con 0x7f9850137a80 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9855f19640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9855f19640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9850121f90 0x7f985009d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9855f19640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9855f19640 1 -- 172.21.15.98:0/1740540544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9850050830 con 0x7f9850121f90 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f983003fdb0 0x7f9830042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(621..621 src has 1..621) ==== 6416+0+0 (secure 0 0 0) 0x7f9838010040 con 0x7f9850137a80 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 10 client.114358.objecter ms_dispatch 0x7f9850000b90 osd_map(621..621 src has 1..621) 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 3 client.114358.objecter handle_osd_map got epochs [621,621] > 0 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 3 client.114358.objecter handle_osd_map decoding full epoch 621 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984f7fe640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9850121f90 0x7f985009d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984f7fe640 10 monclient: get_auth_request con 0x7f9850121f90 auth_method 0 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 20 client.114358.objecter dump_active .. 0 homeless 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984ffff640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f983003fdb0 0x7f9830042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984ffff640 10 monclient: get_auth_request con 0x7f983003fdb0 auth_method 0 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984f7fe640 10 monclient(hunting): handle_auth_done global_id 65970 payload 824 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984f7fe640 1 -- 172.21.15.98:0/1740540544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98500a4d00 con 0x7f9850121f90 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984f7fe640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9850121f90 0x7f985009d250 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9840006fd0 tx=0x7f9840014ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 10 client.114358.objecter ms_handle_connect 0x7f9850121f90 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 10 client.114358.objecter resend_mon_ops 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f984000f040 con 0x7f9850121f90 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 10 monclient: dump: 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.320 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 10 client.114358.objecter ms_handle_connect 0x7f983003fdb0 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f984ffff640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f983003fdb0 0x7f9830042270 secure :-1 s=READY pgs=4281 cs=0 l=1 rev1=1 crypto rx=0x7f9848036000 tx=0x7f9848030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9840012d70 con 0x7f9850121f90 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:37.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9850121f90 msgr2=0x7f985009d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9850121f90 0x7f985009d250 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9840006fd0 tx=0x7f9840014ee0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9850121f90 msgr2=0x7f985009d250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.287+0000 7f9847fff640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9850121f90 0x7f985009d250 secure :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9840006fd0 tx=0x7f9840014ee0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f9855f19640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f9855f19640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f985009d790 0x7f985009dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f9855f19640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f9855f19640 1 -- 172.21.15.98:0/1740540544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98501383a0 con 0x7f985009d790 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f984f7fe640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f985009d790 0x7f985009dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f984f7fe640 10 monclient: get_auth_request con 0x7f985009d790 auth_method 0 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f984f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f984f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f984f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f984f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f984f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f984f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f984f7fe640 10 monclient(hunting): handle_auth_done global_id 65979 payload 824 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f984f7fe640 1 -- 172.21.15.98:0/1740540544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9850122370 con 0x7f985009d790 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f984f7fe640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f985009d790 0x7f985009dba0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9840006fd0 tx=0x7f9840004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f9847fff640 10 client.114358.objecter ms_handle_connect 0x7f985009d790 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f9847fff640 10 client.114358.objecter resend_mon_ops 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9840004490 con 0x7f985009d790 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f9847fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f9847fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.294+0000 7f9847fff640 10 monclient: dump: 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.323 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f9840024570 con 0x7f985009d790 2024-06-11T19:24:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9847fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9847fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f985009d790 msgr2=0x7f985009dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9847fff640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f985009d790 0x7f985009dba0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9840006fd0 tx=0x7f9840004730 comp rx=0 tx=0).stop 2024-06-11T19:24:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9847fff640 1 -- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f985009d790 msgr2=0x7f985009dba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:37.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9847fff640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f985009d790 0x7f985009dba0 secure :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9840006fd0 tx=0x7f9840004730 comp rx=0 tx=0).stop 2024-06-11T19:24:37.325 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 669122, 2024-06-11T19:24:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:37.326 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.327 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.328 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.330 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.331 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.332 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:37.334 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.336 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.337 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:37.339 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:37.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.342 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.344 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:24:37.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.348 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.348 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:37.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.348 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:37.348 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:37.348 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:24:37.348 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.348 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.348 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:37.348 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 10 librados: watch_flush enter 2024-06-11T19:24:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 10 librados: watch_flush exit 2024-06-11T19:24:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 20 client.114358.objecter shutdown clearing up homeless session... 2024-06-11T19:24:37.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 10 client.114358.objecter successfully canceled tick 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 1 -- 172.21.15.98:0/1740540544 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f983003fdb0 msgr2=0x7f9830042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f983003fdb0 0x7f9830042270 secure :-1 s=READY pgs=4281 cs=0 l=1 rev1=1 crypto rx=0x7f9848036000 tx=0x7f9848030000 comp rx=0 tx=0).stop 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 10 monclient: shutdown 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 1 -- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9850137a80 msgr2=0x7f9850137e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 1 --2- 172.21.15.98:0/1740540544 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9850137a80 0x7f9850137e60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9838007310 tx=0x7f98380077d0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f984ffff640 1 -- 172.21.15.98:0/1740540544 reap_dead start 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f984ffff640 1 -- 172.21.15.98:0/1740540544 reap_dead start 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 1 -- 172.21.15.98:0/1740540544 shutdown_connections 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 1 -- 172.21.15.98:0/1740540544 >> 172.21.15.98:0/1740540544 conn(0x7f98500ab850 msgr2=0x7f98500a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 1 -- 172.21.15.98:0/1740540544 shutdown_connections 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 1 -- 172.21.15.98:0/1740540544 wait complete. 2024-06-11T19:24:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.295+0000 7f9855f19640 1 librados: shutdown 2024-06-11T19:24:37.352 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:37.352 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:24:37.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:24:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.453+0000 7f604502e640 10 monclient: get_monmap_and_config 2024-06-11T19:24:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.453+0000 7f604502e640 10 monclient: build_initial_monmap 2024-06-11T19:24:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.453+0000 7f604502e640 10 monclient: monmap: 2024-06-11T19:24:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:37.454178+0000 2024-06-11T19:24:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:37.454178+0000 2024-06-11T19:24:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:37.453 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.453+0000 7f604502e640 1 Processor -- start 2024-06-11T19:24:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.453+0000 7f604502e640 1 -- start start 2024-06-11T19:24:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.453+0000 7f604502e640 10 monclient: init 2024-06-11T19:24:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 10 monclient: _add_conns ranks=[2,10,8,6,5,7,9,4,1,3,0] 2024-06-11T19:24:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60400a2f40 0x7f60400a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 10 monclient(hunting): picked mon.noname-b con 0x7f60400a2f40 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f60400a4df0 0x7f60400a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 10 monclient(hunting): picked mon.noname-f con 0x7f60400a4df0 addr v2:172.21.15.117:3300/0 2024-06-11T19:24:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60401366d0 0x7f6040136b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 10 monclient(hunting): picked mon.noname-e con 0x7f60401366d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f60400a5710 con 0x7f60400a4df0 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f60400a3320 con 0x7f60400a2f40 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f60400a2460 con 0x7f60401366d0 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f604502e640 10 monclient(hunting): authenticate will time out at 1687.261248s 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603f59e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60401366d0 0x7f6040136b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60400a2f40 0x7f60400a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603f59e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60401366d0 0x7f6040136b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:53102/0 (socket says 172.21.15.98:53102) 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60400a2f40 0x7f60400a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57196/0 (socket says 172.21.15.98:57196) 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603f59e640 1 -- 172.21.15.98:0/3513375664 learned_addr learned my addr 172.21.15.98:0/3513375664 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603f59e640 10 monclient(hunting): get_auth_request con 0x7f60401366d0 auth_method 0 2024-06-11T19:24:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603f59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603f59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 10 monclient(hunting): get_auth_request con 0x7f60400a2f40 auth_method 0 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603659c640 1 --2- 172.21.15.98:0/3513375664 >> v2:172.21.15.117:3300/0 conn(0x7f60400a4df0 0x7f60400a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603659c640 10 monclient(hunting): get_auth_request con 0x7f60400a4df0 auth_method 0 2024-06-11T19:24:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 10 monclient(hunting): handle_auth_done global_id 85079 payload 274 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 1 -- 172.21.15.98:0/3513375664 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60401366d0 msgr2=0x7f6040136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 1 --2- 172.21.15.98:0/3513375664 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60401366d0 0x7f6040136b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 1 -- 172.21.15.98:0/3513375664 >> v2:172.21.15.117:3300/0 conn(0x7f60400a4df0 msgr2=0x7f60400a51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 1 --2- 172.21.15.98:0/3513375664 >> v2:172.21.15.117:3300/0 conn(0x7f60400a4df0 0x7f60400a51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 10 monclient: _finish_hunting 0 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 1 monclient: found mon.noname-b 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603ed9d640 1 -- 172.21.15.98:0/3513375664 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60400a0ed0 con 0x7f60400a2f40 2024-06-11T19:24:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.454+0000 7f603f59e640 1 --2- 172.21.15.98:0/3513375664 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60401366d0 0x7f6040136b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603ed9d640 1 --2- 172.21.15.98:0/3513375664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60400a2f40 0x7f60400a48b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f60300097d0 tx=0x7f603002b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=735c5ee21d42d1e7 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 1 -- 172.21.15.98:0/3513375664 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6030037070 con 0x7f60400a2f40 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 10 monclient: dump: 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 10 monclient: _finish_auth 0 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 10 monclient: _check_auth_tickets 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 1 -- 172.21.15.98:0/3513375664 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6030030980 con 0x7f60400a2f40 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f604502e640 5 monclient: authenticate success, global_id 85079 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f604502e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f604502e640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 1 -- 172.21.15.98:0/3513375664 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60300313a0 con 0x7f60400a2f40 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f604502e640 4 set_mon_vals no callback set 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f603e59c640 10 monclient: dump: 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.455+0000 7f604502e640 10 monclient: shutdown 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 1 -- 172.21.15.98:0/3513375664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60400a2f40 msgr2=0x7f60400a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 1 --2- 172.21.15.98:0/3513375664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60400a2f40 0x7f60400a48b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f60300097d0 tx=0x7f603002b6e0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 1 -- 172.21.15.98:0/3513375664 shutdown_connections 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 1 --2- 172.21.15.98:0/3513375664 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60401366d0 0x7f6040136b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 1 --2- 172.21.15.98:0/3513375664 >> v2:172.21.15.117:3300/0 conn(0x7f60400a4df0 0x7f60400a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 1 --2- 172.21.15.98:0/3513375664 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60400a2f40 0x7f60400a48b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 1 -- 172.21.15.98:0/3513375664 >> 172.21.15.98:0/3513375664 conn(0x7f60400a95b0 msgr2=0x7f60400a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 1 -- 172.21.15.98:0/3513375664 shutdown_connections 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 1 -- 172.21.15.98:0/3513375664 wait complete. 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 10 monclient: build_initial_monmap 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 10 monclient: monmap: 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 1 librados: starting msgr at 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.456+0000 7f604502e640 1 librados: starting objecter 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 1 Processor -- start 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 1 -- start start 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 1 librados: setting wanted keys 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 1 librados: calling monclient init 2024-06-11T19:24:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 10 monclient: init 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 10 monclient: _add_conns ranks=[0,6,5,8,3,7,1,4,2] 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60400a2f40 0x7f60401cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 10 monclient(hunting): picked mon.noname-a con 0x7f60400a2f40 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60400a4df0 0x7f60401358e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 10 monclient(hunting): picked mon.noname-d con 0x7f60400a4df0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f60401366d0 0x7f6040135e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 10 monclient(hunting): picked mon.noname-h con 0x7f60401366d0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60400a1250 con 0x7f60400a2f40 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f6040137060 con 0x7f60401366d0 2024-06-11T19:24:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f60400a8150 con 0x7f60400a4df0 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f604502e640 10 monclient(hunting): authenticate will time out at 1687.264756s 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60400a2f40 0x7f60401cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603659c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60400a4df0 0x7f60401358e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60400a2f40 0x7f60401cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53614/0 (socket says 172.21.15.98:53614) 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603ed9d640 1 -- 172.21.15.98:0/1072074628 learned_addr learned my addr 172.21.15.98:0/1072074628 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603ed9d640 10 monclient(hunting): get_auth_request con 0x7f60400a2f40 auth_method 0 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603ed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603659c640 10 monclient(hunting): get_auth_request con 0x7f60400a4df0 auth_method 0 2024-06-11T19:24:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603f59e640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f60401366d0 0x7f6040135e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603f59e640 10 monclient(hunting): get_auth_request con 0x7f60401366d0 auth_method 0 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603f59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.457+0000 7f603f59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603659c640 10 monclient(hunting): handle_auth_done global_id 124125 payload 824 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603659c640 1 -- 172.21.15.98:0/1072074628 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f60401366d0 msgr2=0x7f6040135e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603659c640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f60401366d0 0x7f6040135e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603659c640 1 -- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60400a2f40 msgr2=0x7f60401cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603659c640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60400a2f40 0x7f60401cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603659c640 10 monclient: _finish_hunting 0 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603659c640 1 monclient: found mon.noname-d 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603659c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603659c640 1 -- 172.21.15.98:0/1072074628 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60400a7cb0 con 0x7f60400a4df0 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603ed9d640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60400a2f40 0x7f60401cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603659c640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60400a4df0 0x7f60401358e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f602800d620 tx=0x7f602800dae0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 10 client.?.objecter ms_handle_connect 0x7f60400a4df0 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f602800ba90 con 0x7f60400a4df0 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 10 monclient: dump: 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 10 monclient: _finish_auth 0 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6028019070 con 0x7f60400a4df0 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f604502e640 5 monclient: authenticate success, global_id 124125 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f604502e640 10 monclient: _renew_subs 2024-06-11T19:24:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f604502e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f604502e640 1 -- 172.21.15.98:0/1072074628 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f604006daf0 con 0x7f60400a4df0 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f602800bc70 con 0x7f60400a4df0 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f603dd9b640 4 set_mon_vals no callback set 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f6037fff640 10 monclient: dump: 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f604502e640 10 client.124125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f604502e640 10 monclient: _renew_subs 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f604502e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f604502e640 1 -- 172.21.15.98:0/1072074628 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f6040126a40 con 0x7f60400a4df0 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.458+0000 7f604502e640 1 librados: init done 2024-06-11T19:24:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f604502e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f604502e640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60401cddc0 0x7f604006bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f604502e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f604502e640 1 -- 172.21.15.98:0/1072074628 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f60400a13d0 con 0x7f60401cddc0 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603ed9d640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60401cddc0 0x7f604006bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f602800b620 con 0x7f60400a4df0 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603ed9d640 10 monclient: get_auth_request con 0x7f60401cddc0 auth_method 0 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603ed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f6037fff640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f602003fdb0 0x7f6020042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603ed9d640 10 monclient(hunting): handle_auth_done global_id 85088 payload 824 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(622..622 src has 1..622) ==== 6789+0+0 (secure 0 0 0) 0x7f602800f040 con 0x7f60400a4df0 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f6037fff640 10 client.124125.objecter ms_dispatch 0x7f6040000b90 osd_map(622..622 src has 1..622) 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f6037fff640 3 client.124125.objecter handle_osd_map got epochs [622,622] > 0 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f6037fff640 3 client.124125.objecter handle_osd_map decoding full epoch 622 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603ed9d640 1 -- 172.21.15.98:0/1072074628 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60401373f0 con 0x7f60401cddc0 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f6037fff640 20 client.124125.objecter dump_active .. 0 homeless 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f6037fff640 10 client.124125.objecter ms_handle_connect 0x7f60401cddc0 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f6037fff640 10 client.124125.objecter resend_mon_ops 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603ed9d640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60401cddc0 0x7f604006bf60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f603002b720 tx=0x7f60300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603f59e640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f602003fdb0 0x7f6020042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.459+0000 7f603f59e640 10 monclient: get_auth_request con 0x7f602003fdb0 auth_method 0 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6030037040 con 0x7f60401cddc0 2024-06-11T19:24:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f6037fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f6037fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f6037fff640 10 monclient: dump: 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.483 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.484 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.484 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6030038040 con 0x7f60401cddc0 2024-06-11T19:24:37.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f6037fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:37.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f6037fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:37.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60401cddc0 msgr2=0x7f604006bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f6037fff640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60401cddc0 0x7f604006bf60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f603002b720 tx=0x7f60300046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60401cddc0 msgr2=0x7f604006bf60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:37.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f6037fff640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f60401cddc0 0x7f604006bf60 secure :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f603002b720 tx=0x7f60300046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f6037fff640 10 client.124125.objecter ms_handle_connect 0x7f602003fdb0 2024-06-11T19:24:37.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.460+0000 7f603f59e640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f602003fdb0 0x7f6020042270 secure :-1 s=READY pgs=4282 cs=0 l=1 rev1=1 crypto rx=0x7f60380072e0 tx=0x7f603800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.466+0000 7f604502e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:37.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.466+0000 7f604502e640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604006c4a0 0x7f604006c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.466+0000 7f604502e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.466+0000 7f604502e640 1 -- 172.21.15.98:0/1072074628 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6040050830 con 0x7f604006c4a0 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.466+0000 7f603ed9d640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604006c4a0 0x7f604006c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.466+0000 7f603ed9d640 10 monclient: get_auth_request con 0x7f604006c4a0 auth_method 0 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.466+0000 7f603ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.466+0000 7f603ed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.466+0000 7f603ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.466+0000 7f603ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.466+0000 7f603ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.466+0000 7f603ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f603ed9d640 10 monclient(hunting): handle_auth_done global_id 85097 payload 824 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f603ed9d640 1 -- 172.21.15.98:0/1072074628 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f604006cdf0 con 0x7f604006c4a0 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 10 client.124125.objecter ms_handle_connect 0x7f604006c4a0 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 10 client.124125.objecter resend_mon_ops 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f603ed9d640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604006c4a0 0x7f604006c8b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6040075520 tx=0x7f6030030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6030003c90 con 0x7f604006c4a0 2024-06-11T19:24:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 10 monclient: dump: 2024-06-11T19:24:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.489 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9275 (secure 0 0 0) 0x7f6030003820 con 0x7f604006c4a0 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604006c4a0 msgr2=0x7f604006c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604006c4a0 0x7f604006c8b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6040075520 tx=0x7f6030030000 comp rx=0 tx=0).stop 2024-06-11T19:24:37.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 1 -- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604006c4a0 msgr2=0x7f604006c8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:37.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f6037fff640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f604006c4a0 0x7f604006c8b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.491 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:24:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 549241, 2024-06-11T19:24:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:37.511 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f604502e640 10 librados: watch_flush enter 2024-06-11T19:24:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f604502e640 10 librados: watch_flush exit 2024-06-11T19:24:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f604502e640 20 client.124125.objecter shutdown clearing up homeless session... 2024-06-11T19:24:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f604502e640 10 client.124125.objecter successfully canceled tick 2024-06-11T19:24:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f604502e640 1 -- 172.21.15.98:0/1072074628 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f602003fdb0 msgr2=0x7f6020042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f604502e640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f602003fdb0 0x7f6020042270 secure :-1 s=READY pgs=4282 cs=0 l=1 rev1=1 crypto rx=0x7f60380072e0 tx=0x7f603800b040 comp rx=0 tx=0).stop 2024-06-11T19:24:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f604502e640 10 monclient: shutdown 2024-06-11T19:24:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f604502e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f604502e640 1 -- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60400a4df0 msgr2=0x7f60401358e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f604502e640 1 --2- 172.21.15.98:0/1072074628 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60400a4df0 0x7f60401358e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f602800d620 tx=0x7f602800dae0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f603f59e640 1 -- 172.21.15.98:0/1072074628 reap_dead start 2024-06-11T19:24:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f603f59e640 1 -- 172.21.15.98:0/1072074628 reap_dead start 2024-06-11T19:24:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f604502e640 1 -- 172.21.15.98:0/1072074628 shutdown_connections 2024-06-11T19:24:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.467+0000 7f604502e640 1 -- 172.21.15.98:0/1072074628 >> 172.21.15.98:0/1072074628 conn(0x7f60400a95b0 msgr2=0x7f60401bd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.468+0000 7f604502e640 1 -- 172.21.15.98:0/1072074628 shutdown_connections 2024-06-11T19:24:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.468+0000 7f604502e640 1 -- 172.21.15.98:0/1072074628 wait complete. 2024-06-11T19:24:37.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.468+0000 7f604502e640 1 librados: shutdown 2024-06-11T19:24:37.514 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:37.514 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:24:37.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:24:37.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.614+0000 7fa2bf6a4640 10 monclient: get_monmap_and_config 2024-06-11T19:24:37.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.614+0000 7fa2bf6a4640 10 monclient: build_initial_monmap 2024-06-11T19:24:37.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.614+0000 7fa2bf6a4640 10 monclient: monmap: 2024-06-11T19:24:37.613 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:37.615660+0000 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:37.615660+0000 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:37.615 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:37.615 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:37.615 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 1 Processor -- start 2024-06-11T19:24:37.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 1 -- start start 2024-06-11T19:24:37.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 10 monclient: init 2024-06-11T19:24:37.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:37.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 10 monclient: _add_conns ranks=[0,3,2,5,6,4,10,7,1,8,9] 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa2b80a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2b80a2530 0x7fa2b813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 10 monclient(hunting): picked mon.noname-g con 0x7fa2b80a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2b813e1a0 0x7fa2b8140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 10 monclient(hunting): picked mon.noname-b con 0x7fa2b813e1a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2b80a7140 con 0x7fa2b80a4920 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b80a2b50 con 0x7fa2b813e1a0 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b80a2e30 con 0x7fa2b80a2530 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bf6a4640 10 monclient(hunting): authenticate will time out at 1687.422891s 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bdc41640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2b813e1a0 0x7fa2b8140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.615+0000 7fa2bd440640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa2b80a4920 0x7fa2b80a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53774/0 (socket says 172.21.15.98:53774) 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bd440640 1 -- 172.21.15.98:0/2800625903 learned_addr learned my addr 172.21.15.98:0/2800625903 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 10 monclient(hunting): get_auth_request con 0x7fa2b813e1a0 auth_method 0 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2affff640 1 -- 172.21.15.98:0/2800625903 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690289928 0 0) 0x7fa2b80a7140 con 0x7fa2b80a4920 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2affff640 10 monclient(hunting): _negotiate 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2affff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2affff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2affff640 10 monclient(hunting): my global_id is 65997 2024-06-11T19:24:37.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bcc3f640 1 --2- 172.21.15.98:0/2800625903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2b80a2530 0x7fa2b813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2affff640 1 -- 172.21.15.98:0/2800625903 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2a4003520 con 0x7fa2b80a4920 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 10 monclient(hunting): handle_auth_done global_id 85106 payload 274 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 1 -- 172.21.15.98:0/2800625903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2b80a2530 msgr2=0x7fa2b813dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 1 --2- 172.21.15.98:0/2800625903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2b80a2530 0x7fa2b813dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 1 -- 172.21.15.98:0/2800625903 >> v1:172.21.15.98:6789/0 conn(0x7fa2b80a4920 legacy=0x7fa2b80a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 10 monclient: _finish_hunting 0 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 1 monclient: found mon.noname-b 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bdc41640 1 -- 172.21.15.98:0/2800625903 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b80a3110 con 0x7fa2b813e1a0 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bcc3f640 10 monclient: get_auth_request con 0x7fa2b80a2530 auth_method 0 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.616+0000 7fa2bcc3f640 1 --2- 172.21.15.98:0/2800625903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2b80a2530 0x7fa2b813dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:24:37.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bdc41640 1 --2- 172.21.15.98:0/2800625903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2b813e1a0 0x7fa2b8140590 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009dc0 tx=0x7fa2b002bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db0fc0e76247aa9c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 1 -- 172.21.15.98:0/2800625903 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2b0039070 con 0x7fa2b813e1a0 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 10 monclient: dump: 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.628 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 10 monclient: _finish_auth 0 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 5 monclient: authenticate success, global_id 85106 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 1 -- 172.21.15.98:0/2800625903 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2b002cd70 con 0x7fa2b813e1a0 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:37.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 4 set_mon_vals no callback set 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 1 -- 172.21.15.98:0/2800625903 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2b00327b0 con 0x7fa2b813e1a0 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2affff640 10 monclient: dump: 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.630 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 10 monclient: shutdown 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/2800625903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2b813e1a0 msgr2=0x7fa2b8140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 1 --2- 172.21.15.98:0/2800625903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2b813e1a0 0x7fa2b8140590 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009dc0 tx=0x7fa2b002bd50 comp rx=0 tx=0).stop 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/2800625903 shutdown_connections 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 1 --2- 172.21.15.98:0/2800625903 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2b813e1a0 0x7fa2b8140590 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 1 --2- 172.21.15.98:0/2800625903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2b80a2530 0x7fa2b813dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/2800625903 >> 172.21.15.98:0/2800625903 conn(0x7fa2b80ab850 msgr2=0x7fa2b80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:37.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/2800625903 shutdown_connections 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.617+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/2800625903 wait complete. 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.618+0000 7fa2bf6a4640 10 monclient: build_initial_monmap 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.618+0000 7fa2bf6a4640 10 monclient: monmap: 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:37.632 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.618+0000 7fa2bf6a4640 1 librados: starting msgr at 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.618+0000 7fa2bf6a4640 1 librados: starting objecter 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.618+0000 7fa2bf6a4640 1 Processor -- start 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.618+0000 7fa2bf6a4640 1 -- start start 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.618+0000 7fa2bf6a4640 1 librados: setting wanted keys 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.618+0000 7fa2bf6a4640 1 librados: calling monclient init 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.618+0000 7fa2bf6a4640 10 monclient: init 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 10 monclient: _add_conns ranks=[6,2,7,3,5,1,8,4,0] 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2b80a2530 0x7fa2b81cd620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 10 monclient(hunting): picked mon.noname-d con 0x7fa2b80a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2b80a4920 0x7fa2b8136ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 10 monclient(hunting): picked mon.noname-b con 0x7fa2b80a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa2b813e1a0 0x7fa2b8137000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 10 monclient(hunting): picked mon.noname-i con 0x7fa2b813e1a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b8138e50 con 0x7fa2b80a4920 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa2b80a84b0 con 0x7fa2b80a2530 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa2b80a6da0 con 0x7fa2b813e1a0 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 10 monclient(hunting): authenticate will time out at 1687.426256s 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bcc3f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2b80a4920 0x7fa2b8136ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2b80a2530 0x7fa2b81cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bcc3f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2b80a4920 0x7fa2b8136ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57250/0 (socket says 172.21.15.98:57250) 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2b80a2530 0x7fa2b81cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:35272/0 (socket says 172.21.15.98:35272) 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bcc3f640 1 -- 172.21.15.98:0/1014497763 learned_addr learned my addr 172.21.15.98:0/1014497763 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bcc3f640 10 monclient(hunting): get_auth_request con 0x7fa2b80a4920 auth_method 0 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bcc3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bcc3f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bcc3f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 10 monclient(hunting): get_auth_request con 0x7fa2b80a2530 auth_method 0 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bcc3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bcc3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bcc3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 10 monclient(hunting): handle_auth_done global_id 124134 payload 824 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 1 -- 172.21.15.98:0/1014497763 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa2b813e1a0 msgr2=0x7fa2b8137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa2b813e1a0 0x7fa2b8137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 1 -- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2b80a4920 msgr2=0x7fa2b8136ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2b80a4920 0x7fa2b8136ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 10 monclient: _finish_hunting 0 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 1 monclient: found mon.noname-d 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 1 -- 172.21.15.98:0/1014497763 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b813de80 con 0x7fa2b80a2530 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bcc3f640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2b80a4920 0x7fa2b8136ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bd440640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2b80a2530 0x7fa2b81cd620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400c460 tx=0x7fa2b400c920 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 10 client.?.objecter ms_handle_connect 0x7fa2b80a2530 2024-06-11T19:24:37.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2b4017070 con 0x7fa2b80a2530 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 10 monclient: dump: 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.637 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 10 monclient: _finish_auth 0 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 10 monclient: _check_auth_tickets 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2b4007070 con 0x7fa2b80a2530 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 5 monclient: authenticate success, global_id 124134 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2b40076d0 con 0x7fa2b80a2530 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 10 monclient: _renew_subs 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/1014497763 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b80a1700 con 0x7fa2b80a2530 2024-06-11T19:24:37.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2af7fe640 4 set_mon_vals no callback set 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.619+0000 7fa2adffb640 10 monclient: dump: 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.639 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.620+0000 7fa2bf6a4640 10 client.124134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.620+0000 7fa2bf6a4640 10 monclient: _renew_subs 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.620+0000 7fa2bf6a4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.620+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/1014497763 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b8122bd0 con 0x7fa2b80a2530 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.620+0000 7fa2bf6a4640 1 librados: init done 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.620+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fa2b4007260 con 0x7fa2b80a2530 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.620+0000 7fa2bf6a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.620+0000 7fa2bf6a4640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b809fb20 0x7fa2b809ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.620+0000 7fa2bf6a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.620+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/1014497763 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa2b8050830 con 0x7fa2b809fb20 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa29c03fdb0 0x7fa29c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bcc3f640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b809fb20 0x7fa2b809ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bcc3f640 10 monclient: get_auth_request con 0x7fa2b809fb20 auth_method 0 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bcc3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bcc3f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bcc3f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(622..622 src has 1..622) ==== 6789+0+0 (secure 0 0 0) 0x7fa2b4050dd0 con 0x7fa2b80a2530 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 10 client.124134.objecter ms_dispatch 0x7fa2b8000b90 osd_map(622..622 src has 1..622) 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 3 client.124134.objecter handle_osd_map got epochs [622,622] > 0 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 3 client.124134.objecter handle_osd_map decoding full epoch 622 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bcc3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bcc3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bcc3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 20 client.124134.objecter dump_active .. 0 homeless 2024-06-11T19:24:37.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bdc41640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa29c03fdb0 0x7fa29c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bdc41640 10 monclient: get_auth_request con 0x7fa29c03fdb0 auth_method 0 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bcc3f640 10 monclient(hunting): handle_auth_done global_id 114367 payload 824 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bcc3f640 1 -- 172.21.15.98:0/1014497763 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2b80a4d00 con 0x7fa2b809fb20 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 10 client.124134.objecter ms_handle_connect 0x7fa2b809fb20 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 10 client.124134.objecter resend_mon_ops 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bcc3f640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b809fb20 0x7fa2b809ff00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80a1300 tx=0x7fa2a800d590 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2a800fe70 con 0x7fa2b809fb20 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 10 monclient: dump: 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.642 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 10 client.124134.objecter ms_handle_connect 0x7fa29c03fdb0 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2bdc41640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa29c03fdb0 0x7fa29c042270 secure :-1 s=READY pgs=4283 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0036000 tx=0x7fa2b002e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa2a800fe70 con 0x7fa2b809fb20 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:37.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b809fb20 msgr2=0x7fa2b809ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b809fb20 0x7fa2b809ff00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80a1300 tx=0x7fa2a800d590 comp rx=0 tx=0).stop 2024-06-11T19:24:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b809fb20 msgr2=0x7fa2b809ff00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:37.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.621+0000 7fa2adffb640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b809fb20 0x7fa2b809ff00 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80a1300 tx=0x7fa2a800d590 comp rx=0 tx=0).stop 2024-06-11T19:24:37.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bf6a4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:37.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bf6a4640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b80a0440 0x7fa2b813bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bf6a4640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/1014497763 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa2b8069740 con 0x7fa2b80a0440 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bcc3f640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b80a0440 0x7fa2b813bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bcc3f640 10 monclient: get_auth_request con 0x7fa2b80a0440 auth_method 0 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bcc3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bcc3f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bcc3f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bcc3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bcc3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bcc3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bcc3f640 10 monclient(hunting): handle_auth_done global_id 114376 payload 824 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bcc3f640 1 -- 172.21.15.98:0/1014497763 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa2b80aa5e0 con 0x7fa2b80a0440 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2bcc3f640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b80a0440 0x7fa2b813bc30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80a1300 tx=0x7fa2a8004730 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 10 client.124134.objecter ms_handle_connect 0x7fa2b80a0440 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 10 client.124134.objecter resend_mon_ops 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2a8006690 con 0x7fa2b80a0440 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 10 monclient: dump: 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.647 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fa2a8006690 con 0x7fa2b80a0440 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b80a0440 msgr2=0x7fa2b813bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b80a0440 0x7fa2b813bc30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80a1300 tx=0x7fa2a8004730 comp rx=0 tx=0).stop 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 1 -- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b80a0440 msgr2=0x7fa2b813bc30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:37.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.628+0000 7fa2adffb640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2b80a0440 0x7fa2b813bc30 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80a1300 tx=0x7fa2a8004730 comp rx=0 tx=0).stop 2024-06-11T19:24:37.649 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:37.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:37.649 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:37.649 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:37.649 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:37.649 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 165995, 2024-06-11T19:24:37.649 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:37.649 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:37.649 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:37.649 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:37.649 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:37.649 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.650 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.651 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:37.652 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.653 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.655 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:37.657 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:37.659 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:37.661 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.663 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:37.664 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.665 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:37.666 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.668 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.669 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:37.670 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 10 librados: watch_flush enter 2024-06-11T19:24:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 10 librados: watch_flush exit 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 20 client.124134.objecter shutdown clearing up homeless session... 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 10 client.124134.objecter successfully canceled tick 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/1014497763 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa29c03fdb0 msgr2=0x7fa29c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa29c03fdb0 0x7fa29c042270 secure :-1 s=READY pgs=4283 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0036000 tx=0x7fa2b002e000 comp rx=0 tx=0).stop 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 10 monclient: shutdown 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2b80a2530 msgr2=0x7fa2b81cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2b80a2530 0x7fa2b81cd620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400c460 tx=0x7fa2b400c920 comp rx=0 tx=0).stop 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bdc41640 1 -- 172.21.15.98:0/1014497763 reap_dead start 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/1014497763 shutdown_connections 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 1 --2- 172.21.15.98:0/1014497763 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa2b80a2530 0x7fa2b81cd620 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/1014497763 >> 172.21.15.98:0/1014497763 conn(0x7fa2b80ab850 msgr2=0x7fa2b80a9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/1014497763 shutdown_connections 2024-06-11T19:24:37.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 1 -- 172.21.15.98:0/1014497763 wait complete. 2024-06-11T19:24:37.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.629+0000 7fa2bf6a4640 1 librados: shutdown 2024-06-11T19:24:37.673 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:37.673 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:24:37.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:24:37.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.773+0000 7fd0495e2640 10 monclient: get_monmap_and_config 2024-06-11T19:24:37.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.773+0000 7fd0495e2640 10 monclient: build_initial_monmap 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.773+0000 7fd0495e2640 10 monclient: monmap: 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:37.774741+0000 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:37.774741+0000 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:37.774 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 1 Processor -- start 2024-06-11T19:24:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 1 -- start start 2024-06-11T19:24:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 10 monclient: init 2024-06-11T19:24:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 10 monclient: _add_conns ranks=[1,2,10,0,7,3,4,5,8,9,6] 2024-06-11T19:24:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd0441347f0 addr v1:172.21.15.117:6789/0 2024-06-11T19:24:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd04409e6b0 0x7fd04413dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 10 monclient(hunting): picked mon.noname-b con 0x7fd04409e6b0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:37.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd04413e210 0x7fd044140600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 10 monclient(hunting): picked mon.noname-f con 0x7fd04413e210 addr v2:172.21.15.117:3300/0 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd04409d410 con 0x7fd0441347f0 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fd0440687f0 con 0x7fd04413e210 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd04409eaf0 con 0x7fd04409e6b0 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0495e2640 10 monclient(hunting): authenticate will time out at 1687.581826s 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0427fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd04409e6b0 0x7fd04413dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0427fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd04409e6b0 0x7fd04413dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57258/0 (socket says 172.21.15.98:57258) 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0427fc640 1 -- 172.21.15.98:0/2328030110 learned_addr learned my addr 172.21.15.98:0/2328030110 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0427fc640 10 monclient(hunting): get_auth_request con 0x7fd04409e6b0 auth_method 0 2024-06-11T19:24:37.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0427fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.774+0000 7fd0427fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0437fe640 1 --2- 172.21.15.98:0/2328030110 >> v2:172.21.15.117:3300/0 conn(0x7fd04413e210 0x7fd044140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0437fe640 10 monclient(hunting): get_auth_request con 0x7fd04413e210 auth_method 0 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0437fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0437fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 10 monclient(hunting): handle_auth_done global_id 85124 payload 274 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 1 -- 172.21.15.98:0/2328030110 >> v2:172.21.15.117:3300/0 conn(0x7fd04413e210 msgr2=0x7fd044140600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 1 --2- 172.21.15.98:0/2328030110 >> v2:172.21.15.117:3300/0 conn(0x7fd04413e210 0x7fd044140600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 1 -- 172.21.15.98:0/2328030110 >> v1:172.21.15.117:6789/0 conn(0x7fd0441347f0 legacy=0x7fd044134bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 10 monclient: _finish_hunting 0 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 1 monclient: found mon.noname-b 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 1 -- 172.21.15.98:0/2328030110 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04409d6f0 con 0x7fd04409e6b0 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.775+0000 7fd0427fc640 1 --2- 172.21.15.98:0/2328030110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd04409e6b0 0x7fd04413dcd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd038009870 tx=0x7fd03802b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da7b4801bd7f472d server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 1 -- 172.21.15.98:0/2328030110 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd038038070 con 0x7fd04409e6b0 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 10 monclient: dump: 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.786 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 10 monclient: _finish_auth 0 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:37.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 1 -- 172.21.15.98:0/2328030110 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd038031990 con 0x7fd04409e6b0 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 5 monclient: authenticate success, global_id 85124 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 1 -- 172.21.15.98:0/2328030110 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd0380323b0 con 0x7fd04409e6b0 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 4 set_mon_vals no callback set 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd041ffb640 10 monclient: dump: 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.788 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 10 monclient: shutdown 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 1 -- 172.21.15.98:0/2328030110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd04409e6b0 msgr2=0x7fd04413dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 1 --2- 172.21.15.98:0/2328030110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd04409e6b0 0x7fd04413dcd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd038009870 tx=0x7fd03802b7c0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 1 -- 172.21.15.98:0/2328030110 shutdown_connections 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 1 --2- 172.21.15.98:0/2328030110 >> v2:172.21.15.117:3300/0 conn(0x7fd04413e210 0x7fd044140600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 1 --2- 172.21.15.98:0/2328030110 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd04409e6b0 0x7fd04413dcd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 1 -- 172.21.15.98:0/2328030110 >> 172.21.15.98:0/2328030110 conn(0x7fd0440a4780 msgr2=0x7fd0440a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 1 -- 172.21.15.98:0/2328030110 shutdown_connections 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.776+0000 7fd0495e2640 1 -- 172.21.15.98:0/2328030110 wait complete. 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.777+0000 7fd0495e2640 10 monclient: build_initial_monmap 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.777+0000 7fd0495e2640 10 monclient: monmap: 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:37.790 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.777+0000 7fd0495e2640 1 librados: starting msgr at 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.777+0000 7fd0495e2640 1 librados: starting objecter 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.777+0000 7fd0495e2640 1 Processor -- start 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.777+0000 7fd0495e2640 1 -- start start 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.777+0000 7fd0495e2640 1 librados: setting wanted keys 2024-06-11T19:24:37.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.777+0000 7fd0495e2640 1 librados: calling monclient init 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.777+0000 7fd0495e2640 10 monclient: init 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 10 monclient: _add_conns ranks=[0,6,7,4,2,5,8,3,1] 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd04409e6b0 0x7fd0441cd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 10 monclient(hunting): picked mon.noname-a con 0x7fd04409e6b0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd0441347f0 0x7fd0441339b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 10 monclient(hunting): picked mon.noname-d con 0x7fd0441347f0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd04413e210 0x7fd044133ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 10 monclient(hunting): picked mon.noname-i con 0x7fd04413e210 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0440a56f0 con 0x7fd04409e6b0 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd0440a5570 con 0x7fd0441347f0 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fd044068ac0 con 0x7fd04413e210 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0495e2640 10 monclient(hunting): authenticate will time out at 1687.585280s 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd042ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd04409e6b0 0x7fd0441cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd0441347f0 0x7fd0441339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd042ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd04409e6b0 0x7fd0441cd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53616/0 (socket says 172.21.15.98:53616) 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd042ffd640 1 -- 172.21.15.98:0/3128043624 learned_addr learned my addr 172.21.15.98:0/3128043624 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd042ffd640 10 monclient(hunting): get_auth_request con 0x7fd04409e6b0 auth_method 0 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd042ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd042ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 10 monclient(hunting): get_auth_request con 0x7fd0441347f0 auth_method 0 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd042ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd042ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd042ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0437fe640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd04413e210 0x7fd044133ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0437fe640 10 monclient(hunting): get_auth_request con 0x7fd04413e210 auth_method 0 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0437fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0437fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 10 monclient(hunting): handle_auth_done global_id 124143 payload 824 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 1 -- 172.21.15.98:0/3128043624 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd04413e210 msgr2=0x7fd044133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fd04413e210 0x7fd044133ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 1 -- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd04409e6b0 msgr2=0x7fd0441cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd04409e6b0 0x7fd0441cd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 10 monclient: _finish_hunting 0 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 1 monclient: found mon.noname-d 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 1 -- 172.21.15.98:0/3128043624 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0441cdd40 con 0x7fd0441347f0 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd042ffd640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd04409e6b0 0x7fd0441cd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd0427fc640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd0441347f0 0x7fd0441339b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd038033000 tx=0x7fd0380046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd02b7fe640 10 client.?.objecter ms_handle_connect 0x7fd0441347f0 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd02b7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd038042020 con 0x7fd0441347f0 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd02b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd02b7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd02b7fe640 10 monclient: dump: 2024-06-11T19:24:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.796 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd02b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.778+0000 7fd02b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd02b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd0495e2640 5 monclient: authenticate success, global_id 124143 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd038004240 con 0x7fd0441347f0 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd0495e2640 10 monclient: _renew_subs 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd0495e2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd0495e2640 1 -- 172.21.15.98:0/3128043624 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd04411ebd0 con 0x7fd0441347f0 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd02b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd038038040 con 0x7fd0441347f0 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd02b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd0495e2640 10 client.124143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd02b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd02b7fe640 10 monclient: dump: 2024-06-11T19:24:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd0417fa640 4 set_mon_vals no callback set 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd0495e2640 10 monclient: _renew_subs 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd0495e2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd0495e2640 1 -- 172.21.15.98:0/3128043624 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0440a3a70 con 0x7fd0441347f0 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd0495e2640 1 librados: init done 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd038045020 con 0x7fd0441347f0 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd0495e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.779+0000 7fd0495e2640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04411fac0 0x7fd04413c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd0495e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd0495e2640 1 -- 172.21.15.98:0/3128043624 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd044050830 con 0x7fd04411fac0 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd03003fdd0 0x7fd030042290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd042ffd640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04411fac0 0x7fd04413c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd042ffd640 10 monclient: get_auth_request con 0x7fd04411fac0 auth_method 0 2024-06-11T19:24:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd042ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd042ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(622..622 src has 1..622) ==== 6789+0+0 (secure 0 0 0) 0x7fd038030070 con 0x7fd0441347f0 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 10 client.124143.objecter ms_dispatch 0x7fd044000b90 osd_map(622..622 src has 1..622) 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 3 client.124143.objecter handle_osd_map got epochs [622,622] > 0 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 3 client.124143.objecter handle_osd_map decoding full epoch 622 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd042ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd042ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd042ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 20 client.124143.objecter dump_active .. 0 homeless 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd0437fe640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd03003fdd0 0x7fd030042290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd042ffd640 10 monclient(hunting): handle_auth_done global_id 124152 payload 824 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd042ffd640 1 -- 172.21.15.98:0/3128043624 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd04411fea0 con 0x7fd04411fac0 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd0437fe640 10 monclient: get_auth_request con 0x7fd03003fdd0 auth_method 0 2024-06-11T19:24:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 10 client.124143.objecter ms_handle_connect 0x7fd04411fac0 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 10 client.124143.objecter resend_mon_ops 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd042ffd640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04411fac0 0x7fd04413c590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd03400fb10 tx=0x7fd034004580 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd034018020 con 0x7fd04411fac0 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 10 monclient: dump: 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 10 client.124143.objecter ms_handle_connect 0x7fd03003fdd0 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd0437fe640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd03003fdd0 0x7fd030042290 secure :-1 s=READY pgs=4284 cs=0 l=1 rev1=1 crypto rx=0x7fd0440a3f60 tx=0x7fd03c00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd03400db60 con 0x7fd04411fac0 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04411fac0 msgr2=0x7fd04413c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04411fac0 0x7fd04413c590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd03400fb10 tx=0x7fd034004580 comp rx=0 tx=0).stop 2024-06-11T19:24:37.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04411fac0 msgr2=0x7fd04413c590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:37.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.780+0000 7fd02b7fe640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04411fac0 0x7fd04413c590 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd0495e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:37.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd0495e2640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04402cb40 0x7fd04402cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd0495e2640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd0495e2640 1 -- 172.21.15.98:0/3128043624 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd044134bd0 con 0x7fd04402cb40 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd042ffd640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04402cb40 0x7fd04402cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd042ffd640 10 monclient: get_auth_request con 0x7fd04402cb40 auth_method 0 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd042ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd042ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd042ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd042ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd042ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd042ffd640 10 monclient(hunting): handle_auth_done global_id 124161 payload 824 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd042ffd640 1 -- 172.21.15.98:0/3128043624 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd0440a0790 con 0x7fd04402cb40 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd042ffd640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04402cb40 0x7fd04402cf20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd034019000 tx=0x7fd034003760 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 10 client.124143.objecter ms_handle_connect 0x7fd04402cb40 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 10 client.124143.objecter resend_mon_ops 2024-06-11T19:24:37.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd034018040 con 0x7fd04402cb40 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 10 monclient: dump: 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fd034014d90 con 0x7fd04402cb40 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04402cb40 msgr2=0x7fd04402cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04402cb40 0x7fd04402cf20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd034019000 tx=0x7fd034003760 comp rx=0 tx=0).stop 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 1 -- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04402cb40 msgr2=0x7fd04402cf20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.787+0000 7fd02b7fe640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd04402cb40 0x7fd04402cf20 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd034019000 tx=0x7fd034003760 comp rx=0 tx=0).stop 2024-06-11T19:24:37.807 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 20715, 2024-06-11T19:24:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:24:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:37.829 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 10 librados: watch_flush enter 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 10 librados: watch_flush exit 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 20 client.124143.objecter shutdown clearing up homeless session... 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 10 client.124143.objecter successfully canceled tick 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 1 -- 172.21.15.98:0/3128043624 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd03003fdd0 msgr2=0x7fd030042290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd03003fdd0 0x7fd030042290 secure :-1 s=READY pgs=4284 cs=0 l=1 rev1=1 crypto rx=0x7fd0440a3f60 tx=0x7fd03c00b040 comp rx=0 tx=0).stop 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 10 monclient: shutdown 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 1 -- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd0441347f0 msgr2=0x7fd0441339b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd0441347f0 0x7fd0441339b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd038033000 tx=0x7fd0380046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0437fe640 1 -- 172.21.15.98:0/3128043624 reap_dead start 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 1 -- 172.21.15.98:0/3128043624 shutdown_connections 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 1 --2- 172.21.15.98:0/3128043624 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd0441347f0 0x7fd0441339b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 1 -- 172.21.15.98:0/3128043624 >> 172.21.15.98:0/3128043624 conn(0x7fd0440a4780 msgr2=0x7fd0440a2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 1 -- 172.21.15.98:0/3128043624 shutdown_connections 2024-06-11T19:24:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 1 -- 172.21.15.98:0/3128043624 wait complete. 2024-06-11T19:24:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.788+0000 7fd0495e2640 1 librados: shutdown 2024-06-11T19:24:37.832 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:37.832 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:24:37.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:24:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.932+0000 7f351ea7a640 10 monclient: get_monmap_and_config 2024-06-11T19:24:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.932+0000 7f351ea7a640 10 monclient: build_initial_monmap 2024-06-11T19:24:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.932+0000 7f351ea7a640 10 monclient: monmap: 2024-06-11T19:24:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:37.933276+0000 2024-06-11T19:24:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:37.933276+0000 2024-06-11T19:24:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:37.932 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.932+0000 7f351ea7a640 1 Processor -- start 2024-06-11T19:24:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 1 -- start start 2024-06-11T19:24:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 10 monclient: init 2024-06-11T19:24:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 10 monclient: _add_conns ranks=[9,7,0,2,5,3,8,4,1,10,6] 2024-06-11T19:24:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f35180a4920 0x7f35180a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 10 monclient(hunting): picked mon.noname-a con 0x7f35180a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:24:37.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f35180a5240 0x7f35180a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 10 monclient(hunting): picked mon.noname-i con 0x7f35180a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35180a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:24:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35180a7140 con 0x7f35180a2530 2024-06-11T19:24:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f35180a2b50 con 0x7f35180a4920 2024-06-11T19:24:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f35180a2e30 con 0x7f35180a5240 2024-06-11T19:24:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351ea7a640 10 monclient(hunting): authenticate will time out at 1687.740570s 2024-06-11T19:24:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f35180a4920 0x7f35180a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351d017640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f35180a2530 0x7f35180a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53780/0 (socket says 172.21.15.98:53780) 2024-06-11T19:24:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351d017640 1 -- 172.21.15.98:0/3673989675 learned_addr learned my addr 172.21.15.98:0/3673989675 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 10 monclient(hunting): get_auth_request con 0x7f35180a4920 auth_method 0 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f3517fff640 1 --2- 172.21.15.98:0/3673989675 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f35180a5240 0x7f35180a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 10 monclient(hunting): handle_auth_done global_id 66015 payload 274 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 1 -- 172.21.15.98:0/3673989675 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f35180a5240 msgr2=0x7f35180a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 1 --2- 172.21.15.98:0/3673989675 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f35180a5240 0x7f35180a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 1 -- 172.21.15.98:0/3673989675 >> v1:172.21.15.98:6789/0 conn(0x7f35180a2530 legacy=0x7f35180a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 10 monclient: _finish_hunting 0 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 1 monclient: found mon.noname-a 2024-06-11T19:24:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f351c816640 1 -- 172.21.15.98:0/3673989675 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35180a3110 con 0x7f35180a4920 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f35177fe640 1 -- 172.21.15.98:0/3673989675 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4070743155 0 0) 0x7f35180a7140 con 0x7f35180a2530 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f35177fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.933+0000 7f3517fff640 10 monclient: get_auth_request con 0x7f35180a5240 auth_method 0 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f3517fff640 1 --2- 172.21.15.98:0/3673989675 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f35180a5240 0x7f35180a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f351c816640 1 --2- 172.21.15.98:0/3673989675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35180a4920 0x7f35180a4d00 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f35100097a0 tx=0x7f351002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde8b1b4f835cd4f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 1 -- 172.21.15.98:0/3673989675 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3510039070 con 0x7f35180a4920 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 10 monclient: dump: 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 10 monclient: _finish_auth 0 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 10 monclient: _check_auth_tickets 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 1 -- 172.21.15.98:0/3673989675 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3510031960 con 0x7f35180a4920 2024-06-11T19:24:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f351ea7a640 5 monclient: authenticate success, global_id 66015 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f351ea7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f351ea7a640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 1 -- 172.21.15.98:0/3673989675 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3510032380 con 0x7f35180a4920 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f351ea7a640 4 set_mon_vals no callback set 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.934+0000 7f35177fe640 10 monclient: dump: 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.946 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.935+0000 7f351ea7a640 10 monclient: shutdown 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.935+0000 7f351ea7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.935+0000 7f351ea7a640 1 -- 172.21.15.98:0/3673989675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35180a4920 msgr2=0x7f35180a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.935+0000 7f351ea7a640 1 --2- 172.21.15.98:0/3673989675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35180a4920 0x7f35180a4d00 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f35100097a0 tx=0x7f351002b790 comp rx=0 tx=0).stop 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.935+0000 7f351ea7a640 1 -- 172.21.15.98:0/3673989675 shutdown_connections 2024-06-11T19:24:37.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.935+0000 7f351ea7a640 1 --2- 172.21.15.98:0/3673989675 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f35180a5240 0x7f35180a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.935+0000 7f351ea7a640 1 --2- 172.21.15.98:0/3673989675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35180a4920 0x7f35180a4d00 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.935+0000 7f351ea7a640 1 -- 172.21.15.98:0/3673989675 >> 172.21.15.98:0/3673989675 conn(0x7f35180ab850 msgr2=0x7f35180a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.935+0000 7f351ea7a640 1 -- 172.21.15.98:0/3673989675 shutdown_connections 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.935+0000 7f351ea7a640 1 -- 172.21.15.98:0/3673989675 wait complete. 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.935+0000 7f351ea7a640 10 monclient: build_initial_monmap 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.935+0000 7f351ea7a640 10 monclient: monmap: 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 1 librados: starting msgr at 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 1 librados: starting objecter 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 1 Processor -- start 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 1 -- start start 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 1 librados: setting wanted keys 2024-06-11T19:24:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 1 librados: calling monclient init 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 10 monclient: init 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 10 monclient: _add_conns ranks=[1,5,4,3,2,6,7,8,0] 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35180a2530 0x7f35181cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 10 monclient(hunting): picked mon.noname-f con 0x7f35180a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f35180a4920 0x7f35181ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 10 monclient(hunting): picked mon.noname-h con 0x7f35180a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35180a5240 0x7f3518122d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 10 monclient(hunting): picked mon.noname-c con 0x7f35180a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f35180a6da0 con 0x7f35180a2530 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3518138e50 con 0x7f35180a5240 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f35180a65f0 con 0x7f35180a4920 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.936+0000 7f351ea7a640 10 monclient(hunting): authenticate will time out at 1687.743926s 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35180a5240 0x7f3518122d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35180a5240 0x7f3518122d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46826/0 (socket says 172.21.15.98:46826) 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 1 -- 172.21.15.98:0/3882525610 learned_addr learned my addr 172.21.15.98:0/3882525610 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 10 monclient(hunting): get_auth_request con 0x7f35180a5240 auth_method 0 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351c816640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35180a2530 0x7f35181cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f3517fff640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f35180a4920 0x7f35181ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 10 monclient(hunting): handle_auth_done global_id 114385 payload 824 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 1 -- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f35180a4920 msgr2=0x7f35181ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f35180a4920 0x7f35181ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 1 -- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35180a2530 msgr2=0x7f35181cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35180a2530 0x7f35181cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 10 monclient: _finish_hunting 0 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 1 monclient: found mon.noname-c 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 1 -- 172.21.15.98:0/3882525610 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3518126c70 con 0x7f35180a5240 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351c816640 10 monclient: get_auth_request con 0x7f35180a2530 auth_method 0 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f3517fff640 10 monclient: get_auth_request con 0x7f35180a4920 auth_method 0 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351c816640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35180a2530 0x7f35181cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f3517fff640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f35180a4920 0x7f35181ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:24:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351d017640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35180a5240 0x7f3518122d00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f350000c410 tx=0x7f350000c8d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 10 client.?.objecter ms_handle_connect 0x7f35180a5240 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f35000047e0 con 0x7f35180a5240 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 10 monclient: dump: 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.953 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 10 monclient: _finish_auth 0 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 10 monclient: _check_auth_tickets 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351ea7a640 5 monclient: authenticate success, global_id 114385 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35000049a0 con 0x7f35180a5240 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3500004bd0 con 0x7f35180a5240 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351ea7a640 10 monclient: _renew_subs 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351ea7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351ea7a640 1 -- 172.21.15.98:0/3882525610 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35180aab80 con 0x7f35180a5240 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f3516ffd640 4 set_mon_vals no callback set 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f35157fa640 10 monclient: dump: 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351ea7a640 10 client.114385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351ea7a640 10 monclient: _renew_subs 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351ea7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351ea7a640 1 -- 172.21.15.98:0/3882525610 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f35181266b0 con 0x7f35180a5240 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.937+0000 7f351ea7a640 1 librados: init done 2024-06-11T19:24:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351ea7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351ea7a640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123240 0x7f3518123620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351ea7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351ea7a640 1 -- 172.21.15.98:0/3882525610 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3518050830 con 0x7f3518123240 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3500011070 con 0x7f35180a5240 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351c816640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123240 0x7f3518123620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351c816640 10 monclient: get_auth_request con 0x7f3518123240 auth_method 0 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351c816640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351c816640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351c816640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351c816640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351c816640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351c816640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351c816640 10 monclient(hunting): handle_auth_done global_id 124136 payload 824 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f35157fa640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f34f803fdb0 0x7f34f8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.938+0000 7f351c816640 1 -- 172.21.15.98:0/3882525610 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35181254e0 con 0x7f3518123240 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(622..622 src has 1..622) ==== 6789+0+0 (secure 0 0 0) 0x7f3500051800 con 0x7f35180a5240 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 10 client.114385.objecter ms_dispatch 0x7f3518000b90 osd_map(622..622 src has 1..622) 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 3 client.114385.objecter handle_osd_map got epochs [622,622] > 0 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 3 client.114385.objecter handle_osd_map decoding full epoch 622 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f351c816640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123240 0x7f3518123620 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f35180abfb0 tx=0x7f35100046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 20 client.114385.objecter dump_active .. 0 homeless 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f3517fff640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f34f803fdb0 0x7f34f8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 10 client.114385.objecter ms_handle_connect 0x7f3518123240 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 10 client.114385.objecter resend_mon_ops 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f3517fff640 10 monclient: get_auth_request con 0x7f34f803fdb0 auth_method 0 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3510043070 con 0x7f3518123240 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 10 monclient: dump: 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.959 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.960 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 10 client.114385.objecter ms_handle_connect 0x7f34f803fdb0 2024-06-11T19:24:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f3517fff640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f34f803fdb0 0x7f34f8042270 secure :-1 s=READY pgs=4285 cs=0 l=1 rev1=1 crypto rx=0x7f3518124460 tx=0x7f35080074b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3510003820 con 0x7f3518123240 2024-06-11T19:24:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123240 msgr2=0x7f3518123620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123240 0x7f3518123620 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f35180abfb0 tx=0x7f35100046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123240 msgr2=0x7f3518123620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.939+0000 7f35157fa640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123240 0x7f3518123620 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.945+0000 7f351ea7a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.945+0000 7f351ea7a640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123b60 0x7f351809e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.945+0000 7f351ea7a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.945+0000 7f351ea7a640 1 -- 172.21.15.98:0/3882525610 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f35180a3480 con 0x7f3518123b60 2024-06-11T19:24:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f351c816640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123b60 0x7f351809e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f351c816640 10 monclient: get_auth_request con 0x7f3518123b60 auth_method 0 2024-06-11T19:24:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f351c816640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f351c816640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f351c816640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f351c816640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f351c816640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f351c816640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f351c816640 10 monclient(hunting): handle_auth_done global_id 124145 payload 824 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f351c816640 1 -- 172.21.15.98:0/3882525610 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f35180a4d00 con 0x7f3518123b60 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f351c816640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123b60 0x7f351809e9f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f35100096f0 tx=0x7f3510009190 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 10 client.114385.objecter ms_handle_connect 0x7f3518123b60 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 10 client.114385.objecter resend_mon_ops 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3510039040 con 0x7f3518123b60 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 10 monclient: dump: 2024-06-11T19:24:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:37.965 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f3510034e50 con 0x7f3518123b60 2024-06-11T19:24:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123b60 msgr2=0x7f351809e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123b60 0x7f351809e9f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f35100096f0 tx=0x7f3510009190 comp rx=0 tx=0).stop 2024-06-11T19:24:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 1 -- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123b60 msgr2=0x7f351809e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.946+0000 7f35157fa640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3518123b60 0x7f351809e9f0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f35100096f0 tx=0x7f3510009190 comp rx=0 tx=0).stop 2024-06-11T19:24:37.965 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 20883, 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:37.986 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:37.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 10 librados: watch_flush enter 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 10 librados: watch_flush exit 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 20 client.114385.objecter shutdown clearing up homeless session... 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 10 client.114385.objecter successfully canceled tick 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 1 -- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f34f803fdb0 msgr2=0x7f34f8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f34f803fdb0 0x7f34f8042270 secure :-1 s=READY pgs=4285 cs=0 l=1 rev1=1 crypto rx=0x7f3518124460 tx=0x7f35080074b0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 10 monclient: shutdown 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 1 -- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35180a5240 msgr2=0x7f3518122d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 1 --2- 172.21.15.98:0/3882525610 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35180a5240 0x7f3518122d00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f350000c410 tx=0x7f350000c8d0 comp rx=0 tx=0).stop 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351d017640 1 -- 172.21.15.98:0/3882525610 reap_dead start 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351d017640 1 -- 172.21.15.98:0/3882525610 reap_dead start 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 1 -- 172.21.15.98:0/3882525610 shutdown_connections 2024-06-11T19:24:37.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 1 -- 172.21.15.98:0/3882525610 >> 172.21.15.98:0/3882525610 conn(0x7f35180ab850 msgr2=0x7f35181bffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 1 -- 172.21.15.98:0/3882525610 shutdown_connections 2024-06-11T19:24:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 1 -- 172.21.15.98:0/3882525610 wait complete. 2024-06-11T19:24:37.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:37.947+0000 7f351ea7a640 1 librados: shutdown 2024-06-11T19:24:37.989 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:37.989 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:24:37.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:24:38.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.089+0000 7f0df7087640 10 monclient: get_monmap_and_config 2024-06-11T19:24:38.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.089+0000 7f0df7087640 10 monclient: build_initial_monmap 2024-06-11T19:24:38.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.089+0000 7f0df7087640 10 monclient: monmap: 2024-06-11T19:24:38.089 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:38.089 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.089 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:38.090933+0000 2024-06-11T19:24:38.089 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:38.090933+0000 2024-06-11T19:24:38.089 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:38.089 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.089 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:38.089 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:38.089 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:38.089 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:38.090 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:38.090 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:38.090 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:38.090 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:38.090 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:38.090 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:38.090 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:38.090 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.090+0000 7f0df7087640 1 Processor -- start 2024-06-11T19:24:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.090+0000 7f0df7087640 1 -- start start 2024-06-11T19:24:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.090+0000 7f0df7087640 10 monclient: init 2024-06-11T19:24:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 10 monclient: _add_conns ranks=[0,2,7,4,1,8,9,5,10,6,3] 2024-06-11T19:24:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0df00a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a2530 0x7f0df013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 10 monclient(hunting): picked mon.noname-b con 0x7f0df00a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0df013e1a0 0x7f0df0140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 10 monclient(hunting): picked mon.noname-i con 0x7f0df013e1a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0df00a7140 con 0x7f0df00a4920 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0df00a2b50 con 0x7f0df00a2530 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f0df00a2e30 con 0x7f0df013e1a0 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df7087640 10 monclient(hunting): authenticate will time out at 1687.898254s 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a2530 0x7f0df013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df4e23640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0df00a4920 0x7f0df00a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53792/0 (socket says 172.21.15.98:53792) 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0df4e23640 1 -- 172.21.15.98:0/2070633417 learned_addr learned my addr 172.21.15.98:0/2070633417 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 10 monclient(hunting): get_auth_request con 0x7f0df00a2530 auth_method 0 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 10 monclient(hunting): handle_auth_done global_id 85133 payload 274 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 1 -- 172.21.15.98:0/2070633417 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0df013e1a0 msgr2=0x7f0df0140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 1 --2- 172.21.15.98:0/2070633417 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0df013e1a0 0x7f0df0140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0def7fe640 1 -- 172.21.15.98:0/2070633417 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2623039980 0 0) 0x7f0df00a7140 con 0x7f0df00a4920 2024-06-11T19:24:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 1 -- 172.21.15.98:0/2070633417 >> v1:172.21.15.98:6789/0 conn(0x7f0df00a4920 legacy=0x7f0df00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 1 monclient: found mon.noname-b 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0deffff640 1 -- 172.21.15.98:0/2070633417 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df00a3110 con 0x7f0df00a2530 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.091+0000 7f0def7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0deffff640 1 --2- 172.21.15.98:0/2070633417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a2530 0x7f0df013dc60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0de40097d0 tx=0x7f0de402b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b1716b8854e4e74 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 1 -- 172.21.15.98:0/2070633417 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0de4039070 con 0x7f0df00a2530 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 10 monclient: dump: 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.102 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.103 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 10 monclient: _finish_auth 0 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 1 -- 172.21.15.98:0/2070633417 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0de4030070 con 0x7f0df00a2530 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0df7087640 5 monclient: authenticate success, global_id 85133 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0df7087640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 1 -- 172.21.15.98:0/2070633417 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0de4031db0 con 0x7f0df00a2530 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0def7fe640 10 monclient: dump: 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.104 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.105 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0df7087640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.092+0000 7f0df7087640 4 set_mon_vals no callback set 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 10 monclient: shutdown 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 1 -- 172.21.15.98:0/2070633417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a2530 msgr2=0x7f0df013dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 1 --2- 172.21.15.98:0/2070633417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a2530 0x7f0df013dc60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0de40097d0 tx=0x7f0de402b770 comp rx=0 tx=0).stop 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 1 -- 172.21.15.98:0/2070633417 shutdown_connections 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 1 --2- 172.21.15.98:0/2070633417 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0df013e1a0 0x7f0df0140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 1 --2- 172.21.15.98:0/2070633417 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0df00a2530 0x7f0df013dc60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 1 -- 172.21.15.98:0/2070633417 >> 172.21.15.98:0/2070633417 conn(0x7f0df00ab850 msgr2=0x7f0df00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 1 -- 172.21.15.98:0/2070633417 shutdown_connections 2024-06-11T19:24:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 1 -- 172.21.15.98:0/2070633417 wait complete. 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 10 monclient: build_initial_monmap 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 10 monclient: monmap: 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 1 librados: starting msgr at 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 1 librados: starting objecter 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.093+0000 7f0df7087640 1 Processor -- start 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 1 -- start start 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 1 librados: setting wanted keys 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 1 librados: calling monclient init 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 10 monclient: init 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 10 monclient: _add_conns ranks=[1,7,6,8,3,2,5,0,4] 2024-06-11T19:24:38.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df00a2530 0x7f0df01cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 10 monclient(hunting): picked mon.noname-f con 0x7f0df00a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0df00a4920 0x7f0df01cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 10 monclient(hunting): picked mon.noname-i con 0x7f0df00a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0df013e1a0 0x7f0df009eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 10 monclient(hunting): picked mon.noname-d con 0x7f0df013e1a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0df0138e50 con 0x7f0df00a2530 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0df00a84b0 con 0x7f0df013e1a0 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f0df00a6da0 con 0x7f0df00a4920 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df7087640 10 monclient(hunting): authenticate will time out at 1687.901739s 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df5624640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0df013e1a0 0x7f0df009eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df5624640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0df013e1a0 0x7f0df009eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:35320/0 (socket says 172.21.15.98:35320) 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df5624640 1 -- 172.21.15.98:0/3530536940 learned_addr learned my addr 172.21.15.98:0/3530536940 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df5624640 10 monclient(hunting): get_auth_request con 0x7f0df013e1a0 auth_method 0 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df5624640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df5624640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df5624640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df5624640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df5624640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.094+0000 7f0df5624640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0deffff640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0df00a4920 0x7f0df01cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df4e23640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df00a2530 0x7f0df01cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0deffff640 10 monclient(hunting): get_auth_request con 0x7f0df00a4920 auth_method 0 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0deffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0deffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0deffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df4e23640 10 monclient(hunting): get_auth_request con 0x7f0df00a2530 auth_method 0 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df4e23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df4e23640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df4e23640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df5624640 10 monclient(hunting): handle_auth_done global_id 124170 payload 824 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df5624640 1 -- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0df00a4920 msgr2=0x7f0df01cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df5624640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0df00a4920 0x7f0df01cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df5624640 1 -- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df00a2530 msgr2=0x7f0df01cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df5624640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df00a2530 0x7f0df01cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df5624640 10 monclient: _finish_hunting 0 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df5624640 1 monclient: found mon.noname-d 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df5624640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:38.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df5624640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df5624640 1 -- 172.21.15.98:0/3530536940 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df00a4060 con 0x7f0df013e1a0 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df4e23640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df00a2530 0x7f0df01cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0deffff640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0df00a4920 0x7f0df01cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df5624640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0df013e1a0 0x7f0df009eab0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0de800b9a0 tx=0x7f0de800be60 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 10 client.?.objecter ms_handle_connect 0x7f0df013e1a0 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0de8019070 con 0x7f0df013e1a0 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 10 monclient: dump: 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.112 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 10 monclient: _finish_auth 0 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df7087640 5 monclient: authenticate success, global_id 124170 2024-06-11T19:24:38.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0de800ce10 con 0x7f0df013e1a0 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df7087640 10 monclient: _renew_subs 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df7087640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df7087640 1 -- 172.21.15.98:0/3530536940 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0df0136580 con 0x7f0df013e1a0 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0de80077e0 con 0x7f0df013e1a0 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df7087640 10 client.124170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0deeffd640 4 set_mon_vals no callback set 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0ded7fa640 10 monclient: dump: 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.114 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df7087640 10 monclient: _renew_subs 2024-06-11T19:24:38.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df7087640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df7087640 1 -- 172.21.15.98:0/3530536940 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f0df0121e80 con 0x7f0df013e1a0 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.095+0000 7f0df7087640 1 librados: init done 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f0de8006d10 con 0x7f0df013e1a0 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0df7087640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0df7087640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df01ce320 0x7f0df009eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0df7087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0df7087640 1 -- 172.21.15.98:0/3530536940 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0df0050830 con 0x7f0df01ce320 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0ded7fa640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0ddc03fde0 0x7f0ddc0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(622..622 src has 1..622) ==== 6789+0+0 (secure 0 0 0) 0x7f0de80515b0 con 0x7f0df013e1a0 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0ded7fa640 10 client.124170.objecter ms_dispatch 0x7f0df0000b90 osd_map(622..622 src has 1..622) 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0ded7fa640 3 client.124170.objecter handle_osd_map got epochs [622,622] > 0 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0ded7fa640 3 client.124170.objecter handle_osd_map decoding full epoch 622 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0df4e23640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df01ce320 0x7f0df009eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0ded7fa640 20 client.124170.objecter dump_active .. 0 homeless 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0df4e23640 10 monclient: get_auth_request con 0x7f0df01ce320 auth_method 0 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0df4e23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0df4e23640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0df4e23640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0deffff640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0ddc03fde0 0x7f0ddc0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0deffff640 10 monclient: get_auth_request con 0x7f0ddc03fde0 auth_method 0 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0df4e23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0df4e23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.096+0000 7f0df4e23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0df4e23640 10 monclient(hunting): handle_auth_done global_id 124147 payload 824 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0df4e23640 1 -- 172.21.15.98:0/3530536940 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0df00aa690 con 0x7f0df01ce320 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 10 client.124170.objecter ms_handle_connect 0x7f0ddc03fde0 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0deffff640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0ddc03fde0 0x7f0ddc0422a0 secure :-1 s=READY pgs=4286 cs=0 l=1 rev1=1 crypto rx=0x7f0de4036000 tx=0x7f0de4008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 10 client.124170.objecter ms_handle_connect 0x7f0df01ce320 2024-06-11T19:24:38.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0df4e23640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df01ce320 0x7f0df009eff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0de0000c00 tx=0x7f0de00019b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 10 client.124170.objecter resend_mon_ops 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0de0012a00 con 0x7f0df01ce320 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 10 monclient: dump: 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.118 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0de0018070 con 0x7f0df01ce320 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df01ce320 msgr2=0x7f0df009eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df01ce320 0x7f0df009eff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0de0000c00 tx=0x7f0de00019b0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df01ce320 msgr2=0x7f0df009eff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:38.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.097+0000 7f0ded7fa640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df01ce320 0x7f0df009eff0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df7087640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:38.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df7087640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df009f530 0x7f0df013bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df7087640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df7087640 1 -- 172.21.15.98:0/3530536940 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0df009f910 con 0x7f0df009f530 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df4e23640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df009f530 0x7f0df013bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df4e23640 10 monclient: get_auth_request con 0x7f0df009f530 auth_method 0 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df4e23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df4e23640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df4e23640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df4e23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df4e23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df4e23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df4e23640 10 monclient(hunting): handle_auth_done global_id 124156 payload 824 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df4e23640 1 -- 172.21.15.98:0/3530536940 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0df00a43b0 con 0x7f0df009f530 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0df4e23640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df009f530 0x7f0df013bd30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0de0017000 tx=0x7f0de0000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0ded7fa640 10 client.124170.objecter ms_handle_connect 0x7f0df009f530 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0ded7fa640 10 client.124170.objecter resend_mon_ops 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.104+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0de0027440 con 0x7f0df009f530 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0ded7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0ded7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0ded7fa640 10 monclient: dump: 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f0de0012a10 con 0x7f0df009f530 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0ded7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0ded7fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df009f530 msgr2=0x7f0df013bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0ded7fa640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df009f530 0x7f0df013bd30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0de0017000 tx=0x7f0de0000ed0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0ded7fa640 1 -- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df009f530 msgr2=0x7f0df013bd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0ded7fa640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0df009f530 0x7f0df013bd30 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0de0017000 tx=0x7f0de0000ed0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.125 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 21038, 2024-06-11T19:24:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.133 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:38.135 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:38.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:38.139 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.141 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:38.144 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:38.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:38.146 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:38.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 10 librados: watch_flush enter 2024-06-11T19:24:38.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 10 librados: watch_flush exit 2024-06-11T19:24:38.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 20 client.124170.objecter shutdown clearing up homeless session... 2024-06-11T19:24:38.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 10 client.124170.objecter successfully canceled tick 2024-06-11T19:24:38.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 1 -- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0ddc03fde0 msgr2=0x7f0ddc0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0ddc03fde0 0x7f0ddc0422a0 secure :-1 s=READY pgs=4286 cs=0 l=1 rev1=1 crypto rx=0x7f0de4036000 tx=0x7f0de4008030 comp rx=0 tx=0).stop 2024-06-11T19:24:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 10 monclient: shutdown 2024-06-11T19:24:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 1 -- 172.21.15.98:0/3530536940 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0df013e1a0 msgr2=0x7f0df009eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 1 --2- 172.21.15.98:0/3530536940 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0df013e1a0 0x7f0df009eab0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0de800b9a0 tx=0x7f0de800be60 comp rx=0 tx=0).stop 2024-06-11T19:24:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df5624640 1 -- 172.21.15.98:0/3530536940 reap_dead start 2024-06-11T19:24:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df5624640 1 -- 172.21.15.98:0/3530536940 reap_dead start 2024-06-11T19:24:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 1 -- 172.21.15.98:0/3530536940 shutdown_connections 2024-06-11T19:24:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 1 -- 172.21.15.98:0/3530536940 >> 172.21.15.98:0/3530536940 conn(0x7f0df00ab850 msgr2=0x7f0df00a9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 1 -- 172.21.15.98:0/3530536940 shutdown_connections 2024-06-11T19:24:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.105+0000 7f0df7087640 1 -- 172.21.15.98:0/3530536940 wait complete. 2024-06-11T19:24:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.106+0000 7f0df7087640 1 librados: shutdown 2024-06-11T19:24:38.150 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:38.150 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:24:38.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:24:38.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.250+0000 7f6d4dfa8640 10 monclient: get_monmap_and_config 2024-06-11T19:24:38.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.250+0000 7f6d4dfa8640 10 monclient: build_initial_monmap 2024-06-11T19:24:38.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.250+0000 7f6d4dfa8640 10 monclient: monmap: 2024-06-11T19:24:38.249 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:38.249 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.249 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:38.251170+0000 2024-06-11T19:24:38.249 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:38.251170+0000 2024-06-11T19:24:38.249 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:38.250 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.250+0000 7f6d4dfa8640 1 Processor -- start 2024-06-11T19:24:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 1 -- start start 2024-06-11T19:24:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 10 monclient: init 2024-06-11T19:24:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 10 monclient: _add_conns ranks=[3,9,1,5,0,2,8,4,10,7,6] 2024-06-11T19:24:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d481347f0 0x7f6d4809e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 10 monclient(hunting): picked mon.noname-g con 0x7f6d481347f0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d48134bd0 0x7f6d480a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 10 monclient(hunting): picked mon.noname-a con 0x7f6d48134bd0 addr v2:172.21.15.98:3300/0 2024-06-11T19:24:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6d48135010 addr v1:172.21.15.117:6789/0 2024-06-11T19:24:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d4809d410 con 0x7f6d48135010 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6d480687f0 con 0x7f6d48134bd0 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f6d4809d6f0 con 0x7f6d481347f0 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d4dfa8640 10 monclient(hunting): authenticate will time out at 1688.058636s 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d48134bd0 0x7f6d480a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d48134bd0 0x7f6d480a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53636/0 (socket says 172.21.15.98:53636) 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 1 -- 172.21.15.98:0/2627618800 learned_addr learned my addr 172.21.15.98:0/2627618800 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 10 monclient(hunting): get_auth_request con 0x7f6d48134bd0 auth_method 0 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d477fe640 1 --2- 172.21.15.98:0/2627618800 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d481347f0 0x7f6d4809e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d477fe640 10 monclient(hunting): get_auth_request con 0x7f6d481347f0 auth_method 0 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d477fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d477fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 10 monclient(hunting): handle_auth_done global_id 66042 payload 274 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 1 -- 172.21.15.98:0/2627618800 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d481347f0 msgr2=0x7f6d4809e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 1 --2- 172.21.15.98:0/2627618800 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d481347f0 0x7f6d4809e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.251+0000 7f6d3ffff640 1 -- 172.21.15.98:0/2627618800 >> v1:172.21.15.117:6789/0 conn(0x7f6d48135010 legacy=0x7f6d480a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.252+0000 7f6d3ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.252+0000 7f6d3ffff640 1 monclient: found mon.noname-a 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.252+0000 7f6d3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.252+0000 7f6d3ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.252+0000 7f6d3ffff640 1 -- 172.21.15.98:0/2627618800 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4809d9d0 con 0x7f6d48134bd0 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.252+0000 7f6d3ffff640 1 --2- 172.21.15.98:0/2627618800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d48134bd0 0x7f6d480a3690 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009870 tx=0x7f6d3002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e53044e5f341d02 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.252+0000 7f6d46ffd640 1 -- 172.21.15.98:0/2627618800 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6d30035020 con 0x7f6d48134bd0 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.252+0000 7f6d46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.252+0000 7f6d46ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.252+0000 7f6d46ffd640 10 monclient: dump: 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d46ffd640 10 monclient: _finish_auth 0 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d46ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:24:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 5 monclient: authenticate success, global_id 66042 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d46ffd640 1 -- 172.21.15.98:0/2627618800 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d300318b0 con 0x7f6d48134bd0 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d46ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 4 set_mon_vals no callback set 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d46ffd640 1 -- 172.21.15.98:0/2627618800 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6d3002d040 con 0x7f6d48134bd0 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d46ffd640 10 monclient: dump: 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 10 monclient: shutdown 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/2627618800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d48134bd0 msgr2=0x7f6d480a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 1 --2- 172.21.15.98:0/2627618800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d48134bd0 0x7f6d480a3690 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009870 tx=0x7f6d3002b8f0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/2627618800 shutdown_connections 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 1 --2- 172.21.15.98:0/2627618800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d48134bd0 0x7f6d480a3690 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 1 --2- 172.21.15.98:0/2627618800 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d481347f0 0x7f6d4809e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/2627618800 >> 172.21.15.98:0/2627618800 conn(0x7f6d480a4780 msgr2=0x7f6d480a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/2627618800 shutdown_connections 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.253+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/2627618800 wait complete. 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient: build_initial_monmap 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient: monmap: 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 1 librados: starting msgr at 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 1 librados: starting objecter 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 1 Processor -- start 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 1 -- start start 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 1 librados: setting wanted keys 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 1 librados: calling monclient init 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient: init 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient: _add_conns ranks=[8,4,7,3,0,2,5,1,6] 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6d481347f0 0x7f6d481cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient(hunting): picked mon.noname-e con 0x7f6d481347f0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d48134bd0 0x7f6d481ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient(hunting): picked mon.noname-c con 0x7f6d48134bd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:38.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d48135010 0x7f6d4811ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient(hunting): picked mon.noname-i con 0x7f6d48135010 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f6d48068c40 con 0x7f6d48134bd0 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f6d48068ac0 con 0x7f6d48135010 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6d480a42e0 con 0x7f6d481347f0 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.254+0000 7f6d4dfa8640 10 monclient(hunting): authenticate will time out at 1688.061972s 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d48134bd0 0x7f6d481ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d477fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6d481347f0 0x7f6d481cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d48134bd0 0x7f6d481ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46840/0 (socket says 172.21.15.98:46840) 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d477fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6d481347f0 0x7f6d481cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:53142/0 (socket says 172.21.15.98:53142) 2024-06-11T19:24:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 1 -- 172.21.15.98:0/3630298171 learned_addr learned my addr 172.21.15.98:0/3630298171 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 10 monclient(hunting): get_auth_request con 0x7f6d48134bd0 auth_method 0 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d477fe640 10 monclient(hunting): get_auth_request con 0x7f6d481347f0 auth_method 0 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d477fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d477fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d47fff640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d48135010 0x7f6d4811ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d47fff640 10 monclient(hunting): get_auth_request con 0x7f6d48135010 auth_method 0 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d47fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 10 monclient(hunting): handle_auth_done global_id 114394 payload 824 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 1 -- 172.21.15.98:0/3630298171 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6d481347f0 msgr2=0x7f6d481cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6d481347f0 0x7f6d481cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 1 -- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d48135010 msgr2=0x7f6d4811ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f6d48135010 0x7f6d4811ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 1 monclient: found mon.noname-c 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 1 -- 172.21.15.98:0/3630298171 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d481225f0 con 0x7f6d48134bd0 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d477fe640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6d481347f0 0x7f6d481cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d3ffff640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d48134bd0 0x7f6d481ce130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6d300097c0 tx=0x7f6d300046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 10 client.?.objecter ms_handle_connect 0x7f6d48134bd0 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6d30035040 con 0x7f6d48134bd0 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 10 monclient: dump: 2024-06-11T19:24:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 10 monclient: _finish_auth 0 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d4dfa8640 5 monclient: authenticate success, global_id 114394 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d4dfa8640 10 monclient: _renew_subs 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d30031e30 con 0x7f6d48134bd0 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d4dfa8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/3630298171 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d480a37a0 con 0x7f6d48134bd0 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6d3003a420 con 0x7f6d48134bd0 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d44ff9640 10 monclient: dump: 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.277 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d467fc640 4 set_mon_vals no callback set 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d4dfa8640 10 client.114394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d4dfa8640 10 monclient: _renew_subs 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d4dfa8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/3630298171 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d48122bb0 con 0x7f6d48134bd0 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.255+0000 7f6d4dfa8640 1 librados: init done 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d4dfa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d4dfa8640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811f180 0x7f6d4811f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d4dfa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/3630298171 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f6d48050830 con 0x7f6d4811f180 2024-06-11T19:24:38.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f6d30038020 con 0x7f6d48134bd0 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d477fe640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811f180 0x7f6d4811f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d477fe640 10 monclient: get_auth_request con 0x7f6d4811f180 auth_method 0 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d477fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d477fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d44ff9640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6d2803fdb0 0x7f6d28042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(622..622 src has 1..622) ==== 6789+0+0 (secure 0 0 0) 0x7f6d30073910 con 0x7f6d48134bd0 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d44ff9640 10 client.114394.objecter ms_dispatch 0x7f6d48000b90 osd_map(622..622 src has 1..622) 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d44ff9640 3 client.114394.objecter handle_osd_map got epochs [622,622] > 0 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d44ff9640 3 client.114394.objecter handle_osd_map decoding full epoch 622 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.256+0000 7f6d477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 20 client.114394.objecter dump_active .. 0 homeless 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d47fff640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6d2803fdb0 0x7f6d28042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d47fff640 10 monclient: get_auth_request con 0x7f6d2803fdb0 auth_method 0 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d477fe640 10 monclient(hunting): handle_auth_done global_id 114312 payload 824 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d477fe640 1 -- 172.21.15.98:0/3630298171 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d48122310 con 0x7f6d4811f180 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 10 client.114394.objecter ms_handle_connect 0x7f6d4811f180 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 10 client.114394.objecter resend_mon_ops 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d477fe640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811f180 0x7f6d4811f560 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6d38010bd0 tx=0x7f6d3800e5a0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 10 client.114394.objecter ms_handle_connect 0x7f6d2803fdb0 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d47fff640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6d2803fdb0 0x7f6d28042270 secure :-1 s=READY pgs=4287 cs=0 l=1 rev1=1 crypto rx=0x7f6d481203a0 tx=0x7f6d40009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6d38018020 con 0x7f6d4811f180 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:24:38.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 10 monclient: dump: 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.281 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.282 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6d38004030 con 0x7f6d4811f180 2024-06-11T19:24:38.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:38.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:38.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811f180 msgr2=0x7f6d4811f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811f180 0x7f6d4811f560 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6d38010bd0 tx=0x7f6d3800e5a0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811f180 msgr2=0x7f6d4811f560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:38.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.257+0000 7f6d44ff9640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811f180 0x7f6d4811f560 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d4dfa8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:38.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d4dfa8640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811faa0 0x7f6d481385b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d4dfa8640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/3630298171 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f6d48075170 con 0x7f6d4811faa0 2024-06-11T19:24:38.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d477fe640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811faa0 0x7f6d481385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d477fe640 10 monclient: get_auth_request con 0x7f6d4811faa0 auth_method 0 2024-06-11T19:24:38.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d477fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d477fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d477fe640 10 monclient(hunting): handle_auth_done global_id 114321 payload 824 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d477fe640 1 -- 172.21.15.98:0/3630298171 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6d4809e6b0 con 0x7f6d4811faa0 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d44ff9640 10 client.114394.objecter ms_handle_connect 0x7f6d4811faa0 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d44ff9640 10 client.114394.objecter resend_mon_ops 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.264+0000 7f6d477fe640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811faa0 0x7f6d481385b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6d38019000 tx=0x7f6d380046c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6d38018040 con 0x7f6d4811faa0 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d44ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d44ff9640 10 monclient: dump: 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.286 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.287 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.287 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.287 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.287 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f6d38003bf0 con 0x7f6d4811faa0 2024-06-11T19:24:38.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d44ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:38.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d44ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:38.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811faa0 msgr2=0x7f6d481385b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d44ff9640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811faa0 0x7f6d481385b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6d38019000 tx=0x7f6d380046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d44ff9640 1 -- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811faa0 msgr2=0x7f6d481385b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:38.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d44ff9640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d4811faa0 0x7f6d481385b0 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6d38019000 tx=0x7f6d380046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 166634, 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:38.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.307 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.307 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:38.307 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:38.307 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.307 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.307 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:38.308 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:38.309 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.311 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:38.311 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 10 librados: watch_flush enter 2024-06-11T19:24:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 10 librados: watch_flush exit 2024-06-11T19:24:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 20 client.114394.objecter shutdown clearing up homeless session... 2024-06-11T19:24:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 10 client.114394.objecter successfully canceled tick 2024-06-11T19:24:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6d2803fdb0 msgr2=0x7f6d28042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f6d2803fdb0 0x7f6d28042270 secure :-1 s=READY pgs=4287 cs=0 l=1 rev1=1 crypto rx=0x7f6d481203a0 tx=0x7f6d40009040 comp rx=0 tx=0).stop 2024-06-11T19:24:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 10 monclient: shutdown 2024-06-11T19:24:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/3630298171 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d48134bd0 msgr2=0x7f6d481ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 1 --2- 172.21.15.98:0/3630298171 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6d48134bd0 0x7f6d481ce130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6d300097c0 tx=0x7f6d300046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d47fff640 1 -- 172.21.15.98:0/3630298171 reap_dead start 2024-06-11T19:24:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d47fff640 1 -- 172.21.15.98:0/3630298171 reap_dead start 2024-06-11T19:24:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/3630298171 shutdown_connections 2024-06-11T19:24:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/3630298171 >> 172.21.15.98:0/3630298171 conn(0x7f6d480a4780 msgr2=0x7f6d481c01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/3630298171 shutdown_connections 2024-06-11T19:24:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 1 -- 172.21.15.98:0/3630298171 wait complete. 2024-06-11T19:24:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.265+0000 7f6d4dfa8640 1 librados: shutdown 2024-06-11T19:24:38.314 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:38.314 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:24:38.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:24:38.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.440+0000 7f9d132e1640 10 monclient: get_monmap_and_config 2024-06-11T19:24:38.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.440+0000 7f9d132e1640 10 monclient: build_initial_monmap 2024-06-11T19:24:38.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.440+0000 7f9d132e1640 10 monclient: monmap: 2024-06-11T19:24:38.450 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:38.450 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.450 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:38.441602+0000 2024-06-11T19:24:38.450 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:38.441602+0000 2024-06-11T19:24:38.450 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 1 Processor -- start 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 1 -- start start 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 10 monclient: init 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 10 monclient: _add_conns ranks=[9,8,5,1,10,6,0,4,7,3,2] 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9d0c0a4920 0x7f9d0c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 10 monclient(hunting): picked mon.noname-a con 0x7f9d0c0a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d0c0a5240 0x7f9d0c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 10 monclient(hunting): picked mon.noname-e con 0x7f9d0c0a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c0a2530 0x7f9d0c0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 10 monclient(hunting): picked mon.noname-h con 0x7f9d0c0a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9d0c0a7140 con 0x7f9d0c0a4920 2024-06-11T19:24:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f9d0c0a2b50 con 0x7f9d0c0a2530 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9d0c0a2e30 con 0x7f9d0c0a5240 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d132e1640 10 monclient(hunting): authenticate will time out at 1688.248833s 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1107d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9d0c0a4920 0x7f9d0c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1087c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d0c0a5240 0x7f9d0c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1107d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9d0c0a4920 0x7f9d0c0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53646/0 (socket says 172.21.15.98:53646) 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1107d640 1 -- 172.21.15.98:0/3890378349 learned_addr learned my addr 172.21.15.98:0/3890378349 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1087c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d0c0a5240 0x7f9d0c0a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:53148/0 (socket says 172.21.15.98:53148) 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1107d640 10 monclient(hunting): get_auth_request con 0x7f9d0c0a4920 auth_method 0 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1107d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1107d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1107d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1087c640 10 monclient(hunting): get_auth_request con 0x7f9d0c0a5240 auth_method 0 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1087c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1087c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.441+0000 7f9d1087c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1107d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1107d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1107d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1187e640 1 --2- 172.21.15.98:0/3890378349 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c0a2530 0x7f9d0c0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1187e640 10 monclient(hunting): get_auth_request con 0x7f9d0c0a2530 auth_method 0 2024-06-11T19:24:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1187e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1187e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1187e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1187e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1187e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1187e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 10 monclient(hunting): handle_auth_done global_id 124163 payload 274 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 1 -- 172.21.15.98:0/3890378349 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c0a2530 msgr2=0x7f9d0c0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 1 --2- 172.21.15.98:0/3890378349 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c0a2530 0x7f9d0c0a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 1 -- 172.21.15.98:0/3890378349 >> v2:172.21.15.98:3300/0 conn(0x7f9d0c0a4920 msgr2=0x7f9d0c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 1 --2- 172.21.15.98:0/3890378349 >> v2:172.21.15.98:3300/0 conn(0x7f9d0c0a4920 0x7f9d0c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 10 monclient: _finish_hunting 0 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 1 monclient: found mon.noname-e 2024-06-11T19:24:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 1 -- 172.21.15.98:0/3890378349 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0c0a3110 con 0x7f9d0c0a5240 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1107d640 1 --2- 172.21.15.98:0/3890378349 >> v2:172.21.15.98:3300/0 conn(0x7f9d0c0a4920 0x7f9d0c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.442+0000 7f9d1087c640 1 --2- 172.21.15.98:0/3890378349 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d0c0a5240 0x7f9d0c0a59a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc0097a0 tx=0x7f9cfc02b6f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=933edb3aba5cf2a3 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 1 -- 172.21.15.98:0/3890378349 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9cfc037070 con 0x7f9d0c0a5240 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 10 monclient: dump: 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 10 monclient: _finish_auth 0 2024-06-11T19:24:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 1 -- 172.21.15.98:0/3890378349 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cfc030980 con 0x7f9d0c0a5240 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 5 monclient: authenticate success, global_id 124163 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 1 -- 172.21.15.98:0/3890378349 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9cfc030bb0 con 0x7f9d0c0a5240 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 4 set_mon_vals no callback set 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d03fff640 10 monclient: dump: 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.458 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.459 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 10 monclient: shutdown 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 1 -- 172.21.15.98:0/3890378349 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d0c0a5240 msgr2=0x7f9d0c0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 1 --2- 172.21.15.98:0/3890378349 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d0c0a5240 0x7f9d0c0a59a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc0097a0 tx=0x7f9cfc02b6f0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 1 -- 172.21.15.98:0/3890378349 shutdown_connections 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 1 --2- 172.21.15.98:0/3890378349 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c0a2530 0x7f9d0c0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 1 --2- 172.21.15.98:0/3890378349 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d0c0a5240 0x7f9d0c0a59a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 1 --2- 172.21.15.98:0/3890378349 >> v2:172.21.15.98:3300/0 conn(0x7f9d0c0a4920 0x7f9d0c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 1 -- 172.21.15.98:0/3890378349 >> 172.21.15.98:0/3890378349 conn(0x7f9d0c0ab850 msgr2=0x7f9d0c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 1 -- 172.21.15.98:0/3890378349 shutdown_connections 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.443+0000 7f9d132e1640 1 -- 172.21.15.98:0/3890378349 wait complete. 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.444+0000 7f9d132e1640 10 monclient: build_initial_monmap 2024-06-11T19:24:38.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.444+0000 7f9d132e1640 10 monclient: monmap: 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.444+0000 7f9d132e1640 1 librados: starting msgr at 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.444+0000 7f9d132e1640 1 librados: starting objecter 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.444+0000 7f9d132e1640 1 Processor -- start 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.444+0000 7f9d132e1640 1 -- start start 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.444+0000 7f9d132e1640 1 librados: setting wanted keys 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.444+0000 7f9d132e1640 1 librados: calling monclient init 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.444+0000 7f9d132e1640 10 monclient: init 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 10 monclient: _add_conns ranks=[1,7,4,0,2,8,5,6,3] 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d0c0a2530 0x7f9d0c137700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 10 monclient(hunting): picked mon.noname-f con 0x7f9d0c0a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9d0c0a4920 0x7f9d0c137c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 10 monclient(hunting): picked mon.noname-i con 0x7f9d0c0a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9d0c0a5240 0x7f9d0c123160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 10 monclient(hunting): picked mon.noname-c con 0x7f9d0c0a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c0a6da0 con 0x7f9d0c0a2530 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f9d0c138e50 con 0x7f9d0c0a5240 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f9d0c0a6420 con 0x7f9d0c0a4920 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 10 monclient(hunting): authenticate will time out at 1688.252135s 2024-06-11T19:24:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9d0c0a5240 0x7f9d0c123160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9d0c0a5240 0x7f9d0c123160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46848/0 (socket says 172.21.15.98:46848) 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 1 -- 172.21.15.98:0/3842841583 learned_addr learned my addr 172.21.15.98:0/3842841583 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 10 monclient(hunting): get_auth_request con 0x7f9d0c0a5240 auth_method 0 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1087c640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9d0c0a4920 0x7f9d0c137c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1107d640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d0c0a2530 0x7f9d0c137700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1087c640 10 monclient(hunting): get_auth_request con 0x7f9d0c0a4920 auth_method 0 2024-06-11T19:24:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1087c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1087c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1087c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1107d640 10 monclient(hunting): get_auth_request con 0x7f9d0c0a2530 auth_method 0 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1107d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1107d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1107d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 10 monclient(hunting): handle_auth_done global_id 114403 payload 824 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 1 -- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9d0c0a4920 msgr2=0x7f9d0c137c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9d0c0a4920 0x7f9d0c137c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 1 -- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d0c0a2530 msgr2=0x7f9d0c137700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d0c0a2530 0x7f9d0c137700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 10 monclient: _finish_hunting 0 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 1 monclient: found mon.noname-c 2024-06-11T19:24:38.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 1 -- 172.21.15.98:0/3842841583 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0c126b60 con 0x7f9d0c0a5240 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1107d640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d0c0a2530 0x7f9d0c137700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1087c640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9d0c0a4920 0x7f9d0c137c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d1187e640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9d0c0a5240 0x7f9d0c123160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009dd0 tx=0x7f9d0400c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 10 client.?.objecter ms_handle_connect 0x7f9d0c0a5240 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9d04018020 con 0x7f9d0c0a5240 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 10 monclient: dump: 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.466 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.467 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 10 monclient: _finish_auth 0 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d0400cd30 con 0x7f9d0c0a5240 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 5 monclient: authenticate success, global_id 114403 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 10 monclient: _renew_subs 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9d04014d60 con 0x7f9d0c0a5240 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d132e1640 1 -- 172.21.15.98:0/3842841583 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0c0a90a0 con 0x7f9d0c0a5240 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:38.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d037fe640 4 set_mon_vals no callback set 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.445+0000 7f9d01ffb640 10 monclient: dump: 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.469 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d132e1640 10 client.114403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d132e1640 10 monclient: _renew_subs 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d132e1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d132e1640 1 -- 172.21.15.98:0/3842841583 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d0c124e30 con 0x7f9d0c0a5240 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d132e1640 1 librados: init done 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d132e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d132e1640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c126da0 0x7f9d0c1236a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d132e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d132e1640 1 -- 172.21.15.98:0/3842841583 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f9d0c050830 con 0x7f9d0c126da0 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f9d04005d10 con 0x7f9d0c0a5240 2024-06-11T19:24:38.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d1107d640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c126da0 0x7f9d0c1236a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d1107d640 10 monclient: get_auth_request con 0x7f9d0c126da0 auth_method 0 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d1107d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d1107d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.446+0000 7f9d1107d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d1107d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d1107d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d1107d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9cf003fdb0 0x7f9cf0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(623..623 src has 1..623) ==== 6802+0+0 (secure 0 0 0) 0x7f9d04052a90 con 0x7f9d0c0a5240 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 10 client.114403.objecter ms_dispatch 0x7f9d0c000b90 osd_map(623..623 src has 1..623) 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 3 client.114403.objecter handle_osd_map got epochs [623,623] > 0 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 3 client.114403.objecter handle_osd_map decoding full epoch 623 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d1107d640 10 monclient(hunting): handle_auth_done global_id 124151 payload 824 2024-06-11T19:24:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d1107d640 1 -- 172.21.15.98:0/3842841583 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d0c0a6750 con 0x7f9d0c126da0 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d1087c640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9cf003fdb0 0x7f9cf0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 20 client.114403.objecter dump_active .. 0 homeless 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d1087c640 10 monclient: get_auth_request con 0x7f9cf003fdb0 auth_method 0 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 10 client.114403.objecter ms_handle_connect 0x7f9d0c126da0 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 10 client.114403.objecter resend_mon_ops 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d1107d640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c126da0 0x7f9d0c1236a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9d080115d0 tx=0x7f9d08011a90 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9d080044a0 con 0x7f9d0c126da0 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 10 monclient: dump: 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 10 client.114403.objecter ms_handle_connect 0x7f9cf003fdb0 2024-06-11T19:24:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d1087c640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9cf003fdb0 0x7f9cf0042270 secure :-1 s=READY pgs=4288 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc034000 tx=0x7f9cfc008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9d08004030 con 0x7f9d0c126da0 2024-06-11T19:24:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c126da0 msgr2=0x7f9d0c1236a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c126da0 0x7f9d0c1236a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9d080115d0 tx=0x7f9d08011a90 comp rx=0 tx=0).stop 2024-06-11T19:24:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c126da0 msgr2=0x7f9d0c1236a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.447+0000 7f9d01ffb640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c126da0 0x7f9d0c1236a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d132e1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d132e1640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c123be0 0x7f9d0c123ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d132e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d132e1640 1 -- 172.21.15.98:0/3842841583 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f9d0c124530 con 0x7f9d0c123be0 2024-06-11T19:24:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d1107d640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c123be0 0x7f9d0c123ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d1107d640 10 monclient: get_auth_request con 0x7f9d0c123be0 auth_method 0 2024-06-11T19:24:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d1107d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d1107d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d1107d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d1107d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d1107d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d1107d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d1107d640 10 monclient(hunting): handle_auth_done global_id 124160 payload 824 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d1107d640 1 -- 172.21.15.98:0/3842841583 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9d0c0a61a0 con 0x7f9d0c123be0 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d1107d640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c123be0 0x7f9d0c123ff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9d0801b000 tx=0x7f9d080046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d01ffb640 10 client.114403.objecter ms_handle_connect 0x7f9d0c123be0 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.454+0000 7f9d01ffb640 10 client.114403.objecter resend_mon_ops 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9d08003cb0 con 0x7f9d0c123be0 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d01ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:24:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d01ffb640 10 monclient: dump: 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.479 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.479 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f9d08003cb0 con 0x7f9d0c123be0 2024-06-11T19:24:38.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d01ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:38.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d01ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:38.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c123be0 msgr2=0x7f9d0c123ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d01ffb640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c123be0 0x7f9d0c123ff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9d0801b000 tx=0x7f9d080046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d01ffb640 1 -- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c123be0 msgr2=0x7f9d0c123ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:38.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d01ffb640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f9d0c123be0 0x7f9d0c123ff0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.479 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 21388, 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:38.480 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:38.481 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:38.482 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:38.484 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.485 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:38.488 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.490 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.491 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:38.493 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.496 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:38.498 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.500 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.502 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:38.502 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 10 librados: watch_flush enter 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 10 librados: watch_flush exit 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 20 client.114403.objecter shutdown clearing up homeless session... 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 10 client.114403.objecter successfully canceled tick 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 1 -- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9cf003fdb0 msgr2=0x7f9cf0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9cf003fdb0 0x7f9cf0042270 secure :-1 s=READY pgs=4288 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc034000 tx=0x7f9cfc008030 comp rx=0 tx=0).stop 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 10 monclient: shutdown 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 1 -- 172.21.15.98:0/3842841583 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9d0c0a5240 msgr2=0x7f9d0c123160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 1 --2- 172.21.15.98:0/3842841583 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9d0c0a5240 0x7f9d0c123160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009dd0 tx=0x7f9d0400c490 comp rx=0 tx=0).stop 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d1187e640 1 -- 172.21.15.98:0/3842841583 reap_dead start 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d1187e640 1 -- 172.21.15.98:0/3842841583 reap_dead start 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 1 -- 172.21.15.98:0/3842841583 shutdown_connections 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 1 -- 172.21.15.98:0/3842841583 >> 172.21.15.98:0/3842841583 conn(0x7f9d0c0ab850 msgr2=0x7f9d0c1bff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 1 -- 172.21.15.98:0/3842841583 shutdown_connections 2024-06-11T19:24:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 1 -- 172.21.15.98:0/3842841583 wait complete. 2024-06-11T19:24:38.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.455+0000 7f9d132e1640 1 librados: shutdown 2024-06-11T19:24:38.505 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:38.505 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:24:38.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:24:38.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.605+0000 7fccd501e640 10 monclient: get_monmap_and_config 2024-06-11T19:24:38.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.605+0000 7fccd501e640 10 monclient: build_initial_monmap 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.605+0000 7fccd501e640 10 monclient: monmap: 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:38.606214+0000 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:38.606214+0000 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:38.606 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:38.606 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:38.606 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:38.606 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:38.606 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.605+0000 7fccd501e640 1 Processor -- start 2024-06-11T19:24:38.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.605+0000 7fccd501e640 1 -- start start 2024-06-11T19:24:38.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.605+0000 7fccd501e640 10 monclient: init 2024-06-11T19:24:38.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:38.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 10 monclient: _add_conns ranks=[2,0,3,6,10,7,9,1,5,4,8] 2024-06-11T19:24:38.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd00a2f40 0x7fccd00a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 10 monclient(hunting): picked mon.noname-b con 0x7fccd00a2f40 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:38.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fccd00a4df0 addr v1:172.21.15.98:6789/0 2024-06-11T19:24:38.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccd00a51d0 0x7fccd0136de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 10 monclient(hunting): picked mon.noname-g con 0x7fccd00a51d0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccd00a0ed0 con 0x7fccd00a4df0 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fccd00a3320 con 0x7fccd00a2f40 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fccd00a2460 con 0x7fccd00a51d0 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccd501e640 10 monclient(hunting): authenticate will time out at 1688.413352s 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd00a2f40 0x7fccd00a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccce59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fccd00a4df0 0x7fccd01366d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53794/0 (socket says 172.21.15.98:53794) 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd00a2f40 0x7fccd00a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57276/0 (socket says 172.21.15.98:57276) 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccce59c640 1 -- 172.21.15.98:0/3238206553 learned_addr learned my addr 172.21.15.98:0/3238206553 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:24:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 10 monclient(hunting): get_auth_request con 0x7fccd00a2f40 auth_method 0 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fcccf59e640 1 --2- 172.21.15.98:0/3238206553 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccd00a51d0 0x7fccd0136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fcccf59e640 10 monclient(hunting): get_auth_request con 0x7fccd00a51d0 auth_method 0 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fcccf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fcccf59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fcccf59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 10 monclient(hunting): handle_auth_done global_id 85142 payload 274 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 1 -- 172.21.15.98:0/3238206553 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccd00a51d0 msgr2=0x7fccd0136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 1 --2- 172.21.15.98:0/3238206553 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccd00a51d0 0x7fccd0136de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fcccdd9b640 1 -- 172.21.15.98:0/3238206553 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4262669383 0 0) 0x7fccd00a0ed0 con 0x7fccd00a4df0 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 1 -- 172.21.15.98:0/3238206553 >> v1:172.21.15.98:6789/0 conn(0x7fccd00a4df0 legacy=0x7fccd01366d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 10 monclient: _finish_hunting 0 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 1 monclient: found mon.noname-b 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fccced9d640 1 -- 172.21.15.98:0/3238206553 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd00a2180 con 0x7fccd00a2f40 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.606+0000 7fcccdd9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fccced9d640 1 --2- 172.21.15.98:0/3238206553 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd00a2f40 0x7fccd00a48b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009800 tx=0x7fccc002b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=520786bf10220af3 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 1 -- 172.21.15.98:0/3238206553 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fccc0037070 con 0x7fccd00a2f40 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:24:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 10 monclient: dump: 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 10 monclient: _finish_auth 0 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 10 monclient: _check_auth_tickets 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 1 -- 172.21.15.98:0/3238206553 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccc00044a0 con 0x7fccd00a2f40 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fccd501e640 5 monclient: authenticate success, global_id 85142 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fccd501e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fccd501e640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 1 -- 172.21.15.98:0/3238206553 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fccc0003a80 con 0x7fccd00a2f40 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fccd501e640 4 set_mon_vals no callback set 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.607+0000 7fcccdd9b640 10 monclient: dump: 2024-06-11T19:24:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.608+0000 7fccd501e640 10 monclient: shutdown 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.608+0000 7fccd501e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.608+0000 7fccd501e640 1 -- 172.21.15.98:0/3238206553 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd00a2f40 msgr2=0x7fccd00a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.608+0000 7fccd501e640 1 --2- 172.21.15.98:0/3238206553 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd00a2f40 0x7fccd00a48b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009800 tx=0x7fccc002b760 comp rx=0 tx=0).stop 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.608+0000 7fccd501e640 1 -- 172.21.15.98:0/3238206553 shutdown_connections 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.608+0000 7fccd501e640 1 --2- 172.21.15.98:0/3238206553 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccd00a51d0 0x7fccd0136de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.608+0000 7fccd501e640 1 --2- 172.21.15.98:0/3238206553 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fccd00a2f40 0x7fccd00a48b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.608+0000 7fccd501e640 1 -- 172.21.15.98:0/3238206553 >> 172.21.15.98:0/3238206553 conn(0x7fccd00a95b0 msgr2=0x7fccd00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.608+0000 7fccd501e640 1 -- 172.21.15.98:0/3238206553 shutdown_connections 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.608+0000 7fccd501e640 1 -- 172.21.15.98:0/3238206553 wait complete. 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.608+0000 7fccd501e640 10 monclient: build_initial_monmap 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.608+0000 7fccd501e640 10 monclient: monmap: 2024-06-11T19:24:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 1 librados: starting msgr at 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 1 librados: starting objecter 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 1 Processor -- start 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 1 -- start start 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 1 librados: setting wanted keys 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 1 librados: calling monclient init 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 10 monclient: init 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 10 monclient: _add_conns ranks=[1,3,0,7,8,4,5,2,6] 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccd00a2f40 0x7fccd01cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 10 monclient(hunting): picked mon.noname-f con 0x7fccd00a2f40 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccd00a4df0 0x7fccd01cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 10 monclient(hunting): picked mon.noname-g con 0x7fccd00a4df0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd00a51d0 0x7fccd0123000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 10 monclient(hunting): picked mon.noname-a con 0x7fccd00a51d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fccd00a1250 con 0x7fccd00a51d0 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fccd0137320 con 0x7fccd00a2f40 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fccd0126f40 con 0x7fccd00a4df0 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fccd501e640 10 monclient(hunting): authenticate will time out at 1688.416853s 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fcccf59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd00a51d0 0x7fccd0123000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fcccf59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd00a51d0 0x7fccd0123000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53656/0 (socket says 172.21.15.98:53656) 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fcccf59e640 1 -- 172.21.15.98:0/3321764166 learned_addr learned my addr 172.21.15.98:0/3321764166 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fcccf59e640 10 monclient(hunting): get_auth_request con 0x7fccd00a51d0 auth_method 0 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fcccf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fcccf59e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.609+0000 7fcccf59e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccce59c640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccd00a4df0 0x7fccd01cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccce59c640 10 monclient(hunting): get_auth_request con 0x7fccd00a4df0 auth_method 0 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccce59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccce59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccced9d640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccd00a2f40 0x7fccd01cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccced9d640 10 monclient(hunting): get_auth_request con 0x7fccd00a2f40 auth_method 0 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccced9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccced9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 10 monclient(hunting): handle_auth_done global_id 66069 payload 824 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 1 -- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccd00a4df0 msgr2=0x7fccd01cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccd00a4df0 0x7fccd01cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 1 -- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccd00a2f40 msgr2=0x7fccd01cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccd00a2f40 0x7fccd01cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 10 monclient: _finish_hunting 0 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 1 monclient: found mon.noname-a 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 1 -- 172.21.15.98:0/3321764166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd0125080 con 0x7fccd00a51d0 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccce59c640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccd00a4df0 0x7fccd01cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccf59e640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd00a51d0 0x7fccd0123000 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fccc8007e90 tx=0x7fccc800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 10 client.?.objecter ms_handle_connect 0x7fccd00a51d0 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fccc800ed70 con 0x7fccd00a51d0 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 10 monclient: dump: 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 10 monclient: _finish_auth 0 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 10 monclient: _check_auth_tickets 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccd501e640 5 monclient: authenticate success, global_id 66069 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccc8019070 con 0x7fccd00a51d0 2024-06-11T19:24:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccd501e640 10 monclient: _renew_subs 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccd501e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccd501e640 1 -- 172.21.15.98:0/3321764166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccd0126c60 con 0x7fccd00a51d0 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fccc8014d00 con 0x7fccd00a51d0 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccd501e640 10 client.66069.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccb77fe640 10 monclient: dump: 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.631 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.631 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.631 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.631 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.631 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.631 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.631 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.631 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.631 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.631 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccd501e640 10 monclient: _renew_subs 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccd501e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fcccd59a640 4 set_mon_vals no callback set 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccd501e640 1 -- 172.21.15.98:0/3321764166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccd00a8a10 con 0x7fccd00a51d0 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.610+0000 7fccd501e640 1 librados: init done 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccd501e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccd501e640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd01266a0 0x7fccd0123540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccd501e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccd501e640 1 -- 172.21.15.98:0/3321764166 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fccd00a13d0 con 0x7fccd01266a0 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fccc8014420 con 0x7fccd00a51d0 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccced9d640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd01266a0 0x7fccd0123540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccced9d640 10 monclient: get_auth_request con 0x7fccd01266a0 auth_method 0 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccced9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccced9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccb77fe640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fccbc03fdb0 0x7fccbc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(623..623 src has 1..623) ==== 6802+0+0 (secure 0 0 0) 0x7fccc80524a0 con 0x7fccd00a51d0 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccb77fe640 10 client.66069.objecter ms_dispatch 0x7fccd0000b90 osd_map(623..623 src has 1..623) 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccb77fe640 3 client.66069.objecter handle_osd_map got epochs [623,623] > 0 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccb77fe640 3 client.66069.objecter handle_osd_map decoding full epoch 623 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccb77fe640 20 client.66069.objecter dump_active .. 0 homeless 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccce59c640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fccbc03fdb0 0x7fccbc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.611+0000 7fccce59c640 10 monclient: get_auth_request con 0x7fccbc03fdb0 auth_method 0 2024-06-11T19:24:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccced9d640 10 monclient(hunting): handle_auth_done global_id 84841 payload 824 2024-06-11T19:24:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccced9d640 1 -- 172.21.15.98:0/3321764166 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fccd0126a80 con 0x7fccd01266a0 2024-06-11T19:24:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 10 client.66069.objecter ms_handle_connect 0x7fccd01266a0 2024-06-11T19:24:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 10 client.66069.objecter resend_mon_ops 2024-06-11T19:24:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccced9d640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd01266a0 0x7fccd0123540 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fccc0034000 tx=0x7fccc00046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 10 client.66069.objecter ms_handle_connect 0x7fccbc03fdb0 2024-06-11T19:24:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccce59c640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fccbc03fdb0 0x7fccbc042270 secure :-1 s=READY pgs=4289 cs=0 l=1 rev1=1 crypto rx=0x7fccc4000f60 tx=0x7fccc4007c10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fccc0037040 con 0x7fccd01266a0 2024-06-11T19:24:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:24:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 10 monclient: dump: 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.636 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.636 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fccc0003820 con 0x7fccd01266a0 2024-06-11T19:24:38.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:38.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:38.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd01266a0 msgr2=0x7fccd0123540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd01266a0 0x7fccd0123540 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fccc0034000 tx=0x7fccc00046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd01266a0 msgr2=0x7fccd0123540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:38.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.612+0000 7fccb77fe640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd01266a0 0x7fccd0123540 secure :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fccc0034000 tx=0x7fccc00046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.618+0000 7fccd501e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:38.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.618+0000 7fccd501e640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd0123a80 0x7fccd0123e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.618+0000 7fccd501e640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.618+0000 7fccd501e640 1 -- 172.21.15.98:0/3321764166 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fccd0050830 con 0x7fccd0123a80 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccced9d640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd0123a80 0x7fccd0123e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccced9d640 10 monclient: get_auth_request con 0x7fccd0123a80 auth_method 0 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccced9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccced9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccced9d640 10 monclient(hunting): handle_auth_done global_id 84850 payload 824 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccced9d640 1 -- 172.21.15.98:0/3321764166 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fccd00a84c0 con 0x7fccd0123a80 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccb77fe640 10 client.66069.objecter ms_handle_connect 0x7fccd0123a80 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccb77fe640 10 client.66069.objecter resend_mon_ops 2024-06-11T19:24:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccced9d640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd0123a80 0x7fccd0123e90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fccc002b7a0 tx=0x7fccc0030000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fccc0037050 con 0x7fccd0123a80 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccb77fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.619+0000 7fccb77fe640 10 monclient: dump: 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.641 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7fccc0048030 con 0x7fccd0123a80 2024-06-11T19:24:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccb77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccb77fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd0123a80 msgr2=0x7fccd0123e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccb77fe640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd0123a80 0x7fccd0123e90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fccc002b7a0 tx=0x7fccc0030000 comp rx=0 tx=0).stop 2024-06-11T19:24:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccb77fe640 1 -- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd0123a80 msgr2=0x7fccd0123e90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccb77fe640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fccd0123a80 0x7fccd0123e90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 550389, 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:38.642 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:38.643 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:38.644 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:38.645 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:38.646 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:38.646 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:38.646 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:38.646 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:38.646 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:38.646 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:38.646 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:38.646 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:38.646 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:38.646 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:38.646 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:38.646 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:38.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:38.648 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.649 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.650 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:38.650 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.650 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:38.650 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.650 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.652 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.654 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:38.654 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.654 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.654 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.654 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:38.654 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:38.654 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.654 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.654 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.655 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.657 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:38.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.660 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:38.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:38.662 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:38.664 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:38.665 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:38.665 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:38.665 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:38.665 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:38.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:38.665 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:38.665 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 10 librados: watch_flush enter 2024-06-11T19:24:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 10 librados: watch_flush exit 2024-06-11T19:24:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 20 client.66069.objecter shutdown clearing up homeless session... 2024-06-11T19:24:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 10 client.66069.objecter successfully canceled tick 2024-06-11T19:24:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 1 -- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fccbc03fdb0 msgr2=0x7fccbc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fccbc03fdb0 0x7fccbc042270 secure :-1 s=READY pgs=4289 cs=0 l=1 rev1=1 crypto rx=0x7fccc4000f60 tx=0x7fccc4007c10 comp rx=0 tx=0).stop 2024-06-11T19:24:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 10 monclient: shutdown 2024-06-11T19:24:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 1 -- 172.21.15.98:0/3321764166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd00a51d0 msgr2=0x7fccd0123000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd00a51d0 0x7fccd0123000 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fccc8007e90 tx=0x7fccc800e4b0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fcccf59e640 1 -- 172.21.15.98:0/3321764166 reap_dead start 2024-06-11T19:24:38.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 1 -- 172.21.15.98:0/3321764166 shutdown_connections 2024-06-11T19:24:38.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 1 --2- 172.21.15.98:0/3321764166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccd00a51d0 0x7fccd0123000 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 1 -- 172.21.15.98:0/3321764166 >> 172.21.15.98:0/3321764166 conn(0x7fccd00a95b0 msgr2=0x7fccd009dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:38.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 1 -- 172.21.15.98:0/3321764166 shutdown_connections 2024-06-11T19:24:38.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 1 -- 172.21.15.98:0/3321764166 wait complete. 2024-06-11T19:24:38.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.620+0000 7fccd501e640 1 librados: shutdown 2024-06-11T19:24:38.668 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:38.668 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-06-11T19:24:38.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-06-11T19:24:38.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.769+0000 7f76c9bb5640 10 monclient: get_monmap_and_config 2024-06-11T19:24:38.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.769+0000 7f76c9bb5640 10 monclient: build_initial_monmap 2024-06-11T19:24:38.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.769+0000 7f76c9bb5640 10 monclient: monmap: 2024-06-11T19:24:38.768 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:38.768 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.768 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:38.770120+0000 2024-06-11T19:24:38.768 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:38.770120+0000 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:38.769 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.769+0000 7f76c9bb5640 1 Processor -- start 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.769+0000 7f76c9bb5640 1 -- start start 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.769+0000 7f76c9bb5640 10 monclient: init 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 10 monclient: _add_conns ranks=[3,0,7,4,1,2,8,5,9,10,6] 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c40a4920 0x7f76c40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 10 monclient(hunting): picked mon.noname-g con 0x7f76c40a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76c40a5240 addr v1:172.21.15.98:6789/0 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f76c40a2530 0x7f76c40a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 10 monclient(hunting): picked mon.noname-i con 0x7f76c40a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76c40a7140 con 0x7f76c40a5240 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f76c40a2b50 con 0x7f76c40a4920 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f76c40a2e30 con 0x7f76c40a2530 2024-06-11T19:24:38.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c9bb5640 10 monclient(hunting): authenticate will time out at 1688.577432s 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c2ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f76c40a5240 0x7f76c40a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53798/0 (socket says 172.21.15.98:53798) 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c2ffd640 1 -- 172.21.15.98:0/2224168101 learned_addr learned my addr 172.21.15.98:0/2224168101 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c37fe640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c40a4920 0x7f76c40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1340286952 0 0) 0x7f76c40a7140 con 0x7f76c40a5240 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 10 monclient(hunting): _negotiate 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 10 monclient(hunting): my global_id is 66078 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76ac003520 con 0x7f76c40a5240 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c37fe640 10 monclient(hunting): get_auth_request con 0x7f76c40a4920 auth_method 0 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c3fff640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f76c40a2530 0x7f76c40a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c3fff640 10 monclient(hunting): get_auth_request con 0x7f76c40a2530 auth_method 0 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2930605676 0 0) 0x7f76ac003520 con 0x7f76c40a5240 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f76c40a2530 msgr2=0x7f76c40a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f76c40a2530 0x7f76c40a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c40a4920 msgr2=0x7f76c40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c40a4920 0x7f76c40a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 10 monclient: _finish_hunting 0 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 1 monclient: found mon.noname-a-legacy 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c40a3110 con 0x7f76c40a5240 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.770+0000 7f76c27fc640 10 monclient: _finish_auth 0 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient: _check_auth_tickets 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7f76b8001960 con 0x7f76c40a5240 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c9bb5640 5 monclient: authenticate success, global_id 66078 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c9bb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient: dump: 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.785 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 >> v1:172.21.15.98:6789/0 conn(0x7f76c40a5240 legacy=0x7f76c40a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient: _add_conns ranks=[2,3,1,8,5,4,6,0,7] 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76ac004200 0x7f76ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient(hunting): picked mon.b con 0x7f76ac004200 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c40a4920 0x7f76ac004df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient(hunting): picked mon.g con 0x7f76c40a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76ac0099b0 0x7f76ac00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient(hunting): picked mon.f con 0x7f76ac0099b0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f76c40a2b50 con 0x7f76ac0099b0 2024-06-11T19:24:38.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f76c40a2e30 con 0x7f76ac004200 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f76ac00c310 con 0x7f76c40a4920 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c27fc640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c9bb5640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76ac004200 0x7f76ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 10 monclient(hunting): get_auth_request con 0x7f76ac004200 auth_method 0 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c2ffd640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c40a4920 0x7f76ac004df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c2ffd640 10 monclient(hunting): get_auth_request con 0x7f76c40a4920 auth_method 0 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c3fff640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76ac0099b0 0x7f76ac00bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c3fff640 10 monclient(hunting): get_auth_request con 0x7f76ac0099b0 auth_method 0 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 10 monclient(hunting): handle_auth_done global_id 66078 payload 293 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 1 -- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c40a4920 msgr2=0x7f76ac004df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c40a4920 0x7f76ac004df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 1 -- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76ac0099b0 msgr2=0x7f76ac00bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76ac0099b0 0x7f76ac00bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 10 monclient: _finish_hunting 0 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 1 monclient: found mon.b 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 1 -- 172.21.15.98:0/2224168101 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76ac003520 con 0x7f76ac004200 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 10 monclient: _finish_auth 0 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 10 monclient: _check_auth_tickets 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:38.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.771+0000 7f76c2ffd640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c40a4920 0x7f76ac004df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c37fe640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76ac004200 0x7f76ac004800 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f76b4009a90 tx=0x7f76ac005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=905504ccba0a6834 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76b4039070 con 0x7f76ac004200 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c27fc640 10 monclient: dump: 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.790 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c9bb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b402caf0 con 0x7f76ac004200 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c27fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c27fc640 1 -- 172.21.15.98:0/2224168101 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76b4009690 con 0x7f76ac004200 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c9bb5640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:38.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c9bb5640 4 set_mon_vals no callback set 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c27fc640 10 monclient: dump: 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.792 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c9bb5640 10 monclient: shutdown 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c9bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c9bb5640 1 -- 172.21.15.98:0/2224168101 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76ac004200 msgr2=0x7f76ac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c9bb5640 1 --2- 172.21.15.98:0/2224168101 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f76ac004200 0x7f76ac004800 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f76b4009a90 tx=0x7f76ac005830 comp rx=0 tx=0).stop 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c3fff640 1 -- 172.21.15.98:0/2224168101 reap_dead start 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c9bb5640 1 -- 172.21.15.98:0/2224168101 shutdown_connections 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c9bb5640 1 -- 172.21.15.98:0/2224168101 >> 172.21.15.98:0/2224168101 conn(0x7f76c40ab850 msgr2=0x7f76c40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c9bb5640 1 -- 172.21.15.98:0/2224168101 shutdown_connections 2024-06-11T19:24:38.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.772+0000 7f76c9bb5640 1 -- 172.21.15.98:0/2224168101 wait complete. 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.773+0000 7f76c9bb5640 10 monclient: build_initial_monmap 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.773+0000 7f76c9bb5640 10 monclient: monmap: 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:38.794 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.773+0000 7f76c9bb5640 1 librados: starting msgr at 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.773+0000 7f76c9bb5640 1 librados: starting objecter 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.773+0000 7f76c9bb5640 1 Processor -- start 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 1 -- start start 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 1 librados: setting wanted keys 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 1 librados: calling monclient init 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 10 monclient: init 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 10 monclient: _add_conns ranks=[8,3,7,0,5,4,1,2,6] 2024-06-11T19:24:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76ac004200 0x7f76c41cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 10 monclient(hunting): picked mon.noname-e con 0x7f76ac004200 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c41ce010 0x7f76c4122230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 10 monclient(hunting): picked mon.noname-g con 0x7f76c41ce010 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f76c41ce3f0 0x7f76c4122770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 10 monclient(hunting): picked mon.noname-i con 0x7f76c41ce3f0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f76c40a6da0 con 0x7f76c41ce010 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f76c4138e50 con 0x7f76c41ce3f0 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f76c40a65f0 con 0x7f76ac004200 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:38.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c9bb5640 10 monclient(hunting): authenticate will time out at 1688.581486s 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76ac004200 0x7f76c41cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76ac004200 0x7f76c41cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:53154/0 (socket says 172.21.15.98:53154) 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 1 -- 172.21.15.98:0/3183290281 learned_addr learned my addr 172.21.15.98:0/3183290281 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 10 monclient(hunting): get_auth_request con 0x7f76ac004200 auth_method 0 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c2ffd640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c41ce010 0x7f76c4122230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c3fff640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f76c41ce3f0 0x7f76c4122770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 10 monclient(hunting): handle_auth_done global_id 124172 payload 824 2024-06-11T19:24:38.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 1 -- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f76c41ce3f0 msgr2=0x7f76c4122770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f76c41ce3f0 0x7f76c4122770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 1 -- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c41ce010 msgr2=0x7f76c4122230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c41ce010 0x7f76c4122230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 10 monclient: _finish_hunting 0 2024-06-11T19:24:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 1 monclient: found mon.noname-e 2024-06-11T19:24:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c37fe640 1 -- 172.21.15.98:0/3183290281 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c41241a0 con 0x7f76ac004200 2024-06-11T19:24:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c2ffd640 10 monclient: get_auth_request con 0x7f76c41ce010 auth_method 0 2024-06-11T19:24:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.774+0000 7f76c2ffd640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c41ce010 0x7f76c4122230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:24:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76c37fe640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76ac004200 0x7f76c41cdad0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f76b402c2f0 tx=0x7f76b4004730 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 10 client.?.objecter ms_handle_connect 0x7f76ac004200 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 1 -- 172.21.15.98:0/3183290281 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76b4042020 con 0x7f76ac004200 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 10 monclient: dump: 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 10 monclient: _finish_auth 0 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76c9bb5640 5 monclient: authenticate success, global_id 124172 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 1 -- 172.21.15.98:0/3183290281 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b403a040 con 0x7f76ac004200 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76c9bb5640 10 monclient: _renew_subs 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76c9bb5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76c9bb5640 1 -- 172.21.15.98:0/3183290281 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76c4125d70 con 0x7f76ac004200 2024-06-11T19:24:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 1 -- 172.21.15.98:0/3183290281 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76b4039070 con 0x7f76ac004200 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76c9bb5640 10 client.124172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76abfff640 10 monclient: dump: 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:38.801 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76c9bb5640 10 monclient: _renew_subs 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76c9bb5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76c9bb5640 1 -- 172.21.15.98:0/3183290281 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c40aaa60 con 0x7f76ac004200 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76c1ffb640 4 set_mon_vals no callback set 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.775+0000 7f76c9bb5640 1 librados: init done 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76abfff640 1 -- 172.21.15.98:0/3183290281 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f76b4045020 con 0x7f76ac004200 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76c9bb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:38.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76c9bb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76c9bb5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76c9bb5640 1 -- 172.21.15.98:0/3183290281 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76c40aac90 con 0x7f76ac004200 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76abfff640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f76b003fdb0 0x7f76b0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76abfff640 1 -- 172.21.15.98:0/3183290281 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(623..623 src has 1..623) ==== 6802+0+0 (secure 0 0 0) 0x7f76b400f040 con 0x7f76ac004200 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76abfff640 10 client.124172.objecter ms_dispatch 0x7f76c4000b90 osd_map(623..623 src has 1..623) 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76abfff640 3 client.124172.objecter handle_osd_map got epochs [623,623] > 0 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76abfff640 3 client.124172.objecter handle_osd_map decoding full epoch 623 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76c2ffd640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f76b003fdb0 0x7f76b0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76c2ffd640 10 monclient: get_auth_request con 0x7f76b003fdb0 auth_method 0 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76abfff640 20 client.124172.objecter dump_active .. 0 homeless 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76c2ffd640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f76b003fdb0 0x7f76b0042270 secure :-1 s=READY pgs=4290 cs=0 l=1 rev1=1 crypto rx=0x7f76b8001160 tx=0x7f76b8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.776+0000 7f76abfff640 10 client.124172.objecter ms_handle_connect 0x7f76b003fdb0 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.780+0000 7f76abfff640 1 -- 172.21.15.98:0/3183290281 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f76c40aac90 con 0x7f76ac004200 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.780+0000 7f76abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:38.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:38.780+0000 7f76abfff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:39.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.034+0000 7f76c9bb5640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-06-11T19:24:39.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.034+0000 7f76c9bb5640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-06-11T19:24:39.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.034+0000 7f76c9bb5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:39.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.034+0000 7f76c9bb5640 1 -- 172.21.15.98:0/3183290281 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f76c40a4d00 con 0x7f76ac004200 2024-06-11T19:24:39.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f76abfff640 1 -- 172.21.15.98:0/3183290281 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f76b404c070 con 0x7f76ac004200 2024-06-11T19:24:39.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f76abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-06-11T19:24:39.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f76abfff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:39.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=1648878454,config=692619927,health=1191435648,kv=2000936593,logm=3840723033}) 2024-06-11T19:24:39.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=9,logm=56} crc {auth=466330138,config=692619927,health=4128661616,kv=1471832156,logm=2957545696}) 2024-06-11T19:24:39.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=1648878454,config=692619927,health=1191435648,kv=2000936593,logm=3840723033}) 2024-06-11T19:24:39.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=12,kv=9,logm=56} crc {auth=1869967988,config=692619927,health=4128661616,kv=2000936593,logm=3848033113}) 2024-06-11T19:24:39.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=1648878454,config=692619927,health=1191435648,kv=2000936593,logm=3840723033}) 2024-06-11T19:24:39.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=1499110559,config=692619927,health=3205329809,kv=2236601199,logm=774848917}) 2024-06-11T19:24:39.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=1648878454,config=692619927,health=1191435648,kv=2000936593,logm=3840723033}) 2024-06-11T19:24:39.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=21,config=2,health=12,kv=9,logm=56} crc {auth=1869967988,config=692619927,health=4128661616,kv=1471832156,logm=3899519823}) 2024-06-11T19:24:39.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=1648878454,config=692619927,health=1191435648,kv=2000936593,logm=3840723033}) 2024-06-11T19:24:39.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=19,config=2,health=12,kv=10,logm=57} crc {auth=1013488559,config=692619927,health=4128661616,kv=2000936593,logm=2328527580}) 2024-06-11T19:24:39.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=1648878454,config=692619927,health=1191435648,kv=2000936593,logm=3840723033}) 2024-06-11T19:24:39.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=20,config=2,health=10,kv=8,logm=60} crc {auth=1678599751,config=692619927,health=3038048611,kv=2577091825,logm=1691968988}) 2024-06-11T19:24:39.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=1648878454,config=692619927,health=1191435648,kv=2000936593,logm=3840723033}) 2024-06-11T19:24:39.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=20,config=2,health=11,kv=9,logm=58} crc {auth=818621336,config=692619927,health=963827158,kv=3371622858,logm=1529473103}) 2024-06-11T19:24:39.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=1648878454,config=692619927,health=1191435648,kv=2000936593,logm=3840723033}) 2024-06-11T19:24:39.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.036+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=19,config=2,health=10,kv=9,logm=60} crc {auth=3903544460,config=692619927,health=2903782388,kv=3371622858,logm=1151541780}) 2024-06-11T19:24:39.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4220742180}) 2024-06-11T19:24:39.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f76c9bb5640 10 librados: watch_flush enter 2024-06-11T19:24:39.040 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3044832061}) 2024-06-11T19:24:39.040 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.040 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4220742180}) 2024-06-11T19:24:39.040 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4006062393}) 2024-06-11T19:24:39.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f76c9bb5640 10 librados: watch_flush exit 2024-06-11T19:24:39.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f76c9bb5640 20 client.124172.objecter shutdown clearing up homeless session... 2024-06-11T19:24:39.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f76c9bb5640 10 client.124172.objecter successfully canceled tick 2024-06-11T19:24:39.041 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.041 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4220742180}) 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1871674194}) 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4220742180}) 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1948173069}) 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.038+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4220742180}) 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1467474727}) 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4220742180}) 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3942835655}) 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4220742180}) 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=67112463}) 2024-06-11T19:24:39.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f76c9bb5640 1 -- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f76b003fdb0 msgr2=0x7f76b0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:39.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f76c9bb5640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f76b003fdb0 0x7f76b0042270 secure :-1 s=READY pgs=4290 cs=0 l=1 rev1=1 crypto rx=0x7f76b8001160 tx=0x7f76b8009040 comp rx=0 tx=0).stop 2024-06-11T19:24:39.043 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4220742180}) 2024-06-11T19:24:39.043 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1346977365}) 2024-06-11T19:24:39.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f76c9bb5640 10 monclient: shutdown 2024-06-11T19:24:39.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f76c9bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:39.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f76c9bb5640 1 -- 172.21.15.98:0/3183290281 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76ac004200 msgr2=0x7f76c41cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:39.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f76c9bb5640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76ac004200 0x7f76c41cdad0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f76b402c2f0 tx=0x7f76b4004730 comp rx=0 tx=0).stop 2024-06-11T19:24:39.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f76c9bb5640 1 -- 172.21.15.98:0/3183290281 shutdown_connections 2024-06-11T19:24:39.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f76c9bb5640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f76b003fdb0 0x7f76b0042270 unknown :-1 s=CLOSED pgs=4290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:39.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.039+0000 7f76c9bb5640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f76c41ce3f0 0x7f76c4122770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:39.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f76c9bb5640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76c41ce010 0x7f76c4122230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:39.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f76c9bb5640 1 --2- 172.21.15.98:0/3183290281 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76ac004200 0x7f76c41cdad0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:39.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f76c9bb5640 1 -- 172.21.15.98:0/3183290281 >> 172.21.15.98:0/3183290281 conn(0x7f76c40ab850 msgr2=0x7f76c40726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:39.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f76c9bb5640 1 -- 172.21.15.98:0/3183290281 shutdown_connections 2024-06-11T19:24:39.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f76c9bb5640 1 -- 172.21.15.98:0/3183290281 wait complete. 2024-06-11T19:24:39.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f76c9bb5640 1 librados: shutdown 2024-06-11T19:24:39.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267979718}) 2024-06-11T19:24:39.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416768827}) 2024-06-11T19:24:39.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267979718}) 2024-06-11T19:24:39.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2237264563}) 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267979718}) 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1955625487}) 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267979718}) 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=57210574}) 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267979718}) 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3478445987}) 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267979718}) 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3103183437}) 2024-06-11T19:24:39.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267979718}) 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2017043813}) 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267979718}) 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.040+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3878481065}) 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894103897}) 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2363302002}) 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894103897}) 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465523820}) 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894103897}) 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1915220952}) 2024-06-11T19:24:39.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894103897}) 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3898405738}) 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894103897}) 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3089222073}) 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894103897}) 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3427353836}) 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894103897}) 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=644706673}) 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894103897}) 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.042+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1702960389}) 2024-06-11T19:24:39.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1924327519}) 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2449528906}) 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1924327519}) 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=742280486}) 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1924327519}) 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2929535006}) 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1924327519}) 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=320210891}) 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1924327519}) 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3331165768}) 2024-06-11T19:24:39.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1924327519}) 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4267719088}) 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1924327519}) 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=986774145}) 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1924327519}) 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.044+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2885977025}) 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019712378}) 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2066661412}) 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019712378}) 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1491988212}) 2024-06-11T19:24:39.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019712378}) 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=32838143}) 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019712378}) 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3736108797}) 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019712378}) 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=940050775}) 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019712378}) 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2879220472}) 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019712378}) 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=355183528}) 2024-06-11T19:24:39.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019712378}) 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.046+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1921676387}) 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748181361}) 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3527769106}) 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748181361}) 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1027611962}) 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748181361}) 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3123469676}) 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748181361}) 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4260429118}) 2024-06-11T19:24:39.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748181361}) 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1643558692}) 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748181361}) 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=593337124}) 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748181361}) 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3686241638}) 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748181361}) 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.047+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=100183964}) 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.048+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.048+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332983853}) 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.048+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1149320837}) 2024-06-11T19:24:39.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.048+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.048+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332983853}) 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.048+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2354440265}) 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.048+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.048+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332983853}) 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.048+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3449850862}) 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.048+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332983853}) 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3815082618}) 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332983853}) 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2037844493}) 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332983853}) 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3747957482}) 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332983853}) 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1305650920}) 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332983853}) 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.049+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=530456980}) 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462268772}) 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160543938}) 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462268772}) 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1223658198}) 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462268772}) 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3302192473}) 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462268772}) 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=16043264}) 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462268772}) 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3024104107}) 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462268772}) 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3534158166}) 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462268772}) 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3431757381}) 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462268772}) 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.050+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2594111804}) 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057770797}) 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1337204272}) 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057770797}) 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3533419968}) 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057770797}) 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3133762727}) 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057770797}) 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=590202061}) 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057770797}) 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3596816212}) 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057770797}) 2024-06-11T19:24:39.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=560299426}) 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057770797}) 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3417334713}) 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057770797}) 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.051+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2126197327}) 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433660531}) 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2561545517}) 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433660531}) 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3317182879}) 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433660531}) 2024-06-11T19:24:39.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3834750477}) 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433660531}) 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2675171338}) 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433660531}) 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2079907505}) 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433660531}) 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2084249025}) 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.052+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433660531}) 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.053+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1880152839}) 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.053+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.053+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433660531}) 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.053+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3981754895}) 2024-06-11T19:24:39.072 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532636166}) 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1526701173}) 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532636166}) 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1580291071}) 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532636166}) 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=964295378}) 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532636166}) 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=884853728}) 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532636166}) 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=617707335}) 2024-06-11T19:24:39.073 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532636166}) 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4202999833}) 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532636166}) 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=723594282}) 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532636166}) 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.054+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=352678211}) 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126113902}) 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=877147876}) 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126113902}) 2024-06-11T19:24:39.074 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=238250342}) 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126113902}) 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3400679455}) 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126113902}) 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1398945801}) 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126113902}) 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3538748592}) 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126113902}) 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1938099233}) 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.075 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126113902}) 2024-06-11T19:24:39.076 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1956053798}) 2024-06-11T19:24:39.076 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.076 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126113902}) 2024-06-11T19:24:39.076 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.055+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3659423014}) 2024-06-11T19:24:39.077 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-06-11T19:24:39.325 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103622444}) 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728971601}) 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103622444}) 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2583683896}) 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103622444}) 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3328384176}) 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103622444}) 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1640243923}) 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.326 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103622444}) 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=675680323}) 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103622444}) 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1205893772}) 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103622444}) 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=42435756}) 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103622444}) 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.056+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1004684576}) 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507128429}) 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372473052}) 2024-06-11T19:24:39.327 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507128429}) 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3675246323}) 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507128429}) 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1607066585}) 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507128429}) 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2173315307}) 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507128429}) 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1139562721}) 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507128429}) 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2998761008}) 2024-06-11T19:24:39.328 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507128429}) 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3148185814}) 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507128429}) 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.058+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2921199973}) 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1411804501}) 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915019873}) 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1411804501}) 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2296766397}) 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1411804501}) 2024-06-11T19:24:39.329 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1737513022}) 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1411804501}) 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=192602939}) 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1411804501}) 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1427123682}) 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1411804501}) 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1021832546}) 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1411804501}) 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1278466757}) 2024-06-11T19:24:39.330 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1411804501}) 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.059+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1651921428}) 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2239500398}) 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611790312}) 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2239500398}) 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=13218551}) 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2239500398}) 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1381490443}) 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2239500398}) 2024-06-11T19:24:39.331 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3403034331}) 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2239500398}) 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1143381548}) 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2239500398}) 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1176483681}) 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2239500398}) 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=367088643}) 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2239500398}) 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.060+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3420891446}) 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.332 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368136352}) 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3393988625}) 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368136352}) 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2952105608}) 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368136352}) 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3858048167}) 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368136352}) 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3033920945}) 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.061+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.062+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368136352}) 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.062+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2844900062}) 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.062+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.333 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.062+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368136352}) 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.062+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1159262500}) 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.062+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.062+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368136352}) 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.062+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3426623980}) 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.062+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.062+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368136352}) 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.062+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3535107028}) 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730861187}) 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1198553722}) 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730861187}) 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2550986913}) 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.334 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730861187}) 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3017280138}) 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730861187}) 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3321935861}) 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730861187}) 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2851406145}) 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730861187}) 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1968996126}) 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730861187}) 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=333907740}) 2024-06-11T19:24:39.335 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730861187}) 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.063+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2710903692}) 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458623762}) 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3459468869}) 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458623762}) 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=346975946}) 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458623762}) 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2736959516}) 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458623762}) 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1914376219}) 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.336 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458623762}) 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2786702217}) 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458623762}) 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1217656645}) 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458623762}) 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2434019540}) 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458623762}) 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.064+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3106781553}) 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.065+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.065+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110026047}) 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.065+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543272298}) 2024-06-11T19:24:39.337 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.065+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110026047}) 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1716310302}) 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110026047}) 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3986859281}) 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110026047}) 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=312776471}) 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110026047}) 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2781517057}) 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110026047}) 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=844582555}) 2024-06-11T19:24:39.338 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110026047}) 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2722822911}) 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110026047}) 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.066+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=399575344}) 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001119481}) 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095273893}) 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001119481}) 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3400083639}) 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001119481}) 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2782723093}) 2024-06-11T19:24:39.339 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001119481}) 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1488650634}) 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001119481}) 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2842105705}) 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001119481}) 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3776825224}) 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001119481}) 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=445435879}) 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001119481}) 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.067+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=178961206}) 2024-06-11T19:24:39.340 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702247575}) 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303819924}) 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702247575}) 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584561393}) 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702247575}) 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=467445771}) 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702247575}) 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1660654788}) 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702247575}) 2024-06-11T19:24:39.341 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4016813575}) 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702247575}) 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2669241624}) 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702247575}) 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3791939690}) 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702247575}) 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.068+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1180481189}) 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.069+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.069+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882179557}) 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.069+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2545566861}) 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.069+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.069+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882179557}) 2024-06-11T19:24:39.342 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.069+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3697274599}) 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.069+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.069+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882179557}) 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=680533081}) 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882179557}) 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3311141476}) 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882179557}) 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3390900962}) 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882179557}) 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3487239495}) 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.343 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882179557}) 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1730711314}) 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882179557}) 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.070+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2556884188}) 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753544184}) 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225987871}) 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753544184}) 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2752140971}) 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753544184}) 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=54512118}) 2024-06-11T19:24:39.344 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753544184}) 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3164290086}) 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753544184}) 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1670158995}) 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753544184}) 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=418694500}) 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753544184}) 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3081242375}) 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753544184}) 2024-06-11T19:24:39.345 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.071+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3785796935}) 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247264377}) 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1095298129}) 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247264377}) 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3387138145}) 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247264377}) 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1541542461}) 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247264377}) 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2140076074}) 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247264377}) 2024-06-11T19:24:39.346 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4293993962}) 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247264377}) 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2883192988}) 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247264377}) 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1649705186}) 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247264377}) 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.072+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1999102720}) 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.073+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.073+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413278118}) 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.073+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1272356102}) 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.073+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.073+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413278118}) 2024-06-11T19:24:39.347 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.073+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2268551214}) 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.073+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.073+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413278118}) 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3173744361}) 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413278118}) 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3882274134}) 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413278118}) 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1456030043}) 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413278118}) 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3347026244}) 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413278118}) 2024-06-11T19:24:39.348 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=846985297}) 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413278118}) 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.074+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1569296814}) 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789146769}) 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=862079880}) 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789146769}) 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1363920245}) 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789146769}) 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=311058551}) 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.349 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789146769}) 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2961971283}) 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789146769}) 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1111176368}) 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789146769}) 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=10010044}) 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789146769}) 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=741844429}) 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789146769}) 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.075+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1073384175}) 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.350 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3539067179}) 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2078750528}) 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3539067179}) 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2927804516}) 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3539067179}) 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2387801247}) 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3539067179}) 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1408201786}) 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3539067179}) 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3453055921}) 2024-06-11T19:24:39.351 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3539067179}) 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4263809635}) 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3539067179}) 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=96307123}) 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3539067179}) 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.076+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1331238251}) 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.077+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.077+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2073527269}) 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.077+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3540745014}) 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.077+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.077+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2073527269}) 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.077+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=90749392}) 2024-06-11T19:24:39.352 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.077+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.077+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2073527269}) 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=625523507}) 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2073527269}) 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3942078519}) 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2073527269}) 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1925165670}) 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2073527269}) 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=678179716}) 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2073527269}) 2024-06-11T19:24:39.353 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3200227030}) 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2073527269}) 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.078+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1000045978}) 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516672746}) 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3354668213}) 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516672746}) 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3902865740}) 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516672746}) 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2537384756}) 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.354 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516672746}) 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=562146347}) 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516672746}) 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3539297108}) 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516672746}) 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1954085353}) 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516672746}) 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2683084101}) 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516672746}) 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.079+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2142566797}) 2024-06-11T19:24:39.355 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638636941}) 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1954850959}) 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638636941}) 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2406223624}) 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638636941}) 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2910929830}) 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638636941}) 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1674760291}) 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638636941}) 2024-06-11T19:24:39.356 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=49817025}) 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638636941}) 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2356912291}) 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638636941}) 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2593068510}) 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638636941}) 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.080+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=297464084}) 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493283359}) 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134963469}) 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493283359}) 2024-06-11T19:24:39.357 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1362429664}) 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493283359}) 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3101568922}) 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493283359}) 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1805197984}) 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493283359}) 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=377804374}) 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493283359}) 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1464561260}) 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.358 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493283359}) 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3812317897}) 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493283359}) 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.082+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4117463842}) 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714514469}) 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428959010}) 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714514469}) 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=535682063}) 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714514469}) 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1383270254}) 2024-06-11T19:24:39.359 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714514469}) 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3539302566}) 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714514469}) 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=223558099}) 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714514469}) 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3159271874}) 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714514469}) 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3491501186}) 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714514469}) 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.083+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1136089029}) 2024-06-11T19:24:39.360 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688018990}) 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=937801088}) 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688018990}) 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116127624}) 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688018990}) 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=278804878}) 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688018990}) 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=847309954}) 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688018990}) 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2075493799}) 2024-06-11T19:24:39.361 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688018990}) 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2203539878}) 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688018990}) 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3912682232}) 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688018990}) 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.084+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4126595989}) 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247294561}) 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1032325239}) 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247294561}) 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2738855828}) 2024-06-11T19:24:39.362 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247294561}) 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1164537694}) 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247294561}) 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3525421357}) 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247294561}) 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3677278331}) 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247294561}) 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=885147392}) 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247294561}) 2024-06-11T19:24:39.363 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1523299863}) 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247294561}) 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.086+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3347082069}) 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563299204}) 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3020052525}) 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563299204}) 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3759124079}) 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563299204}) 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=620642239}) 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563299204}) 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2466280946}) 2024-06-11T19:24:39.364 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563299204}) 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2999407987}) 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563299204}) 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=977508623}) 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563299204}) 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=683279810}) 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563299204}) 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.087+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1068854125}) 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228393383}) 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203392857}) 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.365 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228393383}) 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4088062003}) 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228393383}) 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3336353517}) 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228393383}) 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=779742012}) 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228393383}) 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=74452948}) 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228393383}) 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.088+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2444986689}) 2024-06-11T19:24:39.366 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.089+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.089+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228393383}) 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.089+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3893911988}) 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.089+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.089+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228393383}) 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.089+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=540934668}) 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670232298}) 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372948949}) 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670232298}) 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4095151095}) 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670232298}) 2024-06-11T19:24:39.367 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2995300290}) 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670232298}) 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4157461655}) 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670232298}) 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3736015323}) 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670232298}) 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2290563979}) 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670232298}) 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3608946689}) 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670232298}) 2024-06-11T19:24:39.368 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.090+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4269265584}) 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862530530}) 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1078736431}) 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862530530}) 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3502679167}) 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862530530}) 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1396330674}) 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862530530}) 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3012279279}) 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862530530}) 2024-06-11T19:24:39.369 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1482308211}) 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862530530}) 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=139728862}) 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862530530}) 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1816685311}) 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862530530}) 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.091+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1605349475}) 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975020403}) 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290991884}) 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975020403}) 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2369703196}) 2024-06-11T19:24:39.370 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975020403}) 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1965269970}) 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975020403}) 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2840209615}) 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975020403}) 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3910119893}) 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975020403}) 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3428201320}) 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975020403}) 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1505312542}) 2024-06-11T19:24:39.371 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975020403}) 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.093+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1406611305}) 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747566625}) 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2640164810}) 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747566625}) 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=676072903}) 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747566625}) 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2155407041}) 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747566625}) 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1676726548}) 2024-06-11T19:24:39.372 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747566625}) 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1981708930}) 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747566625}) 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3481518295}) 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747566625}) 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1686749888}) 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747566625}) 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.094+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=105610344}) 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604333385}) 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=213828487}) 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.373 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604333385}) 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2612760680}) 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604333385}) 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1922755694}) 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604333385}) 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1072091551}) 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604333385}) 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3976000619}) 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604333385}) 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3725120617}) 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.374 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.095+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604333385}) 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.096+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1069157278}) 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.096+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.096+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604333385}) 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.096+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3867816476}) 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944040704}) 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976912455}) 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944040704}) 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1339915205}) 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944040704}) 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3727295731}) 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944040704}) 2024-06-11T19:24:39.375 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4022532866}) 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944040704}) 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2128809183}) 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944040704}) 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3099843639}) 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944040704}) 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1268526246}) 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944040704}) 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.097+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3106768346}) 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.376 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680610963}) 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211102580}) 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680610963}) 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1969200768}) 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680610963}) 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=754852227}) 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680610963}) 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=141163261}) 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680610963}) 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3430294428}) 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680610963}) 2024-06-11T19:24:39.377 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3143695520}) 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680610963}) 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=48606830}) 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680610963}) 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.098+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3468152663}) 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.099+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.099+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061716441}) 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.099+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2545880767}) 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.099+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.099+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061716441}) 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2895065983}) 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.378 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061716441}) 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2863223605}) 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061716441}) 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=9739080}) 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061716441}) 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2115296241}) 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061716441}) 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2657156894}) 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061716441}) 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=367689419}) 2024-06-11T19:24:39.379 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061716441}) 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.100+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2264508785}) 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224889427}) 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1513073673}) 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224889427}) 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522259324}) 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224889427}) 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3632000456}) 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224889427}) 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1360929527}) 2024-06-11T19:24:39.380 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224889427}) 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2838540839}) 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224889427}) 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1419506922}) 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224889427}) 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2295673422}) 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224889427}) 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.101+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3585873823}) 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794532608}) 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2221203101}) 2024-06-11T19:24:39.381 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794532608}) 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585326152}) 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794532608}) 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4042301209}) 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794532608}) 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3674472503}) 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794532608}) 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1801837984}) 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794532608}) 2024-06-11T19:24:39.382 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=66594958}) 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794532608}) 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1145418478}) 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794532608}) 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.102+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1555316830}) 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.103+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.103+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174333638}) 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.103+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2722763473}) 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.103+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.103+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174333638}) 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.103+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4003936001}) 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174333638}) 2024-06-11T19:24:39.383 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=764786338}) 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174333638}) 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1362839424}) 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174333638}) 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2107090747}) 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174333638}) 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1260899044}) 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174333638}) 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4076587388}) 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174333638}) 2024-06-11T19:24:39.384 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.104+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=992509904}) 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837426672}) 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443008205}) 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837426672}) 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4138929226}) 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837426672}) 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2067794051}) 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837426672}) 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3141272493}) 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.385 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837426672}) 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3413771180}) 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837426672}) 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=8112040}) 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837426672}) 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3913403162}) 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837426672}) 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.105+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3811867696}) 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562340465}) 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715748480}) 2024-06-11T19:24:39.386 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562340465}) 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1522418576}) 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562340465}) 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3633646814}) 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562340465}) 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1845433143}) 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.106+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.107+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562340465}) 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.107+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2303167143}) 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.107+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.387 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.107+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562340465}) 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.107+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4162939518}) 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.107+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.107+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562340465}) 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.107+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=641987152}) 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.107+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.107+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562340465}) 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.107+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3103305779}) 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354506491}) 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250840116}) 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354506491}) 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1894227834}) 2024-06-11T19:24:39.388 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354506491}) 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3452771809}) 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354506491}) 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4002269590}) 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354506491}) 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2698922312}) 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354506491}) 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3837575034}) 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354506491}) 2024-06-11T19:24:39.389 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3623951913}) 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354506491}) 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.108+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3183555551}) 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221676577}) 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145376883}) 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221676577}) 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=576009079}) 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221676577}) 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=806514433}) 2024-06-11T19:24:39.390 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221676577}) 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1890765062}) 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221676577}) 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=671365116}) 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221676577}) 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1785665748}) 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221676577}) 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1826753265}) 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221676577}) 2024-06-11T19:24:39.391 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.109+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3060256936}) 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.110+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.110+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663410500}) 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.110+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725154642}) 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.110+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.110+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663410500}) 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.110+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1397583430}) 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.110+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.110+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663410500}) 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.110+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1728283327}) 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.110+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.110+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663410500}) 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=562202593}) 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663410500}) 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1162232035}) 2024-06-11T19:24:39.392 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663410500}) 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3411391568}) 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663410500}) 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3103526418}) 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663410500}) 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.111+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=71731699}) 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234702171}) 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2491968274}) 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234702171}) 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603384386}) 2024-06-11T19:24:39.393 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234702171}) 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2860903618}) 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234702171}) 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=276608108}) 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234702171}) 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3492453156}) 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234702171}) 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1838779946}) 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234702171}) 2024-06-11T19:24:39.394 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=270747082}) 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234702171}) 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.112+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3761271446}) 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191364729}) 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710622754}) 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191364729}) 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3737347849}) 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191364729}) 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2582565271}) 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191364729}) 2024-06-11T19:24:39.395 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2948001779}) 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191364729}) 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2073190306}) 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191364729}) 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3044630451}) 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191364729}) 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3922155621}) 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191364729}) 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.113+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=327058895}) 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.114+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.114+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=949721706}) 2024-06-11T19:24:39.396 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.114+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432628802}) 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.114+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.114+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=949721706}) 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.114+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083604584}) 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.114+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=949721706}) 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=701069032}) 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=949721706}) 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1875994676}) 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=949721706}) 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1482924605}) 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=949721706}) 2024-06-11T19:24:39.397 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2264284538}) 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=949721706}) 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2496091402}) 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=949721706}) 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.115+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2684748072}) 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570329997}) 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597107304}) 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570329997}) 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3693484521}) 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570329997}) 2024-06-11T19:24:39.398 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523013107}) 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570329997}) 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3617890592}) 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570329997}) 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3887862194}) 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570329997}) 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=437970638}) 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570329997}) 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=247331476}) 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570329997}) 2024-06-11T19:24:39.399 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.116+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=468780299}) 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182318006}) 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1267803137}) 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182318006}) 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3244680093}) 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182318006}) 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4003112120}) 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182318006}) 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3829260167}) 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182318006}) 2024-06-11T19:24:39.400 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1587308376}) 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182318006}) 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2210823354}) 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182318006}) 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2302778685}) 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182318006}) 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.117+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3611697136}) 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.118+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.118+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429484451}) 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.118+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679029494}) 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.118+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429484451}) 2024-06-11T19:24:39.401 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3247267173}) 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429484451}) 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2081567504}) 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429484451}) 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3412429905}) 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429484451}) 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3405435636}) 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429484451}) 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3459646434}) 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.402 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429484451}) 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3584472235}) 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429484451}) 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.119+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2176527614}) 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982534719}) 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=993270831}) 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982534719}) 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495604067}) 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982534719}) 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=334457305}) 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.403 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982534719}) 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3528274316}) 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982534719}) 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2850260626}) 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982534719}) 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4099781678}) 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982534719}) 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=522909710}) 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982534719}) 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.120+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1219440996}) 2024-06-11T19:24:39.404 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456958857}) 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265565687}) 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456958857}) 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=211235553}) 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456958857}) 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1371396414}) 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456958857}) 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1382005074}) 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456958857}) 2024-06-11T19:24:39.405 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4033097694}) 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456958857}) 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1382154044}) 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.121+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456958857}) 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.122+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3916866767}) 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.122+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.122+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456958857}) 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.122+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2933210784}) 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748238281}) 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2223128575}) 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748238281}) 2024-06-11T19:24:39.406 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2010902510}) 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748238281}) 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3985288993}) 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748238281}) 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=842320416}) 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748238281}) 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=876180869}) 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748238281}) 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=938559905}) 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.407 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748238281}) 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1574714524}) 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748238281}) 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.123+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3322477923}) 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118184177}) 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1582072362}) 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118184177}) 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816397855}) 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118184177}) 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1071125028}) 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.408 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118184177}) 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=30621363}) 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118184177}) 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2128561853}) 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118184177}) 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4160562799}) 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.124+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118184177}) 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.125+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=362249919}) 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.125+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.125+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118184177}) 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.125+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1998860470}) 2024-06-11T19:24:39.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681607108}) 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578464156}) 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681607108}) 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=28505491}) 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681607108}) 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=892281071}) 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681607108}) 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1344741317}) 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681607108}) 2024-06-11T19:24:39.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3467920509}) 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681607108}) 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1870746629}) 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681607108}) 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=451476263}) 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681607108}) 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.126+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1191853987}) 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965080771}) 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1766993614}) 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965080771}) 2024-06-11T19:24:39.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3900182350}) 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965080771}) 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3178172396}) 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965080771}) 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2817507794}) 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965080771}) 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3809667957}) 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965080771}) 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3620214048}) 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965080771}) 2024-06-11T19:24:39.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1460547301}) 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965080771}) 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.127+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2039514359}) 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550439489}) 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2035978648}) 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550439489}) 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1396481853}) 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550439489}) 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3745050883}) 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.413 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550439489}) 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2902879621}) 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550439489}) 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1595782328}) 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550439489}) 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=382850391}) 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550439489}) 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=398413027}) 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550439489}) 2024-06-11T19:24:39.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.129+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2448478173}) 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582932971}) 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631410275}) 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582932971}) 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1442041651}) 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582932971}) 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2885250356}) 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582932971}) 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1960274739}) 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582932971}) 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2757304581}) 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582932971}) 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4154615506}) 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582932971}) 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1363087296}) 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582932971}) 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.130+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4118360322}) 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.131+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.131+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758163075}) 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.131+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=596700824}) 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.131+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.131+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758163075}) 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.131+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=768489543}) 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.131+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.131+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758163075}) 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.131+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3581231631}) 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.131+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.131+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758163075}) 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2885799677}) 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758163075}) 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=247814318}) 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758163075}) 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1087510833}) 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758163075}) 2024-06-11T19:24:39.418 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2053686410}) 2024-06-11T19:24:39.418 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.418 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758163075}) 2024-06-11T19:24:39.418 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.132+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1135162041}) 2024-06-11T19:24:39.418 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.418 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134748999}) 2024-06-11T19:24:39.418 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842302204}) 2024-06-11T19:24:39.418 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.418 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134748999}) 2024-06-11T19:24:39.418 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2850423949}) 2024-06-11T19:24:39.418 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134748999}) 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=897571481}) 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134748999}) 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1480646064}) 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134748999}) 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=327658400}) 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134748999}) 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1573860631}) 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134748999}) 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=490795099}) 2024-06-11T19:24:39.419 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134748999}) 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.133+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1520684810}) 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.134+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.134+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1480373051}) 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.134+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1391936136}) 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.134+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.134+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1480373051}) 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=58493445}) 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1480373051}) 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=844561872}) 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1480373051}) 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1613698463}) 2024-06-11T19:24:39.420 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1480373051}) 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3680851178}) 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1480373051}) 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=55534299}) 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1480373051}) 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3201849406}) 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1480373051}) 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.135+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3393466680}) 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439533094}) 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714903078}) 2024-06-11T19:24:39.421 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439533094}) 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3046455477}) 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439533094}) 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2157997238}) 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439533094}) 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1935670033}) 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439533094}) 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1885722692}) 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439533094}) 2024-06-11T19:24:39.422 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=922273059}) 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439533094}) 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=898596637}) 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439533094}) 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.136+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1627776359}) 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.137+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.137+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578213077}) 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.137+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1128007838}) 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.137+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.137+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578213077}) 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.137+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4233528025}) 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.137+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.137+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578213077}) 2024-06-11T19:24:39.423 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.137+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3661847887}) 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.137+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.137+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578213077}) 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4283037333}) 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578213077}) 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=234319607}) 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578213077}) 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1774122203}) 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578213077}) 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2738314816}) 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578213077}) 2024-06-11T19:24:39.424 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.138+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2162314604}) 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620091353}) 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129479248}) 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620091353}) 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2816626008}) 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620091353}) 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=992940208}) 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620091353}) 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3119568597}) 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620091353}) 2024-06-11T19:24:39.425 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2508981292}) 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620091353}) 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3185297199}) 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620091353}) 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2513564960}) 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620091353}) 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.139+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1007296312}) 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270103597}) 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3918118358}) 2024-06-11T19:24:39.426 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270103597}) 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=875686097}) 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270103597}) 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3219802491}) 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270103597}) 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=877418406}) 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270103597}) 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=109707530}) 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270103597}) 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2593152859}) 2024-06-11T19:24:39.427 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270103597}) 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1504536747}) 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270103597}) 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.141+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=278753211}) 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1803083876}) 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=403813781}) 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1803083876}) 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2757943655}) 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1803083876}) 2024-06-11T19:24:39.428 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3304485704}) 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1803083876}) 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1088096537}) 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1803083876}) 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2840214086}) 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1803083876}) 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2401512183}) 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1803083876}) 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4176560173}) 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1803083876}) 2024-06-11T19:24:39.429 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.142+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2324762065}) 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3593721314}) 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388778174}) 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3593721314}) 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=5371818}) 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3593721314}) 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=263462065}) 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3593721314}) 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1549238762}) 2024-06-11T19:24:39.430 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3593721314}) 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.143+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4228052674}) 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.144+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.144+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3593721314}) 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.144+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1720666245}) 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.144+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.144+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3593721314}) 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.144+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3750364961}) 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.144+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.144+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3593721314}) 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.144+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3469036371}) 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2213645781}) 2024-06-11T19:24:39.431 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084790755}) 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2213645781}) 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3192298918}) 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2213645781}) 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3910916704}) 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2213645781}) 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=128356474}) 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2213645781}) 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2739728819}) 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.432 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2213645781}) 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=893685055}) 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2213645781}) 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1689344175}) 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2213645781}) 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.145+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2461772403}) 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512415433}) 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016817621}) 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512415433}) 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1586190110}) 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.433 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512415433}) 2024-06-11T19:24:39.434 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2538249857}) 2024-06-11T19:24:39.434 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.434 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512415433}) 2024-06-11T19:24:39.434 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=728454384}) 2024-06-11T19:24:39.434 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.434 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512415433}) 2024-06-11T19:24:39.434 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2075572112}) 2024-06-11T19:24:39.434 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.434 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512415433}) 2024-06-11T19:24:39.434 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2140601228}) 2024-06-11T19:24:39.434 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.434 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512415433}) 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2321078886}) 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512415433}) 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.146+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3944733634}) 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.147+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.147+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990664057}) 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.147+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1635211994}) 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990664057}) 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105367587}) 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990664057}) 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1831205523}) 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990664057}) 2024-06-11T19:24:39.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1774565334}) 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990664057}) 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=641711861}) 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990664057}) 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=690072584}) 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990664057}) 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1807982533}) 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990664057}) 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.148+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3894765720}) 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3960726888}) 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129334934}) 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3960726888}) 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=280636381}) 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3960726888}) 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3495332580}) 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3960726888}) 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1947839387}) 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3960726888}) 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1856337216}) 2024-06-11T19:24:39.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3960726888}) 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2118465547}) 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3960726888}) 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2297507433}) 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3960726888}) 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.149+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=329395062}) 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897540509}) 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704889490}) 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897540509}) 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=795820775}) 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897540509}) 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3422432338}) 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897540509}) 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3971226562}) 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897540509}) 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1953871085}) 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897540509}) 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=453461474}) 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897540509}) 2024-06-11T19:24:39.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3251981885}) 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897540509}) 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.151+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3299354127}) 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805520755}) 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266492994}) 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805520755}) 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1983375362}) 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805520755}) 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1203922435}) 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805520755}) 2024-06-11T19:24:39.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3105316615}) 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805520755}) 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1481468679}) 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805520755}) 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2148576499}) 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805520755}) 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3190368893}) 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805520755}) 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.152+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=710558097}) 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.153+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.153+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191591475}) 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.153+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3143517938}) 2024-06-11T19:24:39.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.153+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.153+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191591475}) 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=504249173}) 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191591475}) 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1767142274}) 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191591475}) 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3944301707}) 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191591475}) 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3172111217}) 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191591475}) 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3933679862}) 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191591475}) 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1990060591}) 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191591475}) 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.154+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=870534918}) 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681907965}) 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498904372}) 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681907965}) 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2503754088}) 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681907965}) 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=69668950}) 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681907965}) 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1680018523}) 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681907965}) 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3353307961}) 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681907965}) 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2705923779}) 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681907965}) 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1787926755}) 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681907965}) 2024-06-11T19:24:39.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.155+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2474402303}) 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051534467}) 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693447353}) 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051534467}) 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2219428150}) 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051534467}) 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3451034588}) 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051534467}) 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2393415875}) 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.156+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.157+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051534467}) 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.157+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2296188403}) 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.157+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.157+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051534467}) 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.157+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=253575152}) 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.157+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.157+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051534467}) 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.157+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3090315712}) 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.157+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.157+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051534467}) 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.157+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=197048853}) 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820581328}) 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4040952892}) 2024-06-11T19:24:39.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820581328}) 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=940685139}) 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820581328}) 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3649293130}) 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820581328}) 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=900802090}) 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820581328}) 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4171378112}) 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820581328}) 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=908550439}) 2024-06-11T19:24:39.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820581328}) 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=434015810}) 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820581328}) 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.158+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3927903891}) 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288921133}) 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2137474602}) 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288921133}) 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2171401752}) 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288921133}) 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=738201706}) 2024-06-11T19:24:39.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288921133}) 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2834754936}) 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288921133}) 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1945775799}) 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.159+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.160+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288921133}) 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.160+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=570095336}) 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.160+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.160+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288921133}) 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.160+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3134726091}) 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.160+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.160+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288921133}) 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.160+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1769049166}) 2024-06-11T19:24:39.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817089662}) 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564645165}) 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817089662}) 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=244523235}) 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817089662}) 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1452874972}) 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817089662}) 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1332947116}) 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817089662}) 2024-06-11T19:24:39.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2034864810}) 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817089662}) 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3254436641}) 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817089662}) 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2759182006}) 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817089662}) 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.161+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3016207150}) 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926889205}) 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249923792}) 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926889205}) 2024-06-11T19:24:39.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1478657466}) 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926889205}) 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=836269844}) 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926889205}) 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1910931238}) 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926889205}) 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3636489988}) 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.162+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.163+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926889205}) 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.163+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3653748915}) 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.163+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.163+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926889205}) 2024-06-11T19:24:39.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.163+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=551287227}) 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.163+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.163+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926889205}) 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.163+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3295540024}) 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790128824}) 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1640884505}) 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790128824}) 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2071738423}) 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790128824}) 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3413730461}) 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790128824}) 2024-06-11T19:24:39.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=344129270}) 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790128824}) 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2297743851}) 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790128824}) 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1772809654}) 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790128824}) 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3812189963}) 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790128824}) 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.164+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3543370529}) 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440538223}) 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2089664059}) 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440538223}) 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1749982853}) 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440538223}) 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=930227473}) 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440538223}) 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2056002903}) 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.165+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.166+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440538223}) 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.166+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2740441006}) 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.166+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.166+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440538223}) 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.166+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=218340199}) 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.166+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.166+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440538223}) 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.166+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3571741879}) 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.166+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.166+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440538223}) 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.166+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2987385103}) 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=274678836}) 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=184205484}) 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=274678836}) 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3855930420}) 2024-06-11T19:24:39.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=274678836}) 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2164075193}) 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=274678836}) 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4191170967}) 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=274678836}) 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3731860201}) 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=274678836}) 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=64912695}) 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=274678836}) 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=363856792}) 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=274678836}) 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.167+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3302303173}) 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.168+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.168+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=752981599}) 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.168+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=173433672}) 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=752981599}) 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1194890916}) 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=752981599}) 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1261959948}) 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=752981599}) 2024-06-11T19:24:39.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1218628239}) 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=752981599}) 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3927523489}) 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=752981599}) 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=610506442}) 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=752981599}) 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3070168985}) 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=752981599}) 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.169+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=441909504}) 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4031022826}) 2024-06-11T19:24:39.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111196516}) 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4031022826}) 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3975550926}) 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4031022826}) 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3400024720}) 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4031022826}) 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2801474030}) 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4031022826}) 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4017240674}) 2024-06-11T19:24:39.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4031022826}) 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3684798315}) 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4031022826}) 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3916530313}) 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4031022826}) 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.170+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1836212337}) 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352078223}) 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3505424308}) 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352078223}) 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3467130941}) 2024-06-11T19:24:39.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352078223}) 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3353069456}) 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352078223}) 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2526930633}) 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352078223}) 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3486494546}) 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352078223}) 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1421286776}) 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352078223}) 2024-06-11T19:24:39.462 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2359447786}) 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352078223}) 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.172+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=946533794}) 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2825250190}) 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1520144670}) 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2825250190}) 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2716292371}) 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2825250190}) 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2371900394}) 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2825250190}) 2024-06-11T19:24:39.463 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2464575697}) 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2825250190}) 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2361434426}) 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2825250190}) 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2138732723}) 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2825250190}) 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=585310671}) 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2825250190}) 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.173+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1633131833}) 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.174+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.174+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942358812}) 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.174+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893334310}) 2024-06-11T19:24:39.464 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.174+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.174+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942358812}) 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.174+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=14943623}) 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.174+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.174+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942358812}) 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.174+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2476513679}) 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.174+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.174+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942358812}) 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.174+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=106729747}) 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.175+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.175+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942358812}) 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.175+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3298922053}) 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.175+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.175+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942358812}) 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.175+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=834855362}) 2024-06-11T19:24:39.465 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.175+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.175+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942358812}) 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.175+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1839924820}) 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.175+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.175+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942358812}) 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.175+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=431154866}) 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258077653}) 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629794399}) 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258077653}) 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256397451}) 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.466 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258077653}) 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=961886181}) 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258077653}) 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2308971248}) 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258077653}) 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=807606470}) 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258077653}) 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1082003112}) 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258077653}) 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=770032744}) 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.467 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258077653}) 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.176+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2571895527}) 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407681962}) 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547758624}) 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407681962}) 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=731621065}) 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407681962}) 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1972419210}) 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407681962}) 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2132604299}) 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407681962}) 2024-06-11T19:24:39.468 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2013138356}) 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.177+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407681962}) 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.178+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3034061180}) 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.178+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.178+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407681962}) 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.178+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2245470821}) 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.178+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.178+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407681962}) 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.178+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3267724774}) 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12649127}) 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193494782}) 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12649127}) 2024-06-11T19:24:39.469 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=191322267}) 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12649127}) 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2055322950}) 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12649127}) 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4122389064}) 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12649127}) 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2862788877}) 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12649127}) 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2776529598}) 2024-06-11T19:24:39.470 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12649127}) 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1592048493}) 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12649127}) 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.179+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1295839050}) 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829878022}) 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924217225}) 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829878022}) 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3979289100}) 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829878022}) 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1718189781}) 2024-06-11T19:24:39.471 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829878022}) 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3899290282}) 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829878022}) 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2309871258}) 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829878022}) 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1535416047}) 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829878022}) 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1229023413}) 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829878022}) 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.181+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4268785501}) 2024-06-11T19:24:39.472 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.182+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.182+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1151719147}) 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.182+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=590905777}) 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.182+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.182+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1151719147}) 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.182+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2782327851}) 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.182+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.182+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1151719147}) 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.182+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=413476233}) 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.182+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.182+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1151719147}) 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2050355370}) 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1151719147}) 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1646230576}) 2024-06-11T19:24:39.473 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1151719147}) 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3864085311}) 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1151719147}) 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1021774855}) 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1151719147}) 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.183+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1792342873}) 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466753842}) 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1775668687}) 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466753842}) 2024-06-11T19:24:39.474 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2371611983}) 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466753842}) 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=274913339}) 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466753842}) 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2166732979}) 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466753842}) 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2671854278}) 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466753842}) 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2897160137}) 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466753842}) 2024-06-11T19:24:39.475 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4009362069}) 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466753842}) 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.184+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=261495763}) 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.185+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.185+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1382581266}) 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.185+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164591481}) 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.185+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.185+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1382581266}) 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.185+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1119362634}) 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.185+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.185+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1382581266}) 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.185+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1666485721}) 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.185+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.476 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.185+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1382581266}) 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=271104449}) 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1382581266}) 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2404323170}) 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1382581266}) 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=440050915}) 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1382581266}) 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1053157689}) 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1382581266}) 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.186+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=380482996}) 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369233835}) 2024-06-11T19:24:39.477 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2099294299}) 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369233835}) 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787410865}) 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369233835}) 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3690737234}) 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369233835}) 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4057418120}) 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369233835}) 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3529694750}) 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.478 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369233835}) 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1664904579}) 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369233835}) 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1851660963}) 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369233835}) 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.187+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2046691568}) 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362535503}) 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1457754814}) 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362535503}) 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2727763521}) 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.479 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362535503}) 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=79507554}) 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362535503}) 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=179542585}) 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362535503}) 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3120582257}) 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362535503}) 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2026694625}) 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362535503}) 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3276006106}) 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.188+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.189+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362535503}) 2024-06-11T19:24:39.480 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.189+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1840844235}) 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.189+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.189+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310550600}) 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548447743}) 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310550600}) 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2951033530}) 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310550600}) 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3736723837}) 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310550600}) 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=278410636}) 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310550600}) 2024-06-11T19:24:39.481 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1440175808}) 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310550600}) 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2093307114}) 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310550600}) 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3167517473}) 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310550600}) 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.190+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2204294601}) 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734792446}) 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365592739}) 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.482 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734792446}) 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2617326448}) 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734792446}) 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3978113683}) 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734792446}) 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=847667647}) 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734792446}) 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=221323015}) 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734792446}) 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4246504423}) 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.483 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734792446}) 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=403797328}) 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734792446}) 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.191+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4254109821}) 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.192+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.192+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010595816}) 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.192+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2550218737}) 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.192+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.192+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010595816}) 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.192+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3318701624}) 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.192+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.192+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010595816}) 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.192+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2781043911}) 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.192+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.192+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010595816}) 2024-06-11T19:24:39.484 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.192+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2767857195}) 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.193+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.193+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010595816}) 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.193+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2900895475}) 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.193+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.193+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010595816}) 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.193+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1972789400}) 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.193+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.193+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010595816}) 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.193+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3003742540}) 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.193+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.193+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010595816}) 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.193+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4153865173}) 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195115568}) 2024-06-11T19:24:39.485 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4060847433}) 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195115568}) 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=396032459}) 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195115568}) 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1164719819}) 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195115568}) 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1278413994}) 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195115568}) 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3275341702}) 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195115568}) 2024-06-11T19:24:39.486 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3566698662}) 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195115568}) 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1142624286}) 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195115568}) 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.194+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=95105184}) 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.195+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.195+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612799808}) 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.195+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236190298}) 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.195+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.195+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612799808}) 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.195+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3872793104}) 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.195+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.195+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612799808}) 2024-06-11T19:24:39.487 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.195+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3928291729}) 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.195+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.195+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612799808}) 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2944812207}) 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612799808}) 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=162864077}) 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612799808}) 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=598620592}) 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612799808}) 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1055947062}) 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612799808}) 2024-06-11T19:24:39.488 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.196+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=266633715}) 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1048678221}) 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368232592}) 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1048678221}) 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1879158594}) 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1048678221}) 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3702947885}) 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1048678221}) 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1899322717}) 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1048678221}) 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1924402263}) 2024-06-11T19:24:39.489 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1048678221}) 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3195169078}) 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1048678221}) 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3233716989}) 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1048678221}) 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.197+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1745164330}) 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79326034}) 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121067738}) 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79326034}) 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3163115331}) 2024-06-11T19:24:39.490 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79326034}) 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2467874519}) 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79326034}) 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3152801166}) 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79326034}) 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=219106697}) 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79326034}) 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2707487874}) 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79326034}) 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2369620009}) 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.198+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.491 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.199+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79326034}) 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.199+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4267077425}) 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2066261257}) 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2527463161}) 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2066261257}) 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1442282627}) 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2066261257}) 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=224115077}) 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2066261257}) 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=70020910}) 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.492 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2066261257}) 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=139112036}) 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2066261257}) 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=526979122}) 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2066261257}) 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2796840800}) 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2066261257}) 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.200+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4122460934}) 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043527994}) 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62598616}) 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.493 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043527994}) 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3169372316}) 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043527994}) 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4009240412}) 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043527994}) 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=696531606}) 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043527994}) 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=366228976}) 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043527994}) 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=878916926}) 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.494 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043527994}) 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4040897310}) 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043527994}) 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.201+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3545987156}) 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.202+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.202+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731975134}) 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.202+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613400894}) 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.202+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.202+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731975134}) 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.202+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3630357672}) 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.202+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731975134}) 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2104267931}) 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731975134}) 2024-06-11T19:24:39.495 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=344153057}) 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731975134}) 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2600642898}) 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731975134}) 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1708867586}) 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731975134}) 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3406874538}) 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731975134}) 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.203+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2684501933}) 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.496 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019888644}) 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391279266}) 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019888644}) 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1094858059}) 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019888644}) 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4289746821}) 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019888644}) 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1984990170}) 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019888644}) 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3003123077}) 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.497 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019888644}) 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3519396492}) 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019888644}) 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4192289386}) 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019888644}) 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.204+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1222106658}) 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426506680}) 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=543613165}) 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426506680}) 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3404970053}) 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426506680}) 2024-06-11T19:24:39.498 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4246719648}) 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426506680}) 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2148019387}) 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426506680}) 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3473954359}) 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426506680}) 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3099866590}) 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426506680}) 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=798523822}) 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426506680}) 2024-06-11T19:24:39.499 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.205+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1057107280}) 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263580139}) 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4038541933}) 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263580139}) 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40771794}) 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263580139}) 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1074043927}) 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263580139}) 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=239300522}) 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263580139}) 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1322043803}) 2024-06-11T19:24:39.500 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263580139}) 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1234230882}) 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263580139}) 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3416977368}) 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263580139}) 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.207+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=238599273}) 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180569131}) 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125472119}) 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180569131}) 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3973478263}) 2024-06-11T19:24:39.501 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180569131}) 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3605730079}) 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180569131}) 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1441331418}) 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180569131}) 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3502336935}) 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180569131}) 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3784735919}) 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180569131}) 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3427596824}) 2024-06-11T19:24:39.502 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180569131}) 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.208+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1098957246}) 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152482051}) 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4101956423}) 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152482051}) 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=504414716}) 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152482051}) 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=486392660}) 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152482051}) 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1095968183}) 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.503 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152482051}) 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=598593598}) 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152482051}) 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=126757665}) 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152482051}) 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1287369200}) 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152482051}) 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.210+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=517688018}) 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722565413}) 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3959437050}) 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.504 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722565413}) 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3116051644}) 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722565413}) 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3524320502}) 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722565413}) 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2062966999}) 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722565413}) 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4187536396}) 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722565413}) 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=73191871}) 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.505 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722565413}) 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1463813357}) 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722565413}) 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.211+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1481156029}) 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.212+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.212+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1931848340}) 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.212+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983951901}) 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.212+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.212+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1931848340}) 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.212+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4084539087}) 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.212+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1931848340}) 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3842922820}) 2024-06-11T19:24:39.506 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1931848340}) 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4140631354}) 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1931848340}) 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3822733013}) 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1931848340}) 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1639948975}) 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1931848340}) 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=170596246}) 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1931848340}) 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.213+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1162708160}) 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.507 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715231107}) 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107860471}) 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715231107}) 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826240760}) 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715231107}) 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3025115285}) 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715231107}) 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1693769035}) 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715231107}) 2024-06-11T19:24:39.508 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1845860024}) 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715231107}) 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1096991265}) 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715231107}) 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1410506842}) 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715231107}) 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.214+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=262320921}) 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.215+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.215+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238509392}) 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.215+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2546567912}) 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.215+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.509 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.215+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238509392}) 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.215+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1936112115}) 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.215+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.215+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238509392}) 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.215+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4293596420}) 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.215+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238509392}) 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=363739503}) 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238509392}) 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3658021797}) 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238509392}) 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2107279834}) 2024-06-11T19:24:39.510 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238509392}) 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2323014441}) 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238509392}) 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.216+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2931732832}) 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843976255}) 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3731552533}) 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843976255}) 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=817330436}) 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843976255}) 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=512060045}) 2024-06-11T19:24:39.511 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843976255}) 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1571326615}) 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843976255}) 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3159661406}) 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843976255}) 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3776867250}) 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843976255}) 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2476511078}) 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843976255}) 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.217+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=278084147}) 2024-06-11T19:24:39.512 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871146033}) 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372097634}) 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871146033}) 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1097175427}) 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871146033}) 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2024035545}) 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871146033}) 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3868942649}) 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.218+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.219+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871146033}) 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.219+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1338127009}) 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.219+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.513 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.219+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871146033}) 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.219+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1611941307}) 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.219+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.219+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871146033}) 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.219+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2721427637}) 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.219+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.219+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871146033}) 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.219+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1054192957}) 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279985957}) 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2231401811}) 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279985957}) 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2504292569}) 2024-06-11T19:24:39.514 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279985957}) 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3183023003}) 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279985957}) 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2057827234}) 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279985957}) 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1559593648}) 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279985957}) 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1139620770}) 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279985957}) 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3037220193}) 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.515 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279985957}) 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.220+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1345372021}) 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.221+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.221+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=622851563}) 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.221+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850090935}) 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.221+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.221+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=622851563}) 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.221+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3162307728}) 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=622851563}) 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=983075160}) 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=622851563}) 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3660484431}) 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.516 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=622851563}) 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2112079902}) 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=622851563}) 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4106760434}) 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=622851563}) 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=285767829}) 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=622851563}) 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.222+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=750254568}) 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420812843}) 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160881483}) 2024-06-11T19:24:39.517 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420812843}) 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2267968066}) 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420812843}) 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2870392916}) 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420812843}) 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1996605546}) 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420812843}) 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3501378454}) 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420812843}) 2024-06-11T19:24:39.518 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3435245054}) 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420812843}) 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=456779324}) 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420812843}) 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.223+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=915280415}) 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.224+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.224+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2669137473}) 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.224+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705857572}) 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.224+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.224+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2669137473}) 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.224+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1739264082}) 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.224+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.519 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2669137473}) 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1868140705}) 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2669137473}) 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=269000713}) 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2669137473}) 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=205980750}) 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2669137473}) 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1528623437}) 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2669137473}) 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2429814542}) 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.520 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2669137473}) 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.225+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2078245779}) 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.226+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.226+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=1241511303,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3281343581,mon_config_key=2281582573,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1384163437,osd_snap=3612350482}) 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.226+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=31,mdsmap=2,mgr=8,mgr_command_descs=1,mgrstat=8,mon_config_key=10,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=32} crc {logm=3671084354,mdsmap=1401145748,mgr=2102212137,mgr_command_descs=1608769102,mgrstat=3755055681,mon_config_key=574861995,monmap=1530261226,osd_metadata=1390437783,osd_pg_creating=1384163437,osd_snap=4198093272}) 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.226+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=1241511303,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3281343581,mon_config_key=2281582573,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1384163437,osd_snap=3612350482}) 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=29} crc {logm=329500415,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=820786764,mon_config_key=2795952728,monmap=506305176,osd_metadata=1108245517,osd_pg_creating=1384163437,osd_snap=3334144667}) 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=1241511303,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3281343581,mon_config_key=2281582573,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1384163437,osd_snap=3612350482}) 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=25} crc {logm=1241511303,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=820786764,mon_config_key=822367976,monmap=506305176,osd_metadata=1390437783,osd_pg_creating=1384163437,osd_snap=594928598}) 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=1241511303,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3281343581,mon_config_key=2281582573,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1384163437,osd_snap=3612350482}) 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25} crc {logm=676589412,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=820786764,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1384163437,osd_snap=2265701342}) 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=1241511303,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3281343581,mon_config_key=2281582573,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1384163437,osd_snap=3612350482}) 2024-06-11T19:24:39.521 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=26} crc {logm=1241511303,mdsmap=2840952586,mgr=570294379,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=820786764,mon_config_key=798616459,monmap=506305176,osd_metadata=1431320961,osd_pg_creating=1384163438,osd_snap=1225480829}) 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=1241511303,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3281343581,mon_config_key=2281582573,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1384163437,osd_snap=3612350482}) 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=31,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=2105412118,mdsmap=2840952586,mgr=640990267,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2258901651,mon_config_key=455749491,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1384163437,osd_snap=3897989123}) 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=1241511303,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3281343581,mon_config_key=2281582573,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1384163437,osd_snap=3612350482}) 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27} crc {logm=627033763,mdsmap=2840952586,mgr=120143446,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3755055681,mon_config_key=2488442063,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1384163437,osd_snap=4238401951}) 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=1241511303,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3281343581,mon_config_key=2281582573,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1384163437,osd_snap=3612350482}) 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.227+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=34,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=26} crc {logm=3450783417,mdsmap=2840952586,mgr=3166249245,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=820786764,mon_config_key=608068930,monmap=506305176,osd_metadata=352037677,osd_pg_creating=1384163437,osd_snap=310499444}) 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.228+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.228+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=4233033943,osdmap=3734656696}) 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1535626336,osdmap=998205384}) 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.522 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=4233033943,osdmap=3734656696}) 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=4233033943,osdmap=225349643}) 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=4233033943,osdmap=3734656696}) 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=4233033943,osdmap=2460753953}) 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=4233033943,osdmap=3734656696}) 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1535626336,osdmap=3098636511}) 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=4233033943,osdmap=3734656696}) 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=1271290509,osdmap=104391159}) 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=4233033943,osdmap=3734656696}) 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=4233033943,osdmap=3368697476}) 2024-06-11T19:24:39.523 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=4233033943,osdmap=3734656696}) 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1231637184,osdmap=396985994}) 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=4233033943,osdmap=3734656696}) 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.229+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=4233033943,osdmap=1801232169}) 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1982673495}) 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=874314358}) 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1982673495}) 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1162627992}) 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1982673495}) 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3675497256}) 2024-06-11T19:24:39.524 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1982673495}) 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1486938074}) 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1982673495}) 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=881114218}) 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1982673495}) 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1584323386}) 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1982673495}) 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3861720614}) 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.230+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.231+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1982673495}) 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.231+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1308294070}) 2024-06-11T19:24:39.525 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1880108162}) 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=601184656}) 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1880108162}) 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3557664229}) 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1880108162}) 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3143562301}) 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1880108162}) 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2400850489}) 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1880108162}) 2024-06-11T19:24:39.526 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2361429324}) 2024-06-11T19:24:39.527 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.527 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1880108162}) 2024-06-11T19:24:39.527 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=232393567}) 2024-06-11T19:24:39.527 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.527 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1880108162}) 2024-06-11T19:24:39.527 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2837348284}) 2024-06-11T19:24:39.527 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.527 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1880108162}) 2024-06-11T19:24:39.527 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.232+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=728984344}) 2024-06-11T19:24:39.527 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.527 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1741260479}) 2024-06-11T19:24:39.529 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=172111587}) 2024-06-11T19:24:39.529 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.529 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1741260479}) 2024-06-11T19:24:39.529 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=184455791}) 2024-06-11T19:24:39.529 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.529 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1741260479}) 2024-06-11T19:24:39.529 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=121119702}) 2024-06-11T19:24:39.529 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.529 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1741260479}) 2024-06-11T19:24:39.529 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3073993408}) 2024-06-11T19:24:39.529 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.529 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1741260479}) 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3755390365}) 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1741260479}) 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1628513316}) 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1741260479}) 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=956091141}) 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1741260479}) 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.234+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4059170837}) 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.235+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.235+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4091089239}) 2024-06-11T19:24:39.530 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.235+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=784770827}) 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.235+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.235+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4091089239}) 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.235+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=157967762}) 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.235+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.235+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4091089239}) 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.235+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1552939161}) 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.235+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.235+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4091089239}) 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.235+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1110493982}) 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.236+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.236+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4091089239}) 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.236+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=588360719}) 2024-06-11T19:24:39.531 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.236+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.236+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4091089239}) 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.236+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2933226984}) 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.236+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.236+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4091089239}) 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.236+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=4266819521}) 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.236+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.236+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4091089239}) 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.236+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1687689744}) 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3427846541}) 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3260003760}) 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3427846541}) 2024-06-11T19:24:39.532 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3366819394}) 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3427846541}) 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=848494361}) 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3427846541}) 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1655998938}) 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3427846541}) 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=535537674}) 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3427846541}) 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3824817594}) 2024-06-11T19:24:39.533 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3427846541}) 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=769286221}) 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3427846541}) 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.237+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1794261355}) 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3357452715}) 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3537529702}) 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3357452715}) 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2893164199}) 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3357452715}) 2024-06-11T19:24:39.534 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3497618371}) 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3357452715}) 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=209771787}) 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3357452715}) 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1541401218}) 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3357452715}) 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3744932689}) 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3357452715}) 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2780691201}) 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3357452715}) 2024-06-11T19:24:39.535 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.240+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2514511655}) 2024-06-11T19:24:39.536 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.536 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2362539549}) 2024-06-11T19:24:39.536 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=845275740}) 2024-06-11T19:24:39.536 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.536 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2362539549}) 2024-06-11T19:24:39.536 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2295023353}) 2024-06-11T19:24:39.536 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.536 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2362539549}) 2024-06-11T19:24:39.536 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1255434265}) 2024-06-11T19:24:39.536 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.536 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2362539549}) 2024-06-11T19:24:39.536 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=616937842}) 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2362539549}) 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=707380156}) 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2362539549}) 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1725589150}) 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2362539549}) 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1025535670}) 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2362539549}) 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.243+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3361692003}) 2024-06-11T19:24:39.537 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.245+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.245+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2226624608}) 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.245+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1655304496}) 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.245+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.245+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2226624608}) 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.245+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=38711979}) 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.245+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.245+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2226624608}) 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.245+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=350718618}) 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.245+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2226624608}) 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1696863188}) 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.538 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2226624608}) 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3218253119}) 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2226624608}) 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2563828098}) 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2226624608}) 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3992100414}) 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2226624608}) 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.246+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4248465431}) 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=1328203018}) 2024-06-11T19:24:39.539 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=336591515}) 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=1328203018}) 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=20} crc {osdmap=162910276}) 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=1328203018}) 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=20} crc {osdmap=162910276}) 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=1328203018}) 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=18} crc {osdmap=1430963905}) 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=1328203018}) 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=21} crc {osdmap=3109477063}) 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=1328203018}) 2024-06-11T19:24:39.540 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.254+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=20} crc {osdmap=2635284912}) 2024-06-11T19:24:39.541 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.255+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.541 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.255+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=1328203018}) 2024-06-11T19:24:39.541 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.255+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=21} crc {osdmap=2769411282}) 2024-06-11T19:24:39.541 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.255+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:24:39.541 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.255+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=1328203018}) 2024-06-11T19:24:39.541 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:24:39.255+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=21} crc {osdmap=2769411282}) 2024-06-11T19:24:40.077 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:24:40.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:24:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.177+0000 7f20f7f66640 10 monclient: get_monmap_and_config 2024-06-11T19:24:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.177+0000 7f20f7f66640 10 monclient: build_initial_monmap 2024-06-11T19:24:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.177+0000 7f20f7f66640 10 monclient: monmap: 2024-06-11T19:24:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:40.178099+0000 2024-06-11T19:24:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:40.178099+0000 2024-06-11T19:24:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:40.177 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:40.177 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:40.177 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:40.177 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:40.177 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:40.177 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:40.177 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:40.177 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:40.177 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.177+0000 7f20f7f66640 1 Processor -- start 2024-06-11T19:24:40.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.177+0000 7f20f7f66640 1 -- start start 2024-06-11T19:24:40.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.177+0000 7f20f7f66640 10 monclient: init 2024-06-11T19:24:40.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:40.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 10 monclient: _add_conns ranks=[3,4,10,2,1,9,8,7,0,5,6] 2024-06-11T19:24:40.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20f00a6260 0x7f20f00a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 10 monclient(hunting): picked mon.noname-g con 0x7f20f00a6260 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f20f006c890 0x7f20f00a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 10 monclient(hunting): picked mon.noname-c con 0x7f20f006c890 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f20f00a2d00 0x7f20f00a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 10 monclient(hunting): picked mon.noname-f con 0x7f20f00a2d00 addr v2:172.21.15.117:3300/0 2024-06-11T19:24:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f20f00a05b0 con 0x7f20f00a2d00 2024-06-11T19:24:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f20f00a6640 con 0x7f20f00a6260 2024-06-11T19:24:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f20f006c650 con 0x7f20f006c890 2024-06-11T19:24:40.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f7f66640 10 monclient(hunting): authenticate will time out at 1689.985385s 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f20f006c890 0x7f20f00a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f20f006c890 0x7f20f00a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46854/0 (socket says 172.21.15.98:46854) 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 1 -- 172.21.15.98:0/663522923 learned_addr learned my addr 172.21.15.98:0/663522923 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 10 monclient(hunting): get_auth_request con 0x7f20f006c890 auth_method 0 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5d02640 1 --2- 172.21.15.98:0/663522923 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20f00a6260 0x7f20f00a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f6503640 1 --2- 172.21.15.98:0/663522923 >> v2:172.21.15.117:3300/0 conn(0x7f20f00a2d00 0x7f20f00a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f6503640 10 monclient(hunting): get_auth_request con 0x7f20f00a2d00 auth_method 0 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f6503640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f6503640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f6503640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5d02640 10 monclient(hunting): get_auth_request con 0x7f20f00a6260 auth_method 0 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5d02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5d02640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5d02640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 10 monclient(hunting): handle_auth_done global_id 114412 payload 274 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 1 -- 172.21.15.98:0/663522923 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20f00a6260 msgr2=0x7f20f00a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 1 --2- 172.21.15.98:0/663522923 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20f00a6260 0x7f20f00a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 1 -- 172.21.15.98:0/663522923 >> v2:172.21.15.117:3300/0 conn(0x7f20f00a2d00 msgr2=0x7f20f00a4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 1 --2- 172.21.15.98:0/663522923 >> v2:172.21.15.117:3300/0 conn(0x7f20f00a2d00 0x7f20f00a4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 10 monclient: _finish_hunting 0 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 1 monclient: found mon.noname-c 2024-06-11T19:24:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.178+0000 7f20f5501640 1 -- 172.21.15.98:0/663522923 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f0073b60 con 0x7f20f006c890 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f5501640 1 --2- 172.21.15.98:0/663522923 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f20f006c890 0x7f20f00a48b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f20e40098a0 tx=0x7f20e402b7b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=17f41673d40c33c1 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 1 -- 172.21.15.98:0/663522923 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f20e4038070 con 0x7f20f006c890 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 10 monclient: dump: 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 10 monclient: _finish_auth 0 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 10 monclient: _check_auth_tickets 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 1 -- 172.21.15.98:0/663522923 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e4031980 con 0x7f20f006c890 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f7f66640 5 monclient: authenticate success, global_id 114412 2024-06-11T19:24:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f7f66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f7f66640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 1 -- 172.21.15.98:0/663522923 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f20e4031bb0 con 0x7f20f006c890 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f7f66640 4 set_mon_vals no callback set 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.179+0000 7f20f4d00640 10 monclient: dump: 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 10 monclient: shutdown 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 1 -- 172.21.15.98:0/663522923 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f20f006c890 msgr2=0x7f20f00a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 1 --2- 172.21.15.98:0/663522923 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f20f006c890 0x7f20f00a48b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f20e40098a0 tx=0x7f20e402b7b0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 1 -- 172.21.15.98:0/663522923 shutdown_connections 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 1 --2- 172.21.15.98:0/663522923 >> v2:172.21.15.117:3300/0 conn(0x7f20f00a2d00 0x7f20f00a4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 1 --2- 172.21.15.98:0/663522923 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f20f006c890 0x7f20f00a48b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 1 --2- 172.21.15.98:0/663522923 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20f00a6260 0x7f20f00a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 1 -- 172.21.15.98:0/663522923 >> 172.21.15.98:0/663522923 conn(0x7f20f00a95b0 msgr2=0x7f20f00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 1 -- 172.21.15.98:0/663522923 shutdown_connections 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 1 -- 172.21.15.98:0/663522923 wait complete. 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 10 monclient: build_initial_monmap 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.180+0000 7f20f7f66640 10 monclient: monmap: 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 1 librados: starting msgr at 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 1 librados: starting objecter 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 1 Processor -- start 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 1 -- start start 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 1 librados: setting wanted keys 2024-06-11T19:24:40.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 1 librados: calling monclient init 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 10 monclient: init 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 10 monclient: _add_conns ranks=[2,5,6,0,8,7,4,1,3] 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f006c890 0x7f20f01ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 10 monclient(hunting): picked mon.noname-b con 0x7f20f006c890 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f20f00a2d00 0x7f20f01c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 10 monclient(hunting): picked mon.noname-h con 0x7f20f00a2d00 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f20f00a6260 0x7f20f01cd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 10 monclient(hunting): picked mon.noname-d con 0x7f20f00a6260 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f20f00a2410 con 0x7f20f006c890 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f20f00a2290 con 0x7f20f00a2d00 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f20f00a5330 con 0x7f20f00a6260 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f7f66640 10 monclient(hunting): authenticate will time out at 1689.988692s 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5d02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f006c890 0x7f20f01ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f6503640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f20f00a6260 0x7f20f01cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5d02640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f006c890 0x7f20f01ca020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57290/0 (socket says 172.21.15.98:57290) 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5d02640 1 -- 172.21.15.98:0/3113323136 learned_addr learned my addr 172.21.15.98:0/3113323136 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5d02640 10 monclient(hunting): get_auth_request con 0x7f20f006c890 auth_method 0 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5d02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5d02640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5d02640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f6503640 10 monclient(hunting): get_auth_request con 0x7f20f00a6260 auth_method 0 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f6503640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f6503640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f6503640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5d02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5d02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5501640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f20f00a2d00 0x7f20f01c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5d02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f6503640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f6503640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f6503640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5501640 10 monclient(hunting): get_auth_request con 0x7f20f00a2d00 auth_method 0 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5501640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5501640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5501640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5501640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5501640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.181+0000 7f20f5501640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f6503640 10 monclient(hunting): handle_auth_done global_id 124179 payload 824 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f6503640 1 -- 172.21.15.98:0/3113323136 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f20f00a2d00 msgr2=0x7f20f01c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f6503640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f20f00a2d00 0x7f20f01c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f6503640 1 -- 172.21.15.98:0/3113323136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f006c890 msgr2=0x7f20f01ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f6503640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f006c890 0x7f20f01ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f6503640 10 monclient: _finish_hunting 0 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f6503640 1 monclient: found mon.noname-d 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f6503640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f6503640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f6503640 1 -- 172.21.15.98:0/3113323136 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f01ac450 con 0x7f20f00a6260 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f5d02640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f006c890 0x7f20f01ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f6503640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f20f00a6260 0x7f20f01cd560 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f20e800d530 tx=0x7f20e800d9f0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 10 client.?.objecter ms_handle_connect 0x7f20f00a6260 2024-06-11T19:24:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 1 -- 172.21.15.98:0/3113323136 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f20e8012a60 con 0x7f20f00a6260 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 10 monclient: dump: 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 10 monclient: _finish_auth 0 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 1 -- 172.21.15.98:0/3113323136 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e8019070 con 0x7f20f00a6260 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f7f66640 5 monclient: authenticate success, global_id 124179 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f7f66640 10 monclient: _renew_subs 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f7f66640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f7f66640 1 -- 172.21.15.98:0/3113323136 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f01aef10 con 0x7f20f00a6260 2024-06-11T19:24:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 1 -- 172.21.15.98:0/3113323136 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f20e8012c40 con 0x7f20f00a6260 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20ee7fc640 10 monclient: dump: 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f7f66640 10 client.124179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20effff640 4 set_mon_vals no callback set 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f7f66640 10 monclient: _renew_subs 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f7f66640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f7f66640 1 -- 172.21.15.98:0/3113323136 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f00a6ca0 con 0x7f20f00a6260 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.182+0000 7f20f7f66640 1 librados: init done 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20ee7fc640 1 -- 172.21.15.98:0/3113323136 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f20e80125f0 con 0x7f20f00a6260 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20f7f66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20f7f66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20f7f66640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20f7f66640 1 -- 172.21.15.98:0/3113323136 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f00a88c0 con 0x7f20f00a6260 2024-06-11T19:24:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20ee7fc640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f20dc03fe10 0x7f20dc0422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20ee7fc640 1 -- 172.21.15.98:0/3113323136 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(624..624 src has 1..624) ==== 6797+0+0 (secure 0 0 0) 0x7f20e800f040 con 0x7f20f00a6260 2024-06-11T19:24:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20ee7fc640 10 client.124179.objecter ms_dispatch 0x7f20f0000b90 osd_map(624..624 src has 1..624) 2024-06-11T19:24:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20ee7fc640 3 client.124179.objecter handle_osd_map got epochs [624,624] > 0 2024-06-11T19:24:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20ee7fc640 3 client.124179.objecter handle_osd_map decoding full epoch 624 2024-06-11T19:24:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20ee7fc640 20 client.124179.objecter dump_active .. 0 homeless 2024-06-11T19:24:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20f5d02640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f20dc03fe10 0x7f20dc0422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.183+0000 7f20f5d02640 10 monclient: get_auth_request con 0x7f20dc03fe10 auth_method 0 2024-06-11T19:24:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.184+0000 7f20ee7fc640 10 client.124179.objecter ms_handle_connect 0x7f20dc03fe10 2024-06-11T19:24:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.184+0000 7f20f5d02640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f20dc03fe10 0x7f20dc0422d0 secure :-1 s=READY pgs=4291 cs=0 l=1 rev1=1 crypto rx=0x7f20e0003ea0 tx=0x7f20e00098d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.188+0000 7f20ee7fc640 1 -- 172.21.15.98:0/3113323136 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f20e8012de0 con 0x7f20f00a6260 2024-06-11T19:24:40.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.188+0000 7f20ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:40.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.188+0000 7f20ee7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.445+0000 7f20f7f66640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:24:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.445+0000 7f20f7f66640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.445+0000 7f20f7f66640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.445+0000 7f20f7f66640 1 -- 172.21.15.98:0/3113323136 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f20f00a1d50 con 0x7f20f00a6260 2024-06-11T19:24:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.446+0000 7f20ee7fc640 1 -- 172.21.15.98:0/3113323136 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f20e8013390 con 0x7f20f00a6260 2024-06-11T19:24:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.446+0000 7f20ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.446+0000 7f20ee7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:40.445 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":82,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:24:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 10 librados: watch_flush enter 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 10 librados: watch_flush exit 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 20 client.124179.objecter shutdown clearing up homeless session... 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 10 client.124179.objecter successfully canceled tick 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 -- 172.21.15.98:0/3113323136 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f20dc03fe10 msgr2=0x7f20dc0422d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f20dc03fe10 0x7f20dc0422d0 secure :-1 s=READY pgs=4291 cs=0 l=1 rev1=1 crypto rx=0x7f20e0003ea0 tx=0x7f20e00098d0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 10 monclient: shutdown 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 -- 172.21.15.98:0/3113323136 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f20f00a6260 msgr2=0x7f20f01cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f20f00a6260 0x7f20f01cd560 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f20e800d530 tx=0x7f20e800d9f0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 -- 172.21.15.98:0/3113323136 shutdown_connections 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f20dc03fe10 0x7f20dc0422d0 unknown :-1 s=CLOSED pgs=4291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f20f00a6260 0x7f20f01cd560 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f20f00a2d00 0x7f20f01c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 --2- 172.21.15.98:0/3113323136 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f20f006c890 0x7f20f01ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 -- 172.21.15.98:0/3113323136 >> 172.21.15.98:0/3113323136 conn(0x7f20f00a95b0 msgr2=0x7f20f00a72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 -- 172.21.15.98:0/3113323136 shutdown_connections 2024-06-11T19:24:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 -- 172.21.15.98:0/3113323136 wait complete. 2024-06-11T19:24:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.448+0000 7f20f7f66640 1 librados: shutdown 2024-06-11T19:24:40.457 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:24:40.457 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-06-11T19:24:40.458 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:24:40.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.562+0000 7f429ea0a640 10 monclient: get_monmap_and_config 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.562+0000 7f429ea0a640 10 monclient: build_initial_monmap 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.562+0000 7f429ea0a640 10 monclient: monmap: 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:40.563082+0000 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:40.563082+0000 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:40.561 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:40.562 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:40.562 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:40.562 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:40.562 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:40.562 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:40.562 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:40.562 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.562+0000 7f429ea0a640 1 Processor -- start 2024-06-11T19:24:40.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.562+0000 7f429ea0a640 1 -- start start 2024-06-11T19:24:40.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.562+0000 7f429ea0a640 10 monclient: init 2024-06-11T19:24:40.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:40.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 10 monclient: _add_conns ranks=[10,9,3,7,0,4,6,5,1,2,8] 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f42980a52a0 0x7f42980a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 10 monclient(hunting): picked mon.noname-f con 0x7f42980a52a0 addr v2:172.21.15.117:3300/0 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f429809d520 0x7f42980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 10 monclient(hunting): picked mon.noname-a con 0x7f429809d520 addr v2:172.21.15.98:3300/0 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4298073830 0x7f42980a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 10 monclient(hunting): picked mon.noname-g con 0x7f4298073830 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f429809daa0 con 0x7f429809d520 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f429809d2e0 con 0x7f42980a52a0 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f42980a5680 con 0x7f4298073830 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429ea0a640 10 monclient(hunting): authenticate will time out at 1690.370534s 2024-06-11T19:24:40.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f429809d520 0x7f42980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f429809d520 0x7f42980a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53668/0 (socket says 172.21.15.98:53668) 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 1 -- 172.21.15.98:0/3350021262 learned_addr learned my addr 172.21.15.98:0/3350021262 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 10 monclient(hunting): get_auth_request con 0x7f429809d520 auth_method 0 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 10 monclient(hunting): handle_auth_done global_id 66087 payload 274 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f4297fff640 1 --2- 172.21.15.98:0/3350021262 >> v2:172.21.15.117:3300/0 conn(0x7f42980a52a0 0x7f42980a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 1 -- 172.21.15.98:0/3350021262 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4298073830 msgr2=0x7f42980a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429cfa7640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4298073830 0x7f42980a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4298073830 0x7f42980a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 1 -- 172.21.15.98:0/3350021262 >> v2:172.21.15.117:3300/0 conn(0x7f42980a52a0 msgr2=0x7f42980a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 1 --2- 172.21.15.98:0/3350021262 >> v2:172.21.15.117:3300/0 conn(0x7f42980a52a0 0x7f42980a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 1 monclient: found mon.noname-a 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f428ffff640 1 -- 172.21.15.98:0/3350021262 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429809e9d0 con 0x7f429809d520 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429cfa7640 10 monclient: get_auth_request con 0x7f4298073830 auth_method 0 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.563+0000 7f429cfa7640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4298073830 0x7f42980a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.564+0000 7f428ffff640 1 -- 172.21.15.98:0/3350021262 >> v2:172.21.15.98:3300/0 conn(0x7f429809d520 msgr2=0x7f42980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.564+0000 7f428ffff640 1 -- 172.21.15.98:0/3350021262 >> v2:172.21.15.98:3300/0 conn(0x7f429809d520 msgr2=0x7f42980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.564+0000 7f428ffff640 1 --2- 172.21.15.98:0/3350021262 >> v2:172.21.15.98:3300/0 conn(0x7f429809d520 0x7f42980a3690 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f429809cad0 tx=0x7f428002b7e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:40.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.564+0000 7f428ffff640 1 --2- 172.21.15.98:0/3350021262 >> v2:172.21.15.98:3300/0 conn(0x7f429809d520 0x7f42980a3690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:40.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f428ffff640 1 --2- 172.21.15.98:0/3350021262 >> v2:172.21.15.98:3300/0 conn(0x7f429809d520 0x7f42980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f428ffff640 10 monclient: get_auth_request con 0x7f429809d520 auth_method 0 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f428ffff640 0 --2- 172.21.15.98:0/3350021262 >> v2:172.21.15.98:3300/0 conn(0x7f429809d520 0x7f42980a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f428ffff640 1 --2- 172.21.15.98:0/3350021262 >> v2:172.21.15.98:3300/0 conn(0x7f429809d520 0x7f42980a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient: ms_handle_reset current mon v2:172.21.15.98:3300/0 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 1 -- 172.21.15.98:0/3350021262 >> v2:172.21.15.98:3300/0 conn(0x7f429809d520 msgr2=0x7f42980a3690 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 1 --2- 172.21.15.98:0/3350021262 >> v2:172.21.15.98:3300/0 conn(0x7f429809d520 0x7f42980a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient: _add_conns ranks=[6,2,7,5,8,4,0,1] 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4284003630 0x7f4284005a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient(hunting): picked mon.noname-d con 0x7f4284003630 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4284006390 0x7f4284008820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4284006390 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4284009190 0x7f428400b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient(hunting): picked mon.noname-i con 0x7f4284009190 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 1 -- 172.21.15.98:0/3350021262 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f428400bfa0 con 0x7f4284006390 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 1 -- 172.21.15.98:0/3350021262 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f428400c120 con 0x7f4284003630 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 1 -- 172.21.15.98:0/3350021262 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f428400c320 con 0x7f4284009190 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.98:3300/0 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4284003630 0x7f4284005a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f428ffff640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4284006390 0x7f4284008820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 10 monclient(hunting): get_auth_request con 0x7f4284003630 auth_method 0 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:40.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f428ffff640 10 monclient(hunting): get_auth_request con 0x7f4284006390 auth_method 0 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f428ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f428ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f428ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f428ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f428ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f428ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 10 monclient(hunting): handle_auth_done global_id 66087 payload 293 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 1 -- 172.21.15.98:0/3350021262 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4284009190 msgr2=0x7f428400b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4284009190 0x7f428400b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 1 -- 172.21.15.98:0/3350021262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4284006390 msgr2=0x7f4284008820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:40.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4284006390 0x7f4284008820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 1 monclient: found mon.noname-d 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f429cfa7640 1 -- 172.21.15.98:0/3350021262 reap_dead start 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 1 -- 172.21.15.98:0/3350021262 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428400e7f0 con 0x7f4284003630 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f4297fff640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4284003630 0x7f4284005a80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f428800a9a0 tx=0x7f4280009240 comp rx=0 tx=0).ready entity=mon.6 client_cookie=514793981f38cbe8 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 1 -- 172.21.15.98:0/3350021262 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4288015070 con 0x7f4284003630 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient: dump: 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient: _finish_auth 0 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient: _check_auth_tickets 2024-06-11T19:24:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 1 -- 172.21.15.98:0/3350021262 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f428800cba0 con 0x7f4284003630 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f429ea0a640 5 monclient: authenticate success, global_id 66087 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f429ea0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f429ea0a640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 1 -- 172.21.15.98:0/3350021262 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f428800a570 con 0x7f4284003630 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f429ea0a640 4 set_mon_vals no callback set 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.765+0000 7f42977fe640 10 monclient: dump: 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.766+0000 7f429ea0a640 10 monclient: shutdown 2024-06-11T19:24:40.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.766+0000 7f429ea0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:40.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.766+0000 7f429ea0a640 1 -- 172.21.15.98:0/3350021262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4284003630 msgr2=0x7f4284005a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.766+0000 7f429ea0a640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4284003630 0x7f4284005a80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f428800a9a0 tx=0x7f4280009240 comp rx=0 tx=0).stop 2024-06-11T19:24:40.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.766+0000 7f429ea0a640 1 -- 172.21.15.98:0/3350021262 shutdown_connections 2024-06-11T19:24:40.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.766+0000 7f429ea0a640 1 --2- 172.21.15.98:0/3350021262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4284003630 0x7f4284005a80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.766+0000 7f429ea0a640 1 -- 172.21.15.98:0/3350021262 >> 172.21.15.98:0/3350021262 conn(0x7f42980a4780 msgr2=0x7f42980a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:40.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.766+0000 7f429ea0a640 1 -- 172.21.15.98:0/3350021262 shutdown_connections 2024-06-11T19:24:40.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.766+0000 7f429ea0a640 1 -- 172.21.15.98:0/3350021262 wait complete. 2024-06-11T19:24:40.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.767+0000 7f429ea0a640 10 monclient: build_initial_monmap 2024-06-11T19:24:40.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.767+0000 7f429ea0a640 10 monclient: monmap: 2024-06-11T19:24:40.779 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:40.779 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:40.780 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.767+0000 7f429ea0a640 1 librados: starting msgr at 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.767+0000 7f429ea0a640 1 librados: starting objecter 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.767+0000 7f429ea0a640 1 Processor -- start 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.767+0000 7f429ea0a640 1 -- start start 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.767+0000 7f429ea0a640 1 librados: setting wanted keys 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.767+0000 7f429ea0a640 1 librados: calling monclient init 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.767+0000 7f429ea0a640 10 monclient: init 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.767+0000 7f429ea0a640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.767+0000 7f429ea0a640 10 monclient: _add_conns ranks=[3,0,5,6,1,8,7,4,2] 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.767+0000 7f429ea0a640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4284003630 0x7f429811e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 monclient(hunting): picked mon.noname-g con 0x7f4284003630 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f429811e810 0x7f429811ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 monclient(hunting): picked mon.noname-a con 0x7f429811e810 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42981c97b0 0x7f42981c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 monclient(hunting): picked mon.noname-h con 0x7f42981c97b0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42980a1a90 con 0x7f429811e810 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f42980a1910 con 0x7f4284003630 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f42980a1c10 con 0x7f42981c97b0 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 monclient(hunting): authenticate will time out at 1690.575071s 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f429811e810 0x7f429811ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f429811e810 0x7f429811ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53688/0 (socket says 172.21.15.98:53688) 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 1 -- 172.21.15.98:0/2171751368 learned_addr learned my addr 172.21.15.98:0/2171751368 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 10 monclient(hunting): get_auth_request con 0x7f429811e810 auth_method 0 2024-06-11T19:24:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f4297fff640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4284003630 0x7f429811e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 10 monclient(hunting): handle_auth_done global_id 66096 payload 824 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429cfa7640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42981c97b0 0x7f42981c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 1 -- 172.21.15.98:0/2171751368 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42981c97b0 msgr2=0x7f42981c9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f42981c97b0 0x7f42981c9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 1 -- 172.21.15.98:0/2171751368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4284003630 msgr2=0x7f429811e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4284003630 0x7f429811e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 1 monclient: found mon.noname-a 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 1 -- 172.21.15.98:0/2171751368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42981339d0 con 0x7f429811e810 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f4297fff640 10 monclient: get_auth_request con 0x7f4284003630 auth_method 0 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f4297fff640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4284003630 0x7f429811e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f428ffff640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f429811e810 0x7f429811ec30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4280009990 tx=0x7f42800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 10 client.?.objecter ms_handle_connect 0x7f429811e810 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4280043070 con 0x7f429811e810 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 10 monclient: dump: 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.784 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 10 monclient: _finish_auth 0 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 10 monclient: _check_auth_tickets 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 5 monclient: authenticate success, global_id 66096 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4280035b40 con 0x7f429811e810 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f428002f050 con 0x7f429811e810 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 monclient: _renew_subs 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 1 -- 172.21.15.98:0/2171751368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4298121a20 con 0x7f429811e810 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f4296ffd640 4 set_mon_vals no callback set 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f42957fa640 10 monclient: dump: 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.787 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 client.66096.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 monclient: _renew_subs 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 1 -- 172.21.15.98:0/2171751368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4298121740 con 0x7f429811e810 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.768+0000 7f429ea0a640 1 librados: init done 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f429ea0a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f429ea0a640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42981ca170 0x7f42980a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f429ea0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f429ea0a640 1 -- 172.21.15.98:0/2171751368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4298120290 con 0x7f42981ca170 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f4280035cf0 con 0x7f429811e810 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f4297fff640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42981ca170 0x7f42980a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f4297fff640 10 monclient: get_auth_request con 0x7f42981ca170 auth_method 0 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f4297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f4297fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f4297fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f42957fa640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f427803fdb0 0x7f4278042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(625..625 src has 1..625) ==== 6797+0+0 (secure 0 0 0) 0x7f428003b070 con 0x7f429811e810 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f42957fa640 10 client.66096.objecter ms_dispatch 0x7f4298000b90 osd_map(625..625 src has 1..625) 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f42957fa640 3 client.66096.objecter handle_osd_map got epochs [625,625] > 0 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f4297fff640 10 monclient(hunting): handle_auth_done global_id 66105 payload 824 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f42957fa640 3 client.66096.objecter handle_osd_map decoding full epoch 625 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.769+0000 7f4297fff640 1 -- 172.21.15.98:0/2171751368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42980a1fa0 con 0x7f42981ca170 2024-06-11T19:24:40.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 20 client.66096.objecter dump_active .. 0 homeless 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f429cfa7640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f427803fdb0 0x7f4278042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f429cfa7640 10 monclient: get_auth_request con 0x7f427803fdb0 auth_method 0 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 10 client.66096.objecter ms_handle_connect 0x7f42981ca170 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 10 client.66096.objecter resend_mon_ops 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f4297fff640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42981ca170 0x7f42980a5b80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4284003460 tx=0x7f4288004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4288029070 con 0x7f42981ca170 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 10 monclient: dump: 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.790 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 10 client.66096.objecter ms_handle_connect 0x7f427803fdb0 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f429cfa7640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f427803fdb0 0x7f4278042270 secure :-1 s=READY pgs=4292 cs=0 l=1 rev1=1 crypto rx=0x7f4280009870 tx=0x7f42900083c0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f428800c940 con 0x7f42981ca170 2024-06-11T19:24:40.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42981ca170 msgr2=0x7f42980a5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42981ca170 0x7f42980a5b80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4284003460 tx=0x7f4288004730 comp rx=0 tx=0).stop 2024-06-11T19:24:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42981ca170 msgr2=0x7f42980a5b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.770+0000 7f42957fa640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42981ca170 0x7f42980a5b80 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.776+0000 7f429ea0a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.776+0000 7f429ea0a640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42980a60c0 0x7f42980a64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.776+0000 7f429ea0a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.776+0000 7f429ea0a640 1 -- 172.21.15.98:0/2171751368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f429811fa00 con 0x7f42980a60c0 2024-06-11T19:24:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f4297fff640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42980a60c0 0x7f42980a64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f4297fff640 10 monclient: get_auth_request con 0x7f42980a60c0 auth_method 0 2024-06-11T19:24:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f4297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f4297fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f4297fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f4297fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f4297fff640 10 monclient(hunting): handle_auth_done global_id 66114 payload 824 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f4297fff640 1 -- 172.21.15.98:0/2171751368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f42981ca550 con 0x7f42980a60c0 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f4297fff640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42980a60c0 0x7f42980a64f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4288000c00 tx=0x7f4288008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 10 client.66096.objecter ms_handle_connect 0x7f42980a60c0 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 10 client.66096.objecter resend_mon_ops 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4288029050 con 0x7f42980a60c0 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 10 monclient: dump: 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.795 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f428802a040 con 0x7f42980a60c0 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:40.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42980a60c0 msgr2=0x7f42980a64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42980a60c0 0x7f42980a64f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4288000c00 tx=0x7f4288008040 comp rx=0 tx=0).stop 2024-06-11T19:24:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 1 -- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42980a60c0 msgr2=0x7f42980a64f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:40.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.777+0000 7f42957fa640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42980a60c0 0x7f42980a64f0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.797 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 672605, 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-06-11T19:24:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:24:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:40.819 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 10 librados: watch_flush enter 2024-06-11T19:24:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 10 librados: watch_flush exit 2024-06-11T19:24:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 20 client.66096.objecter shutdown clearing up homeless session... 2024-06-11T19:24:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 10 client.66096.objecter successfully canceled tick 2024-06-11T19:24:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 1 -- 172.21.15.98:0/2171751368 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f427803fdb0 msgr2=0x7f4278042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f427803fdb0 0x7f4278042270 secure :-1 s=READY pgs=4292 cs=0 l=1 rev1=1 crypto rx=0x7f4280009870 tx=0x7f42900083c0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 10 monclient: shutdown 2024-06-11T19:24:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 1 -- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f429811e810 msgr2=0x7f429811ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f429811e810 0x7f429811ec30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4280009990 tx=0x7f42800046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429cfa7640 1 -- 172.21.15.98:0/2171751368 reap_dead start 2024-06-11T19:24:40.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 1 -- 172.21.15.98:0/2171751368 shutdown_connections 2024-06-11T19:24:40.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 1 --2- 172.21.15.98:0/2171751368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f429811e810 0x7f429811ec30 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 1 -- 172.21.15.98:0/2171751368 >> 172.21.15.98:0/2171751368 conn(0x7f42980a4780 msgr2=0x7f42980a7470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:40.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 1 -- 172.21.15.98:0/2171751368 shutdown_connections 2024-06-11T19:24:40.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 1 -- 172.21.15.98:0/2171751368 wait complete. 2024-06-11T19:24:40.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.778+0000 7f429ea0a640 1 librados: shutdown 2024-06-11T19:24:40.822 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:40.822 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:24:40.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:24:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.922+0000 7f1fe3887640 10 monclient: get_monmap_and_config 2024-06-11T19:24:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.922+0000 7f1fe3887640 10 monclient: build_initial_monmap 2024-06-11T19:24:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.922+0000 7f1fe3887640 10 monclient: monmap: 2024-06-11T19:24:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:40.923274+0000 2024-06-11T19:24:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:40.923274+0000 2024-06-11T19:24:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:40.922 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.922+0000 7f1fe3887640 1 Processor -- start 2024-06-11T19:24:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 1 -- start start 2024-06-11T19:24:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 10 monclient: init 2024-06-11T19:24:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 10 monclient: _add_conns ranks=[5,1,6,10,3,2,7,0,4,8,9] 2024-06-11T19:24:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1fdc0a4920 0x7f1fdc0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 10 monclient(hunting): picked mon.noname-h con 0x7f1fdc0a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f1fdc0a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:24:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1fdc0a2530 0x7f1fdc0a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 10 monclient(hunting): picked mon.noname-d con 0x7f1fdc0a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fdc0a7140 con 0x7f1fdc0a5240 2024-06-11T19:24:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f1fdc0a2b50 con 0x7f1fdc0a4920 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1fdc0a2e30 con 0x7f1fdc0a2530 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe3887640 10 monclient(hunting): authenticate will time out at 1690.730524s 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1fdc0a2530 0x7f1fdc0a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1fdc0a2530 0x7f1fdc0a60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:35338/0 (socket says 172.21.15.98:35338) 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 1 -- 172.21.15.98:0/318412074 learned_addr learned my addr 172.21.15.98:0/318412074 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 10 monclient(hunting): get_auth_request con 0x7f1fdc0a2530 auth_method 0 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1623640 1 --2- 172.21.15.98:0/318412074 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1fdc0a4920 0x7f1fdc0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1623640 10 monclient(hunting): get_auth_request con 0x7f1fdc0a4920 auth_method 0 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1623640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1623640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1623640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 10 monclient(hunting): handle_auth_done global_id 124188 payload 274 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 1 -- 172.21.15.98:0/318412074 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1fdc0a4920 msgr2=0x7f1fdc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 1 --2- 172.21.15.98:0/318412074 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1fdc0a4920 0x7f1fdc0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 1 -- 172.21.15.98:0/318412074 >> v1:172.21.15.117:6789/0 conn(0x7f1fdc0a5240 legacy=0x7f1fdc0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 10 monclient: _finish_hunting 0 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 1 monclient: found mon.noname-d 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.923+0000 7f1fe1e24640 1 -- 172.21.15.98:0/318412074 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fdc0a3110 con 0x7f1fdc0a2530 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fe1e24640 1 --2- 172.21.15.98:0/318412074 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1fdc0a2530 0x7f1fdc0a60b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009e80 tx=0x7f1fd402bd70 comp rx=0 tx=0).ready entity=mon.6 client_cookie=1a4f170a93a7652c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 1 -- 172.21.15.98:0/318412074 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1fd402cb80 con 0x7f1fdc0a2530 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 10 monclient: dump: 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 10 monclient: _finish_auth 0 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 1 -- 172.21.15.98:0/318412074 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd402cd40 con 0x7f1fdc0a2530 2024-06-11T19:24:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fe3887640 5 monclient: authenticate success, global_id 124188 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fe3887640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fe3887640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fe3887640 4 set_mon_vals no callback set 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 1 -- 172.21.15.98:0/318412074 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1fd4034860 con 0x7f1fdc0a2530 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.924+0000 7f1fcffff640 10 monclient: dump: 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.925+0000 7f1fe3887640 10 monclient: shutdown 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.925+0000 7f1fe3887640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.925+0000 7f1fe3887640 1 -- 172.21.15.98:0/318412074 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1fdc0a2530 msgr2=0x7f1fdc0a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.925+0000 7f1fe3887640 1 --2- 172.21.15.98:0/318412074 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1fdc0a2530 0x7f1fdc0a60b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009e80 tx=0x7f1fd402bd70 comp rx=0 tx=0).stop 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.925+0000 7f1fe3887640 1 -- 172.21.15.98:0/318412074 shutdown_connections 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.925+0000 7f1fe3887640 1 --2- 172.21.15.98:0/318412074 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1fdc0a2530 0x7f1fdc0a60b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.925+0000 7f1fe3887640 1 --2- 172.21.15.98:0/318412074 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f1fdc0a4920 0x7f1fdc0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.925+0000 7f1fe3887640 1 -- 172.21.15.98:0/318412074 >> 172.21.15.98:0/318412074 conn(0x7f1fdc0ab850 msgr2=0x7f1fdc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.925+0000 7f1fe3887640 1 -- 172.21.15.98:0/318412074 shutdown_connections 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.925+0000 7f1fe3887640 1 -- 172.21.15.98:0/318412074 wait complete. 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.925+0000 7f1fe3887640 10 monclient: build_initial_monmap 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.925+0000 7f1fe3887640 10 monclient: monmap: 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 1 librados: starting msgr at 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 1 librados: starting objecter 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 1 Processor -- start 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 1 -- start start 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 1 librados: setting wanted keys 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 1 librados: calling monclient init 2024-06-11T19:24:40.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 10 monclient: init 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 10 monclient: _add_conns ranks=[4,3,8,1,7,6,5,0,2] 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1fdc0a2530 0x7f1fdc137600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 10 monclient(hunting): picked mon.noname-c con 0x7f1fdc0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fdc0a4920 0x7f1fdc137b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 10 monclient(hunting): picked mon.noname-g con 0x7f1fdc0a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1fdc0a5240 0x7f1fdc1230f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 10 monclient(hunting): picked mon.noname-e con 0x7f1fdc0a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f1fdc0a6da0 con 0x7f1fdc0a4920 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1fdc138e50 con 0x7f1fdc0a2530 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f1fdc0a65f0 con 0x7f1fdc0a5240 2024-06-11T19:24:40.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe3887640 10 monclient(hunting): authenticate will time out at 1690.733796s 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1623640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1fdc0a2530 0x7f1fdc137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1623640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1fdc0a2530 0x7f1fdc137600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46880/0 (socket says 172.21.15.98:46880) 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1e24640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1fdc0a5240 0x7f1fdc1230f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1623640 1 -- 172.21.15.98:0/2549244020 learned_addr learned my addr 172.21.15.98:0/2549244020 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1623640 10 monclient(hunting): get_auth_request con 0x7f1fdc0a2530 auth_method 0 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1623640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1623640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1623640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1e24640 10 monclient(hunting): get_auth_request con 0x7f1fdc0a5240 auth_method 0 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1e24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1e24640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1e24640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe0e22640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fdc0a4920 0x7f1fdc137b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1623640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.926+0000 7f1fe1623640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1623640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe0e22640 10 monclient(hunting): get_auth_request con 0x7f1fdc0a4920 auth_method 0 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe0e22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe0e22640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe0e22640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe0e22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe0e22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe0e22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 10 monclient(hunting): handle_auth_done global_id 124181 payload 824 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 1 -- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1fdc0a2530 msgr2=0x7f1fdc137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1fdc0a2530 0x7f1fdc137600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 1 -- 172.21.15.98:0/2549244020 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fdc0a4920 msgr2=0x7f1fdc137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fdc0a4920 0x7f1fdc137b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 10 monclient: _finish_hunting 0 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 1 monclient: found mon.noname-e 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 1 -- 172.21.15.98:0/2549244020 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fdc1c0bf0 con 0x7f1fdc0a5240 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1623640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1fdc0a2530 0x7f1fdc137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe1e24640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1fdc0a5240 0x7f1fdc1230f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4032000 tx=0x7f1fd4004a40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 10 client.?.objecter ms_handle_connect 0x7f1fdc0a5240 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1fd4042020 con 0x7f1fdc0a5240 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 10 monclient: dump: 2024-06-11T19:24:40.944 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.945 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 10 monclient: _finish_auth 0 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 10 monclient: _check_auth_tickets 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe3887640 5 monclient: authenticate success, global_id 124181 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd40046c0 con 0x7f1fdc0a5240 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe3887640 10 monclient: _renew_subs 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1fd403e8c0 con 0x7f1fdc0a5240 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe3887640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe3887640 1 -- 172.21.15.98:0/2549244020 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fdc0aac80 con 0x7f1fdc0a5240 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcf7fe640 4 set_mon_vals no callback set 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fcdffb640 10 monclient: dump: 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe3887640 10 client.124181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe3887640 10 monclient: _renew_subs 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe3887640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe3887640 1 -- 172.21.15.98:0/2549244020 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fdc0a9110 con 0x7f1fdc0a5240 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.927+0000 7f1fe3887640 1 librados: init done 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fe3887640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fe3887640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc126980 0x7f1fdc123630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fe3887640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fe3887640 1 -- 172.21.15.98:0/2549244020 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1fdc125610 con 0x7f1fdc126980 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f1fd4045020 con 0x7f1fdc0a5240 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fe1623640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc126980 0x7f1fdc123630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fe1623640 10 monclient: get_auth_request con 0x7f1fdc126980 auth_method 0 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fe1623640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fe1623640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fe1623640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fe1623640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fe1623640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fe1623640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fcdffb640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1fc003fdb0 0x7f1fc0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(625..625 src has 1..625) ==== 6797+0+0 (secure 0 0 0) 0x7f1fd403a040 con 0x7f1fdc0a5240 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fcdffb640 10 client.124181.objecter ms_dispatch 0x7f1fdc000b90 osd_map(625..625 src has 1..625) 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fcdffb640 3 client.124181.objecter handle_osd_map got epochs [625,625] > 0 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.928+0000 7f1fcdffb640 3 client.124181.objecter handle_osd_map decoding full epoch 625 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fe1623640 10 monclient(hunting): handle_auth_done global_id 85160 payload 824 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fe1623640 1 -- 172.21.15.98:0/2549244020 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1fdc0a4d00 con 0x7f1fdc126980 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 20 client.124181.objecter dump_active .. 0 homeless 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fe0e22640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1fc003fdb0 0x7f1fc0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fe0e22640 10 monclient: get_auth_request con 0x7f1fc003fdb0 auth_method 0 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 10 client.124181.objecter ms_handle_connect 0x7f1fdc126980 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fe1623640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc126980 0x7f1fdc123630 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8010c80 tx=0x7f1fd800e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 10 client.124181.objecter resend_mon_ops 2024-06-11T19:24:40.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1fd8018070 con 0x7f1fdc126980 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 10 monclient: dump: 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.950 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 10 client.124181.objecter ms_handle_connect 0x7f1fc003fdb0 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fe0e22640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1fc003fdb0 0x7f1fc0042270 secure :-1 s=READY pgs=4293 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc06c5b0 tx=0x7f1fd0009060 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1fd801d450 con 0x7f1fdc126980 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc126980 msgr2=0x7f1fdc123630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc126980 0x7f1fdc123630 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8010c80 tx=0x7f1fd800e5a0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc126980 msgr2=0x7f1fdc123630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:40.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.929+0000 7f1fcdffb640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc126980 0x7f1fdc123630 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.935+0000 7f1fe3887640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:40.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.935+0000 7f1fe3887640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc123b70 0x7f1fdc123f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:40.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.935+0000 7f1fe3887640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:40.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fe3887640 1 -- 172.21.15.98:0/2549244020 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1fdc050830 con 0x7f1fdc123b70 2024-06-11T19:24:40.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fe1623640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc123b70 0x7f1fdc123f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:40.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fe1623640 10 monclient: get_auth_request con 0x7f1fdc123b70 auth_method 0 2024-06-11T19:24:40.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fe1623640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:40.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fe1623640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fe1623640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fe1623640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fe1623640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fe1623640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fe1623640 10 monclient(hunting): handle_auth_done global_id 85169 payload 824 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fe1623640 1 -- 172.21.15.98:0/2549244020 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1fdc0692c0 con 0x7f1fdc123b70 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 10 client.124181.objecter ms_handle_connect 0x7f1fdc123b70 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 10 client.124181.objecter resend_mon_ops 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fe1623640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc123b70 0x7f1fdc123f80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8010c80 tx=0x7f1fd80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1fd801f2e0 con 0x7f1fdc123b70 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 10 monclient: dump: 2024-06-11T19:24:40.954 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:40.955 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9275 (secure 0 0 0) 0x7f1fd801f2e0 con 0x7f1fdc123b70 2024-06-11T19:24:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc123b70 msgr2=0x7f1fdc123f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc123b70 0x7f1fdc123f80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8010c80 tx=0x7f1fd80046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 1 -- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc123b70 msgr2=0x7f1fdc123f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:40.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.936+0000 7f1fcdffb640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1fdc123b70 0x7f1fdc123f80 secure :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8010c80 tx=0x7f1fd80046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:40.956 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 552710, 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:24:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:40.977 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 10 librados: watch_flush enter 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 10 librados: watch_flush exit 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 20 client.124181.objecter shutdown clearing up homeless session... 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 10 client.124181.objecter successfully canceled tick 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 1 -- 172.21.15.98:0/2549244020 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1fc003fdb0 msgr2=0x7f1fc0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1fc003fdb0 0x7f1fc0042270 secure :-1 s=READY pgs=4293 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc06c5b0 tx=0x7f1fd0009060 comp rx=0 tx=0).stop 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 10 monclient: shutdown 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 1 -- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1fdc0a5240 msgr2=0x7f1fdc1230f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 1 --2- 172.21.15.98:0/2549244020 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1fdc0a5240 0x7f1fdc1230f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4032000 tx=0x7f1fd4004a40 comp rx=0 tx=0).stop 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe1e24640 1 -- 172.21.15.98:0/2549244020 reap_dead start 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe1e24640 1 -- 172.21.15.98:0/2549244020 reap_dead start 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 1 -- 172.21.15.98:0/2549244020 shutdown_connections 2024-06-11T19:24:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 1 -- 172.21.15.98:0/2549244020 >> 172.21.15.98:0/2549244020 conn(0x7f1fdc0ab850 msgr2=0x7f1fdc1be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:40.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 1 -- 172.21.15.98:0/2549244020 shutdown_connections 2024-06-11T19:24:40.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 1 -- 172.21.15.98:0/2549244020 wait complete. 2024-06-11T19:24:40.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:40.937+0000 7f1fe3887640 1 librados: shutdown 2024-06-11T19:24:40.980 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:40.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:24:40.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:24:41.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.080+0000 7ffa46622640 10 monclient: get_monmap_and_config 2024-06-11T19:24:41.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.080+0000 7ffa46622640 10 monclient: build_initial_monmap 2024-06-11T19:24:41.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.080+0000 7ffa46622640 10 monclient: monmap: 2024-06-11T19:24:41.079 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:41.079 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.079 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:41.081646+0000 2024-06-11T19:24:41.079 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:41.081646+0000 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:41.080 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 1 Processor -- start 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 1 -- start start 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 10 monclient: init 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 10 monclient: _add_conns ranks=[0,1,6,10,7,2,8,3,4,9,5] 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa400a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ffa400a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffa4013f380 0x7ffa40141770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 10 monclient(hunting): picked mon.noname-d con 0x7ffa4013f380 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa400a7140 con 0x7ffa400a4920 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa400a2b50 con 0x7ffa400a2530 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ffa400a2e30 con 0x7ffa4013f380 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.081+0000 7ffa46622640 10 monclient(hunting): authenticate will time out at 1690.888949s 2024-06-11T19:24:41.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa3ffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ffa400a4920 0x7ffa400a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53802/0 (socket says 172.21.15.98:53802) 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa3ffff640 1 -- 172.21.15.98:0/2839094015 learned_addr learned my addr 172.21.15.98:0/2839094015 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffa4013f380 0x7ffa40141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 10 monclient(hunting): get_auth_request con 0x7ffa4013f380 auth_method 0 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 10 monclient(hunting): handle_auth_done global_id 124197 payload 274 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 1 -- 172.21.15.98:0/2839094015 >> v1:172.21.15.117:6789/0 conn(0x7ffa400a2530 legacy=0x7ffa4013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 1 -- 172.21.15.98:0/2839094015 >> v1:172.21.15.98:6789/0 conn(0x7ffa400a4920 legacy=0x7ffa400a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 10 monclient: _finish_hunting 0 2024-06-11T19:24:41.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 1 monclient: found mon.noname-d 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa3effd640 1 -- 172.21.15.98:0/2839094015 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 853052053 0 0) 0x7ffa400a7140 con 0x7ffa400a4920 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa44bbf640 1 -- 172.21.15.98:0/2839094015 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa400a3110 con 0x7ffa4013f380 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.082+0000 7ffa3effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa44bbf640 1 --2- 172.21.15.98:0/2839094015 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffa4013f380 0x7ffa40141770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009dc0 tx=0x7ffa3802bd50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c8d0f48c798df497 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 1 -- 172.21.15.98:0/2839094015 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffa38039070 con 0x7ffa4013f380 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 10 monclient: dump: 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.091 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 10 monclient: _finish_auth 0 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 10 monclient: _check_auth_tickets 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 5 monclient: authenticate success, global_id 124197 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:41.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 1 -- 172.21.15.98:0/2839094015 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa3802cd70 con 0x7ffa4013f380 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 4 set_mon_vals no callback set 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 1 -- 172.21.15.98:0/2839094015 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffa380327b0 con 0x7ffa4013f380 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa3effd640 10 monclient: dump: 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.093 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 10 monclient: shutdown 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 1 -- 172.21.15.98:0/2839094015 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffa4013f380 msgr2=0x7ffa40141770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 1 --2- 172.21.15.98:0/2839094015 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffa4013f380 0x7ffa40141770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009dc0 tx=0x7ffa3802bd50 comp rx=0 tx=0).stop 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 1 -- 172.21.15.98:0/2839094015 shutdown_connections 2024-06-11T19:24:41.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 1 --2- 172.21.15.98:0/2839094015 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffa4013f380 0x7ffa40141770 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 1 -- 172.21.15.98:0/2839094015 >> 172.21.15.98:0/2839094015 conn(0x7ffa400ab850 msgr2=0x7ffa400a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 1 -- 172.21.15.98:0/2839094015 shutdown_connections 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.083+0000 7ffa46622640 1 -- 172.21.15.98:0/2839094015 wait complete. 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.084+0000 7ffa46622640 10 monclient: build_initial_monmap 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.084+0000 7ffa46622640 10 monclient: monmap: 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:41.095 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.084+0000 7ffa46622640 1 librados: starting msgr at 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.084+0000 7ffa46622640 1 librados: starting objecter 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.084+0000 7ffa46622640 1 Processor -- start 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.084+0000 7ffa46622640 1 -- start start 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.084+0000 7ffa46622640 1 librados: setting wanted keys 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.084+0000 7ffa46622640 1 librados: calling monclient init 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.084+0000 7ffa46622640 10 monclient: init 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 10 monclient: _add_conns ranks=[0,4,2,5,1,3,6,7,8] 2024-06-11T19:24:41.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa400a2530 0x7ffa40137720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 10 monclient(hunting): picked mon.noname-a con 0x7ffa400a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a4920 0x7ffa40137c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 10 monclient(hunting): picked mon.noname-c con 0x7ffa400a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa4013f380 0x7ffa4009fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 10 monclient(hunting): picked mon.noname-b con 0x7ffa4013f380 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa40138e50 con 0x7ffa400a2530 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ffa400a84b0 con 0x7ffa4013f380 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ffa400a6da0 con 0x7ffa400a4920 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa46622640 10 monclient(hunting): authenticate will time out at 1690.892379s 2024-06-11T19:24:41.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa400a2530 0x7ffa40137720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa400a2530 0x7ffa40137720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53704/0 (socket says 172.21.15.98:53704) 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa44bbf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa4013f380 0x7ffa4009fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3f7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a4920 0x7ffa40137c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 1 -- 172.21.15.98:0/885694346 learned_addr learned my addr 172.21.15.98:0/885694346 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 10 monclient(hunting): get_auth_request con 0x7ffa400a2530 auth_method 0 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa44bbf640 10 monclient(hunting): get_auth_request con 0x7ffa4013f380 auth_method 0 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa44bbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa44bbf640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa44bbf640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3f7fe640 10 monclient(hunting): get_auth_request con 0x7ffa400a4920 auth_method 0 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa44bbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa44bbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa44bbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 10 monclient(hunting): handle_auth_done global_id 66132 payload 824 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 1 -- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a4920 msgr2=0x7ffa40137c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a4920 0x7ffa40137c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 1 -- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa4013f380 msgr2=0x7ffa4009fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa4013f380 0x7ffa4009fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 1 monclient: found mon.noname-a 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 1 -- 172.21.15.98:0/885694346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa40073310 con 0x7ffa400a2530 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa44bbf640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffa4013f380 0x7ffa4009fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3ffff640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa400a2530 0x7ffa40137720 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000c450 tx=0x7ffa3000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3cff9640 10 client.?.objecter ms_handle_connect 0x7ffa400a2530 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.085+0000 7ffa3cff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffa30018070 con 0x7ffa400a2530 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 10 monclient: dump: 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.100 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 10 monclient: _finish_auth 0 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 10 monclient: _check_auth_tickets 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa300047e0 con 0x7ffa400a2530 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 5 monclient: authenticate success, global_id 66132 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 10 monclient: _renew_subs 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 1 -- 172.21.15.98:0/885694346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa40122f30 con 0x7ffa400a2530 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffa30004e40 con 0x7ffa400a2530 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 10 client.66132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3cff9640 10 monclient: dump: 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.102 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa3e7fc640 4 set_mon_vals no callback set 2024-06-11T19:24:41.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 10 monclient: _renew_subs 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 1 -- 172.21.15.98:0/885694346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa4013f030 con 0x7ffa400a2530 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 1 librados: init done 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0820 0x7ffa4002cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.086+0000 7ffa46622640 1 -- 172.21.15.98:0/885694346 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ffa40050830 con 0x7ffa400a0820 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ffa300049d0 con 0x7ffa400a2530 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3f7fe640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0820 0x7ffa4002cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3f7fe640 10 monclient: get_auth_request con 0x7ffa400a0820 auth_method 0 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_done global_id 114439 payload 824 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffa2c03fdb0 0x7ffa2c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3f7fe640 1 -- 172.21.15.98:0/885694346 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa4013cf10 con 0x7ffa400a0820 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(625..625 src has 1..625) ==== 6797+0+0 (secure 0 0 0) 0x7ffa30050f10 con 0x7ffa400a2530 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 10 client.66132.objecter ms_dispatch 0x7ffa40000b90 osd_map(625..625 src has 1..625) 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 3 client.66132.objecter handle_osd_map got epochs [625,625] > 0 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 3 client.66132.objecter handle_osd_map decoding full epoch 625 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 20 client.66132.objecter dump_active .. 0 homeless 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3f7fe640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0820 0x7ffa4002cb40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400f520 tx=0x7ffa3400f9e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 10 client.66132.objecter ms_handle_connect 0x7ffa400a0820 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 10 client.66132.objecter resend_mon_ops 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffa340052e0 con 0x7ffa400a0820 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa44bbf640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffa2c03fdb0 0x7ffa2c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 10 monclient: dump: 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.107 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa44bbf640 10 monclient: get_auth_request con 0x7ffa2c03fdb0 auth_method 0 2024-06-11T19:24:41.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffa34018070 con 0x7ffa400a0820 2024-06-11T19:24:41.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:41.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:41.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0820 msgr2=0x7ffa4002cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0820 0x7ffa4002cb40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400f520 tx=0x7ffa3400f9e0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0820 msgr2=0x7ffa4002cb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:41.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.087+0000 7ffa3cff9640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0820 0x7ffa4002cb40 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400f520 tx=0x7ffa3400f9e0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.088+0000 7ffa3cff9640 10 client.66132.objecter ms_handle_connect 0x7ffa2c03fdb0 2024-06-11T19:24:41.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.088+0000 7ffa44bbf640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffa2c03fdb0 0x7ffa2c042270 secure :-1 s=READY pgs=4294 cs=0 l=1 rev1=1 crypto rx=0x7ffa38036000 tx=0x7ffa38030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa46622640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa46622640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0c00 0x7ffa4002d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa46622640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa46622640 1 -- 172.21.15.98:0/885694346 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ffa400a4d00 con 0x7ffa400a0c00 2024-06-11T19:24:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3f7fe640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0c00 0x7ffa4002d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3f7fe640 10 monclient: get_auth_request con 0x7ffa400a0c00 auth_method 0 2024-06-11T19:24:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_done global_id 114448 payload 824 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3f7fe640 1 -- 172.21.15.98:0/885694346 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffa4006c300 con 0x7ffa400a0c00 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3cff9640 10 client.66132.objecter ms_handle_connect 0x7ffa400a0c00 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3cff9640 10 client.66132.objecter resend_mon_ops 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3f7fe640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0c00 0x7ffa4002d080 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400f520 tx=0x7ffa34001020 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffa34007030 con 0x7ffa400a0c00 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.094+0000 7ffa3cff9640 10 monclient: dump: 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7ffa34003b50 con 0x7ffa400a0c00 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa3cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa3cff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0c00 msgr2=0x7ffa4002d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa3cff9640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0c00 0x7ffa4002d080 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400f520 tx=0x7ffa34001020 comp rx=0 tx=0).stop 2024-06-11T19:24:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa3cff9640 1 -- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0c00 msgr2=0x7ffa4002d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:41.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa3cff9640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffa400a0c00 0x7ffa4002d080 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400f520 tx=0x7ffa34001020 comp rx=0 tx=0).stop 2024-06-11T19:24:41.112 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:41.112 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:41.112 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:41.112 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:41.112 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:41.112 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 169461, 2024-06-11T19:24:41.112 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:41.112 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:41.112 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:41.112 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.113 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.114 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:41.115 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.116 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:41.117 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:41.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:41.120 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:41.121 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.124 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.127 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.130 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:41.131 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:24:41.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.132 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:41.132 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:41.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 10 librados: watch_flush enter 2024-06-11T19:24:41.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 10 librados: watch_flush exit 2024-06-11T19:24:41.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 20 client.66132.objecter shutdown clearing up homeless session... 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 10 client.66132.objecter successfully canceled tick 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 1 -- 172.21.15.98:0/885694346 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffa2c03fdb0 msgr2=0x7ffa2c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffa2c03fdb0 0x7ffa2c042270 secure :-1 s=READY pgs=4294 cs=0 l=1 rev1=1 crypto rx=0x7ffa38036000 tx=0x7ffa38030000 comp rx=0 tx=0).stop 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 10 monclient: shutdown 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 1 -- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa400a2530 msgr2=0x7ffa40137720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 1 --2- 172.21.15.98:0/885694346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa400a2530 0x7ffa40137720 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000c450 tx=0x7ffa3000c910 comp rx=0 tx=0).stop 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa44bbf640 1 -- 172.21.15.98:0/885694346 reap_dead start 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa44bbf640 1 -- 172.21.15.98:0/885694346 reap_dead start 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 1 -- 172.21.15.98:0/885694346 shutdown_connections 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 1 -- 172.21.15.98:0/885694346 >> 172.21.15.98:0/885694346 conn(0x7ffa400ab850 msgr2=0x7ffa400a9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 1 -- 172.21.15.98:0/885694346 shutdown_connections 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 1 -- 172.21.15.98:0/885694346 wait complete. 2024-06-11T19:24:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.095+0000 7ffa46622640 1 librados: shutdown 2024-06-11T19:24:41.135 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:41.135 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:24:41.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:24:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.235+0000 7f0ce1a42640 10 monclient: get_monmap_and_config 2024-06-11T19:24:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.235+0000 7f0ce1a42640 10 monclient: build_initial_monmap 2024-06-11T19:24:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.235+0000 7f0ce1a42640 10 monclient: monmap: 2024-06-11T19:24:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:41.235 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:41.236957+0000 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:41.236957+0000 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:41.236 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:41.237 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:41.237 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.236+0000 7f0ce1a42640 1 Processor -- start 2024-06-11T19:24:41.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.236+0000 7f0ce1a42640 1 -- start start 2024-06-11T19:24:41.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.236+0000 7f0ce1a42640 10 monclient: init 2024-06-11T19:24:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 10 monclient: _add_conns ranks=[8,2,1,5,10,0,9,7,4,3,6] 2024-06-11T19:24:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cdc0a4920 0x7f0cdc0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 10 monclient(hunting): picked mon.noname-e con 0x7f0cdc0a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cdc0a5240 0x7f0cdc0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 10 monclient(hunting): picked mon.noname-b con 0x7f0cdc0a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f0cdc0a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:24:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cdc0a7140 con 0x7f0cdc0a2530 2024-06-11T19:24:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0cdc0a2b50 con 0x7f0cdc0a5240 2024-06-11T19:24:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0cdc0a2e30 con 0x7f0cdc0a4920 2024-06-11T19:24:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0ce1a42640 10 monclient(hunting): authenticate will time out at 1691.044101s 2024-06-11T19:24:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cda7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cdc0a5240 0x7f0cdc0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cdc0a4920 0x7f0cdc0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cda7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cdc0a5240 0x7f0cdc0a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57340/0 (socket says 172.21.15.98:57340) 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cdc0a4920 0x7f0cdc0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:53170/0 (socket says 172.21.15.98:53170) 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cda7fc640 1 -- 172.21.15.98:0/1215959682 learned_addr learned my addr 172.21.15.98:0/1215959682 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cda7fc640 10 monclient(hunting): get_auth_request con 0x7f0cdc0a5240 auth_method 0 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cda7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cda7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 10 monclient(hunting): get_auth_request con 0x7f0cdc0a4920 auth_method 0 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cda7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cda7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cda7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_done global_id 124190 payload 274 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 1 -- 172.21.15.98:0/1215959682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cdc0a5240 msgr2=0x7f0cdc0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 1 --2- 172.21.15.98:0/1215959682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cdc0a5240 0x7f0cdc0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 1 -- 172.21.15.98:0/1215959682 >> v1:172.21.15.117:6789/0 conn(0x7f0cdc0a2530 legacy=0x7f0cdc0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 10 monclient: _finish_hunting 0 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 1 monclient: found mon.noname-e 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cdaffd640 1 -- 172.21.15.98:0/1215959682 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cdc0a3110 con 0x7f0cdc0a4920 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.237+0000 7f0cda7fc640 1 --2- 172.21.15.98:0/1215959682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cdc0a5240 0x7f0cdc0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cdaffd640 1 --2- 172.21.15.98:0/1215959682 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cdc0a4920 0x7f0cdc0a4d00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc0097d0 tx=0x7f0ccc02b6d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=d6d023711a507e7f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 1 -- 172.21.15.98:0/1215959682 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ccc037070 con 0x7f0cdc0a4920 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 10 monclient: dump: 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.249 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.250 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.250 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.250 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.250 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.250 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.250 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 10 monclient: _finish_auth 0 2024-06-11T19:24:41.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:24:41.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:41.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 1 -- 172.21.15.98:0/1215959682 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ccc030980 con 0x7f0cdc0a4920 2024-06-11T19:24:41.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0ce1a42640 5 monclient: authenticate success, global_id 124190 2024-06-11T19:24:41.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0ce1a42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0ce1a42640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 1 -- 172.21.15.98:0/1215959682 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ccc030bb0 con 0x7f0cdc0a4920 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0ce1a42640 4 set_mon_vals no callback set 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0cd9ffb640 10 monclient: dump: 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.251 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0ce1a42640 10 monclient: shutdown 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0ce1a42640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0ce1a42640 1 -- 172.21.15.98:0/1215959682 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cdc0a4920 msgr2=0x7f0cdc0a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.238+0000 7f0ce1a42640 1 --2- 172.21.15.98:0/1215959682 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cdc0a4920 0x7f0cdc0a4d00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc0097d0 tx=0x7f0ccc02b6d0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.239+0000 7f0ce1a42640 1 -- 172.21.15.98:0/1215959682 shutdown_connections 2024-06-11T19:24:41.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.239+0000 7f0ce1a42640 1 --2- 172.21.15.98:0/1215959682 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cdc0a5240 0x7f0cdc0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.239+0000 7f0ce1a42640 1 --2- 172.21.15.98:0/1215959682 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0cdc0a4920 0x7f0cdc0a4d00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.239+0000 7f0ce1a42640 1 -- 172.21.15.98:0/1215959682 >> 172.21.15.98:0/1215959682 conn(0x7f0cdc0ab850 msgr2=0x7f0cdc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.239+0000 7f0ce1a42640 1 -- 172.21.15.98:0/1215959682 shutdown_connections 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.239+0000 7f0ce1a42640 1 -- 172.21.15.98:0/1215959682 wait complete. 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.239+0000 7f0ce1a42640 10 monclient: build_initial_monmap 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.239+0000 7f0ce1a42640 10 monclient: monmap: 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:41.253 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.239+0000 7f0ce1a42640 1 librados: starting msgr at 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.239+0000 7f0ce1a42640 1 librados: starting objecter 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.239+0000 7f0ce1a42640 1 Processor -- start 2024-06-11T19:24:41.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 1 -- start start 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 1 librados: setting wanted keys 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 1 librados: calling monclient init 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 10 monclient: init 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 10 monclient: _add_conns ranks=[4,0,2,3,5,6,8,1,7] 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cdc0a2530 0x7f0cdc1cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 10 monclient(hunting): picked mon.noname-c con 0x7f0cdc0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cdc0a4920 0x7f0cdc1ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 10 monclient(hunting): picked mon.noname-a con 0x7f0cdc0a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cdc0a5240 0x7f0cdc121fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 10 monclient(hunting): picked mon.noname-b con 0x7f0cdc0a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc0a6da0 con 0x7f0cdc0a4920 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0cdc138e50 con 0x7f0cdc0a5240 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0cdc0a65f0 con 0x7f0cdc0a2530 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0ce1a42640 10 monclient(hunting): authenticate will time out at 1691.047639s 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdaffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cdc0a2530 0x7f0cdc1cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cda7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cdc0a4920 0x7f0cdc1ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdb7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cdc0a5240 0x7f0cdc121fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cda7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cdc0a4920 0x7f0cdc1ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53720/0 (socket says 172.21.15.98:53720) 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdaffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cdc0a2530 0x7f0cdc1cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46928/0 (socket says 172.21.15.98:46928) 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cda7fc640 1 -- 172.21.15.98:0/2205388785 learned_addr learned my addr 172.21.15.98:0/2205388785 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cda7fc640 10 monclient(hunting): get_auth_request con 0x7f0cdc0a4920 auth_method 0 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cda7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cda7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdaffd640 10 monclient(hunting): get_auth_request con 0x7f0cdc0a2530 auth_method 0 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdaffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f0cdc0a5240 auth_method 0 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cda7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cda7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cda7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.240+0000 7f0cdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cda7fc640 10 monclient(hunting): handle_auth_done global_id 66141 payload 824 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cda7fc640 1 -- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cdc0a2530 msgr2=0x7f0cdc1cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cda7fc640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cdc0a2530 0x7f0cdc1cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cda7fc640 1 -- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cdc0a5240 msgr2=0x7f0cdc121fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cda7fc640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cdc0a5240 0x7f0cdc121fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cda7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cda7fc640 1 monclient: found mon.noname-a 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cda7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cda7fc640 1 -- 172.21.15.98:0/2205388785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cdc137720 con 0x7f0cdc0a4920 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cdb7fe640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0cdc0a5240 0x7f0cdc121fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:41.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cdaffd640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0cdc0a2530 0x7f0cdc1cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cda7fc640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cdc0a4920 0x7f0cdc1ce130 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0cd000e8d0 tx=0x7f0cd000ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 10 client.?.objecter ms_handle_connect 0x7f0cdc0a4920 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0cd00044a0 con 0x7f0cdc0a4920 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 10 monclient: dump: 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 10 monclient: _finish_auth 0 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cd000c940 con 0x7f0cdc0a4920 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0ce1a42640 5 monclient: authenticate success, global_id 66141 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0ce1a42640 10 monclient: _renew_subs 2024-06-11T19:24:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0ce1a42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0ce1a42640 1 -- 172.21.15.98:0/2205388785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cdc125160 con 0x7f0cdc0a4920 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0cd0003610 con 0x7f0cdc0a4920 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cc3fff640 10 monclient: dump: 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.261 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0ce1a42640 10 client.66141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0ce1a42640 10 monclient: _renew_subs 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0ce1a42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0ce1a42640 1 -- 172.21.15.98:0/2205388785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cdc1ce670 con 0x7f0cdc0a4920 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0ce1a42640 1 librados: init done 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.241+0000 7f0cd97fa640 4 set_mon_vals no callback set 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0ce1a42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:41.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0ce1a42640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc1224f0 0x7f0cdc1228d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f0cd0004030 con 0x7f0cdc0a4920 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0ce1a42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0ce1a42640 1 -- 172.21.15.98:0/2205388785 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0cdc050830 con 0x7f0cdc1224f0 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdaffd640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc1224f0 0x7f0cdc1228d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdaffd640 10 monclient: get_auth_request con 0x7f0cdc1224f0 auth_method 0 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdaffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cc3fff640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0cc803fde0 0x7f0cc80422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdaffd640 1 -- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc1224f0 msgr2=0x7f0cdc1228d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdaffd640 1 -- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc1224f0 msgr2=0x7f0cdc1228d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:24:41.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdaffd640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc1224f0 0x7f0cdc1228d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdaffd640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc1224f0 0x7f0cdc1228d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(625..625 src has 1..625) ==== 6797+0+0 (secure 0 0 0) 0x7f0cd000f040 con 0x7f0cdc0a4920 2024-06-11T19:24:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cc3fff640 10 client.66141.objecter ms_dispatch 0x7f0cdc000b90 osd_map(625..625 src has 1..625) 2024-06-11T19:24:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cc3fff640 3 client.66141.objecter handle_osd_map got epochs [625,625] > 0 2024-06-11T19:24:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cc3fff640 3 client.66141.objecter handle_osd_map decoding full epoch 625 2024-06-11T19:24:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cc3fff640 20 client.66141.objecter dump_active .. 0 homeless 2024-06-11T19:24:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdb7fe640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0cc803fde0 0x7f0cc80422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.242+0000 7f0cdb7fe640 10 monclient: get_auth_request con 0x7f0cc803fde0 auth_method 0 2024-06-11T19:24:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.243+0000 7f0cc3fff640 10 client.66141.objecter ms_handle_connect 0x7f0cc803fde0 2024-06-11T19:24:41.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.243+0000 7f0cdb7fe640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0cc803fde0 0x7f0cc80422a0 secure :-1 s=READY pgs=4295 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0a2950 tx=0x7f0cd4003d10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cdaffd640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc1224f0 0x7f0cdc1228d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cdaffd640 10 monclient: get_auth_request con 0x7f0cdc1224f0 auth_method 0 2024-06-11T19:24:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cdaffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cdaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_done global_id 124215 payload 824 2024-06-11T19:24:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cdaffd640 1 -- 172.21.15.98:0/2205388785 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0cdc1239d0 con 0x7f0cdc1224f0 2024-06-11T19:24:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cdaffd640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc1224f0 0x7f0cdc1228d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0abf80 tx=0x7f0ccc0046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 10 client.66141.objecter ms_handle_connect 0x7f0cdc1224f0 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 10 client.66141.objecter resend_mon_ops 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ccc037040 con 0x7f0cdc1224f0 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 10 monclient: dump: 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0ccc038040 con 0x7f0cdc1224f0 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc1224f0 msgr2=0x7f0cdc1228d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc1224f0 0x7f0cdc1228d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0abf80 tx=0x7f0ccc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc1224f0 msgr2=0x7f0cdc1228d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.443+0000 7f0cc3fff640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc1224f0 0x7f0cdc1228d0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0abf80 tx=0x7f0ccc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0ce1a42640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:41.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0ce1a42640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc122e10 0x7f0cdc09e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0ce1a42640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0ce1a42640 1 -- 172.21.15.98:0/2205388785 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0cdc0a8f30 con 0x7f0cdc122e10 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cdaffd640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc122e10 0x7f0cdc09e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cdaffd640 10 monclient: get_auth_request con 0x7f0cdc122e10 auth_method 0 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cdaffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cdaffd640 10 monclient(hunting): handle_auth_done global_id 124224 payload 824 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cdaffd640 1 -- 172.21.15.98:0/2205388785 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0cdc0a3480 con 0x7f0cdc122e10 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cdaffd640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc122e10 0x7f0cdc09e9f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc02b710 tx=0x7f0ccc009190 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cc3fff640 10 client.66141.objecter ms_handle_connect 0x7f0cdc122e10 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cc3fff640 10 client.66141.objecter resend_mon_ops 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ccc037040 con 0x7f0cdc122e10 2024-06-11T19:24:41.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cc3fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.450+0000 7f0cc3fff640 10 monclient: dump: 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.453 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f0ccc030eb0 con 0x7f0cdc122e10 2024-06-11T19:24:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0cc3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0cc3fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc122e10 msgr2=0x7f0cdc09e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0cc3fff640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc122e10 0x7f0cdc09e9f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc02b710 tx=0x7f0ccc009190 comp rx=0 tx=0).stop 2024-06-11T19:24:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0cc3fff640 1 -- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc122e10 msgr2=0x7f0cdc09e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0cc3fff640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0cdc122e10 0x7f0cdc09e9f0 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc02b710 tx=0x7f0ccc009190 comp rx=0 tx=0).stop 2024-06-11T19:24:41.453 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:41.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:41.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:41.454 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:41.454 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:41.454 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 24379, 2024-06-11T19:24:41.454 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:41.454 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:41.454 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:41.454 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:41.454 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:41.454 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.458 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:41.460 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.462 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:41.464 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.466 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.468 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:41.469 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:41.471 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:41.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:41.474 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 10 librados: watch_flush enter 2024-06-11T19:24:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 10 librados: watch_flush exit 2024-06-11T19:24:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 20 client.66141.objecter shutdown clearing up homeless session... 2024-06-11T19:24:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 10 client.66141.objecter successfully canceled tick 2024-06-11T19:24:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 1 -- 172.21.15.98:0/2205388785 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0cc803fde0 msgr2=0x7f0cc80422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0cc803fde0 0x7f0cc80422a0 secure :-1 s=READY pgs=4295 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0a2950 tx=0x7f0cd4003d10 comp rx=0 tx=0).stop 2024-06-11T19:24:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 10 monclient: shutdown 2024-06-11T19:24:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 1 -- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cdc0a4920 msgr2=0x7f0cdc1ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 1 --2- 172.21.15.98:0/2205388785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cdc0a4920 0x7f0cdc1ce130 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0cd000e8d0 tx=0x7f0cd000ed90 comp rx=0 tx=0).stop 2024-06-11T19:24:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0cdb7fe640 1 -- 172.21.15.98:0/2205388785 reap_dead start 2024-06-11T19:24:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0cdb7fe640 1 -- 172.21.15.98:0/2205388785 reap_dead start 2024-06-11T19:24:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 1 -- 172.21.15.98:0/2205388785 shutdown_connections 2024-06-11T19:24:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 1 -- 172.21.15.98:0/2205388785 >> 172.21.15.98:0/2205388785 conn(0x7f0cdc0ab850 msgr2=0x7f0cdc137960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 1 -- 172.21.15.98:0/2205388785 shutdown_connections 2024-06-11T19:24:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 1 -- 172.21.15.98:0/2205388785 wait complete. 2024-06-11T19:24:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.451+0000 7f0ce1a42640 1 librados: shutdown 2024-06-11T19:24:41.477 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:41.478 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:24:41.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:24:41.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.580+0000 7f03571f5640 10 monclient: get_monmap_and_config 2024-06-11T19:24:41.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.580+0000 7f03571f5640 10 monclient: build_initial_monmap 2024-06-11T19:24:41.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.580+0000 7f03571f5640 10 monclient: monmap: 2024-06-11T19:24:41.579 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:41.579 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.579 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:41.581236+0000 2024-06-11T19:24:41.579 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:41.581236+0000 2024-06-11T19:24:41.579 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:41.579 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.579 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:41.580 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:41.580 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:41.580 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:41.580 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:41.580 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:41.580 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:41.580 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:41.580 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:41.580 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:41.580 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:41.580 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.580+0000 7f03571f5640 1 Processor -- start 2024-06-11T19:24:41.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 1 -- start start 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 10 monclient: init 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 10 monclient: _add_conns ranks=[4,1,6,9,7,0,10,5,2,8,3] 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f03500a4920 0x7f03500a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 10 monclient(hunting): picked mon.noname-c con 0x7f03500a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f03500a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03500a2530 0x7f03500a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 10 monclient(hunting): picked mon.noname-d con 0x7f03500a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03500a7140 con 0x7f03500a5240 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f03500a2b50 con 0x7f03500a4920 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f03500a2e30 con 0x7f03500a2530 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f03571f5640 10 monclient(hunting): authenticate will time out at 1691.388614s 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f03500a4920 0x7f03500a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0355792640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03500a2530 0x7f03500a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f03500a4920 0x7f03500a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46934/0 (socket says 172.21.15.98:46934) 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 1 -- 172.21.15.98:0/3582664278 learned_addr learned my addr 172.21.15.98:0/3582664278 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 10 monclient(hunting): get_auth_request con 0x7f03500a4920 auth_method 0 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0355792640 10 monclient(hunting): get_auth_request con 0x7f03500a2530 auth_method 0 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0355792640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0355792640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0355792640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0355792640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0355792640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0355792640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 10 monclient(hunting): handle_auth_done global_id 114466 payload 274 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 1 -- 172.21.15.98:0/3582664278 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03500a2530 msgr2=0x7f03500a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 1 --2- 172.21.15.98:0/3582664278 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03500a2530 0x7f03500a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.581+0000 7f0354f91640 1 -- 172.21.15.98:0/3582664278 >> v1:172.21.15.117:6789/0 conn(0x7f03500a5240 legacy=0x7f03500a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f0354f91640 10 monclient: _finish_hunting 0 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f0354f91640 1 monclient: found mon.noname-c 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f0354f91640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f0354f91640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f0354f91640 1 -- 172.21.15.98:0/3582664278 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03500a3110 con 0x7f03500a4920 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f0355792640 1 --2- 172.21.15.98:0/3582664278 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03500a2530 0x7f03500a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f0354f91640 1 --2- 172.21.15.98:0/3582664278 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f03500a4920 0x7f03500a4d00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f03400098a0 tx=0x7f034002b7a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=8698a1ce02ed8527 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 1 -- 172.21.15.98:0/3582664278 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0340038070 con 0x7f03500a4920 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 10 monclient: dump: 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f03571f5640 5 monclient: authenticate success, global_id 114466 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 1 -- 172.21.15.98:0/3582664278 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0340031970 con 0x7f03500a4920 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f03571f5640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f03571f5640 4 set_mon_vals no callback set 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 1 -- 172.21.15.98:0/3582664278 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0340031ba0 con 0x7f03500a4920 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.582+0000 7f034f7fe640 10 monclient: dump: 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.583+0000 7f03571f5640 10 monclient: shutdown 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.583+0000 7f03571f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.583+0000 7f03571f5640 1 -- 172.21.15.98:0/3582664278 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f03500a4920 msgr2=0x7f03500a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.583+0000 7f03571f5640 1 --2- 172.21.15.98:0/3582664278 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f03500a4920 0x7f03500a4d00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f03400098a0 tx=0x7f034002b7a0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.583+0000 7f03571f5640 1 -- 172.21.15.98:0/3582664278 shutdown_connections 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.583+0000 7f03571f5640 1 --2- 172.21.15.98:0/3582664278 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03500a2530 0x7f03500a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.583+0000 7f03571f5640 1 --2- 172.21.15.98:0/3582664278 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f03500a4920 0x7f03500a4d00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.583+0000 7f03571f5640 1 -- 172.21.15.98:0/3582664278 >> 172.21.15.98:0/3582664278 conn(0x7f03500ab850 msgr2=0x7f03500a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.583+0000 7f03571f5640 1 -- 172.21.15.98:0/3582664278 shutdown_connections 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.583+0000 7f03571f5640 1 -- 172.21.15.98:0/3582664278 wait complete. 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.584+0000 7f03571f5640 10 monclient: build_initial_monmap 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.584+0000 7f03571f5640 10 monclient: monmap: 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.584+0000 7f03571f5640 1 librados: starting msgr at 2024-06-11T19:24:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.584+0000 7f03571f5640 1 librados: starting objecter 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.584+0000 7f03571f5640 1 Processor -- start 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.584+0000 7f03571f5640 1 -- start start 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.584+0000 7f03571f5640 1 librados: setting wanted keys 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.584+0000 7f03571f5640 1 librados: calling monclient init 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.584+0000 7f03571f5640 10 monclient: init 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient: _add_conns ranks=[5,2,3,1,4,6,8,0,7] 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f03500a2530 0x7f0350136da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient(hunting): picked mon.noname-h con 0x7f03500a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03500a4920 0x7f03501372e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient(hunting): picked mon.noname-b con 0x7f03500a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03500a5240 0x7f0350122840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient(hunting): picked mon.noname-g con 0x7f03500a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f03500a6da0 con 0x7f03500a4920 2024-06-11T19:24:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f0350138e50 con 0x7f03500a5240 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f03500a65f0 con 0x7f03500a2530 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient(hunting): authenticate will time out at 1691.392082s 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03500a4920 0x7f03501372e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03500a4920 0x7f03501372e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57358/0 (socket says 172.21.15.98:57358) 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 1 -- 172.21.15.98:0/1633690154 learned_addr learned my addr 172.21.15.98:0/1633690154 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 10 monclient(hunting): get_auth_request con 0x7f03500a4920 auth_method 0 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f0355792640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03500a5240 0x7f0350122840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f0354f91640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f03500a2530 0x7f0350136da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f0355792640 10 monclient(hunting): get_auth_request con 0x7f03500a5240 auth_method 0 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f0355792640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f0355792640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f0355792640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f0354f91640 10 monclient(hunting): get_auth_request con 0x7f03500a2530 auth_method 0 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f0354f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f0354f91640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f0354f91640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 10 monclient(hunting): handle_auth_done global_id 85205 payload 824 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 1 -- 172.21.15.98:0/1633690154 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f03500a2530 msgr2=0x7f0350136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f03500a2530 0x7f0350136da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 1 -- 172.21.15.98:0/1633690154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03500a5240 msgr2=0x7f0350122840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03500a5240 0x7f0350122840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 1 monclient: found mon.noname-b 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 1 -- 172.21.15.98:0/1633690154 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03500a9420 con 0x7f03500a4920 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034ffff640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03500a4920 0x7f03501372e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f034400b3e0 tx=0x7f034400b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 10 client.?.objecter ms_handle_connect 0x7f03500a4920 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0344015020 con 0x7f03500a4920 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 10 monclient: dump: 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 10 monclient: _finish_auth 0 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0344011aa0 con 0x7f03500a4920 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 5 monclient: authenticate success, global_id 85205 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient: _renew_subs 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 1 -- 172.21.15.98:0/1633690154 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03500a8f10 con 0x7f03500a4920 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0344011c50 con 0x7f03500a4920 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034d7fa640 10 monclient: dump: 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 client.85205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f034effd640 4 set_mon_vals no callback set 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient: _renew_subs 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:24:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 1 -- 172.21.15.98:0/1633690154 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0350126780 con 0x7f03500a4920 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.585+0000 7f03571f5640 1 librados: init done 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f03571f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f0344018020 con 0x7f03500a4920 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f03571f5640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0350122d80 0x7f0350123160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f03571f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f03571f5640 1 -- 172.21.15.98:0/1633690154 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0350050830 con 0x7f0350122d80 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f0354f91640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0350122d80 0x7f0350123160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f0354f91640 10 monclient: get_auth_request con 0x7f0350122d80 auth_method 0 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f0354f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f0354f91640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f0354f91640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f034d7fa640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f033c03fdb0 0x7f033c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f0354f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f0354f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f0354f91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(626..626 src has 1..626) ==== 6797+0+0 (secure 0 0 0) 0x7f0344051c20 con 0x7f03500a4920 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f034d7fa640 10 client.85205.objecter ms_dispatch 0x7f0350000b90 osd_map(626..626 src has 1..626) 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f034d7fa640 3 client.85205.objecter handle_osd_map got epochs [626,626] > 0 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.586+0000 7f034d7fa640 3 client.85205.objecter handle_osd_map decoding full epoch 626 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 20 client.85205.objecter dump_active .. 0 homeless 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f0354f91640 10 monclient(hunting): handle_auth_done global_id 124199 payload 824 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f0355792640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f033c03fdb0 0x7f033c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f0354f91640 1 -- 172.21.15.98:0/1633690154 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f03500aabc0 con 0x7f0350122d80 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f0355792640 10 monclient: get_auth_request con 0x7f033c03fdb0 auth_method 0 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 10 client.85205.objecter ms_handle_connect 0x7f0350122d80 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 10 client.85205.objecter resend_mon_ops 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f0354f91640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0350122d80 0x7f0350123160 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f03500a8420 tx=0x7f03400046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0340043070 con 0x7f0350122d80 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 10 monclient: dump: 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.607 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 10 client.85205.objecter ms_handle_connect 0x7f033c03fdb0 2024-06-11T19:24:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f0355792640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f033c03fdb0 0x7f033c042270 secure :-1 s=READY pgs=4296 cs=0 l=1 rev1=1 crypto rx=0x7f0350123f20 tx=0x7f034800b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0340031ea0 con 0x7f0350122d80 2024-06-11T19:24:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0350122d80 msgr2=0x7f0350123160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0350122d80 0x7f0350123160 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f03500a8420 tx=0x7f03400046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0350122d80 msgr2=0x7f0350123160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.587+0000 7f034d7fa640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0350122d80 0x7f0350123160 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f03500a8420 tx=0x7f03400046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f03571f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f03571f5640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f03501236a0 0x7f035009e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f03571f5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f03571f5640 1 -- 172.21.15.98:0/1633690154 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0350126590 con 0x7f03501236a0 2024-06-11T19:24:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f0354f91640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f03501236a0 0x7f035009e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f0354f91640 10 monclient: get_auth_request con 0x7f03501236a0 auth_method 0 2024-06-11T19:24:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f0354f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f0354f91640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f0354f91640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f0354f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f0354f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f0354f91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f0354f91640 10 monclient(hunting): handle_auth_done global_id 124208 payload 824 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f0354f91640 1 -- 172.21.15.98:0/1633690154 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0350124d40 con 0x7f03501236a0 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f0354f91640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f03501236a0 0x7f035009e9f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f03400097c0 tx=0x7f0340031000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 10 client.85205.objecter ms_handle_connect 0x7f03501236a0 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 10 client.85205.objecter resend_mon_ops 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0340038040 con 0x7f03501236a0 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 10 monclient: dump: 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f0340003d20 con 0x7f03501236a0 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f03501236a0 msgr2=0x7f035009e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f03501236a0 0x7f035009e9f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f03400097c0 tx=0x7f0340031000 comp rx=0 tx=0).stop 2024-06-11T19:24:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 1 -- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f03501236a0 msgr2=0x7f035009e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.594+0000 7f034d7fa640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f03501236a0 0x7f035009e9f0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f03400097c0 tx=0x7f0340031000 comp rx=0 tx=0).stop 2024-06-11T19:24:41.612 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 24531, 2024-06-11T19:24:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:41.612 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.613 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:41.615 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.616 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:41.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.620 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:41.632 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:41.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 10 librados: watch_flush enter 2024-06-11T19:24:41.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 10 librados: watch_flush exit 2024-06-11T19:24:41.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 20 client.85205.objecter shutdown clearing up homeless session... 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 10 client.85205.objecter successfully canceled tick 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 1 -- 172.21.15.98:0/1633690154 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f033c03fdb0 msgr2=0x7f033c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f033c03fdb0 0x7f033c042270 secure :-1 s=READY pgs=4296 cs=0 l=1 rev1=1 crypto rx=0x7f0350123f20 tx=0x7f034800b040 comp rx=0 tx=0).stop 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 10 monclient: shutdown 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 1 -- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03500a4920 msgr2=0x7f03501372e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 1 --2- 172.21.15.98:0/1633690154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f03500a4920 0x7f03501372e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f034400b3e0 tx=0x7f034400b8a0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f0355792640 1 -- 172.21.15.98:0/1633690154 reap_dead start 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f0355792640 1 -- 172.21.15.98:0/1633690154 reap_dead start 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 1 -- 172.21.15.98:0/1633690154 shutdown_connections 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 1 -- 172.21.15.98:0/1633690154 >> 172.21.15.98:0/1633690154 conn(0x7f03500ab850 msgr2=0x7f0350138150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 1 -- 172.21.15.98:0/1633690154 shutdown_connections 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 1 -- 172.21.15.98:0/1633690154 wait complete. 2024-06-11T19:24:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.595+0000 7f03571f5640 1 librados: shutdown 2024-06-11T19:24:41.635 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:41.635 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:24:41.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:24:41.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.735+0000 7f8d4ff66640 10 monclient: get_monmap_and_config 2024-06-11T19:24:41.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.735+0000 7f8d4ff66640 10 monclient: build_initial_monmap 2024-06-11T19:24:41.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.735+0000 7f8d4ff66640 10 monclient: monmap: 2024-06-11T19:24:41.734 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:41.735 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.735 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:41.736389+0000 2024-06-11T19:24:41.735 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:41.736389+0000 2024-06-11T19:24:41.735 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:41.735 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.735 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:41.735 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:41.735 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:41.735 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:41.735 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:41.735 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:41.736 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:41.736 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:41.736 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:41.736 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:41.736 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:41.736 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.735+0000 7f8d4ff66640 1 Processor -- start 2024-06-11T19:24:41.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 1 -- start start 2024-06-11T19:24:41.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 10 monclient: init 2024-06-11T19:24:41.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:41.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 10 monclient: _add_conns ranks=[3,7,9,10,5,2,6,1,4,8,0] 2024-06-11T19:24:41.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d480a4920 0x7f8d480a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 10 monclient(hunting): picked mon.noname-g con 0x7f8d480a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:41.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d480a5240 0x7f8d480a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 10 monclient(hunting): picked mon.noname-i con 0x7f8d480a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:41.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d480a2530 0x7f8d480a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 10 monclient(hunting): picked mon.noname-a con 0x7f8d480a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:24:41.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8d480a7140 con 0x7f8d480a2530 2024-06-11T19:24:41.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f8d480a2b50 con 0x7f8d480a4920 2024-06-11T19:24:41.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8d480a2e30 con 0x7f8d480a5240 2024-06-11T19:24:41.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4ff66640 10 monclient(hunting): authenticate will time out at 1691.543666s 2024-06-11T19:24:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4e503640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d480a2530 0x7f8d480a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4e503640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d480a2530 0x7f8d480a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53728/0 (socket says 172.21.15.98:53728) 2024-06-11T19:24:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4e503640 1 -- 172.21.15.98:0/4115904045 learned_addr learned my addr 172.21.15.98:0/4115904045 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4e503640 10 monclient(hunting): get_auth_request con 0x7f8d480a2530 auth_method 0 2024-06-11T19:24:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4e503640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4e503640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4e503640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4e503640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4e503640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.736+0000 7f8d4e503640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4dd02640 1 --2- 172.21.15.98:0/4115904045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d480a4920 0x7f8d480a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4e503640 10 monclient(hunting): handle_auth_done global_id 66150 payload 274 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4e503640 1 -- 172.21.15.98:0/4115904045 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d480a5240 msgr2=0x7f8d480a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4e503640 1 --2- 172.21.15.98:0/4115904045 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d480a5240 0x7f8d480a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4e503640 1 -- 172.21.15.98:0/4115904045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d480a4920 msgr2=0x7f8d480a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4e503640 1 --2- 172.21.15.98:0/4115904045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d480a4920 0x7f8d480a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4e503640 10 monclient: _finish_hunting 0 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4e503640 1 monclient: found mon.noname-a 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4e503640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4e503640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4e503640 1 -- 172.21.15.98:0/4115904045 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d480a3110 con 0x7f8d480a2530 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4e503640 1 --2- 172.21.15.98:0/4115904045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d480a2530 0x7f8d480a5ee0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009e50 tx=0x7f8d4002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1801529a1da708ca server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4cd00640 1 -- 172.21.15.98:0/4115904045 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d4002cbd0 con 0x7f8d480a2530 2024-06-11T19:24:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4cd00640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4cd00640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4cd00640 10 monclient: dump: 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4cd00640 10 monclient: _finish_auth 0 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4cd00640 10 monclient: _check_auth_tickets 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4cd00640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4cd00640 1 -- 172.21.15.98:0/4115904045 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d4002cd90 con 0x7f8d480a2530 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4ff66640 5 monclient: authenticate success, global_id 66150 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4ff66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4cd00640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4ff66640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.737+0000 7f8d4cd00640 1 -- 172.21.15.98:0/4115904045 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d40034860 con 0x7f8d480a2530 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4ff66640 4 set_mon_vals no callback set 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4cd00640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4cd00640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4cd00640 10 monclient: dump: 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4ff66640 10 monclient: shutdown 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4ff66640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4ff66640 1 -- 172.21.15.98:0/4115904045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d480a2530 msgr2=0x7f8d480a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4ff66640 1 --2- 172.21.15.98:0/4115904045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d480a2530 0x7f8d480a5ee0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009e50 tx=0x7f8d4002bdc0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4ff66640 1 -- 172.21.15.98:0/4115904045 shutdown_connections 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4ff66640 1 --2- 172.21.15.98:0/4115904045 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d480a5240 0x7f8d480a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4ff66640 1 --2- 172.21.15.98:0/4115904045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d480a2530 0x7f8d480a5ee0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4ff66640 1 --2- 172.21.15.98:0/4115904045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d480a4920 0x7f8d480a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4ff66640 1 -- 172.21.15.98:0/4115904045 >> 172.21.15.98:0/4115904045 conn(0x7f8d480ab850 msgr2=0x7f8d480a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4ff66640 1 -- 172.21.15.98:0/4115904045 shutdown_connections 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.738+0000 7f8d4ff66640 1 -- 172.21.15.98:0/4115904045 wait complete. 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient: build_initial_monmap 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient: monmap: 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:41.754 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 1 librados: starting msgr at 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 1 librados: starting objecter 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 1 Processor -- start 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 1 -- start start 2024-06-11T19:24:41.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 1 librados: setting wanted keys 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 1 librados: calling monclient init 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient: init 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient: _add_conns ranks=[0,7,6,4,2,8,3,5,1] 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d480a2530 0x7f8d481cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient(hunting): picked mon.noname-a con 0x7f8d480a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d480a4920 0x7f8d481ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient(hunting): picked mon.noname-i con 0x7f8d480a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8d480a5240 0x7f8d48122f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient(hunting): picked mon.noname-d con 0x7f8d480a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8d480a6da0 con 0x7f8d480a2530 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8d48138e50 con 0x7f8d480a5240 2024-06-11T19:24:41.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8d480a6420 con 0x7f8d480a4920 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.739+0000 7f8d4ff66640 10 monclient(hunting): authenticate will time out at 1691.546975s 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d480a2530 0x7f8d481cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d480a2530 0x7f8d481cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53734/0 (socket says 172.21.15.98:53734) 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 1 -- 172.21.15.98:0/3257118662 learned_addr learned my addr 172.21.15.98:0/3257118662 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4e503640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8d480a5240 0x7f8d48122f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 10 monclient(hunting): get_auth_request con 0x7f8d480a2530 auth_method 0 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4e503640 10 monclient(hunting): get_auth_request con 0x7f8d480a5240 auth_method 0 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4e503640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4e503640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4e503640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4d501640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d480a4920 0x7f8d481ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4e503640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4e503640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4e503640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4d501640 10 monclient(hunting): get_auth_request con 0x7f8d480a4920 auth_method 0 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4d501640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4d501640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4d501640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4d501640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4d501640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4d501640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 10 monclient(hunting): handle_auth_done global_id 66159 payload 824 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 1 -- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d480a4920 msgr2=0x7f8d481ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8d480a4920 0x7f8d481ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 1 -- 172.21.15.98:0/3257118662 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8d480a5240 msgr2=0x7f8d48122f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8d480a5240 0x7f8d48122f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 10 monclient: _finish_hunting 0 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 1 monclient: found mon.noname-a 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 1 -- 172.21.15.98:0/3257118662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d48126c20 con 0x7f8d480a2530 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4e503640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8d480a5240 0x7f8d48122f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4dd02640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d480a2530 0x7f8d481cdb40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400c7f0 tx=0x7f8d4400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 10 client.?.objecter ms_handle_connect 0x7f8d480a2530 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d44015070 con 0x7f8d480a2530 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 10 monclient: dump: 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4ff66640 5 monclient: authenticate success, global_id 66159 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4ff66640 10 monclient: _renew_subs 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d44007a90 con 0x7f8d480a2530 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4ff66640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4ff66640 1 -- 172.21.15.98:0/3257118662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d48126f00 con 0x7f8d480a2530 2024-06-11T19:24:41.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d44007c40 con 0x7f8d480a2530 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3a7fc640 10 monclient: dump: 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.762 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d3bfff640 4 set_mon_vals no callback set 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4ff66640 10 client.66159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4ff66640 10 monclient: _renew_subs 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4ff66640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4ff66640 1 -- 172.21.15.98:0/3257118662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d48127930 con 0x7f8d480a2530 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.740+0000 7f8d4ff66640 1 librados: init done 2024-06-11T19:24:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d4ff66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d4ff66640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d481234c0 0x7f8d481238a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d4ff66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8d440075c0 con 0x7f8d480a2530 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d4ff66640 1 -- 172.21.15.98:0/3257118662 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8d48050830 con 0x7f8d481234c0 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d3a7fc640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8d2c03fde0 0x7f8d2c0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d4d501640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d481234c0 0x7f8d481238a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d4d501640 10 monclient: get_auth_request con 0x7f8d481234c0 auth_method 0 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d4d501640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d4d501640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d4d501640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(626..626 src has 1..626) ==== 6797+0+0 (secure 0 0 0) 0x7f8d44051890 con 0x7f8d480a2530 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d3a7fc640 10 client.66159.objecter ms_dispatch 0x7f8d48000b90 osd_map(626..626 src has 1..626) 2024-06-11T19:24:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d3a7fc640 3 client.66159.objecter handle_osd_map got epochs [626,626] > 0 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d3a7fc640 3 client.66159.objecter handle_osd_map decoding full epoch 626 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d4d501640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d4d501640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d4d501640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.741+0000 7f8d3a7fc640 20 client.66159.objecter dump_active .. 0 homeless 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d4e503640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8d2c03fde0 0x7f8d2c0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d4e503640 10 monclient: get_auth_request con 0x7f8d2c03fde0 auth_method 0 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d4d501640 10 monclient(hunting): handle_auth_done global_id 124192 payload 824 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d4d501640 1 -- 172.21.15.98:0/3257118662 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d480a1f10 con 0x7f8d481234c0 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 10 client.66159.objecter ms_handle_connect 0x7f8d481234c0 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d4d501640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d481234c0 0x7f8d481238a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8d48073fc0 tx=0x7f8d3c00e9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 10 client.66159.objecter resend_mon_ops 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 10 client.66159.objecter ms_handle_connect 0x7f8d2c03fde0 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d4e503640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8d2c03fde0 0x7f8d2c0422a0 secure :-1 s=READY pgs=4297 cs=0 l=1 rev1=1 crypto rx=0x7f8d40035000 tx=0x7f8d40030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d3c017070 con 0x7f8d481234c0 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 10 monclient: dump: 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.766 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.767 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.767 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.767 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.767 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.767 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8d3c013680 con 0x7f8d481234c0 2024-06-11T19:24:41.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:41.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:41.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d481234c0 msgr2=0x7f8d481238a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d481234c0 0x7f8d481238a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8d48073fc0 tx=0x7f8d3c00e9f0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d481234c0 msgr2=0x7f8d481238a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:41.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.742+0000 7f8d3a7fc640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d481234c0 0x7f8d481238a0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8d48073fc0 tx=0x7f8d3c00e9f0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4ff66640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:41.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4ff66640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d48123de0 0x7f8d4809e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4ff66640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4ff66640 1 -- 172.21.15.98:0/3257118662 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8d480a4d00 con 0x7f8d48123de0 2024-06-11T19:24:41.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4d501640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d48123de0 0x7f8d4809e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4d501640 10 monclient: get_auth_request con 0x7f8d48123de0 auth_method 0 2024-06-11T19:24:41.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4d501640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4d501640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4d501640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4d501640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4d501640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4d501640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4d501640 10 monclient(hunting): handle_auth_done global_id 124201 payload 824 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.749+0000 7f8d4d501640 1 -- 172.21.15.98:0/3257118662 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8d4806c300 con 0x7f8d48123de0 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4d501640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d48123de0 0x7f8d4809e9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8d48073fc0 tx=0x7f8d3c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 10 client.66159.objecter ms_handle_connect 0x7f8d48123de0 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 10 client.66159.objecter resend_mon_ops 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d3c017070 con 0x7f8d48123de0 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 10 monclient: dump: 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.770 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f8d3c006310 con 0x7f8d48123de0 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:41.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:41.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d48123de0 msgr2=0x7f8d4809e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d48123de0 0x7f8d4809e9f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8d48073fc0 tx=0x7f8d3c004730 comp rx=0 tx=0).stop 2024-06-11T19:24:41.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 1 -- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d48123de0 msgr2=0x7f8d4809e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:41.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d3a7fc640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d48123de0 0x7f8d4809e9f0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8d48073fc0 tx=0x7f8d3c004730 comp rx=0 tx=0).stop 2024-06-11T19:24:41.772 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:41.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:41.772 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:41.772 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:41.772 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:41.772 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 24683, 2024-06-11T19:24:41.772 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:41.773 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.774 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:41.775 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:41.776 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:41.776 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:41.776 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:41.776 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:41.776 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:41.776 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:41.776 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:41.776 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:41.776 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:41.776 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:41.776 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.777 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:41.778 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.780 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.782 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:41.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.787 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:41.789 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:41.790 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.792 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.794 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:41.794 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:41.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 10 librados: watch_flush enter 2024-06-11T19:24:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 10 librados: watch_flush exit 2024-06-11T19:24:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 20 client.66159.objecter shutdown clearing up homeless session... 2024-06-11T19:24:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 10 client.66159.objecter successfully canceled tick 2024-06-11T19:24:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 1 -- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8d2c03fde0 msgr2=0x7f8d2c0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8d2c03fde0 0x7f8d2c0422a0 secure :-1 s=READY pgs=4297 cs=0 l=1 rev1=1 crypto rx=0x7f8d40035000 tx=0x7f8d40030000 comp rx=0 tx=0).stop 2024-06-11T19:24:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 10 monclient: shutdown 2024-06-11T19:24:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 1 -- 172.21.15.98:0/3257118662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d480a2530 msgr2=0x7f8d481cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 1 --2- 172.21.15.98:0/3257118662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d480a2530 0x7f8d481cdb40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400c7f0 tx=0x7f8d4400ccb0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4e503640 1 -- 172.21.15.98:0/3257118662 reap_dead start 2024-06-11T19:24:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4e503640 1 -- 172.21.15.98:0/3257118662 reap_dead start 2024-06-11T19:24:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 1 -- 172.21.15.98:0/3257118662 shutdown_connections 2024-06-11T19:24:41.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 1 -- 172.21.15.98:0/3257118662 >> 172.21.15.98:0/3257118662 conn(0x7f8d480ab850 msgr2=0x7f8d480a9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:41.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.750+0000 7f8d4ff66640 1 -- 172.21.15.98:0/3257118662 shutdown_connections 2024-06-11T19:24:41.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.751+0000 7f8d4ff66640 1 -- 172.21.15.98:0/3257118662 wait complete. 2024-06-11T19:24:41.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.751+0000 7f8d4ff66640 1 librados: shutdown 2024-06-11T19:24:41.797 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:41.797 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:24:41.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:24:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.896+0000 7fdeba0e1640 10 monclient: get_monmap_and_config 2024-06-11T19:24:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.896+0000 7fdeba0e1640 10 monclient: build_initial_monmap 2024-06-11T19:24:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.897+0000 7fdeba0e1640 10 monclient: monmap: 2024-06-11T19:24:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:41.898008+0000 2024-06-11T19:24:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:41.898008+0000 2024-06-11T19:24:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:41.897 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:41.897 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:41.897 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:41.897 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:41.897 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:41.897 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:41.897 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:41.897 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:41.897 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:41.897 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:41.897 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.897+0000 7fdeba0e1640 1 Processor -- start 2024-06-11T19:24:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.897+0000 7fdeba0e1640 1 -- start start 2024-06-11T19:24:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.897+0000 7fdeba0e1640 10 monclient: init 2024-06-11T19:24:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 10 monclient: _add_conns ranks=[10,9,4,6,1,8,7,5,3,2,0] 2024-06-11T19:24:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdeb40a4920 0x7fdeb40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 10 monclient(hunting): picked mon.noname-f con 0x7fdeb40a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:24:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdeb40a5240 0x7fdeb40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 10 monclient(hunting): picked mon.noname-a con 0x7fdeb40a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:24:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdeb40a2530 0x7fdeb40a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 10 monclient(hunting): picked mon.noname-c con 0x7fdeb40a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdeb40a7140 con 0x7fdeb40a5240 2024-06-11T19:24:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fdeb40a2b50 con 0x7fdeb40a4920 2024-06-11T19:24:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fdeb40a2e30 con 0x7fdeb40a2530 2024-06-11T19:24:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeba0e1640 10 monclient(hunting): authenticate will time out at 1691.705333s 2024-06-11T19:24:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb2ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdeb40a5240 0x7fdeb40a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdeb40a2530 0x7fdeb40a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb2ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdeb40a5240 0x7fdeb40a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53740/0 (socket says 172.21.15.98:53740) 2024-06-11T19:24:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb2ffd640 1 -- 172.21.15.98:0/1766516408 learned_addr learned my addr 172.21.15.98:0/1766516408 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb2ffd640 10 monclient(hunting): get_auth_request con 0x7fdeb40a5240 auth_method 0 2024-06-11T19:24:41.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdeb40a2530 0x7fdeb40a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46940/0 (socket says 172.21.15.98:46940) 2024-06-11T19:24:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 10 monclient(hunting): get_auth_request con 0x7fdeb40a2530 auth_method 0 2024-06-11T19:24:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb37fe640 1 --2- 172.21.15.98:0/1766516408 >> v2:172.21.15.117:3300/0 conn(0x7fdeb40a4920 0x7fdeb40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb37fe640 10 monclient(hunting): get_auth_request con 0x7fdeb40a4920 auth_method 0 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 10 monclient(hunting): handle_auth_done global_id 114475 payload 274 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 1 -- 172.21.15.98:0/1766516408 >> v2:172.21.15.117:3300/0 conn(0x7fdeb40a4920 msgr2=0x7fdeb40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 1 --2- 172.21.15.98:0/1766516408 >> v2:172.21.15.117:3300/0 conn(0x7fdeb40a4920 0x7fdeb40a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 1 -- 172.21.15.98:0/1766516408 >> v2:172.21.15.98:3300/0 conn(0x7fdeb40a5240 msgr2=0x7fdeb40a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 1 --2- 172.21.15.98:0/1766516408 >> v2:172.21.15.98:3300/0 conn(0x7fdeb40a5240 0x7fdeb40a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 1 monclient: found mon.noname-c 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb3fff640 1 -- 172.21.15.98:0/1766516408 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb40a3110 con 0x7fdeb40a2530 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.898+0000 7fdeb2ffd640 1 --2- 172.21.15.98:0/1766516408 >> v2:172.21.15.98:3300/0 conn(0x7fdeb40a5240 0x7fdeb40a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb3fff640 1 --2- 172.21.15.98:0/1766516408 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdeb40a2530 0x7fdeb40a5ee0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4073de0 tx=0x7fdeac02b710 comp rx=0 tx=0).ready entity=mon.4 client_cookie=510e4ed4407ae0ab server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 1 -- 172.21.15.98:0/1766516408 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdeac037070 con 0x7fdeb40a2530 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 10 monclient: dump: 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 10 monclient: _finish_auth 0 2024-06-11T19:24:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 10 monclient: _check_auth_tickets 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 1 -- 172.21.15.98:0/1766516408 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeac030980 con 0x7fdeb40a2530 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeba0e1640 5 monclient: authenticate success, global_id 114475 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeba0e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeba0e1640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 1 -- 172.21.15.98:0/1766516408 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdeac0313a0 con 0x7fdeb40a2530 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeba0e1640 4 set_mon_vals no callback set 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.899+0000 7fdeb27fc640 10 monclient: dump: 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.913 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 10 monclient: shutdown 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:41.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 1 -- 172.21.15.98:0/1766516408 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdeb40a2530 msgr2=0x7fdeb40a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 1 --2- 172.21.15.98:0/1766516408 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdeb40a2530 0x7fdeb40a5ee0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4073de0 tx=0x7fdeac02b710 comp rx=0 tx=0).stop 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 1 -- 172.21.15.98:0/1766516408 shutdown_connections 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 1 --2- 172.21.15.98:0/1766516408 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdeb40a2530 0x7fdeb40a5ee0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 1 --2- 172.21.15.98:0/1766516408 >> v2:172.21.15.98:3300/0 conn(0x7fdeb40a5240 0x7fdeb40a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 1 --2- 172.21.15.98:0/1766516408 >> v2:172.21.15.117:3300/0 conn(0x7fdeb40a4920 0x7fdeb40a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 1 -- 172.21.15.98:0/1766516408 >> 172.21.15.98:0/1766516408 conn(0x7fdeb40ab850 msgr2=0x7fdeb40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 1 -- 172.21.15.98:0/1766516408 shutdown_connections 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 1 -- 172.21.15.98:0/1766516408 wait complete. 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 10 monclient: build_initial_monmap 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.900+0000 7fdeba0e1640 10 monclient: monmap: 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:41.915 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 1 librados: starting msgr at 2024-06-11T19:24:41.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 1 librados: starting objecter 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 1 Processor -- start 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 1 -- start start 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 1 librados: setting wanted keys 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 1 librados: calling monclient init 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 10 monclient: init 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 10 monclient: _add_conns ranks=[0,7,5,1,3,2,4,8,6] 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdeb40a2530 0x7fdeb41cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 10 monclient(hunting): picked mon.noname-a con 0x7fdeb40a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdeb40a4920 0x7fdeb41cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 10 monclient(hunting): picked mon.noname-i con 0x7fdeb40a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdeb40a5240 0x7fdeb4125050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 10 monclient(hunting): picked mon.noname-h con 0x7fdeb40a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb40a6da0 con 0x7fdeb40a2530 2024-06-11T19:24:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fdeb4138e50 con 0x7fdeb40a5240 2024-06-11T19:24:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fdeb40a6420 con 0x7fdeb40a4920 2024-06-11T19:24:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeba0e1640 10 monclient(hunting): authenticate will time out at 1691.708846s 2024-06-11T19:24:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeb37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdeb40a2530 0x7fdeb41cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeb37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdeb40a2530 0x7fdeb41cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53756/0 (socket says 172.21.15.98:53756) 2024-06-11T19:24:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeb37fe640 1 -- 172.21.15.98:0/2257388136 learned_addr learned my addr 172.21.15.98:0/2257388136 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeb37fe640 10 monclient(hunting): get_auth_request con 0x7fdeb40a2530 auth_method 0 2024-06-11T19:24:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeb37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.901+0000 7fdeb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeb37fe640 10 monclient(hunting): handle_auth_done global_id 66177 payload 824 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeb37fe640 1 -- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdeb40a4920 msgr2=0x7fdeb41cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeb37fe640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdeb40a4920 0x7fdeb41cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeb37fe640 1 -- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdeb40a5240 msgr2=0x7fdeb4125050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeb37fe640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdeb40a5240 0x7fdeb4125050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeb37fe640 10 monclient: _finish_hunting 0 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeb37fe640 1 monclient: found mon.noname-a 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeb37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeb37fe640 1 -- 172.21.15.98:0/2257388136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb41271c0 con 0x7fdeb40a2530 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeb37fe640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdeb40a2530 0x7fdeb41cdaa0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fdea4007ef0 tx=0x7fdea400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 10 client.?.objecter ms_handle_connect 0x7fdeb40a2530 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdea4014890 con 0x7fdeb40a2530 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 10 monclient: dump: 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.920 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 10 monclient: _finish_auth 0 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdea4019070 con 0x7fdeb40a2530 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeba0e1640 5 monclient: authenticate success, global_id 66177 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:41.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdea4013380 con 0x7fdeb40a2530 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeba0e1640 10 monclient: _renew_subs 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeba0e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeba0e1640 1 -- 172.21.15.98:0/2257388136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb4127880 con 0x7fdeb40a2530 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeb1ffb640 4 set_mon_vals no callback set 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeba0e1640 10 client.66177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fde9bfff640 10 monclient: dump: 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.922 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeba0e1640 10 monclient: _renew_subs 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeba0e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeba0e1640 1 -- 172.21.15.98:0/2257388136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb41387f0 con 0x7fdeb40a2530 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.902+0000 7fdeba0e1640 1 librados: init done 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeba0e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeba0e1640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4128770 0x7fdeb4125590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeba0e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeba0e1640 1 -- 172.21.15.98:0/2257388136 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb4050830 con 0x7fdeb4128770 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fdea4014420 con 0x7fdeb40a2530 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeb2ffd640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4128770 0x7fdeb4125590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeb2ffd640 10 monclient: get_auth_request con 0x7fdeb4128770 auth_method 0 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeb2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fde9bfff640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdea003fde0 0x7fdea00422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(626..626 src has 1..626) ==== 6797+0+0 (secure 0 0 0) 0x7fdea4052530 con 0x7fdeb40a2530 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fde9bfff640 10 client.66177.objecter ms_dispatch 0x7fdeb4000b90 osd_map(626..626 src has 1..626) 2024-06-11T19:24:41.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fde9bfff640 3 client.66177.objecter handle_osd_map got epochs [626,626] > 0 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fde9bfff640 3 client.66177.objecter handle_osd_map decoding full epoch 626 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fde9bfff640 20 client.66177.objecter dump_active .. 0 homeless 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeb2ffd640 10 monclient(hunting): handle_auth_done global_id 114384 payload 824 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeb2ffd640 1 -- 172.21.15.98:0/2257388136 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdeb4128b50 con 0x7fdeb4128770 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeb3fff640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdea003fde0 0x7fdea00422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.903+0000 7fdeb3fff640 10 monclient: get_auth_request con 0x7fdea003fde0 auth_method 0 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 10 client.66177.objecter ms_handle_connect 0x7fdeb4128770 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 10 client.66177.objecter resend_mon_ops 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fdeb2ffd640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4128770 0x7fdeb4125590 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdea800e770 tx=0x7fdea800ec30 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdea8017070 con 0x7fdeb4128770 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 10 monclient: dump: 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.925 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.926 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 10 client.66177.objecter ms_handle_connect 0x7fdea003fde0 2024-06-11T19:24:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fdeb3fff640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdea003fde0 0x7fdea00422a0 secure :-1 s=READY pgs=4298 cs=0 l=1 rev1=1 crypto rx=0x7fdeac034000 tx=0x7fdeac02d000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdea800c7a0 con 0x7fdeb4128770 2024-06-11T19:24:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4128770 msgr2=0x7fdeb4125590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4128770 0x7fdeb4125590 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdea800e770 tx=0x7fdea800ec30 comp rx=0 tx=0).stop 2024-06-11T19:24:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4128770 msgr2=0x7fdeb4125590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.904+0000 7fde9bfff640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4128770 0x7fdeb4125590 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdea800e770 tx=0x7fdea800ec30 comp rx=0 tx=0).stop 2024-06-11T19:24:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.910+0000 7fdeba0e1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.910+0000 7fdeba0e1640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4125ad0 0x7fdeb4125ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.910+0000 7fdeba0e1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.910+0000 7fdeba0e1640 1 -- 172.21.15.98:0/2257388136 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb40a66c0 con 0x7fdeb4125ad0 2024-06-11T19:24:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fdeb2ffd640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4125ad0 0x7fdeb4125ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fdeb2ffd640 10 monclient: get_auth_request con 0x7fdeb4125ad0 auth_method 0 2024-06-11T19:24:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fdeb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:41.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fdeb2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fdeb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fdeb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fdeb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fdeb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fdeb2ffd640 10 monclient(hunting): handle_auth_done global_id 114393 payload 824 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fdeb2ffd640 1 -- 172.21.15.98:0/2257388136 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdeb4126420 con 0x7fdeb4125ad0 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fde9bfff640 10 client.66177.objecter ms_handle_connect 0x7fdeb4125ad0 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fdeb2ffd640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4125ad0 0x7fdeb4125ee0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdea800e770 tx=0x7fdea80046c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.911+0000 7fde9bfff640 10 client.66177.objecter resend_mon_ops 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdea8017070 con 0x7fdeb4125ad0 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fde9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fde9bfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fde9bfff640 10 monclient: dump: 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:41.930 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:41.931 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fdea8006370 con 0x7fdeb4125ad0 2024-06-11T19:24:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fde9bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fde9bfff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4125ad0 msgr2=0x7fdeb4125ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fde9bfff640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4125ad0 0x7fdeb4125ee0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdea800e770 tx=0x7fdea80046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fde9bfff640 1 -- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4125ad0 msgr2=0x7fdeb4125ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fde9bfff640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdeb4125ad0 0x7fdeb4125ee0 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdea800e770 tx=0x7fdea80046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.932 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:41.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:41.932 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:41.932 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:41.932 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 170281, 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:41.933 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.934 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.935 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:41.935 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:41.935 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.935 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.935 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.935 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.935 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.935 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.935 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.935 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.935 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:41.937 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:41.937 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:41.937 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:41.937 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:41.937 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:41.937 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:41.937 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:41.937 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:41.937 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:41.937 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:41.937 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:41.937 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:41.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.939 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:41.941 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.944 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:41.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:41.944 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:41.944 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:41.945 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.946 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:41.948 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.949 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.949 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.949 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:41.949 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:41.949 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:41.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.950 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:41.952 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:41.954 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:41.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.955 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:41.955 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:41.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:41.955 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:41.955 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:41.955 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:41.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:41.955 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:41.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:41.955 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:41.955 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:41.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fdeba0e1640 10 librados: watch_flush enter 2024-06-11T19:24:41.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fdeba0e1640 10 librados: watch_flush exit 2024-06-11T19:24:41.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fdeba0e1640 20 client.66177.objecter shutdown clearing up homeless session... 2024-06-11T19:24:41.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fdeba0e1640 10 client.66177.objecter successfully canceled tick 2024-06-11T19:24:41.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.912+0000 7fdeba0e1640 1 -- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdea003fde0 msgr2=0x7fdea00422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.913+0000 7fdeba0e1640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdea003fde0 0x7fdea00422a0 secure :-1 s=READY pgs=4298 cs=0 l=1 rev1=1 crypto rx=0x7fdeac034000 tx=0x7fdeac02d000 comp rx=0 tx=0).stop 2024-06-11T19:24:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.913+0000 7fdeba0e1640 10 monclient: shutdown 2024-06-11T19:24:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.913+0000 7fdeba0e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.913+0000 7fdeba0e1640 1 -- 172.21.15.98:0/2257388136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdeb40a2530 msgr2=0x7fdeb41cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.913+0000 7fdeb3fff640 1 -- 172.21.15.98:0/2257388136 reap_dead start 2024-06-11T19:24:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.913+0000 7fdeba0e1640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdeb40a2530 0x7fdeb41cdaa0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fdea4007ef0 tx=0x7fdea400e4b0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.913+0000 7fdeba0e1640 1 -- 172.21.15.98:0/2257388136 shutdown_connections 2024-06-11T19:24:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.913+0000 7fdeba0e1640 1 --2- 172.21.15.98:0/2257388136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdeb40a2530 0x7fdeb41cdaa0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.913+0000 7fdeba0e1640 1 -- 172.21.15.98:0/2257388136 >> 172.21.15.98:0/2257388136 conn(0x7fdeb40ab850 msgr2=0x7fdeb409c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.913+0000 7fdeba0e1640 1 -- 172.21.15.98:0/2257388136 shutdown_connections 2024-06-11T19:24:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.913+0000 7fdeba0e1640 1 -- 172.21.15.98:0/2257388136 wait complete. 2024-06-11T19:24:41.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:41.913+0000 7fdeba0e1640 1 librados: shutdown 2024-06-11T19:24:41.959 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:41.959 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:24:41.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:24:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.059+0000 7fd189a92640 10 monclient: get_monmap_and_config 2024-06-11T19:24:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.059+0000 7fd189a92640 10 monclient: build_initial_monmap 2024-06-11T19:24:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.059+0000 7fd189a92640 10 monclient: monmap: 2024-06-11T19:24:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:42.060711+0000 2024-06-11T19:24:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:42.060711+0000 2024-06-11T19:24:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.060+0000 7fd189a92640 1 Processor -- start 2024-06-11T19:24:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.060+0000 7fd189a92640 1 -- start start 2024-06-11T19:24:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.060+0000 7fd189a92640 10 monclient: init 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.060+0000 7fd189a92640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.060+0000 7fd189a92640 10 monclient: _add_conns ranks=[3,2,9,0,1,7,6,10,4,8,5] 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.060+0000 7fd189a92640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1840a4920 0x7fd1840a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.060+0000 7fd189a92640 10 monclient(hunting): picked mon.noname-g con 0x7fd1840a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd189a92640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a5240 0x7fd1840a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd189a92640 10 monclient(hunting): picked mon.noname-b con 0x7fd1840a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd189a92640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd1840a2530 0x7fd1840a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd189a92640 10 monclient(hunting): picked mon.noname-a con 0x7fd1840a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd189a92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd189a92640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd1840a7140 con 0x7fd1840a2530 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd189a92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd189a92640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd1840a2b50 con 0x7fd1840a5240 2024-06-11T19:24:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd189a92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd189a92640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd1840a2e30 con 0x7fd1840a4920 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd189a92640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd189a92640 10 monclient(hunting): authenticate will time out at 1691.868074s 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd182ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a5240 0x7fd1840a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd1840a2530 0x7fd1840a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd182ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a5240 0x7fd1840a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57374/0 (socket says 172.21.15.98:57374) 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd182ffd640 1 -- 172.21.15.98:0/1338123643 learned_addr learned my addr 172.21.15.98:0/1338123643 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 10 monclient(hunting): get_auth_request con 0x7fd1840a2530 auth_method 0 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd182ffd640 10 monclient(hunting): get_auth_request con 0x7fd1840a5240 auth_method 0 2024-06-11T19:24:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd182ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd182ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd182ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd1837fe640 1 --2- 172.21.15.98:0/1338123643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1840a4920 0x7fd1840a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd1837fe640 10 monclient(hunting): get_auth_request con 0x7fd1840a4920 auth_method 0 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd1837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd1837fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd1837fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd182ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd182ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd182ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd1837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd1837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd1837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 10 monclient(hunting): handle_auth_done global_id 66186 payload 274 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 1 -- 172.21.15.98:0/1338123643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1840a4920 msgr2=0x7fd1840a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 1 --2- 172.21.15.98:0/1338123643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1840a4920 0x7fd1840a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 1 -- 172.21.15.98:0/1338123643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a5240 msgr2=0x7fd1840a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 1 --2- 172.21.15.98:0/1338123643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a5240 0x7fd1840a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 1 monclient: found mon.noname-a 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:42.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd183fff640 1 -- 172.21.15.98:0/1338123643 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1840a3110 con 0x7fd1840a2530 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.061+0000 7fd182ffd640 1 --2- 172.21.15.98:0/1338123643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a5240 0x7fd1840a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd183fff640 1 --2- 172.21.15.98:0/1338123643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1840a2530 0x7fd1840a5ee0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009d90 tx=0x7fd17c02bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52cc1b846153675 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 1 -- 172.21.15.98:0/1338123643 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd17c039070 con 0x7fd1840a2530 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 10 monclient: dump: 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 10 monclient: _finish_auth 0 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 10 monclient: _check_auth_tickets 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 1 -- 172.21.15.98:0/1338123643 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd17c02cd00 con 0x7fd1840a2530 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 5 monclient: authenticate success, global_id 66186 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 1 -- 172.21.15.98:0/1338123643 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd17c0327b0 con 0x7fd1840a2530 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 4 set_mon_vals no callback set 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd1827fc640 10 monclient: dump: 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 10 monclient: shutdown 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 1 -- 172.21.15.98:0/1338123643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1840a2530 msgr2=0x7fd1840a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 1 --2- 172.21.15.98:0/1338123643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1840a2530 0x7fd1840a5ee0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009d90 tx=0x7fd17c02bce0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 1 -- 172.21.15.98:0/1338123643 shutdown_connections 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 1 --2- 172.21.15.98:0/1338123643 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a5240 0x7fd1840a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 1 --2- 172.21.15.98:0/1338123643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1840a2530 0x7fd1840a5ee0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 1 --2- 172.21.15.98:0/1338123643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1840a4920 0x7fd1840a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 1 -- 172.21.15.98:0/1338123643 >> 172.21.15.98:0/1338123643 conn(0x7fd1840ab850 msgr2=0x7fd1840a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 1 -- 172.21.15.98:0/1338123643 shutdown_connections 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.062+0000 7fd189a92640 1 -- 172.21.15.98:0/1338123643 wait complete. 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.063+0000 7fd189a92640 10 monclient: build_initial_monmap 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.063+0000 7fd189a92640 10 monclient: monmap: 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:42.079 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.063+0000 7fd189a92640 1 librados: starting msgr at 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.063+0000 7fd189a92640 1 librados: starting objecter 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.063+0000 7fd189a92640 1 Processor -- start 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 1 -- start start 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 1 librados: setting wanted keys 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 1 librados: calling monclient init 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 10 monclient: init 2024-06-11T19:24:42.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 10 monclient: _add_conns ranks=[3,2,8,6,7,1,5,0,4] 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1840a2530 0x7fd1841cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 10 monclient(hunting): picked mon.noname-g con 0x7fd1840a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a4920 0x7fd1841cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 10 monclient(hunting): picked mon.noname-b con 0x7fd1840a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1840a5240 0x7fd184124f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 10 monclient(hunting): picked mon.noname-e con 0x7fd1840a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd1840a6da0 con 0x7fd1840a4920 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd184138e50 con 0x7fd1840a2530 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd1840a6420 con 0x7fd1840a5240 2024-06-11T19:24:42.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd189a92640 10 monclient(hunting): authenticate will time out at 1691.871569s 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a4920 0x7fd1841cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a4920 0x7fd1841cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57384/0 (socket says 172.21.15.98:57384) 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 1 -- 172.21.15.98:0/4292210061 learned_addr learned my addr 172.21.15.98:0/4292210061 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 10 monclient(hunting): get_auth_request con 0x7fd1840a4920 auth_method 0 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd183fff640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1840a5240 0x7fd184124f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd183fff640 10 monclient(hunting): get_auth_request con 0x7fd1840a5240 auth_method 0 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd183fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd183fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd183fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd183fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd183fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 10 monclient(hunting): handle_auth_done global_id 85223 payload 824 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 1 -- 172.21.15.98:0/4292210061 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1840a5240 msgr2=0x7fd184124f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1840a5240 0x7fd184124f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 1 -- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1840a2530 msgr2=0x7fd1841cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd1837fe640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1840a2530 0x7fd1841cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1840a2530 0x7fd1841cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 1 monclient: found mon.noname-b 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:42.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.064+0000 7fd182ffd640 1 -- 172.21.15.98:0/4292210061 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1840aacc0 con 0x7fd1840a4920 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd182ffd640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a4920 0x7fd1841cdfe0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd17800e8d0 tx=0x7fd17800ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 10 client.?.objecter ms_handle_connect 0x7fd1840a4920 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1780044a0 con 0x7fd1840a4920 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 10 monclient: dump: 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.084 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 10 monclient: _finish_auth 0 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:42.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd17800c940 con 0x7fd1840a4920 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd189a92640 5 monclient: authenticate success, global_id 85223 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd189a92640 10 monclient: _renew_subs 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd189a92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd189a92640 1 -- 172.21.15.98:0/4292210061 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1840a8ff0 con 0x7fd1840a4920 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd178003610 con 0x7fd1840a4920 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd16bfff640 10 monclient: dump: 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd189a92640 10 client.85223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd189a92640 10 monclient: _renew_subs 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd189a92640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd189a92640 1 -- 172.21.15.98:0/4292210061 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd184128ed0 con 0x7fd1840a4920 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd189a92640 1 librados: init done 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.065+0000 7fd181ffb640 4 set_mon_vals no callback set 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd189a92640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd189a92640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125450 0x7fd184125830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd189a92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd189a92640 1 -- 172.21.15.98:0/4292210061 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd1841277b0 con 0x7fd184125450 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd178004030 con 0x7fd1840a4920 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd1837fe640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125450 0x7fd184125830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd1837fe640 10 monclient: get_auth_request con 0x7fd184125450 auth_method 0 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd1837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd1837fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd1837fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd16bfff640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd17003fdb0 0x7fd170042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd1837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd1837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd1837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(626..626 src has 1..626) ==== 6797+0+0 (secure 0 0 0) 0x7fd17800f040 con 0x7fd1840a4920 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd16bfff640 10 client.85223.objecter ms_dispatch 0x7fd184000b90 osd_map(626..626 src has 1..626) 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd16bfff640 3 client.85223.objecter handle_osd_map got epochs [626,626] > 0 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd16bfff640 3 client.85223.objecter handle_osd_map decoding full epoch 626 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd183fff640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd17003fdb0 0x7fd170042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd16bfff640 20 client.85223.objecter dump_active .. 0 homeless 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd183fff640 10 monclient: get_auth_request con 0x7fd17003fdb0 auth_method 0 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd1837fe640 10 monclient(hunting): handle_auth_done global_id 124196 payload 824 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd1837fe640 1 -- 172.21.15.98:0/4292210061 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd184128a30 con 0x7fd184125450 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd16bfff640 10 client.85223.objecter ms_handle_connect 0x7fd184125450 2024-06-11T19:24:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd16bfff640 10 client.85223.objecter resend_mon_ops 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd1837fe640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125450 0x7fd184125830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd174010750 tx=0x7fd174010c10 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd17400f040 con 0x7fd184125450 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd16bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.066+0000 7fd183fff640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd17003fdb0 0x7fd170042270 secure :-1 s=READY pgs=4299 cs=0 l=1 rev1=1 crypto rx=0x7fd17c036000 tx=0x7fd17c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.067+0000 7fd16bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.067+0000 7fd16bfff640 10 monclient: dump: 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.090 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.067+0000 7fd16bfff640 10 client.85223.objecter ms_handle_connect 0x7fd17003fdb0 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.067+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd174010e60 con 0x7fd184125450 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.067+0000 7fd16bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.067+0000 7fd16bfff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.067+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125450 msgr2=0x7fd184125830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.067+0000 7fd16bfff640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125450 0x7fd184125830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd174010750 tx=0x7fd174010c10 comp rx=0 tx=0).stop 2024-06-11T19:24:42.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.067+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125450 msgr2=0x7fd184125830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:42.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.067+0000 7fd16bfff640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125450 0x7fd184125830 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd174010750 tx=0x7fd174010c10 comp rx=0 tx=0).stop 2024-06-11T19:24:42.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.073+0000 7fd189a92640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:42.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.073+0000 7fd189a92640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125d70 0x7fd18409f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.073+0000 7fd189a92640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.073+0000 7fd189a92640 1 -- 172.21.15.98:0/4292210061 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd184050830 con 0x7fd184125d70 2024-06-11T19:24:42.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.073+0000 7fd1837fe640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125d70 0x7fd18409f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.073+0000 7fd1837fe640 10 monclient: get_auth_request con 0x7fd184125d70 auth_method 0 2024-06-11T19:24:42.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.073+0000 7fd1837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.073+0000 7fd1837fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.073+0000 7fd1837fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.073+0000 7fd1837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.073+0000 7fd1837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.073+0000 7fd1837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd1837fe640 10 monclient(hunting): handle_auth_done global_id 124205 payload 824 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd1837fe640 1 -- 172.21.15.98:0/4292210061 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd1840a6750 con 0x7fd184125d70 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd1837fe640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125d70 0x7fd18409f3d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd174010750 tx=0x7fd1740046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 10 client.85223.objecter ms_handle_connect 0x7fd184125d70 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 10 client.85223.objecter resend_mon_ops 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1740044a0 con 0x7fd184125d70 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 10 monclient: dump: 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.095 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fd174025070 con 0x7fd184125d70 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:42.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125d70 msgr2=0x7fd18409f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125d70 0x7fd18409f3d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd174010750 tx=0x7fd1740046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 1 -- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125d70 msgr2=0x7fd18409f3d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:42.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.074+0000 7fd16bfff640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd184125d70 0x7fd18409f3d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.097 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:42.097 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:42.097 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:42.097 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:42.097 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:42.097 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 25007, 2024-06-11T19:24:42.097 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:42.097 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:42.097 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:42.098 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:42.099 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:42.100 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:42.101 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:42.102 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:42.104 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.105 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.106 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:42.107 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.109 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.112 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:42.113 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.116 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:42.117 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:42.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.117 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:42.117 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:42.117 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:42.117 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.117 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.117 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:42.117 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 10 librados: watch_flush enter 2024-06-11T19:24:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 10 librados: watch_flush exit 2024-06-11T19:24:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 20 client.85223.objecter shutdown clearing up homeless session... 2024-06-11T19:24:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 10 client.85223.objecter successfully canceled tick 2024-06-11T19:24:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 1 -- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd17003fdb0 msgr2=0x7fd170042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd17003fdb0 0x7fd170042270 secure :-1 s=READY pgs=4299 cs=0 l=1 rev1=1 crypto rx=0x7fd17c036000 tx=0x7fd17c030000 comp rx=0 tx=0).stop 2024-06-11T19:24:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 10 monclient: shutdown 2024-06-11T19:24:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:42.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 1 -- 172.21.15.98:0/4292210061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a4920 msgr2=0x7fd1841cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 1 --2- 172.21.15.98:0/4292210061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1840a4920 0x7fd1841cdfe0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd17800e8d0 tx=0x7fd17800ed90 comp rx=0 tx=0).stop 2024-06-11T19:24:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd183fff640 1 -- 172.21.15.98:0/4292210061 reap_dead start 2024-06-11T19:24:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd183fff640 1 -- 172.21.15.98:0/4292210061 reap_dead start 2024-06-11T19:24:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 1 -- 172.21.15.98:0/4292210061 shutdown_connections 2024-06-11T19:24:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 1 -- 172.21.15.98:0/4292210061 >> 172.21.15.98:0/4292210061 conn(0x7fd1840ab850 msgr2=0x7fd1840a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 1 -- 172.21.15.98:0/4292210061 shutdown_connections 2024-06-11T19:24:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 1 -- 172.21.15.98:0/4292210061 wait complete. 2024-06-11T19:24:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.075+0000 7fd189a92640 1 librados: shutdown 2024-06-11T19:24:42.121 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:42.121 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:24:42.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:24:42.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.221+0000 7f9c9a531640 10 monclient: get_monmap_and_config 2024-06-11T19:24:42.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.221+0000 7f9c9a531640 10 monclient: build_initial_monmap 2024-06-11T19:24:42.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.221+0000 7f9c9a531640 10 monclient: monmap: 2024-06-11T19:24:42.221 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:42.221 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.221 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:42.222911+0000 2024-06-11T19:24:42.221 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:42.222911+0000 2024-06-11T19:24:42.221 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:42.221 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.221 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:42.222 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:42.222 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:42.222 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:42.222 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:42.222 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:42.222 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:42.222 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:42.222 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:42.222 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:42.222 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:42.222 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.222+0000 7f9c9a531640 1 Processor -- start 2024-06-11T19:24:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.222+0000 7f9c9a531640 1 -- start start 2024-06-11T19:24:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.222+0000 7f9c9a531640 10 monclient: init 2024-06-11T19:24:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 10 monclient: _add_conns ranks=[4,8,2,9,1,10,5,6,3,7,0] 2024-06-11T19:24:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 0x7f9c940a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 10 monclient(hunting): picked mon.noname-c con 0x7f9c940a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a5240 0x7f9c940a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 10 monclient(hunting): picked mon.noname-e con 0x7f9c940a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 0x7f9c940a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 10 monclient(hunting): picked mon.noname-b con 0x7f9c940a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9c940a7140 con 0x7f9c940a2530 2024-06-11T19:24:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f9c940a2b50 con 0x7f9c940a4920 2024-06-11T19:24:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9c940a2e30 con 0x7f9c940a5240 2024-06-11T19:24:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c9a531640 10 monclient(hunting): authenticate will time out at 1692.030173s 2024-06-11T19:24:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c937fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a5240 0x7f9c940a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 0x7f9c940a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c98ace640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 0x7f9c940a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 0x7f9c940a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46944/0 (socket says 172.21.15.98:46944) 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c98ace640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 0x7f9c940a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57400/0 (socket says 172.21.15.98:57400) 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c937fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a5240 0x7f9c940a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:53204/0 (socket says 172.21.15.98:53204) 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 1 -- 172.21.15.98:0/572741343 learned_addr learned my addr 172.21.15.98:0/572741343 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 10 monclient(hunting): get_auth_request con 0x7f9c940a4920 auth_method 0 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c98ace640 10 monclient(hunting): get_auth_request con 0x7f9c940a2530 auth_method 0 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c98ace640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c98ace640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c98ace640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c937fe640 10 monclient(hunting): get_auth_request con 0x7f9c940a5240 auth_method 0 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c937fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c98ace640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c98ace640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c98ace640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 10 monclient(hunting): handle_auth_done global_id 114484 payload 274 2024-06-11T19:24:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 1 -- 172.21.15.98:0/572741343 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a5240 msgr2=0x7f9c940a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 1 --2- 172.21.15.98:0/572741343 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a5240 0x7f9c940a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 1 -- 172.21.15.98:0/572741343 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 msgr2=0x7f9c940a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 1 --2- 172.21.15.98:0/572741343 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 0x7f9c940a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 1 monclient: found mon.noname-c 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c93fff640 1 -- 172.21.15.98:0/572741343 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c940a3110 con 0x7f9c940a4920 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c937fe640 1 --2- 172.21.15.98:0/572741343 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a5240 0x7f9c940a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.223+0000 7f9c98ace640 1 --2- 172.21.15.98:0/572741343 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 0x7f9c940a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c93fff640 1 --2- 172.21.15.98:0/572741343 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 0x7f9c940a4d00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9c840098a0 tx=0x7f9c8402b7a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=68fd3f91f0bdca24 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 1 -- 172.21.15.98:0/572741343 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c84038070 con 0x7f9c940a4920 2024-06-11T19:24:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 10 monclient: dump: 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 10 monclient: _finish_auth 0 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 1 -- 172.21.15.98:0/572741343 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c84031970 con 0x7f9c940a4920 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c9a531640 5 monclient: authenticate success, global_id 114484 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c9a531640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c9a531640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 1 -- 172.21.15.98:0/572741343 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c84031ba0 con 0x7f9c940a4920 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c9a531640 4 set_mon_vals no callback set 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.224+0000 7f9c92ffd640 10 monclient: dump: 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 10 monclient: shutdown 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 1 -- 172.21.15.98:0/572741343 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 msgr2=0x7f9c940a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 1 --2- 172.21.15.98:0/572741343 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 0x7f9c940a4d00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9c840098a0 tx=0x7f9c8402b7a0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 1 -- 172.21.15.98:0/572741343 shutdown_connections 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 1 --2- 172.21.15.98:0/572741343 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c940a2530 0x7f9c940a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 1 --2- 172.21.15.98:0/572741343 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c940a5240 0x7f9c940a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 1 --2- 172.21.15.98:0/572741343 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 0x7f9c940a4d00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 1 -- 172.21.15.98:0/572741343 >> 172.21.15.98:0/572741343 conn(0x7f9c940ab850 msgr2=0x7f9c940a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 1 -- 172.21.15.98:0/572741343 shutdown_connections 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 1 -- 172.21.15.98:0/572741343 wait complete. 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 10 monclient: build_initial_monmap 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 10 monclient: monmap: 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 1 librados: starting msgr at 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.225+0000 7f9c9a531640 1 librados: starting objecter 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 1 Processor -- start 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 1 -- start start 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 1 librados: setting wanted keys 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 1 librados: calling monclient init 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 10 monclient: init 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 10 monclient: _add_conns ranks=[1,4,6,2,3,5,8,7,0] 2024-06-11T19:24:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c940a2530 0x7f9c94137740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 10 monclient(hunting): picked mon.noname-f con 0x7f9c940a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 0x7f9c94137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 10 monclient(hunting): picked mon.noname-c con 0x7f9c940a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9c940a5240 0x7f9c941232a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 10 monclient(hunting): picked mon.noname-d con 0x7f9c940a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9c940a6da0 con 0x7f9c940a2530 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f9c94138e50 con 0x7f9c940a4920 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f9c940a6420 con 0x7f9c940a5240 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c9a531640 10 monclient(hunting): authenticate will time out at 1692.033727s 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c937fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 0x7f9c94137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c98ace640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9c940a5240 0x7f9c941232a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c937fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 0x7f9c94137c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46950/0 (socket says 172.21.15.98:46950) 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c937fe640 1 -- 172.21.15.98:0/1503823739 learned_addr learned my addr 172.21.15.98:0/1503823739 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c937fe640 10 monclient(hunting): get_auth_request con 0x7f9c940a4920 auth_method 0 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c937fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c98ace640 10 monclient(hunting): get_auth_request con 0x7f9c940a5240 auth_method 0 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c98ace640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c98ace640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c98ace640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c98ace640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c98ace640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.226+0000 7f9c98ace640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c93fff640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c940a2530 0x7f9c94137740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c93fff640 10 monclient(hunting): get_auth_request con 0x7f9c940a2530 auth_method 0 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c93fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c937fe640 10 monclient(hunting): handle_auth_done global_id 114493 payload 824 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c937fe640 1 -- 172.21.15.98:0/1503823739 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9c940a5240 msgr2=0x7f9c941232a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c937fe640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9c940a5240 0x7f9c941232a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c937fe640 1 -- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c940a2530 msgr2=0x7f9c94137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c937fe640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c940a2530 0x7f9c94137740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c937fe640 10 monclient: _finish_hunting 0 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c937fe640 1 monclient: found mon.noname-c 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c937fe640 1 -- 172.21.15.98:0/1503823739 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c941c0d10 con 0x7f9c940a4920 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c937fe640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 0x7f9c94137c80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800e8d0 tx=0x7f9c8800ecd0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 10 client.?.objecter ms_handle_connect 0x7f9c940a4920 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c88017070 con 0x7f9c940a4920 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 10 monclient: dump: 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.247 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 10 monclient: _finish_auth 0 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:24:42.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c8800cdb0 con 0x7f9c940a4920 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c9a531640 5 monclient: authenticate success, global_id 114493 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c9a531640 10 monclient: _renew_subs 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c9a531640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c9a531640 1 -- 172.21.15.98:0/1503823739 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c94126b80 con 0x7f9c940a4920 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c880138e0 con 0x7f9c940a4920 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c90ff9640 10 monclient: dump: 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c9a531640 10 client.114493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c927fc640 4 set_mon_vals no callback set 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c9a531640 10 monclient: _renew_subs 2024-06-11T19:24:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c9a531640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c9a531640 1 -- 172.21.15.98:0/1503823739 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c940a92f0 con 0x7f9c940a4920 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.227+0000 7f9c9a531640 1 librados: init done 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f9c8801a070 con 0x7f9c940a4920 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c9a531640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c9a531640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c94126dc0 0x7f9c94123fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c9a531640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c9a531640 1 -- 172.21.15.98:0/1503823739 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f9c94050830 con 0x7f9c94126dc0 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c90ff9640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c8003fdb0 0x7f9c80042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(626..626 src has 1..626) ==== 6797+0+0 (secure 0 0 0) 0x7f9c8800f040 con 0x7f9c940a4920 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c90ff9640 10 client.114493.objecter ms_dispatch 0x7f9c94000b90 osd_map(626..626 src has 1..626) 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c90ff9640 3 client.114493.objecter handle_osd_map got epochs [626,626] > 0 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c90ff9640 3 client.114493.objecter handle_osd_map decoding full epoch 626 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c90ff9640 20 client.114493.objecter dump_active .. 0 homeless 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c93fff640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c94126dc0 0x7f9c94123fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c98ace640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c8003fdb0 0x7f9c80042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c93fff640 10 monclient: get_auth_request con 0x7f9c94126dc0 auth_method 0 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c93fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c98ace640 10 monclient: get_auth_request con 0x7f9c8003fdb0 auth_method 0 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c93fff640 10 monclient(hunting): handle_auth_done global_id 84877 payload 824 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.228+0000 7f9c93fff640 1 -- 172.21.15.98:0/1503823739 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c941268a0 con 0x7f9c94126dc0 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 10 client.114493.objecter ms_handle_connect 0x7f9c8003fdb0 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c98ace640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c8003fdb0 0x7f9c80042270 secure :-1 s=READY pgs=4300 cs=0 l=1 rev1=1 crypto rx=0x7f9c940a2070 tx=0x7f9c8c00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 10 client.114493.objecter ms_handle_connect 0x7f9c94126dc0 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 10 client.114493.objecter resend_mon_ops 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c93fff640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c94126dc0 0x7f9c94123fa0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9c84009870 tx=0x7f9c840046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c84043070 con 0x7f9c94126dc0 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 10 monclient: dump: 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.254 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9c84031ea0 con 0x7f9c94126dc0 2024-06-11T19:24:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c94126dc0 msgr2=0x7f9c94123fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c94126dc0 0x7f9c94123fa0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9c84009870 tx=0x7f9c840046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c94126dc0 msgr2=0x7f9c94123fa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.229+0000 7f9c90ff9640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c94126dc0 0x7f9c94123fa0 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9c84009870 tx=0x7f9c840046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c9a531640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c9a531640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9402cb40 0x7f9c9402cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c9a531640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c9a531640 1 -- 172.21.15.98:0/1503823739 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f9c94124e40 con 0x7f9c9402cb40 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c93fff640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9402cb40 0x7f9c9402cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c93fff640 10 monclient: get_auth_request con 0x7f9c9402cb40 auth_method 0 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c93fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c93fff640 10 monclient(hunting): handle_auth_done global_id 84886 payload 824 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c93fff640 1 -- 172.21.15.98:0/1503823739 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9c94073550 con 0x7f9c9402cb40 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c93fff640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9402cb40 0x7f9c9402cf50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9c840098a0 tx=0x7f9c84031000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c90ff9640 10 client.114493.objecter ms_handle_connect 0x7f9c9402cb40 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.236+0000 7f9c90ff9640 10 client.114493.objecter resend_mon_ops 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c84038040 con 0x7f9c9402cb40 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c90ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c90ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c90ff9640 10 monclient: dump: 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7f9c84003c10 con 0x7f9c9402cb40 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c90ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c90ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9402cb40 msgr2=0x7f9c9402cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c90ff9640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9402cb40 0x7f9c9402cf50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9c840098a0 tx=0x7f9c84031000 comp rx=0 tx=0).stop 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c90ff9640 1 -- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9402cb40 msgr2=0x7f9c9402cf50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c90ff9640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f9c9402cb40 0x7f9c9402cf50 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9c840098a0 tx=0x7f9c84031000 comp rx=0 tx=0).stop 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 554007, 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:24:42.259 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:42.260 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:42.261 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:42.262 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:42.263 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:42.265 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:42.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:42.269 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.272 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:42.273 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.274 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.274 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:42.274 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:42.274 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.274 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.274 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.274 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:42.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:24:42.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:42.279 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:42.280 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:42.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 10 librados: watch_flush enter 2024-06-11T19:24:42.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 10 librados: watch_flush exit 2024-06-11T19:24:42.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 20 client.114493.objecter shutdown clearing up homeless session... 2024-06-11T19:24:42.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 10 client.114493.objecter successfully canceled tick 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 1 -- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c8003fdb0 msgr2=0x7f9c80042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9c8003fdb0 0x7f9c80042270 secure :-1 s=READY pgs=4300 cs=0 l=1 rev1=1 crypto rx=0x7f9c940a2070 tx=0x7f9c8c00b040 comp rx=0 tx=0).stop 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 10 monclient: shutdown 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 1 -- 172.21.15.98:0/1503823739 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 msgr2=0x7f9c94137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c98ace640 1 -- 172.21.15.98:0/1503823739 reap_dead start 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 0x7f9c94137c80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800e8d0 tx=0x7f9c8800ecd0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 1 -- 172.21.15.98:0/1503823739 shutdown_connections 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 1 --2- 172.21.15.98:0/1503823739 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c940a4920 0x7f9c94137c80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 1 -- 172.21.15.98:0/1503823739 >> 172.21.15.98:0/1503823739 conn(0x7f9c940ab850 msgr2=0x7f9c940a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 1 -- 172.21.15.98:0/1503823739 shutdown_connections 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 1 -- 172.21.15.98:0/1503823739 wait complete. 2024-06-11T19:24:42.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.237+0000 7f9c9a531640 1 librados: shutdown 2024-06-11T19:24:42.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:24:42.283 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['h', 'c', 'i'] 2024-06-11T19:24:42.283 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-06-11T19:24:42.283 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h 2024-06-11T19:24:42.283 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h store 2024-06-11T19:24:42.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h sync_force --yes-i-really-mean-it 2024-06-11T19:24:42.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.386+0000 7ffb16680640 10 monclient: get_monmap_and_config 2024-06-11T19:24:42.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.386+0000 7ffb16680640 10 monclient: build_initial_monmap 2024-06-11T19:24:42.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.386+0000 7ffb16680640 10 monclient: monmap: 2024-06-11T19:24:42.385 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:42.385 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.385 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:42.387361+0000 2024-06-11T19:24:42.385 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:42.387361+0000 2024-06-11T19:24:42.385 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:42.385 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.386 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:42.386 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:42.386 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:42.386 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:42.386 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:42.386 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:42.386 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:42.386 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:42.386 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:42.389 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:42.389 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:42.389 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 1 Processor -- start 2024-06-11T19:24:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 1 -- start start 2024-06-11T19:24:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 10 monclient: init 2024-06-11T19:24:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 10 monclient: _add_conns ranks=[10,4,8,0,7,6,9,5,1,2,3] 2024-06-11T19:24:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ffb100a4920 0x7ffb100a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 10 monclient(hunting): picked mon.noname-f con 0x7ffb100a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:24:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffb100a5240 0x7ffb100a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 10 monclient(hunting): picked mon.noname-c con 0x7ffb100a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb100a2530 0x7ffb100a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 10 monclient(hunting): picked mon.noname-e con 0x7ffb100a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7ffb100a7140 con 0x7ffb100a4920 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ffb100a2b50 con 0x7ffb100a5240 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ffb100a2e30 con 0x7ffb100a2530 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb16680640 10 monclient(hunting): authenticate will time out at 1692.194612s 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb14c1d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb100a2530 0x7ffb100a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb0f7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffb100a5240 0x7ffb100a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb0f7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffb100a5240 0x7ffb100a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46958/0 (socket says 172.21.15.98:46958) 2024-06-11T19:24:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb14c1d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb100a2530 0x7ffb100a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:53214/0 (socket says 172.21.15.98:53214) 2024-06-11T19:24:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb0f7fe640 1 -- 172.21.15.98:0/3997491432 learned_addr learned my addr 172.21.15.98:0/3997491432 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb0f7fe640 10 monclient(hunting): get_auth_request con 0x7ffb100a5240 auth_method 0 2024-06-11T19:24:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb0f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb14c1d640 10 monclient(hunting): get_auth_request con 0x7ffb100a2530 auth_method 0 2024-06-11T19:24:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb14c1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb14c1d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb14c1d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb14c1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb14c1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb14c1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb0ffff640 1 --2- 172.21.15.98:0/3997491432 >> v2:172.21.15.117:3300/0 conn(0x7ffb100a4920 0x7ffb100a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb14c1d640 10 monclient(hunting): handle_auth_done global_id 124235 payload 274 2024-06-11T19:24:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.387+0000 7ffb14c1d640 1 -- 172.21.15.98:0/3997491432 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffb100a5240 msgr2=0x7ffb100a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb14c1d640 1 --2- 172.21.15.98:0/3997491432 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffb100a5240 0x7ffb100a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb14c1d640 1 -- 172.21.15.98:0/3997491432 >> v2:172.21.15.117:3300/0 conn(0x7ffb100a4920 msgr2=0x7ffb100a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb14c1d640 1 --2- 172.21.15.98:0/3997491432 >> v2:172.21.15.117:3300/0 conn(0x7ffb100a4920 0x7ffb100a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb14c1d640 10 monclient: _finish_hunting 0 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb14c1d640 1 monclient: found mon.noname-e 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb14c1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb14c1d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb14c1d640 1 -- 172.21.15.98:0/3997491432 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb100a3110 con 0x7ffb100a2530 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb0ffff640 10 monclient: get_auth_request con 0x7ffb100a4920 auth_method 0 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb0f7fe640 1 --2- 172.21.15.98:0/3997491432 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffb100a5240 0x7ffb100a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb0ffff640 1 --2- 172.21.15.98:0/3997491432 >> v2:172.21.15.117:3300/0 conn(0x7ffb100a4920 0x7ffb100a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb14c1d640 1 --2- 172.21.15.98:0/3997491432 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb100a2530 0x7ffb100a5ee0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009d90 tx=0x7ffb0802bce0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=d76709877dac851b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb0effd640 1 -- 172.21.15.98:0/3997491432 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb08039070 con 0x7ffb100a2530 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb0effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.388+0000 7ffb0effd640 10 monclient: dump: 2024-06-11T19:24:42.403 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.404 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb0effd640 10 monclient: _finish_auth 0 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb0effd640 10 monclient: _check_auth_tickets 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb0effd640 1 -- 172.21.15.98:0/3997491432 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb0802cd00 con 0x7ffb100a2530 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 5 monclient: authenticate success, global_id 124235 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb0effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb0effd640 1 -- 172.21.15.98:0/3997491432 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb080327b0 con 0x7ffb100a2530 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 4 set_mon_vals no callback set 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb0effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb0effd640 10 monclient: dump: 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.405 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.406 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 10 monclient: shutdown 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 1 -- 172.21.15.98:0/3997491432 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb100a2530 msgr2=0x7ffb100a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 1 --2- 172.21.15.98:0/3997491432 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb100a2530 0x7ffb100a5ee0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009d90 tx=0x7ffb0802bce0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 1 -- 172.21.15.98:0/3997491432 shutdown_connections 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 1 --2- 172.21.15.98:0/3997491432 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffb100a5240 0x7ffb100a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 1 --2- 172.21.15.98:0/3997491432 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ffb100a2530 0x7ffb100a5ee0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 1 --2- 172.21.15.98:0/3997491432 >> v2:172.21.15.117:3300/0 conn(0x7ffb100a4920 0x7ffb100a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 1 -- 172.21.15.98:0/3997491432 >> 172.21.15.98:0/3997491432 conn(0x7ffb100ab850 msgr2=0x7ffb100a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 1 -- 172.21.15.98:0/3997491432 shutdown_connections 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.389+0000 7ffb16680640 1 -- 172.21.15.98:0/3997491432 wait complete. 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 10 monclient: build_initial_monmap 2024-06-11T19:24:42.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 10 monclient: monmap: 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:42.408 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 1 librados: starting msgr at 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 1 librados: starting objecter 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 1 Processor -- start 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 1 -- start start 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 1 librados: setting wanted keys 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 1 librados: calling monclient init 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 10 monclient: init 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 10 monclient: _add_conns ranks=[6,3,1,0,4,7,8,5,2] 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffb100a2530 0x7ffb101cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 10 monclient(hunting): picked mon.noname-d con 0x7ffb100a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.390+0000 7ffb16680640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb100a4920 0x7ffb101ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 10 monclient(hunting): picked mon.noname-g con 0x7ffb100a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb100a5240 0x7ffb10122f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 10 monclient(hunting): picked mon.noname-f con 0x7ffb100a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffb100a6da0 con 0x7ffb100a5240 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ffb10138e50 con 0x7ffb100a4920 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ffb100a6420 con 0x7ffb100a2530 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 10 monclient(hunting): authenticate will time out at 1692.198037s 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffb100a2530 0x7ffb101cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffb100a2530 0x7ffb101cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:35430/0 (socket says 172.21.15.98:35430) 2024-06-11T19:24:42.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 1 -- 172.21.15.98:0/389615374 learned_addr learned my addr 172.21.15.98:0/389615374 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 10 monclient(hunting): get_auth_request con 0x7ffb100a2530 auth_method 0 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb14c1d640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb100a5240 0x7ffb10122f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0f7fe640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb100a4920 0x7ffb101ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 10 monclient(hunting): handle_auth_done global_id 124260 payload 824 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 1 -- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb100a4920 msgr2=0x7ffb101ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb100a4920 0x7ffb101ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 1 -- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb100a5240 msgr2=0x7ffb10122f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb100a5240 0x7ffb10122f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 1 monclient: found mon.noname-d 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 1 -- 172.21.15.98:0/389615374 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb100a90d0 con 0x7ffb100a2530 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb14c1d640 10 monclient: get_auth_request con 0x7ffb100a5240 auth_method 0 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb14c1d640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb100a5240 0x7ffb10122f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0f7fe640 10 monclient: get_auth_request con 0x7ffb100a4920 auth_method 0 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0f7fe640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb100a4920 0x7ffb101ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0ffff640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffb100a2530 0x7ffb101cdd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000c850 tx=0x7ffb0000cc50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 10 client.?.objecter ms_handle_connect 0x7ffb100a2530 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:42.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb00015070 con 0x7ffb100a2530 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 10 monclient: dump: 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.413 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 10 monclient: _finish_auth 0 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 10 monclient: _check_auth_tickets 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 5 monclient: authenticate success, global_id 124260 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb00007ca0 con 0x7ffb100a2530 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 10 monclient: _renew_subs 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 1 -- 172.21.15.98:0/389615374 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb10124c20 con 0x7ffb100a2530 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:42.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb0000cea0 con 0x7ffb100a2530 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 10 client.124260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0cff9640 10 monclient: dump: 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.415 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb0e7fc640 4 set_mon_vals no callback set 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 10 monclient: _renew_subs 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 1 -- 172.21.15.98:0/389615374 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb10126c30 con 0x7ffb100a2530 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.391+0000 7ffb16680640 1 librados: init done 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.392+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ffb00018070 con 0x7ffb100a2530 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.392+0000 7ffb16680640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.392+0000 7ffb16680640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123490 0x7ffb10123870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.392+0000 7ffb16680640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.392+0000 7ffb16680640 1 -- 172.21.15.98:0/389615374 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ffb10050830 con 0x7ffb10123490 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.392+0000 7ffb0cff9640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffafc03fdb0 0x7ffafc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.392+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(627..627 src has 1..627) ==== 6823+0+0 (secure 0 0 0) 0x7ffb00051cb0 con 0x7ffb100a2530 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.392+0000 7ffb0cff9640 10 client.124260.objecter ms_dispatch 0x7ffb10000b90 osd_map(627..627 src has 1..627) 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.392+0000 7ffb0cff9640 3 client.124260.objecter handle_osd_map got epochs [627,627] > 0 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.392+0000 7ffb0cff9640 3 client.124260.objecter handle_osd_map decoding full epoch 627 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.392+0000 7ffb0f7fe640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123490 0x7ffb10123870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0f7fe640 10 monclient: get_auth_request con 0x7ffb10123490 auth_method 0 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0cff9640 20 client.124260.objecter dump_active .. 0 homeless 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb14c1d640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffafc03fdb0 0x7ffafc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb14c1d640 10 monclient: get_auth_request con 0x7ffafc03fdb0 auth_method 0 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0f7fe640 10 monclient(hunting): handle_auth_done global_id 124214 payload 824 2024-06-11T19:24:42.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0f7fe640 1 -- 172.21.15.98:0/389615374 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffb100a7360 con 0x7ffb10123490 2024-06-11T19:24:42.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0cff9640 10 client.124260.objecter ms_handle_connect 0x7ffb10123490 2024-06-11T19:24:42.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0cff9640 10 client.124260.objecter resend_mon_ops 2024-06-11T19:24:42.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0f7fe640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123490 0x7ffb10123870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb100a5620 tx=0x7ffb04010c20 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0cff9640 10 client.124260.objecter ms_handle_connect 0x7ffafc03fdb0 2024-06-11T19:24:42.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb14c1d640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffafc03fdb0 0x7ffafc042270 secure :-1 s=READY pgs=4301 cs=0 l=1 rev1=1 crypto rx=0x7ffb08036000 tx=0x7ffb08030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb0400f040 con 0x7ffb10123490 2024-06-11T19:24:42.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0cff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:24:42.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0cff9640 10 monclient: dump: 2024-06-11T19:24:42.418 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.418 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.419 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffb0401c400 con 0x7ffb10123490 2024-06-11T19:24:42.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.393+0000 7ffb0cff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:42.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.394+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123490 msgr2=0x7ffb10123870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.394+0000 7ffb0cff9640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123490 0x7ffb10123870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb100a5620 tx=0x7ffb04010c20 comp rx=0 tx=0).stop 2024-06-11T19:24:42.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.394+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123490 msgr2=0x7ffb10123870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:42.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.394+0000 7ffb0cff9640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123490 0x7ffb10123870 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb100a5620 tx=0x7ffb04010c20 comp rx=0 tx=0).stop 2024-06-11T19:24:42.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb16680640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:24:42.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb16680640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123db0 0x7ffb1009e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb16680640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb16680640 1 -- 172.21.15.98:0/389615374 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ffb10069740 con 0x7ffb10123db0 2024-06-11T19:24:42.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb0f7fe640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123db0 0x7ffb1009e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb0f7fe640 10 monclient: get_auth_request con 0x7ffb10123db0 auth_method 0 2024-06-11T19:24:42.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb0f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb0f7fe640 10 monclient(hunting): handle_auth_done global_id 124223 payload 824 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.401+0000 7ffb0f7fe640 1 -- 172.21.15.98:0/389615374 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7ffb100a6750 con 0x7ffb10123db0 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0f7fe640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123db0 0x7ffb1009e9f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffb100a5620 tx=0x7ffb040046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 10 client.124260.objecter ms_handle_connect 0x7ffb10123db0 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 10 client.124260.objecter resend_mon_ops 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb04005ba0 con 0x7ffb10123db0 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 10 monclient: dump: 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.423 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.424 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7ffb04005ba0 con 0x7ffb10123db0 2024-06-11T19:24:42.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:24:42.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:42.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123db0 msgr2=0x7ffb1009e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123db0 0x7ffb1009e9f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffb100a5620 tx=0x7ffb040046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 1 -- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123db0 msgr2=0x7ffb1009e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:42.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb0cff9640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ffb10123db0 0x7ffb1009e9f0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffb100a5620 tx=0x7ffb040046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.425 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:42.425 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-06-11T19:24:42.425 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-06-11T19:24:42.425 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:42.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb16680640 10 librados: watch_flush enter 2024-06-11T19:24:42.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb16680640 10 librados: watch_flush exit 2024-06-11T19:24:42.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb16680640 20 client.124260.objecter shutdown clearing up homeless session... 2024-06-11T19:24:42.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb16680640 10 client.124260.objecter successfully canceled tick 2024-06-11T19:24:42.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.402+0000 7ffb16680640 1 -- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffafc03fdb0 msgr2=0x7ffafc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.403+0000 7ffb16680640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ffafc03fdb0 0x7ffafc042270 secure :-1 s=READY pgs=4301 cs=0 l=1 rev1=1 crypto rx=0x7ffb08036000 tx=0x7ffb08030000 comp rx=0 tx=0).stop 2024-06-11T19:24:42.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.403+0000 7ffb16680640 10 monclient: shutdown 2024-06-11T19:24:42.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.403+0000 7ffb16680640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:42.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.403+0000 7ffb16680640 1 -- 172.21.15.98:0/389615374 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffb100a2530 msgr2=0x7ffb101cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.403+0000 7ffb16680640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffb100a2530 0x7ffb101cdd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000c850 tx=0x7ffb0000cc50 comp rx=0 tx=0).stop 2024-06-11T19:24:42.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.403+0000 7ffb14c1d640 1 -- 172.21.15.98:0/389615374 reap_dead start 2024-06-11T19:24:42.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.403+0000 7ffb16680640 1 -- 172.21.15.98:0/389615374 shutdown_connections 2024-06-11T19:24:42.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.403+0000 7ffb16680640 1 --2- 172.21.15.98:0/389615374 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffb100a2530 0x7ffb101cdd20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.403+0000 7ffb16680640 1 -- 172.21.15.98:0/389615374 >> 172.21.15.98:0/389615374 conn(0x7ffb100ab850 msgr2=0x7ffb101bff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:42.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.403+0000 7ffb16680640 1 -- 172.21.15.98:0/389615374 shutdown_connections 2024-06-11T19:24:42.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.403+0000 7ffb16680640 1 -- 172.21.15.98:0/389615374 wait complete. 2024-06-11T19:24:42.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.403+0000 7ffb16680640 1 librados: shutdown 2024-06-11T19:24:42.429 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-06-11T19:24:42.429 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-06-11T19:24:42.429 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:24:42.444 INFO:tasks.ceph.mon.h:Stopped 2024-06-11T19:24:42.444 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-06-11T19:24:42.444 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c store 2024-06-11T19:24:42.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c sync_force --yes-i-really-mean-it 2024-06-11T19:24:42.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.546+0000 7fc46be17640 10 monclient: get_monmap_and_config 2024-06-11T19:24:42.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.546+0000 7fc46be17640 10 monclient: build_initial_monmap 2024-06-11T19:24:42.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.546+0000 7fc46be17640 10 monclient: monmap: 2024-06-11T19:24:42.545 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:42.545 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.545 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:42.547170+0000 2024-06-11T19:24:42.545 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:42.547170+0000 2024-06-11T19:24:42.545 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:42.545 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.545 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:42.545 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:42.546 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:42.546 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:42.546 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:42.546 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:42.546 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:42.546 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:42.546 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:42.546 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:42.546 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:42.546 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.546+0000 7fc46be17640 1 Processor -- start 2024-06-11T19:24:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.546+0000 7fc46be17640 1 -- start start 2024-06-11T19:24:42.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.546+0000 7fc46be17640 10 monclient: init 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 10 monclient: _add_conns ranks=[1,4,8,2,6,9,0,3,7,5,10] 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc46409d330 addr v1:172.21.15.117:6789/0 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464136c80 0x7fc46413ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 10 monclient(hunting): picked mon.noname-c con 0x7fc464136c80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc46413bfd0 0x7fc46413e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 10 monclient(hunting): picked mon.noname-e con 0x7fc46413bfd0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc46409b5d0 con 0x7fc46409d330 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc4641376a0 con 0x7fc464136c80 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc46409b890 con 0x7fc46413bfd0 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46be17640 10 monclient(hunting): authenticate will time out at 1692.354404s 2024-06-11T19:24:42.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46a3b4640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc46413bfd0 0x7fc46413e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464136c80 0x7fc46413ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464136c80 0x7fc46413ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46960/0 (socket says 172.21.15.98:46960) 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46a3b4640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc46413bfd0 0x7fc46413e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:53226/0 (socket says 172.21.15.98:53226) 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 1 -- 172.21.15.98:0/1411890883 learned_addr learned my addr 172.21.15.98:0/1411890883 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 10 monclient(hunting): get_auth_request con 0x7fc464136c80 auth_method 0 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46a3b4640 10 monclient(hunting): get_auth_request con 0x7fc46413bfd0 auth_method 0 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46a3b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46a3b4640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46a3b4640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46a3b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46a3b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46a3b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 10 monclient(hunting): handle_auth_done global_id 114511 payload 274 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 1 -- 172.21.15.98:0/1411890883 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc46413bfd0 msgr2=0x7fc46413e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 1 --2- 172.21.15.98:0/1411890883 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc46413bfd0 0x7fc46413e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 1 -- 172.21.15.98:0/1411890883 >> v1:172.21.15.117:6789/0 conn(0x7fc46409d330 legacy=0x7fc46409d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 10 monclient: _finish_hunting 0 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 1 monclient: found mon.noname-c 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc4693b2640 1 -- 172.21.15.98:0/1411890883 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc464137080 con 0x7fc464136c80 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.547+0000 7fc46a3b4640 1 --2- 172.21.15.98:0/1411890883 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc46413bfd0 0x7fc46413e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc4693b2640 1 --2- 172.21.15.98:0/1411890883 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464136c80 0x7fc46413ba90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc454009880 tx=0x7fc45402b790 comp rx=0 tx=0).ready entity=mon.4 client_cookie=20fe8ebc4ca01b7c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 1 -- 172.21.15.98:0/1411890883 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc454038070 con 0x7fc464136c80 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 10 monclient: dump: 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.558 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 10 monclient: _finish_auth 0 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 10 monclient: _check_auth_tickets 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc46be17640 5 monclient: authenticate success, global_id 114511 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc46be17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 1 -- 172.21.15.98:0/1411890883 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4540044a0 con 0x7fc464136c80 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc46be17640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc46be17640 4 set_mon_vals no callback set 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 1 -- 172.21.15.98:0/1411890883 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc454003a80 con 0x7fc464136c80 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.548+0000 7fc468bb1640 10 monclient: dump: 2024-06-11T19:24:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.549+0000 7fc46be17640 10 monclient: shutdown 2024-06-11T19:24:42.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.549+0000 7fc46be17640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.549+0000 7fc46be17640 1 -- 172.21.15.98:0/1411890883 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464136c80 msgr2=0x7fc46413ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.549+0000 7fc46be17640 1 --2- 172.21.15.98:0/1411890883 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464136c80 0x7fc46413ba90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc454009880 tx=0x7fc45402b790 comp rx=0 tx=0).stop 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.549+0000 7fc46be17640 1 -- 172.21.15.98:0/1411890883 shutdown_connections 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.549+0000 7fc46be17640 1 --2- 172.21.15.98:0/1411890883 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc46413bfd0 0x7fc46413e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.549+0000 7fc46be17640 1 --2- 172.21.15.98:0/1411890883 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464136c80 0x7fc46413ba90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.549+0000 7fc46be17640 1 -- 172.21.15.98:0/1411890883 >> 172.21.15.98:0/1411890883 conn(0x7fc4640a7850 msgr2=0x7fc4640a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.549+0000 7fc46be17640 1 -- 172.21.15.98:0/1411890883 shutdown_connections 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.549+0000 7fc46be17640 1 -- 172.21.15.98:0/1411890883 wait complete. 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.549+0000 7fc46be17640 10 monclient: build_initial_monmap 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.549+0000 7fc46be17640 10 monclient: monmap: 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 1 librados: starting msgr at 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 1 librados: starting objecter 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 1 Processor -- start 2024-06-11T19:24:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 1 -- start start 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 1 librados: setting wanted keys 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 1 librados: calling monclient init 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 10 monclient: init 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 10 monclient: _add_conns ranks=[2,6,3,7,0,8,4,1,5] 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc46409d330 0x7fc464096a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 10 monclient(hunting): picked mon.noname-b con 0x7fc46409d330 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc464136c80 0x7fc46406f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 10 monclient(hunting): picked mon.noname-d con 0x7fc464136c80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc46413bfd0 0x7fc46406fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 10 monclient(hunting): picked mon.noname-g con 0x7fc46413bfd0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc46409de20 con 0x7fc46409d330 2024-06-11T19:24:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fc464076a00 con 0x7fc46413bfd0 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc46409e180 con 0x7fc464136c80 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc46be17640 10 monclient(hunting): authenticate will time out at 1692.357828s 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc469bb3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc46409d330 0x7fc464096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc4693b2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc464136c80 0x7fc46406f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc4693b2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc464136c80 0x7fc46406f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:35442/0 (socket says 172.21.15.98:35442) 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc469bb3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc46409d330 0x7fc464096a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57412/0 (socket says 172.21.15.98:57412) 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc4693b2640 1 -- 172.21.15.98:0/2763014729 learned_addr learned my addr 172.21.15.98:0/2763014729 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc4693b2640 10 monclient(hunting): get_auth_request con 0x7fc464136c80 auth_method 0 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc4693b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc4693b2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc4693b2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc469bb3640 10 monclient(hunting): get_auth_request con 0x7fc46409d330 auth_method 0 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc469bb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc469bb3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.550+0000 7fc469bb3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc469bb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc469bb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc469bb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 10 monclient(hunting): handle_auth_done global_id 124269 payload 824 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 1 -- 172.21.15.98:0/2763014729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc46413bfd0 msgr2=0x7fc46406fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc46413bfd0 0x7fc46406fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 1 -- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc46409d330 msgr2=0x7fc464096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc46409d330 0x7fc464096a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 10 monclient: _finish_hunting 0 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 1 monclient: found mon.noname-d 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 1 -- 172.21.15.98:0/2763014729 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc464073400 con 0x7fc464136c80 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc4693b2640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc464136c80 0x7fc46406f8e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc4540090b0 tx=0x7fc4540046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 10 client.?.objecter ms_handle_connect 0x7fc464136c80 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc454042020 con 0x7fc464136c80 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 10 monclient: dump: 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 10 monclient: _finish_auth 0 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:24:42.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc454005e00 con 0x7fc464136c80 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc46be17640 5 monclient: authenticate success, global_id 124269 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc46be17640 10 monclient: _renew_subs 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc46be17640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc46be17640 1 -- 172.21.15.98:0/2763014729 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc464071890 con 0x7fc464136c80 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc454038040 con 0x7fc464136c80 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc46be17640 10 client.124269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45bfff640 4 set_mon_vals no callback set 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc45a7fc640 10 monclient: dump: 2024-06-11T19:24:42.568 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc46be17640 10 monclient: _renew_subs 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc46be17640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc46be17640 1 -- 172.21.15.98:0/2763014729 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4640a7040 con 0x7fc464136c80 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.551+0000 7fc46be17640 1 librados: init done 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc46be17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc46be17640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464070360 0x7fc464138a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc46be17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc46be17640 1 -- 172.21.15.98:0/2763014729 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc464072170 con 0x7fc464070360 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc469bb3640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464070360 0x7fc464138a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc469bb3640 10 monclient: get_auth_request con 0x7fc464070360 auth_method 0 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc469bb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc469bb3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc469bb3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc469bb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc469bb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fc454045020 con 0x7fc464136c80 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc469bb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc469bb3640 10 monclient(hunting): handle_auth_done global_id 114520 payload 824 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc469bb3640 1 -- 172.21.15.98:0/2763014729 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc464070740 con 0x7fc464070360 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc45a7fc640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc44803fdb0 0x7fc448042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(627..627 src has 1..627) ==== 6823+0+0 (secure 0 0 0) 0x7fc454072fe0 con 0x7fc464136c80 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc45a7fc640 10 client.124269.objecter ms_dispatch 0x7fc464000b90 osd_map(627..627 src has 1..627) 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc45a7fc640 3 client.124269.objecter handle_osd_map got epochs [627,627] > 0 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc45a7fc640 3 client.124269.objecter handle_osd_map decoding full epoch 627 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc469bb3640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464070360 0x7fc464138a10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc46000f7a0 tx=0x7fc46000fc60 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc45a7fc640 20 client.124269.objecter dump_active .. 0 homeless 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc45a7fc640 10 client.124269.objecter ms_handle_connect 0x7fc464070360 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc45a7fc640 10 client.124269.objecter resend_mon_ops 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.552+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc460018030 con 0x7fc464070360 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc45a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc45a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc45a7fc640 10 monclient: dump: 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.573 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc46a3b4640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc44803fdb0 0x7fc448042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc46a3b4640 10 monclient: get_auth_request con 0x7fc44803fdb0 auth_method 0 2024-06-11T19:24:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc460003940 con 0x7fc464070360 2024-06-11T19:24:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc45a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc45a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464070360 msgr2=0x7fc464138a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc45a7fc640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464070360 0x7fc464138a10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc46000f7a0 tx=0x7fc46000fc60 comp rx=0 tx=0).stop 2024-06-11T19:24:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464070360 msgr2=0x7fc464138a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc45a7fc640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464070360 0x7fc464138a10 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc46000f7a0 tx=0x7fc46000fc60 comp rx=0 tx=0).stop 2024-06-11T19:24:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc45a7fc640 10 client.124269.objecter ms_handle_connect 0x7fc44803fdb0 2024-06-11T19:24:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.553+0000 7fc46a3b4640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc44803fdb0 0x7fc448042270 secure :-1 s=READY pgs=4302 cs=0 l=1 rev1=1 crypto rx=0x7fc464071ce0 tx=0x7fc45c00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc46be17640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:24:42.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc46be17640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464138f50 0x7fc464139330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc46be17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc46be17640 1 -- 172.21.15.98:0/2763014729 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc464050830 con 0x7fc464138f50 2024-06-11T19:24:42.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc469bb3640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464138f50 0x7fc464139330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc469bb3640 10 monclient: get_auth_request con 0x7fc464138f50 auth_method 0 2024-06-11T19:24:42.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc469bb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc469bb3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc469bb3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc469bb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc469bb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc469bb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc469bb3640 10 monclient(hunting): handle_auth_done global_id 114529 payload 824 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.560+0000 7fc469bb3640 1 -- 172.21.15.98:0/2763014729 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7fc4640a2f20 con 0x7fc464138f50 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 10 client.124269.objecter ms_handle_connect 0x7fc464138f50 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc469bb3640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464138f50 0x7fc464139330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc46000f7a0 tx=0x7fc460003760 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 10 client.124269.objecter resend_mon_ops 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc46000de20 con 0x7fc464138f50 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:24:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 10 monclient: dump: 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.578 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.578 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7fc46000d9b0 con 0x7fc464138f50 2024-06-11T19:24:42.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:24:42.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:42.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464138f50 msgr2=0x7fc464139330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464138f50 0x7fc464139330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc46000f7a0 tx=0x7fc460003760 comp rx=0 tx=0).stop 2024-06-11T19:24:42.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 1 -- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464138f50 msgr2=0x7fc464139330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:42.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc45a7fc640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc464138f50 0x7fc464139330 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.578 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:42.579 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-06-11T19:24:42.579 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-06-11T19:24:42.579 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:42.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 10 librados: watch_flush enter 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 10 librados: watch_flush exit 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 20 client.124269.objecter shutdown clearing up homeless session... 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 10 client.124269.objecter successfully canceled tick 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 1 -- 172.21.15.98:0/2763014729 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc44803fdb0 msgr2=0x7fc448042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc44803fdb0 0x7fc448042270 secure :-1 s=READY pgs=4302 cs=0 l=1 rev1=1 crypto rx=0x7fc464071ce0 tx=0x7fc45c00b040 comp rx=0 tx=0).stop 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 10 monclient: shutdown 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 1 -- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc464136c80 msgr2=0x7fc46406f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 1 --2- 172.21.15.98:0/2763014729 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc464136c80 0x7fc46406f8e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc4540090b0 tx=0x7fc4540046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46a3b4640 1 -- 172.21.15.98:0/2763014729 reap_dead start 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46a3b4640 1 -- 172.21.15.98:0/2763014729 reap_dead start 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 1 -- 172.21.15.98:0/2763014729 shutdown_connections 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 1 -- 172.21.15.98:0/2763014729 >> 172.21.15.98:0/2763014729 conn(0x7fc4640a7850 msgr2=0x7fc46409a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:42.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.561+0000 7fc46be17640 1 -- 172.21.15.98:0/2763014729 shutdown_connections 2024-06-11T19:24:42.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.562+0000 7fc46be17640 1 -- 172.21.15.98:0/2763014729 wait complete. 2024-06-11T19:24:42.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.562+0000 7fc46be17640 1 librados: shutdown 2024-06-11T19:24:42.582 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-06-11T19:24:42.582 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-06-11T19:24:42.582 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:24:42.597 INFO:tasks.ceph.mon.c:Stopped 2024-06-11T19:24:42.597 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-06-11T19:24:42.597 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i store 2024-06-11T19:24:42.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i sync_force --yes-i-really-mean-it 2024-06-11T19:24:42.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.738+0000 7f0218f8c640 10 monclient: get_monmap_and_config 2024-06-11T19:24:42.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.738+0000 7f0218f8c640 10 monclient: build_initial_monmap 2024-06-11T19:24:42.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.738+0000 7f0218f8c640 10 monclient: monmap: 2024-06-11T19:24:42.738 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:42.738 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.738 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:42.739838+0000 2024-06-11T19:24:42.738 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:42.739838+0000 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:42.739 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.739+0000 7f0218f8c640 1 Processor -- start 2024-06-11T19:24:42.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.739+0000 7f0218f8c640 1 -- start start 2024-06-11T19:24:42.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.739+0000 7f0218f8c640 10 monclient: init 2024-06-11T19:24:42.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:42.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 10 monclient: _add_conns ranks=[5,3,6,4,1,10,9,7,8,2,0] 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f02140a4920 0x7f02140a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 10 monclient(hunting): picked mon.noname-h con 0x7f02140a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f02140a5240 0x7f02140a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 10 monclient(hunting): picked mon.noname-g con 0x7f02140a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a2530 0x7f02140a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 10 monclient(hunting): picked mon.noname-d con 0x7f02140a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f02140a7140 con 0x7f02140a5240 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f02140a2b50 con 0x7f02140a4920 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f02140a2e30 con 0x7f02140a2530 2024-06-11T19:24:42.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0218f8c640 10 monclient(hunting): authenticate will time out at 1692.547153s 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f021259c640 1 -- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f02140a4920 msgr2=0x7f02140a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f021259c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f02140a4920 0x7f02140a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a2530 0x7f02140a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a2530 0x7f02140a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:35454/0 (socket says 172.21.15.98:35454) 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 1 -- 172.21.15.98:0/279340954 learned_addr learned my addr 172.21.15.98:0/279340954 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 10 monclient(hunting): get_auth_request con 0x7f02140a2530 auth_method 0 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0211d9b640 1 --2- 172.21.15.98:0/279340954 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f02140a5240 0x7f02140a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0211d9b640 10 monclient(hunting): get_auth_request con 0x7f02140a5240 auth_method 0 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0211d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0211d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0211d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0211d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0211d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0211d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 10 monclient(hunting): handle_auth_done global_id 124278 payload 274 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 1 -- 172.21.15.98:0/279340954 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f02140a4920 msgr2=0x7f02140a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 1 --2- 172.21.15.98:0/279340954 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f02140a4920 0x7f02140a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 1 -- 172.21.15.98:0/279340954 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f02140a5240 msgr2=0x7f02140a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 1 --2- 172.21.15.98:0/279340954 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f02140a5240 0x7f02140a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:24:42.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 1 monclient: found mon.noname-d 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.740+0000 7f0212d9d640 1 -- 172.21.15.98:0/279340954 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02140a3110 con 0x7f02140a2530 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f0212d9d640 1 --2- 172.21.15.98:0/279340954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a2530 0x7f02140a5ee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f020c009e80 tx=0x7f020c02bd90 comp rx=0 tx=0).ready entity=mon.6 client_cookie=fc1f0e1ac535b98e server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 1 -- 172.21.15.98:0/279340954 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f020c02cba0 con 0x7f02140a2530 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 10 monclient: dump: 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.753 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 10 monclient: _finish_auth 0 2024-06-11T19:24:42.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 10 monclient: _check_auth_tickets 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f0218f8c640 5 monclient: authenticate success, global_id 124278 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f0218f8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 1 -- 172.21.15.98:0/279340954 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f020c02cd60 con 0x7f02140a2530 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f0218f8c640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f0218f8c640 4 set_mon_vals no callback set 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 1 -- 172.21.15.98:0/279340954 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f020c034860 con 0x7f02140a2530 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f021159a640 10 monclient: dump: 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.756 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f0218f8c640 10 monclient: shutdown 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f0218f8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f0218f8c640 1 -- 172.21.15.98:0/279340954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a2530 msgr2=0x7f02140a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.741+0000 7f0218f8c640 1 --2- 172.21.15.98:0/279340954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a2530 0x7f02140a5ee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f020c009e80 tx=0x7f020c02bd90 comp rx=0 tx=0).stop 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.742+0000 7f0218f8c640 1 -- 172.21.15.98:0/279340954 shutdown_connections 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.742+0000 7f0218f8c640 1 --2- 172.21.15.98:0/279340954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a2530 0x7f02140a5ee0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.742+0000 7f0218f8c640 1 --2- 172.21.15.98:0/279340954 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f02140a5240 0x7f02140a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.742+0000 7f0218f8c640 1 --2- 172.21.15.98:0/279340954 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f02140a4920 0x7f02140a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.742+0000 7f0218f8c640 1 -- 172.21.15.98:0/279340954 >> 172.21.15.98:0/279340954 conn(0x7f02140ab850 msgr2=0x7f02140a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.742+0000 7f0218f8c640 1 -- 172.21.15.98:0/279340954 shutdown_connections 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.742+0000 7f0218f8c640 1 -- 172.21.15.98:0/279340954 wait complete. 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.742+0000 7f0218f8c640 10 monclient: build_initial_monmap 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.742+0000 7f0218f8c640 10 monclient: monmap: 2024-06-11T19:24:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.742+0000 7f0218f8c640 1 librados: starting msgr at 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.742+0000 7f0218f8c640 1 librados: starting objecter 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 1 Processor -- start 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 1 -- start start 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 1 librados: setting wanted keys 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 1 librados: calling monclient init 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 10 monclient: init 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 10 monclient: _add_conns ranks=[4,6,0,1,3,8,5,7,2] 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f02140a2530 0x7f02141cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 10 monclient(hunting): picked mon.noname-c con 0x7f02140a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a4920 0x7f02141ce050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 10 monclient(hunting): picked mon.noname-d con 0x7f02140a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02140a5240 0x7f0214124fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 10 monclient(hunting): picked mon.noname-a con 0x7f02140a5240 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02140a6da0 con 0x7f02140a5240 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0214138e50 con 0x7f02140a2530 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f02140a6420 con 0x7f02140a4920 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0218f8c640 10 monclient(hunting): authenticate will time out at 1692.550572s 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f021259c640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f02140a2530 msgr2=0x7f02141cdb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f021259c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f02140a2530 0x7f02141cdb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0211d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a4920 0x7f02141ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02140a5240 0x7f0214124fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0211d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a4920 0x7f02141ce050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:35464/0 (socket says 172.21.15.98:35464) 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0211d9b640 1 -- 172.21.15.98:0/2085128130 learned_addr learned my addr 172.21.15.98:0/2085128130 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0211d9b640 10 monclient(hunting): get_auth_request con 0x7f02140a4920 auth_method 0 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0211d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0211d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0211d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 10 monclient(hunting): get_auth_request con 0x7f02140a5240 auth_method 0 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0211d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0211d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0211d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 10 monclient(hunting): handle_auth_done global_id 66195 payload 824 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 1 -- 172.21.15.98:0/2085128130 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a4920 msgr2=0x7f02141ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a4920 0x7f02141ce050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 1 -- 172.21.15.98:0/2085128130 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f02140a2530 msgr2=0x7f02141cdb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f02140a2530 0x7f02141cdb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 1 monclient: found mon.noname-a 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0212d9d640 1 -- 172.21.15.98:0/2085128130 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0214128ef0 con 0x7f02140a5240 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.743+0000 7f0211d9b640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02140a4920 0x7f02141ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0212d9d640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02140a5240 0x7f0214124fb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f020c032000 tx=0x7f020c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 10 client.?.objecter ms_handle_connect 0x7f02140a5240 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f020c042020 con 0x7f02140a5240 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 10 monclient: dump: 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.763 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 10 monclient: _finish_auth 0 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f020c0046c0 con 0x7f02140a5240 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0218f8c640 5 monclient: authenticate success, global_id 66195 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f020c03e8c0 con 0x7f02140a5240 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0202ffd640 10 monclient: dump: 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.765 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0210d99640 4 set_mon_vals no callback set 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0218f8c640 10 monclient: _renew_subs 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0218f8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0218f8c640 1 -- 172.21.15.98:0/2085128130 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02140aabc0 con 0x7f02140a5240 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0218f8c640 10 client.66195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0218f8c640 10 monclient: _renew_subs 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0218f8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0218f8c640 1 -- 172.21.15.98:0/2085128130 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0214128c10 con 0x7f02140a5240 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.744+0000 7f0218f8c640 1 librados: init done 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0218f8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f020c045020 con 0x7f02140a5240 2024-06-11T19:24:42.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0218f8c640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f02141254f0 0x7f02141258d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0218f8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0218f8c640 1 -- 172.21.15.98:0/2085128130 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f0214050830 con 0x7f02141254f0 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0202ffd640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f01f403fde0 0x7f01f40422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(627..627 src has 1..627) ==== 6823+0+0 (secure 0 0 0) 0x7f020c03a040 con 0x7f02140a5240 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0202ffd640 10 client.66195.objecter ms_dispatch 0x7f0214000b90 osd_map(627..627 src has 1..627) 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0202ffd640 3 client.66195.objecter handle_osd_map got epochs [627,627] > 0 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0202ffd640 3 client.66195.objecter handle_osd_map decoding full epoch 627 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f021259c640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f02141254f0 0x7f02141258d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0202ffd640 20 client.66195.objecter dump_active .. 0 homeless 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f021259c640 10 monclient: get_auth_request con 0x7f02141254f0 auth_method 0 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f021259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f021259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f021259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0211d9b640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f01f403fde0 0x7f01f40422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f0211d9b640 10 monclient: get_auth_request con 0x7f01f403fde0 auth_method 0 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f021259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f021259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f021259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f021259c640 10 monclient(hunting): handle_auth_done global_id 84895 payload 824 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.745+0000 7f021259c640 1 -- 172.21.15.98:0/2085128130 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02140a6720 con 0x7f02141254f0 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 10 client.66195.objecter ms_handle_connect 0x7f02141254f0 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 10 client.66195.objecter resend_mon_ops 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 10 client.66195.objecter ms_handle_connect 0x7f01f403fde0 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f021259c640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f02141254f0 0x7f02141258d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f020400b450 tx=0x7f020400b910 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0211d9b640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f01f403fde0 0x7f01f40422a0 secure :-1 s=READY pgs=4303 cs=0 l=1 rev1=1 crypto rx=0x7f0214126790 tx=0x7f0208009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0204003bc0 con 0x7f02141254f0 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:24:42.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 10 monclient: dump: 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.769 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.770 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.770 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0204015070 con 0x7f02141254f0 2024-06-11T19:24:42.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:42.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f02141254f0 msgr2=0x7f02141258d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f02141254f0 0x7f02141258d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f020400b450 tx=0x7f020400b910 comp rx=0 tx=0).stop 2024-06-11T19:24:42.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f02141254f0 msgr2=0x7f02141258d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:42.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.746+0000 7f0202ffd640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f02141254f0 0x7f02141258d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.753+0000 7f0218f8c640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:24:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f0218f8c640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0214125e10 0x7f02141215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f0218f8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f0218f8c640 1 -- 172.21.15.98:0/2085128130 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f02140a8fc0 con 0x7f0214125e10 2024-06-11T19:24:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f021259c640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0214125e10 0x7f02141215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f021259c640 10 monclient: get_auth_request con 0x7f0214125e10 auth_method 0 2024-06-11T19:24:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f021259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f021259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f021259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f021259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f021259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f021259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f021259c640 10 monclient(hunting): handle_auth_done global_id 84904 payload 824 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f021259c640 1 -- 172.21.15.98:0/2085128130 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f021406c300 con 0x7f0214125e10 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f0202ffd640 10 client.66195.objecter ms_handle_connect 0x7f0214125e10 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f021259c640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0214125e10 0x7f02141215f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f020400bfd0 tx=0x7f02040059d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f0202ffd640 10 client.66195.objecter resend_mon_ops 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0204028080 con 0x7f0214125e10 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f0202ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f0202ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.754+0000 7f0202ffd640 10 monclient: dump: 2024-06-11T19:24:42.773 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.774 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.775 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f02040055b0 con 0x7f0214125e10 2024-06-11T19:24:42.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0202ffd640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-06-11T19:24:42.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0202ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:42.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0214125e10 msgr2=0x7f02141215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0202ffd640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0214125e10 0x7f02141215f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f020400bfd0 tx=0x7f02040059d0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0202ffd640 1 -- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0214125e10 msgr2=0x7f02141215f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:42.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0202ffd640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f0214125e10 0x7f02141215f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.775 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:42.775 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-06-11T19:24:42.775 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-06-11T19:24:42.775 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 10 librados: watch_flush enter 2024-06-11T19:24:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 10 librados: watch_flush exit 2024-06-11T19:24:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 20 client.66195.objecter shutdown clearing up homeless session... 2024-06-11T19:24:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 10 client.66195.objecter successfully canceled tick 2024-06-11T19:24:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 1 -- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f01f403fde0 msgr2=0x7f01f40422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f01f403fde0 0x7f01f40422a0 secure :-1 s=READY pgs=4303 cs=0 l=1 rev1=1 crypto rx=0x7f0214126790 tx=0x7f0208009190 comp rx=0 tx=0).stop 2024-06-11T19:24:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 10 monclient: shutdown 2024-06-11T19:24:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:42.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 1 -- 172.21.15.98:0/2085128130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02140a5240 msgr2=0x7f0214124fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 1 --2- 172.21.15.98:0/2085128130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02140a5240 0x7f0214124fb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f020c032000 tx=0x7f020c004a40 comp rx=0 tx=0).stop 2024-06-11T19:24:42.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0212d9d640 1 -- 172.21.15.98:0/2085128130 reap_dead start 2024-06-11T19:24:42.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0212d9d640 1 -- 172.21.15.98:0/2085128130 reap_dead start 2024-06-11T19:24:42.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 1 -- 172.21.15.98:0/2085128130 shutdown_connections 2024-06-11T19:24:42.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 1 -- 172.21.15.98:0/2085128130 >> 172.21.15.98:0/2085128130 conn(0x7f02140ab850 msgr2=0x7f021409c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:42.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 1 -- 172.21.15.98:0/2085128130 shutdown_connections 2024-06-11T19:24:42.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 1 -- 172.21.15.98:0/2085128130 wait complete. 2024-06-11T19:24:42.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.755+0000 7f0218f8c640 1 librados: shutdown 2024-06-11T19:24:42.778 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-06-11T19:24:42.778 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-06-11T19:24:42.779 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:24:42.798 INFO:tasks.ceph.mon.i:Stopped 2024-06-11T19:24:42.798 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 6 2024-06-11T19:24:42.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:24:42.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.898+0000 7fc2389da640 10 monclient: get_monmap_and_config 2024-06-11T19:24:42.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.898+0000 7fc2389da640 10 monclient: build_initial_monmap 2024-06-11T19:24:42.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.898+0000 7fc2389da640 10 monclient: monmap: 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:42.899850+0000 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:42.899850+0000 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:42.899 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:42.899 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:42.899 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:42.899 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 1 Processor -- start 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 1 -- start start 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 10 monclient: init 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 10 monclient: _add_conns ranks=[8,9,0,5,4,2,10,3,6,1,7] 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc2340a4920 0x7fc2340a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 10 monclient(hunting): picked mon.noname-e con 0x7fc2340a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2340a5240 0x7fc2340a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 10 monclient(hunting): picked mon.noname-a con 0x7fc2340a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2340a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2340a7140 con 0x7fc2340a2530 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc2340a2b50 con 0x7fc2340a5240 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc2340a2e30 con 0x7fc2340a4920 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.899+0000 7fc2389da640 10 monclient(hunting): authenticate will time out at 1692.706991s 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc23259c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc2340a4920 0x7fc2340a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2340a5240 0x7fc2340a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc23259c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc2340a4920 0x7fc2340a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:53236/0 (socket says 172.21.15.98:53236) 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2340a5240 0x7fc2340a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53804/0 (socket says 172.21.15.98:53804) 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc23259c640 1 -- 172.21.15.98:0/1229176350 learned_addr learned my addr 172.21.15.98:0/1229176350 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc23259c640 10 monclient(hunting): get_auth_request con 0x7fc2340a4920 auth_method 0 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc23259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc23259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc23259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 10 monclient(hunting): get_auth_request con 0x7fc2340a5240 auth_method 0 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc23259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc23259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc23259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 10 monclient(hunting): handle_auth_done global_id 66204 payload 274 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 1 -- 172.21.15.98:0/1229176350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc2340a4920 msgr2=0x7fc2340a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 1 --2- 172.21.15.98:0/1229176350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc2340a4920 0x7fc2340a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 1 -- 172.21.15.98:0/1229176350 >> v1:172.21.15.98:6789/0 conn(0x7fc2340a2530 legacy=0x7fc2340a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 1 monclient: found mon.noname-a 2024-06-11T19:24:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc229d9b640 1 -- 172.21.15.98:0/1229176350 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2340a3110 con 0x7fc2340a5240 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.900+0000 7fc23259c640 1 --2- 172.21.15.98:0/1229176350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc2340a4920 0x7fc2340a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc229d9b640 1 --2- 172.21.15.98:0/1229176350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2340a5240 0x7fc2340a59a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc21c0097a0 tx=0x7fc21c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d129d834dfbf055c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 1 -- 172.21.15.98:0/1229176350 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc21c039070 con 0x7fc2340a5240 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 10 monclient: dump: 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 10 monclient: _finish_auth 0 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 1 -- 172.21.15.98:0/1229176350 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc21c031960 con 0x7fc2340a5240 2024-06-11T19:24:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 5 monclient: authenticate success, global_id 66204 2024-06-11T19:24:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 1 -- 172.21.15.98:0/1229176350 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc21c032380 con 0x7fc2340a5240 2024-06-11T19:24:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc231d9b640 10 monclient: dump: 2024-06-11T19:24:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 4 set_mon_vals no callback set 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 10 monclient: shutdown 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 1 -- 172.21.15.98:0/1229176350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2340a5240 msgr2=0x7fc2340a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 1 --2- 172.21.15.98:0/1229176350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2340a5240 0x7fc2340a59a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc21c0097a0 tx=0x7fc21c02b790 comp rx=0 tx=0).stop 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 1 -- 172.21.15.98:0/1229176350 shutdown_connections 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 1 --2- 172.21.15.98:0/1229176350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2340a5240 0x7fc2340a59a0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 1 --2- 172.21.15.98:0/1229176350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc2340a4920 0x7fc2340a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 1 -- 172.21.15.98:0/1229176350 >> 172.21.15.98:0/1229176350 conn(0x7fc2340ab850 msgr2=0x7fc2340a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 1 -- 172.21.15.98:0/1229176350 shutdown_connections 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.901+0000 7fc2389da640 1 -- 172.21.15.98:0/1229176350 wait complete. 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.902+0000 7fc2389da640 10 monclient: build_initial_monmap 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.902+0000 7fc2389da640 10 monclient: monmap: 2024-06-11T19:24:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.902+0000 7fc2389da640 1 librados: starting msgr at 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.902+0000 7fc2389da640 1 librados: starting objecter 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 1 Processor -- start 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 1 -- start start 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 1 librados: setting wanted keys 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 1 librados: calling monclient init 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 10 monclient: init 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 10 monclient: _add_conns ranks=[6,0,5,4,2,3,1,7,8] 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc2340a2530 0x7fc234137170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 10 monclient(hunting): picked mon.noname-d con 0x7fc2340a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2340a4920 0x7fc2341376b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 10 monclient(hunting): picked mon.noname-a con 0x7fc2340a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:42.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2340a5240 0x7fc234123130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 10 monclient(hunting): picked mon.noname-h con 0x7fc2340a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2340a6da0 con 0x7fc2340a4920 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fc234138e50 con 0x7fc2340a5240 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc2340a65f0 con 0x7fc2340a2530 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc2389da640 10 monclient(hunting): authenticate will time out at 1692.710632s 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc23259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc2340a2530 0x7fc234137170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2340a4920 0x7fc2341376b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc23259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc2340a2530 0x7fc234137170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:35480/0 (socket says 172.21.15.98:35480) 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc23259c640 1 -- 172.21.15.98:0/4267153626 learned_addr learned my addr 172.21.15.98:0/4267153626 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 10 monclient(hunting): get_auth_request con 0x7fc2340a4920 auth_method 0 2024-06-11T19:24:42.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc23259c640 10 monclient(hunting): get_auth_request con 0x7fc2340a2530 auth_method 0 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc23259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc23259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc23259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc232d9d640 1 -- 172.21.15.98:0/4267153626 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2340a5240 msgr2=0x7fc234123130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc232d9d640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2340a5240 0x7fc234123130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc23259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc23259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc23259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 10 monclient(hunting): handle_auth_done global_id 66213 payload 824 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 1 -- 172.21.15.98:0/4267153626 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc2340a2530 msgr2=0x7fc234137170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc2340a2530 0x7fc234137170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 1 -- 172.21.15.98:0/4267153626 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2340a5240 msgr2=0x7fc234123130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2340a5240 0x7fc234123130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 1 monclient: found mon.noname-a 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.903+0000 7fc229d9b640 1 -- 172.21.15.98:0/4267153626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc234126d70 con 0x7fc2340a4920 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc229d9b640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2340a4920 0x7fc2341376b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc21c033000 tx=0x7fc21c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 10 client.?.objecter ms_handle_connect 0x7fc2340a4920 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 1 -- 172.21.15.98:0/4267153626 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc21c043030 con 0x7fc2340a4920 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 10 monclient: dump: 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.920 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 1 -- 172.21.15.98:0/4267153626 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc21c0044b0 con 0x7fc2340a4920 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc2389da640 5 monclient: authenticate success, global_id 66213 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc2389da640 10 monclient: _renew_subs 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc2389da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc2389da640 1 -- 172.21.15.98:0/4267153626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc234127050 con 0x7fc2340a4920 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 1 -- 172.21.15.98:0/4267153626 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc21c039040 con 0x7fc2340a4920 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc22b7fe640 10 monclient: dump: 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc23159a640 4 set_mon_vals no callback set 2024-06-11T19:24:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc2389da640 10 client.66213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc2389da640 10 monclient: _renew_subs 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc2389da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc2389da640 1 -- 172.21.15.98:0/4267153626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2340aace0 con 0x7fc2340a4920 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.904+0000 7fc2389da640 1 librados: init done 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc22b7fe640 1 -- 172.21.15.98:0/4267153626 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fc21c046070 con 0x7fc2340a4920 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc2389da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc2389da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc2389da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc2389da640 1 -- 172.21.15.98:0/4267153626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2340a7360 con 0x7fc2340a4920 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc22b7fe640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc21403fdb0 0x7fc214042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc22b7fe640 1 -- 172.21.15.98:0/4267153626 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(627..627 src has 1..627) ==== 6823+0+0 (secure 0 0 0) 0x7fc21c044070 con 0x7fc2340a4920 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc22b7fe640 10 client.66213.objecter ms_dispatch 0x7fc234000b90 osd_map(627..627 src has 1..627) 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc22b7fe640 3 client.66213.objecter handle_osd_map got epochs [627,627] > 0 2024-06-11T19:24:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc22b7fe640 3 client.66213.objecter handle_osd_map decoding full epoch 627 2024-06-11T19:24:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc22b7fe640 20 client.66213.objecter dump_active .. 0 homeless 2024-06-11T19:24:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc23259c640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc21403fdb0 0x7fc214042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.905+0000 7fc23259c640 10 monclient: get_auth_request con 0x7fc21403fdb0 auth_method 0 2024-06-11T19:24:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.906+0000 7fc22b7fe640 10 client.66213.objecter ms_handle_connect 0x7fc21403fdb0 2024-06-11T19:24:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.906+0000 7fc23259c640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc21403fdb0 0x7fc214042270 secure :-1 s=READY pgs=4304 cs=0 l=1 rev1=1 crypto rx=0x7fc224009630 tx=0x7fc224009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.910+0000 7fc22b7fe640 1 -- 172.21.15.98:0/4267153626 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc2340a7360 con 0x7fc2340a4920 2024-06-11T19:24:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.910+0000 7fc22b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:42.910+0000 7fc22b7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:43.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.161+0000 7fc2389da640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:24:43.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.161+0000 7fc2389da640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:43.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.161+0000 7fc2389da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:43.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.161+0000 7fc2389da640 1 -- 172.21.15.98:0/4267153626 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc234137f80 con 0x7fc2340a4920 2024-06-11T19:24:43.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.161+0000 7fc22b7fe640 1 -- 172.21.15.98:0/4267153626 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fc21c04d020 con 0x7fc2340a4920 2024-06-11T19:24:43.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.161+0000 7fc22b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:43.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.161+0000 7fc22b7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:43.161 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":82,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:24:43.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 10 librados: watch_flush enter 2024-06-11T19:24:43.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 10 librados: watch_flush exit 2024-06-11T19:24:43.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 20 client.66213.objecter shutdown clearing up homeless session... 2024-06-11T19:24:43.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 10 client.66213.objecter successfully canceled tick 2024-06-11T19:24:43.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 1 -- 172.21.15.98:0/4267153626 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc21403fdb0 msgr2=0x7fc214042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc21403fdb0 0x7fc214042270 secure :-1 s=READY pgs=4304 cs=0 l=1 rev1=1 crypto rx=0x7fc224009630 tx=0x7fc224009240 comp rx=0 tx=0).stop 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 10 monclient: shutdown 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 1 -- 172.21.15.98:0/4267153626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2340a4920 msgr2=0x7fc2341376b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2340a4920 0x7fc2341376b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc21c033000 tx=0x7fc21c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 1 -- 172.21.15.98:0/4267153626 shutdown_connections 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc21403fdb0 0x7fc214042270 unknown :-1 s=CLOSED pgs=4304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fc2340a5240 0x7fc234123130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2340a4920 0x7fc2341376b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 1 --2- 172.21.15.98:0/4267153626 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc2340a2530 0x7fc234137170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 1 -- 172.21.15.98:0/4267153626 >> 172.21.15.98:0/4267153626 conn(0x7fc2340ab850 msgr2=0x7fc2341be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 1 -- 172.21.15.98:0/4267153626 shutdown_connections 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.163+0000 7fc2389da640 1 -- 172.21.15.98:0/4267153626 wait complete. 2024-06-11T19:24:43.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:43.164+0000 7fc2389da640 1 librados: shutdown 2024-06-11T19:24:46.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:24:46.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.286+0000 7fe4b21a0640 10 monclient: get_monmap_and_config 2024-06-11T19:24:46.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.286+0000 7fe4b21a0640 10 monclient: build_initial_monmap 2024-06-11T19:24:46.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.286+0000 7fe4b21a0640 10 monclient: monmap: 2024-06-11T19:24:46.285 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:46.285 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:46.285 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:46.287060+0000 2024-06-11T19:24:46.285 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:46.287060+0000 2024-06-11T19:24:46.285 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:46.285 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:46.286 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:46.286 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:46.286 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:46.286 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:46.286 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:46.286 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:46.286 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:46.286 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:46.286 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:46.286 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:46.286 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:46.286 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.286+0000 7fe4b21a0640 1 Processor -- start 2024-06-11T19:24:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.286+0000 7fe4b21a0640 1 -- start start 2024-06-11T19:24:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.286+0000 7fe4b21a0640 10 monclient: init 2024-06-11T19:24:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 10 monclient: _add_conns ranks=[8,7,1,2,9,5,0,6,10,3,4] 2024-06-11T19:24:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4ac0a48f0 0x7fe4ac0a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 10 monclient(hunting): picked mon.noname-e con 0x7fe4ac0a48f0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4ac0a4cd0 0x7fe4ac138710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 10 monclient(hunting): picked mon.noname-i con 0x7fe4ac0a4cd0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe4ac0a50b0 addr v1:172.21.15.117:6789/0 2024-06-11T19:24:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4ac073aa0 con 0x7fe4ac0a50b0 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fe4ac0a2bf0 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe4ac0a2ed0 con 0x7fe4ac0a48f0 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b21a0640 10 monclient(hunting): authenticate will time out at 1696.094425s 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4ac0a48f0 0x7fe4ac0a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4ac0a48f0 0x7fe4ac0a5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:52974/0 (socket says 172.21.15.98:52974) 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 1 -- 172.21.15.98:0/1961879550 learned_addr learned my addr 172.21.15.98:0/1961879550 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 10 monclient(hunting): get_auth_request con 0x7fe4ac0a48f0 auth_method 0 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b099d640 1 -- 172.21.15.98:0/1961879550 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4ac0a4cd0 msgr2=0x7fe4ac138710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:24:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b099d640 1 --2- 172.21.15.98:0/1961879550 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4ac0a4cd0 0x7fe4ac138710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 10 monclient(hunting): handle_auth_done global_id 124262 payload 274 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 1 -- 172.21.15.98:0/1961879550 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4ac0a4cd0 msgr2=0x7fe4ac138710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 1 --2- 172.21.15.98:0/1961879550 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4ac0a4cd0 0x7fe4ac138710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 1 -- 172.21.15.98:0/1961879550 >> v1:172.21.15.117:6789/0 conn(0x7fe4ac0a50b0 legacy=0x7fe4ac138c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 10 monclient: _finish_hunting 0 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 1 monclient: found mon.noname-e 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.287+0000 7fe4b119e640 1 -- 172.21.15.98:0/1961879550 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4ac0a31b0 con 0x7fe4ac0a48f0 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4b119e640 1 --2- 172.21.15.98:0/1961879550 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4ac0a48f0 0x7fe4ac0a5ca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40098a0 tx=0x7fe4a402b7c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b481bc5c10e64cf0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 1 -- 172.21.15.98:0/1961879550 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a4038070 con 0x7fe4ac0a48f0 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:46.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 10 monclient: dump: 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:46.297 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 10 monclient: _finish_auth 0 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4b21a0640 5 monclient: authenticate success, global_id 124262 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 1 -- 172.21.15.98:0/1961879550 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4a4031990 con 0x7fe4ac0a48f0 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4b21a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4b21a0640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4b21a0640 4 set_mon_vals no callback set 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 1 -- 172.21.15.98:0/1961879550 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a40323b0 con 0x7fe4ac0a48f0 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:46.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.288+0000 7fe4abfff640 10 monclient: dump: 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:46.299 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.289+0000 7fe4b21a0640 10 monclient: shutdown 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.289+0000 7fe4b21a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.289+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1961879550 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4ac0a48f0 msgr2=0x7fe4ac0a5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.289+0000 7fe4b21a0640 1 --2- 172.21.15.98:0/1961879550 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4ac0a48f0 0x7fe4ac0a5ca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40098a0 tx=0x7fe4a402b7c0 comp rx=0 tx=0).stop 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.289+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1961879550 shutdown_connections 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.289+0000 7fe4b21a0640 1 --2- 172.21.15.98:0/1961879550 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4ac0a4cd0 0x7fe4ac138710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.289+0000 7fe4b21a0640 1 --2- 172.21.15.98:0/1961879550 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4ac0a48f0 0x7fe4ac0a5ca0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.289+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1961879550 >> 172.21.15.98:0/1961879550 conn(0x7fe4ac0ab580 msgr2=0x7fe4ac0a78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.289+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1961879550 shutdown_connections 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.289+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1961879550 wait complete. 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.289+0000 7fe4b21a0640 10 monclient: build_initial_monmap 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.289+0000 7fe4b21a0640 10 monclient: monmap: 2024-06-11T19:24:46.300 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:46.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 1 librados: starting msgr at 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 1 librados: starting objecter 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 1 Processor -- start 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 1 -- start start 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 1 librados: setting wanted keys 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 1 librados: calling monclient init 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 10 monclient: init 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 10 monclient: _add_conns ranks=[7,0,4,3,5,6,8,2,1] 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4ac0a48f0 0x7fe4ac1cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 10 monclient(hunting): picked mon.noname-i con 0x7fe4ac0a48f0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4ac0a4cd0 0x7fe4ac137a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 10 monclient(hunting): picked mon.noname-a con 0x7fe4ac0a4cd0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0a50b0 0x7fe4ac137fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:46.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 10 monclient(hunting): picked mon.noname-c con 0x7fe4ac0a50b0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ac139360 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe4ac0a6d90 con 0x7fe4ac0a50b0 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fe4ac1394e0 con 0x7fe4ac0a48f0 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b21a0640 10 monclient(hunting): authenticate will time out at 1696.097803s 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b199f640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0a50b0 msgr2=0x7fe4ac137fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b199f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0a50b0 0x7fe4ac137fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b119e640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4ac0a48f0 msgr2=0x7fe4ac1cd900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b119e640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4ac0a48f0 0x7fe4ac1cd900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b099d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4ac0a4cd0 0x7fe4ac137a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b099d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4ac0a4cd0 0x7fe4ac137a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41928/0 (socket says 172.21.15.98:41928) 2024-06-11T19:24:46.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b099d640 1 -- 172.21.15.98:0/1701507762 learned_addr learned my addr 172.21.15.98:0/1701507762 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b099d640 10 monclient(hunting): get_auth_request con 0x7fe4ac0a4cd0 auth_method 0 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b099d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b099d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.290+0000 7fe4b099d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 10 monclient(hunting): handle_auth_done global_id 66222 payload 824 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 1 -- 172.21.15.98:0/1701507762 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4ac0a48f0 msgr2=0x7fe4ac1cd900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 1 --2- 172.21.15.98:0/1701507762 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4ac0a48f0 0x7fe4ac1cd900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 1 -- 172.21.15.98:0/1701507762 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0a50b0 msgr2=0x7fe4ac137fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 1 --2- 172.21.15.98:0/1701507762 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0a50b0 0x7fe4ac137fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 10 monclient: _finish_hunting 0 2024-06-11T19:24:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 1 monclient: found mon.noname-a 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 1 -- 172.21.15.98:0/1701507762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4ac127640 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b099d640 1 --2- 172.21.15.98:0/1701507762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4ac0a4cd0 0x7fe4ac137a90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000be50 tx=0x7fe4a000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 10 client.?.objecter ms_handle_connect 0x7fe4ac0a4cd0 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1701507762 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a000cdb0 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 10 monclient: dump: 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 10 monclient: _finish_auth 0 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1701507762 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4a0010070 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b21a0640 5 monclient: authenticate success, global_id 66222 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b21a0640 10 monclient: _renew_subs 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b21a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1701507762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4ac0a9850 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1701507762 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a0015d50 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4a9ffb640 10 monclient: dump: 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:46.307 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b21a0640 10 client.66222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b21a0640 10 monclient: _renew_subs 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b21a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4ab7fe640 4 set_mon_vals no callback set 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1701507762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4ac1271a0 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.291+0000 7fe4b21a0640 1 librados: init done 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.292+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1701507762 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fe4a0015470 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.292+0000 7fe4a9ffb640 1 --2- 172.21.15.98:0/1701507762 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe49803fdb0 0x7fe498042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.292+0000 7fe4b21a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.292+0000 7fe4b21a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.292+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1701507762 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(627..627 src has 1..627) ==== 6823+0+0 (secure 0 0 0) 0x7fe4a0052290 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.292+0000 7fe4b21a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.292+0000 7fe4a9ffb640 10 client.66222.objecter ms_dispatch 0x7fe4ac000b90 osd_map(627..627 src has 1..627) 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.292+0000 7fe4a9ffb640 3 client.66222.objecter handle_osd_map got epochs [627,627] > 0 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.292+0000 7fe4a9ffb640 3 client.66222.objecter handle_osd_map decoding full epoch 627 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.292+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1701507762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4ac0503d0 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.293+0000 7fe4a9ffb640 20 client.66222.objecter dump_active .. 0 homeless 2024-06-11T19:24:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.293+0000 7fe4b119e640 1 --2- 172.21.15.98:0/1701507762 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe49803fdb0 0x7fe498042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:46.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.293+0000 7fe4b119e640 10 monclient: get_auth_request con 0x7fe49803fdb0 auth_method 0 2024-06-11T19:24:46.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.293+0000 7fe4a9ffb640 10 client.66222.objecter ms_handle_connect 0x7fe49803fdb0 2024-06-11T19:24:46.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.293+0000 7fe4b119e640 1 --2- 172.21.15.98:0/1701507762 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe49803fdb0 0x7fe498042270 secure :-1 s=READY pgs=4305 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4035000 tx=0x7fe4a4030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:46.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.297+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1701507762 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe4a0006e20 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.297+0000 7fe4a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:46.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.297+0000 7fe4a9ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.550+0000 7fe4b21a0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:24:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.550+0000 7fe4b21a0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:46.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.550+0000 7fe4b21a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.550+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1701507762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe4ac0a1b50 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.551+0000 7fe4a9ffb640 1 -- 172.21.15.98:0/1701507762 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fe4ac0a1b50 con 0x7fe4ac0a4cd0 2024-06-11T19:24:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.551+0000 7fe4a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:46.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.551+0000 7fe4a9ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:46.550 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":82,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:24:46.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.552+0000 7fe4b21a0640 10 librados: watch_flush enter 2024-06-11T19:24:46.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.552+0000 7fe4b21a0640 10 librados: watch_flush exit 2024-06-11T19:24:46.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.552+0000 7fe4b21a0640 20 client.66222.objecter shutdown clearing up homeless session... 2024-06-11T19:24:46.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.552+0000 7fe4b21a0640 10 client.66222.objecter successfully canceled tick 2024-06-11T19:24:46.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1701507762 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe49803fdb0 msgr2=0x7fe498042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 --2- 172.21.15.98:0/1701507762 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe49803fdb0 0x7fe498042270 secure :-1 s=READY pgs=4305 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4035000 tx=0x7fe4a4030000 comp rx=0 tx=0).stop 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 10 monclient: shutdown 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1701507762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4ac0a4cd0 msgr2=0x7fe4ac137a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 --2- 172.21.15.98:0/1701507762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4ac0a4cd0 0x7fe4ac137a90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000be50 tx=0x7fe4a000bf40 comp rx=0 tx=0).stop 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1701507762 shutdown_connections 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 --2- 172.21.15.98:0/1701507762 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fe49803fdb0 0x7fe498042270 unknown :-1 s=CLOSED pgs=4305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 --2- 172.21.15.98:0/1701507762 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4ac0a50b0 0x7fe4ac137fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 --2- 172.21.15.98:0/1701507762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4ac0a4cd0 0x7fe4ac137a90 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 --2- 172.21.15.98:0/1701507762 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fe4ac0a48f0 0x7fe4ac1cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1701507762 >> 172.21.15.98:0/1701507762 conn(0x7fe4ac0ab580 msgr2=0x7fe4ac0a5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1701507762 shutdown_connections 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 -- 172.21.15.98:0/1701507762 wait complete. 2024-06-11T19:24:46.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:46.553+0000 7fe4b21a0640 1 librados: shutdown 2024-06-11T19:24:49.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:24:49.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.673+0000 7fd1c0bfc640 10 monclient: get_monmap_and_config 2024-06-11T19:24:49.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.673+0000 7fd1c0bfc640 10 monclient: build_initial_monmap 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.673+0000 7fd1c0bfc640 10 monclient: monmap: 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:49.674754+0000 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:49.674754+0000 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:49.674 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.674+0000 7fd1c0bfc640 1 Processor -- start 2024-06-11T19:24:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.674+0000 7fd1c0bfc640 1 -- start start 2024-06-11T19:24:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.674+0000 7fd1c0bfc640 10 monclient: init 2024-06-11T19:24:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.674+0000 7fd1c0bfc640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.674+0000 7fd1c0bfc640 10 monclient: _add_conns ranks=[0,4,8,1,2,9,3,10,6,5,7] 2024-06-11T19:24:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1c0bfc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1bc0a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:24:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1c0bfc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1bc0a2530 0x7fd1bc13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1c0bfc640 10 monclient(hunting): picked mon.noname-c con 0x7fd1bc0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1c0bfc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc13e1a0 0x7fd1bc140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1c0bfc640 10 monclient(hunting): picked mon.noname-e con 0x7fd1bc13e1a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1c0bfc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1bc0a7140 con 0x7fd1bc0a4920 2024-06-11T19:24:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1c0bfc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1c0bfc640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd1bc0a2b50 con 0x7fd1bc0a2530 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1c0bfc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1c0bfc640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd1bc0a2e30 con 0x7fd1bc13e1a0 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1c0bfc640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1b3fff640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1bc0a2530 msgr2=0x7fd1bc13dc60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1c0bfc640 10 monclient(hunting): authenticate will time out at 1699.482119s 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1b3fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1bc0a2530 0x7fd1bc13dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1ba59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd1bc0a4920 0x7fd1bc0a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42344/0 (socket says 172.21.15.98:42344) 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1ba59c640 1 -- 172.21.15.98:0/41866309 learned_addr learned my addr 172.21.15.98:0/41866309 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 1 --2- 172.21.15.98:0/41866309 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc13e1a0 0x7fd1bc140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 10 monclient(hunting): get_auth_request con 0x7fd1bc13e1a0 auth_method 0 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 10 monclient(hunting): handle_auth_done global_id 124271 payload 274 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 1 -- 172.21.15.98:0/41866309 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1bc0a2530 msgr2=0x7fd1bc13dc60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 1 --2- 172.21.15.98:0/41866309 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1bc0a2530 0x7fd1bc13dc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 1 -- 172.21.15.98:0/41866309 >> v1:172.21.15.98:6789/0 conn(0x7fd1bc0a4920 legacy=0x7fd1bc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 1 monclient: found mon.noname-e 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.675+0000 7fd1bad9d640 1 -- 172.21.15.98:0/41866309 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1bc0a3110 con 0x7fd1bc13e1a0 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1bad9d640 1 --2- 172.21.15.98:0/41866309 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc13e1a0 0x7fd1bc140590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4009dc0 tx=0x7fd1b402bd70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=bb4f3ad7b00657c3 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 1 -- 172.21.15.98:0/41866309 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1b4039070 con 0x7fd1bc13e1a0 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 10 monclient: dump: 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 10 monclient: _finish_auth 0 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1c0bfc640 5 monclient: authenticate success, global_id 124271 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 1 -- 172.21.15.98:0/41866309 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b402cd90 con 0x7fd1bc13e1a0 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1c0bfc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1c0bfc640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1c0bfc640 4 set_mon_vals no callback set 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 1 -- 172.21.15.98:0/41866309 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1b40327b0 con 0x7fd1bc13e1a0 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1b9d9b640 10 monclient: dump: 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1c0bfc640 10 monclient: shutdown 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1c0bfc640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/41866309 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc13e1a0 msgr2=0x7fd1bc140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.676+0000 7fd1c0bfc640 1 --2- 172.21.15.98:0/41866309 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc13e1a0 0x7fd1bc140590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4009dc0 tx=0x7fd1b402bd70 comp rx=0 tx=0).stop 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.677+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/41866309 shutdown_connections 2024-06-11T19:24:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.677+0000 7fd1c0bfc640 1 --2- 172.21.15.98:0/41866309 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1bc0a2530 0x7fd1bc13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.677+0000 7fd1c0bfc640 1 --2- 172.21.15.98:0/41866309 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc13e1a0 0x7fd1bc140590 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.677+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/41866309 >> 172.21.15.98:0/41866309 conn(0x7fd1bc0ab850 msgr2=0x7fd1bc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.677+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/41866309 shutdown_connections 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.677+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/41866309 wait complete. 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.677+0000 7fd1c0bfc640 10 monclient: build_initial_monmap 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.677+0000 7fd1c0bfc640 10 monclient: monmap: 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 1 librados: starting msgr at 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 1 librados: starting objecter 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 1 Processor -- start 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 1 -- start start 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 1 librados: setting wanted keys 2024-06-11T19:24:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 1 librados: calling monclient init 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 10 monclient: init 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 10 monclient: _add_conns ranks=[2,8,3,7,5,4,1,6,0] 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1bc0a2530 0x7fd1bc0a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 10 monclient(hunting): picked mon.noname-b con 0x7fd1bc0a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc0a4920 0x7fd1bc1ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 10 monclient(hunting): picked mon.noname-e con 0x7fd1bc0a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1bc13e1a0 0x7fd1bc135e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 10 monclient(hunting): picked mon.noname-g con 0x7fd1bc13e1a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd1bc138e50 con 0x7fd1bc0a2530 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd1bc0a84b0 con 0x7fd1bc13e1a0 2024-06-11T19:24:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd1bc0a6da0 con 0x7fd1bc0a4920 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1c0bfc640 10 monclient(hunting): authenticate will time out at 1699.485713s 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1b3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc0a4920 0x7fd1bc1ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1ba59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1bc0a2530 0x7fd1bc0a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1b3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc0a4920 0x7fd1bc1ca9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:52996/0 (socket says 172.21.15.98:52996) 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1b3fff640 1 -- 172.21.15.98:0/789049685 learned_addr learned my addr 172.21.15.98:0/789049685 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1b3fff640 10 monclient(hunting): get_auth_request con 0x7fd1bc0a4920 auth_method 0 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1b3fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1ba59c640 10 monclient(hunting): get_auth_request con 0x7fd1bc0a2530 auth_method 0 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1ba59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.678+0000 7fd1bad9d640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1bc13e1a0 0x7fd1bc135e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1bad9d640 10 monclient(hunting): get_auth_request con 0x7fd1bc13e1a0 auth_method 0 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1bad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b3fff640 10 monclient(hunting): handle_auth_done global_id 124280 payload 824 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b3fff640 1 -- 172.21.15.98:0/789049685 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1bc13e1a0 msgr2=0x7fd1bc135e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b3fff640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1bc13e1a0 0x7fd1bc135e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b3fff640 1 -- 172.21.15.98:0/789049685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1bc0a2530 msgr2=0x7fd1bc0a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b3fff640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1bc0a2530 0x7fd1bc0a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b3fff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b3fff640 1 monclient: found mon.noname-e 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b3fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b3fff640 1 -- 172.21.15.98:0/789049685 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1bc13dd60 con 0x7fd1bc0a4920 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1bad9d640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1bc13e1a0 0x7fd1bc135e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1ba59c640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1bc0a2530 0x7fd1bc0a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b3fff640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc0a4920 0x7fd1bc1ca9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd1a400b450 tx=0x7fd1a400b910 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 10 client.?.objecter ms_handle_connect 0x7fd1bc0a4920 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 1 -- 172.21.15.98:0/789049685 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1a4003bc0 con 0x7fd1bc0a4920 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 10 monclient: dump: 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 10 monclient: _finish_auth 0 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1c0bfc640 5 monclient: authenticate success, global_id 124280 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 1 -- 172.21.15.98:0/789049685 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1a4015070 con 0x7fd1bc0a4920 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1c0bfc640 10 monclient: _renew_subs 2024-06-11T19:24:49.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1c0bfc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/789049685 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1bc0a04b0 con 0x7fd1bc0a4920 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 1 -- 172.21.15.98:0/789049685 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1a4003da0 con 0x7fd1bc0a4920 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1c0bfc640 10 client.124280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b959a640 4 set_mon_vals no callback set 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1b2ffd640 10 monclient: dump: 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1c0bfc640 10 monclient: _renew_subs 2024-06-11T19:24:49.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1c0bfc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/789049685 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1bc122820 con 0x7fd1bc0a4920 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.679+0000 7fd1c0bfc640 1 librados: init done 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1b2ffd640 1 -- 172.21.15.98:0/789049685 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd1a400f070 con 0x7fd1bc0a4920 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1c0bfc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1c0bfc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1c0bfc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/789049685 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1bc138860 con 0x7fd1bc0a4920 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1b2ffd640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd19c03fdb0 0x7fd19c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1b2ffd640 1 -- 172.21.15.98:0/789049685 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(627..627 src has 1..627) ==== 6823+0+0 (secure 0 0 0) 0x7fd1a40514c0 con 0x7fd1bc0a4920 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1b2ffd640 10 client.124280.objecter ms_dispatch 0x7fd1bc000b90 osd_map(627..627 src has 1..627) 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1b2ffd640 3 client.124280.objecter handle_osd_map got epochs [627,627] > 0 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1b2ffd640 3 client.124280.objecter handle_osd_map decoding full epoch 627 2024-06-11T19:24:49.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1b2ffd640 20 client.124280.objecter dump_active .. 0 homeless 2024-06-11T19:24:49.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1ba59c640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd19c03fdb0 0x7fd19c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:49.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.680+0000 7fd1ba59c640 10 monclient: get_auth_request con 0x7fd19c03fdb0 auth_method 0 2024-06-11T19:24:49.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.681+0000 7fd1b2ffd640 10 client.124280.objecter ms_handle_connect 0x7fd19c03fdb0 2024-06-11T19:24:49.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.681+0000 7fd1ba59c640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd19c03fdb0 0x7fd19c042270 secure :-1 s=READY pgs=4306 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac000f60 tx=0x7fd1ac007ea0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:49.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.685+0000 7fd1b2ffd640 1 -- 172.21.15.98:0/789049685 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd1bc138860 con 0x7fd1bc0a4920 2024-06-11T19:24:49.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.685+0000 7fd1b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:49.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.685+0000 7fd1b2ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:49.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.936+0000 7fd1c0bfc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:24:49.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.936+0000 7fd1c0bfc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:49.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.936+0000 7fd1c0bfc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:49.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.936+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/789049685 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd1bc0a4d00 con 0x7fd1bc0a4920 2024-06-11T19:24:49.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.937+0000 7fd1b2ffd640 1 -- 172.21.15.98:0/789049685 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7fd1bc13dd60 con 0x7fd1bc0a4920 2024-06-11T19:24:49.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.937+0000 7fd1b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:49.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.937+0000 7fd1b2ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:49.937 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":82,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 10 librados: watch_flush enter 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 10 librados: watch_flush exit 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 20 client.124280.objecter shutdown clearing up homeless session... 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 10 client.124280.objecter successfully canceled tick 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/789049685 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd19c03fdb0 msgr2=0x7fd19c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd19c03fdb0 0x7fd19c042270 secure :-1 s=READY pgs=4306 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac000f60 tx=0x7fd1ac007ea0 comp rx=0 tx=0).stop 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 10 monclient: shutdown 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/789049685 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc0a4920 msgr2=0x7fd1bc1ca9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc0a4920 0x7fd1bc1ca9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd1a400b450 tx=0x7fd1a400b910 comp rx=0 tx=0).stop 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/789049685 shutdown_connections 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd19c03fdb0 0x7fd19c042270 unknown :-1 s=CLOSED pgs=4306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1bc13e1a0 0x7fd1bc135e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:49.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1bc0a4920 0x7fd1bc1ca9a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:49.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 --2- 172.21.15.98:0/789049685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1bc0a2530 0x7fd1bc0a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:49.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/789049685 >> 172.21.15.98:0/789049685 conn(0x7fd1bc0ab850 msgr2=0x7fd1bc0a90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:49.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/789049685 shutdown_connections 2024-06-11T19:24:49.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 -- 172.21.15.98:0/789049685 wait complete. 2024-06-11T19:24:49.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:49.939+0000 7fd1c0bfc640 1 librados: shutdown 2024-06-11T19:24:52.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:24:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.060+0000 7f42909fe640 10 monclient: get_monmap_and_config 2024-06-11T19:24:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.060+0000 7f42909fe640 10 monclient: build_initial_monmap 2024-06-11T19:24:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.060+0000 7f42909fe640 10 monclient: monmap: 2024-06-11T19:24:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:53.061231+0000 2024-06-11T19:24:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:53.061231+0000 2024-06-11T19:24:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:53.060 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 1 Processor -- start 2024-06-11T19:24:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 1 -- start start 2024-06-11T19:24:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 10 monclient: init 2024-06-11T19:24:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 10 monclient: _add_conns ranks=[8,9,10,0,7,6,1,2,5,3,4] 2024-06-11T19:24:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 10 monclient(hunting): picked mon.noname-e con 0x7f428c1347f0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 10 monclient(hunting): picked mon.noname-a con 0x7f428c134bd0 addr v2:172.21.15.98:3300/0 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 10 monclient(hunting): picked mon.noname-f con 0x7f428c135010 addr v2:172.21.15.117:3300/0 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f428c0687f0 con 0x7f428c134bd0 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f428c09d410 con 0x7f428c135010 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f428c09d6f0 con 0x7f428c1347f0 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f42909fe640 10 monclient(hunting): authenticate will time out at 1702.868593s 2024-06-11T19:24:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f428a59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f4283fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f4283fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58172/0 (socket says 172.21.15.98:58172) 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f428a59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:37336/0 (socket says 172.21.15.98:37336) 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 learned_addr learned my addr 172.21.15.98:0/3001957311 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f4283fff640 10 monclient(hunting): get_auth_request con 0x7f428c134bd0 auth_method 0 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f4283fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f4283fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f4283fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f428a59c640 10 monclient(hunting): get_auth_request con 0x7f428c1347f0 auth_method 0 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f428a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f428a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f428a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 msgr2=0x7f428c0a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:24:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 msgr2=0x7f428c0a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 msgr2=0x7f428c09e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 msgr2=0x7f428c09e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.061+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.062+0000 7f428ad9d640 10 monclient(hunting): get_auth_request con 0x7f428c135010 auth_method 0 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.062+0000 7f428ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.062+0000 7f428ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.062+0000 7f428ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.062+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 msgr2=0x7f428c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.062+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 msgr2=0x7f428c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.062+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:53.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.062+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f428a59c640 10 monclient(hunting): get_auth_request con 0x7f428c1347f0 auth_method 0 2024-06-11T19:24:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f428a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f428a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f428a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f4283fff640 10 monclient(hunting): get_auth_request con 0x7f428c134bd0 auth_method 0 2024-06-11T19:24:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f4283fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f4283fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f4283fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 msgr2=0x7f428c09e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:24:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 msgr2=0x7f428c09e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 msgr2=0x7f428c0a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:24:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 msgr2=0x7f428c0a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:24:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.262+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:24:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.263+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.263+0000 7f428ad9d640 10 monclient(hunting): get_auth_request con 0x7f428c135010 auth_method 0 2024-06-11T19:24:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.263+0000 7f428ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.263+0000 7f428ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.263+0000 7f428ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.263+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 msgr2=0x7f428c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:24:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.263+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 msgr2=0x7f428c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.263+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.263+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:24:53.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:53.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:53.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f428a59c640 10 monclient(hunting): get_auth_request con 0x7f428c1347f0 auth_method 0 2024-06-11T19:24:53.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f428a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:53.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f428a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:53.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f428a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:53.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f4283fff640 10 monclient(hunting): get_auth_request con 0x7f428c134bd0 auth_method 0 2024-06-11T19:24:53.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f4283fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:53.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f4283fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:53.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f4283fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:53.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 msgr2=0x7f428c09e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:24:53.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 msgr2=0x7f428c09e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:53.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:53.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 msgr2=0x7f428c0a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:24:53.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 msgr2=0x7f428c0a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:53.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:53.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:24:53.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.663+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:24:53.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.664+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:53.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.664+0000 7f428ad9d640 10 monclient(hunting): get_auth_request con 0x7f428c135010 auth_method 0 2024-06-11T19:24:53.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.664+0000 7f428ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:53.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.664+0000 7f428ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:53.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.664+0000 7f428ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:53.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.664+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 msgr2=0x7f428c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:24:53.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.664+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 msgr2=0x7f428c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:53.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.664+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:53.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:53.664+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:24:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f4283fff640 10 monclient(hunting): get_auth_request con 0x7f428c134bd0 auth_method 0 2024-06-11T19:24:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f4283fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f4283fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f4283fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f428a59c640 10 monclient(hunting): get_auth_request con 0x7f428c1347f0 auth_method 0 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f428a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f428a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f428a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 msgr2=0x7f428c0a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 msgr2=0x7f428c0a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 msgr2=0x7f428c09e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 msgr2=0x7f428c09e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.464+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:24:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.465+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.465+0000 7f428ad9d640 10 monclient(hunting): get_auth_request con 0x7f428c135010 auth_method 0 2024-06-11T19:24:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.465+0000 7f428ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.465+0000 7f428ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.465+0000 7f428ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.465+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 msgr2=0x7f428c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:24:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.465+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 msgr2=0x7f428c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.465+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:54.465+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:24:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 10 monclient(hunting): tick 2024-06-11T19:24:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 10 monclient(hunting): _check_auth_tickets 2024-06-11T19:24:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 monclient(hunting): continuing hunt 2024-06-11T19:24:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 10 monclient(hunting): _reopen_session rank -1 2024-06-11T19:24:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 msgr2=0x7f428c09e6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c1347f0 0x7f428c09e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 msgr2=0x7f428c0a3bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.117:3300/0 conn(0x7f428c135010 0x7f428c0a3bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 -- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 msgr2=0x7f428c0a3690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 --2- 172.21.15.98:0/3001957311 >> v2:172.21.15.98:3300/0 conn(0x7f428c134bd0 0x7f428c0a3690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:56.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 10 monclient: _add_conns ranks=[2,7,4,1,3,5,6,0] 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 10 monclient(hunting): picked mon.noname-b con 0x7f42780037c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f42780061a0 0x7f4278008610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 10 monclient(hunting): picked mon.noname-i con 0x7f42780061a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4278008d00 0x7f427800b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 10 monclient(hunting): picked mon.noname-c con 0x7f4278008d00 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 -- 172.21.15.98:0/3001957311 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f428c09d410 con 0x7f42780037c0 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 -- 172.21.15.98:0/3001957311 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f428c09d6f0 con 0x7f4278008d00 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 1 -- 172.21.15.98:0/3001957311 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f427800b9e0 con 0x7f42780061a0 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f4288d99640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:56.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4278008d00 msgr2=0x7f427800b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:24:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.061+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4278008d00 0x7f427800b190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.062+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.062+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f42780061a0 msgr2=0x7f4278008610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:24:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.062+0000 7f428a59c640 10 monclient(hunting): get_auth_request con 0x7f42780037c0 auth_method 0 2024-06-11T19:24:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.062+0000 7f428a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.062+0000 7f428a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.062+0000 7f428a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.062+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f42780061a0 0x7f4278008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.062+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 msgr2=0x7f4278005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:24:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.062+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 msgr2=0x7f4278005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.062+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:56.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.062+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:56.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.262+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4278008d00 msgr2=0x7f427800b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:24:56.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.262+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4278008d00 0x7f427800b190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:24:56.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.263+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f42780061a0 msgr2=0x7f4278008610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:24:56.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.263+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f42780061a0 0x7f4278008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:24:56.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.263+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.263+0000 7f428a59c640 10 monclient(hunting): get_auth_request con 0x7f42780037c0 auth_method 0 2024-06-11T19:24:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.263+0000 7f428a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.263+0000 7f428a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.263+0000 7f428a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.263+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 msgr2=0x7f4278005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:24:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.263+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 msgr2=0x7f4278005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.263+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:56.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.263+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:24:56.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.663+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4278008d00 msgr2=0x7f427800b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:24:56.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.663+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4278008d00 0x7f427800b190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:24:56.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.664+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f42780061a0 msgr2=0x7f4278008610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:24:56.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.664+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f42780061a0 0x7f4278008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:24:56.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.664+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:56.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.664+0000 7f428a59c640 10 monclient(hunting): get_auth_request con 0x7f42780037c0 auth_method 0 2024-06-11T19:24:56.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.664+0000 7f428a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:56.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.664+0000 7f428a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:56.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.664+0000 7f428a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:56.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.664+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 msgr2=0x7f4278005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:24:56.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.664+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 msgr2=0x7f4278005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:24:56.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.664+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:24:56.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:56.664+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:24:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.464+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4278008d00 msgr2=0x7f427800b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:24:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.464+0000 7f428ad9d640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4278008d00 0x7f427800b190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:24:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f4283fff640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f42780061a0 msgr2=0x7f4278008610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:24:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f4283fff640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f42780061a0 0x7f4278008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:24:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 10 monclient(hunting): get_auth_request con 0x7f42780037c0 auth_method 0 2024-06-11T19:24:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 10 monclient(hunting): handle_auth_done global_id 85259 payload 274 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f42780061a0 msgr2=0x7f4278008610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f42780061a0 0x7f4278008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4278008d00 msgr2=0x7f427800b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4278008d00 0x7f427800b190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 10 monclient: _finish_hunting 0 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 1 monclient: found mon.noname-b 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428ad9d640 1 -- 172.21.15.98:0/3001957311 reap_dead start 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:24:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.465+0000 7f428a59c640 1 -- 172.21.15.98:0/3001957311 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428c09d9d0 con 0x7f42780037c0 2024-06-11T19:24:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.466+0000 7f428a59c640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f427c017a50 tx=0x7f427c02d5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f11aa0e85223d74 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.466+0000 7f4289d9b640 1 -- 172.21.15.98:0/3001957311 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f427c0044a0 con 0x7f42780037c0 2024-06-11T19:24:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.466+0000 7f4289d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.466+0000 7f4289d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.466+0000 7f4289d9b640 10 monclient: dump: 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f4289d9b640 10 monclient: _finish_auth 0 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f4289d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f4289d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 5 monclient: authenticate success, global_id 85259 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f4289d9b640 1 -- 172.21.15.98:0/3001957311 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f427c003610 con 0x7f42780037c0 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f4289d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 4 set_mon_vals no callback set 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f4289d9b640 1 -- 172.21.15.98:0/3001957311 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f427c003c30 con 0x7f42780037c0 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f4289d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f4289d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f4289d9b640 10 monclient: dump: 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 10 monclient: shutdown 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 1 -- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 msgr2=0x7f4278005bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f427c017a50 tx=0x7f427c02d5b0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 1 -- 172.21.15.98:0/3001957311 shutdown_connections 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 1 --2- 172.21.15.98:0/3001957311 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f42780037c0 0x7f4278005bf0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 1 -- 172.21.15.98:0/3001957311 >> 172.21.15.98:0/3001957311 conn(0x7f428c0a4780 msgr2=0x7f428c0a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 1 -- 172.21.15.98:0/3001957311 shutdown_connections 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.467+0000 7f42909fe640 1 -- 172.21.15.98:0/3001957311 wait complete. 2024-06-11T19:24:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 10 monclient: build_initial_monmap 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 10 monclient: monmap: 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 1 librados: starting msgr at 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 1 librados: starting objecter 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 1 Processor -- start 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 1 -- start start 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 1 librados: setting wanted keys 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 1 librados: calling monclient init 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 10 monclient: init 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 10 monclient: _add_conns ranks=[1,5,8,7,2,6,0,3,4] 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42780037c0 0x7f428c133260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 10 monclient(hunting): picked mon.noname-f con 0x7f42780037c0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f428c1337a0 0x7f428c11ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 10 monclient(hunting): picked mon.noname-h con 0x7f428c1337a0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c11f210 0x7f428c11f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 10 monclient(hunting): picked mon.noname-e con 0x7f428c11f210 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.469+0000 7f42909fe640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f428c068c40 con 0x7f42780037c0 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f42909fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f42909fe640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f428c068ac0 con 0x7f428c1337a0 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f42909fe640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f42909fe640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f428c09e6b0 con 0x7f428c11f210 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f42909fe640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f42909fe640 10 monclient(hunting): authenticate will time out at 1707.277024s 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c11f210 0x7f428c11f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4283fff640 1 -- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f428c1337a0 msgr2=0x7f428c11ecd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c11f210 0x7f428c11f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:37354/0 (socket says 172.21.15.98:37354) 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4283fff640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f428c1337a0 0x7f428c11ecd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 1 -- 172.21.15.98:0/1417894624 learned_addr learned my addr 172.21.15.98:0/1417894624 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 10 monclient(hunting): get_auth_request con 0x7f428c11f210 auth_method 0 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428a59c640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42780037c0 0x7f428c133260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428a59c640 10 monclient(hunting): get_auth_request con 0x7f42780037c0 auth_method 0 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 10 monclient(hunting): handle_auth_done global_id 124289 payload 824 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 1 -- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f428c1337a0 msgr2=0x7f428c11ecd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f428c1337a0 0x7f428c11ecd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 1 -- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42780037c0 msgr2=0x7f428c133260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42780037c0 0x7f428c133260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 1 monclient: found mon.noname-e 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 1 -- 172.21.15.98:0/1417894624 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428c1c0d00 con 0x7f428c11f210 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428ad9d640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c11f210 0x7f428c11f650 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4284000f80 tx=0x7f428400ad60 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 10 client.?.objecter ms_handle_connect 0x7f428c11f210 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 1 -- 172.21.15.98:0/1417894624 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f428400ae90 con 0x7f428c11f210 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 10 monclient: dump: 2024-06-11T19:24:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:57.483 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 10 monclient: _finish_auth 0 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 1 -- 172.21.15.98:0/1417894624 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f428401e070 con 0x7f428c11f210 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f42909fe640 5 monclient: authenticate success, global_id 124289 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 1 -- 172.21.15.98:0/1417894624 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4284012e70 con 0x7f428c11f210 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f4282ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.471+0000 7f4282ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.471+0000 7f4282ffd640 10 monclient: dump: 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.471+0000 7f42909fe640 10 monclient: _renew_subs 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.470+0000 7f428959a640 4 set_mon_vals no callback set 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.471+0000 7f42909fe640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.471+0000 7f42909fe640 1 -- 172.21.15.98:0/1417894624 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428c122c40 con 0x7f428c11f210 2024-06-11T19:24:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.471+0000 7f42909fe640 10 client.124289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.471+0000 7f42909fe640 10 monclient: _renew_subs 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.471+0000 7f42909fe640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.471+0000 7f42909fe640 1 -- 172.21.15.98:0/1417894624 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f428c0a39b0 con 0x7f428c11f210 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.471+0000 7f42909fe640 1 librados: init done 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f4282ffd640 1 -- 172.21.15.98:0/1417894624 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f4284012a00 con 0x7f428c11f210 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f42909fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f42909fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f42909fe640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f42909fe640 1 -- 172.21.15.98:0/1417894624 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f428c075170 con 0x7f428c11f210 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f4282ffd640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f42780413f0 0x7f42780438b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f4282ffd640 1 -- 172.21.15.98:0/1417894624 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(627..627 src has 1..627) ==== 6823+0+0 (secure 0 0 0) 0x7f42840527a0 con 0x7f428c11f210 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f4282ffd640 10 client.124289.objecter ms_dispatch 0x7f428c000b90 osd_map(627..627 src has 1..627) 2024-06-11T19:24:57.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f4282ffd640 3 client.124289.objecter handle_osd_map got epochs [627,627] > 0 2024-06-11T19:24:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f4282ffd640 3 client.124289.objecter handle_osd_map decoding full epoch 627 2024-06-11T19:24:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f4282ffd640 20 client.124289.objecter dump_active .. 0 homeless 2024-06-11T19:24:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f428a59c640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f42780413f0 0x7f42780438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.472+0000 7f428a59c640 10 monclient: get_auth_request con 0x7f42780413f0 auth_method 0 2024-06-11T19:24:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.473+0000 7f4282ffd640 10 client.124289.objecter ms_handle_connect 0x7f42780413f0 2024-06-11T19:24:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.473+0000 7f428a59c640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f42780413f0 0x7f42780438b0 secure :-1 s=READY pgs=4307 cs=0 l=1 rev1=1 crypto rx=0x7f427c036000 tx=0x7f427c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.477+0000 7f4282ffd640 1 -- 172.21.15.98:0/1417894624 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f428c1c0d00 con 0x7f428c11f210 2024-06-11T19:24:57.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.477+0000 7f4282ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:57.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.477+0000 7f4282ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.736+0000 7f42909fe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:24:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.736+0000 7f42909fe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.736+0000 7f42909fe640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.736+0000 7f42909fe640 1 -- 172.21.15.98:0/1417894624 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f428c09e9b0 con 0x7f428c11f210 2024-06-11T19:24:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.736+0000 7f4282ffd640 1 -- 172.21.15.98:0/1417894624 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3174 (secure 0 0 0) 0x7f4284021d40 con 0x7f428c11f210 2024-06-11T19:24:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.736+0000 7f4282ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:24:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.736+0000 7f4282ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:57.736 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":86,"quorum":[0,1,2,3,6,8],"quorum_names":["a","f","b","g","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:24:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 10 librados: watch_flush enter 2024-06-11T19:24:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 10 librados: watch_flush exit 2024-06-11T19:24:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 20 client.124289.objecter shutdown clearing up homeless session... 2024-06-11T19:24:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 10 client.124289.objecter successfully canceled tick 2024-06-11T19:24:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 1 -- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f42780413f0 msgr2=0x7f42780438b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f42780413f0 0x7f42780438b0 secure :-1 s=READY pgs=4307 cs=0 l=1 rev1=1 crypto rx=0x7f427c036000 tx=0x7f427c030000 comp rx=0 tx=0).stop 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 10 monclient: shutdown 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 1 -- 172.21.15.98:0/1417894624 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c11f210 msgr2=0x7f428c11f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c11f210 0x7f428c11f650 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4284000f80 tx=0x7f428400ad60 comp rx=0 tx=0).stop 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 1 -- 172.21.15.98:0/1417894624 shutdown_connections 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f42780413f0 0x7f42780438b0 unknown :-1 s=CLOSED pgs=4307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f428c11f210 0x7f428c11f650 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f428c1337a0 0x7f428c11ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 1 --2- 172.21.15.98:0/1417894624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42780037c0 0x7f428c133260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 1 -- 172.21.15.98:0/1417894624 >> 172.21.15.98:0/1417894624 conn(0x7f428c0a4780 msgr2=0x7f428c1c0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.738+0000 7f42909fe640 1 -- 172.21.15.98:0/1417894624 shutdown_connections 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.739+0000 7f42909fe640 1 -- 172.21.15.98:0/1417894624 wait complete. 2024-06-11T19:24:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.739+0000 7f42909fe640 1 librados: shutdown 2024-06-11T19:24:57.749 INFO:tasks.mon_thrash.ceph_manager:quorum is size 6 2024-06-11T19:24:57.749 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon a 2024-06-11T19:24:57.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:24:57.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.849+0000 7fdfe8c99640 10 monclient: get_monmap_and_config 2024-06-11T19:24:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.849+0000 7fdfe8c99640 10 monclient: build_initial_monmap 2024-06-11T19:24:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.849+0000 7fdfe8c99640 10 monclient: monmap: 2024-06-11T19:24:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:57.850246+0000 2024-06-11T19:24:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:57.850246+0000 2024-06-11T19:24:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:57.853 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.849+0000 7fdfe8c99640 1 Processor -- start 2024-06-11T19:24:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.849+0000 7fdfe8c99640 1 -- start start 2024-06-11T19:24:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.849+0000 7fdfe8c99640 10 monclient: init 2024-06-11T19:24:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 10 monclient: _add_conns ranks=[7,4,10,8,6,1,3,0,2,5,9] 2024-06-11T19:24:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdfe40a4920 0x7fdfe40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 10 monclient(hunting): picked mon.noname-i con 0x7fdfe40a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdfe40a5240 0x7fdfe40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 10 monclient(hunting): picked mon.noname-c con 0x7fdfe40a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdfe40a2530 0x7fdfe40a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 10 monclient(hunting): picked mon.noname-f con 0x7fdfe40a2530 addr v2:172.21.15.117:3300/0 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fdfe40a7140 con 0x7fdfe40a2530 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fdfe40a2b50 con 0x7fdfe40a5240 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fdfe40a2e30 con 0x7fdfe40a4920 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe8c99640 10 monclient(hunting): authenticate will time out at 1707.657506s 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe1d9b640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdfe40a5240 msgr2=0x7fdfe40a59a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:24:57.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe1d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdfe40a5240 0x7fdfe40a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe259c640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdfe40a4920 msgr2=0x7fdfe40a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe259c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdfe40a4920 0x7fdfe40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe2d9d640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdfe40a2530 0x7fdfe40a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe2d9d640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdfe40a2530 0x7fdfe40a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:44604/0 (socket says 172.21.15.98:44604) 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe2d9d640 1 -- 172.21.15.98:0/3367127590 learned_addr learned my addr 172.21.15.98:0/3367127590 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe2d9d640 10 monclient(hunting): get_auth_request con 0x7fdfe40a2530 auth_method 0 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe2d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.850+0000 7fdfe2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:57.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.851+0000 7fdfe2d9d640 10 monclient(hunting): handle_auth_done global_id 124237 payload 274 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.851+0000 7fdfe2d9d640 1 -- 172.21.15.98:0/3367127590 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdfe40a4920 msgr2=0x7fdfe40a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.851+0000 7fdfe2d9d640 1 --2- 172.21.15.98:0/3367127590 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdfe40a4920 0x7fdfe40a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.851+0000 7fdfe2d9d640 1 -- 172.21.15.98:0/3367127590 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdfe40a5240 msgr2=0x7fdfe40a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.851+0000 7fdfe2d9d640 1 --2- 172.21.15.98:0/3367127590 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdfe40a5240 0x7fdfe40a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.851+0000 7fdfe2d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.851+0000 7fdfe2d9d640 1 monclient: found mon.noname-f 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.851+0000 7fdfe2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.851+0000 7fdfe2d9d640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.851+0000 7fdfe2d9d640 1 -- 172.21.15.98:0/3367127590 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe40a3110 con 0x7fdfe40a2530 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.851+0000 7fdfe2d9d640 1 --2- 172.21.15.98:0/3367127590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe40a2530 0x7fdfe40a5ee0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009e50 tx=0x7fdfdc02bde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2374b17ef68ff37 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 1 -- 172.21.15.98:0/3367127590 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdfdc02cbf0 con 0x7fdfe40a2530 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:57.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 10 monclient: dump: 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 10 monclient: _finish_auth 0 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 10 monclient: _check_auth_tickets 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 1 -- 172.21.15.98:0/3367127590 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfdc02cdb0 con 0x7fdfe40a2530 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 5 monclient: authenticate success, global_id 124237 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 1 -- 172.21.15.98:0/3367127590 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdfdc034860 con 0x7fdfe40a2530 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 4 set_mon_vals no callback set 2024-06-11T19:24:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe159a640 10 monclient: dump: 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:57.871 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:57.872 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 10 monclient: shutdown 2024-06-11T19:24:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 1 -- 172.21.15.98:0/3367127590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe40a2530 msgr2=0x7fdfe40a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 1 --2- 172.21.15.98:0/3367127590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe40a2530 0x7fdfe40a5ee0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009e50 tx=0x7fdfdc02bde0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 1 -- 172.21.15.98:0/3367127590 shutdown_connections 2024-06-11T19:24:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 1 --2- 172.21.15.98:0/3367127590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe40a2530 0x7fdfe40a5ee0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 1 --2- 172.21.15.98:0/3367127590 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdfe40a5240 0x7fdfe40a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 1 --2- 172.21.15.98:0/3367127590 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fdfe40a4920 0x7fdfe40a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 1 -- 172.21.15.98:0/3367127590 >> 172.21.15.98:0/3367127590 conn(0x7fdfe40ab850 msgr2=0x7fdfe40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 1 -- 172.21.15.98:0/3367127590 shutdown_connections 2024-06-11T19:24:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.852+0000 7fdfe8c99640 1 -- 172.21.15.98:0/3367127590 wait complete. 2024-06-11T19:24:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.853+0000 7fdfe8c99640 10 monclient: build_initial_monmap 2024-06-11T19:24:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.853+0000 7fdfe8c99640 10 monclient: monmap: 2024-06-11T19:24:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.853+0000 7fdfe8c99640 1 librados: starting msgr at 2024-06-11T19:24:57.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.853+0000 7fdfe8c99640 1 librados: starting objecter 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.853+0000 7fdfe8c99640 1 Processor -- start 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.853+0000 7fdfe8c99640 1 -- start start 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.853+0000 7fdfe8c99640 1 librados: setting wanted keys 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.853+0000 7fdfe8c99640 1 librados: calling monclient init 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.853+0000 7fdfe8c99640 10 monclient: init 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 10 monclient: _add_conns ranks=[8,6,1,2,7,5,3,4,0] 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdfe40a2530 0x7fdfe41cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 10 monclient(hunting): picked mon.noname-e con 0x7fdfe40a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdfe40a4920 0x7fdfe41cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 10 monclient(hunting): picked mon.noname-d con 0x7fdfe40a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe40a5240 0x7fdfe41ce400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 10 monclient(hunting): picked mon.noname-f con 0x7fdfe40a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe40a6da0 con 0x7fdfe40a5240 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fdfe4138e50 con 0x7fdfe40a4920 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fdfe40a6420 con 0x7fdfe40a2530 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe8c99640 10 monclient(hunting): authenticate will time out at 1707.661315s 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdfe40a2530 0x7fdfe41cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdfe40a2530 0x7fdfe41cd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:37368/0 (socket says 172.21.15.98:37368) 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 1 -- 172.21.15.98:0/2104756912 learned_addr learned my addr 172.21.15.98:0/2104756912 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 10 monclient(hunting): get_auth_request con 0x7fdfe40a2530 auth_method 0 2024-06-11T19:24:57.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe1d9b640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdfe40a4920 0x7fdfe41cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe1d9b640 10 monclient(hunting): get_auth_request con 0x7fdfe40a4920 auth_method 0 2024-06-11T19:24:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe1d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe2d9d640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe40a5240 0x7fdfe41ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:57.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe2d9d640 10 monclient(hunting): get_auth_request con 0x7fdfe40a5240 auth_method 0 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe2d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 10 monclient(hunting): handle_auth_done global_id 124298 payload 824 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 1 -- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdfe40a4920 msgr2=0x7fdfe41cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdfe40a4920 0x7fdfe41cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 1 -- 172.21.15.98:0/2104756912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe40a5240 msgr2=0x7fdfe41ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe40a5240 0x7fdfe41ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 10 monclient: _finish_hunting 0 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 1 monclient: found mon.noname-e 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 1 -- 172.21.15.98:0/2104756912 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe40a8fc0 con 0x7fdfe40a2530 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfe259c640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdfe40a2530 0x7fdfe41cd980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400b550 tx=0x7fdfd400ba10 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfd2ffd640 10 client.?.objecter ms_handle_connect 0x7fdfe40a2530 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.854+0000 7fdfd2ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdfd4013020 con 0x7fdfe40a2530 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 10 monclient: dump: 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:57.879 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:57.880 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 10 monclient: _finish_auth 0 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfd4003bc0 con 0x7fdfe40a2530 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfe8c99640 5 monclient: authenticate success, global_id 124298 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdfd4003d70 con 0x7fdfe40a2530 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfe0d99640 4 set_mon_vals no callback set 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfd2ffd640 10 monclient: dump: 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.881 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfe8c99640 10 monclient: _renew_subs 2024-06-11T19:24:57.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfe8c99640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfe8c99640 1 -- 172.21.15.98:0/2104756912 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfe4124a70 con 0x7fdfe40a2530 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfe8c99640 10 client.124298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfe8c99640 10 monclient: _renew_subs 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfe8c99640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfe8c99640 1 -- 172.21.15.98:0/2104756912 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfe4126530 con 0x7fdfe40a2530 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfe8c99640 1 librados: init done 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.855+0000 7fdfe8c99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe8c99640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe41271f0 0x7fdfe4123120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe8c99640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe8c99640 1 -- 172.21.15.98:0/2104756912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe4050830 con 0x7fdfe41271f0 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fdfd4016020 con 0x7fdfe40a2530 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe1d9b640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe41271f0 0x7fdfe4123120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe1d9b640 10 monclient: get_auth_request con 0x7fdfe41271f0 auth_method 0 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:57.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe1d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdfc403fde0 0x7fdfc40422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe1d9b640 10 monclient(hunting): handle_auth_done global_id 66231 payload 824 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(627..627 src has 1..627) ==== 6823+0+0 (secure 0 0 0) 0x7fdfd40518c0 con 0x7fdfe40a2530 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 10 client.124298.objecter ms_dispatch 0x7fdfe4000b90 osd_map(627..627 src has 1..627) 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 3 client.124298.objecter handle_osd_map got epochs [627,627] > 0 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 3 client.124298.objecter handle_osd_map decoding full epoch 627 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe1d9b640 1 -- 172.21.15.98:0/2104756912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdfe4126a50 con 0x7fdfe41271f0 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 20 client.124298.objecter dump_active .. 0 homeless 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe1d9b640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe41271f0 0x7fdfe4123120 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40a5620 tx=0x7fdfd8009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 10 client.124298.objecter ms_handle_connect 0x7fdfe41271f0 2024-06-11T19:24:57.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 10 client.124298.objecter resend_mon_ops 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe2d9d640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdfc403fde0 0x7fdfc40422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe2d9d640 10 monclient: get_auth_request con 0x7fdfc403fde0 auth_method 0 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdfd8016070 con 0x7fdfe41271f0 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 10 monclient: dump: 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:57.885 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdfd8012600 con 0x7fdfe41271f0 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe41271f0 msgr2=0x7fdfe4123120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:57.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe41271f0 0x7fdfe4123120 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40a5620 tx=0x7fdfd8009ac0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe41271f0 msgr2=0x7fdfe4123120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe41271f0 0x7fdfe4123120 secure :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40a5620 tx=0x7fdfd8009ac0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfd2ffd640 10 client.124298.objecter ms_handle_connect 0x7fdfc403fde0 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.856+0000 7fdfe2d9d640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdfc403fde0 0x7fdfc40422a0 secure :-1 s=READY pgs=4308 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc035000 tx=0x7fdfdc030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe8c99640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe8c99640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4123660 0x7fdfe4123a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe8c99640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe8c99640 1 -- 172.21.15.98:0/2104756912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe4123fb0 con 0x7fdfe4123660 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe1d9b640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4123660 0x7fdfe4123a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe1d9b640 10 monclient: get_auth_request con 0x7fdfe4123660 auth_method 0 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe1d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe1d9b640 10 monclient(hunting): handle_auth_done global_id 66240 payload 824 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe1d9b640 1 -- 172.21.15.98:0/2104756912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdfe40a7360 con 0x7fdfe4123660 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfd2ffd640 10 client.124298.objecter ms_handle_connect 0x7fdfe4123660 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfd2ffd640 10 client.124298.objecter resend_mon_ops 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfe1d9b640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4123660 0x7fdfe4123a70 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8018000 tx=0x7fdfd80059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdfd8016040 con 0x7fdfe4123660 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.863+0000 7fdfd2ffd640 10 monclient: dump: 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:57.888 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9410 (secure 0 0 0) 0x7fdfd801dac0 con 0x7fdfe4123660 2024-06-11T19:24:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfd2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:57.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfd2ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:57.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4123660 msgr2=0x7fdfe4123a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:57.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfd2ffd640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4123660 0x7fdfe4123a70 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8018000 tx=0x7fdfd80059d0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfd2ffd640 1 -- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4123660 msgr2=0x7fdfe4123a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:57.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfd2ffd640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdfe4123660 0x7fdfe4123a70 secure :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8018000 tx=0x7fdfd80059d0 comp rx=0 tx=0).stop 2024-06-11T19:24:57.890 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:57.890 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 86, 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 689691, 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:57.891 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:57.892 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:57.893 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:57.894 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:57.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:57.897 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:57.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:57.899 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:57.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:57.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:57.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:57.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:57.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:57.907 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:57.909 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.910 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:24:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:57.913 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 10 librados: watch_flush enter 2024-06-11T19:24:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 10 librados: watch_flush exit 2024-06-11T19:24:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 20 client.124298.objecter shutdown clearing up homeless session... 2024-06-11T19:24:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 10 client.124298.objecter successfully canceled tick 2024-06-11T19:24:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 1 -- 172.21.15.98:0/2104756912 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdfc403fde0 msgr2=0x7fdfc40422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdfc403fde0 0x7fdfc40422a0 secure :-1 s=READY pgs=4308 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc035000 tx=0x7fdfdc030000 comp rx=0 tx=0).stop 2024-06-11T19:24:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 10 monclient: shutdown 2024-06-11T19:24:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 1 -- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdfe40a2530 msgr2=0x7fdfe41cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 1 --2- 172.21.15.98:0/2104756912 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdfe40a2530 0x7fdfe41cd980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400b550 tx=0x7fdfd400ba10 comp rx=0 tx=0).stop 2024-06-11T19:24:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe2d9d640 1 -- 172.21.15.98:0/2104756912 reap_dead start 2024-06-11T19:24:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe2d9d640 1 -- 172.21.15.98:0/2104756912 reap_dead start 2024-06-11T19:24:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 1 -- 172.21.15.98:0/2104756912 shutdown_connections 2024-06-11T19:24:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 1 -- 172.21.15.98:0/2104756912 >> 172.21.15.98:0/2104756912 conn(0x7fdfe40ab850 msgr2=0x7fdfe40a9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 1 -- 172.21.15.98:0/2104756912 shutdown_connections 2024-06-11T19:24:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 1 -- 172.21.15.98:0/2104756912 wait complete. 2024-06-11T19:24:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:57.864+0000 7fdfe8c99640 1 librados: shutdown 2024-06-11T19:24:57.916 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-06-11T19:24:57.916 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon b 2024-06-11T19:24:57.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:24:58.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.018+0000 7f84c797d640 10 monclient: get_monmap_and_config 2024-06-11T19:24:58.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.018+0000 7f84c797d640 10 monclient: build_initial_monmap 2024-06-11T19:24:58.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.018+0000 7f84c797d640 10 monclient: monmap: 2024-06-11T19:24:58.017 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:58.017 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.017 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:58.019089+0000 2024-06-11T19:24:58.018 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:58.019089+0000 2024-06-11T19:24:58.018 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:58.018 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.018 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:58.018 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:58.018 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:58.018 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:58.018 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:58.018 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:58.019 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:58.019 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:58.019 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:58.019 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:58.019 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:58.019 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.018+0000 7f84c797d640 1 Processor -- start 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.018+0000 7f84c797d640 1 -- start start 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.018+0000 7f84c797d640 10 monclient: init 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 10 monclient: _add_conns ranks=[2,7,8,9,10,4,0,1,3,5,6] 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c00a0fa0 0x7f84c00a1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 10 monclient(hunting): picked mon.noname-b con 0x7f84c00a0fa0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84c0138cf0 0x7f84c00a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 10 monclient(hunting): picked mon.noname-i con 0x7f84c0138cf0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84c01390f0 0x7f84c00a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 10 monclient(hunting): picked mon.noname-e con 0x7f84c01390f0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f84c009f380 con 0x7f84c00a0fa0 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f84c009ed30 con 0x7f84c0138cf0 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f84c009f010 con 0x7f84c01390f0 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c797d640 10 monclient(hunting): authenticate will time out at 1707.826429s 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5719640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c00a0fa0 0x7f84c00a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84c01390f0 0x7f84c00a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5719640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c00a0fa0 0x7f84c00a1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50680/0 (socket says 172.21.15.98:50680) 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5719640 1 -- 172.21.15.98:0/1954555305 learned_addr learned my addr 172.21.15.98:0/1954555305 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5719640 10 monclient(hunting): get_auth_request con 0x7f84c00a0fa0 auth_method 0 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5719640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5719640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5719640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 10 monclient(hunting): get_auth_request con 0x7f84c01390f0 auth_method 0 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c4f18640 1 -- 172.21.15.98:0/1954555305 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84c0138cf0 msgr2=0x7f84c00a22b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c4f18640 1 --2- 172.21.15.98:0/1954555305 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84c0138cf0 0x7f84c00a22b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5719640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5719640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5719640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 10 monclient(hunting): handle_auth_done global_id 124307 payload 274 2024-06-11T19:24:58.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 1 -- 172.21.15.98:0/1954555305 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84c0138cf0 msgr2=0x7f84c00a22b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 1 --2- 172.21.15.98:0/1954555305 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84c0138cf0 0x7f84c00a22b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 1 -- 172.21.15.98:0/1954555305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c00a0fa0 msgr2=0x7f84c00a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 1 --2- 172.21.15.98:0/1954555305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c00a0fa0 0x7f84c00a1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 10 monclient: _finish_hunting 0 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 1 monclient: found mon.noname-e 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5f1a640 1 -- 172.21.15.98:0/1954555305 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c00a3cd0 con 0x7f84c01390f0 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.019+0000 7f84c5719640 1 --2- 172.21.15.98:0/1954555305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c00a0fa0 0x7f84c00a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84c5f1a640 1 --2- 172.21.15.98:0/1954555305 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84c01390f0 0x7f84c00a27f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009dc0 tx=0x7f84b802bce0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=521aca88f164016b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 1 -- 172.21.15.98:0/1954555305 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84b8039070 con 0x7f84c01390f0 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 10 monclient: dump: 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.032 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 10 monclient: _finish_auth 0 2024-06-11T19:24:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 1 -- 172.21.15.98:0/1954555305 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84b802cd00 con 0x7f84c01390f0 2024-06-11T19:24:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84c797d640 5 monclient: authenticate success, global_id 124307 2024-06-11T19:24:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84c797d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84c797d640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 1 -- 172.21.15.98:0/1954555305 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84b80327b0 con 0x7f84c01390f0 2024-06-11T19:24:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84c797d640 4 set_mon_vals no callback set 2024-06-11T19:24:58.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.020+0000 7f84b7fff640 10 monclient: dump: 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.035 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 10 monclient: shutdown 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 1 -- 172.21.15.98:0/1954555305 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84c01390f0 msgr2=0x7f84c00a27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 1 --2- 172.21.15.98:0/1954555305 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84c01390f0 0x7f84c00a27f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009dc0 tx=0x7f84b802bce0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 1 -- 172.21.15.98:0/1954555305 shutdown_connections 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 1 --2- 172.21.15.98:0/1954555305 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84c01390f0 0x7f84c00a27f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 1 --2- 172.21.15.98:0/1954555305 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84c0138cf0 0x7f84c00a22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 1 --2- 172.21.15.98:0/1954555305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c00a0fa0 0x7f84c00a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 1 -- 172.21.15.98:0/1954555305 >> 172.21.15.98:0/1954555305 conn(0x7f84c00a7850 msgr2=0x7f84c00a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 1 -- 172.21.15.98:0/1954555305 shutdown_connections 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 1 -- 172.21.15.98:0/1954555305 wait complete. 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 10 monclient: build_initial_monmap 2024-06-11T19:24:58.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.021+0000 7f84c797d640 10 monclient: monmap: 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:58.037 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 1 librados: starting msgr at 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 1 librados: starting objecter 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 1 Processor -- start 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 1 -- start start 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 1 librados: setting wanted keys 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 1 librados: calling monclient init 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 10 monclient: init 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 10 monclient: _add_conns ranks=[0,7,2,6,4,3,8,1,5] 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84c00a0fa0 0x7f84c01cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 10 monclient(hunting): picked mon.noname-a con 0x7f84c00a0fa0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84c0138cf0 0x7f84c01358d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 10 monclient(hunting): picked mon.noname-i con 0x7f84c0138cf0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c01390f0 0x7f84c0135e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 10 monclient(hunting): picked mon.noname-b con 0x7f84c01390f0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84c00a2eb0 con 0x7f84c00a0fa0 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f84c00a2d30 con 0x7f84c01390f0 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f84c00a3030 con 0x7f84c0138cf0 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c797d640 10 monclient(hunting): authenticate will time out at 1707.829870s 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c5719640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84c00a0fa0 0x7f84c01cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c4f18640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84c0138cf0 msgr2=0x7f84c01358d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c5719640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84c00a0fa0 0x7f84c01cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58220/0 (socket says 172.21.15.98:58220) 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c5719640 1 -- 172.21.15.98:0/3111498744 learned_addr learned my addr 172.21.15.98:0/3111498744 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c5719640 10 monclient(hunting): get_auth_request con 0x7f84c00a0fa0 auth_method 0 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c5719640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c5719640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c5719640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c4f18640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84c0138cf0 0x7f84c01358d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c5f1a640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c01390f0 0x7f84c0135e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c5f1a640 10 monclient(hunting): get_auth_request con 0x7f84c01390f0 auth_method 0 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c5f1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c5f1a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.022+0000 7f84c5f1a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5f1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5f1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5f1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 10 monclient(hunting): handle_auth_done global_id 66249 payload 824 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 1 -- 172.21.15.98:0/3111498744 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84c0138cf0 msgr2=0x7f84c01358d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:24:58.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f84c0138cf0 0x7f84c01358d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 1 -- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c01390f0 msgr2=0x7f84c0135e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c01390f0 0x7f84c0135e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 10 monclient: _finish_hunting 0 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 1 monclient: found mon.noname-a 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 1 -- 172.21.15.98:0/3111498744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c012fda0 con 0x7f84c00a0fa0 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5f1a640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c01390f0 0x7f84c0135e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c5719640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84c00a0fa0 0x7f84c01cd9b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00e9d0 tx=0x7f84bc00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 10 client.?.objecter ms_handle_connect 0x7f84c00a0fa0 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84bc017070 con 0x7f84c00a0fa0 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:58.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 10 monclient: dump: 2024-06-11T19:24:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.042 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.043 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 10 monclient: _finish_auth 0 2024-06-11T19:24:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:24:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84bc013470 con 0x7f84c00a0fa0 2024-06-11T19:24:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c797d640 5 monclient: authenticate success, global_id 66249 2024-06-11T19:24:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c797d640 10 monclient: _renew_subs 2024-06-11T19:24:58.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84bc013a90 con 0x7f84c00a0fa0 2024-06-11T19:24:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c797d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c797d640 1 -- 172.21.15.98:0/3111498744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c01cdef0 con 0x7f84c00a0fa0 2024-06-11T19:24:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b77fe640 4 set_mon_vals no callback set 2024-06-11T19:24:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84b5ffb640 10 monclient: dump: 2024-06-11T19:24:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.044 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c797d640 10 client.66249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c797d640 10 monclient: _renew_subs 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c797d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:58.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c797d640 1 -- 172.21.15.98:0/3111498744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c0130080 con 0x7f84c00a0fa0 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.023+0000 7f84c797d640 1 librados: init done 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84c797d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84c797d640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012c4f0 0x7f84c012c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f84bc01a070 con 0x7f84c00a0fa0 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84c797d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84c797d640 1 -- 172.21.15.98:0/3111498744 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f84c0050830 con 0x7f84c012c4f0 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84c4f18640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012c4f0 0x7f84c012c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84c4f18640 10 monclient: get_auth_request con 0x7f84c012c4f0 auth_method 0 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84c4f18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84c4f18640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84c4f18640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84c4f18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84c4f18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.024+0000 7f84c4f18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f84a403fde0 0x7f84a40422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(627..627 src has 1..627) ==== 6823+0+0 (secure 0 0 0) 0x7f84bc00f040 con 0x7f84c00a0fa0 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 10 client.66249.objecter ms_dispatch 0x7f84c0000b90 osd_map(627..627 src has 1..627) 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 3 client.66249.objecter handle_osd_map got epochs [627,627] > 0 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 3 client.66249.objecter handle_osd_map decoding full epoch 627 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84c4f18640 10 monclient(hunting): handle_auth_done global_id 85286 payload 824 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84c4f18640 1 -- 172.21.15.98:0/3111498744 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84c00a64d0 con 0x7f84c012c4f0 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 20 client.66249.objecter dump_active .. 0 homeless 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 10 client.66249.objecter ms_handle_connect 0x7f84c012c4f0 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 10 client.66249.objecter resend_mon_ops 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84c4f18640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012c4f0 0x7f84c012c8d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f84c012db50 tx=0x7f84b000b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84b0005070 con 0x7f84c012c4f0 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84c5f1a640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f84a403fde0 0x7f84a40422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 10 monclient: dump: 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.049 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84c5f1a640 10 monclient: get_auth_request con 0x7f84a403fde0 auth_method 0 2024-06-11T19:24:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f84b0003b70 con 0x7f84c012c4f0 2024-06-11T19:24:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012c4f0 msgr2=0x7f84c012c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012c4f0 0x7f84c012c8d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f84c012db50 tx=0x7f84b000b810 comp rx=0 tx=0).stop 2024-06-11T19:24:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012c4f0 msgr2=0x7f84c012c8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012c4f0 0x7f84c012c8d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84b5ffb640 10 client.66249.objecter ms_handle_connect 0x7f84a403fde0 2024-06-11T19:24:58.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.025+0000 7f84c5f1a640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f84a403fde0 0x7f84a40422a0 secure :-1 s=READY pgs=4309 cs=0 l=1 rev1=1 crypto rx=0x7f84b8036000 tx=0x7f84b8030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c797d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c797d640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012ce10 0x7f84c01be430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c797d640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c797d640 1 -- 172.21.15.98:0/3111498744 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f84c00a1380 con 0x7f84c012ce10 2024-06-11T19:24:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c4f18640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012ce10 0x7f84c01be430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c4f18640 10 monclient: get_auth_request con 0x7f84c012ce10 auth_method 0 2024-06-11T19:24:58.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c4f18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c4f18640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c4f18640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c4f18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c4f18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c4f18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c4f18640 10 monclient(hunting): handle_auth_done global_id 85295 payload 824 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c4f18640 1 -- 172.21.15.98:0/3111498744 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f84c0073920 con 0x7f84c012ce10 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 10 client.66249.objecter ms_handle_connect 0x7f84c012ce10 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84c4f18640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012ce10 0x7f84c01be430 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f84b0000c00 tx=0x7f84b00059a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 10 client.66249.objecter resend_mon_ops 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84b0019be0 con 0x7f84c012ce10 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:58.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 10 monclient: dump: 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.053 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.054 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.054 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.054 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9412 (secure 0 0 0) 0x7f84b0011400 con 0x7f84c012ce10 2024-06-11T19:24:58.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:58.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:58.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012ce10 msgr2=0x7f84c01be430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012ce10 0x7f84c01be430 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f84b0000c00 tx=0x7f84b00059a0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 1 -- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012ce10 msgr2=0x7f84c01be430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:58.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.032+0000 7f84b5ffb640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84c012ce10 0x7f84c01be430 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.055 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:24:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 86, 2024-06-11T19:24:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 569806, 2024-06-11T19:24:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:58.055 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.056 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.057 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:58.058 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.059 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:58.060 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.061 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.064 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.067 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.070 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.074 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:58.075 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:24:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.076 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:58.076 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 10 librados: watch_flush enter 2024-06-11T19:24:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 10 librados: watch_flush exit 2024-06-11T19:24:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 20 client.66249.objecter shutdown clearing up homeless session... 2024-06-11T19:24:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 10 client.66249.objecter successfully canceled tick 2024-06-11T19:24:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 1 -- 172.21.15.98:0/3111498744 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f84a403fde0 msgr2=0x7f84a40422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f84a403fde0 0x7f84a40422a0 secure :-1 s=READY pgs=4309 cs=0 l=1 rev1=1 crypto rx=0x7f84b8036000 tx=0x7f84b8030000 comp rx=0 tx=0).stop 2024-06-11T19:24:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 10 monclient: shutdown 2024-06-11T19:24:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 1 -- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84c00a0fa0 msgr2=0x7f84c01cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 1 --2- 172.21.15.98:0/3111498744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84c00a0fa0 0x7f84c01cd9b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00e9d0 tx=0x7f84bc00ee90 comp rx=0 tx=0).stop 2024-06-11T19:24:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c5f1a640 1 -- 172.21.15.98:0/3111498744 reap_dead start 2024-06-11T19:24:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c5f1a640 1 -- 172.21.15.98:0/3111498744 reap_dead start 2024-06-11T19:24:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 1 -- 172.21.15.98:0/3111498744 shutdown_connections 2024-06-11T19:24:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 1 -- 172.21.15.98:0/3111498744 >> 172.21.15.98:0/3111498744 conn(0x7f84c00a7850 msgr2=0x7f84c01371c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 1 -- 172.21.15.98:0/3111498744 shutdown_connections 2024-06-11T19:24:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 1 -- 172.21.15.98:0/3111498744 wait complete. 2024-06-11T19:24:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.033+0000 7f84c797d640 1 librados: shutdown 2024-06-11T19:24:58.079 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-06-11T19:24:58.079 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon d 2024-06-11T19:24:58.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:24:58.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.179+0000 7fea2f874640 10 monclient: get_monmap_and_config 2024-06-11T19:24:58.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.179+0000 7fea2f874640 10 monclient: build_initial_monmap 2024-06-11T19:24:58.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.179+0000 7fea2f874640 10 monclient: monmap: 2024-06-11T19:24:58.178 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:58.178 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.178 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:58.180242+0000 2024-06-11T19:24:58.178 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:58.180242+0000 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:58.179 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.179+0000 7fea2f874640 1 Processor -- start 2024-06-11T19:24:58.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 1 -- start start 2024-06-11T19:24:58.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 10 monclient: init 2024-06-11T19:24:58.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:58.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 10 monclient: _add_conns ranks=[3,1,7,4,10,6,2,0,8,5,9] 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea280a52a0 0x7fea280a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 10 monclient(hunting): picked mon.noname-g con 0x7fea280a52a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fea2809d520 addr v1:172.21.15.117:6789/0 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fea28073830 0x7fea280a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 10 monclient(hunting): picked mon.noname-i con 0x7fea28073830 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea2809d2e0 con 0x7fea2809d520 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fea2809daa0 con 0x7fea280a52a0 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fea280a5680 con 0x7fea28073830 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2f874640 10 monclient(hunting): authenticate will time out at 1707.987487s 2024-06-11T19:24:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2de11640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fea28073830 msgr2=0x7fea280a3da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2de11640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fea28073830 0x7fea280a3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2d610640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea280a52a0 0x7fea280a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2ce0f640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fea2809d520 0x7fea280a3690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:37790/0 (socket says 172.21.15.98:37790) 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2ce0f640 1 -- 172.21.15.98:0/1976300430 learned_addr learned my addr 172.21.15.98:0/1976300430 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2d610640 10 monclient(hunting): get_auth_request con 0x7fea280a52a0 auth_method 0 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2d610640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2d610640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2d610640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2d610640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2d610640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.180+0000 7fea2d610640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.181+0000 7fea1bfff640 1 -- 172.21.15.98:0/1976300430 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1202335850 0 0) 0x7fea2809d2e0 con 0x7fea2809d520 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.181+0000 7fea2d610640 10 monclient(hunting): handle_auth_done global_id 114429 payload 274 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.181+0000 7fea2d610640 1 -- 172.21.15.98:0/1976300430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fea28073830 msgr2=0x7fea280a3da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.181+0000 7fea2d610640 1 --2- 172.21.15.98:0/1976300430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fea28073830 0x7fea280a3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.181+0000 7fea2d610640 1 -- 172.21.15.98:0/1976300430 >> v1:172.21.15.117:6789/0 conn(0x7fea2809d520 legacy=0x7fea280a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.181+0000 7fea2d610640 10 monclient: _finish_hunting 0 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.181+0000 7fea2d610640 1 monclient: found mon.noname-g 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.181+0000 7fea2d610640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.181+0000 7fea2d610640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.181+0000 7fea2d610640 1 -- 172.21.15.98:0/1976300430 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2809e9d0 con 0x7fea280a52a0 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.181+0000 7fea1bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.181+0000 7fea2d610640 1 --2- 172.21.15.98:0/1976300430 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea280a52a0 0x7fea280a1910 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0097d0 tx=0x7fea1c02b7a0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=dcc05c5fbee346ad server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 1 -- 172.21.15.98:0/1976300430 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea1c038070 con 0x7fea280a52a0 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 10 monclient: dump: 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.193 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 10 monclient: _finish_auth 0 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 1 -- 172.21.15.98:0/1976300430 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea1c031970 con 0x7fea280a52a0 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 5 monclient: authenticate success, global_id 114429 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 1 -- 172.21.15.98:0/1976300430 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea1c032390 con 0x7fea280a52a0 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 4 set_mon_vals no callback set 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea1bfff640 10 monclient: dump: 2024-06-11T19:24:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.196 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 10 monclient: shutdown 2024-06-11T19:24:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 1 -- 172.21.15.98:0/1976300430 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea280a52a0 msgr2=0x7fea280a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 1 --2- 172.21.15.98:0/1976300430 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea280a52a0 0x7fea280a1910 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0097d0 tx=0x7fea1c02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 1 -- 172.21.15.98:0/1976300430 shutdown_connections 2024-06-11T19:24:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 1 --2- 172.21.15.98:0/1976300430 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fea28073830 0x7fea280a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 1 --2- 172.21.15.98:0/1976300430 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea280a52a0 0x7fea280a1910 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 1 -- 172.21.15.98:0/1976300430 >> 172.21.15.98:0/1976300430 conn(0x7fea280a4780 msgr2=0x7fea280a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 1 -- 172.21.15.98:0/1976300430 shutdown_connections 2024-06-11T19:24:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.182+0000 7fea2f874640 1 -- 172.21.15.98:0/1976300430 wait complete. 2024-06-11T19:24:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 10 monclient: build_initial_monmap 2024-06-11T19:24:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 10 monclient: monmap: 2024-06-11T19:24:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 1 librados: starting msgr at 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 1 librados: starting objecter 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 1 Processor -- start 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 1 -- start start 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 1 librados: setting wanted keys 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 1 librados: calling monclient init 2024-06-11T19:24:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 10 monclient: init 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 10 monclient: _add_conns ranks=[2,4,8,7,1,6,5,0,3] 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea28073830 0x7fea281226e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 10 monclient(hunting): picked mon.noname-b con 0x7fea28073830 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fea2809d520 0x7fea28122c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 10 monclient(hunting): picked mon.noname-c con 0x7fea2809d520 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.183+0000 7fea2f874640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fea280a52a0 0x7fea28123160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 10 monclient(hunting): picked mon.noname-e con 0x7fea280a52a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fea280a4460 con 0x7fea28073830 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fea280a42e0 con 0x7fea2809d520 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fea280a1e50 con 0x7fea280a52a0 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2ce0f640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fea2809d520 msgr2=0x7fea28122c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 10 monclient(hunting): authenticate will time out at 1707.991037s 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2ce0f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fea2809d520 0x7fea28122c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea28073830 0x7fea281226e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2de11640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fea280a52a0 0x7fea28123160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea28073830 0x7fea281226e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50714/0 (socket says 172.21.15.98:50714) 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 1 -- 172.21.15.98:0/1782954493 learned_addr learned my addr 172.21.15.98:0/1782954493 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 10 monclient(hunting): get_auth_request con 0x7fea28073830 auth_method 0 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2de11640 10 monclient(hunting): get_auth_request con 0x7fea280a52a0 auth_method 0 2024-06-11T19:24:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2de11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2de11640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2de11640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2de11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2de11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2de11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 10 monclient(hunting): handle_auth_done global_id 85304 payload 824 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 1 -- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fea280a52a0 msgr2=0x7fea28123160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fea280a52a0 0x7fea28123160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 1 -- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fea2809d520 msgr2=0x7fea28122c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fea2809d520 0x7fea28122c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 10 monclient: _finish_hunting 0 2024-06-11T19:24:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 1 monclient: found mon.noname-b 2024-06-11T19:24:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:24:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 1 -- 172.21.15.98:0/1782954493 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea28072db0 con 0x7fea28073830 2024-06-11T19:24:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2de11640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fea280a52a0 0x7fea28123160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2d610640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea28073830 0x7fea281226e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fea1c000c00 tx=0x7fea1c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 10 client.?.objecter ms_handle_connect 0x7fea28073830 2024-06-11T19:24:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea1c041020 con 0x7fea28073830 2024-06-11T19:24:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:24:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 10 monclient: dump: 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.203 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 10 monclient: _finish_auth 0 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea1c004470 con 0x7fea28073830 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 5 monclient: authenticate success, global_id 85304 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea1c038040 con 0x7fea28073830 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 10 monclient: _renew_subs 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea2f874640 1 -- 172.21.15.98:0/1782954493 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea28124f80 con 0x7fea28073830 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea1b7fe640 4 set_mon_vals no callback set 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:24:58.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.184+0000 7fea19ffb640 10 monclient: dump: 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.205 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2f874640 10 client.85304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2f874640 10 monclient: _renew_subs 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2f874640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2f874640 1 -- 172.21.15.98:0/1782954493 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea28123810 con 0x7fea28073830 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2f874640 1 librados: init done 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2f874640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2f874640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281cd7e0 0x7fea281cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2f874640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2f874640 1 -- 172.21.15.98:0/1782954493 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fea280a4d10 con 0x7fea281cd7e0 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fea1c044020 con 0x7fea28073830 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2ce0f640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281cd7e0 0x7fea281cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2ce0f640 10 monclient: get_auth_request con 0x7fea281cd7e0 auth_method 0 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2ce0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2ce0f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2ce0f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2ce0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2ce0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.185+0000 7fea2ce0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea2ce0f640 10 monclient(hunting): handle_auth_done global_id 124314 payload 824 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fea0c03fdb0 0x7fea0c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea2ce0f640 1 -- 172.21.15.98:0/1782954493 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fea28069290 con 0x7fea281cd7e0 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(627..627 src has 1..627) ==== 6823+0+0 (secure 0 0 0) 0x7fea1c0731f0 con 0x7fea28073830 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 10 client.85304.objecter ms_dispatch 0x7fea28000b90 osd_map(627..627 src has 1..627) 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 3 client.85304.objecter handle_osd_map got epochs [627,627] > 0 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 3 client.85304.objecter handle_osd_map decoding full epoch 627 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea2ce0f640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281cd7e0 0x7fea281cdbc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fea28072c60 tx=0x7fea2400c810 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 20 client.85304.objecter dump_active .. 0 homeless 2024-06-11T19:24:58.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 10 client.85304.objecter ms_handle_connect 0x7fea281cd7e0 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 10 client.85304.objecter resend_mon_ops 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea2de11640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fea0c03fdb0 0x7fea0c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea2de11640 10 monclient: get_auth_request con 0x7fea0c03fdb0 auth_method 0 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea240047e0 con 0x7fea281cd7e0 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 10 monclient: dump: 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.208 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fea240072e0 con 0x7fea281cd7e0 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281cd7e0 msgr2=0x7fea281cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281cd7e0 0x7fea281cdbc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fea28072c60 tx=0x7fea2400c810 comp rx=0 tx=0).stop 2024-06-11T19:24:58.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281cd7e0 msgr2=0x7fea281cdbc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:58.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281cd7e0 0x7fea281cdbc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea19ffb640 10 client.85304.objecter ms_handle_connect 0x7fea0c03fdb0 2024-06-11T19:24:58.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.186+0000 7fea2de11640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fea0c03fdb0 0x7fea0c042270 secure :-1 s=READY pgs=4310 cs=0 l=1 rev1=1 crypto rx=0x7fea28072960 tx=0x7fea200095f0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.192+0000 7fea2f874640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.192+0000 7fea2f874640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281ce100 0x7fea280a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.192+0000 7fea2f874640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.192+0000 7fea2f874640 1 -- 172.21.15.98:0/1782954493 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fea28075170 con 0x7fea281ce100 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea2ce0f640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281ce100 0x7fea280a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea2ce0f640 10 monclient: get_auth_request con 0x7fea281ce100 auth_method 0 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea2ce0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea2ce0f640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea2ce0f640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea2ce0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea2ce0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea2ce0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea2ce0f640 10 monclient(hunting): handle_auth_done global_id 124323 payload 824 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea2ce0f640 1 -- 172.21.15.98:0/1782954493 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fea280a1910 con 0x7fea281ce100 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea2ce0f640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281ce100 0x7fea280a5b60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fea24013000 tx=0x7fea24000bc0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 10 client.85304.objecter ms_handle_connect 0x7fea281ce100 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 10 client.85304.objecter resend_mon_ops 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea24018040 con 0x7fea281ce100 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 10 monclient: dump: 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.213 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9409 (secure 0 0 0) 0x7fea24013410 con 0x7fea281ce100 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281ce100 msgr2=0x7fea280a5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281ce100 0x7fea280a5b60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fea24013000 tx=0x7fea24000bc0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 1 -- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281ce100 msgr2=0x7fea280a5b60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:58.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.193+0000 7fea19ffb640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea281ce100 0x7fea280a5b60 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fea24013000 tx=0x7fea24000bc0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.215 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 86, 2024-06-11T19:24:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 41121, 2024-06-11T19:24:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.225 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.228 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:58.231 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:58.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.234 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.235 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:58.235 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:58.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 10 librados: watch_flush enter 2024-06-11T19:24:58.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 10 librados: watch_flush exit 2024-06-11T19:24:58.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 20 client.85304.objecter shutdown clearing up homeless session... 2024-06-11T19:24:58.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 10 client.85304.objecter successfully canceled tick 2024-06-11T19:24:58.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 1 -- 172.21.15.98:0/1782954493 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fea0c03fdb0 msgr2=0x7fea0c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fea0c03fdb0 0x7fea0c042270 secure :-1 s=READY pgs=4310 cs=0 l=1 rev1=1 crypto rx=0x7fea28072960 tx=0x7fea200095f0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 10 monclient: shutdown 2024-06-11T19:24:58.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:58.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 1 -- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea28073830 msgr2=0x7fea281226e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 1 --2- 172.21.15.98:0/1782954493 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fea28073830 0x7fea281226e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fea1c000c00 tx=0x7fea1c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2de11640 1 -- 172.21.15.98:0/1782954493 reap_dead start 2024-06-11T19:24:58.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2de11640 1 -- 172.21.15.98:0/1782954493 reap_dead start 2024-06-11T19:24:58.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 1 -- 172.21.15.98:0/1782954493 shutdown_connections 2024-06-11T19:24:58.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 1 -- 172.21.15.98:0/1782954493 >> 172.21.15.98:0/1782954493 conn(0x7fea280a4780 msgr2=0x7fea28133ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:58.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 1 -- 172.21.15.98:0/1782954493 shutdown_connections 2024-06-11T19:24:58.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 1 -- 172.21.15.98:0/1782954493 wait complete. 2024-06-11T19:24:58.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.194+0000 7fea2f874640 1 librados: shutdown 2024-06-11T19:24:58.238 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-06-11T19:24:58.238 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon e 2024-06-11T19:24:58.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.340+0000 7f603eca1640 10 monclient: get_monmap_and_config 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.340+0000 7f603eca1640 10 monclient: build_initial_monmap 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.341+0000 7f603eca1640 10 monclient: monmap: 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:58.342046+0000 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:58.342046+0000 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:58.340 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:58.341 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:58.341 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:58.341 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:58.341 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:58.341 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:58.341 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:58.341 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:58.341 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 1 Processor -- start 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 1 -- start start 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 10 monclient: init 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 10 monclient: _add_conns ranks=[8,9,0,6,1,3,10,7,2,4,5] 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60380a6260 0x7f60380a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 10 monclient(hunting): picked mon.noname-e con 0x7f60380a6260 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f603806c890 0x7f60380a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 10 monclient(hunting): picked mon.noname-a con 0x7f603806c890 addr v2:172.21.15.98:3300/0 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60380a2d00 addr v1:172.21.15.98:6789/0 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f603806c650 con 0x7f60380a2d00 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f60380a05b0 con 0x7f603806c890 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f60380a6640 con 0x7f60380a6260 2024-06-11T19:24:58.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:58.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603eca1640 10 monclient(hunting): authenticate will time out at 1708.149894s 2024-06-11T19:24:58.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f6037fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f603806c890 0x7f60380a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f603ca3d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60380a6260 0x7f60380a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f6037fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f603806c890 0x7f60380a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58226/0 (socket says 172.21.15.98:58226) 2024-06-11T19:24:58.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f6037fff640 1 -- 172.21.15.98:0/1643050117 learned_addr learned my addr 172.21.15.98:0/1643050117 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:58.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.342+0000 7f6037fff640 10 monclient(hunting): get_auth_request con 0x7f603806c890 auth_method 0 2024-06-11T19:24:58.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f6037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f6037fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f6037fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 10 monclient(hunting): get_auth_request con 0x7f60380a6260 auth_method 0 2024-06-11T19:24:58.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 10 monclient(hunting): handle_auth_done global_id 124325 payload 274 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 1 -- 172.21.15.98:0/1643050117 >> v2:172.21.15.98:3300/0 conn(0x7f603806c890 msgr2=0x7f60380a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 1 --2- 172.21.15.98:0/1643050117 >> v2:172.21.15.98:3300/0 conn(0x7f603806c890 0x7f60380a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 1 -- 172.21.15.98:0/1643050117 >> v1:172.21.15.98:6789/0 conn(0x7f60380a2d00 legacy=0x7f60380a4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 10 monclient: _finish_hunting 0 2024-06-11T19:24:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 1 monclient: found mon.noname-e 2024-06-11T19:24:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 1 -- 172.21.15.98:0/1643050117 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6038073b60 con 0x7f60380a6260 2024-06-11T19:24:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f6037fff640 1 --2- 172.21.15.98:0/1643050117 >> v2:172.21.15.98:3300/0 conn(0x7f603806c890 0x7f60380a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:58.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.343+0000 7f603ca3d640 1 --2- 172.21.15.98:0/1643050117 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60380a6260 0x7f60380a1d50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f602c0097a0 tx=0x7f602c02b7a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=f4ff2564e1e74ba3 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 1 -- 172.21.15.98:0/1643050117 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f602c038070 con 0x7f60380a6260 2024-06-11T19:24:58.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:58.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 10 monclient: dump: 2024-06-11T19:24:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.353 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.354 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 10 monclient: _finish_auth 0 2024-06-11T19:24:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 10 monclient: _check_auth_tickets 2024-06-11T19:24:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 1 -- 172.21.15.98:0/1643050117 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f602c031970 con 0x7f60380a6260 2024-06-11T19:24:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f603eca1640 5 monclient: authenticate success, global_id 124325 2024-06-11T19:24:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f603eca1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:58.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f603eca1640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 1 -- 172.21.15.98:0/1643050117 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f602c032390 con 0x7f60380a6260 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f603eca1640 4 set_mon_vals no callback set 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.345+0000 7f60377fe640 10 monclient: dump: 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.355 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.346+0000 7f603eca1640 10 monclient: shutdown 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.346+0000 7f603eca1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.346+0000 7f603eca1640 1 -- 172.21.15.98:0/1643050117 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60380a6260 msgr2=0x7f60380a1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.346+0000 7f603eca1640 1 --2- 172.21.15.98:0/1643050117 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60380a6260 0x7f60380a1d50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f602c0097a0 tx=0x7f602c02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.346+0000 7f603eca1640 1 -- 172.21.15.98:0/1643050117 shutdown_connections 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.346+0000 7f603eca1640 1 --2- 172.21.15.98:0/1643050117 >> v2:172.21.15.98:3300/0 conn(0x7f603806c890 0x7f60380a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.346+0000 7f603eca1640 1 --2- 172.21.15.98:0/1643050117 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60380a6260 0x7f60380a1d50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.346+0000 7f603eca1640 1 -- 172.21.15.98:0/1643050117 >> 172.21.15.98:0/1643050117 conn(0x7f60380a95b0 msgr2=0x7f60380a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.346+0000 7f603eca1640 1 -- 172.21.15.98:0/1643050117 shutdown_connections 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.346+0000 7f603eca1640 1 -- 172.21.15.98:0/1643050117 wait complete. 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.347+0000 7f603eca1640 10 monclient: build_initial_monmap 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.347+0000 7f603eca1640 10 monclient: monmap: 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:58.357 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.347+0000 7f603eca1640 1 librados: starting msgr at 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.347+0000 7f603eca1640 1 librados: starting objecter 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.347+0000 7f603eca1640 1 Processor -- start 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.347+0000 7f603eca1640 1 -- start start 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.347+0000 7f603eca1640 1 librados: setting wanted keys 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.347+0000 7f603eca1640 1 librados: calling monclient init 2024-06-11T19:24:58.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.347+0000 7f603eca1640 10 monclient: init 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 10 monclient: _add_conns ranks=[8,4,5,0,3,6,1,7,2] 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603806c890 0x7f60381ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 10 monclient(hunting): picked mon.noname-e con 0x7f603806c890 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f60380a2d00 0x7f60381c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 10 monclient(hunting): picked mon.noname-c con 0x7f60380a2d00 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f60380a6260 0x7f60381cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 10 monclient(hunting): picked mon.noname-h con 0x7f60380a6260 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f60380a2410 con 0x7f60380a2d00 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f60380a2290 con 0x7f60380a6260 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f60380a5500 con 0x7f603806c890 2024-06-11T19:24:58.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f6037fff640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f60380a2d00 msgr2=0x7f60381c75a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603eca1640 10 monclient(hunting): authenticate will time out at 1708.155298s 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f6037fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f60380a2d00 0x7f60381c75a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603806c890 0x7f60381ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603806c890 0x7f60381ca010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:37392/0 (socket says 172.21.15.98:37392) 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 1 -- 172.21.15.98:0/3183255438 learned_addr learned my addr 172.21.15.98:0/3183255438 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 10 monclient(hunting): get_auth_request con 0x7f603806c890 auth_method 0 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603d23e640 1 -- 172.21.15.98:0/3183255438 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f60380a6260 msgr2=0x7f60381cd550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3302/0 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603d23e640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f60380a6260 0x7f60381cd550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:58.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 10 monclient(hunting): handle_auth_done global_id 124334 payload 824 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 1 -- 172.21.15.98:0/3183255438 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f60380a6260 msgr2=0x7f60381cd550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f60380a6260 0x7f60381cd550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 1 -- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f60380a2d00 msgr2=0x7f60381c75a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f60380a2d00 0x7f60381c75a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 10 monclient: _finish_hunting 0 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 1 monclient: found mon.noname-e 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 1 -- 172.21.15.98:0/3183255438 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60381ac490 con 0x7f603806c890 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f603ca3d640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603806c890 0x7f60381ca010 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f602c0096f0 tx=0x7f602c0046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f60357fa640 10 client.?.objecter ms_handle_connect 0x7f603806c890 2024-06-11T19:24:58.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f60357fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f602c038050 con 0x7f603806c890 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f60357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f60357fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f60357fa640 10 monclient: dump: 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.362 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f60357fa640 10 monclient: _finish_auth 0 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f60357fa640 10 monclient: _check_auth_tickets 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f60357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f603eca1640 5 monclient: authenticate success, global_id 124334 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.348+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f602c044070 con 0x7f603806c890 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f603eca1640 10 monclient: _renew_subs 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f603eca1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:58.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f603eca1640 1 -- 172.21.15.98:0/3183255438 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60381aef00 con 0x7f603806c890 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f60357fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f602c031e20 con 0x7f603806c890 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f60357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f60357fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f60357fa640 10 monclient: dump: 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.364 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f603eca1640 10 client.124334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f603eca1640 10 monclient: _renew_subs 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f603eca1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f603eca1640 1 -- 172.21.15.98:0/3183255438 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f60380a8880 con 0x7f603806c890 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f603eca1640 1 librados: init done 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.349+0000 7f6036ffd640 4 set_mon_vals no callback set 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f603eca1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f603eca1640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60381cda90 0x7f60381cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f603eca1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f603eca1640 1 -- 172.21.15.98:0/3183255438 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6038050830 con 0x7f60381cda90 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f6037fff640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60381cda90 0x7f60381cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f6037fff640 10 monclient: get_auth_request con 0x7f60381cda90 auth_method 0 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f6037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f6037fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f6037fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f602c003610 con 0x7f603806c890 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f6037fff640 10 monclient(hunting): handle_auth_done global_id 124343 payload 824 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f60357fa640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f601c03ff40 0x7f601c042400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f6037fff640 1 -- 172.21.15.98:0/3183255438 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60380a70f0 con 0x7f60381cda90 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f60357fa640 10 client.124334.objecter ms_handle_connect 0x7f60381cda90 2024-06-11T19:24:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f6037fff640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60381cda90 0x7f60381cde70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f60381aafb0 tx=0x7f602400eb40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f60357fa640 10 client.124334.objecter resend_mon_ops 2024-06-11T19:24:58.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f603d23e640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f601c03ff40 0x7f601c042400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.350+0000 7f603d23e640 10 monclient: get_auth_request con 0x7f601c03ff40 auth_method 0 2024-06-11T19:24:58.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.351+0000 7f60357fa640 10 client.124334.objecter ms_handle_connect 0x7f601c03ff40 2024-06-11T19:24:58.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.351+0000 7f603d23e640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f601c03ff40 0x7f601c042400 secure :-1 s=READY pgs=4311 cs=0 l=1 rev1=1 crypto rx=0x7f6030000f80 tx=0x7f6030007d10 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.357+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6024017070 con 0x7f60381cda90 2024-06-11T19:24:58.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.357+0000 7f60357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.357+0000 7f60357fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:58.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.357+0000 7f60357fa640 10 monclient: dump: 2024-06-11T19:24:58.369 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.369 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.357+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(628..628 src has 1..628) ==== 6823+0+0 (secure 0 0 0) 0x7f602c033cf0 con 0x7f603806c890 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.357+0000 7f60357fa640 10 client.124334.objecter ms_dispatch 0x7f6038000b90 osd_map(628..628 src has 1..628) 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.357+0000 7f60357fa640 3 client.124334.objecter handle_osd_map got epochs [628,628] > 0 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.357+0000 7f60357fa640 3 client.124334.objecter handle_osd_map decoding full epoch 628 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.358+0000 7f60357fa640 20 client.124334.objecter dump_active .. 0 homeless 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.358+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f602400c710 con 0x7f60381cda90 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.358+0000 7f60357fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.358+0000 7f60357fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.358+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60381cda90 msgr2=0x7f60381cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.358+0000 7f60357fa640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60381cda90 0x7f60381cde70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f60381aafb0 tx=0x7f602400eb40 comp rx=0 tx=0).stop 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.358+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60381cda90 msgr2=0x7f60381cde70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.358+0000 7f60357fa640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60381cda90 0x7f60381cde70 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f60381aafb0 tx=0x7f602400eb40 comp rx=0 tx=0).stop 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.364+0000 7f603eca1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.364+0000 7f603eca1640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603809d290 0x7f603809d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.364+0000 7f603eca1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.364+0000 7f603eca1640 1 -- 172.21.15.98:0/3183255438 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6038069740 con 0x7f603809d290 2024-06-11T19:24:58.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.364+0000 7f6037fff640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603809d290 0x7f603809d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.364+0000 7f6037fff640 10 monclient: get_auth_request con 0x7f603809d290 auth_method 0 2024-06-11T19:24:58.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.364+0000 7f6037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.364+0000 7f6037fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.364+0000 7f6037fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f6037fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f6037fff640 10 monclient(hunting): handle_auth_done global_id 124352 payload 824 2024-06-11T19:24:58.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f6037fff640 1 -- 172.21.15.98:0/3183255438 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f60381ae7f0 con 0x7f603809d290 2024-06-11T19:24:58.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 10 client.124334.objecter ms_handle_connect 0x7f603809d290 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f6037fff640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603809d290 0x7f603809d6a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f60381aafb0 tx=0x7f6024004730 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 10 client.124334.objecter resend_mon_ops 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6024017070 con 0x7f603809d290 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 10 monclient: dump: 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.373 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.374 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9239 (secure 0 0 0) 0x7f6024006370 con 0x7f603809d290 2024-06-11T19:24:58.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603809d290 msgr2=0x7f603809d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603809d290 0x7f603809d6a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f60381aafb0 tx=0x7f6024004730 comp rx=0 tx=0).stop 2024-06-11T19:24:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 1 -- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603809d290 msgr2=0x7f603809d6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:58.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f60357fa640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603809d290 0x7f603809d6a0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f60381aafb0 tx=0x7f6024004730 comp rx=0 tx=0).stop 2024-06-11T19:24:58.375 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:58.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:58.375 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:58.375 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:58.375 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 86, 2024-06-11T19:24:58.375 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 41302, 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.377 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:58.378 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.379 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.380 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.381 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.388 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.389 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.390 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.392 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:58.394 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:58.395 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:58.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f603eca1640 10 librados: watch_flush enter 2024-06-11T19:24:58.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f603eca1640 10 librados: watch_flush exit 2024-06-11T19:24:58.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.365+0000 7f603eca1640 20 client.124334.objecter shutdown clearing up homeless session... 2024-06-11T19:24:58.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603eca1640 10 client.124334.objecter successfully canceled tick 2024-06-11T19:24:58.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603eca1640 1 -- 172.21.15.98:0/3183255438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f601c03ff40 msgr2=0x7f601c042400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603eca1640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f601c03ff40 0x7f601c042400 secure :-1 s=READY pgs=4311 cs=0 l=1 rev1=1 crypto rx=0x7f6030000f80 tx=0x7f6030007d10 comp rx=0 tx=0).stop 2024-06-11T19:24:58.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603eca1640 10 monclient: shutdown 2024-06-11T19:24:58.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603eca1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:58.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603eca1640 1 -- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603806c890 msgr2=0x7f60381ca010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603eca1640 1 --2- 172.21.15.98:0/3183255438 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f603806c890 0x7f60381ca010 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f602c0096f0 tx=0x7f602c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603d23e640 1 -- 172.21.15.98:0/3183255438 reap_dead start 2024-06-11T19:24:58.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603d23e640 1 -- 172.21.15.98:0/3183255438 reap_dead start 2024-06-11T19:24:58.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603eca1640 1 -- 172.21.15.98:0/3183255438 shutdown_connections 2024-06-11T19:24:58.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603eca1640 1 -- 172.21.15.98:0/3183255438 >> 172.21.15.98:0/3183255438 conn(0x7f60380a95b0 msgr2=0x7f60380725f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:58.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603eca1640 1 -- 172.21.15.98:0/3183255438 shutdown_connections 2024-06-11T19:24:58.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603eca1640 1 -- 172.21.15.98:0/3183255438 wait complete. 2024-06-11T19:24:58.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.366+0000 7f603eca1640 1 librados: shutdown 2024-06-11T19:24:58.399 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-06-11T19:24:58.399 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon f 2024-06-11T19:24:58.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:24:58.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.499+0000 7f745ba20640 10 monclient: get_monmap_and_config 2024-06-11T19:24:58.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.499+0000 7f745ba20640 10 monclient: build_initial_monmap 2024-06-11T19:24:58.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.499+0000 7f745ba20640 10 monclient: monmap: 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:58.500841+0000 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:58.500841+0000 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:58.499 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:58.500 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:58.500 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:58.500 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.500+0000 7f745ba20640 1 Processor -- start 2024-06-11T19:24:58.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.500+0000 7f745ba20640 1 -- start start 2024-06-11T19:24:58.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.500+0000 7f745ba20640 10 monclient: init 2024-06-11T19:24:58.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:58.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 10 monclient: _add_conns ranks=[7,9,0,5,3,6,10,1,8,4,2] 2024-06-11T19:24:58.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74540a4920 0x7f74540a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 10 monclient(hunting): picked mon.noname-i con 0x7f74540a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:24:58.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74540a5240 0x7f74540a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 10 monclient(hunting): picked mon.noname-a con 0x7f74540a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f74540a2530 addr v1:172.21.15.98:6789/0 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74540a7140 con 0x7f74540a2530 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f74540a2b50 con 0x7f74540a5240 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f74540a2e30 con 0x7f74540a4920 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f745ba20640 10 monclient(hunting): authenticate will time out at 1708.308156s 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74540a5240 0x7f74540a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74540a5240 0x7f74540a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58242/0 (socket says 172.21.15.98:58242) 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f74597bc640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74540a4920 msgr2=0x7f74540a4d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:24:58.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 1 -- 172.21.15.98:0/4226334025 learned_addr learned my addr 172.21.15.98:0/4226334025 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 10 monclient(hunting): get_auth_request con 0x7f74540a5240 auth_method 0 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f74597bc640 1 --2- 172.21.15.98:0/4226334025 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74540a4920 0x7f74540a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 10 monclient(hunting): handle_auth_done global_id 66267 payload 274 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 1 -- 172.21.15.98:0/4226334025 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74540a4920 msgr2=0x7f74540a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 1 --2- 172.21.15.98:0/4226334025 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74540a4920 0x7f74540a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 1 -- 172.21.15.98:0/4226334025 >> v1:172.21.15.98:6789/0 conn(0x7f74540a2530 legacy=0x7f74540a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 10 monclient: _finish_hunting 0 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 1 monclient: found mon.noname-a 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.501+0000 7f7458fbb640 1 -- 172.21.15.98:0/4226334025 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74540a3110 con 0x7f74540a5240 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f7458fbb640 1 --2- 172.21.15.98:0/4226334025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74540a5240 0x7f74540a59a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7444009870 tx=0x7f744402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef001ad023dcd75 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 1 -- 172.21.15.98:0/4226334025 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f744402e070 con 0x7f74540a5240 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 10 monclient: dump: 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.513 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 10 monclient: _finish_auth 0 2024-06-11T19:24:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:24:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f745ba20640 5 monclient: authenticate success, global_id 66267 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 1 -- 172.21.15.98:0/4226334025 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7444037070 con 0x7f74540a5240 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f745ba20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f745ba20640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 1 -- 172.21.15.98:0/4226334025 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7444032470 con 0x7f74540a5240 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f745ba20640 4 set_mon_vals no callback set 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f744bfff640 10 monclient: dump: 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f745ba20640 10 monclient: shutdown 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f745ba20640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f745ba20640 1 -- 172.21.15.98:0/4226334025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74540a5240 msgr2=0x7f74540a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.502+0000 7f745ba20640 1 --2- 172.21.15.98:0/4226334025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74540a5240 0x7f74540a59a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7444009870 tx=0x7f744402b880 comp rx=0 tx=0).stop 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.503+0000 7f745ba20640 1 -- 172.21.15.98:0/4226334025 shutdown_connections 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.503+0000 7f745ba20640 1 --2- 172.21.15.98:0/4226334025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74540a5240 0x7f74540a59a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.503+0000 7f745ba20640 1 --2- 172.21.15.98:0/4226334025 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f74540a4920 0x7f74540a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.503+0000 7f745ba20640 1 -- 172.21.15.98:0/4226334025 >> 172.21.15.98:0/4226334025 conn(0x7f74540ab850 msgr2=0x7f74540a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.503+0000 7f745ba20640 1 -- 172.21.15.98:0/4226334025 shutdown_connections 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.503+0000 7f745ba20640 1 -- 172.21.15.98:0/4226334025 wait complete. 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.503+0000 7f745ba20640 10 monclient: build_initial_monmap 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.503+0000 7f745ba20640 10 monclient: monmap: 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.503+0000 7f745ba20640 1 librados: starting msgr at 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.503+0000 7f745ba20640 1 librados: starting objecter 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 1 Processor -- start 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 1 -- start start 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 1 librados: setting wanted keys 2024-06-11T19:24:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 1 librados: calling monclient init 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 10 monclient: init 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 10 monclient: _add_conns ranks=[4,3,8,5,0,2,6,1,7] 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f74540a2530 0x7f7454137160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 10 monclient(hunting): picked mon.noname-c con 0x7f74540a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74540a4920 0x7f74541376a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 10 monclient(hunting): picked mon.noname-g con 0x7f74540a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f74540a5240 0x7f7454123120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 10 monclient(hunting): picked mon.noname-e con 0x7f74540a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f74540a6da0 con 0x7f74540a4920 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f74597bc640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f74540a2530 msgr2=0x7f7454137160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:24:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7454138e50 con 0x7f74540a2530 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f74540a65f0 con 0x7f74540a5240 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f745ba20640 10 monclient(hunting): authenticate will time out at 1708.311586s 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f74597bc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f74540a2530 0x7f7454137160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f74540a5240 0x7f7454123120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f74540a5240 0x7f7454123120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:37426/0 (socket says 172.21.15.98:37426) 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 1 -- 172.21.15.98:0/605594724 learned_addr learned my addr 172.21.15.98:0/605594724 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 10 monclient(hunting): get_auth_request con 0x7f74540a5240 auth_method 0 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 10 monclient(hunting): handle_auth_done global_id 124361 payload 824 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 1 -- 172.21.15.98:0/605594724 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f74540a2530 msgr2=0x7f7454137160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f74540a2530 0x7f7454137160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7458fbb640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74540a4920 0x7f74541376a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 1 -- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74540a4920 msgr2=0x7f74541376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74540a4920 0x7f74541376a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 10 monclient: _finish_hunting 0 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.504+0000 7f7459fbd640 1 monclient: found mon.noname-e 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7459fbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7459fbd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7459fbd640 1 -- 172.21.15.98:0/605594724 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7454126d60 con 0x7f74540a5240 2024-06-11T19:24:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7459fbd640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f74540a5240 0x7f7454123120 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f744c00c7d0 tx=0x7f744c00cc90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 10 client.?.objecter ms_handle_connect 0x7f74540a5240 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f744c001860 con 0x7f74540a5240 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 10 monclient: dump: 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 10 monclient: _finish_auth 0 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f744c003de0 con 0x7f74540a5240 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f745ba20640 5 monclient: authenticate success, global_id 124361 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f745ba20640 10 monclient: _renew_subs 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f745ba20640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f745ba20640 1 -- 172.21.15.98:0/605594724 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7454127040 con 0x7f74540a5240 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f744c00acc0 con 0x7f74540a5240 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f7449ffb640 10 monclient: dump: 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.523 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f744b7fe640 4 set_mon_vals no callback set 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f745ba20640 10 client.124361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f745ba20640 10 monclient: _renew_subs 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f745ba20640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f745ba20640 1 -- 172.21.15.98:0/605594724 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f74540aab50 con 0x7f74540a5240 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.505+0000 7f745ba20640 1 librados: init done 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f745ba20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f744c00a850 con 0x7f74540a5240 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f745ba20640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123660 0x7f7454123a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f745ba20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f745ba20640 1 -- 172.21.15.98:0/605594724 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f74541256e0 con 0x7f7454123660 2024-06-11T19:24:58.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f7449ffb640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f743803fdb0 0x7f7438042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f74597bc640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123660 0x7f7454123a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(628..628 src has 1..628) ==== 6823+0+0 (secure 0 0 0) 0x7f744c051830 con 0x7f74540a5240 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f7449ffb640 10 client.124361.objecter ms_dispatch 0x7f7454000b90 osd_map(628..628 src has 1..628) 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f7449ffb640 3 client.124361.objecter handle_osd_map got epochs [628,628] > 0 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f7449ffb640 3 client.124361.objecter handle_osd_map decoding full epoch 628 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f74597bc640 10 monclient: get_auth_request con 0x7f7454123660 auth_method 0 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f74597bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f74597bc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f74597bc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f7449ffb640 20 client.124361.objecter dump_active .. 0 homeless 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f74597bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f74597bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f74597bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f7458fbb640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f743803fdb0 0x7f7438042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f7458fbb640 10 monclient: get_auth_request con 0x7f743803fdb0 auth_method 0 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f74597bc640 10 monclient(hunting): handle_auth_done global_id 124264 payload 824 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.506+0000 7f74597bc640 1 -- 172.21.15.98:0/605594724 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7454137f90 con 0x7f7454123660 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f74597bc640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123660 0x7f7454123a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f745000b450 tx=0x7f745000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 10 client.124361.objecter ms_handle_connect 0x7f7454123660 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 10 client.124361.objecter resend_mon_ops 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 10 client.124361.objecter ms_handle_connect 0x7f743803fdb0 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7458fbb640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f743803fdb0 0x7f7438042270 secure :-1 s=READY pgs=4312 cs=0 l=1 rev1=1 crypto rx=0x7f7444036000 tx=0x7f7444008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7450003bc0 con 0x7f7454123660 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 10 monclient: dump: 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.526 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.527 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7450015070 con 0x7f7454123660 2024-06-11T19:24:58.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:58.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:58.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123660 msgr2=0x7f7454123a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123660 0x7f7454123a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f745000b450 tx=0x7f745000b910 comp rx=0 tx=0).stop 2024-06-11T19:24:58.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123660 msgr2=0x7f7454123a40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:58.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.507+0000 7f7449ffb640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123660 0x7f7454123a40 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f745000b450 tx=0x7f745000b910 comp rx=0 tx=0).stop 2024-06-11T19:24:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.513+0000 7f745ba20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.513+0000 7f745ba20640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123f80 0x7f745409e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.513+0000 7f745ba20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.513+0000 7f745ba20640 1 -- 172.21.15.98:0/605594724 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7454050830 con 0x7f7454123f80 2024-06-11T19:24:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f74597bc640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123f80 0x7f745409e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f74597bc640 10 monclient: get_auth_request con 0x7f7454123f80 auth_method 0 2024-06-11T19:24:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f74597bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f74597bc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f74597bc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f74597bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f74597bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f74597bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f74597bc640 10 monclient(hunting): handle_auth_done global_id 124273 payload 824 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f74597bc640 1 -- 172.21.15.98:0/605594724 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f74540aa970 con 0x7f7454123f80 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f74597bc640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123f80 0x7f745409e9f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f745000b450 tx=0x7f74500059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f7449ffb640 10 client.124361.objecter ms_handle_connect 0x7f7454123f80 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f7449ffb640 10 client.124361.objecter resend_mon_ops 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7450010400 con 0x7f7454123f80 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f7449ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f7449ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.514+0000 7f7449ffb640 10 monclient: dump: 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9239 (secure 0 0 0) 0x7f7450005410 con 0x7f7454123f80 2024-06-11T19:24:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f7449ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f7449ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123f80 msgr2=0x7f745409e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f7449ffb640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123f80 0x7f745409e9f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f745000b450 tx=0x7f74500059d0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f7449ffb640 1 -- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123f80 msgr2=0x7f745409e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f7449ffb640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7454123f80 0x7f745409e9f0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f745000b450 tx=0x7f74500059d0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.533 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:58.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:58.533 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:58.533 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 86, 2024-06-11T19:24:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 41448, 2024-06-11T19:24:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:24:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:58.537 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.538 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:58.539 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.541 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:58.542 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.544 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.548 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:58.549 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.552 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.553 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:58.553 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:58.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 10 librados: watch_flush enter 2024-06-11T19:24:58.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 10 librados: watch_flush exit 2024-06-11T19:24:58.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 20 client.124361.objecter shutdown clearing up homeless session... 2024-06-11T19:24:58.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 10 client.124361.objecter successfully canceled tick 2024-06-11T19:24:58.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 1 -- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f743803fdb0 msgr2=0x7f7438042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f743803fdb0 0x7f7438042270 secure :-1 s=READY pgs=4312 cs=0 l=1 rev1=1 crypto rx=0x7f7444036000 tx=0x7f7444008030 comp rx=0 tx=0).stop 2024-06-11T19:24:58.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 10 monclient: shutdown 2024-06-11T19:24:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 1 -- 172.21.15.98:0/605594724 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f74540a5240 msgr2=0x7f7454123120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 1 --2- 172.21.15.98:0/605594724 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f74540a5240 0x7f7454123120 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f744c00c7d0 tx=0x7f744c00cc90 comp rx=0 tx=0).stop 2024-06-11T19:24:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f7459fbd640 1 -- 172.21.15.98:0/605594724 reap_dead start 2024-06-11T19:24:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f7459fbd640 1 -- 172.21.15.98:0/605594724 reap_dead start 2024-06-11T19:24:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 1 -- 172.21.15.98:0/605594724 shutdown_connections 2024-06-11T19:24:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 1 -- 172.21.15.98:0/605594724 >> 172.21.15.98:0/605594724 conn(0x7f74540ab850 msgr2=0x7f74541be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 1 -- 172.21.15.98:0/605594724 shutdown_connections 2024-06-11T19:24:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 1 -- 172.21.15.98:0/605594724 wait complete. 2024-06-11T19:24:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.515+0000 7f745ba20640 1 librados: shutdown 2024-06-11T19:24:58.556 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-06-11T19:24:58.556 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon g 2024-06-11T19:24:58.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:24:58.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.656+0000 7fb55b042640 10 monclient: get_monmap_and_config 2024-06-11T19:24:58.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.656+0000 7fb55b042640 10 monclient: build_initial_monmap 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.656+0000 7fb55b042640 10 monclient: monmap: 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:24:58.657750+0000 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:24:58.657750+0000 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:24:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:24:58.657 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.657+0000 7fb55b042640 1 Processor -- start 2024-06-11T19:24:58.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.657+0000 7fb55b042640 1 -- start start 2024-06-11T19:24:58.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.657+0000 7fb55b042640 10 monclient: init 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.657+0000 7fb55b042640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.657+0000 7fb55b042640 10 monclient: _add_conns ranks=[9,6,8,2,7,5,3,4,0,1,10] 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.657+0000 7fb55b042640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5540a4920 0x7fb5540a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.657+0000 7fb55b042640 10 monclient(hunting): picked mon.noname-a con 0x7fb5540a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.657+0000 7fb55b042640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5540a5240 0x7fb5540a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb55b042640 10 monclient(hunting): picked mon.noname-d con 0x7fb5540a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb55b042640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5540a2530 0x7fb5540a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb55b042640 10 monclient(hunting): picked mon.noname-e con 0x7fb5540a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb55b042640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb55b042640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb5540a7140 con 0x7fb5540a4920 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb55b042640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb55b042640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb5540a2b50 con 0x7fb5540a5240 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb55b042640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb55b042640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb5540a2e30 con 0x7fb5540a2530 2024-06-11T19:24:58.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb55b042640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:58.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb55b042640 10 monclient(hunting): authenticate will time out at 1708.465060s 2024-06-11T19:24:58.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb54bfff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5540a5240 0x7fb5540a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb558dde640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5540a4920 0x7fb5540a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5540a2530 0x7fb5540a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb54bfff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5540a5240 0x7fb5540a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34866/0 (socket says 172.21.15.98:34866) 2024-06-11T19:24:58.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb54bfff640 1 -- 172.21.15.98:0/263324229 learned_addr learned my addr 172.21.15.98:0/263324229 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:58.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb54bfff640 10 monclient(hunting): get_auth_request con 0x7fb5540a5240 auth_method 0 2024-06-11T19:24:58.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb54bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb54bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb54bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb558dde640 10 monclient(hunting): get_auth_request con 0x7fb5540a4920 auth_method 0 2024-06-11T19:24:58.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb558dde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb558dde640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb558dde640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 10 monclient(hunting): get_auth_request con 0x7fb5540a2530 auth_method 0 2024-06-11T19:24:58.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb558dde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb558dde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb558dde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb54bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb54bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb54bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 10 monclient(hunting): handle_auth_done global_id 124370 payload 274 2024-06-11T19:24:58.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 1 -- 172.21.15.98:0/263324229 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5540a5240 msgr2=0x7fb5540a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:58.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 1 --2- 172.21.15.98:0/263324229 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5540a5240 0x7fb5540a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 1 -- 172.21.15.98:0/263324229 >> v2:172.21.15.98:3300/0 conn(0x7fb5540a4920 msgr2=0x7fb5540a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 1 --2- 172.21.15.98:0/263324229 >> v2:172.21.15.98:3300/0 conn(0x7fb5540a4920 0x7fb5540a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 10 monclient: _finish_hunting 0 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 1 monclient: found mon.noname-e 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb5595df640 1 -- 172.21.15.98:0/263324229 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5540a3110 con 0x7fb5540a2530 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.658+0000 7fb558dde640 1 --2- 172.21.15.98:0/263324229 >> v2:172.21.15.98:3300/0 conn(0x7fb5540a4920 0x7fb5540a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb5595df640 1 --2- 172.21.15.98:0/263324229 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5540a2530 0x7fb5540a5ee0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009d90 tx=0x7fb54c0344f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=ea6a2e9e18901642 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 1 -- 172.21.15.98:0/263324229 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb54c0079b0 con 0x7fb5540a2530 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 10 monclient: dump: 2024-06-11T19:24:58.671 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.672 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.673 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 10 monclient: _finish_auth 0 2024-06-11T19:24:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:24:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 1 -- 172.21.15.98:0/263324229 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb54c007b70 con 0x7fb5540a2530 2024-06-11T19:24:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb55b042640 5 monclient: authenticate success, global_id 124370 2024-06-11T19:24:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb55b042640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:24:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:58.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb55b042640 10 monclient: get_monmap_and_config success 2024-06-11T19:24:58.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 1 -- 172.21.15.98:0/263324229 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb54c007d60 con 0x7fb5540a2530 2024-06-11T19:24:58.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb55b042640 4 set_mon_vals no callback set 2024-06-11T19:24:58.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:24:58.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb54b7fe640 10 monclient: dump: 2024-06-11T19:24:58.674 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.674 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.674 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.674 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.674 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.674 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.675 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.675 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.675 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.675 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.675 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.675 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.675 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.675 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.675 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb55b042640 10 monclient: shutdown 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb55b042640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb55b042640 1 -- 172.21.15.98:0/263324229 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5540a2530 msgr2=0x7fb5540a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.659+0000 7fb55b042640 1 --2- 172.21.15.98:0/263324229 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5540a2530 0x7fb5540a5ee0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009d90 tx=0x7fb54c0344f0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 -- 172.21.15.98:0/263324229 shutdown_connections 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 --2- 172.21.15.98:0/263324229 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5540a2530 0x7fb5540a5ee0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 --2- 172.21.15.98:0/263324229 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5540a5240 0x7fb5540a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 --2- 172.21.15.98:0/263324229 >> v2:172.21.15.98:3300/0 conn(0x7fb5540a4920 0x7fb5540a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 -- 172.21.15.98:0/263324229 >> 172.21.15.98:0/263324229 conn(0x7fb5540ab850 msgr2=0x7fb5540a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 -- 172.21.15.98:0/263324229 shutdown_connections 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 -- 172.21.15.98:0/263324229 wait complete. 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 10 monclient: build_initial_monmap 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 10 monclient: monmap: 2024-06-11T19:24:58.676 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:24:58.677 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 librados: starting msgr at 2024-06-11T19:24:58.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 librados: starting objecter 2024-06-11T19:24:58.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 Processor -- start 2024-06-11T19:24:58.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 -- start start 2024-06-11T19:24:58.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 librados: setting wanted keys 2024-06-11T19:24:58.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 1 librados: calling monclient init 2024-06-11T19:24:58.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.660+0000 7fb55b042640 10 monclient: init 2024-06-11T19:24:58.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 10 monclient: _reopen_session rank -1 2024-06-11T19:24:58.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 10 monclient: _add_conns ranks=[3,0,2,7,5,1,6,8,4] 2024-06-11T19:24:58.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5540a2530 0x7fb5541215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 10 monclient(hunting): picked mon.noname-g con 0x7fb5540a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:24:58.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5540a4920 0x7fb554121b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 10 monclient(hunting): picked mon.noname-a con 0x7fb5540a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:24:58.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5540a5240 0x7fb554122070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 10 monclient(hunting): picked mon.noname-b con 0x7fb5540a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:24:58.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb5540a6da0 con 0x7fb5540a4920 2024-06-11T19:24:58.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb554138e50 con 0x7fb5540a5240 2024-06-11T19:24:58.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fb5540a6420 con 0x7fb5540a2530 2024-06-11T19:24:58.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 10 monclient(hunting): _renew_subs 2024-06-11T19:24:58.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb55b042640 10 monclient(hunting): authenticate will time out at 1708.468575s 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5540a4920 0x7fb554121b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5540a4920 0x7fb554121b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58252/0 (socket says 172.21.15.98:58252) 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 1 -- 172.21.15.98:0/219140684 learned_addr learned my addr 172.21.15.98:0/219140684 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 10 monclient(hunting): get_auth_request con 0x7fb5540a4920 auth_method 0 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb5595df640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5540a5240 0x7fb554122070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb5595df640 10 monclient(hunting): get_auth_request con 0x7fb5540a5240 auth_method 0 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb5595df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb5595df640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb5595df640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb5595df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb5595df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb5595df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 10 monclient(hunting): handle_auth_done global_id 66294 payload 824 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 1 -- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5540a2530 msgr2=0x7fb5541215f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb558dde640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5540a2530 0x7fb5541215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5540a2530 0x7fb5541215f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 1 -- 172.21.15.98:0/219140684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5540a5240 msgr2=0x7fb554122070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5540a5240 0x7fb554122070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 1 monclient: found mon.noname-a 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:24:58.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.661+0000 7fb54bfff640 1 -- 172.21.15.98:0/219140684 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb554137d60 con 0x7fb5540a4920 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb558dde640 10 monclient: get_auth_request con 0x7fb5540a2530 auth_method 0 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb558dde640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5540a2530 0x7fb5541215f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5595df640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5540a5240 0x7fb554122070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb54bfff640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5540a4920 0x7fb554121b30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb54400e840 tx=0x7fb54400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 10 client.?.objecter ms_handle_connect 0x7fb5540a4920 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb544017070 con 0x7fb5540a4920 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 10 monclient: dump: 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.682 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 10 monclient: _finish_auth 0 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 10 monclient: _check_auth_tickets 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:24:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb54400cdb0 con 0x7fb5540a4920 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb55b042640 5 monclient: authenticate success, global_id 66294 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb55b042640 10 monclient: _renew_subs 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb55b042640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb55b042640 1 -- 172.21.15.98:0/219140684 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb554124d80 con 0x7fb5540a4920 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5440138e0 con 0x7fb5540a4920 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb5497fa640 10 monclient: dump: 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb55b042640 10 client.66294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb55b042640 10 monclient: _renew_subs 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb55b042640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:24:58.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb54affd640 4 set_mon_vals no callback set 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb55b042640 1 -- 172.21.15.98:0/219140684 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5540a8da0 con 0x7fb5540a4920 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.662+0000 7fb55b042640 1 librados: init done 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb55b042640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb55b042640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb554124720 0x7fb5541c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb55b042640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb55b042640 1 -- 172.21.15.98:0/219140684 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fb554050830 con 0x7fb554124720 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fb54401a070 con 0x7fb5540a4920 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb5497fa640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb54003fdb0 0x7fb540042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb558dde640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb554124720 0x7fb5541c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(628..628 src has 1..628) ==== 6823+0+0 (secure 0 0 0) 0x7fb54400f040 con 0x7fb5540a4920 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb5497fa640 10 client.66294.objecter ms_dispatch 0x7fb554000b90 osd_map(628..628 src has 1..628) 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb5497fa640 3 client.66294.objecter handle_osd_map got epochs [628,628] > 0 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb5497fa640 3 client.66294.objecter handle_osd_map decoding full epoch 628 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb558dde640 10 monclient: get_auth_request con 0x7fb554124720 auth_method 0 2024-06-11T19:24:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb558dde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb558dde640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb558dde640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb5497fa640 20 client.66294.objecter dump_active .. 0 homeless 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb558dde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb558dde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb558dde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb5595df640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb54003fdb0 0x7fb540042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb5595df640 10 monclient: get_auth_request con 0x7fb54003fdb0 auth_method 0 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.663+0000 7fb558dde640 10 monclient(hunting): handle_auth_done global_id 114438 payload 824 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb558dde640 1 -- 172.21.15.98:0/219140684 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb554071400 con 0x7fb554124720 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 10 client.66294.objecter ms_handle_connect 0x7fb554124720 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 10 client.66294.objecter resend_mon_ops 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb558dde640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb554124720 0x7fb5541c97c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb550010720 tx=0x7fb550010be0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 10 client.66294.objecter ms_handle_connect 0x7fb54003fdb0 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5595df640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb54003fdb0 0x7fb540042270 secure :-1 s=READY pgs=4313 cs=0 l=1 rev1=1 crypto rx=0x7fb54c03e000 tx=0x7fb54c039000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb55000f040 con 0x7fb554124720 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 10 monclient: dump: 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.688 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.689 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb55001c400 con 0x7fb554124720 2024-06-11T19:24:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:24:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:24:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb554124720 msgr2=0x7fb5541c97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb554124720 0x7fb5541c97c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb550010720 tx=0x7fb550010be0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb554124720 msgr2=0x7fb5541c97c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.664+0000 7fb5497fa640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb554124720 0x7fb5541c97c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb55b042640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:24:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb55b042640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5541c9d00 0x7fb5541ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:24:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb55b042640 10 monclient(hunting): start opening mon connection 2024-06-11T19:24:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb55b042640 1 -- 172.21.15.98:0/219140684 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fb5541ca650 con 0x7fb5541c9d00 2024-06-11T19:24:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb558dde640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5541c9d00 0x7fb5541ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:24:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb558dde640 10 monclient: get_auth_request con 0x7fb5541c9d00 auth_method 0 2024-06-11T19:24:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb558dde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:24:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb558dde640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:24:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb558dde640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb558dde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb558dde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb558dde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb558dde640 10 monclient(hunting): handle_auth_done global_id 114447 payload 824 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.671+0000 7fb558dde640 1 -- 172.21.15.98:0/219140684 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb554123580 con 0x7fb5541c9d00 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb558dde640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5541c9d00 0x7fb5541ca110 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb550010720 tx=0x7fb5500046c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 10 client.66294.objecter ms_handle_connect 0x7fb5541c9d00 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 10 client.66294.objecter resend_mon_ops 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5500212d0 con 0x7fb5541c9d00 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 10 monclient: dump: 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:24:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:24:58.693 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:24:58.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9240 (secure 0 0 0) 0x7fb550025070 con 0x7fb5541c9d00 2024-06-11T19:24:58.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:24:58.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:24:58.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5541c9d00 msgr2=0x7fb5541ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5541c9d00 0x7fb5541ca110 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb550010720 tx=0x7fb5500046c0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 1 -- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5541c9d00 msgr2=0x7fb5541ca110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:24:58.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb5497fa640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5541c9d00 0x7fb5541ca110 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:24:58.694 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:24:58.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:58.694 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:58.694 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 86, 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 187041, 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:24:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:24:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:24:58.697 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:24:58.698 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.699 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.700 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:24:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:24:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:24:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:24:58.705 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.708 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:24:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:24:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:24:58.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:24:58.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:24:58.713 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:24:58.714 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:24:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:24:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:24:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:24:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:24:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:24:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:24:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:24:58.715 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:24:58.715 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:24:58.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb55b042640 10 librados: watch_flush enter 2024-06-11T19:24:58.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb55b042640 10 librados: watch_flush exit 2024-06-11T19:24:58.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb55b042640 20 client.66294.objecter shutdown clearing up homeless session... 2024-06-11T19:24:58.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.672+0000 7fb55b042640 10 client.66294.objecter successfully canceled tick 2024-06-11T19:24:58.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb55b042640 1 -- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb54003fdb0 msgr2=0x7fb540042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb55b042640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb54003fdb0 0x7fb540042270 secure :-1 s=READY pgs=4313 cs=0 l=1 rev1=1 crypto rx=0x7fb54c03e000 tx=0x7fb54c039000 comp rx=0 tx=0).stop 2024-06-11T19:24:58.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb55b042640 10 monclient: shutdown 2024-06-11T19:24:58.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb55b042640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:24:58.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb55b042640 1 -- 172.21.15.98:0/219140684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5540a4920 msgr2=0x7fb554121b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:24:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb55b042640 1 --2- 172.21.15.98:0/219140684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5540a4920 0x7fb554121b30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb54400e840 tx=0x7fb54400ed00 comp rx=0 tx=0).stop 2024-06-11T19:24:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb5595df640 1 -- 172.21.15.98:0/219140684 reap_dead start 2024-06-11T19:24:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb5595df640 1 -- 172.21.15.98:0/219140684 reap_dead start 2024-06-11T19:24:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb55b042640 1 -- 172.21.15.98:0/219140684 shutdown_connections 2024-06-11T19:24:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb55b042640 1 -- 172.21.15.98:0/219140684 >> 172.21.15.98:0/219140684 conn(0x7fb5540ab850 msgr2=0x7fb554070cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:24:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb55b042640 1 -- 172.21.15.98:0/219140684 shutdown_connections 2024-06-11T19:24:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb55b042640 1 -- 172.21.15.98:0/219140684 wait complete. 2024-06-11T19:24:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:24:58.673+0000 7fb55b042640 1 librados: shutdown 2024-06-11T19:24:58.718 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-06-11T19:24:58.718 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-06-11T19:26:21.396 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7993 ms) 2024-06-11T19:26:21.396 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-06-11T19:26:21.396 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13031 ms) 2024-06-11T19:26:21.396 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7986 ms) 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13010 ms) 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8018 ms) 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (7965 ms) 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: hmm, no HitSet yet 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: ok, hit_set contains 332:602f83fe:::foo:head 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (23040 ms) 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133899,0 2024-06-11T19:26:21.397 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: first is 1718133899 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133899,0 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133899,0 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133899,1718133901,1718133902,0 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133899,1718133901,1718133902,0 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133899,1718133901,1718133902,0 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133899,1718133901,1718133902,1718133904,1718133905,0 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133899,1718133901,1718133902,1718133904,1718133905,0 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133899,1718133901,1718133902,1718133904,1718133905,0 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133899,1718133901,1718133902,1718133904,1718133905,1718133907,1718133908,0 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133899,1718133901,1718133902,1718133904,1718133905,1718133907,1718133908,0 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133899,1718133901,1718133902,1718133904,1718133905,1718133907,1718133908,0 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: got ls 1718133902,1718133904,1718133905,1718133907,1718133908,1718133910,1718133911,0 2024-06-11T19:26:21.398 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: first now 1718133902, trimmed 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (35348 ms) 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: foo0 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13635 ms) 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17361 ms) 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (21658 ms) 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5040 ms) 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3036 ms) 2024-06-11T19:26:21.399 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-06-11T19:26:21.400 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (2918 ms) 2024-06-11T19:26:21.400 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-06-11T19:26:21.400 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3021 ms) 2024-06-11T19:26:21.400 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (276343 ms total) 2024-06-11T19:26:21.400 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: 2024-06-11T19:26:21.400 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-06-11T19:26:21.400 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (947176 ms total) 2024-06-11T19:26:21.400 INFO:tasks.workunit.client.0.smithi117.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-06-11T19:26:21.402 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.402 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36449 2024-06-11T19:26:21.402 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36449 2024-06-11T19:26:21.402 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.402 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36687 2024-06-11T19:26:21.402 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36687 2024-06-11T19:26:21.402 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.402 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37174 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37174 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36919 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36919 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36799 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36799 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36523 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36523 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36851 2024-06-11T19:26:21.403 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36851 2024-06-11T19:26:21.404 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.404 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37359 2024-06-11T19:26:21.404 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37359 2024-06-11T19:26:21.404 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.404 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37433 2024-06-11T19:26:21.404 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37433 2024-06-11T19:26:21.404 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.404 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36573 2024-06-11T19:26:21.404 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36573 2024-06-11T19:26:21.404 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.404 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37242 2024-06-11T19:26:21.404 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37242 2024-06-11T19:26:21.405 INFO:tasks.workunit.client.0.smithi117.stdout:error in cls (37242) 2024-06-11T19:26:21.405 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'error in cls (37242)' 2024-06-11T19:26:21.405 INFO:tasks.workunit.client.0.smithi117.stderr:+ ret=1 2024-06-11T19:26:21.406 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.406 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36424 2024-06-11T19:26:21.406 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36424 2024-06-11T19:26:21.406 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.406 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36487 2024-06-11T19:26:21.406 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36487 2024-06-11T19:26:21.406 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.406 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37107 2024-06-11T19:26:21.406 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37107 2024-06-11T19:26:21.406 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.406 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36415 2024-06-11T19:26:21.406 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36415 2024-06-11T19:26:21.407 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.407 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36435 2024-06-11T19:26:21.407 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36435 2024-06-11T19:26:21.407 INFO:tasks.workunit.client.0.smithi117.stdout:error in api_io_pp (36435) 2024-06-11T19:26:21.410 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'error in api_io_pp (36435)' 2024-06-11T19:26:21.410 INFO:tasks.workunit.client.0.smithi117.stderr:+ ret=1 2024-06-11T19:26:21.410 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.410 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37550 2024-06-11T19:26:21.410 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37550 2024-06-11T19:26:21.410 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.410 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36707 2024-06-11T19:26:21.410 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36707 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37612 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37612 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36838 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36838 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37326 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37326 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37706 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37706 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.411 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36721 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36721 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37146 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37146 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37495 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37495 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37302 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37302 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37591 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37591 2024-06-11T19:26:21.412 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37059 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37059 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36629 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36629 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37659 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37659 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36940 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36940 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36408 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36408 2024-06-11T19:26:21.413 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36466 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36466 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36870 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36870 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36506 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36506 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=36964 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 36964 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37034 2024-06-11T19:26:21.414 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37034 2024-06-11T19:26:21.415 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.415 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37475 2024-06-11T19:26:21.415 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37475 2024-06-11T19:26:21.415 INFO:tasks.workunit.client.0.smithi117.stderr:+ for t in "${!pids[@]}" 2024-06-11T19:26:21.415 INFO:tasks.workunit.client.0.smithi117.stderr:+ pid=37757 2024-06-11T19:26:21.415 INFO:tasks.workunit.client.0.smithi117.stderr:+ wait 37757 2024-06-11T19:26:21.415 INFO:tasks.workunit.client.0.smithi117.stderr:+ exit 1 2024-06-11T19:26:21.415 INFO:tasks.workunit.client.0.smithi117.stderr:+ cleanup 2024-06-11T19:26:21.415 INFO:tasks.workunit.client.0.smithi117.stderr:+ pkill -P 36402 2024-06-11T19:26:21.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T19:26:21.416 INFO:tasks.workunit.client.0.smithi117.stderr:+ true 2024-06-11T19:26:22.440 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2024-06-11T19:26:22.441 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-06-11T19:26:22.703 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 529, in run_unit_test r = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37bbe1d8d788b7c9a71280e7d510c295a5853f0b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa/tasks/workunit.py", line 426, in _run_tests remote.run_unit_test( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 534, in run_unit_test raise UnitTestError( teuthology.exceptions.UnitTestError: Unit test failed (workunit test rados/test.sh) on smithi117 with status 1: '(total 4 failed) FAILURE: Test `CrcZeroWrite` of `LibRadosIoECPP`. Reason: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-4216-g37bbe1d8/rpm/el9/BUILD/ceph-19.0....' 2024-06-11T19:26:22.871 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=525c3dbf9f57437b9f48582eef8032aa Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 529, in run_unit_test r = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37bbe1d8d788b7c9a71280e7d510c295a5853f0b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_37bbe1d8d788b7c9a71280e7d510c295a5853f0b/qa/tasks/workunit.py", line 426, in _run_tests remote.run_unit_test( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 534, in run_unit_test raise UnitTestError( teuthology.exceptions.UnitTestError: Unit test failed (workunit test rados/test.sh) on smithi117 with status 1: '(total 4 failed) FAILURE: Test `CrcZeroWrite` of `LibRadosIoECPP`. Reason: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-4216-g37bbe1d8/rpm/el9/BUILD/ceph-19.0....' 2024-06-11T19:26:22.873 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-06-11T19:26:22.884 INFO:tasks.mon_thrash:joining mon_thrasher 2024-06-11T19:26:28.719 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-06-11T19:26:28.719 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.h 2024-06-11T19:26:28.719 INFO:tasks.ceph.mon.h:Restarting daemon 2024-06-11T19:26:28.720 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-06-11T19:26:28.722 INFO:tasks.ceph.mon.h:Started 2024-06-11T19:26:28.723 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-06-11T19:26:28.723 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-06-11T19:26:28.723 INFO:tasks.ceph.mon.c:Restarting daemon 2024-06-11T19:26:28.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-06-11T19:26:28.725 INFO:tasks.ceph.mon.c:Started 2024-06-11T19:26:28.726 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-06-11T19:26:28.726 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-06-11T19:26:28.726 INFO:tasks.ceph.mon.i:Restarting daemon 2024-06-11T19:26:28.726 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-06-11T19:26:28.728 INFO:tasks.ceph.mon.i:Started 2024-06-11T19:26:28.728 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:26:28.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:26:28.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.830+0000 7f8bb98c7640 10 monclient: get_monmap_and_config 2024-06-11T19:26:28.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.830+0000 7f8bb98c7640 10 monclient: build_initial_monmap 2024-06-11T19:26:28.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.830+0000 7f8bb98c7640 10 monclient: monmap: 2024-06-11T19:26:28.830 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:28.830 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:28.830 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:28.831849+0000 2024-06-11T19:26:28.830 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:28.831849+0000 2024-06-11T19:26:28.830 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:28.831 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:28.831 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:28.831 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:28.831 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:28.831 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:28.831 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:28.831 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:28.831 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:28.831 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:28.832 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:28.832 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:28.832 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:28.832 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 1 Processor -- start 2024-06-11T19:26:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 1 -- start start 2024-06-11T19:26:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 10 monclient: init 2024-06-11T19:26:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 10 monclient: _add_conns ranks=[4,6,8,0,5,1,9,3,7,10,2] 2024-06-11T19:26:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8bb40a52a0 0x7f8bb40a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:28.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 10 monclient(hunting): picked mon.noname-c con 0x7f8bb40a52a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:28.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8bb409d520 0x7f8bb40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:28.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 10 monclient(hunting): picked mon.noname-d con 0x7f8bb409d520 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:28.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb4073830 0x7f8bb40a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:28.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 10 monclient(hunting): picked mon.noname-e con 0x7f8bb4073830 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:28.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f8bb409daa0 con 0x7f8bb40a52a0 2024-06-11T19:26:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8bb409d2e0 con 0x7f8bb409d520 2024-06-11T19:26:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8bb40a5680 con 0x7f8bb4073830 2024-06-11T19:26:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.831+0000 7f8bb98c7640 10 monclient(hunting): authenticate will time out at 1798.638978s 2024-06-11T19:26:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb2ffd640 1 -- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8bb40a52a0 msgr2=0x7f8bb40a1910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3302/0 2024-06-11T19:26:28.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb2ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8bb40a52a0 0x7f8bb40a1910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:26:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8bb409d520 0x7f8bb40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb37fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb4073830 0x7f8bb40a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8bb409d520 0x7f8bb40a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:57900/0 (socket says 172.21.15.98:57900) 2024-06-11T19:26:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 1 -- 172.21.15.98:0/302035468 learned_addr learned my addr 172.21.15.98:0/302035468 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 10 monclient(hunting): get_auth_request con 0x7f8bb409d520 auth_method 0 2024-06-11T19:26:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:28.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb37fe640 10 monclient(hunting): get_auth_request con 0x7f8bb4073830 auth_method 0 2024-06-11T19:26:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:28.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 10 monclient(hunting): handle_auth_done global_id 124341 payload 274 2024-06-11T19:26:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 1 -- 172.21.15.98:0/302035468 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb4073830 msgr2=0x7f8bb40a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 1 --2- 172.21.15.98:0/302035468 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb4073830 0x7f8bb40a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 1 -- 172.21.15.98:0/302035468 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8bb40a52a0 msgr2=0x7f8bb40a1910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:26:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 1 --2- 172.21.15.98:0/302035468 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8bb40a52a0 0x7f8bb40a1910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 10 monclient: _finish_hunting 0 2024-06-11T19:26:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 1 monclient: found mon.noname-d 2024-06-11T19:26:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:28.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb27fc640 1 -- 172.21.15.98:0/302035468 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb409e9d0 con 0x7f8bb409d520 2024-06-11T19:26:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.832+0000 7f8bb37fe640 1 --2- 172.21.15.98:0/302035468 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb4073830 0x7f8bb40a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb27fc640 1 --2- 172.21.15.98:0/302035468 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8bb409d520 0x7f8bb40a3690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8ba80098a0 tx=0x7f8ba802b7c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=a065fe678ead3223 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 1 -- 172.21.15.98:0/302035468 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ba8038070 con 0x7f8bb409d520 2024-06-11T19:26:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 10 monclient: dump: 2024-06-11T19:26:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:28.847 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:28.848 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:28.849 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 10 monclient: _finish_auth 0 2024-06-11T19:26:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:26:28.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 1 -- 172.21.15.98:0/302035468 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba8031990 con 0x7f8bb409d520 2024-06-11T19:26:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb98c7640 5 monclient: authenticate success, global_id 124341 2024-06-11T19:26:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb98c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb98c7640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 1 -- 172.21.15.98:0/302035468 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ba80323b0 con 0x7f8bb409d520 2024-06-11T19:26:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:28.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb98c7640 4 set_mon_vals no callback set 2024-06-11T19:26:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.833+0000 7f8bb1ffb640 10 monclient: dump: 2024-06-11T19:26:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:28.851 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:28.852 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:28.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 10 monclient: shutdown 2024-06-11T19:26:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 1 -- 172.21.15.98:0/302035468 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8bb409d520 msgr2=0x7f8bb40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 1 --2- 172.21.15.98:0/302035468 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8bb409d520 0x7f8bb40a3690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8ba80098a0 tx=0x7f8ba802b7c0 comp rx=0 tx=0).stop 2024-06-11T19:26:28.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 1 -- 172.21.15.98:0/302035468 shutdown_connections 2024-06-11T19:26:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 1 --2- 172.21.15.98:0/302035468 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb4073830 0x7f8bb40a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 1 --2- 172.21.15.98:0/302035468 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8bb409d520 0x7f8bb40a3690 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 1 --2- 172.21.15.98:0/302035468 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8bb40a52a0 0x7f8bb40a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 1 -- 172.21.15.98:0/302035468 >> 172.21.15.98:0/302035468 conn(0x7f8bb40a4780 msgr2=0x7f8bb40a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 1 -- 172.21.15.98:0/302035468 shutdown_connections 2024-06-11T19:26:28.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 1 -- 172.21.15.98:0/302035468 wait complete. 2024-06-11T19:26:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 10 monclient: build_initial_monmap 2024-06-11T19:26:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.834+0000 7f8bb98c7640 10 monclient: monmap: 2024-06-11T19:26:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:28.855 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:28.856 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:28.856 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:28.856 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:28.856 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:28.856 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:28.856 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:28.856 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:28.856 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:28.857 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:28.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 1 librados: starting msgr at 2024-06-11T19:26:28.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 1 librados: starting objecter 2024-06-11T19:26:28.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 1 Processor -- start 2024-06-11T19:26:28.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 1 -- start start 2024-06-11T19:26:28.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 1 librados: setting wanted keys 2024-06-11T19:26:28.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 1 librados: calling monclient init 2024-06-11T19:26:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 10 monclient: init 2024-06-11T19:26:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 10 monclient: _add_conns ranks=[1,8,7,3,5,0,2,6,4] 2024-06-11T19:26:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bb4073830 0x7f8bb4133e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 10 monclient(hunting): picked mon.noname-f con 0x7f8bb4073830 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:26:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb409d520 0x7f8bb4134370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 10 monclient(hunting): picked mon.noname-e con 0x7f8bb409d520 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8bb40a52a0 0x7f8bb411ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 10 monclient(hunting): picked mon.noname-i con 0x7f8bb40a52a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb40a1fd0 con 0x7f8bb4073830 2024-06-11T19:26:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f8bb40a1e50 con 0x7f8bb40a52a0 2024-06-11T19:26:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8bb40a2150 con 0x7f8bb409d520 2024-06-11T19:26:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb98c7640 10 monclient(hunting): authenticate will time out at 1798.642911s 2024-06-11T19:26:28.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb37fe640 1 -- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8bb40a52a0 msgr2=0x7f8bb411ff50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:3303/0 2024-06-11T19:26:28.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.835+0000 7f8bb37fe640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8bb40a52a0 0x7f8bb411ff50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:26:28.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb409d520 0x7f8bb4134370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:28.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb409d520 0x7f8bb4134370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34798/0 (socket says 172.21.15.98:34798) 2024-06-11T19:26:28.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 1 -- 172.21.15.98:0/1291189590 learned_addr learned my addr 172.21.15.98:0/1291189590 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:28.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 10 monclient(hunting): get_auth_request con 0x7f8bb409d520 auth_method 0 2024-06-11T19:26:28.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:28.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb2ffd640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bb4073830 0x7f8bb4133e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb2ffd640 10 monclient(hunting): get_auth_request con 0x7f8bb4073830 auth_method 0 2024-06-11T19:26:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 10 monclient(hunting): handle_auth_done global_id 124388 payload 824 2024-06-11T19:26:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 1 -- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8bb40a52a0 msgr2=0x7f8bb411ff50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:26:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8bb40a52a0 0x7f8bb411ff50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 1 -- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bb4073830 msgr2=0x7f8bb4133e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bb4073830 0x7f8bb4133e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 10 monclient: _finish_hunting 0 2024-06-11T19:26:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 1 monclient: found mon.noname-e 2024-06-11T19:26:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:26:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 1 -- 172.21.15.98:0/1291189590 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb40a1430 con 0x7f8bb409d520 2024-06-11T19:26:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb27fc640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb409d520 0x7f8bb4134370 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8000c00 tx=0x7f8ba80046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 10 client.?.objecter ms_handle_connect 0x7f8bb409d520 2024-06-11T19:26:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 1 -- 172.21.15.98:0/1291189590 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ba8042020 con 0x7f8bb409d520 2024-06-11T19:26:28.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:28.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:26:28.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 10 monclient: dump: 2024-06-11T19:26:28.866 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:28.866 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:28.866 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:28.866 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:28.866 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:28.867 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:28.867 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:28.867 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:28.867 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:28.867 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:28.867 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:28.867 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:28.867 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:28.867 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:28.868 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:28.868 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:28.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 10 monclient: _finish_auth 0 2024-06-11T19:26:28.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:26:28.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:28.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb98c7640 5 monclient: authenticate success, global_id 124388 2024-06-11T19:26:28.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 1 -- 172.21.15.98:0/1291189590 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba8003610 con 0x7f8bb409d520 2024-06-11T19:26:28.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:28.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 1 -- 172.21.15.98:0/1291189590 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ba8038040 con 0x7f8bb409d520 2024-06-11T19:26:28.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb98c7640 10 monclient: _renew_subs 2024-06-11T19:26:28.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb98c7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:28.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb98c7640 1 -- 172.21.15.98:0/1291189590 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb4123dd0 con 0x7f8bb409d520 2024-06-11T19:26:28.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:28.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:28.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb17fa640 4 set_mon_vals no callback set 2024-06-11T19:26:28.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8b9bfff640 10 monclient: dump: 2024-06-11T19:26:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:28.871 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb98c7640 10 client.124388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb98c7640 10 monclient: _renew_subs 2024-06-11T19:26:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb98c7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb98c7640 1 -- 172.21.15.98:0/1291189590 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb4123530 con 0x7f8bb409d520 2024-06-11T19:26:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.836+0000 7f8bb98c7640 1 librados: init done 2024-06-11T19:26:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.837+0000 7f8b9bfff640 1 -- 172.21.15.98:0/1291189590 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f8ba8045020 con 0x7f8bb409d520 2024-06-11T19:26:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.837+0000 7f8b9bfff640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8ba003fdb0 0x7f8ba0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8bb98c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8bb98c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8bb98c7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8b9bfff640 1 -- 172.21.15.98:0/1291189590 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f8ba8030070 con 0x7f8bb409d520 2024-06-11T19:26:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8b9bfff640 10 client.124388.objecter ms_dispatch 0x7f8bb4000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8b9bfff640 3 client.124388.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8b9bfff640 3 client.124388.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8bb98c7640 1 -- 172.21.15.98:0/1291189590 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bb40a1910 con 0x7f8bb409d520 2024-06-11T19:26:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8b9bfff640 20 client.124388.objecter dump_active .. 0 homeless 2024-06-11T19:26:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8bb2ffd640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8ba003fdb0 0x7f8ba0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8bb2ffd640 10 monclient: get_auth_request con 0x7f8ba003fdb0 auth_method 0 2024-06-11T19:26:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8b9bfff640 10 client.124388.objecter ms_handle_connect 0x7f8ba003fdb0 2024-06-11T19:26:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.838+0000 7f8bb2ffd640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8ba003fdb0 0x7f8ba0042270 secure :-1 s=READY pgs=4314 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4003980 tx=0x7f8ba4009110 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.842+0000 7f8b9bfff640 1 -- 172.21.15.98:0/1291189590 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8ba804c030 con 0x7f8bb409d520 2024-06-11T19:26:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.842+0000 7f8b9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:28.842+0000 7f8b9bfff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:28.875 INFO:tasks.ceph.mon.h.smithi117.stderr:2024-06-11T19:26:28.854+0000 7f6406a48b00 -1 mon.h@-1(???).log v0 log_external_backlog rewinding external_log_to from 642 -> 0 (sync_force? mon rebuild?) 2024-06-11T19:26:28.876 INFO:tasks.ceph.mon.c.smithi098.stderr:2024-06-11T19:26:28.843+0000 7f6f22318b00 -1 mon.c@-1(???).log v0 log_external_backlog rewinding external_log_to from 642 -> 0 (sync_force? mon rebuild?) 2024-06-11T19:26:28.876 INFO:tasks.ceph.mon.i.smithi117.stderr:2024-06-11T19:26:28.855+0000 7fe52ec74b00 -1 mon.i@-1(???).log v0 log_external_backlog rewinding external_log_to from 642 -> 0 (sync_force? mon rebuild?) 2024-06-11T19:26:29.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.104+0000 7f8bb98c7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:26:29.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.104+0000 7f8bb98c7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:26:29.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.104+0000 7f8bb98c7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:29.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.104+0000 7f8bb98c7640 1 -- 172.21.15.98:0/1291189590 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8bb4123a50 con 0x7f8bb409d520 2024-06-11T19:26:29.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.104+0000 7f8b9bfff640 1 -- 172.21.15.98:0/1291189590 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3175 (secure 0 0 0) 0x7f8ba8003b40 con 0x7f8bb409d520 2024-06-11T19:26:29.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.104+0000 7f8b9bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:26:29.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.104+0000 7f8b9bfff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:29.105 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":86,"quorum":[0,1,2,3,6,8],"quorum_names":["a","f","b","g","d","e"],"quorum_leader_name":"b","quorum_age":91,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:26:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.106+0000 7f8bb98c7640 10 librados: watch_flush enter 2024-06-11T19:26:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.106+0000 7f8bb98c7640 10 librados: watch_flush exit 2024-06-11T19:26:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.106+0000 7f8bb98c7640 20 client.124388.objecter shutdown clearing up homeless session... 2024-06-11T19:26:29.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.106+0000 7f8bb98c7640 10 client.124388.objecter successfully canceled tick 2024-06-11T19:26:29.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 -- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8ba003fdb0 msgr2=0x7f8ba0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:29.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8ba003fdb0 0x7f8ba0042270 secure :-1 s=READY pgs=4314 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4003980 tx=0x7f8ba4009110 comp rx=0 tx=0).stop 2024-06-11T19:26:29.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 10 monclient: shutdown 2024-06-11T19:26:29.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:29.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 -- 172.21.15.98:0/1291189590 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb409d520 msgr2=0x7f8bb4134370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:29.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb409d520 0x7f8bb4134370 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8000c00 tx=0x7f8ba80046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:29.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 -- 172.21.15.98:0/1291189590 shutdown_connections 2024-06-11T19:26:29.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f8ba003fdb0 0x7f8ba0042270 unknown :-1 s=CLOSED pgs=4314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:29.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f8bb40a52a0 0x7f8bb411ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:29.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8bb409d520 0x7f8bb4134370 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:29.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 --2- 172.21.15.98:0/1291189590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bb4073830 0x7f8bb4133e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:29.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 -- 172.21.15.98:0/1291189590 >> 172.21.15.98:0/1291189590 conn(0x7f8bb40a4780 msgr2=0x7f8bb40a5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:29.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 -- 172.21.15.98:0/1291189590 shutdown_connections 2024-06-11T19:26:29.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 -- 172.21.15.98:0/1291189590 wait complete. 2024-06-11T19:26:29.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:29.107+0000 7f8bb98c7640 1 librados: shutdown 2024-06-11T19:26:32.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:26:32.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.227+0000 7f1cd4cda640 10 monclient: get_monmap_and_config 2024-06-11T19:26:32.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.227+0000 7f1cd4cda640 10 monclient: build_initial_monmap 2024-06-11T19:26:32.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.227+0000 7f1cd4cda640 10 monclient: monmap: 2024-06-11T19:26:32.227 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:32.227 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:32.227 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:32.228310+0000 2024-06-11T19:26:32.227 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:32.228310+0000 2024-06-11T19:26:32.228 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:32.228 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:32.228 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:32.228 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:32.228 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:32.228 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:32.228 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:32.228 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:32.228 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:32.229 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:32.229 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:32.229 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:32.229 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:32.229 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:32.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.227+0000 7f1cd4cda640 1 Processor -- start 2024-06-11T19:26:32.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.227+0000 7f1cd4cda640 1 -- start start 2024-06-11T19:26:32.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.227+0000 7f1cd4cda640 10 monclient: init 2024-06-11T19:26:32.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:32.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 10 monclient: _add_conns ranks=[9,6,7,8,5,10,0,2,1,3,4] 2024-06-11T19:26:32.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1cd00a4920 0x7f1cd00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:32.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 10 monclient(hunting): picked mon.noname-a con 0x7f1cd00a4920 addr v2:172.21.15.98:3300/0 2024-06-11T19:26:32.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cd00a5240 0x7f1cd00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:32.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 10 monclient(hunting): picked mon.noname-d con 0x7f1cd00a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:32.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 0x7f1cd00a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 10 monclient(hunting): picked mon.noname-i con 0x7f1cd00a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1cd00a7140 con 0x7f1cd00a4920 2024-06-11T19:26:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1cd00a2b50 con 0x7f1cd00a5240 2024-06-11T19:26:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f1cd00a2e30 con 0x7f1cd00a2530 2024-06-11T19:26:32.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cd4cda640 10 monclient(hunting): authenticate will time out at 1802.035574s 2024-06-11T19:26:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cce59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1cd00a4920 0x7f1cd00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cc5d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cd00a5240 0x7f1cd00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cce59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1cd00a4920 0x7f1cd00a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45880/0 (socket says 172.21.15.98:45880) 2024-06-11T19:26:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cce59c640 1 -- 172.21.15.98:0/3449644957 learned_addr learned my addr 172.21.15.98:0/3449644957 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cc5d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cd00a5240 0x7f1cd00a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:57952/0 (socket says 172.21.15.98:57952) 2024-06-11T19:26:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cce59c640 10 monclient(hunting): get_auth_request con 0x7f1cd00a4920 auth_method 0 2024-06-11T19:26:32.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:32.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cce59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:32.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cce59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:32.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cc5d9b640 10 monclient(hunting): get_auth_request con 0x7f1cd00a5240 auth_method 0 2024-06-11T19:26:32.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cc5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:32.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cc5d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:32.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:32.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:32.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:32.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:32.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cc5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:32.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cc5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:32.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cc5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:32.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cce59c640 10 monclient(hunting): handle_auth_done global_id 66303 payload 274 2024-06-11T19:26:32.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cced9d640 1 --2- 172.21.15.98:0/3449644957 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 0x7f1cd00a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:32.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.228+0000 7f1cce59c640 1 -- 172.21.15.98:0/3449644957 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 msgr2=0x7f1cd00a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:32.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1cce59c640 1 --2- 172.21.15.98:0/3449644957 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 0x7f1cd00a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1cce59c640 1 -- 172.21.15.98:0/3449644957 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cd00a5240 msgr2=0x7f1cd00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:32.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1cce59c640 1 --2- 172.21.15.98:0/3449644957 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cd00a5240 0x7f1cd00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1cce59c640 10 monclient: _finish_hunting 0 2024-06-11T19:26:32.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1cce59c640 1 monclient: found mon.noname-a 2024-06-11T19:26:32.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1cce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:32.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1cce59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:26:32.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1cce59c640 1 -- 172.21.15.98:0/3449644957 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd00a3110 con 0x7f1cd00a4920 2024-06-11T19:26:32.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1cced9d640 10 monclient: get_auth_request con 0x7f1cd00a2530 auth_method 0 2024-06-11T19:26:32.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1cc5d9b640 1 --2- 172.21.15.98:0/3449644957 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cd00a5240 0x7f1cd00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:32.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1cced9d640 1 --2- 172.21.15.98:0/3449644957 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 0x7f1cd00a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:26:32.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1cce59c640 1 --2- 172.21.15.98:0/3449644957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cd00a4920 0x7f1cd00a4d00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009870 tx=0x7f1cb802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf87ba93d1bd9d7 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:32.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1ccdd9b640 1 -- 172.21.15.98:0/3449644957 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1cb8038070 con 0x7f1cd00a4920 2024-06-11T19:26:32.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1ccdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:32.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1ccdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:26:32.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1ccdd9b640 10 monclient: dump: 2024-06-11T19:26:32.248 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:32.248 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:32.248 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:32.249 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:32.249 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:32.249 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:32.249 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:32.249 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:32.249 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:32.249 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:32.249 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:32.249 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:32.250 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:32.250 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:32.250 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:32.250 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:32.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1ccdd9b640 10 monclient: _finish_auth 0 2024-06-11T19:26:32.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1ccdd9b640 10 monclient: _check_auth_tickets 2024-06-11T19:26:32.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.229+0000 7f1ccdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:32.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1ccdd9b640 1 -- 172.21.15.98:0/3449644957 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cb8031990 con 0x7f1cd00a4920 2024-06-11T19:26:32.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1ccdd9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:32.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1ccdd9b640 1 -- 172.21.15.98:0/3449644957 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1cb80323b0 con 0x7f1cd00a4920 2024-06-11T19:26:32.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 5 monclient: authenticate success, global_id 66303 2024-06-11T19:26:32.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:32.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 4 set_mon_vals no callback set 2024-06-11T19:26:32.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1ccdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:32.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1ccdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:32.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1ccdd9b640 10 monclient: dump: 2024-06-11T19:26:32.251 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:32.252 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:32.252 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:32.252 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:32.252 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:32.252 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:32.252 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:32.252 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:32.252 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:32.253 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:32.253 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:32.253 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:32.253 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:32.253 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:32.253 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:32.253 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:32.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 10 monclient: shutdown 2024-06-11T19:26:32.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:32.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 1 -- 172.21.15.98:0/3449644957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cd00a4920 msgr2=0x7f1cd00a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:32.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 1 --2- 172.21.15.98:0/3449644957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cd00a4920 0x7f1cd00a4d00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009870 tx=0x7f1cb802b7c0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 1 -- 172.21.15.98:0/3449644957 shutdown_connections 2024-06-11T19:26:32.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 1 --2- 172.21.15.98:0/3449644957 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 0x7f1cd00a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 1 --2- 172.21.15.98:0/3449644957 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1cd00a5240 0x7f1cd00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 1 --2- 172.21.15.98:0/3449644957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cd00a4920 0x7f1cd00a4d00 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 1 -- 172.21.15.98:0/3449644957 >> 172.21.15.98:0/3449644957 conn(0x7f1cd00ab850 msgr2=0x7f1cd00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:32.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 1 -- 172.21.15.98:0/3449644957 shutdown_connections 2024-06-11T19:26:32.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.230+0000 7f1cd4cda640 1 -- 172.21.15.98:0/3449644957 wait complete. 2024-06-11T19:26:32.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.231+0000 7f1cd4cda640 10 monclient: build_initial_monmap 2024-06-11T19:26:32.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.231+0000 7f1cd4cda640 10 monclient: monmap: 2024-06-11T19:26:32.255 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:32.255 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:32.255 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:32.255 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:32.255 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:32.256 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:32.256 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:32.256 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:32.256 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:32.256 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:32.256 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:32.256 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:32.256 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:32.257 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:32.257 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:32.257 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:32.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.231+0000 7f1cd4cda640 1 librados: starting msgr at 2024-06-11T19:26:32.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.231+0000 7f1cd4cda640 1 librados: starting objecter 2024-06-11T19:26:32.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.231+0000 7f1cd4cda640 1 Processor -- start 2024-06-11T19:26:32.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 1 -- start start 2024-06-11T19:26:32.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 1 librados: setting wanted keys 2024-06-11T19:26:32.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 1 librados: calling monclient init 2024-06-11T19:26:32.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 10 monclient: init 2024-06-11T19:26:32.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:32.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 10 monclient: _add_conns ranks=[7,1,3,6,5,2,0,8,4] 2024-06-11T19:26:32.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 0x7f1cd01cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:32.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 10 monclient(hunting): picked mon.noname-i con 0x7f1cd00a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:32.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cd00a4920 0x7f1cd01ce160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:32.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 10 monclient(hunting): picked mon.noname-f con 0x7f1cd00a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:26:32.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd00a5240 0x7f1cd009f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 10 monclient(hunting): picked mon.noname-g con 0x7f1cd00a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd00a6da0 con 0x7f1cd00a4920 2024-06-11T19:26:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f1cd0138e50 con 0x7f1cd00a5240 2024-06-11T19:26:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f1cd00a6420 con 0x7f1cd00a2530 2024-06-11T19:26:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:32.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cd4cda640 10 monclient(hunting): authenticate will time out at 1802.039509s 2024-06-11T19:26:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cced9d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd00a5240 0x7f1cd009f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cc5d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cd00a4920 0x7f1cd01ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cce59c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 0x7f1cd01cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cc5d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cd00a4920 0x7f1cd01ce160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:47582/0 (socket says 172.21.15.98:47582) 2024-06-11T19:26:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cc5d9b640 1 -- 172.21.15.98:0/2416332886 learned_addr learned my addr 172.21.15.98:0/2416332886 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cc5d9b640 10 monclient(hunting): get_auth_request con 0x7f1cd00a4920 auth_method 0 2024-06-11T19:26:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cced9d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd00a5240 0x7f1cd009f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:54120/0 (socket says 172.21.15.98:54120) 2024-06-11T19:26:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cce59c640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 0x7f1cd01cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:44868/0 (socket says 172.21.15.98:44868) 2024-06-11T19:26:32.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cc5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:32.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cc5d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:32.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:32.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cced9d640 10 monclient(hunting): get_auth_request con 0x7f1cd00a5240 auth_method 0 2024-06-11T19:26:32.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:32.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cced9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:32.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cced9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:32.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cce59c640 10 monclient(hunting): get_auth_request con 0x7f1cd00a2530 auth_method 0 2024-06-11T19:26:32.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:32.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cce59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cce59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.232+0000 7f1cc5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cce59c640 1 -- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 msgr2=0x7f1cd01cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:26:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cce59c640 1 -- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 msgr2=0x7f1cd01cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:26:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cce59c640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 0x7f1cd01cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:32.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:32.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cce59c640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 0x7f1cd01cdc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:26:32.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 10 monclient(hunting): handle_auth_done global_id 124291 payload 824 2024-06-11T19:26:32.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 1 -- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 msgr2=0x7f1cd01cdc20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:26:32.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 0x7f1cd01cdc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 1 -- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd00a5240 msgr2=0x7f1cd009f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:32.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd00a5240 0x7f1cd009f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:26:32.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 1 monclient: found mon.noname-f 2024-06-11T19:26:32.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:32.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:26:32.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 1 -- 172.21.15.98:0/2416332886 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd0122d60 con 0x7f1cd00a4920 2024-06-11T19:26:32.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cced9d640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd00a5240 0x7f1cd009f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:32.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc5d9b640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cd00a4920 0x7f1cd01ce160 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1cc000e900 tx=0x7f1cc000edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:32.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 10 client.?.objecter ms_handle_connect 0x7f1cd00a4920 2024-06-11T19:26:32.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 1 -- 172.21.15.98:0/2416332886 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1cc0017070 con 0x7f1cd00a4920 2024-06-11T19:26:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:26:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 10 monclient: dump: 2024-06-11T19:26:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:32.267 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 10 monclient: _finish_auth 0 2024-06-11T19:26:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 10 monclient: _check_auth_tickets 2024-06-11T19:26:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 1 -- 172.21.15.98:0/2416332886 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cc000ce70 con 0x7f1cd00a4920 2024-06-11T19:26:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cd4cda640 5 monclient: authenticate success, global_id 124291 2024-06-11T19:26:32.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:32.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 1 -- 172.21.15.98:0/2416332886 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1cc00138e0 con 0x7f1cd00a4920 2024-06-11T19:26:32.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cd4cda640 10 monclient: _renew_subs 2024-06-11T19:26:32.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cd4cda640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:26:32.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cd4cda640 1 -- 172.21.15.98:0/2416332886 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd00a92b0 con 0x7f1cd00a4920 2024-06-11T19:26:32.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:32.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:26:32.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1ccd59a640 4 set_mon_vals no callback set 2024-06-11T19:26:32.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cc77fe640 10 monclient: dump: 2024-06-11T19:26:32.269 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:32.269 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:32.269 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:32.269 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:32.269 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:32.269 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:32.269 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:32.269 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:32.270 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:32.270 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:32.270 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:32.270 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:32.270 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:32.270 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:32.270 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:32.270 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:32.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cd4cda640 10 client.124291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:32.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cd4cda640 10 monclient: _renew_subs 2024-06-11T19:26:32.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cd4cda640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:26:32.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cd4cda640 1 -- 172.21.15.98:0/2416332886 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cd00a1090 con 0x7f1cd00a4920 2024-06-11T19:26:32.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.233+0000 7f1cd4cda640 1 librados: init done 2024-06-11T19:26:32.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.234+0000 7f1cd4cda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:32.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.234+0000 7f1cd4cda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:32.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.234+0000 7f1cd4cda640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:26:32.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.234+0000 7f1cd4cda640 1 -- 172.21.15.98:0/2416332886 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cd00a6750 con 0x7f1cd00a4920 2024-06-11T19:26:32.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.234+0000 7f1cc77fe640 1 -- 172.21.15.98:0/2416332886 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f1cc001a070 con 0x7f1cd00a4920 2024-06-11T19:26:32.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.234+0000 7f1cc77fe640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1cb403fdb0 0x7f1cb4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:32.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.235+0000 7f1cc77fe640 1 -- 172.21.15.98:0/2416332886 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f1cc000f040 con 0x7f1cd00a4920 2024-06-11T19:26:32.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.235+0000 7f1cc77fe640 10 client.124291.objecter ms_dispatch 0x7f1cd0000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:32.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.235+0000 7f1cc77fe640 3 client.124291.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:32.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.235+0000 7f1cc77fe640 3 client.124291.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:32.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.235+0000 7f1cc77fe640 20 client.124291.objecter dump_active .. 0 homeless 2024-06-11T19:26:32.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.235+0000 7f1cce59c640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1cb403fdb0 0x7f1cb4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:32.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.235+0000 7f1cce59c640 10 monclient: get_auth_request con 0x7f1cb403fdb0 auth_method 0 2024-06-11T19:26:32.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.235+0000 7f1cc77fe640 10 client.124291.objecter ms_handle_connect 0x7f1cb403fdb0 2024-06-11T19:26:32.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.235+0000 7f1cce59c640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1cb403fdb0 0x7f1cb4042270 secure :-1 s=READY pgs=4318 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8035000 tx=0x7f1cb8030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:32.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.240+0000 7f1cc77fe640 1 -- 172.21.15.98:0/2416332886 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1cc0021020 con 0x7f1cd00a4920 2024-06-11T19:26:32.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.240+0000 7f1cc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:32.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.240+0000 7f1cc77fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.493+0000 7f1cd4cda640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:26:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.493+0000 7f1cd4cda640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:26:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.493+0000 7f1cd4cda640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.117:3300/0 2024-06-11T19:26:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.493+0000 7f1cd4cda640 1 -- 172.21.15.98:0/2416332886 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1cd01215f0 con 0x7f1cd00a4920 2024-06-11T19:26:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.493+0000 7f1cc77fe640 1 -- 172.21.15.98:0/2416332886 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3175 (secure 0 0 0) 0x7f1cc0011350 con 0x7f1cd00a4920 2024-06-11T19:26:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.493+0000 7f1cc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:26:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.493+0000 7f1cc77fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:32.494 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":86,"quorum":[0,1,2,3,6,8],"quorum_names":["a","f","b","g","d","e"],"quorum_leader_name":"b","quorum_age":95,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:26:32.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 10 librados: watch_flush enter 2024-06-11T19:26:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 10 librados: watch_flush exit 2024-06-11T19:26:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 20 client.124291.objecter shutdown clearing up homeless session... 2024-06-11T19:26:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 10 client.124291.objecter successfully canceled tick 2024-06-11T19:26:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 1 -- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1cb403fdb0 msgr2=0x7f1cb4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1cb403fdb0 0x7f1cb4042270 secure :-1 s=READY pgs=4318 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8035000 tx=0x7f1cb8030000 comp rx=0 tx=0).stop 2024-06-11T19:26:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 10 monclient: shutdown 2024-06-11T19:26:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 1 -- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cd00a4920 msgr2=0x7f1cd01ce160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cd00a4920 0x7f1cd01ce160 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1cc000e900 tx=0x7f1cc000edc0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 1 -- 172.21.15.98:0/2416332886 shutdown_connections 2024-06-11T19:26:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f1cb403fdb0 0x7f1cb4042270 unknown :-1 s=CLOSED pgs=4318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd00a5240 0x7f1cd009f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cd00a4920 0x7f1cd01ce160 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 1 --2- 172.21.15.98:0/2416332886 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f1cd00a2530 0x7f1cd01cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.495+0000 7f1cd4cda640 1 -- 172.21.15.98:0/2416332886 >> 172.21.15.98:0/2416332886 conn(0x7f1cd00ab850 msgr2=0x7f1cd0138130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:32.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.496+0000 7f1cd4cda640 1 -- 172.21.15.98:0/2416332886 shutdown_connections 2024-06-11T19:26:32.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.496+0000 7f1cd4cda640 1 -- 172.21.15.98:0/2416332886 wait complete. 2024-06-11T19:26:32.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:32.496+0000 7f1cd4cda640 1 librados: shutdown 2024-06-11T19:26:35.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:26:35.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.617+0000 7fa7bb8d1640 10 monclient: get_monmap_and_config 2024-06-11T19:26:35.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.617+0000 7fa7bb8d1640 10 monclient: build_initial_monmap 2024-06-11T19:26:35.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.617+0000 7fa7bb8d1640 10 monclient: monmap: 2024-06-11T19:26:35.616 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:35.617 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:35.617 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:35.618227+0000 2024-06-11T19:26:35.617 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:35.618227+0000 2024-06-11T19:26:35.617 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:35.617 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:35.617 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:35.617 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:35.617 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:35.617 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:35.618 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:35.618 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:35.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.617+0000 7fa7bb8d1640 1 Processor -- start 2024-06-11T19:26:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 1 -- start start 2024-06-11T19:26:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 10 monclient: init 2024-06-11T19:26:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 10 monclient: _add_conns ranks=[6,10,8,1,0,5,2,4,7,3,9] 2024-06-11T19:26:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a4920 0x7fa7b40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 10 monclient(hunting): picked mon.noname-d con 0x7fa7b40a4920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa7b40a5240 0x7fa7b40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:35.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 10 monclient(hunting): picked mon.noname-f con 0x7fa7b40a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:26:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7b40a2530 0x7fa7b40a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 10 monclient(hunting): picked mon.noname-e con 0x7fa7b40a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fa7b40a7140 con 0x7fa7b40a5240 2024-06-11T19:26:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa7b40a2b50 con 0x7fa7b40a4920 2024-06-11T19:26:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:35.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa7b40a2e30 con 0x7fa7b40a2530 2024-06-11T19:26:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7bb8d1640 10 monclient(hunting): authenticate will time out at 1805.425562s 2024-06-11T19:26:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b9e6e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7b40a2530 0x7fa7b40a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b966d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a4920 0x7fa7b40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b966d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a4920 0x7fa7b40a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46948/0 (socket says 172.21.15.98:46948) 2024-06-11T19:26:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b9e6e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7b40a2530 0x7fa7b40a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:38230/0 (socket says 172.21.15.98:38230) 2024-06-11T19:26:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b966d640 1 -- 172.21.15.98:0/3204953726 learned_addr learned my addr 172.21.15.98:0/3204953726 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b966d640 10 monclient(hunting): get_auth_request con 0x7fa7b40a4920 auth_method 0 2024-06-11T19:26:35.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b966d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b966d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b966d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b9e6e640 10 monclient(hunting): get_auth_request con 0x7fa7b40a2530 auth_method 0 2024-06-11T19:26:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b9e6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b9e6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b9e6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b966d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:35.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b966d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b966d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b9e6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b9e6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b9e6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b8e6c640 1 --2- 172.21.15.98:0/3204953726 >> v2:172.21.15.117:3300/0 conn(0x7fa7b40a5240 0x7fa7b40a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b8e6c640 10 monclient(hunting): get_auth_request con 0x7fa7b40a5240 auth_method 0 2024-06-11T19:26:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b8e6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:35.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b8e6c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:35.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b8e6c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:35.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b966d640 10 monclient(hunting): handle_auth_done global_id 124359 payload 274 2024-06-11T19:26:35.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.618+0000 7fa7b966d640 1 -- 172.21.15.98:0/3204953726 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7b40a2530 msgr2=0x7fa7b40a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:35.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7b966d640 1 --2- 172.21.15.98:0/3204953726 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7b40a2530 0x7fa7b40a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:35.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7b966d640 1 -- 172.21.15.98:0/3204953726 >> v2:172.21.15.117:3300/0 conn(0x7fa7b40a5240 msgr2=0x7fa7b40a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:35.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7b966d640 1 --2- 172.21.15.98:0/3204953726 >> v2:172.21.15.117:3300/0 conn(0x7fa7b40a5240 0x7fa7b40a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:35.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7b966d640 10 monclient: _finish_hunting 0 2024-06-11T19:26:35.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7b966d640 1 monclient: found mon.noname-d 2024-06-11T19:26:35.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7b966d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:35.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7b966d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:35.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7b966d640 1 -- 172.21.15.98:0/3204953726 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b40a3110 con 0x7fa7b40a4920 2024-06-11T19:26:35.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7b8e6c640 1 --2- 172.21.15.98:0/3204953726 >> v2:172.21.15.117:3300/0 conn(0x7fa7b40a5240 0x7fa7b40a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:26:35.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7b9e6e640 1 --2- 172.21.15.98:0/3204953726 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7b40a2530 0x7fa7b40a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:35.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7b966d640 1 --2- 172.21.15.98:0/3204953726 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a4920 0x7fa7b40a4d00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8009870 tx=0x7fa7a802b7c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=af862078df660f36 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:35.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 1 -- 172.21.15.98:0/3204953726 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7a8038070 con 0x7fa7b40a4920 2024-06-11T19:26:35.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 10 monclient: dump: 2024-06-11T19:26:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:35.634 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:35.635 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:35.636 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 10 monclient: _finish_auth 0 2024-06-11T19:26:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:26:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 1 -- 172.21.15.98:0/3204953726 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a8031990 con 0x7fa7b40a4920 2024-06-11T19:26:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7bb8d1640 5 monclient: authenticate success, global_id 124359 2024-06-11T19:26:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7bb8d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:35.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:35.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7bb8d1640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:35.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 1 -- 172.21.15.98:0/3204953726 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7a80323b0 con 0x7fa7b40a4920 2024-06-11T19:26:35.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:35.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7bb8d1640 4 set_mon_vals no callback set 2024-06-11T19:26:35.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:35.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.619+0000 7fa7a7fff640 10 monclient: dump: 2024-06-11T19:26:35.637 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:35.637 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:35.638 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:35.639 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.620+0000 7fa7bb8d1640 10 monclient: shutdown 2024-06-11T19:26:35.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.620+0000 7fa7bb8d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.620+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3204953726 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a4920 msgr2=0x7fa7b40a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.620+0000 7fa7bb8d1640 1 --2- 172.21.15.98:0/3204953726 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a4920 0x7fa7b40a4d00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8009870 tx=0x7fa7a802b7c0 comp rx=0 tx=0).stop 2024-06-11T19:26:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.620+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3204953726 shutdown_connections 2024-06-11T19:26:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.620+0000 7fa7bb8d1640 1 --2- 172.21.15.98:0/3204953726 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7b40a2530 0x7fa7b40a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.620+0000 7fa7bb8d1640 1 --2- 172.21.15.98:0/3204953726 >> v2:172.21.15.117:3300/0 conn(0x7fa7b40a5240 0x7fa7b40a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.620+0000 7fa7bb8d1640 1 --2- 172.21.15.98:0/3204953726 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a4920 0x7fa7b40a4d00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.620+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3204953726 >> 172.21.15.98:0/3204953726 conn(0x7fa7b40ab850 msgr2=0x7fa7b40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:35.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.620+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3204953726 shutdown_connections 2024-06-11T19:26:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.620+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3204953726 wait complete. 2024-06-11T19:26:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 10 monclient: build_initial_monmap 2024-06-11T19:26:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 10 monclient: monmap: 2024-06-11T19:26:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:35.641 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:35.643 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 1 librados: starting msgr at 2024-06-11T19:26:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 1 librados: starting objecter 2024-06-11T19:26:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 1 Processor -- start 2024-06-11T19:26:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 1 -- start start 2024-06-11T19:26:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 1 librados: setting wanted keys 2024-06-11T19:26:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 1 librados: calling monclient init 2024-06-11T19:26:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 10 monclient: init 2024-06-11T19:26:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 10 monclient: _add_conns ranks=[7,3,6,8,0,2,1,5,4] 2024-06-11T19:26:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7b40a2530 0x7fa7b41cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 10 monclient(hunting): picked mon.noname-i con 0x7fa7b40a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7b40a4920 0x7fa7b4137990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.621+0000 7fa7bb8d1640 10 monclient(hunting): picked mon.noname-g con 0x7fa7b40a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a5240 0x7fa7b4137ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 10 monclient(hunting): picked mon.noname-d con 0x7fa7b40a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b40a6da0 con 0x7fa7b40a4920 2024-06-11T19:26:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa7b4138e50 con 0x7fa7b40a5240 2024-06-11T19:26:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa7b40a6420 con 0x7fa7b40a2530 2024-06-11T19:26:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:35.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 10 monclient(hunting): authenticate will time out at 1805.429027s 2024-06-11T19:26:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a5240 0x7fa7b4137ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a5240 0x7fa7b4137ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46952/0 (socket says 172.21.15.98:46952) 2024-06-11T19:26:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 1 -- 172.21.15.98:0/3050119146 learned_addr learned my addr 172.21.15.98:0/3050119146 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 10 monclient(hunting): get_auth_request con 0x7fa7b40a5240 auth_method 0 2024-06-11T19:26:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b966d640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7b40a2530 0x7fa7b41cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b966d640 10 monclient(hunting): get_auth_request con 0x7fa7b40a2530 auth_method 0 2024-06-11T19:26:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b966d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b966d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b966d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:35.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b8e6c640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7b40a4920 0x7fa7b4137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b8e6c640 10 monclient(hunting): get_auth_request con 0x7fa7b40a4920 auth_method 0 2024-06-11T19:26:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b8e6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b8e6c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b8e6c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b966d640 1 -- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7b40a2530 msgr2=0x7fa7b41cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:26:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b966d640 1 -- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7b40a2530 msgr2=0x7fa7b41cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:26:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b966d640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7b40a2530 0x7fa7b41cdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:35.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b966d640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7b40a2530 0x7fa7b41cdaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:26:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b8e6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b8e6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b8e6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 10 monclient(hunting): handle_auth_done global_id 124368 payload 824 2024-06-11T19:26:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 1 -- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7b40a2530 msgr2=0x7fa7b41cdaf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:26:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7b40a2530 0x7fa7b41cdaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 1 -- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7b40a4920 msgr2=0x7fa7b4137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:35.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7b40a4920 0x7fa7b4137990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 10 monclient: _finish_hunting 0 2024-06-11T19:26:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 1 monclient: found mon.noname-d 2024-06-11T19:26:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 1 -- 172.21.15.98:0/3050119146 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b40a92f0 con 0x7fa7b40a5240 2024-06-11T19:26:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7b9e6e640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a5240 0x7fa7b4137ed0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00efc0 tx=0x7fa7ac009f90 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa7b40a5240 2024-06-11T19:26:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 1 -- 172.21.15.98:0/3050119146 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7ac018030 con 0x7fa7b40a5240 2024-06-11T19:26:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 10 monclient: dump: 2024-06-11T19:26:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:35.652 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:35.653 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 10 monclient: _finish_auth 0 2024-06-11T19:26:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:26:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 1 -- 172.21.15.98:0/3050119146 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7ac00cde0 con 0x7fa7b40a5240 2024-06-11T19:26:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 5 monclient: authenticate success, global_id 124368 2024-06-11T19:26:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 10 monclient: _renew_subs 2024-06-11T19:26:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3050119146 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b41ce1a0 con 0x7fa7b40a5240 2024-06-11T19:26:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 1 -- 172.21.15.98:0/3050119146 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7ac014d60 con 0x7fa7b40a5240 2024-06-11T19:26:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a5ffb640 10 monclient: dump: 2024-06-11T19:26:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:35.656 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:35.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.622+0000 7fa7a77fe640 4 set_mon_vals no callback set 2024-06-11T19:26:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.623+0000 7fa7bb8d1640 10 client.124368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.623+0000 7fa7bb8d1640 10 monclient: _renew_subs 2024-06-11T19:26:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.623+0000 7fa7bb8d1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.623+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3050119146 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b4127090 con 0x7fa7b40a5240 2024-06-11T19:26:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.623+0000 7fa7bb8d1640 1 librados: init done 2024-06-11T19:26:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.623+0000 7fa7a5ffb640 1 -- 172.21.15.98:0/3050119146 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fa7ac005d10 con 0x7fa7b40a5240 2024-06-11T19:26:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.623+0000 7fa7a5ffb640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa79803fdb0 0x7fa798042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.623+0000 7fa7a5ffb640 1 -- 172.21.15.98:0/3050119146 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7fa7ac019070 con 0x7fa7b40a5240 2024-06-11T19:26:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.623+0000 7fa7a5ffb640 10 client.124368.objecter ms_dispatch 0x7fa7b4000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.623+0000 7fa7a5ffb640 3 client.124368.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.623+0000 7fa7a5ffb640 3 client.124368.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.623+0000 7fa7bb8d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.624+0000 7fa7bb8d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.624+0000 7fa7bb8d1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.624+0000 7fa7a5ffb640 20 client.124368.objecter dump_active .. 0 homeless 2024-06-11T19:26:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.624+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3050119146 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b40a7360 con 0x7fa7b40a5240 2024-06-11T19:26:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.624+0000 7fa7b966d640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa79803fdb0 0x7fa798042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.624+0000 7fa7b966d640 10 monclient: get_auth_request con 0x7fa79803fdb0 auth_method 0 2024-06-11T19:26:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.624+0000 7fa7a5ffb640 10 client.124368.objecter ms_handle_connect 0x7fa79803fdb0 2024-06-11T19:26:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.624+0000 7fa7b966d640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa79803fdb0 0x7fa798042270 secure :-1 s=READY pgs=4319 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8035000 tx=0x7fa7a8030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:35.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.628+0000 7fa7a5ffb640 1 -- 172.21.15.98:0/3050119146 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa7b40a7360 con 0x7fa7b40a5240 2024-06-11T19:26:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.628+0000 7fa7a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:35.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.628+0000 7fa7a5ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:35.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.880+0000 7fa7bb8d1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:26:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.880+0000 7fa7bb8d1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:26:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.880+0000 7fa7bb8d1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.880+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3050119146 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa7b40754f0 con 0x7fa7b40a5240 2024-06-11T19:26:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.881+0000 7fa7a5ffb640 1 -- 172.21.15.98:0/3050119146 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3175 (secure 0 0 0) 0x7fa7b40754f0 con 0x7fa7b40a5240 2024-06-11T19:26:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.881+0000 7fa7a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:26:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.881+0000 7fa7a5ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:35.881 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":86,"quorum":[0,1,2,3,6,8],"quorum_names":["a","f","b","g","d","e"],"quorum_leader_name":"b","quorum_age":98,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:26:35.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 10 librados: watch_flush enter 2024-06-11T19:26:35.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 10 librados: watch_flush exit 2024-06-11T19:26:35.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 20 client.124368.objecter shutdown clearing up homeless session... 2024-06-11T19:26:35.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 10 client.124368.objecter successfully canceled tick 2024-06-11T19:26:35.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa79803fdb0 msgr2=0x7fa798042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:35.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa79803fdb0 0x7fa798042270 secure :-1 s=READY pgs=4319 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8035000 tx=0x7fa7a8030000 comp rx=0 tx=0).stop 2024-06-11T19:26:35.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 10 monclient: shutdown 2024-06-11T19:26:35.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:35.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3050119146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a5240 msgr2=0x7fa7b4137ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:35.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a5240 0x7fa7b4137ed0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00efc0 tx=0x7fa7ac009f90 comp rx=0 tx=0).stop 2024-06-11T19:26:35.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3050119146 shutdown_connections 2024-06-11T19:26:35.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa79803fdb0 0x7fa798042270 unknown :-1 s=CLOSED pgs=4319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:35.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7b40a4920 0x7fa7b4137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:35.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7b40a5240 0x7fa7b4137ed0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:35.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 --2- 172.21.15.98:0/3050119146 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7b40a2530 0x7fa7b41cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:35.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3050119146 >> 172.21.15.98:0/3050119146 conn(0x7fa7b40ab850 msgr2=0x7fa7b40726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:35.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3050119146 shutdown_connections 2024-06-11T19:26:35.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 -- 172.21.15.98:0/3050119146 wait complete. 2024-06-11T19:26:35.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:35.883+0000 7fa7bb8d1640 1 librados: shutdown 2024-06-11T19:26:38.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:26:39.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.003+0000 7ff0d15b5640 10 monclient: get_monmap_and_config 2024-06-11T19:26:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.003+0000 7ff0d15b5640 10 monclient: build_initial_monmap 2024-06-11T19:26:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.003+0000 7ff0d15b5640 10 monclient: monmap: 2024-06-11T19:26:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:39.004609+0000 2024-06-11T19:26:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:39.004609+0000 2024-06-11T19:26:39.003 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:39.004 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:39.005 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:39.005 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:39.005 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:39.005 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:39.005 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:39.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 1 Processor -- start 2024-06-11T19:26:39.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 1 -- start start 2024-06-11T19:26:39.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 10 monclient: init 2024-06-11T19:26:39.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:39.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 10 monclient: _add_conns ranks=[4,5,1,7,6,3,2,9,0,10,8] 2024-06-11T19:26:39.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 0x7ff0cc0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:39.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 10 monclient(hunting): picked mon.noname-c con 0x7ff0cc0a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:39.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 0x7ff0cc0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:39.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 10 monclient(hunting): picked mon.noname-h con 0x7ff0cc0a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:39.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff0cc0a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:26:39.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0cc0a7140 con 0x7ff0cc0a2530 2024-06-11T19:26:39.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:39.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff0cc0a2b50 con 0x7ff0cc0a4920 2024-06-11T19:26:39.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:39.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ff0cc0a2e30 con 0x7ff0cc0a5240 2024-06-11T19:26:39.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:39.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0d15b5640 10 monclient(hunting): authenticate will time out at 1808.811756s 2024-06-11T19:26:39.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0caffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 0x7ff0cc0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:39.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0caffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 0x7ff0cc0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38564/0 (socket says 172.21.15.98:38564) 2024-06-11T19:26:39.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0caffd640 1 -- 172.21.15.98:0/3919486920 learned_addr learned my addr 172.21.15.98:0/3919486920 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:39.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0caffd640 10 monclient(hunting): get_auth_request con 0x7ff0cc0a4920 auth_method 0 2024-06-11T19:26:39.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:39.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0caffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:39.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0caffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:39.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0caffd640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 msgr2=0x7ff0cc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:26:39.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0caffd640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 msgr2=0x7ff0cc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:26:39.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.004+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 0x7ff0cc0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:39.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.005+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 0x7ff0cc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:26:39.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.005+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 0x7ff0cc0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:39.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.005+0000 7ff0c27fc640 10 monclient(hunting): get_auth_request con 0x7ff0cc0a5240 auth_method 0 2024-06-11T19:26:39.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.005+0000 7ff0c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:39.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.005+0000 7ff0c27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:39.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.005+0000 7ff0c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:39.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.005+0000 7ff0c27fc640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 msgr2=0x7ff0cc0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:26:39.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.005+0000 7ff0c27fc640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 msgr2=0x7ff0cc0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:26:39.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.005+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 0x7ff0cc0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:39.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.005+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 0x7ff0cc0a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:26:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 0x7ff0cc0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0caffd640 10 monclient(hunting): get_auth_request con 0x7ff0cc0a4920 auth_method 0 2024-06-11T19:26:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0caffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:39.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0caffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:26:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0caffd640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 msgr2=0x7ff0cc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:26:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0caffd640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 msgr2=0x7ff0cc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:26:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 0x7ff0cc0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 0x7ff0cc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:26:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 0x7ff0cc0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0c27fc640 10 monclient(hunting): get_auth_request con 0x7ff0cc0a5240 auth_method 0 2024-06-11T19:26:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0c27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:39.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0c27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:26:39.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0c27fc640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 msgr2=0x7ff0cc0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:26:39.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0c27fc640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 msgr2=0x7ff0cc0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:26:39.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 0x7ff0cc0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:39.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.206+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 0x7ff0cc0a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:26:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 0x7ff0cc0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0caffd640 10 monclient(hunting): get_auth_request con 0x7ff0cc0a4920 auth_method 0 2024-06-11T19:26:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0caffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0caffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:26:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0caffd640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 msgr2=0x7ff0cc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-06-11T19:26:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0caffd640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 msgr2=0x7ff0cc0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:26:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 0x7ff0cc0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 0x7ff0cc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:26:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 0x7ff0cc0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0c27fc640 10 monclient(hunting): get_auth_request con 0x7ff0cc0a5240 auth_method 0 2024-06-11T19:26:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0c27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0c27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:26:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0c27fc640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 msgr2=0x7ff0cc0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-11T19:26:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0c27fc640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 msgr2=0x7ff0cc0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:26:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 0x7ff0cc0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:39.607+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 0x7ff0cc0a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:26:40.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.000+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 252374885 0 0) 0x7ff0cc0a7140 con 0x7ff0cc0a2530 2024-06-11T19:26:40.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.000+0000 7ff0ca7fc640 10 monclient(hunting): _negotiate 2024-06-11T19:26:40.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.000+0000 7ff0ca7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.000+0000 7ff0ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.000+0000 7ff0ca7fc640 10 monclient(hunting): my global_id is 124309 2024-06-11T19:26:40.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.000+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0b8003520 con 0x7ff0cc0a2530 2024-06-11T19:26:40.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2829157504 0 0) 0x7ff0b8003520 con 0x7ff0cc0a2530 2024-06-11T19:26:40.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 msgr2=0x7ff0cc0a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:26:40.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff0cc0a5240 0x7ff0cc0a59a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 msgr2=0x7ff0cc0a4d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:26:40.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff0cc0a4920 0x7ff0cc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:26:40.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 monclient: found mon.noname-f-legacy 2024-06-11T19:26:40.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:40.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.117:6789/0 2024-06-11T19:26:40.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0cc0a3110 con 0x7ff0cc0a2530 2024-06-11T19:26:40.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient: _finish_auth 0 2024-06-11T19:26:40.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:26:40.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:40.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2697086589 0 0) 0x7ff0c4001ce0 con 0x7ff0cc0a2530 2024-06-11T19:26:40.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2024-06-11T19:26:40.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient: dump: 2024-06-11T19:26:40.007 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.007 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.007 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.007 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.007 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.007 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.007 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.007 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.007 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.008 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.008 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.008 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.008 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.008 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.008 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.008 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 monclient: mon.f has (v2) addrs [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] but i'm connected to v1:172.21.15.117:6789/0, reconnecting 2024-06-11T19:26:40.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:40.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 >> v1:172.21.15.117:6789/0 conn(0x7ff0cc0a2530 legacy=0x7ff0cc0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient: _add_conns ranks=[2,6,7,1,8,4,5,0,3] 2024-06-11T19:26:40.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0b8004200 0x7ff0b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient(hunting): picked mon.b con 0x7ff0b8004200 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:40.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff0b8004e10 0x7ff0b8005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient(hunting): picked mon.d con 0x7ff0b8004e10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:40.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff0b8009bf0 0x7ff0b800c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient(hunting): picked mon.i con 0x7ff0b8009bf0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:40.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff0cc0a2b50 con 0x7ff0b8004200 2024-06-11T19:26:40.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff0cc0a2e30 con 0x7ff0b8004e10 2024-06-11T19:26:40.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff0b800c990 con 0x7ff0b8009bf0 2024-06-11T19:26:40.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:40.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0ca7fc640 10 monclient(hunting): _finish_auth 0 2024-06-11T19:26:40.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff0b8004e10 0x7ff0b8005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0c27fc640 10 monclient(hunting): get_auth_request con 0x7ff0b8004e10 auth_method 0 2024-06-11T19:26:40.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0c27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0c27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:26:40.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0cb7fe640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff0b8009bf0 0x7ff0b800c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:26:40.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0cb7fe640 10 monclient(hunting): get_auth_request con 0x7ff0b8009bf0 auth_method 0 2024-06-11T19:26:40.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0cb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0cb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:26:40.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.001+0000 7ff0cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-06-11T19:26:40.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 10 monclient(hunting): handle_auth_done global_id 124309 payload 293 2024-06-11T19:26:40.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff0b8009bf0 msgr2=0x7ff0b800c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:40.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff0b8009bf0 0x7ff0b800c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0b8004200 0x7ff0b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0b8004200 msgr2=0x7ff0b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:40.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0b8004200 0x7ff0b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 10 monclient: _finish_hunting 0 2024-06-11T19:26:40.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 1 monclient: found mon.d 2024-06-11T19:26:40.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:40.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:40.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 1 -- 172.21.15.98:0/3919486920 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b8003520 con 0x7ff0b8004e10 2024-06-11T19:26:40.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 10 monclient: _finish_auth 0 2024-06-11T19:26:40.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 10 monclient: _check_auth_tickets 2024-06-11T19:26:40.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:40.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0cb7fe640 1 -- 172.21.15.98:0/3919486920 reap_dead start 2024-06-11T19:26:40.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0caffd640 10 monclient: get_auth_request con 0x7ff0b8004200 auth_method 0 2024-06-11T19:26:40.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0b8004200 0x7ff0b8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:26:40.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0d15b5640 5 monclient: authenticate success, global_id 124309 2024-06-11T19:26:40.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.002+0000 7ff0d15b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:40.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff0b8004e10 0x7ff0b8005290 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc073fc0 tx=0x7ff0b8005830 comp rx=0 tx=0).ready entity=mon.6 client_cookie=ecc9a2c020951085 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0b4032070 con 0x7ff0b8004e10 2024-06-11T19:26:40.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0ca7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:40.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0ca7fc640 10 monclient: dump: 2024-06-11T19:26:40.015 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.015 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.015 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.017 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0d15b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b40353b0 con 0x7ff0b8004e10 2024-06-11T19:26:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0ca7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0d15b5640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0ca7fc640 1 -- 172.21.15.98:0/3919486920 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0b400f710 con 0x7ff0b8004e10 2024-06-11T19:26:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0d15b5640 4 set_mon_vals no callback set 2024-06-11T19:26:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0ca7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.003+0000 7ff0ca7fc640 10 monclient: dump: 2024-06-11T19:26:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.020 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.004+0000 7ff0d15b5640 10 monclient: shutdown 2024-06-11T19:26:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.004+0000 7ff0d15b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.004+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff0b8004e10 msgr2=0x7ff0b8005290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.004+0000 7ff0d15b5640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff0b8004e10 0x7ff0b8005290 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc073fc0 tx=0x7ff0b8005830 comp rx=0 tx=0).stop 2024-06-11T19:26:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.004+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3919486920 shutdown_connections 2024-06-11T19:26:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.004+0000 7ff0d15b5640 1 --2- 172.21.15.98:0/3919486920 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff0b8004e10 0x7ff0b8005290 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.004+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3919486920 >> 172.21.15.98:0/3919486920 conn(0x7ff0cc0ab850 msgr2=0x7ff0cc0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.004+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3919486920 shutdown_connections 2024-06-11T19:26:40.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.004+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3919486920 wait complete. 2024-06-11T19:26:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.005+0000 7ff0d15b5640 10 monclient: build_initial_monmap 2024-06-11T19:26:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.005+0000 7ff0d15b5640 10 monclient: monmap: 2024-06-11T19:26:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:40.026 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:40.027 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:40.028 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.006+0000 7ff0d15b5640 1 librados: starting msgr at 2024-06-11T19:26:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.006+0000 7ff0d15b5640 1 librados: starting objecter 2024-06-11T19:26:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.006+0000 7ff0d15b5640 1 Processor -- start 2024-06-11T19:26:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.006+0000 7ff0d15b5640 1 -- start start 2024-06-11T19:26:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.006+0000 7ff0d15b5640 1 librados: setting wanted keys 2024-06-11T19:26:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.006+0000 7ff0d15b5640 1 librados: calling monclient init 2024-06-11T19:26:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.006+0000 7ff0d15b5640 10 monclient: init 2024-06-11T19:26:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 10 monclient: _add_conns ranks=[2,3,6,4,5,1,7,8,0] 2024-06-11T19:26:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0b8004e10 0x7ff0cc1cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 10 monclient(hunting): picked mon.noname-b con 0x7ff0b8004e10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0cc1ce010 0x7ff0cc124130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 10 monclient(hunting): picked mon.noname-g con 0x7ff0cc1ce010 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff0cc1ce3f0 0x7ff0cc124670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 10 monclient(hunting): picked mon.noname-d con 0x7ff0cc1ce3f0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff0cc0a6da0 con 0x7ff0b8004e10 2024-06-11T19:26:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7ff0cc138e50 con 0x7ff0cc1ce010 2024-06-11T19:26:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff0cc0a4d00 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0d15b5640 10 monclient(hunting): authenticate will time out at 1809.814186s 2024-06-11T19:26:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0caffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0b8004e10 0x7ff0cc1cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff0cc1ce3f0 0x7ff0cc124670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0caffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0b8004e10 0x7ff0cc1cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40258/0 (socket says 172.21.15.98:40258) 2024-06-11T19:26:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0caffd640 1 -- 172.21.15.98:0/3632980960 learned_addr learned my addr 172.21.15.98:0/3632980960 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 10 monclient(hunting): get_auth_request con 0x7ff0cc1ce3f0 auth_method 0 2024-06-11T19:26:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0caffd640 10 monclient(hunting): get_auth_request con 0x7ff0b8004e10 auth_method 0 2024-06-11T19:26:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0caffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0caffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0c27fc640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0cc1ce010 0x7ff0cc124130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0c27fc640 10 monclient(hunting): get_auth_request con 0x7ff0cc1ce010 auth_method 0 2024-06-11T19:26:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0c27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 10 monclient(hunting): handle_auth_done global_id 124377 payload 824 2024-06-11T19:26:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 1 -- 172.21.15.98:0/3632980960 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0cc1ce010 msgr2=0x7ff0cc124130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0cc1ce010 0x7ff0cc124130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 1 -- 172.21.15.98:0/3632980960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0b8004e10 msgr2=0x7ff0cc1cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0b8004e10 0x7ff0cc1cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:26:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 1 monclient: found mon.noname-d 2024-06-11T19:26:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0cb7fe640 1 -- 172.21.15.98:0/3632980960 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0cc127a90 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.007+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0b8004e10 0x7ff0cc1cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0cb7fe640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff0cc1ce3f0 0x7ff0cc124670 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0a8420 tx=0x7ff0c400d520 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 10 client.?.objecter ms_handle_connect 0x7ff0cc1ce3f0 2024-06-11T19:26:40.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:40.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 1 -- 172.21.15.98:0/3632980960 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0c4003970 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:40.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 10 monclient: dump: 2024-06-11T19:26:40.036 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.036 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.038 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 10 monclient: _finish_auth 0 2024-06-11T19:26:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 10 monclient: _check_auth_tickets 2024-06-11T19:26:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 1 -- 172.21.15.98:0/3632980960 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c4018070 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0d15b5640 5 monclient: authenticate success, global_id 124377 2024-06-11T19:26:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0d15b5640 10 monclient: _renew_subs 2024-06-11T19:26:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0d15b5640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3632980960 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0cc126200 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 1 -- 172.21.15.98:0/3632980960 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0c40050d0 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0d15b5640 10 client.124377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c3fff640 10 monclient: dump: 2024-06-11T19:26:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.042 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0c9ffb640 4 set_mon_vals no callback set 2024-06-11T19:26:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0d15b5640 10 monclient: _renew_subs 2024-06-11T19:26:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0d15b5640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3632980960 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0cc127d70 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.008+0000 7ff0d15b5640 1 librados: init done 2024-06-11T19:26:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.009+0000 7ff0c3fff640 1 -- 172.21.15.98:0/3632980960 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff0c4003b30 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0c3fff640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0ac03fdb0 0x7ff0ac042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0c3fff640 1 -- 172.21.15.98:0/3632980960 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7ff0c4010040 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0c3fff640 10 client.124377.objecter ms_dispatch 0x7ff0cc000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:40.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0c3fff640 3 client.124377.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:40.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0c3fff640 3 client.124377.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:40.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0c3fff640 20 client.124377.objecter dump_active .. 0 homeless 2024-06-11T19:26:40.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0ac03fdb0 0x7ff0ac042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0caffd640 10 monclient: get_auth_request con 0x7ff0ac03fdb0 auth_method 0 2024-06-11T19:26:40.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0d15b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:40.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0d15b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:40.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0d15b5640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:40.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3632980960 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0cc072f80 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0c3fff640 10 client.124377.objecter ms_handle_connect 0x7ff0ac03fdb0 2024-06-11T19:26:40.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.010+0000 7ff0caffd640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0ac03fdb0 0x7ff0ac042270 secure :-1 s=READY pgs=4320 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc006a90 tx=0x7ff0bc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.019+0000 7ff0c3fff640 1 -- 172.21.15.98:0/3632980960 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff0cc072f80 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.019+0000 7ff0c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:40.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.019+0000 7ff0c3fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:40.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.301+0000 7ff0d15b5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:26:40.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.301+0000 7ff0d15b5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:26:40.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.301+0000 7ff0d15b5640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:40.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.301+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3632980960 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff0cc0aabd0 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.302+0000 7ff0c3fff640 1 -- 172.21.15.98:0/3632980960 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7ff0c40055f0 con 0x7ff0cc1ce3f0 2024-06-11T19:26:40.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.302+0000 7ff0c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:26:40.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.302+0000 7ff0c3fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:40.302 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":92,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:26:40.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 10 librados: watch_flush enter 2024-06-11T19:26:40.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 10 librados: watch_flush exit 2024-06-11T19:26:40.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 20 client.124377.objecter shutdown clearing up homeless session... 2024-06-11T19:26:40.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 10 client.124377.objecter successfully canceled tick 2024-06-11T19:26:40.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3632980960 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0ac03fdb0 msgr2=0x7ff0ac042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0ac03fdb0 0x7ff0ac042270 secure :-1 s=READY pgs=4320 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc006a90 tx=0x7ff0bc008040 comp rx=0 tx=0).stop 2024-06-11T19:26:40.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 10 monclient: shutdown 2024-06-11T19:26:40.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:40.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3632980960 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff0cc1ce3f0 msgr2=0x7ff0cc124670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff0cc1ce3f0 0x7ff0cc124670 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0a8420 tx=0x7ff0c400d520 comp rx=0 tx=0).stop 2024-06-11T19:26:40.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3632980960 shutdown_connections 2024-06-11T19:26:40.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff0ac03fdb0 0x7ff0ac042270 unknown :-1 s=CLOSED pgs=4320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff0cc1ce3f0 0x7ff0cc124670 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0cc1ce010 0x7ff0cc124130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 --2- 172.21.15.98:0/3632980960 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff0b8004e10 0x7ff0cc1cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3632980960 >> 172.21.15.98:0/3632980960 conn(0x7ff0cc0ab850 msgr2=0x7ff0cc0726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:40.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3632980960 shutdown_connections 2024-06-11T19:26:40.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 -- 172.21.15.98:0/3632980960 wait complete. 2024-06-11T19:26:40.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.304+0000 7ff0d15b5640 1 librados: shutdown 2024-06-11T19:26:40.314 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:26:40.314 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-06-11T19:26:40.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-11T19:26:40.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.414+0000 7f087285b640 10 monclient: get_monmap_and_config 2024-06-11T19:26:40.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.414+0000 7f087285b640 10 monclient: build_initial_monmap 2024-06-11T19:26:40.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.414+0000 7f087285b640 10 monclient: monmap: 2024-06-11T19:26:40.414 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:40.414 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.414 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:40.415660+0000 2024-06-11T19:26:40.414 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:40.415660+0000 2024-06-11T19:26:40.415 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:40.415 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.415 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:40.415 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:40.415 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:40.415 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:40.415 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:40.415 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:40.416 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:40.416 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:40.416 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:40.416 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:40.416 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:40.416 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 1 Processor -- start 2024-06-11T19:26:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 1 -- start start 2024-06-11T19:26:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 10 monclient: init 2024-06-11T19:26:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 10 monclient: _add_conns ranks=[2,10,4,7,9,8,1,0,3,5,6] 2024-06-11T19:26:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f086c0a4920 0x7f086c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 10 monclient(hunting): picked mon.noname-b con 0x7f086c0a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f086c0a5240 0x7f086c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 10 monclient(hunting): picked mon.noname-f con 0x7f086c0a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:26:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a2530 0x7f086c0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 10 monclient(hunting): picked mon.noname-c con 0x7f086c0a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f086c0a7140 con 0x7f086c0a5240 2024-06-11T19:26:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f086c0a2b50 con 0x7f086c0a4920 2024-06-11T19:26:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f086c0a2e30 con 0x7f086c0a2530 2024-06-11T19:26:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.415+0000 7f087285b640 10 monclient(hunting): authenticate will time out at 1810.222959s 2024-06-11T19:26:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f086c0a4920 0x7f086c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f0870df8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a2530 0x7f086c0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f086c0a4920 0x7f086c0a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40284/0 (socket says 172.21.15.98:40284) 2024-06-11T19:26:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 1 -- 172.21.15.98:0/2004053252 learned_addr learned my addr 172.21.15.98:0/2004053252 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 10 monclient(hunting): get_auth_request con 0x7f086c0a4920 auth_method 0 2024-06-11T19:26:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f0870df8640 10 monclient(hunting): get_auth_request con 0x7f086c0a2530 auth_method 0 2024-06-11T19:26:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f0870df8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f0870df8640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f0870df8640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086b7fe640 1 --2- 172.21.15.98:0/2004053252 >> v2:172.21.15.117:3300/0 conn(0x7f086c0a5240 0x7f086c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086b7fe640 10 monclient(hunting): get_auth_request con 0x7f086c0a5240 auth_method 0 2024-06-11T19:26:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f0870df8640 1 -- 172.21.15.98:0/2004053252 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a2530 msgr2=0x7f086c0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:26:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f0870df8640 1 -- 172.21.15.98:0/2004053252 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a2530 msgr2=0x7f086c0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:26:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f0870df8640 1 --2- 172.21.15.98:0/2004053252 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a2530 0x7f086c0a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f0870df8640 1 --2- 172.21.15.98:0/2004053252 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a2530 0x7f086c0a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:26:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 10 monclient(hunting): handle_auth_done global_id 85331 payload 274 2024-06-11T19:26:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 1 -- 172.21.15.98:0/2004053252 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a2530 msgr2=0x7f086c0a5ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-11T19:26:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 1 --2- 172.21.15.98:0/2004053252 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a2530 0x7f086c0a5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 1 -- 172.21.15.98:0/2004053252 >> v2:172.21.15.117:3300/0 conn(0x7f086c0a5240 msgr2=0x7f086c0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 1 --2- 172.21.15.98:0/2004053252 >> v2:172.21.15.117:3300/0 conn(0x7f086c0a5240 0x7f086c0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:26:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 1 monclient: found mon.noname-b 2024-06-11T19:26:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:26:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.416+0000 7f086bfff640 1 -- 172.21.15.98:0/2004053252 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086c0a3110 con 0x7f086c0a4920 2024-06-11T19:26:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086bfff640 1 --2- 172.21.15.98:0/2004053252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f086c0a4920 0x7f086c0a4d00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f085c009870 tx=0x7f085c02b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d61a17a43da76f28 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 1 -- 172.21.15.98:0/2004053252 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f085c038070 con 0x7f086c0a4920 2024-06-11T19:26:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:26:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 10 monclient: dump: 2024-06-11T19:26:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.437 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 10 monclient: _finish_auth 0 2024-06-11T19:26:40.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 10 monclient: _check_auth_tickets 2024-06-11T19:26:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 1 -- 172.21.15.98:0/2004053252 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085c031990 con 0x7f086c0a4920 2024-06-11T19:26:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 5 monclient: authenticate success, global_id 85331 2024-06-11T19:26:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 1 -- 172.21.15.98:0/2004053252 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f085c0323b0 con 0x7f086c0a4920 2024-06-11T19:26:40.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 4 set_mon_vals no callback set 2024-06-11T19:26:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:26:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f086affd640 10 monclient: dump: 2024-06-11T19:26:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.439 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.441 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 10 monclient: shutdown 2024-06-11T19:26:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 1 -- 172.21.15.98:0/2004053252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f086c0a4920 msgr2=0x7f086c0a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 1 --2- 172.21.15.98:0/2004053252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f086c0a4920 0x7f086c0a4d00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f085c009870 tx=0x7f085c02b7c0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 1 -- 172.21.15.98:0/2004053252 shutdown_connections 2024-06-11T19:26:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 1 --2- 172.21.15.98:0/2004053252 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a2530 0x7f086c0a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 1 --2- 172.21.15.98:0/2004053252 >> v2:172.21.15.117:3300/0 conn(0x7f086c0a5240 0x7f086c0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 1 --2- 172.21.15.98:0/2004053252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f086c0a4920 0x7f086c0a4d00 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 1 -- 172.21.15.98:0/2004053252 >> 172.21.15.98:0/2004053252 conn(0x7f086c0ab850 msgr2=0x7f086c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 1 -- 172.21.15.98:0/2004053252 shutdown_connections 2024-06-11T19:26:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.417+0000 7f087285b640 1 -- 172.21.15.98:0/2004053252 wait complete. 2024-06-11T19:26:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.418+0000 7f087285b640 10 monclient: build_initial_monmap 2024-06-11T19:26:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.418+0000 7f087285b640 10 monclient: monmap: 2024-06-11T19:26:40.442 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:40.444 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.418+0000 7f087285b640 1 librados: starting msgr at 2024-06-11T19:26:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.418+0000 7f087285b640 1 librados: starting objecter 2024-06-11T19:26:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.418+0000 7f087285b640 1 Processor -- start 2024-06-11T19:26:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.418+0000 7f087285b640 1 -- start start 2024-06-11T19:26:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.418+0000 7f087285b640 1 librados: setting wanted keys 2024-06-11T19:26:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.418+0000 7f087285b640 1 librados: calling monclient init 2024-06-11T19:26:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.418+0000 7f087285b640 10 monclient: init 2024-06-11T19:26:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 10 monclient: _add_conns ranks=[6,4,2,1,8,0,7,5,3] 2024-06-11T19:26:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f086c0a2530 0x7f086c137520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 10 monclient(hunting): picked mon.noname-d con 0x7f086c0a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a4920 0x7f086c137a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 10 monclient(hunting): picked mon.noname-c con 0x7f086c0a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f086c0a5240 0x7f086c122fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 10 monclient(hunting): picked mon.noname-b con 0x7f086c0a5240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f086c0a6da0 con 0x7f086c0a5240 2024-06-11T19:26:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f086c138e50 con 0x7f086c0a4920 2024-06-11T19:26:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f086c0a6420 con 0x7f086c0a2530 2024-06-11T19:26:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f087285b640 10 monclient(hunting): authenticate will time out at 1810.226389s 2024-06-11T19:26:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f086c0a2530 0x7f086c137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086b7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a4920 0x7f086c137a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f0870df8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f086c0a5240 0x7f086c122fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f086c0a2530 0x7f086c137520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46970/0 (socket says 172.21.15.98:46970) 2024-06-11T19:26:40.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 1 -- 172.21.15.98:0/3686185637 learned_addr learned my addr 172.21.15.98:0/3686185637 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 10 monclient(hunting): get_auth_request con 0x7f086c0a2530 auth_method 0 2024-06-11T19:26:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086b7fe640 10 monclient(hunting): get_auth_request con 0x7f086c0a4920 auth_method 0 2024-06-11T19:26:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f0870df8640 10 monclient(hunting): get_auth_request con 0x7f086c0a5240 auth_method 0 2024-06-11T19:26:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f0870df8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f0870df8640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f0870df8640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086b7fe640 1 -- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a4920 msgr2=0x7f086c137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-06-11T19:26:40.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f0870df8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f0870df8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086b7fe640 1 -- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a4920 msgr2=0x7f086c137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:26:40.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086b7fe640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a4920 0x7f086c137a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:40.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f0870df8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086b7fe640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a4920 0x7f086c137a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:26:40.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 10 monclient(hunting): handle_auth_done global_id 124386 payload 824 2024-06-11T19:26:40.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 1 -- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a4920 msgr2=0x7f086c137a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:26:40.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f086c0a4920 0x7f086c137a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 1 -- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f086c0a5240 msgr2=0x7f086c122fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f086c0a5240 0x7f086c122fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 10 monclient: _finish_hunting 0 2024-06-11T19:26:40.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 1 monclient: found mon.noname-d 2024-06-11T19:26:40.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:40.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:40.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 1 -- 172.21.15.98:0/3686185637 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086c0a8ea0 con 0x7f086c0a2530 2024-06-11T19:26:40.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f0870df8640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f086c0a5240 0x7f086c122fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:40.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f086bfff640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f086c0a2530 0x7f086c137520 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f085c009870 tx=0x7f085c0046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f0868ff9640 10 client.?.objecter ms_handle_connect 0x7f086c0a2530 2024-06-11T19:26:40.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.419+0000 7f0868ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:40.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f085c042020 con 0x7f086c0a2530 2024-06-11T19:26:40.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:40.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 10 monclient: dump: 2024-06-11T19:26:40.452 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.453 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.453 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.453 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.453 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.453 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.453 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.453 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.453 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.454 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.454 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.454 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.454 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.454 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.454 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.454 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 10 monclient: _finish_auth 0 2024-06-11T19:26:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:26:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085c003610 con 0x7f086c0a2530 2024-06-11T19:26:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f087285b640 5 monclient: authenticate success, global_id 124386 2024-06-11T19:26:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f087285b640 10 monclient: _renew_subs 2024-06-11T19:26:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f087285b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f087285b640 1 -- 172.21.15.98:0/3686185637 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f086c0a91d0 con 0x7f086c0a2530 2024-06-11T19:26:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:40.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f085c038040 con 0x7f086c0a2530 2024-06-11T19:26:40.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:40.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f0868ff9640 10 monclient: dump: 2024-06-11T19:26:40.456 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.456 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.456 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.456 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.457 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.457 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.457 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.457 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.457 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.457 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.457 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.457 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.458 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.458 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.458 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.458 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f087285b640 10 client.124386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:40.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f087285b640 10 monclient: _renew_subs 2024-06-11T19:26:40.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f087285b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:40.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f087285b640 1 -- 172.21.15.98:0/3686185637 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f086c126c20 con 0x7f086c0a2530 2024-06-11T19:26:40.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f087285b640 1 librados: init done 2024-06-11T19:26:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f086a7fc640 4 set_mon_vals no callback set 2024-06-11T19:26:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.420+0000 7f087285b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f087285b640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123520 0x7f086c123900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f087285b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f087285b640 1 -- 172.21.15.98:0/3686185637 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f086c050830 con 0x7f086c123520 2024-06-11T19:26:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f085c045020 con 0x7f086c0a2530 2024-06-11T19:26:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f086b7fe640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123520 0x7f086c123900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f086b7fe640 10 monclient: get_auth_request con 0x7f086c123520 auth_method 0 2024-06-11T19:26:40.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f086b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f086b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f086b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f086b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f086b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f086b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f085803fdb0 0x7f0858042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f085c030070 con 0x7f086c0a2530 2024-06-11T19:26:40.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 10 client.124386.objecter ms_dispatch 0x7f086c000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:40.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 3 client.124386.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:40.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 3 client.124386.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:40.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f086b7fe640 10 monclient(hunting): handle_auth_done global_id 66312 payload 824 2024-06-11T19:26:40.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 20 client.124386.objecter dump_active .. 0 homeless 2024-06-11T19:26:40.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f086b7fe640 1 -- 172.21.15.98:0/3686185637 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f086c0a6750 con 0x7f086c123520 2024-06-11T19:26:40.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0870df8640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f085803fdb0 0x7f0858042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 10 client.124386.objecter ms_handle_connect 0x7f086c123520 2024-06-11T19:26:40.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 10 client.124386.objecter resend_mon_ops 2024-06-11T19:26:40.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f086b7fe640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123520 0x7f086c123900 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f086c0a5620 tx=0x7f0860011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0870df8640 10 monclient: get_auth_request con 0x7f085803fdb0 auth_method 0 2024-06-11T19:26:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0860015ae0 con 0x7f086c123520 2024-06-11T19:26:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 10 monclient: dump: 2024-06-11T19:26:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.462 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.463 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.464 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.464 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.464 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.464 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0860015670 con 0x7f086c123520 2024-06-11T19:26:40.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:40.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:40.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123520 msgr2=0x7f086c123900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123520 0x7f086c123900 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f086c0a5620 tx=0x7f0860011920 comp rx=0 tx=0).stop 2024-06-11T19:26:40.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123520 msgr2=0x7f086c123900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:40.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.421+0000 7f0868ff9640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123520 0x7f086c123900 secure :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f086c0a5620 tx=0x7f0860011920 comp rx=0 tx=0).stop 2024-06-11T19:26:40.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.422+0000 7f0868ff9640 10 client.124386.objecter ms_handle_connect 0x7f085803fdb0 2024-06-11T19:26:40.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.422+0000 7f0870df8640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f085803fdb0 0x7f0858042270 secure :-1 s=READY pgs=4321 cs=0 l=1 rev1=1 crypto rx=0x7f0864009610 tx=0x7f08640097d0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f087285b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:26:40.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f087285b640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123e40 0x7f086c09e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f087285b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f087285b640 1 -- 172.21.15.98:0/3686185637 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1c0c40 con 0x7f086c123e40 2024-06-11T19:26:40.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f086b7fe640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123e40 0x7f086c09e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f086b7fe640 10 monclient: get_auth_request con 0x7f086c123e40 auth_method 0 2024-06-11T19:26:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f086b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f086b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f086b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f086b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f086b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f086b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f086b7fe640 10 monclient(hunting): handle_auth_done global_id 66321 payload 824 2024-06-11T19:26:40.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f086b7fe640 1 -- 172.21.15.98:0/3686185637 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f086c126a50 con 0x7f086c123e40 2024-06-11T19:26:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f0868ff9640 10 client.124386.objecter ms_handle_connect 0x7f086c123e40 2024-06-11T19:26:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f0868ff9640 10 client.124386.objecter resend_mon_ops 2024-06-11T19:26:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f086b7fe640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123e40 0x7f086c09e9f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f086c0a5620 tx=0x7f08600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08600222c0 con 0x7f086c123e40 2024-06-11T19:26:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f0868ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f0868ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.428+0000 7f0868ff9640 10 monclient: dump: 2024-06-11T19:26:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.469 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.471 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f08600222c0 con 0x7f086c123e40 2024-06-11T19:26:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f0868ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:26:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f0868ff9640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123e40 msgr2=0x7f086c09e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f0868ff9640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123e40 0x7f086c09e9f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f086c0a5620 tx=0x7f08600046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f0868ff9640 1 -- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123e40 msgr2=0x7f086c09e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f0868ff9640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f086c123e40 0x7f086c09e9f0 secure :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f086c0a5620 tx=0x7f08600046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.472 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:40.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:26:40.473 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:26:40.473 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:26:40.473 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 92, 2024-06-11T19:26:40.473 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 792256, 2024-06-11T19:26:40.473 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:26:40.473 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:26:40.473 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:26:40.473 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:26:40.474 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:26:40.474 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:26:40.474 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:26:40.474 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:26:40.474 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:26:40.474 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:26:40.474 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:40.474 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:26:40.475 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:40.475 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:26:40.475 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:26:40.475 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:40.475 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:40.475 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:40.475 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:40.475 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:40.476 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:40.476 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:40.476 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:40.476 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:40.476 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:40.476 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:40.476 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:40.476 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:26:40.477 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:26:40.477 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:40.477 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:40.477 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:40.477 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:40.477 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:40.477 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:40.477 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:40.477 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:40.478 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:40.478 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:40.478 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:40.478 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.478 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:26:40.478 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:26:40.478 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:26:40.479 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:26:40.479 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:26:40.479 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:26:40.479 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:40.479 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:40.479 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:26:40.479 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:26:40.479 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:26:40.479 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:26:40.480 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:26:40.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:26:40.480 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:26:40.480 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:40.480 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:26:40.480 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:40.480 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:40.480 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:40.481 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:40.481 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:40.481 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:40.481 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:40.481 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:40.481 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:40.481 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:40.481 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:40.481 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:40.482 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:26:40.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.482 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:26:40.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.482 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:26:40.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:26:40.482 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.483 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.483 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:26:40.483 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.483 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:26:40.484 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.484 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.484 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:26:40.484 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:26:40.484 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.484 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.484 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.485 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:26:40.485 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:26:40.485 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.485 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.485 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:26:40.486 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.486 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:26:40.486 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.486 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:26:40.487 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:26:40.487 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.487 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.487 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.487 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:26:40.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:26:40.488 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:26:40.488 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.489 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:26:40.489 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.489 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.489 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:26:40.489 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:26:40.490 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.490 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.490 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.490 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:26:40.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:26:40.490 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.491 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.491 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:26:40.491 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:26:40.492 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.492 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.492 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:26:40.492 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:26:40.492 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.492 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.493 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.493 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:26:40.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:26:40.493 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:26:40.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:26:40.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.495 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:26:40.495 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:26:40.495 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.495 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.495 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.496 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:26:40.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:26:40.496 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:26:40.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:26:40.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.497 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.497 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:26:40.498 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:26:40.498 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.498 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.498 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.498 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:26:40.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:26:40.499 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:26:40.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:26:40.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.500 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:26:40.500 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:26:40.500 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.501 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.501 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.501 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:26:40.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:26:40.501 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:26:40.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:26:40.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.503 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:26:40.503 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:26:40.503 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.503 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.503 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.504 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:26:40.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:26:40.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:26:40.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:26:40.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.505 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.505 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:26:40.506 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:26:40.506 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.506 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.506 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.506 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.507 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:26:40.507 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:26:40.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.507 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:40.507 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:40.507 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:40.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.507 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:40.508 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:26:40.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.508 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:40.508 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:40.508 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:26:40.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.508 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:40.508 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:26:40.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.509 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:40.509 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:40.509 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-06-11T19:26:40.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.509 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.509 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:26:40.510 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:40.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 10 librados: watch_flush enter 2024-06-11T19:26:40.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 10 librados: watch_flush exit 2024-06-11T19:26:40.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 20 client.124386.objecter shutdown clearing up homeless session... 2024-06-11T19:26:40.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 10 client.124386.objecter successfully canceled tick 2024-06-11T19:26:40.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 1 -- 172.21.15.98:0/3686185637 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f085803fdb0 msgr2=0x7f0858042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f085803fdb0 0x7f0858042270 secure :-1 s=READY pgs=4321 cs=0 l=1 rev1=1 crypto rx=0x7f0864009610 tx=0x7f08640097d0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 10 monclient: shutdown 2024-06-11T19:26:40.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:40.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 1 -- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f086c0a2530 msgr2=0x7f086c137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 1 --2- 172.21.15.98:0/3686185637 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f086c0a2530 0x7f086c137520 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f085c009870 tx=0x7f085c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f0870df8640 1 -- 172.21.15.98:0/3686185637 reap_dead start 2024-06-11T19:26:40.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f0870df8640 1 -- 172.21.15.98:0/3686185637 reap_dead start 2024-06-11T19:26:40.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 1 -- 172.21.15.98:0/3686185637 shutdown_connections 2024-06-11T19:26:40.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 1 -- 172.21.15.98:0/3686185637 >> 172.21.15.98:0/3686185637 conn(0x7f086c0ab850 msgr2=0x7f086c1be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:40.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 1 -- 172.21.15.98:0/3686185637 shutdown_connections 2024-06-11T19:26:40.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 1 -- 172.21.15.98:0/3686185637 wait complete. 2024-06-11T19:26:40.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.429+0000 7f087285b640 1 librados: shutdown 2024-06-11T19:26:40.514 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:26:40.514 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-06-11T19:26:40.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-11T19:26:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.614+0000 7fa013ff5640 10 monclient: get_monmap_and_config 2024-06-11T19:26:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.614+0000 7fa013ff5640 10 monclient: build_initial_monmap 2024-06-11T19:26:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.614+0000 7fa013ff5640 10 monclient: monmap: 2024-06-11T19:26:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:40.615802+0000 2024-06-11T19:26:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:40.615802+0000 2024-06-11T19:26:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:40.616 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.615+0000 7fa013ff5640 1 Processor -- start 2024-06-11T19:26:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.615+0000 7fa013ff5640 1 -- start start 2024-06-11T19:26:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.615+0000 7fa013ff5640 10 monclient: init 2024-06-11T19:26:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.615+0000 7fa013ff5640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.615+0000 7fa013ff5640 10 monclient: _add_conns ranks=[10,9,5,6,0,2,8,4,1,3,7] 2024-06-11T19:26:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.615+0000 7fa013ff5640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa00c1347f0 0x7fa00c09e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.615+0000 7fa013ff5640 10 monclient(hunting): picked mon.noname-f con 0x7fa00c1347f0 addr v2:172.21.15.117:3300/0 2024-06-11T19:26:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa013ff5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa00c134bd0 0x7fa00c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa013ff5640 10 monclient(hunting): picked mon.noname-a con 0x7fa00c134bd0 addr v2:172.21.15.98:3300/0 2024-06-11T19:26:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa013ff5640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa00c135010 0x7fa00c0a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa013ff5640 10 monclient(hunting): picked mon.noname-h con 0x7fa00c135010 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa013ff5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa013ff5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa00c0687f0 con 0x7fa00c134bd0 2024-06-11T19:26:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa013ff5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa013ff5640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fa00c09d410 con 0x7fa00c1347f0 2024-06-11T19:26:40.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa013ff5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa013ff5640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fa00c09d6f0 con 0x7fa00c135010 2024-06-11T19:26:40.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa013ff5640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:40.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa013ff5640 10 monclient(hunting): authenticate will time out at 1810.423043s 2024-06-11T19:26:40.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa00c134bd0 0x7fa00c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa00c134bd0 0x7fa00c0a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55432/0 (socket says 172.21.15.98:55432) 2024-06-11T19:26:40.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 1 -- 172.21.15.98:0/17580540 learned_addr learned my addr 172.21.15.98:0/17580540 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 10 monclient(hunting): get_auth_request con 0x7fa00c134bd0 auth_method 0 2024-06-11T19:26:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011d91640 1 --2- 172.21.15.98:0/17580540 >> v2:172.21.15.117:3300/0 conn(0x7fa00c1347f0 0x7fa00c09e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa012592640 1 --2- 172.21.15.98:0/17580540 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa00c135010 0x7fa00c0a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011d91640 10 monclient(hunting): get_auth_request con 0x7fa00c1347f0 auth_method 0 2024-06-11T19:26:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011d91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011d91640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011d91640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa012592640 10 monclient(hunting): get_auth_request con 0x7fa00c135010 auth_method 0 2024-06-11T19:26:40.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa012592640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa012592640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa012592640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011d91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011d91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011d91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa012592640 1 -- 172.21.15.98:0/17580540 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa00c135010 msgr2=0x7fa00c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-11T19:26:40.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa012592640 1 -- 172.21.15.98:0/17580540 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa00c135010 msgr2=0x7fa00c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-11T19:26:40.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa012592640 1 --2- 172.21.15.98:0/17580540 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa00c135010 0x7fa00c0a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:40.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 10 monclient(hunting): handle_auth_done global_id 66330 payload 274 2024-06-11T19:26:40.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 1 -- 172.21.15.98:0/17580540 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa00c135010 msgr2=0x7fa00c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:40.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa012592640 1 --2- 172.21.15.98:0/17580540 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa00c135010 0x7fa00c0a3bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:26:40.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 1 --2- 172.21.15.98:0/17580540 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa00c135010 0x7fa00c0a3bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 1 -- 172.21.15.98:0/17580540 >> v2:172.21.15.117:3300/0 conn(0x7fa00c1347f0 msgr2=0x7fa00c09e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:40.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 1 --2- 172.21.15.98:0/17580540 >> v2:172.21.15.117:3300/0 conn(0x7fa00c1347f0 0x7fa00c09e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 10 monclient: _finish_hunting 0 2024-06-11T19:26:40.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 1 monclient: found mon.noname-a 2024-06-11T19:26:40.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:40.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:26:40.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.616+0000 7fa011590640 1 -- 172.21.15.98:0/17580540 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00c09d9d0 con 0x7fa00c134bd0 2024-06-11T19:26:40.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa011590640 1 --2- 172.21.15.98:0/17580540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa00c134bd0 0x7fa00c0a3690 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa00c138910 tx=0x7f9ffc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d0f0df05a9099f server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 1 -- 172.21.15.98:0/17580540 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ffc02e070 con 0x7fa00c134bd0 2024-06-11T19:26:40.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:26:40.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 10 monclient: dump: 2024-06-11T19:26:40.634 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.634 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.634 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.634 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.638 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.638 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.638 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.638 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.638 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.638 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.638 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.639 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.639 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.639 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.639 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.639 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 10 monclient: _finish_auth 0 2024-06-11T19:26:40.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 10 monclient: _check_auth_tickets 2024-06-11T19:26:40.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:40.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 1 -- 172.21.15.98:0/17580540 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ffc037070 con 0x7fa00c134bd0 2024-06-11T19:26:40.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 5 monclient: authenticate success, global_id 66330 2024-06-11T19:26:40.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:40.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:40.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:40.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 1 -- 172.21.15.98:0/17580540 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ffc032470 con 0x7fa00c134bd0 2024-06-11T19:26:40.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 4 set_mon_vals no callback set 2024-06-11T19:26:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa010d8f640 10 monclient: dump: 2024-06-11T19:26:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.641 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.642 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.643 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.643 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.643 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 10 monclient: shutdown 2024-06-11T19:26:40.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:40.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 1 -- 172.21.15.98:0/17580540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa00c134bd0 msgr2=0x7fa00c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 1 --2- 172.21.15.98:0/17580540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa00c134bd0 0x7fa00c0a3690 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa00c138910 tx=0x7f9ffc02b880 comp rx=0 tx=0).stop 2024-06-11T19:26:40.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 1 -- 172.21.15.98:0/17580540 shutdown_connections 2024-06-11T19:26:40.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 1 --2- 172.21.15.98:0/17580540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa00c134bd0 0x7fa00c0a3690 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 1 --2- 172.21.15.98:0/17580540 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fa00c135010 0x7fa00c0a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 1 --2- 172.21.15.98:0/17580540 >> v2:172.21.15.117:3300/0 conn(0x7fa00c1347f0 0x7fa00c09e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.617+0000 7fa013ff5640 1 -- 172.21.15.98:0/17580540 >> 172.21.15.98:0/17580540 conn(0x7fa00c0a4780 msgr2=0x7fa00c0a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:40.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.618+0000 7fa013ff5640 1 -- 172.21.15.98:0/17580540 shutdown_connections 2024-06-11T19:26:40.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.618+0000 7fa013ff5640 1 -- 172.21.15.98:0/17580540 wait complete. 2024-06-11T19:26:40.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.618+0000 7fa013ff5640 10 monclient: build_initial_monmap 2024-06-11T19:26:40.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.618+0000 7fa013ff5640 10 monclient: monmap: 2024-06-11T19:26:40.645 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:40.645 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:40.645 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:40.645 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:40.645 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:40.645 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:40.645 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:40.645 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:40.646 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:40.646 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:40.646 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:40.646 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:40.646 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:40.646 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:40.646 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:40.646 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.618+0000 7fa013ff5640 1 librados: starting msgr at 2024-06-11T19:26:40.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.618+0000 7fa013ff5640 1 librados: starting objecter 2024-06-11T19:26:40.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.618+0000 7fa013ff5640 1 Processor -- start 2024-06-11T19:26:40.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.618+0000 7fa013ff5640 1 -- start start 2024-06-11T19:26:40.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.618+0000 7fa013ff5640 1 librados: setting wanted keys 2024-06-11T19:26:40.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.618+0000 7fa013ff5640 1 librados: calling monclient init 2024-06-11T19:26:40.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.618+0000 7fa013ff5640 10 monclient: init 2024-06-11T19:26:40.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 10 monclient: _add_conns ranks=[1,2,7,5,8,3,0,4,6] 2024-06-11T19:26:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa00c1347f0 0x7fa00c1cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 10 monclient(hunting): picked mon.noname-f con 0x7fa00c1347f0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:26:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c134bd0 0x7fa00c1ce160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 10 monclient(hunting): picked mon.noname-b con 0x7fa00c134bd0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa00c135010 0x7fa00c11ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 10 monclient(hunting): picked mon.noname-i con 0x7fa00c135010 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa00c068c40 con 0x7fa00c1347f0 2024-06-11T19:26:40.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa00c068ac0 con 0x7fa00c134bd0 2024-06-11T19:26:40.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa00c0a4110 con 0x7fa00c135010 2024-06-11T19:26:40.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:40.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa013ff5640 10 monclient(hunting): authenticate will time out at 1810.426399s 2024-06-11T19:26:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c134bd0 0x7fa00c1ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c134bd0 0x7fa00c1ce160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40306/0 (socket says 172.21.15.98:40306) 2024-06-11T19:26:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 1 -- 172.21.15.98:0/2318964568 learned_addr learned my addr 172.21.15.98:0/2318964568 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 10 monclient(hunting): get_auth_request con 0x7fa00c134bd0 auth_method 0 2024-06-11T19:26:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011d91640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa00c1347f0 0x7fa00c1cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa012592640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa00c135010 0x7fa00c11ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011d91640 10 monclient(hunting): get_auth_request con 0x7fa00c1347f0 auth_method 0 2024-06-11T19:26:40.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011d91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011d91640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011d91640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa012592640 10 monclient(hunting): get_auth_request con 0x7fa00c135010 auth_method 0 2024-06-11T19:26:40.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa012592640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa012592640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa012592640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011d91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011d91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011d91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa012592640 1 -- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa00c135010 msgr2=0x7fa00c11ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:26:40.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa012592640 1 -- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa00c135010 msgr2=0x7fa00c11ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:26:40.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa012592640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa00c135010 0x7fa00c11ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:40.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa012592640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa00c135010 0x7fa00c11ea50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:26:40.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 10 monclient(hunting): handle_auth_done global_id 85349 payload 824 2024-06-11T19:26:40.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 1 -- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa00c135010 msgr2=0x7fa00c11ea50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:26:40.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa00c135010 0x7fa00c11ea50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 1 -- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa00c1347f0 msgr2=0x7fa00c1cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa00c1347f0 0x7fa00c1cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 10 monclient: _finish_hunting 0 2024-06-11T19:26:40.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 1 monclient: found mon.noname-b 2024-06-11T19:26:40.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:40.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:26:40.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.619+0000 7fa011590640 1 -- 172.21.15.98:0/2318964568 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00c122730 con 0x7fa00c134bd0 2024-06-11T19:26:40.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa011590640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c134bd0 0x7fa00c1ce160 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc000c00 tx=0x7f9ffc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 10 client.?.objecter ms_handle_connect 0x7fa00c134bd0 2024-06-11T19:26:40.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:40.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ffc041070 con 0x7fa00c134bd0 2024-06-11T19:26:40.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:26:40.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 10 monclient: dump: 2024-06-11T19:26:40.655 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.655 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.655 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.655 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.655 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.656 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.656 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.656 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.656 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.656 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.656 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.656 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.656 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.657 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.657 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.657 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 10 monclient: _finish_auth 0 2024-06-11T19:26:40.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 10 monclient: _check_auth_tickets 2024-06-11T19:26:40.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:40.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ffc0315e0 con 0x7fa00c134bd0 2024-06-11T19:26:40.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa013ff5640 5 monclient: authenticate success, global_id 85349 2024-06-11T19:26:40.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa013ff5640 10 monclient: _renew_subs 2024-06-11T19:26:40.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa013ff5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:40.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa013ff5640 1 -- 172.21.15.98:0/2318964568 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa00c121280 con 0x7fa00c134bd0 2024-06-11T19:26:40.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:40.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ffc031c00 con 0x7fa00c134bd0 2024-06-11T19:26:40.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:26:40.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa0027fc640 10 monclient: dump: 2024-06-11T19:26:40.659 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.659 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.659 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.659 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.659 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.659 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.659 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.659 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.659 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.660 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.660 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.660 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.660 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.660 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.660 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.660 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa003fff640 4 set_mon_vals no callback set 2024-06-11T19:26:40.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa013ff5640 10 client.85349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:40.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa013ff5640 10 monclient: _renew_subs 2024-06-11T19:26:40.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa013ff5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:40.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa013ff5640 1 -- 172.21.15.98:0/2318964568 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa00c0a3730 con 0x7fa00c134bd0 2024-06-11T19:26:40.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa013ff5640 1 librados: init done 2024-06-11T19:26:40.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa013ff5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:40.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.620+0000 7fa013ff5640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c1222f0 0x7fa00c11ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa013ff5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa013ff5640 1 -- 172.21.15.98:0/2318964568 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa00c1214a0 con 0x7fa00c1222f0 2024-06-11T19:26:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f9ffc04c060 con 0x7fa00c134bd0 2024-06-11T19:26:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa011d91640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c1222f0 0x7fa00c11ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa011d91640 10 monclient: get_auth_request con 0x7fa00c1222f0 auth_method 0 2024-06-11T19:26:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa011d91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa011d91640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa011d91640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa011d91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa011d91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa011d91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9ff003fdb0 0x7f9ff0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa011d91640 10 monclient(hunting): handle_auth_done global_id 85358 payload 824 2024-06-11T19:26:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f9ffc072980 con 0x7fa00c134bd0 2024-06-11T19:26:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 10 client.85349.objecter ms_dispatch 0x7fa00c000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:40.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 3 client.85349.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:40.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 3 client.85349.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:40.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa011d91640 1 -- 172.21.15.98:0/2318964568 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa00c09e6b0 con 0x7fa00c1222f0 2024-06-11T19:26:40.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 20 client.85349.objecter dump_active .. 0 homeless 2024-06-11T19:26:40.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa012592640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9ff003fdb0 0x7f9ff0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa012592640 10 monclient: get_auth_request con 0x7f9ff003fdb0 auth_method 0 2024-06-11T19:26:40.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 10 client.85349.objecter ms_handle_connect 0x7fa00c1222f0 2024-06-11T19:26:40.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 10 client.85349.objecter resend_mon_ops 2024-06-11T19:26:40.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa011d91640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c1222f0 0x7fa00c11ef90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa00800f410 tx=0x7fa00800f8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa008019070 con 0x7fa00c1222f0 2024-06-11T19:26:40.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:26:40.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 10 monclient: dump: 2024-06-11T19:26:40.665 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.665 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.665 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.665 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.666 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.666 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.666 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.666 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.666 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.666 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.666 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.666 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.667 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 10 client.85349.objecter ms_handle_connect 0x7f9ff003fdb0 2024-06-11T19:26:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa012592640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9ff003fdb0 0x7f9ff0042270 secure :-1 s=READY pgs=4322 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0a4750 tx=0x7fa004008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa008003990 con 0x7fa00c1222f0 2024-06-11T19:26:40.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.621+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c1222f0 msgr2=0x7fa00c11ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.622+0000 7fa0027fc640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c1222f0 0x7fa00c11ef90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa00800f410 tx=0x7fa00800f8d0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.622+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c1222f0 msgr2=0x7fa00c11ef90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:40.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.622+0000 7fa0027fc640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c1222f0 0x7fa00c11ef90 secure :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa00800f410 tx=0x7fa00800f8d0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa013ff5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:26:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa013ff5640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c11f4d0 0x7fa00c11f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa013ff5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa013ff5640 1 -- 172.21.15.98:0/2318964568 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa00c050830 con 0x7fa00c11f4d0 2024-06-11T19:26:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa011d91640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c11f4d0 0x7fa00c11f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa011d91640 10 monclient: get_auth_request con 0x7fa00c11f4d0 auth_method 0 2024-06-11T19:26:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa011d91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa011d91640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa011d91640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa011d91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa011d91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa011d91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa011d91640 10 monclient(hunting): handle_auth_done global_id 85367 payload 824 2024-06-11T19:26:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa011d91640 1 -- 172.21.15.98:0/2318964568 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa00c11fdf0 con 0x7fa00c11f4d0 2024-06-11T19:26:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa0027fc640 10 client.85349.objecter ms_handle_connect 0x7fa00c11f4d0 2024-06-11T19:26:40.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa0027fc640 10 client.85349.objecter resend_mon_ops 2024-06-11T19:26:40.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa011d91640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c11f4d0 0x7fa00c11f8b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa008012010 tx=0x7fa008004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa00801ead0 con 0x7fa00c11f4d0 2024-06-11T19:26:40.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa0027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa0027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:26:40.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.628+0000 7fa0027fc640 10 monclient: dump: 2024-06-11T19:26:40.673 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.673 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.673 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.674 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.675 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7fa00801e660 con 0x7fa00c11f4d0 2024-06-11T19:26:40.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa0027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:26:40.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa0027fc640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:40.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c11f4d0 msgr2=0x7fa00c11f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa0027fc640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c11f4d0 0x7fa00c11f8b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa008012010 tx=0x7fa008004a40 comp rx=0 tx=0).stop 2024-06-11T19:26:40.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa0027fc640 1 -- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c11f4d0 msgr2=0x7fa00c11f8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:40.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa0027fc640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c11f4d0 0x7fa00c11f8b0 secure :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa008012010 tx=0x7fa008004a40 comp rx=0 tx=0).stop 2024-06-11T19:26:40.676 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:40.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:26:40.677 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:26:40.677 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-06-11T19:26:40.677 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 92, 2024-06-11T19:26:40.677 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 672402, 2024-06-11T19:26:40.677 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:26:40.677 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:26:40.678 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:26:40.678 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:26:40.678 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:26:40.678 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:26:40.678 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:26:40.678 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:26:40.678 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:26:40.678 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:26:40.679 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:40.679 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-06-11T19:26:40.679 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:40.679 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:26:40.679 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:26:40.679 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:40.679 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:40.679 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:40.680 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:40.680 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:40.680 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:40.680 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:40.680 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:40.680 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:40.680 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:40.680 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:40.681 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:40.681 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:26:40.681 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:26:40.681 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:40.681 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:40.681 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:40.681 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:40.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:40.682 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:40.682 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:40.682 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:40.682 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:40.682 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:40.682 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:40.682 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.683 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:26:40.683 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:26:40.683 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:26:40.683 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:26:40.683 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:26:40.683 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:26:40.683 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:40.683 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:40.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:26:40.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:26:40.684 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:26:40.684 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:26:40.684 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:26:40.684 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:26:40.684 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:26:40.685 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:40.685 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:26:40.685 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:40.685 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:40.685 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:40.685 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:40.685 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:40.685 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:40.685 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:40.686 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:40.686 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:40.686 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:40.686 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:40.686 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:40.686 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:26:40.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.687 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:26:40.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.687 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:26:40.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:26:40.687 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:26:40.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:26:40.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:26:40.689 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:26:40.689 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.689 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.689 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.690 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:26:40.690 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:26:40.690 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:26:40.690 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:26:40.691 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.691 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.691 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:26:40.692 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:26:40.692 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.692 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.692 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.692 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:26:40.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:26:40.693 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:26:40.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:26:40.694 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:26:40.694 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:26:40.694 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.695 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.695 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.695 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:26:40.695 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:26:40.695 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.695 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:26:40.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:26:40.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.697 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:26:40.697 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:26:40.697 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.697 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.697 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.698 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:26:40.698 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:26:40.698 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:26:40.699 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:26:40.699 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.699 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.699 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:26:40.700 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:26:40.700 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.700 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.700 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.701 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:26:40.701 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:26:40.701 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:26:40.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:26:40.702 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.702 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.702 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:26:40.703 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:26:40.703 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.703 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.703 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.703 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:26:40.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:26:40.704 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:26:40.704 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:26:40.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.705 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:26:40.705 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:26:40.706 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.706 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.706 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.706 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:26:40.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:26:40.706 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:26:40.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:26:40.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.708 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.708 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:26:40.708 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:26:40.708 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.708 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.709 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.709 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:26:40.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:26:40.709 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:40.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:40.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:40.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:26:40.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:40.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:26:40.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:40.711 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:26:40.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:26:40.711 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:40.711 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:40.711 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:40.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.712 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.712 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:26:40.712 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:26:40.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.712 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:40.712 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:40.712 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:40.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.713 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:40.713 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-06-11T19:26:40.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.713 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:40.713 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:40.713 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:26:40.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.714 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:40.714 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:26:40.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:40.714 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:40.714 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:40.714 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-06-11T19:26:40.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:40.714 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:40.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:40.715 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:26:40.715 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:40.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 10 librados: watch_flush enter 2024-06-11T19:26:40.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 10 librados: watch_flush exit 2024-06-11T19:26:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 20 client.85349.objecter shutdown clearing up homeless session... 2024-06-11T19:26:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 10 client.85349.objecter successfully canceled tick 2024-06-11T19:26:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 1 -- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9ff003fdb0 msgr2=0x7f9ff0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f9ff003fdb0 0x7f9ff0042270 secure :-1 s=READY pgs=4322 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0a4750 tx=0x7fa004008040 comp rx=0 tx=0).stop 2024-06-11T19:26:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 10 monclient: shutdown 2024-06-11T19:26:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:40.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 1 -- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c134bd0 msgr2=0x7fa00c1ce160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c134bd0 0x7fa00c1ce160 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc000c00 tx=0x7f9ffc0046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa012592640 1 -- 172.21.15.98:0/2318964568 reap_dead start 2024-06-11T19:26:40.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 1 -- 172.21.15.98:0/2318964568 shutdown_connections 2024-06-11T19:26:40.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 1 --2- 172.21.15.98:0/2318964568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa00c134bd0 0x7fa00c1ce160 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 1 -- 172.21.15.98:0/2318964568 >> 172.21.15.98:0/2318964568 conn(0x7fa00c0a4780 msgr2=0x7fa00c1c01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:40.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 1 -- 172.21.15.98:0/2318964568 shutdown_connections 2024-06-11T19:26:40.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 1 -- 172.21.15.98:0/2318964568 wait complete. 2024-06-11T19:26:40.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.629+0000 7fa013ff5640 1 librados: shutdown 2024-06-11T19:26:40.719 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:26:40.719 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-06-11T19:26:40.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-11T19:26:40.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.819+0000 7fdac8dda640 10 monclient: get_monmap_and_config 2024-06-11T19:26:40.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.819+0000 7fdac8dda640 10 monclient: build_initial_monmap 2024-06-11T19:26:40.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.819+0000 7fdac8dda640 10 monclient: monmap: 2024-06-11T19:26:40.818 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:40.819 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.819 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:40.820172+0000 2024-06-11T19:26:40.819 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:40.820172+0000 2024-06-11T19:26:40.819 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:40.819 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.819 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:40.819 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:40.819 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:40.820 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:40.821 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.819+0000 7fdac8dda640 1 Processor -- start 2024-06-11T19:26:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.819+0000 7fdac8dda640 1 -- start start 2024-06-11T19:26:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.819+0000 7fdac8dda640 10 monclient: init 2024-06-11T19:26:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 10 monclient: _add_conns ranks=[2,9,5,6,8,7,3,10,4,0,1] 2024-06-11T19:26:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdac4136c90 0x7fdac4137070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 10 monclient(hunting): picked mon.noname-b con 0x7fdac4136c90 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdac41367d0 0x7fdac40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 10 monclient(hunting): picked mon.noname-a con 0x7fdac41367d0 addr v2:172.21.15.98:3300/0 2024-06-11T19:26:40.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdac40a3bd0 0x7fdac40a4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 10 monclient(hunting): picked mon.noname-h con 0x7fdac40a3bd0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:40.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdac4068180 con 0x7fdac41367d0 2024-06-11T19:26:40.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdac40a0270 con 0x7fdac4136c90 2024-06-11T19:26:40.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fdac409fc20 con 0x7fdac40a3bd0 2024-06-11T19:26:40.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:40.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac8dda640 10 monclient(hunting): authenticate will time out at 1810.627286s 2024-06-11T19:26:40.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdac4136c90 0x7fdac4137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdac41367d0 0x7fdac40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdac4136c90 0x7fdac4137070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40338/0 (socket says 172.21.15.98:40338) 2024-06-11T19:26:40.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdac41367d0 0x7fdac40a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55436/0 (socket says 172.21.15.98:55436) 2024-06-11T19:26:40.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac259c640 1 -- 172.21.15.98:0/1097772801 learned_addr learned my addr 172.21.15.98:0/1097772801 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:40.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac259c640 10 monclient(hunting): get_auth_request con 0x7fdac4136c90 auth_method 0 2024-06-11T19:26:40.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 10 monclient(hunting): get_auth_request con 0x7fdac41367d0 auth_method 0 2024-06-11T19:26:40.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac2d9d640 1 --2- 172.21.15.98:0/1097772801 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdac40a3bd0 0x7fdac40a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac2d9d640 10 monclient(hunting): get_auth_request con 0x7fdac40a3bd0 auth_method 0 2024-06-11T19:26:40.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac2d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_done global_id 66339 payload 274 2024-06-11T19:26:40.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 1 -- 172.21.15.98:0/1097772801 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdac40a3bd0 msgr2=0x7fdac40a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:40.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 1 --2- 172.21.15.98:0/1097772801 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdac40a3bd0 0x7fdac40a4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 1 -- 172.21.15.98:0/1097772801 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdac4136c90 msgr2=0x7fdac4137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:40.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 1 --2- 172.21.15.98:0/1097772801 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdac4136c90 0x7fdac4137070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:26:40.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 1 monclient: found mon.noname-a 2024-06-11T19:26:40.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:40.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:26:40.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdab9d9b640 1 -- 172.21.15.98:0/1097772801 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac409ff00 con 0x7fdac41367d0 2024-06-11T19:26:40.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.820+0000 7fdac259c640 1 --2- 172.21.15.98:0/1097772801 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdac4136c90 0x7fdac4137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:40.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdab9d9b640 1 --2- 172.21.15.98:0/1097772801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac41367d0 0x7fdac40a3690 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0097a0 tx=0x7fdaac02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c35b0a920ed32ae server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 1 -- 172.21.15.98:0/1097772801 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdaac039070 con 0x7fdac41367d0 2024-06-11T19:26:40.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:26:40.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 10 monclient: dump: 2024-06-11T19:26:40.837 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.837 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.837 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.837 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.837 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.837 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.837 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.838 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.838 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.838 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.838 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.838 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.838 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.838 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.838 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.839 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 10 monclient: _finish_auth 0 2024-06-11T19:26:40.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 10 monclient: _check_auth_tickets 2024-06-11T19:26:40.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:40.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 1 -- 172.21.15.98:0/1097772801 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaac031960 con 0x7fdac41367d0 2024-06-11T19:26:40.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac8dda640 5 monclient: authenticate success, global_id 66339 2024-06-11T19:26:40.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac8dda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:40.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:40.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 1 -- 172.21.15.98:0/1097772801 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdaac032380 con 0x7fdac41367d0 2024-06-11T19:26:40.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac8dda640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:40.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac8dda640 4 set_mon_vals no callback set 2024-06-11T19:26:40.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:40.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.821+0000 7fdac1d9b640 10 monclient: dump: 2024-06-11T19:26:40.840 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.840 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.841 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.841 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.841 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.841 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.841 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.841 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.841 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.841 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.841 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.842 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.842 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.842 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.842 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.842 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 10 monclient: shutdown 2024-06-11T19:26:40.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:40.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 1 -- 172.21.15.98:0/1097772801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac41367d0 msgr2=0x7fdac40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 1 --2- 172.21.15.98:0/1097772801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac41367d0 0x7fdac40a3690 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0097a0 tx=0x7fdaac02b790 comp rx=0 tx=0).stop 2024-06-11T19:26:40.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 1 -- 172.21.15.98:0/1097772801 shutdown_connections 2024-06-11T19:26:40.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 1 --2- 172.21.15.98:0/1097772801 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdac40a3bd0 0x7fdac40a4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 1 --2- 172.21.15.98:0/1097772801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac41367d0 0x7fdac40a3690 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 1 --2- 172.21.15.98:0/1097772801 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdac4136c90 0x7fdac4137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 1 -- 172.21.15.98:0/1097772801 >> 172.21.15.98:0/1097772801 conn(0x7fdac40a4780 msgr2=0x7fdac40a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:40.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 1 -- 172.21.15.98:0/1097772801 shutdown_connections 2024-06-11T19:26:40.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 1 -- 172.21.15.98:0/1097772801 wait complete. 2024-06-11T19:26:40.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 10 monclient: build_initial_monmap 2024-06-11T19:26:40.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.822+0000 7fdac8dda640 10 monclient: monmap: 2024-06-11T19:26:40.847 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:40.847 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:40.847 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:40.847 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:40.847 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:40.847 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:40.847 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:40.847 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:40.847 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:40.848 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:40.848 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:40.848 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:40.848 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:40.848 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:40.848 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:40.848 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 1 librados: starting msgr at 2024-06-11T19:26:40.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 1 librados: starting objecter 2024-06-11T19:26:40.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 1 Processor -- start 2024-06-11T19:26:40.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 1 -- start start 2024-06-11T19:26:40.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 1 librados: setting wanted keys 2024-06-11T19:26:40.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 1 librados: calling monclient init 2024-06-11T19:26:40.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 10 monclient: init 2024-06-11T19:26:40.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:40.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 10 monclient: _add_conns ranks=[3,6,5,7,2,4,1,8,0] 2024-06-11T19:26:40.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdac40a3bd0 0x7fdac41cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 10 monclient(hunting): picked mon.noname-g con 0x7fdac40a3bd0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:40.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdac41367d0 0x7fdac41cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 10 monclient(hunting): picked mon.noname-d con 0x7fdac41367d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:40.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdac4136c90 0x7fdac4120b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 10 monclient(hunting): picked mon.noname-h con 0x7fdac4136c90 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:40.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fdac41375b0 con 0x7fdac40a3bd0 2024-06-11T19:26:40.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fdac4068560 con 0x7fdac4136c90 2024-06-11T19:26:40.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fdac4123330 con 0x7fdac41367d0 2024-06-11T19:26:40.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:40.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.823+0000 7fdac8dda640 10 monclient(hunting): authenticate will time out at 1810.630981s 2024-06-11T19:26:40.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdac41367d0 0x7fdac41cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdac41367d0 0x7fdac41cbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46984/0 (socket says 172.21.15.98:46984) 2024-06-11T19:26:40.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 1 -- 172.21.15.98:0/3228699659 learned_addr learned my addr 172.21.15.98:0/3228699659 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:40.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 10 monclient(hunting): get_auth_request con 0x7fdac41367d0 auth_method 0 2024-06-11T19:26:40.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:40.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:40.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:40.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac2d9d640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdac4136c90 0x7fdac4120b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdac40a3bd0 0x7fdac41cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac2d9d640 10 monclient(hunting): get_auth_request con 0x7fdac4136c90 auth_method 0 2024-06-11T19:26:40.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac2d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_done global_id 124395 payload 824 2024-06-11T19:26:40.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdac4136c90 msgr2=0x7fdac4120b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fdac4136c90 0x7fdac4120b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdac40a3bd0 msgr2=0x7fdac41cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:40.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdac40a3bd0 0x7fdac41cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:40.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:26:40.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 1 monclient: found mon.noname-d 2024-06-11T19:26:40.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:40.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:40.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 1 -- 172.21.15.98:0/3228699659 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac40a4bf0 con 0x7fdac41367d0 2024-06-11T19:26:40.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac259c640 10 monclient: get_auth_request con 0x7fdac40a3bd0 auth_method 0 2024-06-11T19:26:40.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdac40a3bd0 0x7fdac41cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:26:40.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdab9d9b640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdac41367d0 0x7fdac41cbed0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0096f0 tx=0x7fdaac0046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:40.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 10 client.?.objecter ms_handle_connect 0x7fdac41367d0 2024-06-11T19:26:40.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:40.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdaac043030 con 0x7fdac41367d0 2024-06-11T19:26:40.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:40.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 10 monclient: dump: 2024-06-11T19:26:40.855 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.855 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.856 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.856 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.856 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.856 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.856 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.856 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.856 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.856 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.857 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.857 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.857 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.857 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.857 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.857 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 10 monclient: _finish_auth 0 2024-06-11T19:26:40.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:26:40.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:40.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaac0044b0 con 0x7fdac41367d0 2024-06-11T19:26:40.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac8dda640 5 monclient: authenticate success, global_id 124395 2024-06-11T19:26:40.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac8dda640 10 monclient: _renew_subs 2024-06-11T19:26:40.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac8dda640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:40.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac8dda640 1 -- 172.21.15.98:0/3228699659 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac40a7850 con 0x7fdac41367d0 2024-06-11T19:26:40.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:40.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdaac039040 con 0x7fdac41367d0 2024-06-11T19:26:40.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:40.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:40.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdabb7fe640 10 monclient: dump: 2024-06-11T19:26:40.859 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:40.859 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:40.859 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.859 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:40.859 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:40.860 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:40.860 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:40.860 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:40.860 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:40.860 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:40.860 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:40.860 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:40.860 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:40.861 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:40.861 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:40.861 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:40.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac8dda640 10 client.124395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:40.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac8dda640 10 monclient: _renew_subs 2024-06-11T19:26:40.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac159a640 4 set_mon_vals no callback set 2024-06-11T19:26:40.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac8dda640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:40.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac8dda640 1 -- 172.21.15.98:0/3228699659 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac41247e0 con 0x7fdac41367d0 2024-06-11T19:26:40.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.824+0000 7fdac8dda640 1 librados: init done 2024-06-11T19:26:40.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac8dda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:40.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac8dda640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac8dda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:40.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac8dda640 1 -- 172.21.15.98:0/3228699659 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fdac4137730 con 0x7fdac4121040 2024-06-11T19:26:40.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac259c640 10 monclient: get_auth_request con 0x7fdac4121040 auth_method 0 2024-06-11T19:26:40.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fdaac046070 con 0x7fdac41367d0 2024-06-11T19:26:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac259c640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 msgr2=0x7fdac4121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-06-11T19:26:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac259c640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 msgr2=0x7fdac4121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:26:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:26:40.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.825+0000 7fdabb7fe640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdaa403fdb0 0x7fdaa4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.826+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7fdaac044070 con 0x7fdac41367d0 2024-06-11T19:26:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.826+0000 7fdabb7fe640 10 client.124395.objecter ms_dispatch 0x7fdac4000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.826+0000 7fdabb7fe640 3 client.124395.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.826+0000 7fdabb7fe640 3 client.124395.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.826+0000 7fdabb7fe640 20 client.124395.objecter dump_active .. 0 homeless 2024-06-11T19:26:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.826+0000 7fdac2d9d640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdaa403fdb0 0x7fdaa4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.826+0000 7fdac2d9d640 10 monclient: get_auth_request con 0x7fdaa403fdb0 auth_method 0 2024-06-11T19:26:40.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.826+0000 7fdabb7fe640 10 client.124395.objecter ms_handle_connect 0x7fdaa403fdb0 2024-06-11T19:26:40.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:40.826+0000 7fdac2d9d640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdaa403fdb0 0x7fdaa4042270 secure :-1 s=READY pgs=4323 cs=0 l=1 rev1=1 crypto rx=0x7fdac4122880 tx=0x7fdabc00b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:41.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.026+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.026+0000 7fdac259c640 10 monclient: get_auth_request con 0x7fdac4121040 auth_method 0 2024-06-11T19:26:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.026+0000 7fdac259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.026+0000 7fdac259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.026+0000 7fdac259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:26:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.026+0000 7fdac259c640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 msgr2=0x7fdac4121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:26:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.026+0000 7fdac259c640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 msgr2=0x7fdac4121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:26:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.026+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:41.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.026+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-11T19:26:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.427+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:41.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.427+0000 7fdac259c640 10 monclient: get_auth_request con 0x7fdac4121040 auth_method 0 2024-06-11T19:26:41.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.427+0000 7fdac259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:41.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.427+0000 7fdac259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:41.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.427+0000 7fdac259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:26:41.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.427+0000 7fdac259c640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 msgr2=0x7fdac4121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:26:41.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.427+0000 7fdac259c640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 msgr2=0x7fdac4121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:26:41.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.427+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:41.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:41.427+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-11T19:26:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:42.228+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:42.228+0000 7fdac259c640 10 monclient: get_auth_request con 0x7fdac4121040 auth_method 0 2024-06-11T19:26:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:42.228+0000 7fdac259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:42.228+0000 7fdac259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:42.228+0000 7fdac259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:26:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:42.228+0000 7fdac259c640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 msgr2=0x7fdac4121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:26:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:42.228+0000 7fdac259c640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 msgr2=0x7fdac4121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:26:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:42.228+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:42.228+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-11T19:26:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.824+0000 7fdaba7fc640 10 monclient: tick 2024-06-11T19:26:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.824+0000 7fdaba7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:26:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.824+0000 7fdaba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.824+0000 7fdaba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.830+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:43.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.830+0000 7fdac259c640 10 monclient: get_auth_request con 0x7fdac4121040 auth_method 0 2024-06-11T19:26:43.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.830+0000 7fdac259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:43.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.830+0000 7fdac259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:43.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.830+0000 7fdac259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-06-11T19:26:43.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.830+0000 7fdac259c640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 msgr2=0x7fdac4121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:26:43.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.830+0000 7fdac259c640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 msgr2=0x7fdac4121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:26:43.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.830+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:43.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:43.830+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-06-11T19:26:44.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdaba7fc640 10 monclient: tick 2024-06-11T19:26:44.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdaba7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:26:44.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdaba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:44.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdaba7fc640 5 monclient: _check_tell_commands timeout tell command 1 2024-06-11T19:26:44.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdaba7fc640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 msgr2=0x7fdac4121420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:26:44.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdaba7fc640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:44.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdaba7fc640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda9c003540 0x7fda9c005a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:44.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdaba7fc640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 msgr2=0x7fdac4121420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:44.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdaba7fc640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121040 0x7fdac4121420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:44.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdaba7fc640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:44.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdaba7fc640 1 -- 172.21.15.98:0/3228699659 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fda9c0060b0 con 0x7fda9c003540 2024-06-11T19:26:44.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda9c003540 0x7fda9c005a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:44.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdac259c640 10 monclient: get_auth_request con 0x7fda9c003540 auth_method 0 2024-06-11T19:26:44.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdac259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:44.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdac259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdac259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdac259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdac259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdac259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdac259c640 10 monclient(hunting): handle_auth_done global_id 134104 payload 824 2024-06-11T19:26:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdac259c640 1 -- 172.21.15.98:0/3228699659 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda9c006260 con 0x7fda9c003540 2024-06-11T19:26:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda9c003540 0x7fda9c005a00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdab4006c90 tx=0x7fdab4018bd0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdabb7fe640 10 client.124395.objecter ms_handle_connect 0x7fda9c003540 2024-06-11T19:26:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.824+0000 7fdabb7fe640 10 client.124395.objecter resend_mon_ops 2024-06-11T19:26:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.825+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdab4026070 con 0x7fda9c003540 2024-06-11T19:26:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.825+0000 7fdabb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.825+0000 7fdabb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:26:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.825+0000 7fdabb7fe640 10 monclient: dump: 2024-06-11T19:26:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:44.829 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:44.829 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:44.829 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:44.829 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:44.829 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:44.829 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:44.829 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:44.829 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:44.829 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:44.830 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.825+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdab4016bb0 con 0x7fda9c003540 2024-06-11T19:26:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.825+0000 7fdabb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.825+0000 7fdabb7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.825+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda9c003540 msgr2=0x7fda9c005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.825+0000 7fdabb7fe640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda9c003540 0x7fda9c005a00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdab4006c90 tx=0x7fdab4018bd0 comp rx=0 tx=0).stop 2024-06-11T19:26:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.825+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda9c003540 msgr2=0x7fda9c005a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.825+0000 7fdabb7fe640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda9c003540 0x7fda9c005a00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.831+0000 7fdac8dda640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:26:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac8dda640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121960 0x7fdac4071540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac8dda640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac8dda640 1 -- 172.21.15.98:0/3228699659 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fdac4050830 con 0x7fdac4121960 2024-06-11T19:26:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121960 0x7fdac4071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac259c640 10 monclient: get_auth_request con 0x7fdac4121960 auth_method 0 2024-06-11T19:26:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac259c640 10 monclient(hunting): handle_auth_done global_id 134113 payload 824 2024-06-11T19:26:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac259c640 1 -- 172.21.15.98:0/3228699659 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdac41230b0 con 0x7fdac4121960 2024-06-11T19:26:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac259c640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121960 0x7fdac4071540 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdab4000c00 tx=0x7fdab4004730 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 10 client.124395.objecter ms_handle_connect 0x7fdac4121960 2024-06-11T19:26:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 10 client.124395.objecter resend_mon_ops 2024-06-11T19:26:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdab4026040 con 0x7fdac4121960 2024-06-11T19:26:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:26:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 10 monclient: dump: 2024-06-11T19:26:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:44.837 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fdab40066c0 con 0x7fdac4121960 2024-06-11T19:26:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:26:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121960 msgr2=0x7fdac4071540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121960 0x7fdac4071540 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdab4000c00 tx=0x7fdab4004730 comp rx=0 tx=0).stop 2024-06-11T19:26:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121960 msgr2=0x7fdac4071540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdabb7fe640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdac4121960 0x7fdac4071540 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdab4000c00 tx=0x7fdab4004730 comp rx=0 tx=0).stop 2024-06-11T19:26:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.832+0000 7fdac2d9d640 1 -- 172.21.15.98:0/3228699659 reap_dead start 2024-06-11T19:26:44.838 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:44.839 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:26:44.839 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:26:44.839 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:26:44.839 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 92, 2024-06-11T19:26:44.839 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 16009, 2024-06-11T19:26:44.839 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:26:44.839 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:26:44.839 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:26:44.840 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:26:44.840 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:26:44.840 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:26:44.840 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:26:44.840 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:26:44.840 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:26:44.840 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:26:44.840 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:44.841 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-06-11T19:26:44.841 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:44.841 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:26:44.841 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:26:44.841 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:44.841 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:44.841 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:44.841 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:44.842 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:44.842 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:44.842 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:44.842 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:44.842 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:44.842 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:44.842 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:44.842 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:44.843 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:26:44.843 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:26:44.843 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:44.843 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:44.843 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:44.843 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:44.843 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:44.843 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:44.844 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:44.844 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:44.844 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:44.844 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:44.844 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:44.844 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:44.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.844 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:26:44.844 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:26:44.845 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:26:44.845 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:26:44.845 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:26:44.845 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:26:44.845 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:44.845 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:44.845 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:26:44.845 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:26:44.845 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:26:44.846 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:26:44.846 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:26:44.846 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:26:44.846 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:26:44.846 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:44.846 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:26:44.846 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:44.846 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:44.847 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:44.847 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:44.847 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:44.847 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:44.847 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:44.847 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:44.847 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:44.847 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:44.847 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:44.848 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:44.848 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:26:44.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.848 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:26:44.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.848 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:26:44.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:26:44.848 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:44.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:44.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:44.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:26:44.849 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:44.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:26:44.849 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.850 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:44.850 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:44.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.850 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:26:44.850 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:26:44.850 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:44.850 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:44.850 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:44.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.851 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:26:44.851 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:26:44.851 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:44.851 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:44.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.851 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:44.851 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:26:44.851 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:44.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:26:44.852 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:44.852 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:44.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:26:44.853 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:26:44.853 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:44.853 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:44.853 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:44.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.853 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:26:44.853 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:26:44.853 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:44.853 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:44.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:44.854 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:26:44.854 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:44.854 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:26:44.855 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.855 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:44.855 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:44.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.855 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:26:44.855 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:26:44.855 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:44.855 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:44.855 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:44.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.856 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:26:44.856 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:26:44.856 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:44.856 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:44.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:44.856 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:26:44.856 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:44.857 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:26:44.857 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.857 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:44.857 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:44.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.857 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:26:44.857 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:26:44.858 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:44.858 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:44.858 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:44.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.858 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:26:44.858 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:26:44.858 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:44.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:44.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:44.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:26:44.859 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:44.860 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:26:44.860 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.860 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:44.860 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:44.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.860 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:26:44.860 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:26:44.860 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:44.860 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:44.861 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:44.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.861 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:26:44.861 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:26:44.861 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:44.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:44.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:44.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:26:44.862 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:44.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:26:44.862 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.862 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:44.862 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:44.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:26:44.863 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:26:44.863 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:44.863 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:44.863 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:44.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.863 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:26:44.864 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:26:44.864 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:44.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:44.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:44.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:26:44.864 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:44.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:26:44.865 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:44.865 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:44.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:26:44.865 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:26:44.865 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:44.865 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:44.866 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:44.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.866 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:26:44.866 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:26:44.866 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:44.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:44.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:44.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:26:44.867 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:44.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:26:44.867 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.867 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:44.868 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:44.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:26:44.868 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:26:44.868 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:44.868 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:44.868 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:44.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.869 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:26:44.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:26:44.869 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:44.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:44.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:44.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:26:44.869 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:44.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:26:44.870 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:44.870 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:44.870 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:44.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:26:44.871 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:26:44.871 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:44.871 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:44.871 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:44.871 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:44.871 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:44.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.871 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:26:44.871 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:26:44.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:26:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:44.873 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:44.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:44.873 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:44.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:44.873 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:26:44.873 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:44.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 10 librados: watch_flush enter 2024-06-11T19:26:44.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 10 librados: watch_flush exit 2024-06-11T19:26:44.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 20 client.124395.objecter shutdown clearing up homeless session... 2024-06-11T19:26:44.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 10 client.124395.objecter successfully canceled tick 2024-06-11T19:26:44.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdaa403fdb0 msgr2=0x7fdaa4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:44.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdaa403fdb0 0x7fdaa4042270 secure :-1 s=READY pgs=4323 cs=0 l=1 rev1=1 crypto rx=0x7fdac4122880 tx=0x7fdabc00b040 comp rx=0 tx=0).stop 2024-06-11T19:26:44.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 10 monclient: shutdown 2024-06-11T19:26:44.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:44.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 1 -- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdac41367d0 msgr2=0x7fdac41cbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:44.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdac41367d0 0x7fdac41cbed0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0096f0 tx=0x7fdaac0046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:44.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 1 -- 172.21.15.98:0/3228699659 shutdown_connections 2024-06-11T19:26:44.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fdaa403fdb0 0x7fdaa4042270 unknown :-1 s=CLOSED pgs=4323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:44.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 1 --2- 172.21.15.98:0/3228699659 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdac41367d0 0x7fdac41cbed0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:44.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 1 -- 172.21.15.98:0/3228699659 >> 172.21.15.98:0/3228699659 conn(0x7fdac40a4780 msgr2=0x7fdac41bd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:44.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 1 -- 172.21.15.98:0/3228699659 shutdown_connections 2024-06-11T19:26:44.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 1 -- 172.21.15.98:0/3228699659 wait complete. 2024-06-11T19:26:44.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.833+0000 7fdac8dda640 1 librados: shutdown 2024-06-11T19:26:44.877 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:26:44.877 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-06-11T19:26:44.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-06-11T19:26:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.978+0000 7fc05c9e5640 10 monclient: get_monmap_and_config 2024-06-11T19:26:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.978+0000 7fc05c9e5640 10 monclient: build_initial_monmap 2024-06-11T19:26:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.978+0000 7fc05c9e5640 10 monclient: monmap: 2024-06-11T19:26:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:44.979128+0000 2024-06-11T19:26:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:44.979128+0000 2024-06-11T19:26:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:44.979 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:44.979 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:44.979 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:44.979 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:44.979 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:44.979 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:44.979 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.978+0000 7fc05c9e5640 1 Processor -- start 2024-06-11T19:26:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.978+0000 7fc05c9e5640 1 -- start start 2024-06-11T19:26:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.978+0000 7fc05c9e5640 10 monclient: init 2024-06-11T19:26:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 10 monclient: _add_conns ranks=[1,10,3,2,7,9,8,5,4,0,6] 2024-06-11T19:26:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc0580a2f40 addr v1:172.21.15.117:6789/0 2024-06-11T19:26:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc0580a4fc0 0x7fc05813b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 10 monclient(hunting): picked mon.noname-f con 0x7fc0580a4fc0 addr v2:172.21.15.117:3300/0 2024-06-11T19:26:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0580a53d0 0x7fc05813df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 10 monclient(hunting): picked mon.noname-g con 0x7fc0580a53d0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0580a0ed0 con 0x7fc0580a2f40 2024-06-11T19:26:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fc0580a3320 con 0x7fc0580a4fc0 2024-06-11T19:26:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fc0580a2460 con 0x7fc0580a53d0 2024-06-11T19:26:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc05c9e5640 10 monclient(hunting): authenticate will time out at 1814.786471s 2024-06-11T19:26:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc056d9d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0580a53d0 0x7fc05813df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc055d9b640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc0580a4fc0 0x7fc05813b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc056d9d640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0580a53d0 0x7fc05813df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.98:49236/0 (socket says 172.21.15.98:49236) 2024-06-11T19:26:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc056d9d640 1 -- 172.21.15.98:0/3325200312 learned_addr learned my addr 172.21.15.98:0/3325200312 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc056d9d640 10 monclient(hunting): get_auth_request con 0x7fc0580a53d0 auth_method 0 2024-06-11T19:26:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc056d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc056d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc056d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc055d9b640 10 monclient(hunting): get_auth_request con 0x7fc0580a4fc0 auth_method 0 2024-06-11T19:26:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc055d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc055d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.979+0000 7fc055d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc055d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc055d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc055d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc056d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc056d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc056d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:44.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc055d9b640 10 monclient(hunting): handle_auth_done global_id 124345 payload 274 2024-06-11T19:26:44.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc055d9b640 1 -- 172.21.15.98:0/3325200312 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0580a53d0 msgr2=0x7fc05813df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:44.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc055d9b640 1 --2- 172.21.15.98:0/3325200312 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0580a53d0 0x7fc05813df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:44.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc055d9b640 1 -- 172.21.15.98:0/3325200312 >> v1:172.21.15.117:6789/0 conn(0x7fc0580a2f40 legacy=0x7fc0580a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:44.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc055d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:26:44.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc055d9b640 1 monclient: found mon.noname-f 2024-06-11T19:26:44.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc055d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:44.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc05559a640 1 -- 172.21.15.98:0/3325200312 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2034478186 0 0) 0x7fc0580a0ed0 con 0x7fc0580a2f40 2024-06-11T19:26:44.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc055d9b640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:26:44.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc055d9b640 1 -- 172.21.15.98:0/3325200312 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0580a2180 con 0x7fc0580a4fc0 2024-06-11T19:26:44.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc056d9d640 1 --2- 172.21.15.98:0/3325200312 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0580a53d0 0x7fc05813df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:44.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.980+0000 7fc05559a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:26:44.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc055d9b640 1 --2- 172.21.15.98:0/3325200312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0580a4fc0 0x7fc05813b9f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0097d0 tx=0x7fc04c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d24a7aa8e6505eb server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:44.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 1 -- 172.21.15.98:0/3325200312 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc04c039070 con 0x7fc0580a4fc0 2024-06-11T19:26:44.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:44.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:26:44.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 10 monclient: dump: 2024-06-11T19:26:44.994 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:44.994 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:44.998 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 10 monclient: _finish_auth 0 2024-06-11T19:26:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 10 monclient: _check_auth_tickets 2024-06-11T19:26:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 1 -- 172.21.15.98:0/3325200312 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc04c031960 con 0x7fc0580a4fc0 2024-06-11T19:26:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 1 -- 172.21.15.98:0/3325200312 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc04c032380 con 0x7fc0580a4fc0 2024-06-11T19:26:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:26:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05559a640 10 monclient: dump: 2024-06-11T19:26:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.000 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.000 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.000 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.000 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.000 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.000 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.000 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.000 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.001 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.001 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.001 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.001 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.001 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 5 monclient: authenticate success, global_id 124345 2024-06-11T19:26:45.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:45.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 4 set_mon_vals no callback set 2024-06-11T19:26:45.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 10 monclient: shutdown 2024-06-11T19:26:45.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:45.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 1 -- 172.21.15.98:0/3325200312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0580a4fc0 msgr2=0x7fc05813b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 1 --2- 172.21.15.98:0/3325200312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0580a4fc0 0x7fc05813b9f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0097d0 tx=0x7fc04c02b790 comp rx=0 tx=0).stop 2024-06-11T19:26:45.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 1 -- 172.21.15.98:0/3325200312 shutdown_connections 2024-06-11T19:26:45.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 1 --2- 172.21.15.98:0/3325200312 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0580a53d0 0x7fc05813df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 1 --2- 172.21.15.98:0/3325200312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0580a4fc0 0x7fc05813b9f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 1 -- 172.21.15.98:0/3325200312 >> 172.21.15.98:0/3325200312 conn(0x7fc0580a95b0 msgr2=0x7fc0580a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:45.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 1 -- 172.21.15.98:0/3325200312 shutdown_connections 2024-06-11T19:26:45.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.981+0000 7fc05c9e5640 1 -- 172.21.15.98:0/3325200312 wait complete. 2024-06-11T19:26:45.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.982+0000 7fc05c9e5640 10 monclient: build_initial_monmap 2024-06-11T19:26:45.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.982+0000 7fc05c9e5640 10 monclient: monmap: 2024-06-11T19:26:45.003 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:45.003 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:45.003 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:45.003 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:45.003 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:45.004 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:45.004 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:45.004 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:45.004 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:45.004 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:45.004 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:45.004 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:45.004 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:45.004 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:45.005 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:45.005 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.982+0000 7fc05c9e5640 1 librados: starting msgr at 2024-06-11T19:26:45.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.982+0000 7fc05c9e5640 1 librados: starting objecter 2024-06-11T19:26:45.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.982+0000 7fc05c9e5640 1 Processor -- start 2024-06-11T19:26:45.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.982+0000 7fc05c9e5640 1 -- start start 2024-06-11T19:26:45.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.982+0000 7fc05c9e5640 1 librados: setting wanted keys 2024-06-11T19:26:45.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.982+0000 7fc05c9e5640 1 librados: calling monclient init 2024-06-11T19:26:45.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.982+0000 7fc05c9e5640 10 monclient: init 2024-06-11T19:26:45.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:45.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 10 monclient: _add_conns ranks=[3,6,0,2,7,8,5,4,1] 2024-06-11T19:26:45.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0580a2f40 0x7fc0581cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 10 monclient(hunting): picked mon.noname-g con 0x7fc0580a2f40 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:45.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0580a4fc0 0x7fc0581cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 10 monclient(hunting): picked mon.noname-d con 0x7fc0580a4fc0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:45.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0580a53d0 0x7fc0581cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 10 monclient(hunting): picked mon.noname-a con 0x7fc0580a53d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:26:45.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc05813e600 con 0x7fc0580a53d0 2024-06-11T19:26:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fc05813e480 con 0x7fc0580a2f40 2024-06-11T19:26:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc05813e780 con 0x7fc0580a4fc0 2024-06-11T19:26:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05c9e5640 10 monclient(hunting): authenticate will time out at 1814.790388s 2024-06-11T19:26:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0580a4fc0 0x7fc0581cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc056d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0580a53d0 0x7fc0581cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0580a4fc0 0x7fc0581cbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58840/0 (socket says 172.21.15.98:58840) 2024-06-11T19:26:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 1 -- 172.21.15.98:0/1718258508 learned_addr learned my addr 172.21.15.98:0/1718258508 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 10 monclient(hunting): get_auth_request con 0x7fc0580a4fc0 auth_method 0 2024-06-11T19:26:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc056d9d640 10 monclient(hunting): get_auth_request con 0x7fc0580a53d0 auth_method 0 2024-06-11T19:26:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc056d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc056d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc056d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05659c640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0580a2f40 0x7fc0581cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05659c640 10 monclient(hunting): get_auth_request con 0x7fc0580a2f40 auth_method 0 2024-06-11T19:26:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc056d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc056d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc056d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc05659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 10 monclient(hunting): handle_auth_done global_id 124404 payload 824 2024-06-11T19:26:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 1 -- 172.21.15.98:0/1718258508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0580a2f40 msgr2=0x7fc0581cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0580a2f40 0x7fc0581cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 1 -- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0580a53d0 msgr2=0x7fc0581cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0580a53d0 0x7fc0581cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:26:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 1 monclient: found mon.noname-d 2024-06-11T19:26:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc055d9b640 1 -- 172.21.15.98:0/1718258508 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc05813d0e0 con 0x7fc0580a4fc0 2024-06-11T19:26:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.983+0000 7fc056d9d640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0580a53d0 0x7fc0581cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc055d9b640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0580a4fc0 0x7fc0581cbbd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc04c037040 tx=0x7fc04c0046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 10 client.?.objecter ms_handle_connect 0x7fc0580a4fc0 2024-06-11T19:26:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc04c043030 con 0x7fc0580a4fc0 2024-06-11T19:26:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 10 monclient: dump: 2024-06-11T19:26:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.014 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 10 monclient: _finish_auth 0 2024-06-11T19:26:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:26:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc04c0044b0 con 0x7fc0580a4fc0 2024-06-11T19:26:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc05c9e5640 5 monclient: authenticate success, global_id 124404 2024-06-11T19:26:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc04c039040 con 0x7fc0580a4fc0 2024-06-11T19:26:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc046ffd640 10 monclient: dump: 2024-06-11T19:26:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.016 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.016 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.016 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.016 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.016 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.016 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.016 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.016 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.016 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.016 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.017 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc054d99640 4 set_mon_vals no callback set 2024-06-11T19:26:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc05c9e5640 10 monclient: _renew_subs 2024-06-11T19:26:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc05c9e5640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc05c9e5640 1 -- 172.21.15.98:0/1718258508 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc05809f300 con 0x7fc0580a4fc0 2024-06-11T19:26:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc05c9e5640 10 client.124404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc05c9e5640 10 monclient: _renew_subs 2024-06-11T19:26:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc05c9e5640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc05c9e5640 1 -- 172.21.15.98:0/1718258508 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fc058120360 con 0x7fc0580a4fc0 2024-06-11T19:26:45.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.984+0000 7fc05c9e5640 1 librados: init done 2024-06-11T19:26:45.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05c9e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:45.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05c9e5640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581377a0 0x7fc058137b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05c9e5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05c9e5640 1 -- 172.21.15.98:0/1718258508 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc05813d5d0 con 0x7fc0581377a0 2024-06-11T19:26:45.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05659c640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581377a0 0x7fc058137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05659c640 10 monclient: get_auth_request con 0x7fc0581377a0 auth_method 0 2024-06-11T19:26:45.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fc04c046070 con 0x7fc0580a4fc0 2024-06-11T19:26:45.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05659c640 10 monclient(hunting): handle_auth_done global_id 124413 payload 824 2024-06-11T19:26:45.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05659c640 1 -- 172.21.15.98:0/1718258508 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0580a45b0 con 0x7fc0581377a0 2024-06-11T19:26:45.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc03803fdb0 0x7fc038042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc05659c640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581377a0 0x7fc058137b80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc04800fae0 tx=0x7fc0480020d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 10 client.124404.objecter ms_handle_connect 0x7fc0581377a0 2024-06-11T19:26:45.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 10 client.124404.objecter resend_mon_ops 2024-06-11T19:26:45.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7fc04c044070 con 0x7fc0580a4fc0 2024-06-11T19:26:45.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 10 client.124404.objecter ms_dispatch 0x7fc058000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:45.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 3 client.124404.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:45.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 3 client.124404.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:45.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 20 client.124404.objecter dump_active .. 0 homeless 2024-06-11T19:26:45.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc0480050d0 con 0x7fc0581377a0 2024-06-11T19:26:45.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:45.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc046ffd640 10 monclient: dump: 2024-06-11T19:26:45.021 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.021 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.021 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.021 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.021 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.021 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.021 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.022 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc056d9d640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc03803fdb0 0x7fc038042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.985+0000 7fc056d9d640 10 monclient: get_auth_request con 0x7fc03803fdb0 auth_method 0 2024-06-11T19:26:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.986+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc048018070 con 0x7fc0581377a0 2024-06-11T19:26:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.986+0000 7fc046ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.986+0000 7fc046ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.986+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581377a0 msgr2=0x7fc058137b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.986+0000 7fc046ffd640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581377a0 0x7fc058137b80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc04800fae0 tx=0x7fc0480020d0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.986+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581377a0 msgr2=0x7fc058137b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.986+0000 7fc046ffd640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581377a0 0x7fc058137b80 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc04800fae0 tx=0x7fc0480020d0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.986+0000 7fc046ffd640 10 client.124404.objecter ms_handle_connect 0x7fc03803fdb0 2024-06-11T19:26:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.986+0000 7fc056d9d640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc03803fdb0 0x7fc038042270 secure :-1 s=READY pgs=4324 cs=0 l=1 rev1=1 crypto rx=0x7fc0500096b0 tx=0x7fc050009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05c9e5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:26:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05c9e5640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581380c0 0x7fc05813afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05c9e5640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05c9e5640 1 -- 172.21.15.98:0/1718258508 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc058050830 con 0x7fc0581380c0 2024-06-11T19:26:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05659c640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581380c0 0x7fc05813afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05659c640 10 monclient: get_auth_request con 0x7fc0581380c0 auth_method 0 2024-06-11T19:26:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05659c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05659c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05659c640 10 monclient(hunting): handle_auth_done global_id 124422 payload 824 2024-06-11T19:26:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05659c640 1 -- 172.21.15.98:0/1718258508 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc0580a48b0 con 0x7fc0581380c0 2024-06-11T19:26:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc05659c640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581380c0 0x7fc05813afb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc048001330 tx=0x7fc048001250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc046ffd640 10 client.124404.objecter ms_handle_connect 0x7fc0581380c0 2024-06-11T19:26:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.992+0000 7fc046ffd640 10 client.124404.objecter resend_mon_ops 2024-06-11T19:26:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc048005810 con 0x7fc0581380c0 2024-06-11T19:26:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc046ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc046ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc046ffd640 10 monclient: dump: 2024-06-11T19:26:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.030 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fc0480053a0 con 0x7fc0581380c0 2024-06-11T19:26:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc046ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:26:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc046ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581380c0 msgr2=0x7fc05813afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc046ffd640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581380c0 0x7fc05813afb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc048001330 tx=0x7fc048001250 comp rx=0 tx=0).stop 2024-06-11T19:26:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc046ffd640 1 -- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581380c0 msgr2=0x7fc05813afb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc046ffd640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0581380c0 0x7fc05813afb0 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc048001330 tx=0x7fc048001250 comp rx=0 tx=0).stop 2024-06-11T19:26:45.031 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:45.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:26:45.031 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:26:45.032 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:26:45.032 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 92, 2024-06-11T19:26:45.032 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 147921, 2024-06-11T19:26:45.032 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:26:45.032 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:26:45.032 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:26:45.032 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:26:45.032 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:26:45.032 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:26:45.033 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:26:45.033 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:26:45.033 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:26:45.033 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:26:45.033 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.033 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:26:45.033 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:45.033 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:26:45.033 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:26:45.034 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.034 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.034 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.034 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.034 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.034 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.034 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.034 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.034 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.035 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.035 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.035 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.035 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:26:45.035 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:26:45.035 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.035 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.035 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.036 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.036 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.036 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.036 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.036 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.036 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.036 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.036 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.036 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.037 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:26:45.037 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:26:45.037 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:26:45.037 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:26:45.037 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:26:45.037 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:26:45.037 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:45.038 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:45.038 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:26:45.038 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:26:45.038 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:26:45.038 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:26:45.038 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:26:45.038 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:26:45.038 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:26:45.039 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:45.039 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:26:45.039 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.039 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.039 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.039 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.039 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.039 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.039 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.040 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.040 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.040 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.040 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.040 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.040 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:26:45.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.040 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:26:45.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.041 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:26:45.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:26:45.041 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.041 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:26:45.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:26:45.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.043 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.043 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:26:45.043 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:26:45.043 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.043 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.043 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.044 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:26:45.044 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:26:45.044 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.044 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.044 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:26:45.044 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.045 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:26:45.045 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.045 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.045 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.045 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:26:45.045 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:26:45.046 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.046 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.046 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.046 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:26:45.046 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:26:45.046 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:26:45.047 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:26:45.048 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.048 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.048 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:26:45.048 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:26:45.048 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.048 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.048 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.049 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:26:45.049 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:26:45.049 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:26:45.050 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.050 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:26:45.050 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.050 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.050 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.051 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:26:45.051 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:26:45.051 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.051 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.051 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.052 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:26:45.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:26:45.052 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:26:45.052 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.053 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.053 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:26:45.053 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.053 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.053 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.053 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:26:45.053 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:26:45.054 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.054 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.054 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.054 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:26:45.054 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:26:45.054 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.055 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.055 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:26:45.055 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.055 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:26:45.056 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.056 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.056 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:26:45.056 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:26:45.056 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.056 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.057 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.057 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:26:45.057 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:26:45.057 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.057 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.058 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:26:45.058 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.058 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:26:45.058 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.058 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.059 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.059 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:26:45.059 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:26:45.059 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.059 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.059 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.060 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:26:45.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:26:45.060 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:26:45.060 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:26:45.061 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.061 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.061 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:26:45.061 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:26:45.062 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.062 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.062 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.062 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:26:45.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:26:45.062 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:26:45.063 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:26:45.063 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.064 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.064 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.064 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:26:45.064 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:26:45.064 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.064 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.064 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.064 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.065 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.065 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:26:45.065 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:26:45.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.065 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:45.065 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:45.065 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:45.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.066 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.066 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:26:45.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.066 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:45.066 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:45.066 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-06-11T19:26:45.066 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.066 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.066 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-06-11T19:26:45.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.067 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:45.067 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:45.067 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:45.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.067 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.067 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:26:45.067 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 10 librados: watch_flush enter 2024-06-11T19:26:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 10 librados: watch_flush exit 2024-06-11T19:26:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 20 client.124404.objecter shutdown clearing up homeless session... 2024-06-11T19:26:45.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 10 client.124404.objecter successfully canceled tick 2024-06-11T19:26:45.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 1 -- 172.21.15.98:0/1718258508 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc03803fdb0 msgr2=0x7fc038042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fc03803fdb0 0x7fc038042270 secure :-1 s=READY pgs=4324 cs=0 l=1 rev1=1 crypto rx=0x7fc0500096b0 tx=0x7fc050009240 comp rx=0 tx=0).stop 2024-06-11T19:26:45.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 10 monclient: shutdown 2024-06-11T19:26:45.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:45.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 1 -- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0580a4fc0 msgr2=0x7fc0581cbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0580a4fc0 0x7fc0581cbbd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc04c037040 tx=0x7fc04c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc056d9d640 1 -- 172.21.15.98:0/1718258508 reap_dead start 2024-06-11T19:26:45.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 1 -- 172.21.15.98:0/1718258508 shutdown_connections 2024-06-11T19:26:45.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 1 --2- 172.21.15.98:0/1718258508 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc0580a4fc0 0x7fc0581cbbd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 1 -- 172.21.15.98:0/1718258508 >> 172.21.15.98:0/1718258508 conn(0x7fc0580a95b0 msgr2=0x7fc0580725f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:45.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 1 -- 172.21.15.98:0/1718258508 shutdown_connections 2024-06-11T19:26:45.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 1 -- 172.21.15.98:0/1718258508 wait complete. 2024-06-11T19:26:45.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:44.993+0000 7fc05c9e5640 1 librados: shutdown 2024-06-11T19:26:45.071 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:26:45.072 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-06-11T19:26:45.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-06-11T19:26:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.172+0000 7fa7d1984640 10 monclient: get_monmap_and_config 2024-06-11T19:26:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.172+0000 7fa7d1984640 10 monclient: build_initial_monmap 2024-06-11T19:26:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.172+0000 7fa7d1984640 10 monclient: monmap: 2024-06-11T19:26:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:45.171 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:45.173186+0000 2024-06-11T19:26:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:45.173186+0000 2024-06-11T19:26:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:45.172 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:45.173 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:45.173 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.172+0000 7fa7d1984640 1 Processor -- start 2024-06-11T19:26:45.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.172+0000 7fa7d1984640 1 -- start start 2024-06-11T19:26:45.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.172+0000 7fa7d1984640 10 monclient: init 2024-06-11T19:26:45.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:45.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 10 monclient: _add_conns ranks=[3,10,0,2,5,8,9,7,6,1,4] 2024-06-11T19:26:45.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7cc0a2f40 0x7fa7cc0a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 10 monclient(hunting): picked mon.noname-g con 0x7fa7cc0a2f40 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:45.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa7cc0a4df0 0x7fa7cc0a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 10 monclient(hunting): picked mon.noname-f con 0x7fa7cc0a4df0 addr v2:172.21.15.117:3300/0 2024-06-11T19:26:45.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7cc1366d0 addr v1:172.21.15.98:6789/0 2024-06-11T19:26:45.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7cc0a0ed0 con 0x7fa7cc1366d0 2024-06-11T19:26:45.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fa7cc0a5710 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fa7cc0a3320 con 0x7fa7cc0a2f40 2024-06-11T19:26:45.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:45.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7d1984640 10 monclient(hunting): authenticate will time out at 1814.980297s 2024-06-11T19:26:45.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7cb7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa7cc1366d0 0x7fa7cc136ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51366/0 (socket says 172.21.15.98:51366) 2024-06-11T19:26:45.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7cb7fe640 1 -- 172.21.15.98:0/2918006665 learned_addr learned my addr 172.21.15.98:0/2918006665 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:26:45.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 1 --2- 172.21.15.98:0/2918006665 >> v2:172.21.15.117:3300/0 conn(0x7fa7cc0a4df0 0x7fa7cc0a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7caffd640 1 --2- 172.21.15.98:0/2918006665 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7cc0a2f40 0x7fa7cc0a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 10 monclient(hunting): get_auth_request con 0x7fa7cc0a4df0 auth_method 0 2024-06-11T19:26:45.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7caffd640 10 monclient(hunting): get_auth_request con 0x7fa7cc0a2f40 auth_method 0 2024-06-11T19:26:45.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7caffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7caffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7c9ffb640 1 -- 172.21.15.98:0/2918006665 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2034489136 0 0) 0x7fa7cc0a0ed0 con 0x7fa7cc1366d0 2024-06-11T19:26:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7c9ffb640 10 monclient(hunting): _negotiate 2024-06-11T19:26:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7c9ffb640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7c9ffb640 10 monclient(hunting): my global_id is 66357 2024-06-11T19:26:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7c9ffb640 1 -- 172.21.15.98:0/2918006665 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7b4003520 con 0x7fa7cc1366d0 2024-06-11T19:26:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 10 monclient(hunting): handle_auth_done global_id 124363 payload 274 2024-06-11T19:26:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7c9ffb640 1 -- 172.21.15.98:0/2918006665 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 690309448 0 0) 0x7fa7b4003520 con 0x7fa7cc1366d0 2024-06-11T19:26:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 1 -- 172.21.15.98:0/2918006665 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7cc0a2f40 msgr2=0x7fa7cc0a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 1 --2- 172.21.15.98:0/2918006665 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7cc0a2f40 0x7fa7cc0a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 1 -- 172.21.15.98:0/2918006665 >> v1:172.21.15.98:6789/0 conn(0x7fa7cc1366d0 legacy=0x7fa7cc136ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:26:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 1 monclient: found mon.noname-f 2024-06-11T19:26:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:26:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7ca7fc640 1 -- 172.21.15.98:0/2918006665 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7cc0a2180 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.173+0000 7fa7c9ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:26:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7ca7fc640 1 --2- 172.21.15.98:0/2918006665 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7cc0a4df0 0x7fa7cc0a51d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa7c00097a0 tx=0x7fa7c002b660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9c84b64127cb240 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 1 -- 172.21.15.98:0/2918006665 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7c0037070 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:26:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 10 monclient: dump: 2024-06-11T19:26:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.190 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 10 monclient: _finish_auth 0 2024-06-11T19:26:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:26:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 1 -- 172.21.15.98:0/2918006665 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c0030980 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7d1984640 5 monclient: authenticate success, global_id 124363 2024-06-11T19:26:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7d1984640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 1 -- 172.21.15.98:0/2918006665 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7c002be60 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:26:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7c9ffb640 10 monclient: dump: 2024-06-11T19:26:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.193 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7d1984640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.174+0000 7fa7d1984640 4 set_mon_vals no callback set 2024-06-11T19:26:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.175+0000 7fa7d1984640 10 monclient: shutdown 2024-06-11T19:26:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.175+0000 7fa7d1984640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.175+0000 7fa7d1984640 1 -- 172.21.15.98:0/2918006665 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7cc0a4df0 msgr2=0x7fa7cc0a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.175+0000 7fa7d1984640 1 --2- 172.21.15.98:0/2918006665 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7cc0a4df0 0x7fa7cc0a51d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa7c00097a0 tx=0x7fa7c002b660 comp rx=0 tx=0).stop 2024-06-11T19:26:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.175+0000 7fa7d1984640 1 -- 172.21.15.98:0/2918006665 shutdown_connections 2024-06-11T19:26:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.175+0000 7fa7d1984640 1 --2- 172.21.15.98:0/2918006665 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7cc0a4df0 0x7fa7cc0a51d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.175+0000 7fa7d1984640 1 --2- 172.21.15.98:0/2918006665 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7cc0a2f40 0x7fa7cc0a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.175+0000 7fa7d1984640 1 -- 172.21.15.98:0/2918006665 >> 172.21.15.98:0/2918006665 conn(0x7fa7cc0a95b0 msgr2=0x7fa7cc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.175+0000 7fa7d1984640 1 -- 172.21.15.98:0/2918006665 shutdown_connections 2024-06-11T19:26:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.175+0000 7fa7d1984640 1 -- 172.21.15.98:0/2918006665 wait complete. 2024-06-11T19:26:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.176+0000 7fa7d1984640 10 monclient: build_initial_monmap 2024-06-11T19:26:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.176+0000 7fa7d1984640 10 monclient: monmap: 2024-06-11T19:26:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:45.197 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.176+0000 7fa7d1984640 1 librados: starting msgr at 2024-06-11T19:26:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.176+0000 7fa7d1984640 1 librados: starting objecter 2024-06-11T19:26:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.176+0000 7fa7d1984640 1 Processor -- start 2024-06-11T19:26:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.176+0000 7fa7d1984640 1 -- start start 2024-06-11T19:26:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.176+0000 7fa7d1984640 1 librados: setting wanted keys 2024-06-11T19:26:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.176+0000 7fa7d1984640 1 librados: calling monclient init 2024-06-11T19:26:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.176+0000 7fa7d1984640 10 monclient: init 2024-06-11T19:26:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient: _add_conns ranks=[4,6,7,1,2,3,5,8,0] 2024-06-11T19:26:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc0a2f40 0x7fa7cc1cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient(hunting): picked mon.noname-c con 0x7fa7cc0a2f40 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7cc0a4df0 0x7fa7cc1cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient(hunting): picked mon.noname-d con 0x7fa7cc0a4df0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7cc1366d0 0x7fa7cc06bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient(hunting): picked mon.noname-i con 0x7fa7cc1366d0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa7cc1371f0 con 0x7fa7cc0a2f40 2024-06-11T19:26:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa7cc0a2460 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fa7cc137370 con 0x7fa7cc1366d0 2024-06-11T19:26:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient(hunting): authenticate will time out at 1814.984133s 2024-06-11T19:26:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7caffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc0a2f40 0x7fa7cc1cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7cc0a4df0 0x7fa7cc1cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7caffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc0a2f40 0x7fa7cc1cd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:37200/0 (socket says 172.21.15.98:37200) 2024-06-11T19:26:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7caffd640 1 -- 172.21.15.98:0/4151035383 learned_addr learned my addr 172.21.15.98:0/4151035383 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 10 monclient(hunting): get_auth_request con 0x7fa7cc0a4df0 auth_method 0 2024-06-11T19:26:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7caffd640 10 monclient(hunting): get_auth_request con 0x7fa7cc0a2f40 auth_method 0 2024-06-11T19:26:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7caffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7caffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7cb7fe640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7cc1366d0 0x7fa7cc06bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7cb7fe640 10 monclient(hunting): get_auth_request con 0x7fa7cc1366d0 auth_method 0 2024-06-11T19:26:45.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7cb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 10 monclient(hunting): handle_auth_done global_id 124431 payload 824 2024-06-11T19:26:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 1 -- 172.21.15.98:0/4151035383 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7cc1366d0 msgr2=0x7fa7cc06bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fa7cc1366d0 0x7fa7cc06bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 1 -- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc0a2f40 msgr2=0x7fa7cc1cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc0a2f40 0x7fa7cc1cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 10 monclient: _finish_hunting 0 2024-06-11T19:26:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 1 monclient: found mon.noname-d 2024-06-11T19:26:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 1 -- 172.21.15.98:0/4151035383 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7cc0a7cb0 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7caffd640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7cc0a2f40 0x7fa7cc1cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7ca7fc640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7cc0a4df0 0x7fa7cc1cdd60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0032000 tx=0x7fa7c00046c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 10 client.?.objecter ms_handle_connect 0x7fa7cc0a4df0 2024-06-11T19:26:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7c0037050 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 10 monclient: dump: 2024-06-11T19:26:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.206 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 10 monclient: _finish_auth 0 2024-06-11T19:26:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 10 monclient: _check_auth_tickets 2024-06-11T19:26:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 5 monclient: authenticate success, global_id 124431 2024-06-11T19:26:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c0038040 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient: _renew_subs 2024-06-11T19:26:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 1 -- 172.21.15.98:0/4151035383 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7cc09f550 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7c4001960 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 client.124431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7b3fff640 10 monclient: dump: 2024-06-11T19:26:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.209 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient: _renew_subs 2024-06-11T19:26:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7c97fa640 4 set_mon_vals no callback set 2024-06-11T19:26:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 1 -- 172.21.15.98:0/4151035383 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7cc126c00 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.177+0000 7fa7d1984640 1 librados: init done 2024-06-11T19:26:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.178+0000 7fa7d1984640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.178+0000 7fa7d1984640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06c4a0 0x7fa7cc06c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.178+0000 7fa7d1984640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.178+0000 7fa7d1984640 1 -- 172.21.15.98:0/4151035383 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa7cc050830 con 0x7fa7cc06c4a0 2024-06-11T19:26:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.178+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fa7c0045030 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.178+0000 7fa7caffd640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06c4a0 0x7fa7cc06c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.178+0000 7fa7caffd640 10 monclient: get_auth_request con 0x7fa7cc06c4a0 auth_method 0 2024-06-11T19:26:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.178+0000 7fa7caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.178+0000 7fa7caffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.178+0000 7fa7caffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa7b803fde0 0x7fa7b80422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7caffd640 10 monclient(hunting): handle_auth_done global_id 124406 payload 824 2024-06-11T19:26:45.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7caffd640 1 -- 172.21.15.98:0/4151035383 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7cc126640 con 0x7fa7cc06c4a0 2024-06-11T19:26:45.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7fa7c0043020 con 0x7fa7cc0a4df0 2024-06-11T19:26:45.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 10 client.124431.objecter ms_dispatch 0x7fa7cc000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:45.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 3 client.124431.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:45.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 3 client.124431.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:45.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 20 client.124431.objecter dump_active .. 0 homeless 2024-06-11T19:26:45.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7cb7fe640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa7b803fde0 0x7fa7b80422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 10 client.124431.objecter ms_handle_connect 0x7fa7cc06c4a0 2024-06-11T19:26:45.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 10 client.124431.objecter resend_mon_ops 2024-06-11T19:26:45.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7caffd640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06c4a0 0x7fa7cc06c880 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0075b0 tx=0x7fa7bc007a70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7cb7fe640 10 monclient: get_auth_request con 0x7fa7b803fde0 auth_method 0 2024-06-11T19:26:45.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7bc01b070 con 0x7fa7cc06c4a0 2024-06-11T19:26:45.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:45.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 10 monclient: dump: 2024-06-11T19:26:45.213 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.213 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.213 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.214 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.214 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.214 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.214 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.214 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.214 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.214 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.214 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.214 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.214 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.215 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.215 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.215 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 10 client.124431.objecter ms_handle_connect 0x7fa7b803fde0 2024-06-11T19:26:45.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7cb7fe640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa7b803fde0 0x7fa7b80422a0 secure :-1 s=READY pgs=4325 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4001530 tx=0x7fa7c40014c0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa7bc011790 con 0x7fa7cc06c4a0 2024-06-11T19:26:45.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:45.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:45.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06c4a0 msgr2=0x7fa7cc06c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06c4a0 0x7fa7cc06c880 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0075b0 tx=0x7fa7bc007a70 comp rx=0 tx=0).stop 2024-06-11T19:26:45.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06c4a0 msgr2=0x7fa7cc06c880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:45.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.179+0000 7fa7b3fff640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06c4a0 0x7fa7cc06c880 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7d1984640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:26:45.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7d1984640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06cdc0 0x7fa7cc0705e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7d1984640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7d1984640 1 -- 172.21.15.98:0/4151035383 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa7cc0a48b0 con 0x7fa7cc06cdc0 2024-06-11T19:26:45.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7caffd640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06cdc0 0x7fa7cc0705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7caffd640 10 monclient: get_auth_request con 0x7fa7cc06cdc0 auth_method 0 2024-06-11T19:26:45.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7caffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7caffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7caffd640 10 monclient(hunting): handle_auth_done global_id 124415 payload 824 2024-06-11T19:26:45.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7caffd640 1 -- 172.21.15.98:0/4151035383 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7cc135e50 con 0x7fa7cc06cdc0 2024-06-11T19:26:45.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 10 client.124431.objecter ms_handle_connect 0x7fa7cc06cdc0 2024-06-11T19:26:45.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 10 client.124431.objecter resend_mon_ops 2024-06-11T19:26:45.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7caffd640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06cdc0 0x7fa7cc0705e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000f80 tx=0x7fa7bc004ab0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7bc017ac0 con 0x7fa7cc06cdc0 2024-06-11T19:26:45.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:45.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 10 monclient: dump: 2024-06-11T19:26:45.220 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.220 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.220 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.220 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.220 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.220 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.220 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.220 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.220 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.221 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.221 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.221 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.221 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.221 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.221 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.221 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fa7bc017650 con 0x7fa7cc06cdc0 2024-06-11T19:26:45.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:26:45.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:45.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06cdc0 msgr2=0x7fa7cc0705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06cdc0 0x7fa7cc0705e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000f80 tx=0x7fa7bc004ab0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 1 -- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06cdc0 msgr2=0x7fa7cc0705e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:45.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.186+0000 7fa7b3fff640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7cc06cdc0 0x7fa7cc0705e0 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000f80 tx=0x7fa7bc004ab0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.222 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:45.222 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:26:45.223 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:26:45.223 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:26:45.223 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 92, 2024-06-11T19:26:45.223 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 148123, 2024-06-11T19:26:45.223 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:26:45.223 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:26:45.223 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:26:45.223 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:26:45.223 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:26:45.224 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:26:45.224 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:26:45.224 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:26:45.224 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:26:45.224 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:26:45.224 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.224 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:26:45.224 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:45.224 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:26:45.225 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:26:45.225 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.225 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.225 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.225 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.225 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.225 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.225 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.225 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.226 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.226 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.226 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.226 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.226 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:26:45.226 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:26:45.226 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.226 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.226 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.227 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.227 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.227 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.227 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.227 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.227 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.227 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.227 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.228 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:26:45.228 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:26:45.228 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:26:45.228 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:26:45.228 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:26:45.228 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:26:45.228 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:45.228 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:45.228 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:26:45.229 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:26:45.229 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:26:45.229 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:26:45.229 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:26:45.229 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:26:45.229 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:26:45.229 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:45.229 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:26:45.229 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.230 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.230 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.230 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.230 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.230 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.230 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.230 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.230 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.231 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.231 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.231 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:26:45.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.231 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:26:45.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:26:45.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:26:45.231 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:26:45.232 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:26:45.232 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.233 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:26:45.233 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:26:45.233 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.233 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.233 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:26:45.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:26:45.234 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.234 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:26:45.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:26:45.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.235 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.235 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:26:45.236 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:26:45.236 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.236 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.236 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:26:45.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:26:45.237 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:26:45.237 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.238 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:26:45.238 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.238 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.238 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.238 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:26:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:26:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.239 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:26:45.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:26:45.240 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:26:45.240 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:26:45.241 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.241 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.241 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:26:45.242 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:26:45.242 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.242 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.242 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:26:45.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:26:45.243 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:26:45.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:26:45.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:26:45.244 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:26:45.244 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.244 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.245 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:26:45.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:26:45.245 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:26:45.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:26:45.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.246 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.247 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:26:45.247 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:26:45.247 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.247 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.247 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:26:45.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:26:45.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:26:45.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:26:45.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:26:45.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:26:45.250 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.250 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.250 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:26:45.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:26:45.251 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:26:45.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:26:45.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.252 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:26:45.252 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:26:45.252 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.252 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.253 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:26:45.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:26:45.253 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:26:45.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:26:45.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.254 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:26:45.255 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:26:45.255 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.255 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.255 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.255 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.255 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.256 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:26:45.256 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:26:45.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.256 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:45.256 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:45.256 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:45.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.256 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.256 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:26:45.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.257 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:45.257 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:45.257 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:45.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.257 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.257 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:26:45.257 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:45.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 10 librados: watch_flush enter 2024-06-11T19:26:45.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 10 librados: watch_flush exit 2024-06-11T19:26:45.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 20 client.124431.objecter shutdown clearing up homeless session... 2024-06-11T19:26:45.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 10 client.124431.objecter successfully canceled tick 2024-06-11T19:26:45.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 1 -- 172.21.15.98:0/4151035383 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa7b803fde0 msgr2=0x7fa7b80422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fa7b803fde0 0x7fa7b80422a0 secure :-1 s=READY pgs=4325 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4001530 tx=0x7fa7c40014c0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 10 monclient: shutdown 2024-06-11T19:26:45.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:45.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 1 -- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7cc0a4df0 msgr2=0x7fa7cc1cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7cb7fe640 1 -- 172.21.15.98:0/4151035383 reap_dead start 2024-06-11T19:26:45.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7cc0a4df0 0x7fa7cc1cdd60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0032000 tx=0x7fa7c00046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 1 -- 172.21.15.98:0/4151035383 shutdown_connections 2024-06-11T19:26:45.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 1 --2- 172.21.15.98:0/4151035383 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7cc0a4df0 0x7fa7cc1cdd60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 1 -- 172.21.15.98:0/4151035383 >> 172.21.15.98:0/4151035383 conn(0x7fa7cc0a95b0 msgr2=0x7fa7cc0a0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:45.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 1 -- 172.21.15.98:0/4151035383 shutdown_connections 2024-06-11T19:26:45.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 1 -- 172.21.15.98:0/4151035383 wait complete. 2024-06-11T19:26:45.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.187+0000 7fa7d1984640 1 librados: shutdown 2024-06-11T19:26:45.261 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:26:45.261 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-06-11T19:26:45.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-06-11T19:26:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.362+0000 7fb20d99c640 10 monclient: get_monmap_and_config 2024-06-11T19:26:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.362+0000 7fb20d99c640 10 monclient: build_initial_monmap 2024-06-11T19:26:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.362+0000 7fb20d99c640 10 monclient: monmap: 2024-06-11T19:26:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:45.363160+0000 2024-06-11T19:26:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:45.363160+0000 2024-06-11T19:26:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:45.363 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:45.363 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:45.363 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:45.363 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:45.363 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:45.363 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:45.363 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.362+0000 7fb20d99c640 1 Processor -- start 2024-06-11T19:26:45.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.362+0000 7fb20d99c640 1 -- start start 2024-06-11T19:26:45.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.362+0000 7fb20d99c640 10 monclient: init 2024-06-11T19:26:45.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:45.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 10 monclient: _add_conns ranks=[0,8,3,1,7,2,4,10,6,5,9] 2024-06-11T19:26:45.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2080a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:26:45.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2080a2530 0x7fb20813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 10 monclient(hunting): picked mon.noname-e con 0x7fb2080a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:45.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb20813e1a0 0x7fb208140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 10 monclient(hunting): picked mon.noname-g con 0x7fb20813e1a0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:45.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2080a7140 con 0x7fb2080a4920 2024-06-11T19:26:45.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fb2080a2b50 con 0x7fb20813e1a0 2024-06-11T19:26:45.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb2080a2e30 con 0x7fb2080a2530 2024-06-11T19:26:45.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:45.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb20d99c640 10 monclient(hunting): authenticate will time out at 1815.170239s 2024-06-11T19:26:45.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2080a2530 0x7fb20813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb206ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb2080a4920 0x7fb2080a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51376/0 (socket says 172.21.15.98:51376) 2024-06-11T19:26:45.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb206ffd640 1 -- 172.21.15.98:0/608402459 learned_addr learned my addr 172.21.15.98:0/608402459 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:26:45.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 10 monclient(hunting): get_auth_request con 0x7fb2080a2530 auth_method 0 2024-06-11T19:26:45.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 10 monclient(hunting): handle_auth_done global_id 124424 payload 274 2024-06-11T19:26:45.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 1 -- 172.21.15.98:0/608402459 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb20813e1a0 msgr2=0x7fb208140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:45.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 1 --2- 172.21.15.98:0/608402459 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb20813e1a0 0x7fb208140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 1 -- 172.21.15.98:0/608402459 >> v1:172.21.15.98:6789/0 conn(0x7fb2080a4920 legacy=0x7fb2080a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 10 monclient: _finish_hunting 0 2024-06-11T19:26:45.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb205ffb640 1 -- 172.21.15.98:0/608402459 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2036111504 0 0) 0x7fb2080a7140 con 0x7fb2080a4920 2024-06-11T19:26:45.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 1 monclient: found mon.noname-e 2024-06-11T19:26:45.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:45.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:26:45.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb2067fc640 1 -- 172.21.15.98:0/608402459 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2080a3110 con 0x7fb2080a2530 2024-06-11T19:26:45.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.363+0000 7fb205ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:26:45.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb2067fc640 1 --2- 172.21.15.98:0/608402459 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2080a2530 0x7fb20813dc60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009dc0 tx=0x7fb1fc02bd50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=415d0ecb03797153 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 1 -- 172.21.15.98:0/608402459 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1fc039070 con 0x7fb2080a2530 2024-06-11T19:26:45.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:26:45.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 10 monclient: dump: 2024-06-11T19:26:45.378 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.378 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.378 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.378 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.379 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.379 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.379 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.379 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.379 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.379 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.379 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.379 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.379 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.379 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.380 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.380 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 10 monclient: _finish_auth 0 2024-06-11T19:26:45.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:26:45.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:45.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 1 -- 172.21.15.98:0/608402459 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1fc02cd70 con 0x7fb2080a2530 2024-06-11T19:26:45.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb20d99c640 5 monclient: authenticate success, global_id 124424 2024-06-11T19:26:45.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb20d99c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:45.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:45.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb20d99c640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:45.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 1 -- 172.21.15.98:0/608402459 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1fc0327b0 con 0x7fb2080a2530 2024-06-11T19:26:45.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb20d99c640 4 set_mon_vals no callback set 2024-06-11T19:26:45.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:45.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.364+0000 7fb205ffb640 10 monclient: dump: 2024-06-11T19:26:45.381 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.381 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.381 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.382 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.382 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.382 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.382 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.382 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.382 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.382 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.382 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.383 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.383 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.383 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.383 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.383 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 10 monclient: shutdown 2024-06-11T19:26:45.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:45.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 1 -- 172.21.15.98:0/608402459 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2080a2530 msgr2=0x7fb20813dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 1 --2- 172.21.15.98:0/608402459 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2080a2530 0x7fb20813dc60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009dc0 tx=0x7fb1fc02bd50 comp rx=0 tx=0).stop 2024-06-11T19:26:45.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 1 -- 172.21.15.98:0/608402459 shutdown_connections 2024-06-11T19:26:45.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 1 --2- 172.21.15.98:0/608402459 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb20813e1a0 0x7fb208140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 1 --2- 172.21.15.98:0/608402459 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb2080a2530 0x7fb20813dc60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 1 -- 172.21.15.98:0/608402459 >> 172.21.15.98:0/608402459 conn(0x7fb2080ab850 msgr2=0x7fb2080a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:45.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 1 -- 172.21.15.98:0/608402459 shutdown_connections 2024-06-11T19:26:45.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 1 -- 172.21.15.98:0/608402459 wait complete. 2024-06-11T19:26:45.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 10 monclient: build_initial_monmap 2024-06-11T19:26:45.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 10 monclient: monmap: 2024-06-11T19:26:45.385 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:45.385 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:45.385 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:45.385 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:45.385 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:45.385 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:45.385 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:45.385 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:45.386 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:45.386 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:45.386 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:45.386 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:45.386 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:45.386 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:45.386 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:45.386 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 1 librados: starting msgr at 2024-06-11T19:26:45.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.365+0000 7fb20d99c640 1 librados: starting objecter 2024-06-11T19:26:45.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 1 Processor -- start 2024-06-11T19:26:45.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 1 -- start start 2024-06-11T19:26:45.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 1 librados: setting wanted keys 2024-06-11T19:26:45.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 1 librados: calling monclient init 2024-06-11T19:26:45.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 10 monclient: init 2024-06-11T19:26:45.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:45.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 10 monclient: _add_conns ranks=[0,4,7,3,6,2,8,5,1] 2024-06-11T19:26:45.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2080a2530 0x7fb2081cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 10 monclient(hunting): picked mon.noname-a con 0x7fb2080a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:26:45.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb2080a4920 0x7fb208136b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 10 monclient(hunting): picked mon.noname-c con 0x7fb2080a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:45.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb20813e1a0 0x7fb2081370d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 10 monclient(hunting): picked mon.noname-i con 0x7fb20813e1a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:45.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb208138e50 con 0x7fb2080a2530 2024-06-11T19:26:45.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb2080a84b0 con 0x7fb2080a4920 2024-06-11T19:26:45.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fb2080a6da0 con 0x7fb20813e1a0 2024-06-11T19:26:45.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:45.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb20d99c640 10 monclient(hunting): authenticate will time out at 1815.173781s 2024-06-11T19:26:45.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb206ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2080a2530 0x7fb2081cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb206ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2080a2530 0x7fb2081cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41778/0 (socket says 172.21.15.98:41778) 2024-06-11T19:26:45.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb2067fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb2080a4920 0x7fb208136b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb206ffd640 1 -- 172.21.15.98:0/550824389 learned_addr learned my addr 172.21.15.98:0/550824389 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:45.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb206ffd640 10 monclient(hunting): get_auth_request con 0x7fb2080a2530 auth_method 0 2024-06-11T19:26:45.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb206ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb206ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb2067fc640 10 monclient(hunting): get_auth_request con 0x7fb2080a4920 auth_method 0 2024-06-11T19:26:45.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb2067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb2067fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb2067fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb2067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb2067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb2067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb206ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb206ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.366+0000 7fb206ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb2077fe640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb20813e1a0 0x7fb2081370d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb2077fe640 10 monclient(hunting): get_auth_request con 0x7fb20813e1a0 auth_method 0 2024-06-11T19:26:45.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb2077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb2077fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb2077fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb2077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb2077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb2077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb206ffd640 10 monclient(hunting): handle_auth_done global_id 66375 payload 824 2024-06-11T19:26:45.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb206ffd640 1 -- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb20813e1a0 msgr2=0x7fb2081370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb206ffd640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb20813e1a0 0x7fb2081370d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb206ffd640 1 -- 172.21.15.98:0/550824389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb2080a4920 msgr2=0x7fb208136b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb206ffd640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb2080a4920 0x7fb208136b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb206ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:26:45.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb206ffd640 1 monclient: found mon.noname-a 2024-06-11T19:26:45.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb206ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:45.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb206ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:26:45.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb206ffd640 1 -- 172.21.15.98:0/550824389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb208123260 con 0x7fb2080a2530 2024-06-11T19:26:45.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb2067fc640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb2080a4920 0x7fb208136b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:45.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb206ffd640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2080a2530 0x7fb2081cd6b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb20000c450 tx=0x7fb20000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 10 client.?.objecter ms_handle_connect 0x7fb2080a2530 2024-06-11T19:26:45.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:45.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb200018070 con 0x7fb2080a2530 2024-06-11T19:26:45.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:26:45.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 10 monclient: dump: 2024-06-11T19:26:45.394 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.394 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.395 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.395 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.395 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.395 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.395 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.395 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.395 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.395 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.395 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.396 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.396 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.396 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.396 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.396 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 10 monclient: _finish_auth 0 2024-06-11T19:26:45.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 10 monclient: _check_auth_tickets 2024-06-11T19:26:45.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:45.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb20d99c640 5 monclient: authenticate success, global_id 66375 2024-06-11T19:26:45.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2000047e0 con 0x7fb2080a2530 2024-06-11T19:26:45.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:45.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb200004e40 con 0x7fb2080a2530 2024-06-11T19:26:45.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:45.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb2057fa640 4 set_mon_vals no callback set 2024-06-11T19:26:45.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb1effff640 10 monclient: dump: 2024-06-11T19:26:45.397 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.397 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.398 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.398 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.398 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.398 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.398 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.398 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.398 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.398 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.398 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.399 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.399 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.399 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.399 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.399 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb20d99c640 10 monclient: _renew_subs 2024-06-11T19:26:45.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb20d99c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:45.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb20d99c640 1 -- 172.21.15.98:0/550824389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb208121db0 con 0x7fb2080a2530 2024-06-11T19:26:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb20d99c640 10 client.66375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb20d99c640 10 monclient: _renew_subs 2024-06-11T19:26:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb20d99c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb20d99c640 1 -- 172.21.15.98:0/550824389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb20813ddf0 con 0x7fb2080a2530 2024-06-11T19:26:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.367+0000 7fb20d99c640 1 librados: init done 2024-06-11T19:26:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb20d99c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb20d99c640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb20809fb20 0x7fb20809ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb20d99c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb20d99c640 1 -- 172.21.15.98:0/550824389 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb208074fc0 con 0x7fb20809fb20 2024-06-11T19:26:45.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fb2000049d0 con 0x7fb2080a2530 2024-06-11T19:26:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb2067fc640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb20809fb20 0x7fb20809ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb2067fc640 10 monclient: get_auth_request con 0x7fb20809fb20 auth_method 0 2024-06-11T19:26:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb2067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb2067fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb2067fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb1effff640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb1f403fdb0 0x7fb1f4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7fb200050a80 con 0x7fb2080a2530 2024-06-11T19:26:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb1effff640 10 client.66375.objecter ms_dispatch 0x7fb208000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb1effff640 3 client.66375.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb1effff640 3 client.66375.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb2067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb2067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb2067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb1effff640 20 client.66375.objecter dump_active .. 0 homeless 2024-06-11T19:26:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb2077fe640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb1f403fdb0 0x7fb1f4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb2067fc640 10 monclient(hunting): handle_auth_done global_id 124372 payload 824 2024-06-11T19:26:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb2067fc640 1 -- 172.21.15.98:0/550824389 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2080a4d00 con 0x7fb20809fb20 2024-06-11T19:26:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.368+0000 7fb2077fe640 10 monclient: get_auth_request con 0x7fb1f403fdb0 auth_method 0 2024-06-11T19:26:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 10 client.66375.objecter ms_handle_connect 0x7fb20809fb20 2024-06-11T19:26:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb2067fc640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb20809fb20 0x7fb20809ff00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb208073de0 tx=0x7fb1fc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 10 client.66375.objecter resend_mon_ops 2024-06-11T19:26:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1fc044070 con 0x7fb20809fb20 2024-06-11T19:26:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:26:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 10 monclient: dump: 2024-06-11T19:26:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.405 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.405 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.405 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.405 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.405 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.405 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 10 client.66375.objecter ms_handle_connect 0x7fb1f403fdb0 2024-06-11T19:26:45.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb2077fe640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb1f403fdb0 0x7fb1f4042270 secure :-1 s=READY pgs=4326 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8006a00 tx=0x7fb1f8006990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb1fc004370 con 0x7fb20809fb20 2024-06-11T19:26:45.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:45.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:45.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb20809fb20 msgr2=0x7fb20809ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb20809fb20 0x7fb20809ff00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb208073de0 tx=0x7fb1fc004a40 comp rx=0 tx=0).stop 2024-06-11T19:26:45.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb20809fb20 msgr2=0x7fb20809ff00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:45.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.369+0000 7fb1effff640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb20809fb20 0x7fb20809ff00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.375+0000 7fb20d99c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:26:45.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.375+0000 7fb20d99c640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2080a0440 0x7fb20813bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb20d99c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb20d99c640 1 -- 172.21.15.98:0/550824389 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb208050830 con 0x7fb2080a0440 2024-06-11T19:26:45.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb2067fc640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2080a0440 0x7fb20813bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb2067fc640 10 monclient: get_auth_request con 0x7fb2080a0440 auth_method 0 2024-06-11T19:26:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb2067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb2067fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb2067fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb2067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb2067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb2067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb2067fc640 10 monclient(hunting): handle_auth_done global_id 124381 payload 824 2024-06-11T19:26:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb2067fc640 1 -- 172.21.15.98:0/550824389 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb208121b00 con 0x7fb2080a0440 2024-06-11T19:26:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb2067fc640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2080a0440 0x7fb20813bcc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc037040 tx=0x7fb1fc031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb1effff640 10 client.66375.objecter ms_handle_connect 0x7fb2080a0440 2024-06-11T19:26:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.376+0000 7fb1effff640 10 client.66375.objecter resend_mon_ops 2024-06-11T19:26:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1fc039040 con 0x7fb2080a0440 2024-06-11T19:26:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb1effff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb1effff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:26:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb1effff640 10 monclient: dump: 2024-06-11T19:26:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.412 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.412 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.412 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.412 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.412 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.412 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 <== mon.1 v2:172.21.15.117:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fb1fc034440 con 0x7fb2080a0440 2024-06-11T19:26:45.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb1effff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:26:45.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb1effff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:45.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2080a0440 msgr2=0x7fb20813bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb1effff640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2080a0440 0x7fb20813bcc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc037040 tx=0x7fb1fc031000 comp rx=0 tx=0).stop 2024-06-11T19:26:45.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb1effff640 1 -- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2080a0440 msgr2=0x7fb20813bcc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:45.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb1effff640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2080a0440 0x7fb20813bcc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.413 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:45.414 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:26:45.414 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:26:45.414 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:26:45.414 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 92, 2024-06-11T19:26:45.414 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 148309, 2024-06-11T19:26:45.414 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:26:45.414 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:26:45.414 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:26:45.415 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:26:45.415 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:26:45.415 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:26:45.415 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:26:45.415 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:26:45.415 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:26:45.415 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:26:45.415 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.415 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:26:45.416 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:45.416 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:26:45.416 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:26:45.416 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.416 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.416 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.416 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.416 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.417 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.417 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.417 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.417 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.417 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.417 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.417 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:26:45.417 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:26:45.417 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.418 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.418 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.418 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.418 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.418 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.418 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.418 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.419 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.419 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.419 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.419 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:26:45.419 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:26:45.419 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:26:45.419 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:26:45.419 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:26:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:26:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:26:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:26:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:26:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:26:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:26:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:26:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:26:45.421 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:45.421 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:26:45.421 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.421 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.421 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.421 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.421 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.421 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.422 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.422 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.422 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.422 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.422 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.422 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:26:45.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.422 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:26:45.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:26:45.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:26:45.423 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:26:45.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:26:45.424 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.424 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:26:45.425 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:26:45.425 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.425 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.425 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:26:45.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:26:45.426 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:26:45.426 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:26:45.427 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.427 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:26:45.427 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:26:45.427 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.428 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.428 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.428 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:26:45.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:26:45.428 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.429 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:26:45.429 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.429 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:26:45.429 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:26:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:26:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:26:45.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:26:45.431 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:26:45.431 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:26:45.432 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.432 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.432 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:26:45.433 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:26:45.433 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.433 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.433 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.433 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:26:45.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:26:45.433 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:26:45.434 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:26:45.435 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.435 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.435 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:26:45.435 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:26:45.435 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.436 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.436 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:26:45.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:26:45.436 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:26:45.437 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:26:45.437 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.438 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:26:45.438 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:26:45.438 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.438 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.438 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:26:45.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:26:45.439 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:26:45.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:26:45.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.440 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.441 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:26:45.441 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:26:45.441 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.441 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.441 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:26:45.442 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:26:45.442 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.442 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.442 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:26:45.443 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.443 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:26:45.443 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.443 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.443 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.444 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:26:45.444 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:26:45.444 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.444 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.444 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:26:45.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:26:45.445 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:26:45.445 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:26:45.446 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.446 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:26:45.447 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:26:45.447 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.447 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.447 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.447 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.447 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.447 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:26:45.448 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:26:45.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.448 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:45.448 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:45.448 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:45.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.448 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.448 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:26:45.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.449 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:45.449 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:45.449 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:45.449 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.449 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.449 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:26:45.450 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:45.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 10 librados: watch_flush enter 2024-06-11T19:26:45.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 10 librados: watch_flush exit 2024-06-11T19:26:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 20 client.66375.objecter shutdown clearing up homeless session... 2024-06-11T19:26:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 10 client.66375.objecter successfully canceled tick 2024-06-11T19:26:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 1 -- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb1f403fdb0 msgr2=0x7fb1f4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb1f403fdb0 0x7fb1f4042270 secure :-1 s=READY pgs=4326 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8006a00 tx=0x7fb1f8006990 comp rx=0 tx=0).stop 2024-06-11T19:26:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 10 monclient: shutdown 2024-06-11T19:26:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 1 -- 172.21.15.98:0/550824389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2080a2530 msgr2=0x7fb2081cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 1 --2- 172.21.15.98:0/550824389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2080a2530 0x7fb2081cd6b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb20000c450 tx=0x7fb20000c910 comp rx=0 tx=0).stop 2024-06-11T19:26:45.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb2077fe640 1 -- 172.21.15.98:0/550824389 reap_dead start 2024-06-11T19:26:45.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb2077fe640 1 -- 172.21.15.98:0/550824389 reap_dead start 2024-06-11T19:26:45.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 1 -- 172.21.15.98:0/550824389 shutdown_connections 2024-06-11T19:26:45.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 1 -- 172.21.15.98:0/550824389 >> 172.21.15.98:0/550824389 conn(0x7fb2080ab850 msgr2=0x7fb2080a8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:45.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 1 -- 172.21.15.98:0/550824389 shutdown_connections 2024-06-11T19:26:45.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 1 -- 172.21.15.98:0/550824389 wait complete. 2024-06-11T19:26:45.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.377+0000 7fb20d99c640 1 librados: shutdown 2024-06-11T19:26:45.454 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:26:45.454 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-06-11T19:26:45.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-06-11T19:26:45.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.554+0000 7fceff72a640 10 monclient: get_monmap_and_config 2024-06-11T19:26:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.554+0000 7fceff72a640 10 monclient: build_initial_monmap 2024-06-11T19:26:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.554+0000 7fceff72a640 10 monclient: monmap: 2024-06-11T19:26:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:45.555611+0000 2024-06-11T19:26:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:45.555611+0000 2024-06-11T19:26:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:45.554 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.555 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:45.555 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:45.555 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:45.555 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:45.555 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:45.555 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:45.555 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:45.555 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:45.556 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:45.556 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:45.556 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:45.556 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 1 Processor -- start 2024-06-11T19:26:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 1 -- start start 2024-06-11T19:26:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 10 monclient: init 2024-06-11T19:26:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 10 monclient: _add_conns ranks=[2,8,9,5,4,1,7,6,3,0,10] 2024-06-11T19:26:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80a4920 0x7fcef80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 10 monclient(hunting): picked mon.noname-b con 0x7fcef80a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:45.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a5240 0x7fcef80a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 10 monclient(hunting): picked mon.noname-e con 0x7fcef80a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:45.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcef80a2530 0x7fcef80a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 10 monclient(hunting): picked mon.noname-a con 0x7fcef80a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:26:45.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcef80a7140 con 0x7fcef80a2530 2024-06-11T19:26:45.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcef80a2b50 con 0x7fcef80a4920 2024-06-11T19:26:45.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcef80a2e30 con 0x7fcef80a5240 2024-06-11T19:26:45.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:45.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fceff72a640 10 monclient(hunting): authenticate will time out at 1815.362845s 2024-06-11T19:26:45.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fcefd4c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80a4920 0x7fcef80a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fcefccc5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a5240 0x7fcef80a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.555+0000 7fcefdcc7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcef80a2530 0x7fcef80a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a5240 0x7fcef80a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54442/0 (socket says 172.21.15.98:54442) 2024-06-11T19:26:45.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefd4c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80a4920 0x7fcef80a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:43090/0 (socket says 172.21.15.98:43090) 2024-06-11T19:26:45.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefdcc7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcef80a2530 0x7fcef80a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41794/0 (socket says 172.21.15.98:41794) 2024-06-11T19:26:45.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 1 -- 172.21.15.98:0/323590127 learned_addr learned my addr 172.21.15.98:0/323590127 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:45.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefdcc7640 10 monclient(hunting): get_auth_request con 0x7fcef80a2530 auth_method 0 2024-06-11T19:26:45.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefdcc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefdcc7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefdcc7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 10 monclient(hunting): get_auth_request con 0x7fcef80a5240 auth_method 0 2024-06-11T19:26:45.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefd4c6640 10 monclient(hunting): get_auth_request con 0x7fcef80a4920 auth_method 0 2024-06-11T19:26:45.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefd4c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefd4c6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefd4c6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefdcc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefdcc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefdcc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefd4c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefd4c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefd4c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 10 monclient(hunting): handle_auth_done global_id 124433 payload 274 2024-06-11T19:26:45.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 1 -- 172.21.15.98:0/323590127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80a4920 msgr2=0x7fcef80a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 1 --2- 172.21.15.98:0/323590127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80a4920 0x7fcef80a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 1 -- 172.21.15.98:0/323590127 >> v2:172.21.15.98:3300/0 conn(0x7fcef80a2530 msgr2=0x7fcef80a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 1 --2- 172.21.15.98:0/323590127 >> v2:172.21.15.98:3300/0 conn(0x7fcef80a2530 0x7fcef80a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 10 monclient: _finish_hunting 0 2024-06-11T19:26:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 1 monclient: found mon.noname-e 2024-06-11T19:26:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:26:45.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefccc5640 1 -- 172.21.15.98:0/323590127 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef80a3110 con 0x7fcef80a5240 2024-06-11T19:26:45.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefdcc7640 1 --2- 172.21.15.98:0/323590127 >> v2:172.21.15.98:3300/0 conn(0x7fcef80a2530 0x7fcef80a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:45.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.556+0000 7fcefd4c6640 1 --2- 172.21.15.98:0/323590127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80a4920 0x7fcef80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:45.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fcefccc5640 1 --2- 172.21.15.98:0/323590127 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a5240 0x7fcef80a59a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009870 tx=0x7fcee802b7c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=ff96657fd76daf4e server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 1 -- 172.21.15.98:0/323590127 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcee8038070 con 0x7fcef80a5240 2024-06-11T19:26:45.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:26:45.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 10 monclient: dump: 2024-06-11T19:26:45.572 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.572 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.573 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.574 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.574 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.574 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.574 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.574 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 10 monclient: _finish_auth 0 2024-06-11T19:26:45.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 10 monclient: _check_auth_tickets 2024-06-11T19:26:45.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:45.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 1 -- 172.21.15.98:0/323590127 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee8031990 con 0x7fcef80a5240 2024-06-11T19:26:45.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 5 monclient: authenticate success, global_id 124433 2024-06-11T19:26:45.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:45.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:45.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:45.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 1 -- 172.21.15.98:0/323590127 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcee80323b0 con 0x7fcef80a5240 2024-06-11T19:26:45.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 4 set_mon_vals no callback set 2024-06-11T19:26:45.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceeffff640 10 monclient: dump: 2024-06-11T19:26:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.576 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.577 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.577 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 10 monclient: shutdown 2024-06-11T19:26:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 1 -- 172.21.15.98:0/323590127 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a5240 msgr2=0x7fcef80a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 1 --2- 172.21.15.98:0/323590127 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a5240 0x7fcef80a59a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009870 tx=0x7fcee802b7c0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 1 -- 172.21.15.98:0/323590127 shutdown_connections 2024-06-11T19:26:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 1 --2- 172.21.15.98:0/323590127 >> v2:172.21.15.98:3300/0 conn(0x7fcef80a2530 0x7fcef80a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 1 --2- 172.21.15.98:0/323590127 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a5240 0x7fcef80a59a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 1 --2- 172.21.15.98:0/323590127 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcef80a4920 0x7fcef80a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 1 -- 172.21.15.98:0/323590127 >> 172.21.15.98:0/323590127 conn(0x7fcef80ab850 msgr2=0x7fcef80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 1 -- 172.21.15.98:0/323590127 shutdown_connections 2024-06-11T19:26:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.557+0000 7fceff72a640 1 -- 172.21.15.98:0/323590127 wait complete. 2024-06-11T19:26:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.558+0000 7fceff72a640 10 monclient: build_initial_monmap 2024-06-11T19:26:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.558+0000 7fceff72a640 10 monclient: monmap: 2024-06-11T19:26:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:45.579 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:45.580 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:45.581 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.558+0000 7fceff72a640 1 librados: starting msgr at 2024-06-11T19:26:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.558+0000 7fceff72a640 1 librados: starting objecter 2024-06-11T19:26:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.558+0000 7fceff72a640 1 Processor -- start 2024-06-11T19:26:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.558+0000 7fceff72a640 1 -- start start 2024-06-11T19:26:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.558+0000 7fceff72a640 1 librados: setting wanted keys 2024-06-11T19:26:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.558+0000 7fceff72a640 1 librados: calling monclient init 2024-06-11T19:26:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.558+0000 7fceff72a640 10 monclient: init 2024-06-11T19:26:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 10 monclient: _add_conns ranks=[5,8,7,0,1,6,2,4,3] 2024-06-11T19:26:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcef80a2530 0x7fcef81cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 10 monclient(hunting): picked mon.noname-h con 0x7fcef80a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a4920 0x7fcef8137990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 10 monclient(hunting): picked mon.noname-e con 0x7fcef80a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcef80a5240 0x7fcef8137ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 10 monclient(hunting): picked mon.noname-i con 0x7fcef80a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fcef80a6da0 con 0x7fcef80a2530 2024-06-11T19:26:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fcef8138e50 con 0x7fcef80a5240 2024-06-11T19:26:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcef80a6420 con 0x7fcef80a4920 2024-06-11T19:26:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceff72a640 10 monclient(hunting): authenticate will time out at 1815.366368s 2024-06-11T19:26:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a4920 0x7fcef8137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a4920 0x7fcef8137990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54448/0 (socket says 172.21.15.98:54448) 2024-06-11T19:26:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 1 -- 172.21.15.98:0/3039465461 learned_addr learned my addr 172.21.15.98:0/3039465461 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 10 monclient(hunting): get_auth_request con 0x7fcef80a4920 auth_method 0 2024-06-11T19:26:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefdcc7640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcef80a5240 0x7fcef8137ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefdcc7640 10 monclient(hunting): get_auth_request con 0x7fcef80a5240 auth_method 0 2024-06-11T19:26:45.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefdcc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefdcc7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefdcc7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefd4c6640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcef80a2530 0x7fcef81cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 10 monclient(hunting): handle_auth_done global_id 124442 payload 824 2024-06-11T19:26:45.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 1 -- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcef80a5240 msgr2=0x7fcef8137ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcef80a5240 0x7fcef8137ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 1 -- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcef80a2530 msgr2=0x7fcef81cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcef80a2530 0x7fcef81cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 10 monclient: _finish_hunting 0 2024-06-11T19:26:45.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 1 monclient: found mon.noname-e 2024-06-11T19:26:45.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:45.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:26:45.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 1 -- 172.21.15.98:0/3039465461 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef81290b0 con 0x7fcef80a4920 2024-06-11T19:26:45.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefd4c6640 10 monclient: get_auth_request con 0x7fcef80a2530 auth_method 0 2024-06-11T19:26:45.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefd4c6640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fcef80a2530 0x7fcef81cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:26:45.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefdcc7640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fcef80a5240 0x7fcef8137ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:26:45.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fcefccc5640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a4920 0x7fcef8137990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcee80090b0 tx=0x7fcee80046c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceedffb640 10 client.?.objecter ms_handle_connect 0x7fcef80a4920 2024-06-11T19:26:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.559+0000 7fceedffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcee8038050 con 0x7fcef80a4920 2024-06-11T19:26:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:26:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 10 monclient: dump: 2024-06-11T19:26:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.589 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.589 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.589 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.589 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.589 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.589 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.589 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.589 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.590 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.590 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.590 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.590 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 10 monclient: _finish_auth 0 2024-06-11T19:26:45.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 10 monclient: _check_auth_tickets 2024-06-11T19:26:45.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:45.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceff72a640 5 monclient: authenticate success, global_id 124442 2024-06-11T19:26:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee8044070 con 0x7fcef80a4920 2024-06-11T19:26:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcee8004350 con 0x7fcef80a4920 2024-06-11T19:26:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceff72a640 10 monclient: _renew_subs 2024-06-11T19:26:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceff72a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceff72a640 1 -- 172.21.15.98:0/3039465461 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef81cdf50 con 0x7fcef80a4920 2024-06-11T19:26:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceef7fe640 4 set_mon_vals no callback set 2024-06-11T19:26:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceedffb640 10 monclient: dump: 2024-06-11T19:26:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.593 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.593 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.593 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.593 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.593 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.593 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.593 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.593 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.594 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceff72a640 10 client.124442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceff72a640 10 monclient: _renew_subs 2024-06-11T19:26:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceff72a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceff72a640 1 -- 172.21.15.98:0/3039465461 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef8127540 con 0x7fcef80a4920 2024-06-11T19:26:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.560+0000 7fceff72a640 1 librados: init done 2024-06-11T19:26:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceff72a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:45.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceff72a640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8128c10 0x7fcef81253d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceff72a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceff72a640 1 -- 172.21.15.98:0/3039465461 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fcef8050830 con 0x7fcef8128c10 2024-06-11T19:26:45.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fcee8003610 con 0x7fcef80a4920 2024-06-11T19:26:45.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefd4c6640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8128c10 0x7fcef81253d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefd4c6640 10 monclient: get_auth_request con 0x7fcef8128c10 auth_method 0 2024-06-11T19:26:45.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefd4c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefd4c6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefd4c6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcedc03fdb0 0x7fcedc042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefd4c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefd4c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefd4c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7fcee8030070 con 0x7fcef80a4920 2024-06-11T19:26:45.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 10 client.124442.objecter ms_dispatch 0x7fcef8000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:45.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 3 client.124442.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:45.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 3 client.124442.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:45.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 20 client.124442.objecter dump_active .. 0 homeless 2024-06-11T19:26:45.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefd4c6640 10 monclient(hunting): handle_auth_done global_id 114510 payload 824 2024-06-11T19:26:45.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefd4c6640 1 -- 172.21.15.98:0/3039465461 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcef8073550 con 0x7fcef8128c10 2024-06-11T19:26:45.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefdcc7640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcedc03fdb0 0x7fcedc042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefdcc7640 10 monclient: get_auth_request con 0x7fcedc03fdb0 auth_method 0 2024-06-11T19:26:45.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 10 client.124442.objecter ms_handle_connect 0x7fcef8128c10 2024-06-11T19:26:45.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 10 client.124442.objecter resend_mon_ops 2024-06-11T19:26:45.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fcefd4c6640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8128c10 0x7fcef81253d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcef40107c0 tx=0x7fcef4010c80 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcef400f040 con 0x7fcef8128c10 2024-06-11T19:26:45.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:26:45.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.561+0000 7fceedffb640 10 monclient: dump: 2024-06-11T19:26:45.598 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.598 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.598 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.598 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.598 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.599 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.599 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.599 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.599 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.599 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.599 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.599 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.599 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.600 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.562+0000 7fceedffb640 10 client.124442.objecter ms_handle_connect 0x7fcedc03fdb0 2024-06-11T19:26:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.562+0000 7fcefdcc7640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcedc03fdb0 0x7fcedc042270 secure :-1 s=READY pgs=4327 cs=0 l=1 rev1=1 crypto rx=0x7fcef80a2950 tx=0x7fcef000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.562+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcef401c400 con 0x7fcef8128c10 2024-06-11T19:26:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.562+0000 7fceedffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:45.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.562+0000 7fceedffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.562+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8128c10 msgr2=0x7fcef81253d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.562+0000 7fceedffb640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8128c10 0x7fcef81253d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcef40107c0 tx=0x7fcef4010c80 comp rx=0 tx=0).stop 2024-06-11T19:26:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.562+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8128c10 msgr2=0x7fcef81253d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:45.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.562+0000 7fceedffb640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8128c10 0x7fcef81253d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.568+0000 7fceff72a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:26:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.568+0000 7fceff72a640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8125910 0x7fcef8125d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.568+0000 7fceff72a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.568+0000 7fceff72a640 1 -- 172.21.15.98:0/3039465461 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fcef8126260 con 0x7fcef8125910 2024-06-11T19:26:45.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.568+0000 7fcefd4c6640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8125910 0x7fcef8125d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.568+0000 7fcefd4c6640 10 monclient: get_auth_request con 0x7fcef8125910 auth_method 0 2024-06-11T19:26:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.568+0000 7fcefd4c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.568+0000 7fcefd4c6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.568+0000 7fcefd4c6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.568+0000 7fcefd4c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.568+0000 7fcefd4c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.568+0000 7fcefd4c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fcefd4c6640 10 monclient(hunting): handle_auth_done global_id 114519 payload 824 2024-06-11T19:26:45.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fcefd4c6640 1 -- 172.21.15.98:0/3039465461 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcef80a9240 con 0x7fcef8125910 2024-06-11T19:26:45.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fcefd4c6640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8125910 0x7fcef8125d20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcef40107c0 tx=0x7fcef40046c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 10 client.124442.objecter ms_handle_connect 0x7fcef8125910 2024-06-11T19:26:45.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 10 client.124442.objecter resend_mon_ops 2024-06-11T19:26:45.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcef400ffe0 con 0x7fcef8125910 2024-06-11T19:26:45.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:26:45.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 10 monclient: dump: 2024-06-11T19:26:45.605 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.606 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.606 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.606 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.606 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.606 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.606 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.606 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.606 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.607 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.607 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.607 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.607 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.607 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.607 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.607 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 <== mon.3 v2:172.21.15.117:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fcef400ffe0 con 0x7fcef8125910 2024-06-11T19:26:45.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:26:45.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:45.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8125910 msgr2=0x7fcef8125d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8125910 0x7fcef8125d20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcef40107c0 tx=0x7fcef40046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 1 -- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8125910 msgr2=0x7fcef8125d20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:45.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceedffb640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcef8125910 0x7fcef8125d20 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcef40107c0 tx=0x7fcef40046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.608 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:45.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:26:45.608 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:26:45.609 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:26:45.609 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 92, 2024-06-11T19:26:45.609 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 293937, 2024-06-11T19:26:45.609 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:26:45.609 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:26:45.609 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:26:45.609 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:26:45.609 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:26:45.610 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:26:45.610 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:26:45.610 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:26:45.610 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:26:45.610 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:26:45.610 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.610 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:26:45.610 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:45.610 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:26:45.611 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:26:45.611 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.611 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.611 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.611 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.611 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.611 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.611 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.611 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.612 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.612 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.612 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.612 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.612 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:26:45.612 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:26:45.612 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.612 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.612 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.613 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.613 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.613 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.613 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.613 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.613 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.613 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.613 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.614 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:26:45.614 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:26:45.614 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:26:45.614 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:26:45.614 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:26:45.614 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:26:45.614 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:45.615 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:45.615 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:26:45.615 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:26:45.615 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:26:45.615 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:26:45.615 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:26:45.615 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:26:45.615 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:26:45.615 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:45.616 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:26:45.616 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.616 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.616 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.616 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.616 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.616 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.616 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.616 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:26:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:26:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:26:45.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:26:45.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:26:45.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:26:45.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:26:45.619 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:26:45.619 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.619 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.620 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.620 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:26:45.620 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:26:45.620 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:26:45.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:26:45.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.621 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.621 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:26:45.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:26:45.622 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.622 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.622 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.622 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:26:45.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:26:45.623 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:26:45.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:26:45.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.624 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:26:45.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:26:45.624 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.625 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.625 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.625 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:26:45.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:26:45.625 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:26:45.626 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:26:45.626 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.627 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:26:45.627 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:26:45.627 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.627 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.627 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.628 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:26:45.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:26:45.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:26:45.628 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:26:45.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:26:45.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:26:45.630 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.630 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.630 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.630 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:26:45.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:26:45.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:26:45.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:26:45.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.632 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:26:45.632 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:26:45.632 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.632 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.632 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.633 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:26:45.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:26:45.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:26:45.634 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:26:45.634 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.634 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:26:45.635 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:26:45.635 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.635 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.635 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.635 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:26:45.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:26:45.636 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:26:45.636 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:26:45.637 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.637 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:26:45.637 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:26:45.637 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.638 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.638 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.638 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:26:45.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:26:45.638 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:26:45.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:26:45.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.640 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.640 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.640 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:26:45.640 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:26:45.640 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.640 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.640 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.640 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.641 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.641 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:26:45.641 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:26:45.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.641 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:45.641 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:45.641 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:45.642 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.642 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.642 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:26:45.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.642 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:45.642 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:45.642 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:45.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.643 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.643 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:26:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:45.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.569+0000 7fceff72a640 10 librados: watch_flush enter 2024-06-11T19:26:45.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 10 librados: watch_flush exit 2024-06-11T19:26:45.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 20 client.124442.objecter shutdown clearing up homeless session... 2024-06-11T19:26:45.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 10 client.124442.objecter successfully canceled tick 2024-06-11T19:26:45.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 1 -- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcedc03fdb0 msgr2=0x7fcedc042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fcedc03fdb0 0x7fcedc042270 secure :-1 s=READY pgs=4327 cs=0 l=1 rev1=1 crypto rx=0x7fcef80a2950 tx=0x7fcef000b040 comp rx=0 tx=0).stop 2024-06-11T19:26:45.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 10 monclient: shutdown 2024-06-11T19:26:45.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:45.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 1 -- 172.21.15.98:0/3039465461 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a4920 msgr2=0x7fcef8137990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 1 --2- 172.21.15.98:0/3039465461 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcef80a4920 0x7fcef8137990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcee80090b0 tx=0x7fcee80046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fcefdcc7640 1 -- 172.21.15.98:0/3039465461 reap_dead start 2024-06-11T19:26:45.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fcefdcc7640 1 -- 172.21.15.98:0/3039465461 reap_dead start 2024-06-11T19:26:45.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 1 -- 172.21.15.98:0/3039465461 shutdown_connections 2024-06-11T19:26:45.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 1 -- 172.21.15.98:0/3039465461 >> 172.21.15.98:0/3039465461 conn(0x7fcef80ab850 msgr2=0x7fcef80a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:45.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 1 -- 172.21.15.98:0/3039465461 shutdown_connections 2024-06-11T19:26:45.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 1 -- 172.21.15.98:0/3039465461 wait complete. 2024-06-11T19:26:45.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.570+0000 7fceff72a640 1 librados: shutdown 2024-06-11T19:26:45.647 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:26:45.647 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-06-11T19:26:45.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-06-11T19:26:45.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.747+0000 7f8660f03640 10 monclient: get_monmap_and_config 2024-06-11T19:26:45.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.747+0000 7f8660f03640 10 monclient: build_initial_monmap 2024-06-11T19:26:45.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.747+0000 7f8660f03640 10 monclient: monmap: 2024-06-11T19:26:45.750 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:45.750 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.750 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:45.748302+0000 2024-06-11T19:26:45.750 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:45.748302+0000 2024-06-11T19:26:45.750 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:45.750 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.750 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:45.751 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:45.751 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:45.751 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:45.751 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:45.751 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:45.751 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:45.751 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:45.751 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:45.752 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:45.752 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:45.752 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.747+0000 7f8660f03640 1 Processor -- start 2024-06-11T19:26:45.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.747+0000 7f8660f03640 1 -- start start 2024-06-11T19:26:45.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.747+0000 7f8660f03640 10 monclient: init 2024-06-11T19:26:45.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:45.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 10 monclient: _add_conns ranks=[3,1,5,7,0,4,2,10,8,9,6] 2024-06-11T19:26:45.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f865c0a4920 0x7f865c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 10 monclient(hunting): picked mon.noname-g con 0x7f865c0a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f865c0a5240 addr v1:172.21.15.117:6789/0 2024-06-11T19:26:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c0a2530 0x7f865c0a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 10 monclient(hunting): picked mon.noname-h con 0x7f865c0a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f865c0a7140 con 0x7f865c0a5240 2024-06-11T19:26:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f865c0a2b50 con 0x7f865c0a4920 2024-06-11T19:26:45.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f865c0a2e30 con 0x7f865c0a2530 2024-06-11T19:26:45.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:45.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8660f03640 10 monclient(hunting): authenticate will time out at 1815.555596s 2024-06-11T19:26:45.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f865ad9d640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c0a2530 0x7f865c0a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f865a59c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f865c0a4920 0x7f865c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8659d9b640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f865c0a5240 0x7f865c0a59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:60694/0 (socket says 172.21.15.98:60694) 2024-06-11T19:26:45.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f8659d9b640 1 -- 172.21.15.98:0/2115125268 learned_addr learned my addr 172.21.15.98:0/2115125268 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:26:45.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f865ad9d640 10 monclient(hunting): get_auth_request con 0x7f865c0a2530 auth_method 0 2024-06-11T19:26:45.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f865ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f865ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f865ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f865a59c640 10 monclient(hunting): get_auth_request con 0x7f865c0a4920 auth_method 0 2024-06-11T19:26:45.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f865a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f865a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.748+0000 7f865a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865ad9d640 10 monclient(hunting): handle_auth_done global_id 134105 payload 274 2024-06-11T19:26:45.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865ad9d640 1 -- 172.21.15.98:0/2115125268 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f865c0a4920 msgr2=0x7f865c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:45.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865ad9d640 1 --2- 172.21.15.98:0/2115125268 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f865c0a4920 0x7f865c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865ad9d640 1 -- 172.21.15.98:0/2115125268 >> v1:172.21.15.117:6789/0 conn(0x7f865c0a5240 legacy=0x7f865c0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865ad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:26:45.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865ad9d640 1 monclient: found mon.noname-h 2024-06-11T19:26:45.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:45.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865ad9d640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.117:3302/0 2024-06-11T19:26:45.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865959a640 1 -- 172.21.15.98:0/2115125268 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 947869471 0 0) 0x7f865c0a7140 con 0x7f865c0a5240 2024-06-11T19:26:45.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865ad9d640 1 -- 172.21.15.98:0/2115125268 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865c0a3110 con 0x7f865c0a2530 2024-06-11T19:26:45.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.749+0000 7f865959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:26:45.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865ad9d640 1 --2- 172.21.15.98:0/2115125268 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c0a2530 0x7f865c0a60b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8654009dc0 tx=0x7f865402bd50 comp rx=0 tx=0).ready entity=mon.5 client_cookie=70dbb5c20d868c16 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 1 -- 172.21.15.98:0/2115125268 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8654039070 con 0x7f865c0a2530 2024-06-11T19:26:45.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-06-11T19:26:45.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 10 monclient: dump: 2024-06-11T19:26:45.770 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.770 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.770 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.770 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.770 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.771 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.771 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.771 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.771 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.771 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.771 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.771 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.771 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.771 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.772 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.772 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 10 monclient: _finish_auth 0 2024-06-11T19:26:45.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 10 monclient: _check_auth_tickets 2024-06-11T19:26:45.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:45.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 1 -- 172.21.15.98:0/2115125268 <== mon.5 v2:172.21.15.117:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f865402cd70 con 0x7f865c0a2530 2024-06-11T19:26:45.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 5 monclient: authenticate success, global_id 134105 2024-06-11T19:26:45.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:45.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:45.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 1 -- 172.21.15.98:0/2115125268 <== mon.5 v2:172.21.15.117:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86540327b0 con 0x7f865c0a2530 2024-06-11T19:26:45.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:45.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 4 set_mon_vals no callback set 2024-06-11T19:26:45.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:26:45.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f865959a640 10 monclient: dump: 2024-06-11T19:26:45.773 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.774 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.774 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.774 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.774 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.774 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.774 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.774 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.774 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.775 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.775 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.775 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.775 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.775 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.775 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.775 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 10 monclient: shutdown 2024-06-11T19:26:45.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 1 -- 172.21.15.98:0/2115125268 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c0a2530 msgr2=0x7f865c0a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 1 --2- 172.21.15.98:0/2115125268 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c0a2530 0x7f865c0a60b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8654009dc0 tx=0x7f865402bd50 comp rx=0 tx=0).stop 2024-06-11T19:26:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 1 -- 172.21.15.98:0/2115125268 shutdown_connections 2024-06-11T19:26:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 1 --2- 172.21.15.98:0/2115125268 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c0a2530 0x7f865c0a60b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 1 --2- 172.21.15.98:0/2115125268 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f865c0a4920 0x7f865c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 1 -- 172.21.15.98:0/2115125268 >> 172.21.15.98:0/2115125268 conn(0x7f865c0ab850 msgr2=0x7f865c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 1 -- 172.21.15.98:0/2115125268 shutdown_connections 2024-06-11T19:26:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.750+0000 7f8660f03640 1 -- 172.21.15.98:0/2115125268 wait complete. 2024-06-11T19:26:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.751+0000 7f8660f03640 10 monclient: build_initial_monmap 2024-06-11T19:26:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.751+0000 7f8660f03640 10 monclient: monmap: 2024-06-11T19:26:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:45.779 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.751+0000 7f8660f03640 1 librados: starting msgr at 2024-06-11T19:26:45.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.751+0000 7f8660f03640 1 librados: starting objecter 2024-06-11T19:26:45.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.751+0000 7f8660f03640 1 Processor -- start 2024-06-11T19:26:45.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 1 -- start start 2024-06-11T19:26:45.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 1 librados: setting wanted keys 2024-06-11T19:26:45.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 1 librados: calling monclient init 2024-06-11T19:26:45.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 10 monclient: init 2024-06-11T19:26:45.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 10 monclient: _add_conns ranks=[1,0,5,2,6,7,8,4,3] 2024-06-11T19:26:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f865c0a2530 0x7f865c1cda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 10 monclient(hunting): picked mon.noname-f con 0x7f865c0a2530 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:26:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f865c0a4920 0x7f865c137a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 10 monclient(hunting): picked mon.noname-a con 0x7f865c0a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:26:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c0a5240 0x7f865c137f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 10 monclient(hunting): picked mon.noname-h con 0x7f865c0a5240 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f865c0a6da0 con 0x7f865c0a4920 2024-06-11T19:26:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f865c138e50 con 0x7f865c0a2530 2024-06-11T19:26:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f865c0a65f0 con 0x7f865c0a5240 2024-06-11T19:26:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8660f03640 10 monclient(hunting): authenticate will time out at 1815.559510s 2024-06-11T19:26:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f865c0a4920 0x7f865c137a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f865c0a4920 0x7f865c137a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41808/0 (socket says 172.21.15.98:41808) 2024-06-11T19:26:45.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 1 -- 172.21.15.98:0/592525468 learned_addr learned my addr 172.21.15.98:0/592525468 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:45.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 10 monclient(hunting): get_auth_request con 0x7f865c0a4920 auth_method 0 2024-06-11T19:26:45.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865ad9d640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c0a5240 0x7f865c137f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865ad9d640 10 monclient(hunting): get_auth_request con 0x7f865c0a5240 auth_method 0 2024-06-11T19:26:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865a59c640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f865c0a2530 0x7f865c1cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865a59c640 10 monclient(hunting): get_auth_request con 0x7f865c0a2530 auth_method 0 2024-06-11T19:26:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 10 monclient(hunting): handle_auth_done global_id 66393 payload 824 2024-06-11T19:26:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 1 -- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c0a5240 msgr2=0x7f865c137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c0a5240 0x7f865c137f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 1 -- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f865c0a2530 msgr2=0x7f865c1cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f865c0a2530 0x7f865c1cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:26:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 1 monclient: found mon.noname-a 2024-06-11T19:26:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:26:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f8659d9b640 1 -- 172.21.15.98:0/592525468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865c129450 con 0x7f865c0a4920 2024-06-11T19:26:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.752+0000 7f865a59c640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f865c0a2530 0x7f865c1cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:26:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8659d9b640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f865c0a4920 0x7f865c137a50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f864c00c340 tx=0x7f864c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 10 client.?.objecter ms_handle_connect 0x7f865c0a4920 2024-06-11T19:26:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f864c0047e0 con 0x7f865c0a4920 2024-06-11T19:26:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:26:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 10 monclient: dump: 2024-06-11T19:26:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.788 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 10 monclient: _finish_auth 0 2024-06-11T19:26:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:26:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f864c0049a0 con 0x7f865c0a4920 2024-06-11T19:26:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8660f03640 5 monclient: authenticate success, global_id 66393 2024-06-11T19:26:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8660f03640 10 monclient: _renew_subs 2024-06-11T19:26:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8660f03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8660f03640 1 -- 172.21.15.98:0/592525468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f865c127600 con 0x7f865c0a4920 2024-06-11T19:26:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f864c0133c0 con 0x7f865c0a4920 2024-06-11T19:26:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8658d99640 4 set_mon_vals no callback set 2024-06-11T19:26:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8660f03640 10 client.66393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8652ffd640 10 monclient: dump: 2024-06-11T19:26:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.791 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.792 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8660f03640 10 monclient: _renew_subs 2024-06-11T19:26:45.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8660f03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:45.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8660f03640 1 -- 172.21.15.98:0/592525468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f865c0aaa80 con 0x7f865c0a4920 2024-06-11T19:26:45.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8660f03640 1 librados: init done 2024-06-11T19:26:45.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.753+0000 7f8660f03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:45.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f8660f03640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c128f70 0x7f865c125490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f8660f03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f8660f03640 1 -- 172.21.15.98:0/592525468 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f865c050830 con 0x7f865c128f70 2024-06-11T19:26:45.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f864c011070 con 0x7f865c0a4920 2024-06-11T19:26:45.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865a59c640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c128f70 0x7f865c125490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865a59c640 10 monclient: get_auth_request con 0x7f865c128f70 auth_method 0 2024-06-11T19:26:45.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f8652ffd640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f863c03fdb0 0x7f863c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f864c051290 con 0x7f865c0a4920 2024-06-11T19:26:45.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f8652ffd640 10 client.66393.objecter ms_dispatch 0x7f865c000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:45.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f8652ffd640 3 client.66393.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:45.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f8652ffd640 3 client.66393.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:45.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f8652ffd640 20 client.66393.objecter dump_active .. 0 homeless 2024-06-11T19:26:45.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865ad9d640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f863c03fdb0 0x7f863c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865a59c640 10 monclient(hunting): handle_auth_done global_id 134123 payload 824 2024-06-11T19:26:45.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865a59c640 1 -- 172.21.15.98:0/592525468 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f865c0aa660 con 0x7f865c128f70 2024-06-11T19:26:45.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865ad9d640 10 monclient: get_auth_request con 0x7f863c03fdb0 auth_method 0 2024-06-11T19:26:45.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865ad9d640 1 -- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f863c03fdb0 msgr2=0x7f863c042270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-11T19:26:45.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865ad9d640 1 -- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f863c03fdb0 msgr2=0x7f863c042270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-11T19:26:45.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865ad9d640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f863c03fdb0 0x7f863c042270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-11T19:26:45.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865ad9d640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f863c03fdb0 0x7f863c042270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-11T19:26:45.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f8652ffd640 10 client.66393.objecter ms_handle_connect 0x7f865c128f70 2024-06-11T19:26:45.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f8652ffd640 10 client.66393.objecter resend_mon_ops 2024-06-11T19:26:45.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.754+0000 7f865a59c640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c128f70 0x7f865c125490 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8644011830 tx=0x7f8644011cf0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.755+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f864400fca0 con 0x7f865c128f70 2024-06-11T19:26:45.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.755+0000 7f8652ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.755+0000 7f8652ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:26:45.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.755+0000 7f8652ffd640 10 monclient: dump: 2024-06-11T19:26:45.796 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.796 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.796 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.797 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.797 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.797 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.797 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.797 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.797 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.797 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.797 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.797 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.798 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.798 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.798 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.798 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.755+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f864400f830 con 0x7f865c128f70 2024-06-11T19:26:45.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.755+0000 7f8652ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:45.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.755+0000 7f8652ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:45.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.755+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c128f70 msgr2=0x7f865c125490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.755+0000 7f8652ffd640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c128f70 0x7f865c125490 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8644011830 tx=0x7f8644011cf0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.755+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c128f70 msgr2=0x7f865c125490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:45.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.755+0000 7f8652ffd640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c128f70 0x7f865c125490 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8644011830 tx=0x7f8644011cf0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.761+0000 7f8660f03640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:26:45.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.761+0000 7f8660f03640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c1259d0 0x7f865c125db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.761+0000 7f8660f03640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.761+0000 7f8660f03640 1 -- 172.21.15.98:0/592525468 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f865c1262f0 con 0x7f865c1259d0 2024-06-11T19:26:45.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f865a59c640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c1259d0 0x7f865c125db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f865a59c640 10 monclient: get_auth_request con 0x7f865c1259d0 auth_method 0 2024-06-11T19:26:45.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f865a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f865a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f865a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f865a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f865a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f865a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f865a59c640 10 monclient(hunting): handle_auth_done global_id 134132 payload 824 2024-06-11T19:26:45.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f865a59c640 1 -- 172.21.15.98:0/592525468 --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f865c0a4d00 con 0x7f865c1259d0 2024-06-11T19:26:45.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f865a59c640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c1259d0 0x7f865c125db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8644011830 tx=0x7f86440046c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f8652ffd640 10 client.66393.objecter ms_handle_connect 0x7f865c1259d0 2024-06-11T19:26:45.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f8652ffd640 10 client.66393.objecter resend_mon_ops 2024-06-11T19:26:45.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 <== mon.5 v2:172.21.15.117:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8644006570 con 0x7f865c1259d0 2024-06-11T19:26:45.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f8652ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f8652ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-06-11T19:26:45.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.762+0000 7f8652ffd640 10 monclient: dump: 2024-06-11T19:26:45.802 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.802 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.802 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.802 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.802 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.802 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.802 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.802 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.802 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.803 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.803 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.803 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.803 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.803 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.803 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.803 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 <== mon.5 v2:172.21.15.117:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f8644006100 con 0x7f865c1259d0 2024-06-11T19:26:45.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8652ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:26:45.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8652ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:45.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c1259d0 msgr2=0x7f865c125db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8652ffd640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c1259d0 0x7f865c125db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8644011830 tx=0x7f86440046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8652ffd640 1 -- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c1259d0 msgr2=0x7f865c125db0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:45.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8652ffd640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f865c1259d0 0x7f865c125db0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8644011830 tx=0x7f86440046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.805 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:45.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:26:45.805 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:26:45.805 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:26:45.805 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 92, 2024-06-11T19:26:45.805 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 16930, 2024-06-11T19:26:45.805 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:26:45.806 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:26:45.806 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:26:45.806 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:26:45.806 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:26:45.806 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:26:45.806 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:26:45.806 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:26:45.806 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:26:45.807 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:26:45.807 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.807 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:26:45.807 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:45.807 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:26:45.807 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:26:45.807 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.807 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.808 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.808 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.808 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.808 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.808 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.808 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.808 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.809 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.809 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.809 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:26:45.809 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:26:45.809 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.809 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.809 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.809 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.811 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:26:45.811 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:26:45.811 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:26:45.811 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:26:45.811 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:26:45.811 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:26:45.811 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:26:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:26:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:26:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:26:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:26:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:26:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:26:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:26:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:45.814 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:45.814 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:45.814 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:45.814 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:45.814 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:45.814 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.814 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:26:45.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.815 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:26:45.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.815 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:26:45.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:26:45.815 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:26:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:26:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.817 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:26:45.817 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:26:45.817 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.817 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.817 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.818 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:26:45.818 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:26:45.818 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:26:45.818 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:26:45.819 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:26:45.820 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:26:45.820 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.820 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.820 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.820 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:26:45.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:26:45.821 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:26:45.821 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:26:45.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.822 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:26:45.822 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:26:45.823 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.823 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.823 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.823 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:26:45.823 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:26:45.823 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:26:45.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:26:45.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.825 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:26:45.825 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:26:45.825 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.825 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.825 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.826 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:26:45.826 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:26:45.826 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:26:45.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:26:45.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:26:45.828 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:26:45.828 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.828 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.828 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.828 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:26:45.828 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:26:45.829 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:26:45.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:26:45.830 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.830 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.830 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:26:45.830 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:26:45.830 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.830 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.831 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.831 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:26:45.831 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:26:45.831 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:26:45.832 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.832 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:26:45.832 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.832 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:26:45.833 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:26:45.833 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.833 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.833 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.833 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:26:45.833 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:26:45.834 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:26:45.834 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:26:45.835 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.835 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.835 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:26:45.835 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:26:45.835 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.835 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.836 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.836 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:26:45.836 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:26:45.836 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:45.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:45.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:45.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:26:45.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:45.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:26:45.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:45.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.837 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:26:45.838 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:26:45.838 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:45.838 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:45.838 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:45.838 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.838 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.839 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:26:45.839 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:26:45.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.839 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:45.839 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:45.839 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:45.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.839 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:45.839 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:26:45.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:45.840 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:45.840 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:45.840 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:45.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:45.840 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:45.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:45.840 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:26:45.840 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:45.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 10 librados: watch_flush enter 2024-06-11T19:26:45.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 10 librados: watch_flush exit 2024-06-11T19:26:45.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 20 client.66393.objecter shutdown clearing up homeless session... 2024-06-11T19:26:45.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 10 client.66393.objecter successfully canceled tick 2024-06-11T19:26:45.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 1 -- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f863c03fdb0 msgr2=0x7f863c042270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-11T19:26:45.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f863c03fdb0 0x7f863c042270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 10 monclient: shutdown 2024-06-11T19:26:45.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:45.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 1 -- 172.21.15.98:0/592525468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f865c0a4920 msgr2=0x7f865c137a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f865ad9d640 1 -- 172.21.15.98:0/592525468 reap_dead start 2024-06-11T19:26:45.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f865c0a4920 0x7f865c137a50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f864c00c340 tx=0x7f864c00c800 comp rx=0 tx=0).stop 2024-06-11T19:26:45.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 1 -- 172.21.15.98:0/592525468 shutdown_connections 2024-06-11T19:26:45.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 1 --2- 172.21.15.98:0/592525468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f865c0a4920 0x7f865c137a50 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 1 -- 172.21.15.98:0/592525468 >> 172.21.15.98:0/592525468 conn(0x7f865c0ab850 msgr2=0x7f865c09c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:45.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 1 -- 172.21.15.98:0/592525468 shutdown_connections 2024-06-11T19:26:45.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 1 -- 172.21.15.98:0/592525468 wait complete. 2024-06-11T19:26:45.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.763+0000 7f8660f03640 1 librados: shutdown 2024-06-11T19:26:45.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:26:45.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-06-11T19:26:45.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-06-11T19:26:45.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.946+0000 7fb54087a640 10 monclient: get_monmap_and_config 2024-06-11T19:26:45.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.946+0000 7fb54087a640 10 monclient: build_initial_monmap 2024-06-11T19:26:45.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.946+0000 7fb54087a640 10 monclient: monmap: 2024-06-11T19:26:45.946 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:45.946 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.946 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:45.947877+0000 2024-06-11T19:26:45.946 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:45.947877+0000 2024-06-11T19:26:45.946 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:45.947 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.947 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:45.947 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:45.947 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:45.947 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:45.947 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:45.947 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:45.947 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:45.947 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:45.948 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:45.948 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:45.948 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:45.948 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.947+0000 7fb54087a640 1 Processor -- start 2024-06-11T19:26:45.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.947+0000 7fb54087a640 1 -- start start 2024-06-11T19:26:45.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.947+0000 7fb54087a640 10 monclient: init 2024-06-11T19:26:45.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:45.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 10 monclient: _add_conns ranks=[7,4,9,8,3,0,5,2,10,6,1] 2024-06-11T19:26:45.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c0a4920 0x7fb53c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 10 monclient(hunting): picked mon.noname-i con 0x7fb53c0a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:45.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb53c0a5240 0x7fb53c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 10 monclient(hunting): picked mon.noname-c con 0x7fb53c0a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:45.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb53c0a2530 0x7fb53c0a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 10 monclient(hunting): picked mon.noname-a con 0x7fb53c0a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:26:45.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb53c0a7140 con 0x7fb53c0a2530 2024-06-11T19:26:45.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb53c0a2b50 con 0x7fb53c0a5240 2024-06-11T19:26:45.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fb53c0a2e30 con 0x7fb53c0a4920 2024-06-11T19:26:45.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:45.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb54087a640 10 monclient(hunting): authenticate will time out at 1815.755156s 2024-06-11T19:26:45.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb53c0a2530 0x7fb53c0a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb539d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb53c0a5240 0x7fb53c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb53c0a2530 0x7fb53c0a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41818/0 (socket says 172.21.15.98:41818) 2024-06-11T19:26:45.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 1 -- 172.21.15.98:0/1559779354 learned_addr learned my addr 172.21.15.98:0/1559779354 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:45.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb539d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb53c0a5240 0x7fb53c0a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:37214/0 (socket says 172.21.15.98:37214) 2024-06-11T19:26:45.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 10 monclient(hunting): get_auth_request con 0x7fb53c0a2530 auth_method 0 2024-06-11T19:26:45.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb539d9b640 10 monclient(hunting): get_auth_request con 0x7fb53c0a5240 auth_method 0 2024-06-11T19:26:45.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb539d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb539d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb539d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb539d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb539d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb539d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53a59c640 1 --2- 172.21.15.98:0/1559779354 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c0a4920 0x7fb53c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53a59c640 10 monclient(hunting): get_auth_request con 0x7fb53c0a4920 auth_method 0 2024-06-11T19:26:45.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 10 monclient(hunting): handle_auth_done global_id 66402 payload 274 2024-06-11T19:26:45.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 1 -- 172.21.15.98:0/1559779354 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c0a4920 msgr2=0x7fb53c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:45.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 1 --2- 172.21.15.98:0/1559779354 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c0a4920 0x7fb53c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 1 -- 172.21.15.98:0/1559779354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb53c0a5240 msgr2=0x7fb53c0a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:45.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 1 --2- 172.21.15.98:0/1559779354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb53c0a5240 0x7fb53c0a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:26:45.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 1 monclient: found mon.noname-a 2024-06-11T19:26:45.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:45.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:26:45.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb53ad9d640 1 -- 172.21.15.98:0/1559779354 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53c0a3110 con 0x7fb53c0a2530 2024-06-11T19:26:45.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.948+0000 7fb539d9b640 1 --2- 172.21.15.98:0/1559779354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb53c0a5240 0x7fb53c0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:45.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53ad9d640 1 --2- 172.21.15.98:0/1559779354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb53c0a2530 0x7fb53c0a5ee0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb534009e50 tx=0x7fb53402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f92f319b321c7 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 1 -- 172.21.15.98:0/1559779354 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb53402cbc0 con 0x7fb53c0a2530 2024-06-11T19:26:45.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:26:45.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 10 monclient: dump: 2024-06-11T19:26:45.966 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.966 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.966 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.966 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.967 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.967 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.967 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.967 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.967 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.967 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.967 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.967 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.968 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.968 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.968 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.968 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 10 monclient: _finish_auth 0 2024-06-11T19:26:45.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 10 monclient: _check_auth_tickets 2024-06-11T19:26:45.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:45.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 1 -- 172.21.15.98:0/1559779354 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb53402cd80 con 0x7fb53c0a2530 2024-06-11T19:26:45.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb54087a640 5 monclient: authenticate success, global_id 66402 2024-06-11T19:26:45.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:45.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb54087a640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:45.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 1 -- 172.21.15.98:0/1559779354 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb534034860 con 0x7fb53c0a2530 2024-06-11T19:26:45.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb54087a640 4 set_mon_vals no callback set 2024-06-11T19:26:45.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:45.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb53959a640 10 monclient: dump: 2024-06-11T19:26:45.969 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.969 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.970 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.970 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.970 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.970 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.970 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.970 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.970 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.970 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.971 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.971 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.971 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.971 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.971 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.971 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb54087a640 10 monclient: shutdown 2024-06-11T19:26:45.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb54087a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:45.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb54087a640 1 -- 172.21.15.98:0/1559779354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb53c0a2530 msgr2=0x7fb53c0a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.949+0000 7fb54087a640 1 --2- 172.21.15.98:0/1559779354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb53c0a2530 0x7fb53c0a5ee0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb534009e50 tx=0x7fb53402bdb0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.950+0000 7fb54087a640 1 -- 172.21.15.98:0/1559779354 shutdown_connections 2024-06-11T19:26:45.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.950+0000 7fb54087a640 1 --2- 172.21.15.98:0/1559779354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb53c0a2530 0x7fb53c0a5ee0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.950+0000 7fb54087a640 1 --2- 172.21.15.98:0/1559779354 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb53c0a5240 0x7fb53c0a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.950+0000 7fb54087a640 1 --2- 172.21.15.98:0/1559779354 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c0a4920 0x7fb53c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.950+0000 7fb54087a640 1 -- 172.21.15.98:0/1559779354 >> 172.21.15.98:0/1559779354 conn(0x7fb53c0ab850 msgr2=0x7fb53c0a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:45.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.950+0000 7fb54087a640 1 -- 172.21.15.98:0/1559779354 shutdown_connections 2024-06-11T19:26:45.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.950+0000 7fb54087a640 1 -- 172.21.15.98:0/1559779354 wait complete. 2024-06-11T19:26:45.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.950+0000 7fb54087a640 10 monclient: build_initial_monmap 2024-06-11T19:26:45.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.950+0000 7fb54087a640 10 monclient: monmap: 2024-06-11T19:26:45.973 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:45.973 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:45.973 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:45.973 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:45.973 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:45.973 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:45.974 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:45.974 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:45.974 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:45.974 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:45.974 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:45.974 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:45.974 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:45.974 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:45.975 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:45.975 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 1 librados: starting msgr at 2024-06-11T19:26:45.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 1 librados: starting objecter 2024-06-11T19:26:45.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 1 Processor -- start 2024-06-11T19:26:45.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 1 -- start start 2024-06-11T19:26:45.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 1 librados: setting wanted keys 2024-06-11T19:26:45.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 1 librados: calling monclient init 2024-06-11T19:26:45.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 10 monclient: init 2024-06-11T19:26:45.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:45.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 10 monclient: _add_conns ranks=[8,0,1,5,6,2,7,4,3] 2024-06-11T19:26:45.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb53c0a2530 0x7fb53c1cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 10 monclient(hunting): picked mon.noname-e con 0x7fb53c0a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:45.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb53c0a4920 0x7fb53c1ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 10 monclient(hunting): picked mon.noname-a con 0x7fb53c0a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:26:45.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb53c0a5240 0x7fb53c125140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 10 monclient(hunting): picked mon.noname-f con 0x7fb53c0a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:26:45.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c0a6da0 con 0x7fb53c0a4920 2024-06-11T19:26:45.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c138e50 con 0x7fb53c0a5240 2024-06-11T19:26:45.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb53c0a6420 con 0x7fb53c0a2530 2024-06-11T19:26:45.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:45.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb54087a640 10 monclient(hunting): authenticate will time out at 1815.758646s 2024-06-11T19:26:45.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb53c0a4920 0x7fb53c1ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb53a59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb53c0a2530 0x7fb53c1cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb53c0a4920 0x7fb53c1ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41832/0 (socket says 172.21.15.98:41832) 2024-06-11T19:26:45.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 1 -- 172.21.15.98:0/3875813335 learned_addr learned my addr 172.21.15.98:0/3875813335 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:45.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 10 monclient(hunting): get_auth_request con 0x7fb53c0a4920 auth_method 0 2024-06-11T19:26:45.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb53a59c640 10 monclient(hunting): get_auth_request con 0x7fb53c0a2530 auth_method 0 2024-06-11T19:26:45.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb53a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb53a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb53a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb53a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb53a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb53a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 10 monclient(hunting): handle_auth_done global_id 66411 payload 824 2024-06-11T19:26:45.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 1 -- 172.21.15.98:0/3875813335 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb53c0a2530 msgr2=0x7fb53c1cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb53c0a2530 0x7fb53c1cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 1 -- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb53c0a5240 msgr2=0x7fb53c125140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb53c0a5240 0x7fb53c125140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:45.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:26:45.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 1 monclient: found mon.noname-a 2024-06-11T19:26:45.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:45.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:26:45.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.951+0000 7fb539d9b640 1 -- 172.21.15.98:0/3875813335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53c128ac0 con 0x7fb53c0a4920 2024-06-11T19:26:45.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb539d9b640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb53c0a4920 0x7fb53c1ce010 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb53000e900 tx=0x7fb53000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 10 client.?.objecter ms_handle_connect 0x7fb53c0a4920 2024-06-11T19:26:45.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:45.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb530017070 con 0x7fb53c0a4920 2024-06-11T19:26:45.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:26:45.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 10 monclient: dump: 2024-06-11T19:26:45.982 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.982 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.982 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.982 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.982 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.982 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.983 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.983 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.983 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.983 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.983 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.983 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.983 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.983 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.984 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.984 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 10 monclient: _finish_auth 0 2024-06-11T19:26:45.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 10 monclient: _check_auth_tickets 2024-06-11T19:26:45.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:45.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb54087a640 5 monclient: authenticate success, global_id 66411 2024-06-11T19:26:45.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb53000ce70 con 0x7fb53c0a4920 2024-06-11T19:26:45.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb54087a640 10 monclient: _renew_subs 2024-06-11T19:26:45.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb54087a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:45.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb54087a640 1 -- 172.21.15.98:0/3875813335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb53c1271d0 con 0x7fb53c0a4920 2024-06-11T19:26:45.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:45.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5300138e0 con 0x7fb53c0a4920 2024-06-11T19:26:45.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb54087a640 10 client.66411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:45.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:45.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb538d99640 4 set_mon_vals no callback set 2024-06-11T19:26:45.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb52affd640 10 monclient: dump: 2024-06-11T19:26:45.986 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.986 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.986 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.986 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.986 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.986 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.986 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.986 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.987 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.987 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.987 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.987 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.987 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.987 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.987 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.987 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb54087a640 10 monclient: _renew_subs 2024-06-11T19:26:45.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb54087a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:45.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb54087a640 1 -- 172.21.15.98:0/3875813335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb53c0a8f30 con 0x7fb53c0a4920 2024-06-11T19:26:45.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.952+0000 7fb54087a640 1 librados: init done 2024-06-11T19:26:45.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb54087a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:45.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb54087a640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125680 0x7fb53c125a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb54087a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb54087a640 1 -- 172.21.15.98:0/3875813335 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fb53c050830 con 0x7fb53c125680 2024-06-11T19:26:45.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fb53001a070 con 0x7fb53c0a4920 2024-06-11T19:26:45.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53a59c640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125680 0x7fb53c125a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb52affd640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb51c03fdb0 0x7fb51c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53a59c640 10 monclient: get_auth_request con 0x7fb53c125680 auth_method 0 2024-06-11T19:26:45.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7fb53000f040 con 0x7fb53c0a4920 2024-06-11T19:26:45.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb52affd640 10 client.66411.objecter ms_dispatch 0x7fb53c000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:45.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb52affd640 3 client.66411.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:45.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb52affd640 3 client.66411.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:45.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb52affd640 20 client.66411.objecter dump_active .. 0 homeless 2024-06-11T19:26:45.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53ad9d640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb51c03fdb0 0x7fb51c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53ad9d640 10 monclient: get_auth_request con 0x7fb51c03fdb0 auth_method 0 2024-06-11T19:26:45.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53a59c640 10 monclient(hunting): handle_auth_done global_id 134143 payload 824 2024-06-11T19:26:45.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53a59c640 1 -- 172.21.15.98:0/3875813335 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb53c0a0aa0 con 0x7fb53c125680 2024-06-11T19:26:45.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb52affd640 10 client.66411.objecter ms_handle_connect 0x7fb53c125680 2024-06-11T19:26:45.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb52affd640 10 client.66411.objecter resend_mon_ops 2024-06-11T19:26:45.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.953+0000 7fb53a59c640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125680 0x7fb53c125a60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb52c010780 tx=0x7fb52c010c40 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb52c018020 con 0x7fb53c125680 2024-06-11T19:26:45.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb52affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb52affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:26:45.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb52affd640 10 monclient: dump: 2024-06-11T19:26:45.992 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:45.992 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:45.992 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.992 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:45.992 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:45.992 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:45.993 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:45.993 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:45.993 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:45.993 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:45.993 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:45.993 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:45.993 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:45.993 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:45.994 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:45.994 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:45.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb52affd640 10 client.66411.objecter ms_handle_connect 0x7fb51c03fdb0 2024-06-11T19:26:45.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb53ad9d640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb51c03fdb0 0x7fb51c042270 secure :-1 s=READY pgs=4328 cs=0 l=1 rev1=1 crypto rx=0x7fb534035000 tx=0x7fb53402e000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb52c004240 con 0x7fb53c125680 2024-06-11T19:26:45.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb52affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:45.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb52affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:45.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125680 msgr2=0x7fb53c125a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:45.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb52affd640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125680 0x7fb53c125a60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb52c010780 tx=0x7fb52c010c40 comp rx=0 tx=0).stop 2024-06-11T19:26:45.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125680 msgr2=0x7fb53c125a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:45.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.954+0000 7fb52affd640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125680 0x7fb53c125a60 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb52c010780 tx=0x7fb52c010c40 comp rx=0 tx=0).stop 2024-06-11T19:26:45.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.960+0000 7fb54087a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-06-11T19:26:45.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.960+0000 7fb54087a640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125fa0 0x7fb53c09f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:45.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.960+0000 7fb54087a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:45.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.960+0000 7fb54087a640 1 -- 172.21.15.98:0/3875813335 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7fb53c0aaba0 con 0x7fb53c125fa0 2024-06-11T19:26:45.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb53a59c640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125fa0 0x7fb53c09f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:45.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb53a59c640 10 monclient: get_auth_request con 0x7fb53c125fa0 auth_method 0 2024-06-11T19:26:45.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb53a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:45.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb53a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:45.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb53a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:45.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb53a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:45.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb53a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:45.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb53a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:45.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb53a59c640 10 monclient(hunting): handle_auth_done global_id 134152 payload 824 2024-06-11T19:26:45.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb53a59c640 1 -- 172.21.15.98:0/3875813335 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb53c0a4d00 con 0x7fb53c125fa0 2024-06-11T19:26:45.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb53a59c640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125fa0 0x7fb53c09f380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb52c010780 tx=0x7fb52c0046c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:45.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb52affd640 10 client.66411.objecter ms_handle_connect 0x7fb53c125fa0 2024-06-11T19:26:45.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb52affd640 10 client.66411.objecter resend_mon_ops 2024-06-11T19:26:45.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb52c018040 con 0x7fb53c125fa0 2024-06-11T19:26:45.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb52affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:45.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb52affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:26:45.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.961+0000 7fb52affd640 10 monclient: dump: 2024-06-11T19:26:46.000 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:46.000 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:46.000 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:46.000 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:46.000 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:46.000 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:46.000 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:46.000 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:46.000 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:46.001 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:46.001 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:46.001 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:46.001 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:46.001 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:46.001 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:46.001 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:46.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 <== mon.7 v2:172.21.15.117:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fb52c00ee00 con 0x7fb53c125fa0 2024-06-11T19:26:46.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb52affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-06-11T19:26:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb52affd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125fa0 msgr2=0x7fb53c09f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb52affd640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125fa0 0x7fb53c09f380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb52c010780 tx=0x7fb52c0046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb52affd640 1 -- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125fa0 msgr2=0x7fb53c09f380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-11T19:26:46.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb52affd640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7fb53c125fa0 0x7fb53c09f380 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.002 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:26:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:26:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-06-11T19:26:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 92, 2024-06-11T19:26:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 17127, 2024-06-11T19:26:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-06-11T19:26:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-06-11T19:26:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-06-11T19:26:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-06-11T19:26:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-06-11T19:26:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-06-11T19:26:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-06-11T19:26:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-06-11T19:26:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-06-11T19:26:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-06-11T19:26:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-06-11T19:26:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-06-11T19:26:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-06-11T19:26:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540701547738038271", 2024-06-11T19:26:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-06-11T19:26:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-06-11T19:26:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-06-11T19:26:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-06-11T19:26:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-06-11T19:26:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-06-11T19:26:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "9daa5251-9ae7-4a07-8f69-80112d2163e1", 2024-06-11T19:26:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-06-11T19:08:43.892957Z", 2024-06-11T19:26:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-06-11T19:26:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-06-11T19:26:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-06-11T19:26:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-06-11T19:26:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-06-11T19:26:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-06-11T19:26:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-06-11T19:26:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-06-11T19:26:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-06-11T19:26:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-06-11T19:26:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-06-11T19:26:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-06-11T19:26:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-06-11T19:26:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-06-11T19:26:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-06-11T19:26:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-06-11T19:26:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-06-11T19:26:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-06-11T19:26:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-06-11T19:26:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-06-11T19:26:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-06-11T19:26:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-06-11T19:26:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-06-11T19:26:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-06-11T19:26:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-06-11T19:26:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-06-11T19:26:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-06-11T19:26:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-06-11T19:26:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-06-11T19:26:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-06-11T19:26:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3300", 2024-06-11T19:26:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789", 2024-06-11T19:26:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6789/0", 2024-06-11T19:26:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6789/0", 2024-06-11T19:26:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-06-11T19:26:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-06-11T19:26:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-06-11T19:26:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-06-11T19:26:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-06-11T19:26:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-06-11T19:26:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-06-11T19:26:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-06-11T19:26:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3301", 2024-06-11T19:26:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790", 2024-06-11T19:26:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6790/0", 2024-06-11T19:26:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6790/0", 2024-06-11T19:26:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-06-11T19:26:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-06-11T19:26:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-06-11T19:26:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-06-11T19:26:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-06-11T19:26:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-06-11T19:26:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-06-11T19:26:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-06-11T19:26:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3302", 2024-06-11T19:26:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791", 2024-06-11T19:26:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6791/0", 2024-06-11T19:26:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6791/0", 2024-06-11T19:26:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-06-11T19:26:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-06-11T19:26:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-06-11T19:26:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-06-11T19:26:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-06-11T19:26:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-06-11T19:26:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-06-11T19:26:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-06-11T19:26:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:3303", 2024-06-11T19:26:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792", 2024-06-11T19:26:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.117:6792/0", 2024-06-11T19:26:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.117:6792/0", 2024-06-11T19:26:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-06-11T19:26:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-06-11T19:26:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-06-11T19:26:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-06-11T19:26:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-06-11T19:26:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-06-11T19:26:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-06-11T19:26:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-06-11T19:26:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-06-11T19:26:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-06-11T19:26:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-06-11T19:26:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-06-11T19:26:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-06-11T19:26:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-06-11T19:26:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-06-11T19:26:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-06-11T19:26:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-06-11T19:26:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-06-11T19:26:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-06-11T19:26:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03cffffffdffff", 2024-06-11T19:26:46.039 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2024-06-11T19:26:46.039 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-06-11T19:26:46.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-06-11T19:26:46.039 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-06-11T19:26:46.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-06-11T19:26:46.039 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-06-11T19:26:46.039 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:46.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 10 librados: watch_flush enter 2024-06-11T19:26:46.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 10 librados: watch_flush exit 2024-06-11T19:26:46.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 20 client.66411.objecter shutdown clearing up homeless session... 2024-06-11T19:26:46.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 10 client.66411.objecter successfully canceled tick 2024-06-11T19:26:46.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 1 -- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb51c03fdb0 msgr2=0x7fb51c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:46.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fb51c03fdb0 0x7fb51c042270 secure :-1 s=READY pgs=4328 cs=0 l=1 rev1=1 crypto rx=0x7fb534035000 tx=0x7fb53402e000 comp rx=0 tx=0).stop 2024-06-11T19:26:46.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 10 monclient: shutdown 2024-06-11T19:26:46.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:46.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 1 -- 172.21.15.98:0/3875813335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb53c0a4920 msgr2=0x7fb53c1ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:46.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb53ad9d640 1 -- 172.21.15.98:0/3875813335 reap_dead start 2024-06-11T19:26:46.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb53c0a4920 0x7fb53c1ce010 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb53000e900 tx=0x7fb53000edc0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 1 -- 172.21.15.98:0/3875813335 shutdown_connections 2024-06-11T19:26:46.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 1 --2- 172.21.15.98:0/3875813335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb53c0a4920 0x7fb53c1ce010 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 1 -- 172.21.15.98:0/3875813335 >> 172.21.15.98:0/3875813335 conn(0x7fb53c0ab850 msgr2=0x7fb53c0a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:46.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 1 -- 172.21.15.98:0/3875813335 shutdown_connections 2024-06-11T19:26:46.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 1 -- 172.21.15.98:0/3875813335 wait complete. 2024-06-11T19:26:46.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:45.962+0000 7fb54087a640 1 librados: shutdown 2024-06-11T19:26:46.043 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-06-11T19:26:46.044 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-06-11T19:26:46.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-06-11T19:26:46.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.143+0000 7f7919c17640 10 monclient: get_monmap_and_config 2024-06-11T19:26:46.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.143+0000 7f7919c17640 10 monclient: build_initial_monmap 2024-06-11T19:26:46.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.143+0000 7f7919c17640 10 monclient: monmap: 2024-06-11T19:26:46.142 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:46.142 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:46.143 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:46.144211+0000 2024-06-11T19:26:46.143 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:46.144211+0000 2024-06-11T19:26:46.143 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:46.143 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:46.143 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:46.143 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:46.143 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:46.144 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:46.144 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:46.144 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:46.144 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:46.144 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:46.144 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:46.144 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:46.144 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:46.144 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.143+0000 7f7919c17640 1 Processor -- start 2024-06-11T19:26:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.143+0000 7f7919c17640 1 -- start start 2024-06-11T19:26:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.143+0000 7f7919c17640 10 monclient: init 2024-06-11T19:26:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:46.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 10 monclient: _add_conns ranks=[0,10,4,7,6,5,2,8,9,1,3] 2024-06-11T19:26:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79140a0cd0 addr v1:172.21.15.98:6789/0 2024-06-11T19:26:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7914138c80 0x7f7914139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 10 monclient(hunting): picked mon.noname-f con 0x7f7914138c80 addr v2:172.21.15.117:3300/0 2024-06-11T19:26:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f791413dc50 0x7f7914140040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 10 monclient(hunting): picked mon.noname-c con 0x7f791413dc50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79140a17c0 con 0x7f79140a0cd0 2024-06-11T19:26:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f79140739b0 con 0x7f7914138c80 2024-06-11T19:26:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f791409ecf0 con 0x7f791413dc50 2024-06-11T19:26:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7919c17640 10 monclient(hunting): authenticate will time out at 1815.951443s 2024-06-11T19:26:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f791413dc50 0x7f7914140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f79137fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f79140a0cd0 0x7f79140a10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51390/0 (socket says 172.21.15.98:51390) 2024-06-11T19:26:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f79137fe640 1 -- 172.21.15.98:0/656700495 learned_addr learned my addr 172.21.15.98:0/656700495 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:26:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 10 monclient(hunting): get_auth_request con 0x7f791413dc50 auth_method 0 2024-06-11T19:26:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:46.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:46.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:46.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:46.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:46.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:46.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7912ffd640 1 --2- 172.21.15.98:0/656700495 >> v2:172.21.15.117:3300/0 conn(0x7f7914138c80 0x7f7914139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:46.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7912ffd640 10 monclient(hunting): get_auth_request con 0x7f7914138c80 auth_method 0 2024-06-11T19:26:46.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7912ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7912ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f79127fc640 1 -- 172.21.15.98:0/656700495 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2887347336 0 0) 0x7f79140a17c0 con 0x7f79140a0cd0 2024-06-11T19:26:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f79127fc640 10 monclient(hunting): _negotiate 2024-06-11T19:26:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f79127fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f79127fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f79127fc640 10 monclient(hunting): my global_id is 66420 2024-06-11T19:26:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f79127fc640 1 -- 172.21.15.98:0/656700495 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7900003520 con 0x7f79140a0cd0 2024-06-11T19:26:46.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 10 monclient(hunting): handle_auth_done global_id 134149 payload 274 2024-06-11T19:26:46.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 1 -- 172.21.15.98:0/656700495 >> v2:172.21.15.117:3300/0 conn(0x7f7914138c80 msgr2=0x7f7914139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:46.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 1 --2- 172.21.15.98:0/656700495 >> v2:172.21.15.117:3300/0 conn(0x7f7914138c80 0x7f7914139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 1 -- 172.21.15.98:0/656700495 >> v1:172.21.15.98:6789/0 conn(0x7f79140a0cd0 legacy=0x7f79140a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:46.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 10 monclient: _finish_hunting 0 2024-06-11T19:26:46.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 1 monclient: found mon.noname-c 2024-06-11T19:26:46.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:46.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:26:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.144+0000 7f7913fff640 1 -- 172.21.15.98:0/656700495 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791409efd0 con 0x7f791413dc50 2024-06-11T19:26:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f7913fff640 1 --2- 172.21.15.98:0/656700495 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f791413dc50 0x7f7914140040 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f790c009d90 tx=0x7f790c02bd80 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ee2591e590886341 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 1 -- 172.21.15.98:0/656700495 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f790c039070 con 0x7f791413dc50 2024-06-11T19:26:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:26:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 10 monclient: dump: 2024-06-11T19:26:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:46.159 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:46.159 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:46.159 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:46.159 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:46.159 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:46.159 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:46.159 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:46.159 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:46.160 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:46.160 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:46.160 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:46.160 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:46.160 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:46.160 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:46.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 10 monclient: _finish_auth 0 2024-06-11T19:26:46.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 10 monclient: _check_auth_tickets 2024-06-11T19:26:46.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:46.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 1 -- 172.21.15.98:0/656700495 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f790c02cda0 con 0x7f791413dc50 2024-06-11T19:26:46.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f7919c17640 5 monclient: authenticate success, global_id 134149 2024-06-11T19:26:46.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f7919c17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:46.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:46.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 1 -- 172.21.15.98:0/656700495 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f790c0327b0 con 0x7f791413dc50 2024-06-11T19:26:46.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f7919c17640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:46.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f7919c17640 4 set_mon_vals no callback set 2024-06-11T19:26:46.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:46.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:26:46.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.145+0000 7f79127fc640 10 monclient: dump: 2024-06-11T19:26:46.162 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:46.162 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:46.162 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:46.162 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:46.162 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:46.162 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:46.162 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:46.162 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:46.163 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:46.163 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:46.163 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:46.163 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:46.163 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:46.163 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:46.163 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:46.163 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:46.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.146+0000 7f7919c17640 10 monclient: shutdown 2024-06-11T19:26:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.146+0000 7f7919c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.146+0000 7f7919c17640 1 -- 172.21.15.98:0/656700495 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f791413dc50 msgr2=0x7f7914140040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.146+0000 7f7919c17640 1 --2- 172.21.15.98:0/656700495 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f791413dc50 0x7f7914140040 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f790c009d90 tx=0x7f790c02bd80 comp rx=0 tx=0).stop 2024-06-11T19:26:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.146+0000 7f7919c17640 1 -- 172.21.15.98:0/656700495 shutdown_connections 2024-06-11T19:26:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.146+0000 7f7919c17640 1 --2- 172.21.15.98:0/656700495 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f791413dc50 0x7f7914140040 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.146+0000 7f7919c17640 1 --2- 172.21.15.98:0/656700495 >> v2:172.21.15.117:3300/0 conn(0x7f7914138c80 0x7f7914139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.146+0000 7f7919c17640 1 -- 172.21.15.98:0/656700495 >> 172.21.15.98:0/656700495 conn(0x7f79140a7850 msgr2=0x7f79140a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.146+0000 7f7919c17640 1 -- 172.21.15.98:0/656700495 shutdown_connections 2024-06-11T19:26:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.146+0000 7f7919c17640 1 -- 172.21.15.98:0/656700495 wait complete. 2024-06-11T19:26:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.146+0000 7f7919c17640 10 monclient: build_initial_monmap 2024-06-11T19:26:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.146+0000 7f7919c17640 10 monclient: monmap: 2024-06-11T19:26:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:46.166 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 1 librados: starting msgr at 2024-06-11T19:26:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 1 librados: starting objecter 2024-06-11T19:26:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 1 Processor -- start 2024-06-11T19:26:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 1 -- start start 2024-06-11T19:26:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 1 librados: setting wanted keys 2024-06-11T19:26:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 1 librados: calling monclient init 2024-06-11T19:26:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 10 monclient: init 2024-06-11T19:26:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 10 monclient: _add_conns ranks=[6,0,4,2,8,3,5,1,7] 2024-06-11T19:26:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79140a0cd0 0x7f79141358e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 10 monclient(hunting): picked mon.noname-d con 0x7f79140a0cd0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7914138c80 0x7f7914135e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.147+0000 7f7919c17640 10 monclient(hunting): picked mon.noname-a con 0x7f7914138c80 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:26:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f791413dc50 0x7f791413ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 10 monclient(hunting): picked mon.noname-c con 0x7f791413dc50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7914140580 con 0x7f7914138c80 2024-06-11T19:26:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7914076a00 con 0x7f791413dc50 2024-06-11T19:26:46.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:46.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f7914140700 con 0x7f79140a0cd0 2024-06-11T19:26:46.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:46.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 10 monclient(hunting): authenticate will time out at 1815.955041s 2024-06-11T19:26:46.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7912ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7914138c80 0x7f7914135e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:46.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f79137fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79140a0cd0 0x7f79141358e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:46.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7912ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7914138c80 0x7f7914135e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41848/0 (socket says 172.21.15.98:41848) 2024-06-11T19:26:46.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7912ffd640 1 -- 172.21.15.98:0/1286132610 learned_addr learned my addr 172.21.15.98:0/1286132610 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:46.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7912ffd640 10 monclient(hunting): get_auth_request con 0x7f7914138c80 auth_method 0 2024-06-11T19:26:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7912ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7912ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f791413dc50 0x7f791413ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f79137fe640 10 monclient(hunting): get_auth_request con 0x7f79140a0cd0 auth_method 0 2024-06-11T19:26:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f79137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f79137fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f79137fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 10 monclient(hunting): get_auth_request con 0x7f791413dc50 auth_method 0 2024-06-11T19:26:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7912ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7912ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7912ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f79137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f79137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f79137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 10 monclient(hunting): handle_auth_done global_id 134158 payload 824 2024-06-11T19:26:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 1 -- 172.21.15.98:0/1286132610 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79140a0cd0 msgr2=0x7f79141358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79140a0cd0 0x7f79141358e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 1 -- 172.21.15.98:0/1286132610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7914138c80 msgr2=0x7f7914135e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7914138c80 0x7f7914135e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 10 monclient: _finish_hunting 0 2024-06-11T19:26:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 1 monclient: found mon.noname-c 2024-06-11T19:26:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:26:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 1 -- 172.21.15.98:0/1286132610 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791413c680 con 0x7f791413dc50 2024-06-11T19:26:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7912ffd640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7914138c80 0x7f7914135e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7913fff640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f791413dc50 0x7f791413ac80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f790c033000 tx=0x7f790c004a40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 10 client.?.objecter ms_handle_connect 0x7f791413dc50 2024-06-11T19:26:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 1 -- 172.21.15.98:0/1286132610 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f790c039050 con 0x7f791413dc50 2024-06-11T19:26:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:26:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 10 monclient: dump: 2024-06-11T19:26:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:46.176 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 10 monclient: _finish_auth 0 2024-06-11T19:26:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:26:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 1 -- 172.21.15.98:0/1286132610 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f790c045070 con 0x7f791413dc50 2024-06-11T19:26:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 5 monclient: authenticate success, global_id 134158 2024-06-11T19:26:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 10 monclient: _renew_subs 2024-06-11T19:26:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 1 -- 172.21.15.98:0/1286132610 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f791412d340 con 0x7f791413dc50 2024-06-11T19:26:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 1 -- 172.21.15.98:0/1286132610 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f790c0046c0 con 0x7f791413dc50 2024-06-11T19:26:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:26:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f790bfff640 10 monclient: dump: 2024-06-11T19:26:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:46.178 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:46.178 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:46.178 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:46.178 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:46.178 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:46.178 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:46.178 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:46.178 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:46.178 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:46.179 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:46.179 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:46.179 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:46.179 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:46.179 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:46.179 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:46.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7911ffb640 4 set_mon_vals no callback set 2024-06-11T19:26:46.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 10 client.134158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:46.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 10 monclient: _renew_subs 2024-06-11T19:26:46.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:46.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 1 -- 172.21.15.98:0/1286132610 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f7914140050 con 0x7f791413dc50 2024-06-11T19:26:46.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.148+0000 7f7919c17640 1 librados: init done 2024-06-11T19:26:46.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.149+0000 7f790bfff640 1 -- 172.21.15.98:0/1286132610 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f790c004250 con 0x7f791413dc50 2024-06-11T19:26:46.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.149+0000 7f7919c17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:46.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.149+0000 7f7919c17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:46.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.149+0000 7f7919c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:46.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.149+0000 7f7919c17640 1 -- 172.21.15.98:0/1286132610 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79140a49a0 con 0x7f791413dc50 2024-06-11T19:26:46.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.149+0000 7f790bfff640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f78f403fde0 0x7f78f40422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.149+0000 7f790bfff640 1 -- 172.21.15.98:0/1286132610 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f790c073bf0 con 0x7f791413dc50 2024-06-11T19:26:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.150+0000 7f790bfff640 10 client.134158.objecter ms_dispatch 0x7f7914000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.150+0000 7f790bfff640 3 client.134158.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.150+0000 7f790bfff640 3 client.134158.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.150+0000 7f790bfff640 20 client.134158.objecter dump_active .. 0 homeless 2024-06-11T19:26:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.150+0000 7f79137fe640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f78f403fde0 0x7f78f40422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.150+0000 7f79137fe640 10 monclient: get_auth_request con 0x7f78f403fde0 auth_method 0 2024-06-11T19:26:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.150+0000 7f790bfff640 10 client.134158.objecter ms_handle_connect 0x7f78f403fde0 2024-06-11T19:26:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.150+0000 7f79137fe640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f78f403fde0 0x7f78f40422a0 secure :-1 s=READY pgs=4329 cs=0 l=1 rev1=1 crypto rx=0x7f7904007f50 tx=0x7f7904007ee0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.154+0000 7f790bfff640 1 -- 172.21.15.98:0/1286132610 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f791409efd0 con 0x7f791413dc50 2024-06-11T19:26:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.154+0000 7f790bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.154+0000 7f790bfff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:46.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.407+0000 7f7919c17640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-06-11T19:26:46.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.407+0000 7f7919c17640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-06-11T19:26:46.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.407+0000 7f7919c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:46.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.407+0000 7f7919c17640 1 -- 172.21.15.98:0/1286132610 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f7914139060 con 0x7f791413dc50 2024-06-11T19:26:46.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.408+0000 7f790bfff640 1 -- 172.21.15.98:0/1286132610 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f790c02cd70 con 0x7f791413dc50 2024-06-11T19:26:46.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.408+0000 7f790bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-06-11T19:26:46.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.408+0000 7f790bfff640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:46.409 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=1648761865,config=692619927,health=1415816619,kv=2000936593,logm=4225422029}) 2024-06-11T19:26:46.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=9,logm=57} crc {auth=2608861333,config=692619927,health=1415816619,kv=1471832156,logm=3094195325}) 2024-06-11T19:26:46.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=1648761865,config=692619927,health=1415816619,kv=2000936593,logm=4225422029}) 2024-06-11T19:26:46.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=11,kv=9,logm=55} crc {auth=3171991178,config=692619927,health=3966421950,kv=1736684318,logm=4206056412}) 2024-06-11T19:26:46.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.410 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=1648761865,config=692619927,health=1415816619,kv=2000936593,logm=4225422029}) 2024-06-11T19:26:46.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=3171991178,config=692619927,health=1415816619,kv=2000936593,logm=3554922327}) 2024-06-11T19:26:46.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=1648761865,config=692619927,health=1415816619,kv=2000936593,logm=4225422029}) 2024-06-11T19:26:46.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=3171991178,config=692619927,health=1415816619,kv=2000936593,logm=2026671735}) 2024-06-11T19:26:46.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=1648761865,config=692619927,health=1415816619,kv=2000936593,logm=4225422029}) 2024-06-11T19:26:46.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=21,config=2,health=12,kv=10,logm=55} crc {auth=2534986448,config=692619927,health=4022453781,kv=1048552374,logm=666404538}) 2024-06-11T19:26:46.411 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=1648761865,config=692619927,health=1415816619,kv=2000936593,logm=4225422029}) 2024-06-11T19:26:46.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=250127918,config=692619927,health=4102591616,kv=2000936593,logm=237780503}) 2024-06-11T19:26:46.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=1648761865,config=692619927,health=1415816619,kv=2000936593,logm=4225422029}) 2024-06-11T19:26:46.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=1852687347,config=692619927,health=1415816619,kv=2000936593,logm=2253288825}) 2024-06-11T19:26:46.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=1648761865,config=692619927,health=1415816619,kv=2000936593,logm=4225422029}) 2024-06-11T19:26:46.412 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.409+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=3171991178,config=692619927,health=1415816619,kv=2000936593,logm=302942161}) 2024-06-11T19:26:46.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452322454}) 2024-06-11T19:26:46.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219538222}) 2024-06-11T19:26:46.414 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452322454}) 2024-06-11T19:26:46.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2890847273}) 2024-06-11T19:26:46.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452322454}) 2024-06-11T19:26:46.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1647413851}) 2024-06-11T19:26:46.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452322454}) 2024-06-11T19:26:46.415 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1381812249}) 2024-06-11T19:26:46.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452322454}) 2024-06-11T19:26:46.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=230543767}) 2024-06-11T19:26:46.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452322454}) 2024-06-11T19:26:46.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4038294767}) 2024-06-11T19:26:46.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.416 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452322454}) 2024-06-11T19:26:46.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=590732388}) 2024-06-11T19:26:46.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452322454}) 2024-06-11T19:26:46.417 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=611301685}) 2024-06-11T19:26:46.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.411+0000 7f7919c17640 10 librados: watch_flush enter 2024-06-11T19:26:46.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 10 librados: watch_flush exit 2024-06-11T19:26:46.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 20 client.134158.objecter shutdown clearing up homeless session... 2024-06-11T19:26:46.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 10 client.134158.objecter successfully canceled tick 2024-06-11T19:26:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 -- 172.21.15.98:0/1286132610 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f78f403fde0 msgr2=0x7f78f40422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f78f403fde0 0x7f78f40422a0 secure :-1 s=READY pgs=4329 cs=0 l=1 rev1=1 crypto rx=0x7f7904007f50 tx=0x7f7904007ee0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 10 monclient: shutdown 2024-06-11T19:26:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 -- 172.21.15.98:0/1286132610 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f791413dc50 msgr2=0x7f791413ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f791413dc50 0x7f791413ac80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f790c033000 tx=0x7f790c004a40 comp rx=0 tx=0).stop 2024-06-11T19:26:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 -- 172.21.15.98:0/1286132610 shutdown_connections 2024-06-11T19:26:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f78f403fde0 0x7f78f40422a0 unknown :-1 s=CLOSED pgs=4329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f791413dc50 0x7f791413ac80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7914138c80 0x7f7914135e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 --2- 172.21.15.98:0/1286132610 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79140a0cd0 0x7f79141358e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:46.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 -- 172.21.15.98:0/1286132610 >> 172.21.15.98:0/1286132610 conn(0x7f79140a7850 msgr2=0x7f79140a21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:46.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 -- 172.21.15.98:0/1286132610 shutdown_connections 2024-06-11T19:26:46.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 -- 172.21.15.98:0/1286132610 wait complete. 2024-06-11T19:26:46.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:46.412+0000 7f7919c17640 1 librados: shutdown 2024-06-11T19:26:46.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794504838}) 2024-06-11T19:26:46.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231968356}) 2024-06-11T19:26:46.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794504838}) 2024-06-11T19:26:46.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2066217737}) 2024-06-11T19:26:46.435 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794504838}) 2024-06-11T19:26:46.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1938968297}) 2024-06-11T19:26:46.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794504838}) 2024-06-11T19:26:46.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2983166010}) 2024-06-11T19:26:46.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794504838}) 2024-06-11T19:26:46.436 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4138818911}) 2024-06-11T19:26:46.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794504838}) 2024-06-11T19:26:46.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1447844919}) 2024-06-11T19:26:46.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794504838}) 2024-06-11T19:26:46.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.413+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3230163186}) 2024-06-11T19:26:46.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.414+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.414+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794504838}) 2024-06-11T19:26:46.437 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.414+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4087130957}) 2024-06-11T19:26:46.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423345421}) 2024-06-11T19:26:46.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918394792}) 2024-06-11T19:26:46.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423345421}) 2024-06-11T19:26:46.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3964908579}) 2024-06-11T19:26:46.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.438 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423345421}) 2024-06-11T19:26:46.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=319285894}) 2024-06-11T19:26:46.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423345421}) 2024-06-11T19:26:46.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2628263698}) 2024-06-11T19:26:46.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423345421}) 2024-06-11T19:26:46.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2421525133}) 2024-06-11T19:26:46.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.439 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423345421}) 2024-06-11T19:26:46.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=550131623}) 2024-06-11T19:26:46.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423345421}) 2024-06-11T19:26:46.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2902648851}) 2024-06-11T19:26:46.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423345421}) 2024-06-11T19:26:46.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.416+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=61483028}) 2024-06-11T19:26:46.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.440 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3519932779}) 2024-06-11T19:26:46.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1506240081}) 2024-06-11T19:26:46.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3519932779}) 2024-06-11T19:26:46.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4006244971}) 2024-06-11T19:26:46.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3519932779}) 2024-06-11T19:26:46.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3203113720}) 2024-06-11T19:26:46.441 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3519932779}) 2024-06-11T19:26:46.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2287274504}) 2024-06-11T19:26:46.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3519932779}) 2024-06-11T19:26:46.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2907545524}) 2024-06-11T19:26:46.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3519932779}) 2024-06-11T19:26:46.442 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3365405586}) 2024-06-11T19:26:46.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3519932779}) 2024-06-11T19:26:46.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3649671734}) 2024-06-11T19:26:46.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3519932779}) 2024-06-11T19:26:46.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.418+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=842229589}) 2024-06-11T19:26:46.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.419+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.443 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.419+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105849624}) 2024-06-11T19:26:46.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3886021596}) 2024-06-11T19:26:46.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105849624}) 2024-06-11T19:26:46.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3123683372}) 2024-06-11T19:26:46.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105849624}) 2024-06-11T19:26:46.444 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3694853655}) 2024-06-11T19:26:46.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105849624}) 2024-06-11T19:26:46.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4257590595}) 2024-06-11T19:26:46.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105849624}) 2024-06-11T19:26:46.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=539071826}) 2024-06-11T19:26:46.445 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105849624}) 2024-06-11T19:26:46.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1028982323}) 2024-06-11T19:26:46.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105849624}) 2024-06-11T19:26:46.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=989785616}) 2024-06-11T19:26:46.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105849624}) 2024-06-11T19:26:46.446 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.420+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2503429791}) 2024-06-11T19:26:46.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145982093}) 2024-06-11T19:26:46.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3967847837}) 2024-06-11T19:26:46.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145982093}) 2024-06-11T19:26:46.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=625670414}) 2024-06-11T19:26:46.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.447 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145982093}) 2024-06-11T19:26:46.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3658221945}) 2024-06-11T19:26:46.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145982093}) 2024-06-11T19:26:46.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1630322065}) 2024-06-11T19:26:46.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145982093}) 2024-06-11T19:26:46.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4183548130}) 2024-06-11T19:26:46.448 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145982093}) 2024-06-11T19:26:46.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3426371947}) 2024-06-11T19:26:46.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145982093}) 2024-06-11T19:26:46.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2832111224}) 2024-06-11T19:26:46.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145982093}) 2024-06-11T19:26:46.449 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.421+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2882575083}) 2024-06-11T19:26:46.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480728983}) 2024-06-11T19:26:46.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4167568269}) 2024-06-11T19:26:46.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480728983}) 2024-06-11T19:26:46.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=959911358}) 2024-06-11T19:26:46.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.450 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480728983}) 2024-06-11T19:26:46.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1698156533}) 2024-06-11T19:26:46.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480728983}) 2024-06-11T19:26:46.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1081533905}) 2024-06-11T19:26:46.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480728983}) 2024-06-11T19:26:46.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=214895033}) 2024-06-11T19:26:46.451 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480728983}) 2024-06-11T19:26:46.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1378229187}) 2024-06-11T19:26:46.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480728983}) 2024-06-11T19:26:46.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3599628378}) 2024-06-11T19:26:46.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480728983}) 2024-06-11T19:26:46.452 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.422+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1041161759}) 2024-06-11T19:26:46.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.423+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673156235}) 2024-06-11T19:26:46.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912990592}) 2024-06-11T19:26:46.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673156235}) 2024-06-11T19:26:46.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1212486346}) 2024-06-11T19:26:46.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673156235}) 2024-06-11T19:26:46.453 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=980928058}) 2024-06-11T19:26:46.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673156235}) 2024-06-11T19:26:46.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2550138148}) 2024-06-11T19:26:46.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673156235}) 2024-06-11T19:26:46.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2788473873}) 2024-06-11T19:26:46.454 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673156235}) 2024-06-11T19:26:46.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2494048238}) 2024-06-11T19:26:46.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673156235}) 2024-06-11T19:26:46.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1667676771}) 2024-06-11T19:26:46.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673156235}) 2024-06-11T19:26:46.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.424+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1047316970}) 2024-06-11T19:26:46.455 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000073786}) 2024-06-11T19:26:46.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1909052750}) 2024-06-11T19:26:46.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000073786}) 2024-06-11T19:26:46.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781287020}) 2024-06-11T19:26:46.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000073786}) 2024-06-11T19:26:46.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=563767232}) 2024-06-11T19:26:46.456 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000073786}) 2024-06-11T19:26:46.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=408418279}) 2024-06-11T19:26:46.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000073786}) 2024-06-11T19:26:46.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=932749750}) 2024-06-11T19:26:46.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000073786}) 2024-06-11T19:26:46.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=579808483}) 2024-06-11T19:26:46.457 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000073786}) 2024-06-11T19:26:46.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3653244186}) 2024-06-11T19:26:46.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000073786}) 2024-06-11T19:26:46.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.425+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1597134551}) 2024-06-11T19:26:46.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537234215}) 2024-06-11T19:26:46.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016462303}) 2024-06-11T19:26:46.458 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537234215}) 2024-06-11T19:26:46.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2564300135}) 2024-06-11T19:26:46.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537234215}) 2024-06-11T19:26:46.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3958533721}) 2024-06-11T19:26:46.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537234215}) 2024-06-11T19:26:46.459 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=816306255}) 2024-06-11T19:26:46.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537234215}) 2024-06-11T19:26:46.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1421559282}) 2024-06-11T19:26:46.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537234215}) 2024-06-11T19:26:46.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2380601736}) 2024-06-11T19:26:46.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537234215}) 2024-06-11T19:26:46.460 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2676558155}) 2024-06-11T19:26:46.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537234215}) 2024-06-11T19:26:46.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.426+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=800853376}) 2024-06-11T19:26:46.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2171035995}) 2024-06-11T19:26:46.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365344577}) 2024-06-11T19:26:46.461 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.462 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-06-11T19:26:46.708 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2171035995}) 2024-06-11T19:26:46.708 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1102609496}) 2024-06-11T19:26:46.709 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.709 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2171035995}) 2024-06-11T19:26:46.709 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=440020821}) 2024-06-11T19:26:46.709 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.709 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2171035995}) 2024-06-11T19:26:46.709 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4252989647}) 2024-06-11T19:26:46.709 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.709 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2171035995}) 2024-06-11T19:26:46.710 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=93631509}) 2024-06-11T19:26:46.710 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.710 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2171035995}) 2024-06-11T19:26:46.710 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3517857677}) 2024-06-11T19:26:46.710 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.710 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2171035995}) 2024-06-11T19:26:46.710 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=988190235}) 2024-06-11T19:26:46.711 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.711 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2171035995}) 2024-06-11T19:26:46.711 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.427+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2137851073}) 2024-06-11T19:26:46.711 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.711 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082717415}) 2024-06-11T19:26:46.711 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1587029761}) 2024-06-11T19:26:46.711 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.711 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082717415}) 2024-06-11T19:26:46.711 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1192735281}) 2024-06-11T19:26:46.712 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.712 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082717415}) 2024-06-11T19:26:46.712 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2881754317}) 2024-06-11T19:26:46.712 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.712 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082717415}) 2024-06-11T19:26:46.712 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2841842959}) 2024-06-11T19:26:46.712 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.712 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082717415}) 2024-06-11T19:26:46.713 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1044033009}) 2024-06-11T19:26:46.713 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.713 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082717415}) 2024-06-11T19:26:46.713 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4100903185}) 2024-06-11T19:26:46.713 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.713 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082717415}) 2024-06-11T19:26:46.713 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3437942904}) 2024-06-11T19:26:46.713 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.714 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082717415}) 2024-06-11T19:26:46.714 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.429+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2418284559}) 2024-06-11T19:26:46.714 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.714 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518097853}) 2024-06-11T19:26:46.714 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1482279351}) 2024-06-11T19:26:46.714 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.714 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518097853}) 2024-06-11T19:26:46.714 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=548021423}) 2024-06-11T19:26:46.715 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.715 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518097853}) 2024-06-11T19:26:46.715 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=999218098}) 2024-06-11T19:26:46.715 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.715 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518097853}) 2024-06-11T19:26:46.715 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2749905940}) 2024-06-11T19:26:46.715 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.715 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518097853}) 2024-06-11T19:26:46.716 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3309813012}) 2024-06-11T19:26:46.716 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.716 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518097853}) 2024-06-11T19:26:46.716 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3275108035}) 2024-06-11T19:26:46.716 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.716 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518097853}) 2024-06-11T19:26:46.716 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1637868786}) 2024-06-11T19:26:46.716 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.716 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518097853}) 2024-06-11T19:26:46.717 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.430+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1334060223}) 2024-06-11T19:26:46.717 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.717 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360359995}) 2024-06-11T19:26:46.717 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929457564}) 2024-06-11T19:26:46.717 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.717 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360359995}) 2024-06-11T19:26:46.717 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335481155}) 2024-06-11T19:26:46.717 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.718 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360359995}) 2024-06-11T19:26:46.718 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3212906103}) 2024-06-11T19:26:46.718 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.718 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360359995}) 2024-06-11T19:26:46.718 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=245559882}) 2024-06-11T19:26:46.718 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.718 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360359995}) 2024-06-11T19:26:46.718 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=797638939}) 2024-06-11T19:26:46.719 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.719 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360359995}) 2024-06-11T19:26:46.719 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3486944272}) 2024-06-11T19:26:46.719 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.719 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360359995}) 2024-06-11T19:26:46.719 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3672991507}) 2024-06-11T19:26:46.719 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.719 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360359995}) 2024-06-11T19:26:46.719 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.431+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3950210856}) 2024-06-11T19:26:46.720 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.720 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2253548034}) 2024-06-11T19:26:46.720 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3411683178}) 2024-06-11T19:26:46.720 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.720 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2253548034}) 2024-06-11T19:26:46.720 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3321515328}) 2024-06-11T19:26:46.720 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.720 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2253548034}) 2024-06-11T19:26:46.721 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1239106760}) 2024-06-11T19:26:46.721 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.721 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2253548034}) 2024-06-11T19:26:46.721 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1132783493}) 2024-06-11T19:26:46.721 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.721 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2253548034}) 2024-06-11T19:26:46.721 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4162569263}) 2024-06-11T19:26:46.721 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.721 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2253548034}) 2024-06-11T19:26:46.722 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3194420184}) 2024-06-11T19:26:46.722 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.432+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.722 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.433+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2253548034}) 2024-06-11T19:26:46.722 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.433+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1320942709}) 2024-06-11T19:26:46.722 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.433+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.722 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.433+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2253548034}) 2024-06-11T19:26:46.722 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.433+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=468227325}) 2024-06-11T19:26:46.722 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.722 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149942451}) 2024-06-11T19:26:46.723 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2330586836}) 2024-06-11T19:26:46.723 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.723 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149942451}) 2024-06-11T19:26:46.723 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=978381138}) 2024-06-11T19:26:46.723 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.723 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149942451}) 2024-06-11T19:26:46.723 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=376033894}) 2024-06-11T19:26:46.723 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.724 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149942451}) 2024-06-11T19:26:46.724 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=661688116}) 2024-06-11T19:26:46.724 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.724 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149942451}) 2024-06-11T19:26:46.724 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1292418957}) 2024-06-11T19:26:46.724 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.724 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149942451}) 2024-06-11T19:26:46.724 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2658374771}) 2024-06-11T19:26:46.725 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.725 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149942451}) 2024-06-11T19:26:46.725 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=482454233}) 2024-06-11T19:26:46.725 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.725 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149942451}) 2024-06-11T19:26:46.725 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.434+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4080798294}) 2024-06-11T19:26:46.725 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.725 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233831817}) 2024-06-11T19:26:46.725 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=58004388}) 2024-06-11T19:26:46.726 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.726 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233831817}) 2024-06-11T19:26:46.726 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149963024}) 2024-06-11T19:26:46.726 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.726 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233831817}) 2024-06-11T19:26:46.726 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2491618151}) 2024-06-11T19:26:46.726 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.726 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233831817}) 2024-06-11T19:26:46.727 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3140922177}) 2024-06-11T19:26:46.727 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.727 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233831817}) 2024-06-11T19:26:46.727 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=706101156}) 2024-06-11T19:26:46.727 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.727 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233831817}) 2024-06-11T19:26:46.727 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4078355028}) 2024-06-11T19:26:46.727 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.727 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233831817}) 2024-06-11T19:26:46.728 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=284210079}) 2024-06-11T19:26:46.728 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.728 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233831817}) 2024-06-11T19:26:46.728 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.435+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=690162234}) 2024-06-11T19:26:46.728 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.728 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737150859}) 2024-06-11T19:26:46.728 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=590873191}) 2024-06-11T19:26:46.728 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.729 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737150859}) 2024-06-11T19:26:46.729 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2448568313}) 2024-06-11T19:26:46.729 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.729 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737150859}) 2024-06-11T19:26:46.729 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=80103518}) 2024-06-11T19:26:46.729 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.729 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737150859}) 2024-06-11T19:26:46.729 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2039776684}) 2024-06-11T19:26:46.729 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.730 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737150859}) 2024-06-11T19:26:46.730 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=375042171}) 2024-06-11T19:26:46.730 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.730 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737150859}) 2024-06-11T19:26:46.730 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.436+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2752930231}) 2024-06-11T19:26:46.730 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.437+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.730 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.437+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737150859}) 2024-06-11T19:26:46.730 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.437+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2309547726}) 2024-06-11T19:26:46.731 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.437+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.731 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.437+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737150859}) 2024-06-11T19:26:46.731 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.437+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3560562706}) 2024-06-11T19:26:46.731 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.731 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1963201414}) 2024-06-11T19:26:46.731 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472943016}) 2024-06-11T19:26:46.731 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.731 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1963201414}) 2024-06-11T19:26:46.731 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2737420874}) 2024-06-11T19:26:46.732 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.732 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1963201414}) 2024-06-11T19:26:46.732 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1261954500}) 2024-06-11T19:26:46.732 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.732 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1963201414}) 2024-06-11T19:26:46.732 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1026474645}) 2024-06-11T19:26:46.732 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.732 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1963201414}) 2024-06-11T19:26:46.733 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=558384930}) 2024-06-11T19:26:46.733 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.733 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1963201414}) 2024-06-11T19:26:46.733 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4203548195}) 2024-06-11T19:26:46.733 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.733 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1963201414}) 2024-06-11T19:26:46.733 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2686271242}) 2024-06-11T19:26:46.733 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.734 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1963201414}) 2024-06-11T19:26:46.734 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.438+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3935499150}) 2024-06-11T19:26:46.734 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.734 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907931157}) 2024-06-11T19:26:46.734 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085525825}) 2024-06-11T19:26:46.734 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.734 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907931157}) 2024-06-11T19:26:46.734 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4167892216}) 2024-06-11T19:26:46.734 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.735 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907931157}) 2024-06-11T19:26:46.735 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=59923314}) 2024-06-11T19:26:46.735 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.735 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907931157}) 2024-06-11T19:26:46.735 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3161276649}) 2024-06-11T19:26:46.735 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.735 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907931157}) 2024-06-11T19:26:46.735 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2105962072}) 2024-06-11T19:26:46.736 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.736 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907931157}) 2024-06-11T19:26:46.736 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1154606043}) 2024-06-11T19:26:46.736 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.736 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907931157}) 2024-06-11T19:26:46.736 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3599783260}) 2024-06-11T19:26:46.736 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.736 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907931157}) 2024-06-11T19:26:46.736 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.439+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=208432505}) 2024-06-11T19:26:46.737 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.440+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.737 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.440+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996214172}) 2024-06-11T19:26:46.737 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.440+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563729108}) 2024-06-11T19:26:46.737 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.440+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.737 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.440+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996214172}) 2024-06-11T19:26:46.737 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.440+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2197444856}) 2024-06-11T19:26:46.737 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.737 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996214172}) 2024-06-11T19:26:46.738 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=898887658}) 2024-06-11T19:26:46.738 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.738 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996214172}) 2024-06-11T19:26:46.738 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3369457572}) 2024-06-11T19:26:46.738 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.738 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996214172}) 2024-06-11T19:26:46.738 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2755762344}) 2024-06-11T19:26:46.738 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.738 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996214172}) 2024-06-11T19:26:46.739 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3356184143}) 2024-06-11T19:26:46.739 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.739 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996214172}) 2024-06-11T19:26:46.739 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2647668488}) 2024-06-11T19:26:46.739 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.739 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996214172}) 2024-06-11T19:26:46.739 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.441+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3570102095}) 2024-06-11T19:26:46.739 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.740 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047071448}) 2024-06-11T19:26:46.740 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528252777}) 2024-06-11T19:26:46.740 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.740 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047071448}) 2024-06-11T19:26:46.740 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3839887227}) 2024-06-11T19:26:46.740 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.740 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047071448}) 2024-06-11T19:26:46.740 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3891721601}) 2024-06-11T19:26:46.740 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.741 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047071448}) 2024-06-11T19:26:46.741 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3834408952}) 2024-06-11T19:26:46.741 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.741 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047071448}) 2024-06-11T19:26:46.741 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3624132825}) 2024-06-11T19:26:46.741 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.741 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047071448}) 2024-06-11T19:26:46.741 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3729999841}) 2024-06-11T19:26:46.742 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.742 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047071448}) 2024-06-11T19:26:46.742 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=868804795}) 2024-06-11T19:26:46.742 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.742 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047071448}) 2024-06-11T19:26:46.742 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.442+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1956770705}) 2024-06-11T19:26:46.742 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.742 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316722283}) 2024-06-11T19:26:46.743 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144774385}) 2024-06-11T19:26:46.743 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.743 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316722283}) 2024-06-11T19:26:46.743 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2304539173}) 2024-06-11T19:26:46.743 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.743 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316722283}) 2024-06-11T19:26:46.743 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3698576379}) 2024-06-11T19:26:46.743 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.743 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316722283}) 2024-06-11T19:26:46.744 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=679542444}) 2024-06-11T19:26:46.744 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.744 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316722283}) 2024-06-11T19:26:46.744 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2297826724}) 2024-06-11T19:26:46.744 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.744 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316722283}) 2024-06-11T19:26:46.744 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3724202689}) 2024-06-11T19:26:46.744 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.744 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316722283}) 2024-06-11T19:26:46.745 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3670132782}) 2024-06-11T19:26:46.745 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.745 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316722283}) 2024-06-11T19:26:46.745 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.443+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3565028031}) 2024-06-11T19:26:46.745 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.745 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1438228184}) 2024-06-11T19:26:46.745 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502166529}) 2024-06-11T19:26:46.746 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.746 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1438228184}) 2024-06-11T19:26:46.746 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1657928892}) 2024-06-11T19:26:46.746 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.746 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1438228184}) 2024-06-11T19:26:46.746 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=154153510}) 2024-06-11T19:26:46.746 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.746 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1438228184}) 2024-06-11T19:26:46.747 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4049233854}) 2024-06-11T19:26:46.747 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.747 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1438228184}) 2024-06-11T19:26:46.747 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=255227636}) 2024-06-11T19:26:46.747 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.747 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1438228184}) 2024-06-11T19:26:46.747 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=506754731}) 2024-06-11T19:26:46.747 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.748 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1438228184}) 2024-06-11T19:26:46.748 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1285747836}) 2024-06-11T19:26:46.748 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.748 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1438228184}) 2024-06-11T19:26:46.748 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.444+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=737258999}) 2024-06-11T19:26:46.748 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.445+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.748 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.445+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732977597}) 2024-06-11T19:26:46.748 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.445+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762219275}) 2024-06-11T19:26:46.748 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.445+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.749 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.445+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732977597}) 2024-06-11T19:26:46.749 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3935419879}) 2024-06-11T19:26:46.749 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.749 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732977597}) 2024-06-11T19:26:46.749 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1742903055}) 2024-06-11T19:26:46.749 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.749 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732977597}) 2024-06-11T19:26:46.749 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3592509947}) 2024-06-11T19:26:46.750 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.750 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732977597}) 2024-06-11T19:26:46.750 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2621142429}) 2024-06-11T19:26:46.750 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.750 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732977597}) 2024-06-11T19:26:46.750 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1899434430}) 2024-06-11T19:26:46.750 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.750 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732977597}) 2024-06-11T19:26:46.751 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1795824269}) 2024-06-11T19:26:46.751 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.751 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732977597}) 2024-06-11T19:26:46.751 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.446+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2417841493}) 2024-06-11T19:26:46.751 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.751 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2478549856}) 2024-06-11T19:26:46.751 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1525943436}) 2024-06-11T19:26:46.751 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.751 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2478549856}) 2024-06-11T19:26:46.752 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=702670859}) 2024-06-11T19:26:46.752 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.752 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2478549856}) 2024-06-11T19:26:46.752 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1166613912}) 2024-06-11T19:26:46.752 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.752 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2478549856}) 2024-06-11T19:26:46.752 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3486395475}) 2024-06-11T19:26:46.752 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.753 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2478549856}) 2024-06-11T19:26:46.753 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2548309250}) 2024-06-11T19:26:46.753 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.753 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2478549856}) 2024-06-11T19:26:46.753 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=147519161}) 2024-06-11T19:26:46.753 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.753 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2478549856}) 2024-06-11T19:26:46.753 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1768315713}) 2024-06-11T19:26:46.753 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.754 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2478549856}) 2024-06-11T19:26:46.754 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.447+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4036690688}) 2024-06-11T19:26:46.754 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.754 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596246712}) 2024-06-11T19:26:46.754 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3906688095}) 2024-06-11T19:26:46.754 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.754 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596246712}) 2024-06-11T19:26:46.754 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=703919527}) 2024-06-11T19:26:46.755 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.755 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596246712}) 2024-06-11T19:26:46.755 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2848910467}) 2024-06-11T19:26:46.755 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.755 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596246712}) 2024-06-11T19:26:46.755 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3074777557}) 2024-06-11T19:26:46.755 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.755 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596246712}) 2024-06-11T19:26:46.755 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2907090494}) 2024-06-11T19:26:46.756 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.756 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596246712}) 2024-06-11T19:26:46.756 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=932574761}) 2024-06-11T19:26:46.756 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.756 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596246712}) 2024-06-11T19:26:46.756 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=580726982}) 2024-06-11T19:26:46.756 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.756 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596246712}) 2024-06-11T19:26:46.757 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.448+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=533553349}) 2024-06-11T19:26:46.757 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.757 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845946243}) 2024-06-11T19:26:46.757 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516636888}) 2024-06-11T19:26:46.757 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.757 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845946243}) 2024-06-11T19:26:46.757 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=814173986}) 2024-06-11T19:26:46.757 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.758 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845946243}) 2024-06-11T19:26:46.758 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2919774945}) 2024-06-11T19:26:46.758 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.758 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845946243}) 2024-06-11T19:26:46.758 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2027251279}) 2024-06-11T19:26:46.758 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.758 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845946243}) 2024-06-11T19:26:46.758 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.449+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=747042480}) 2024-06-11T19:26:46.758 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.450+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.759 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.450+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845946243}) 2024-06-11T19:26:46.759 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.450+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2034618110}) 2024-06-11T19:26:46.759 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.450+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.759 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.450+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845946243}) 2024-06-11T19:26:46.759 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.450+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3097304727}) 2024-06-11T19:26:46.759 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.450+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.759 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.450+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845946243}) 2024-06-11T19:26:46.759 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.450+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3873319092}) 2024-06-11T19:26:46.760 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.760 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2144245545}) 2024-06-11T19:26:46.760 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656579619}) 2024-06-11T19:26:46.760 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.760 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2144245545}) 2024-06-11T19:26:46.760 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1186314966}) 2024-06-11T19:26:46.760 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.760 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2144245545}) 2024-06-11T19:26:46.760 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=84878754}) 2024-06-11T19:26:46.761 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.761 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2144245545}) 2024-06-11T19:26:46.761 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1850471603}) 2024-06-11T19:26:46.761 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.761 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2144245545}) 2024-06-11T19:26:46.761 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1719636484}) 2024-06-11T19:26:46.761 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.761 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2144245545}) 2024-06-11T19:26:46.762 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2700505399}) 2024-06-11T19:26:46.762 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.762 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2144245545}) 2024-06-11T19:26:46.762 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3431562702}) 2024-06-11T19:26:46.762 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.762 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2144245545}) 2024-06-11T19:26:46.762 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.451+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2410247520}) 2024-06-11T19:26:46.762 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.762 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=154290632}) 2024-06-11T19:26:46.763 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2368710036}) 2024-06-11T19:26:46.763 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.763 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=154290632}) 2024-06-11T19:26:46.763 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=549886110}) 2024-06-11T19:26:46.763 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.763 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=154290632}) 2024-06-11T19:26:46.763 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=687391487}) 2024-06-11T19:26:46.763 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.764 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=154290632}) 2024-06-11T19:26:46.764 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3764217822}) 2024-06-11T19:26:46.764 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.764 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=154290632}) 2024-06-11T19:26:46.764 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=380570869}) 2024-06-11T19:26:46.764 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.764 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=154290632}) 2024-06-11T19:26:46.764 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3730998444}) 2024-06-11T19:26:46.764 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.765 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=154290632}) 2024-06-11T19:26:46.765 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3261712692}) 2024-06-11T19:26:46.765 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.765 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=154290632}) 2024-06-11T19:26:46.765 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.452+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2693021073}) 2024-06-11T19:26:46.765 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.765 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788575028}) 2024-06-11T19:26:46.765 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3465099270}) 2024-06-11T19:26:46.766 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.766 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788575028}) 2024-06-11T19:26:46.766 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=632441373}) 2024-06-11T19:26:46.766 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.766 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788575028}) 2024-06-11T19:26:46.766 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=883128636}) 2024-06-11T19:26:46.766 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.766 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788575028}) 2024-06-11T19:26:46.767 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3516990009}) 2024-06-11T19:26:46.767 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.767 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788575028}) 2024-06-11T19:26:46.767 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.453+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=410144373}) 2024-06-11T19:26:46.767 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.454+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.767 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.454+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788575028}) 2024-06-11T19:26:46.767 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.454+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2302940500}) 2024-06-11T19:26:46.767 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.454+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.767 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.454+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788575028}) 2024-06-11T19:26:46.768 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.454+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1070869467}) 2024-06-11T19:26:46.768 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.454+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.768 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.454+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788575028}) 2024-06-11T19:26:46.768 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.454+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2074612396}) 2024-06-11T19:26:46.768 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.768 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679765100}) 2024-06-11T19:26:46.768 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99685085}) 2024-06-11T19:26:46.768 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.769 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679765100}) 2024-06-11T19:26:46.769 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4281135127}) 2024-06-11T19:26:46.769 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.769 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679765100}) 2024-06-11T19:26:46.769 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1072585108}) 2024-06-11T19:26:46.769 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.769 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679765100}) 2024-06-11T19:26:46.769 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=893894224}) 2024-06-11T19:26:46.769 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.770 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679765100}) 2024-06-11T19:26:46.770 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=256829887}) 2024-06-11T19:26:46.770 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.770 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679765100}) 2024-06-11T19:26:46.770 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=379805364}) 2024-06-11T19:26:46.770 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.770 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679765100}) 2024-06-11T19:26:46.770 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1285309554}) 2024-06-11T19:26:46.771 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.771 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679765100}) 2024-06-11T19:26:46.771 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.455+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1571123827}) 2024-06-11T19:26:46.771 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.771 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=813559988}) 2024-06-11T19:26:46.771 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856365696}) 2024-06-11T19:26:46.771 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.771 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=813559988}) 2024-06-11T19:26:46.771 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2102702556}) 2024-06-11T19:26:46.771 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.772 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=813559988}) 2024-06-11T19:26:46.772 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4235311210}) 2024-06-11T19:26:46.772 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.772 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=813559988}) 2024-06-11T19:26:46.772 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2839965890}) 2024-06-11T19:26:46.772 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.772 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=813559988}) 2024-06-11T19:26:46.772 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=985337541}) 2024-06-11T19:26:46.772 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.773 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=813559988}) 2024-06-11T19:26:46.773 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=936120869}) 2024-06-11T19:26:46.773 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.773 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=813559988}) 2024-06-11T19:26:46.773 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3235902402}) 2024-06-11T19:26:46.773 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.773 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=813559988}) 2024-06-11T19:26:46.773 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.456+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=545707346}) 2024-06-11T19:26:46.773 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.774 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2661217371}) 2024-06-11T19:26:46.774 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2541740736}) 2024-06-11T19:26:46.774 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.774 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2661217371}) 2024-06-11T19:26:46.774 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2203214742}) 2024-06-11T19:26:46.774 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.774 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2661217371}) 2024-06-11T19:26:46.774 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2438240925}) 2024-06-11T19:26:46.775 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.775 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2661217371}) 2024-06-11T19:26:46.775 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1540781939}) 2024-06-11T19:26:46.775 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.775 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2661217371}) 2024-06-11T19:26:46.775 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2289506965}) 2024-06-11T19:26:46.775 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.775 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2661217371}) 2024-06-11T19:26:46.775 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3512929870}) 2024-06-11T19:26:46.776 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.776 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2661217371}) 2024-06-11T19:26:46.776 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1965294034}) 2024-06-11T19:26:46.776 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.776 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2661217371}) 2024-06-11T19:26:46.776 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.458+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=750203237}) 2024-06-11T19:26:46.776 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.776 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=746477356}) 2024-06-11T19:26:46.776 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170835441}) 2024-06-11T19:26:46.777 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.777 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=746477356}) 2024-06-11T19:26:46.777 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=705707937}) 2024-06-11T19:26:46.777 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.777 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=746477356}) 2024-06-11T19:26:46.777 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3162840039}) 2024-06-11T19:26:46.777 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.777 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=746477356}) 2024-06-11T19:26:46.777 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2749956019}) 2024-06-11T19:26:46.778 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.778 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=746477356}) 2024-06-11T19:26:46.778 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2860216645}) 2024-06-11T19:26:46.778 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.778 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=746477356}) 2024-06-11T19:26:46.778 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3157229507}) 2024-06-11T19:26:46.778 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.778 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=746477356}) 2024-06-11T19:26:46.779 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=175843138}) 2024-06-11T19:26:46.779 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.779 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=746477356}) 2024-06-11T19:26:46.779 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.459+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1623603186}) 2024-06-11T19:26:46.779 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.779 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4246658027}) 2024-06-11T19:26:46.779 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3375525033}) 2024-06-11T19:26:46.779 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.779 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4246658027}) 2024-06-11T19:26:46.780 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1974601133}) 2024-06-11T19:26:46.780 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.780 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4246658027}) 2024-06-11T19:26:46.780 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3126946232}) 2024-06-11T19:26:46.780 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.780 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4246658027}) 2024-06-11T19:26:46.780 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1724730413}) 2024-06-11T19:26:46.780 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.780 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4246658027}) 2024-06-11T19:26:46.781 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2724614419}) 2024-06-11T19:26:46.781 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.781 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4246658027}) 2024-06-11T19:26:46.781 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2383540992}) 2024-06-11T19:26:46.781 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.781 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4246658027}) 2024-06-11T19:26:46.781 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1777758197}) 2024-06-11T19:26:46.781 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.781 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4246658027}) 2024-06-11T19:26:46.782 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.461+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3106955153}) 2024-06-11T19:26:46.782 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.782 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734708230}) 2024-06-11T19:26:46.782 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862452837}) 2024-06-11T19:26:46.782 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.782 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734708230}) 2024-06-11T19:26:46.782 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2990107837}) 2024-06-11T19:26:46.782 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.783 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734708230}) 2024-06-11T19:26:46.783 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2207113887}) 2024-06-11T19:26:46.783 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.783 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734708230}) 2024-06-11T19:26:46.783 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3772208054}) 2024-06-11T19:26:46.783 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.783 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734708230}) 2024-06-11T19:26:46.783 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3631292239}) 2024-06-11T19:26:46.783 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.783 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734708230}) 2024-06-11T19:26:46.784 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3478259765}) 2024-06-11T19:26:46.784 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.784 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734708230}) 2024-06-11T19:26:46.784 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4224728046}) 2024-06-11T19:26:46.784 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.784 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734708230}) 2024-06-11T19:26:46.784 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.462+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1820490132}) 2024-06-11T19:26:46.784 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.784 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3324044113}) 2024-06-11T19:26:46.785 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=216495955}) 2024-06-11T19:26:46.785 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.785 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3324044113}) 2024-06-11T19:26:46.785 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2356518222}) 2024-06-11T19:26:46.785 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.785 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3324044113}) 2024-06-11T19:26:46.785 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=932244689}) 2024-06-11T19:26:46.785 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.785 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3324044113}) 2024-06-11T19:26:46.786 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2700593827}) 2024-06-11T19:26:46.786 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.786 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3324044113}) 2024-06-11T19:26:46.786 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3330959148}) 2024-06-11T19:26:46.786 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.786 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3324044113}) 2024-06-11T19:26:46.786 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=829510993}) 2024-06-11T19:26:46.786 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.786 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.463+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3324044113}) 2024-06-11T19:26:46.787 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.464+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3599635137}) 2024-06-11T19:26:46.787 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.464+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.787 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.464+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3324044113}) 2024-06-11T19:26:46.787 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.464+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3601752877}) 2024-06-11T19:26:46.787 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.787 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181792140}) 2024-06-11T19:26:46.787 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390535877}) 2024-06-11T19:26:46.787 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.787 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181792140}) 2024-06-11T19:26:46.788 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3841399516}) 2024-06-11T19:26:46.788 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.788 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181792140}) 2024-06-11T19:26:46.788 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1451139271}) 2024-06-11T19:26:46.788 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.788 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181792140}) 2024-06-11T19:26:46.788 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1298966679}) 2024-06-11T19:26:46.788 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.788 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181792140}) 2024-06-11T19:26:46.789 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1146312010}) 2024-06-11T19:26:46.789 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.789 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181792140}) 2024-06-11T19:26:46.789 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2054487823}) 2024-06-11T19:26:46.789 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.789 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181792140}) 2024-06-11T19:26:46.789 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=283113770}) 2024-06-11T19:26:46.789 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.789 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181792140}) 2024-06-11T19:26:46.790 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.465+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2087348530}) 2024-06-11T19:26:46.790 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.790 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087737350}) 2024-06-11T19:26:46.790 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1756072685}) 2024-06-11T19:26:46.790 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.790 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087737350}) 2024-06-11T19:26:46.790 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4060022213}) 2024-06-11T19:26:46.790 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.790 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087737350}) 2024-06-11T19:26:46.791 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=366335541}) 2024-06-11T19:26:46.791 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.791 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087737350}) 2024-06-11T19:26:46.791 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1929694087}) 2024-06-11T19:26:46.791 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.791 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087737350}) 2024-06-11T19:26:46.791 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=215766687}) 2024-06-11T19:26:46.791 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.791 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087737350}) 2024-06-11T19:26:46.792 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4232405304}) 2024-06-11T19:26:46.792 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.792 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087737350}) 2024-06-11T19:26:46.792 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3385657684}) 2024-06-11T19:26:46.792 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.466+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.792 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.467+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087737350}) 2024-06-11T19:26:46.792 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.467+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2992787447}) 2024-06-11T19:26:46.792 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.792 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287417486}) 2024-06-11T19:26:46.793 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3037532998}) 2024-06-11T19:26:46.793 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.793 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287417486}) 2024-06-11T19:26:46.793 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2683255641}) 2024-06-11T19:26:46.793 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.793 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287417486}) 2024-06-11T19:26:46.793 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1540833802}) 2024-06-11T19:26:46.793 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.793 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287417486}) 2024-06-11T19:26:46.794 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=360510096}) 2024-06-11T19:26:46.794 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.794 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287417486}) 2024-06-11T19:26:46.794 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=6576642}) 2024-06-11T19:26:46.794 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.794 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287417486}) 2024-06-11T19:26:46.794 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2743539693}) 2024-06-11T19:26:46.794 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.795 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287417486}) 2024-06-11T19:26:46.795 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3855934736}) 2024-06-11T19:26:46.795 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.795 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287417486}) 2024-06-11T19:26:46.795 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.468+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1865102651}) 2024-06-11T19:26:46.795 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.795 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381174953}) 2024-06-11T19:26:46.795 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=887692208}) 2024-06-11T19:26:46.796 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.796 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381174953}) 2024-06-11T19:26:46.796 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316364847}) 2024-06-11T19:26:46.796 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.796 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381174953}) 2024-06-11T19:26:46.796 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=101125994}) 2024-06-11T19:26:46.796 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.796 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381174953}) 2024-06-11T19:26:46.796 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=492759253}) 2024-06-11T19:26:46.797 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.797 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381174953}) 2024-06-11T19:26:46.797 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=159258801}) 2024-06-11T19:26:46.797 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.797 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381174953}) 2024-06-11T19:26:46.797 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=889321290}) 2024-06-11T19:26:46.797 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.797 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381174953}) 2024-06-11T19:26:46.798 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2677150568}) 2024-06-11T19:26:46.798 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.798 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381174953}) 2024-06-11T19:26:46.798 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.469+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3785537127}) 2024-06-11T19:26:46.798 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.470+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.798 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.470+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333950321}) 2024-06-11T19:26:46.798 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.470+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2604139801}) 2024-06-11T19:26:46.798 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.470+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.799 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.470+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333950321}) 2024-06-11T19:26:46.799 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.470+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2263369869}) 2024-06-11T19:26:46.799 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.470+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.799 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.470+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333950321}) 2024-06-11T19:26:46.799 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.470+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=347272342}) 2024-06-11T19:26:46.799 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.470+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.799 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.470+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333950321}) 2024-06-11T19:26:46.799 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.470+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=89284152}) 2024-06-11T19:26:46.800 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.471+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.800 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.471+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333950321}) 2024-06-11T19:26:46.800 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.471+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1183135628}) 2024-06-11T19:26:46.800 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.471+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.800 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.471+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333950321}) 2024-06-11T19:26:46.800 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.471+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3878031503}) 2024-06-11T19:26:46.800 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.471+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.800 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.471+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333950321}) 2024-06-11T19:26:46.800 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.471+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3029811777}) 2024-06-11T19:26:46.801 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.471+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.801 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.471+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333950321}) 2024-06-11T19:26:46.801 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.471+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=738268099}) 2024-06-11T19:26:46.801 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.801 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705244435}) 2024-06-11T19:26:46.801 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924793203}) 2024-06-11T19:26:46.801 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.801 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705244435}) 2024-06-11T19:26:46.802 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2826240556}) 2024-06-11T19:26:46.802 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.802 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705244435}) 2024-06-11T19:26:46.802 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2005523516}) 2024-06-11T19:26:46.802 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.802 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705244435}) 2024-06-11T19:26:46.802 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1817294311}) 2024-06-11T19:26:46.802 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.802 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705244435}) 2024-06-11T19:26:46.803 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1908623969}) 2024-06-11T19:26:46.803 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.803 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705244435}) 2024-06-11T19:26:46.803 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1189596662}) 2024-06-11T19:26:46.803 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.803 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705244435}) 2024-06-11T19:26:46.803 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=897337798}) 2024-06-11T19:26:46.803 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.804 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705244435}) 2024-06-11T19:26:46.804 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.472+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2730272838}) 2024-06-11T19:26:46.804 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.804 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4234013455}) 2024-06-11T19:26:46.804 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2321115620}) 2024-06-11T19:26:46.804 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.804 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4234013455}) 2024-06-11T19:26:46.804 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2189092344}) 2024-06-11T19:26:46.804 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.805 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4234013455}) 2024-06-11T19:26:46.805 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=430939958}) 2024-06-11T19:26:46.805 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.805 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4234013455}) 2024-06-11T19:26:46.805 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=738303383}) 2024-06-11T19:26:46.805 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.805 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4234013455}) 2024-06-11T19:26:46.805 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4112171885}) 2024-06-11T19:26:46.805 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.806 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.473+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4234013455}) 2024-06-11T19:26:46.806 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.474+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=625389758}) 2024-06-11T19:26:46.806 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.474+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.806 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.474+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4234013455}) 2024-06-11T19:26:46.806 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.474+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1113008893}) 2024-06-11T19:26:46.806 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.474+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.806 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.474+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4234013455}) 2024-06-11T19:26:46.807 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.474+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1627729674}) 2024-06-11T19:26:46.807 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.807 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252644674}) 2024-06-11T19:26:46.807 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2840659593}) 2024-06-11T19:26:46.807 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.807 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252644674}) 2024-06-11T19:26:46.807 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1276560414}) 2024-06-11T19:26:46.807 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.808 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252644674}) 2024-06-11T19:26:46.808 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1838714732}) 2024-06-11T19:26:46.808 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.808 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252644674}) 2024-06-11T19:26:46.808 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=812508464}) 2024-06-11T19:26:46.808 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.808 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252644674}) 2024-06-11T19:26:46.808 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2547463782}) 2024-06-11T19:26:46.808 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.809 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252644674}) 2024-06-11T19:26:46.809 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=270363856}) 2024-06-11T19:26:46.809 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.809 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252644674}) 2024-06-11T19:26:46.809 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3686429882}) 2024-06-11T19:26:46.809 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.809 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252644674}) 2024-06-11T19:26:46.809 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.475+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=474482778}) 2024-06-11T19:26:46.810 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.476+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.810 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.476+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27628127}) 2024-06-11T19:26:46.810 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.476+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314091640}) 2024-06-11T19:26:46.810 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.476+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.810 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.476+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27628127}) 2024-06-11T19:26:46.810 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.476+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3723118792}) 2024-06-11T19:26:46.810 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.476+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.810 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.476+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27628127}) 2024-06-11T19:26:46.811 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.476+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2654638666}) 2024-06-11T19:26:46.811 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.476+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.811 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27628127}) 2024-06-11T19:26:46.811 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2576170558}) 2024-06-11T19:26:46.811 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.811 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27628127}) 2024-06-11T19:26:46.811 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2044770657}) 2024-06-11T19:26:46.811 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.812 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27628127}) 2024-06-11T19:26:46.812 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2222500493}) 2024-06-11T19:26:46.812 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.812 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27628127}) 2024-06-11T19:26:46.812 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=153882280}) 2024-06-11T19:26:46.812 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.812 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27628127}) 2024-06-11T19:26:46.812 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.477+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1458109841}) 2024-06-11T19:26:46.812 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.812 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1022368765}) 2024-06-11T19:26:46.813 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3113613295}) 2024-06-11T19:26:46.813 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.813 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1022368765}) 2024-06-11T19:26:46.813 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2141771768}) 2024-06-11T19:26:46.813 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.813 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1022368765}) 2024-06-11T19:26:46.813 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3563706196}) 2024-06-11T19:26:46.813 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.814 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1022368765}) 2024-06-11T19:26:46.814 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3607211898}) 2024-06-11T19:26:46.814 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.814 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1022368765}) 2024-06-11T19:26:46.814 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2021932768}) 2024-06-11T19:26:46.814 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.814 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1022368765}) 2024-06-11T19:26:46.814 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2325700056}) 2024-06-11T19:26:46.815 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.815 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1022368765}) 2024-06-11T19:26:46.815 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2829385221}) 2024-06-11T19:26:46.815 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.815 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1022368765}) 2024-06-11T19:26:46.815 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.478+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2079259946}) 2024-06-11T19:26:46.815 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.815 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557053213}) 2024-06-11T19:26:46.815 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440176215}) 2024-06-11T19:26:46.816 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.816 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557053213}) 2024-06-11T19:26:46.816 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=396775158}) 2024-06-11T19:26:46.816 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.816 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557053213}) 2024-06-11T19:26:46.816 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2401759864}) 2024-06-11T19:26:46.816 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.816 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557053213}) 2024-06-11T19:26:46.816 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3537004976}) 2024-06-11T19:26:46.817 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.817 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557053213}) 2024-06-11T19:26:46.817 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3286657242}) 2024-06-11T19:26:46.817 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.817 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557053213}) 2024-06-11T19:26:46.817 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2278051018}) 2024-06-11T19:26:46.817 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.817 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557053213}) 2024-06-11T19:26:46.817 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2268776421}) 2024-06-11T19:26:46.818 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.818 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557053213}) 2024-06-11T19:26:46.818 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.479+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2587489605}) 2024-06-11T19:26:46.818 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.818 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45811802}) 2024-06-11T19:26:46.818 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160125710}) 2024-06-11T19:26:46.818 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.818 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45811802}) 2024-06-11T19:26:46.819 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3519774546}) 2024-06-11T19:26:46.819 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.819 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45811802}) 2024-06-11T19:26:46.819 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1469853988}) 2024-06-11T19:26:46.819 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.819 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45811802}) 2024-06-11T19:26:46.819 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=722757920}) 2024-06-11T19:26:46.819 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.820 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45811802}) 2024-06-11T19:26:46.820 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3593907729}) 2024-06-11T19:26:46.820 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.820 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45811802}) 2024-06-11T19:26:46.820 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1270751023}) 2024-06-11T19:26:46.820 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.820 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45811802}) 2024-06-11T19:26:46.820 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2251647900}) 2024-06-11T19:26:46.821 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.821 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45811802}) 2024-06-11T19:26:46.821 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.481+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1724790811}) 2024-06-11T19:26:46.821 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.821 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614248246}) 2024-06-11T19:26:46.821 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2126075289}) 2024-06-11T19:26:46.821 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.821 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614248246}) 2024-06-11T19:26:46.821 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928630041}) 2024-06-11T19:26:46.822 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.822 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614248246}) 2024-06-11T19:26:46.822 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=554993814}) 2024-06-11T19:26:46.822 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.822 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614248246}) 2024-06-11T19:26:46.822 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1822917536}) 2024-06-11T19:26:46.822 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.822 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614248246}) 2024-06-11T19:26:46.823 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=19397538}) 2024-06-11T19:26:46.823 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.823 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614248246}) 2024-06-11T19:26:46.823 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3173771493}) 2024-06-11T19:26:46.823 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.823 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614248246}) 2024-06-11T19:26:46.823 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1671375433}) 2024-06-11T19:26:46.823 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.823 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614248246}) 2024-06-11T19:26:46.824 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.483+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3428281527}) 2024-06-11T19:26:46.824 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.824 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641941479}) 2024-06-11T19:26:46.824 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1796156472}) 2024-06-11T19:26:46.824 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.824 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641941479}) 2024-06-11T19:26:46.824 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3740243201}) 2024-06-11T19:26:46.824 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.825 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641941479}) 2024-06-11T19:26:46.825 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2590559198}) 2024-06-11T19:26:46.825 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.825 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641941479}) 2024-06-11T19:26:46.825 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2702644806}) 2024-06-11T19:26:46.825 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.825 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641941479}) 2024-06-11T19:26:46.825 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=877576078}) 2024-06-11T19:26:46.825 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.826 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641941479}) 2024-06-11T19:26:46.826 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=6507647}) 2024-06-11T19:26:46.826 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.826 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641941479}) 2024-06-11T19:26:46.826 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3496093230}) 2024-06-11T19:26:46.826 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.826 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641941479}) 2024-06-11T19:26:46.826 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.484+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=190760632}) 2024-06-11T19:26:46.827 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.827 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685751331}) 2024-06-11T19:26:46.827 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1064841261}) 2024-06-11T19:26:46.827 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.827 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685751331}) 2024-06-11T19:26:46.827 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4210472548}) 2024-06-11T19:26:46.827 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.827 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685751331}) 2024-06-11T19:26:46.828 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3480163250}) 2024-06-11T19:26:46.828 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.828 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685751331}) 2024-06-11T19:26:46.828 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3433209629}) 2024-06-11T19:26:46.828 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.828 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685751331}) 2024-06-11T19:26:46.828 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=622475602}) 2024-06-11T19:26:46.828 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.828 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685751331}) 2024-06-11T19:26:46.829 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=261086122}) 2024-06-11T19:26:46.829 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.829 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685751331}) 2024-06-11T19:26:46.829 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3861641502}) 2024-06-11T19:26:46.829 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.829 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685751331}) 2024-06-11T19:26:46.829 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.485+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1800478127}) 2024-06-11T19:26:46.829 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.486+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.830 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.486+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166187747}) 2024-06-11T19:26:46.830 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.486+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2815390940}) 2024-06-11T19:26:46.830 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.486+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.830 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166187747}) 2024-06-11T19:26:46.830 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107045249}) 2024-06-11T19:26:46.830 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.830 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166187747}) 2024-06-11T19:26:46.830 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2085332776}) 2024-06-11T19:26:46.831 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.831 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166187747}) 2024-06-11T19:26:46.831 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3553511433}) 2024-06-11T19:26:46.831 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.831 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166187747}) 2024-06-11T19:26:46.831 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1542044423}) 2024-06-11T19:26:46.831 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.831 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166187747}) 2024-06-11T19:26:46.831 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=581791897}) 2024-06-11T19:26:46.832 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.832 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166187747}) 2024-06-11T19:26:46.832 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1918752716}) 2024-06-11T19:26:46.832 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.832 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166187747}) 2024-06-11T19:26:46.832 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.487+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=957316486}) 2024-06-11T19:26:46.832 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.832 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481335317}) 2024-06-11T19:26:46.833 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2788549325}) 2024-06-11T19:26:46.833 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.833 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481335317}) 2024-06-11T19:26:46.833 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1951335033}) 2024-06-11T19:26:46.833 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.833 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481335317}) 2024-06-11T19:26:46.833 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2232008295}) 2024-06-11T19:26:46.833 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.834 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481335317}) 2024-06-11T19:26:46.834 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1484873182}) 2024-06-11T19:26:46.834 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.834 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481335317}) 2024-06-11T19:26:46.834 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3598004059}) 2024-06-11T19:26:46.834 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.834 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481335317}) 2024-06-11T19:26:46.834 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3170451635}) 2024-06-11T19:26:46.835 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.835 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481335317}) 2024-06-11T19:26:46.835 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2745587321}) 2024-06-11T19:26:46.835 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.835 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481335317}) 2024-06-11T19:26:46.835 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.488+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2024164304}) 2024-06-11T19:26:46.835 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.835 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363363736}) 2024-06-11T19:26:46.836 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2492032824}) 2024-06-11T19:26:46.836 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.836 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363363736}) 2024-06-11T19:26:46.836 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=315736261}) 2024-06-11T19:26:46.836 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.836 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363363736}) 2024-06-11T19:26:46.836 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=499479995}) 2024-06-11T19:26:46.836 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.836 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363363736}) 2024-06-11T19:26:46.837 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=966241892}) 2024-06-11T19:26:46.837 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.837 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363363736}) 2024-06-11T19:26:46.837 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2089890901}) 2024-06-11T19:26:46.837 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.837 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363363736}) 2024-06-11T19:26:46.837 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=646144825}) 2024-06-11T19:26:46.837 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.837 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363363736}) 2024-06-11T19:26:46.838 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3513861777}) 2024-06-11T19:26:46.838 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.838 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363363736}) 2024-06-11T19:26:46.838 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.489+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=354282902}) 2024-06-11T19:26:46.838 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.838 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625157048}) 2024-06-11T19:26:46.838 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=508537833}) 2024-06-11T19:26:46.838 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.839 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625157048}) 2024-06-11T19:26:46.839 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3868598735}) 2024-06-11T19:26:46.839 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.839 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625157048}) 2024-06-11T19:26:46.839 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2231202672}) 2024-06-11T19:26:46.839 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.839 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625157048}) 2024-06-11T19:26:46.839 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4127729477}) 2024-06-11T19:26:46.839 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.840 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.490+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625157048}) 2024-06-11T19:26:46.840 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.491+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2438580864}) 2024-06-11T19:26:46.840 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.491+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.840 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.491+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625157048}) 2024-06-11T19:26:46.840 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.491+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3373950032}) 2024-06-11T19:26:46.840 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.491+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.840 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.491+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625157048}) 2024-06-11T19:26:46.840 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.491+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3930046630}) 2024-06-11T19:26:46.840 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.491+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.841 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.491+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625157048}) 2024-06-11T19:26:46.841 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.491+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3170980455}) 2024-06-11T19:26:46.841 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.841 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494371333}) 2024-06-11T19:26:46.841 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421825978}) 2024-06-11T19:26:46.841 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.841 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494371333}) 2024-06-11T19:26:46.841 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=961563408}) 2024-06-11T19:26:46.841 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.841 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494371333}) 2024-06-11T19:26:46.842 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3793047050}) 2024-06-11T19:26:46.842 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.842 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494371333}) 2024-06-11T19:26:46.842 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=923288047}) 2024-06-11T19:26:46.842 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.842 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494371333}) 2024-06-11T19:26:46.842 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3109800118}) 2024-06-11T19:26:46.842 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.843 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494371333}) 2024-06-11T19:26:46.843 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=723291310}) 2024-06-11T19:26:46.843 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.843 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494371333}) 2024-06-11T19:26:46.843 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3397788196}) 2024-06-11T19:26:46.843 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.843 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494371333}) 2024-06-11T19:26:46.843 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.492+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2825057556}) 2024-06-11T19:26:46.843 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.844 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280784526}) 2024-06-11T19:26:46.844 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884990669}) 2024-06-11T19:26:46.844 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.844 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280784526}) 2024-06-11T19:26:46.844 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3523838074}) 2024-06-11T19:26:46.844 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.844 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280784526}) 2024-06-11T19:26:46.844 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=78728679}) 2024-06-11T19:26:46.845 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.845 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280784526}) 2024-06-11T19:26:46.845 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2636327033}) 2024-06-11T19:26:46.845 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.845 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280784526}) 2024-06-11T19:26:46.845 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3979454971}) 2024-06-11T19:26:46.845 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.846 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280784526}) 2024-06-11T19:26:46.846 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=686930091}) 2024-06-11T19:26:46.846 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.846 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280784526}) 2024-06-11T19:26:46.846 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=142204042}) 2024-06-11T19:26:46.846 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.846 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280784526}) 2024-06-11T19:26:46.846 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.493+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2038172856}) 2024-06-11T19:26:46.847 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.847 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442202929}) 2024-06-11T19:26:46.847 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106858573}) 2024-06-11T19:26:46.847 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.847 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442202929}) 2024-06-11T19:26:46.847 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1502384624}) 2024-06-11T19:26:46.847 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.847 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442202929}) 2024-06-11T19:26:46.848 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3299244792}) 2024-06-11T19:26:46.848 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.848 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442202929}) 2024-06-11T19:26:46.848 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1646780410}) 2024-06-11T19:26:46.848 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.848 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442202929}) 2024-06-11T19:26:46.848 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3554040411}) 2024-06-11T19:26:46.848 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.848 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442202929}) 2024-06-11T19:26:46.849 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2754469115}) 2024-06-11T19:26:46.849 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.849 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442202929}) 2024-06-11T19:26:46.849 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1506771083}) 2024-06-11T19:26:46.849 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.849 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442202929}) 2024-06-11T19:26:46.849 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.495+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2271332005}) 2024-06-11T19:26:46.849 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.850 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623929762}) 2024-06-11T19:26:46.850 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3192953690}) 2024-06-11T19:26:46.850 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.850 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623929762}) 2024-06-11T19:26:46.850 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=729014687}) 2024-06-11T19:26:46.850 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.850 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623929762}) 2024-06-11T19:26:46.850 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3528514665}) 2024-06-11T19:26:46.851 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.851 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623929762}) 2024-06-11T19:26:46.851 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=53545304}) 2024-06-11T19:26:46.851 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.851 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623929762}) 2024-06-11T19:26:46.851 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=241327405}) 2024-06-11T19:26:46.851 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.851 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623929762}) 2024-06-11T19:26:46.852 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=287750009}) 2024-06-11T19:26:46.852 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.852 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623929762}) 2024-06-11T19:26:46.852 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2813896113}) 2024-06-11T19:26:46.852 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.852 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623929762}) 2024-06-11T19:26:46.852 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.496+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=151663629}) 2024-06-11T19:26:46.852 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.853 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871841546}) 2024-06-11T19:26:46.853 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858214858}) 2024-06-11T19:26:46.853 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.853 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871841546}) 2024-06-11T19:26:46.853 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=84265587}) 2024-06-11T19:26:46.853 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.853 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871841546}) 2024-06-11T19:26:46.853 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=416474587}) 2024-06-11T19:26:46.854 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.854 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871841546}) 2024-06-11T19:26:46.854 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=76922634}) 2024-06-11T19:26:46.854 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.854 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871841546}) 2024-06-11T19:26:46.854 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1577851145}) 2024-06-11T19:26:46.854 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.854 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871841546}) 2024-06-11T19:26:46.854 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=838073660}) 2024-06-11T19:26:46.855 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.855 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871841546}) 2024-06-11T19:26:46.855 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3505473595}) 2024-06-11T19:26:46.855 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.855 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871841546}) 2024-06-11T19:26:46.855 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.497+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2131423330}) 2024-06-11T19:26:46.855 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.498+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.855 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.498+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605137243}) 2024-06-11T19:26:46.855 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.498+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565717158}) 2024-06-11T19:26:46.856 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.498+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.856 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.498+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605137243}) 2024-06-11T19:26:46.856 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.498+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=161000355}) 2024-06-11T19:26:46.856 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.498+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.856 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.498+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605137243}) 2024-06-11T19:26:46.856 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.498+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1402403751}) 2024-06-11T19:26:46.856 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.498+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.856 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605137243}) 2024-06-11T19:26:46.856 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2331445257}) 2024-06-11T19:26:46.857 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.857 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605137243}) 2024-06-11T19:26:46.857 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1128751152}) 2024-06-11T19:26:46.857 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.857 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605137243}) 2024-06-11T19:26:46.857 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2468697796}) 2024-06-11T19:26:46.857 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.857 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605137243}) 2024-06-11T19:26:46.857 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3122047467}) 2024-06-11T19:26:46.858 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.858 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605137243}) 2024-06-11T19:26:46.858 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.499+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=31542558}) 2024-06-11T19:26:46.858 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.858 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984888557}) 2024-06-11T19:26:46.858 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1293317985}) 2024-06-11T19:26:46.858 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.858 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984888557}) 2024-06-11T19:26:46.858 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2795727787}) 2024-06-11T19:26:46.859 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.859 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984888557}) 2024-06-11T19:26:46.859 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2189042958}) 2024-06-11T19:26:46.859 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.859 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984888557}) 2024-06-11T19:26:46.859 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=293752949}) 2024-06-11T19:26:46.859 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.859 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984888557}) 2024-06-11T19:26:46.860 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2274341060}) 2024-06-11T19:26:46.860 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.860 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984888557}) 2024-06-11T19:26:46.860 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3603684548}) 2024-06-11T19:26:46.860 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.860 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984888557}) 2024-06-11T19:26:46.860 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3375268391}) 2024-06-11T19:26:46.860 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.860 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984888557}) 2024-06-11T19:26:46.861 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.500+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=303494567}) 2024-06-11T19:26:46.861 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.861 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322259972}) 2024-06-11T19:26:46.861 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132124089}) 2024-06-11T19:26:46.861 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.861 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322259972}) 2024-06-11T19:26:46.861 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1839124242}) 2024-06-11T19:26:46.861 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.861 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322259972}) 2024-06-11T19:26:46.862 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3550228733}) 2024-06-11T19:26:46.862 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.862 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322259972}) 2024-06-11T19:26:46.862 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2656183321}) 2024-06-11T19:26:46.862 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.862 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322259972}) 2024-06-11T19:26:46.862 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4144161620}) 2024-06-11T19:26:46.862 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.862 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322259972}) 2024-06-11T19:26:46.863 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1297777032}) 2024-06-11T19:26:46.863 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.863 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322259972}) 2024-06-11T19:26:46.863 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1075756478}) 2024-06-11T19:26:46.863 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.863 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322259972}) 2024-06-11T19:26:46.863 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.501+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1021680448}) 2024-06-11T19:26:46.863 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.864 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605280527}) 2024-06-11T19:26:46.864 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997471495}) 2024-06-11T19:26:46.864 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.864 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605280527}) 2024-06-11T19:26:46.864 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1319900188}) 2024-06-11T19:26:46.864 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.864 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605280527}) 2024-06-11T19:26:46.864 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=42184220}) 2024-06-11T19:26:46.865 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.865 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605280527}) 2024-06-11T19:26:46.865 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3711089585}) 2024-06-11T19:26:46.865 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.865 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605280527}) 2024-06-11T19:26:46.865 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.502+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3573612244}) 2024-06-11T19:26:46.865 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.503+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.865 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.503+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605280527}) 2024-06-11T19:26:46.865 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.503+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1354661459}) 2024-06-11T19:26:46.866 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.503+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.866 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.503+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605280527}) 2024-06-11T19:26:46.866 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.503+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1678836593}) 2024-06-11T19:26:46.866 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.503+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.866 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.503+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605280527}) 2024-06-11T19:26:46.866 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.503+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=897272018}) 2024-06-11T19:26:46.866 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.866 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765974432}) 2024-06-11T19:26:46.867 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779981161}) 2024-06-11T19:26:46.867 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.867 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765974432}) 2024-06-11T19:26:46.867 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=231224573}) 2024-06-11T19:26:46.867 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.867 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765974432}) 2024-06-11T19:26:46.867 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1701696088}) 2024-06-11T19:26:46.867 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.868 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765974432}) 2024-06-11T19:26:46.868 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2593463906}) 2024-06-11T19:26:46.868 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.868 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765974432}) 2024-06-11T19:26:46.868 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2321071186}) 2024-06-11T19:26:46.868 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.868 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765974432}) 2024-06-11T19:26:46.868 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2254882832}) 2024-06-11T19:26:46.868 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.869 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765974432}) 2024-06-11T19:26:46.869 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2092391363}) 2024-06-11T19:26:46.869 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.869 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765974432}) 2024-06-11T19:26:46.869 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.504+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2621964427}) 2024-06-11T19:26:46.869 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.869 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334038435}) 2024-06-11T19:26:46.869 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2494291280}) 2024-06-11T19:26:46.870 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.870 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334038435}) 2024-06-11T19:26:46.870 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2919713230}) 2024-06-11T19:26:46.870 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.870 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334038435}) 2024-06-11T19:26:46.870 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2910751589}) 2024-06-11T19:26:46.870 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.870 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334038435}) 2024-06-11T19:26:46.871 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=751067836}) 2024-06-11T19:26:46.871 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.871 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334038435}) 2024-06-11T19:26:46.871 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=5553501}) 2024-06-11T19:26:46.871 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.871 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334038435}) 2024-06-11T19:26:46.871 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3704861624}) 2024-06-11T19:26:46.871 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.871 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334038435}) 2024-06-11T19:26:46.872 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2440014254}) 2024-06-11T19:26:46.872 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.872 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334038435}) 2024-06-11T19:26:46.872 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.505+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1767044548}) 2024-06-11T19:26:46.872 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.506+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.872 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.506+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37814938}) 2024-06-11T19:26:46.872 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.506+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4162197744}) 2024-06-11T19:26:46.872 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.873 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37814938}) 2024-06-11T19:26:46.873 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=662739472}) 2024-06-11T19:26:46.873 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.873 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37814938}) 2024-06-11T19:26:46.873 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=660260502}) 2024-06-11T19:26:46.873 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.873 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37814938}) 2024-06-11T19:26:46.873 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2750545008}) 2024-06-11T19:26:46.874 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.874 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37814938}) 2024-06-11T19:26:46.874 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2029579418}) 2024-06-11T19:26:46.874 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.874 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37814938}) 2024-06-11T19:26:46.874 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=835208764}) 2024-06-11T19:26:46.874 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.874 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37814938}) 2024-06-11T19:26:46.875 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1144571643}) 2024-06-11T19:26:46.875 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.875 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37814938}) 2024-06-11T19:26:46.875 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.507+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1928131576}) 2024-06-11T19:26:46.875 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.875 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726609416}) 2024-06-11T19:26:46.875 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481569665}) 2024-06-11T19:26:46.875 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.876 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726609416}) 2024-06-11T19:26:46.876 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=49595363}) 2024-06-11T19:26:46.876 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.876 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726609416}) 2024-06-11T19:26:46.876 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4283962444}) 2024-06-11T19:26:46.876 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.876 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726609416}) 2024-06-11T19:26:46.877 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2268868704}) 2024-06-11T19:26:46.877 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.877 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726609416}) 2024-06-11T19:26:46.877 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3434549804}) 2024-06-11T19:26:46.877 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.877 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726609416}) 2024-06-11T19:26:46.877 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1175003247}) 2024-06-11T19:26:46.877 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.878 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726609416}) 2024-06-11T19:26:46.878 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1989294592}) 2024-06-11T19:26:46.878 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.878 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726609416}) 2024-06-11T19:26:46.878 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.508+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4075758811}) 2024-06-11T19:26:46.878 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.878 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419659301}) 2024-06-11T19:26:46.878 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921997002}) 2024-06-11T19:26:46.879 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.879 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419659301}) 2024-06-11T19:26:46.879 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3663895326}) 2024-06-11T19:26:46.879 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.879 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419659301}) 2024-06-11T19:26:46.879 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2813011983}) 2024-06-11T19:26:46.879 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.879 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419659301}) 2024-06-11T19:26:46.879 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=380039351}) 2024-06-11T19:26:46.880 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.880 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419659301}) 2024-06-11T19:26:46.880 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3278482416}) 2024-06-11T19:26:46.880 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.880 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419659301}) 2024-06-11T19:26:46.880 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1903099861}) 2024-06-11T19:26:46.880 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.880 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419659301}) 2024-06-11T19:26:46.881 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2469404600}) 2024-06-11T19:26:46.881 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.881 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419659301}) 2024-06-11T19:26:46.881 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.509+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2751585642}) 2024-06-11T19:26:46.881 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.510+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.881 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.510+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735803337}) 2024-06-11T19:26:46.881 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.510+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=271465104}) 2024-06-11T19:26:46.881 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.510+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.881 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.510+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735803337}) 2024-06-11T19:26:46.882 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.510+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=80780916}) 2024-06-11T19:26:46.882 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.510+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.882 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.510+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735803337}) 2024-06-11T19:26:46.882 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=781270945}) 2024-06-11T19:26:46.882 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.882 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735803337}) 2024-06-11T19:26:46.882 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=859892878}) 2024-06-11T19:26:46.882 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.883 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735803337}) 2024-06-11T19:26:46.883 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2127051392}) 2024-06-11T19:26:46.883 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.883 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735803337}) 2024-06-11T19:26:46.883 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2131927885}) 2024-06-11T19:26:46.883 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.883 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735803337}) 2024-06-11T19:26:46.883 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2799494466}) 2024-06-11T19:26:46.884 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.884 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735803337}) 2024-06-11T19:26:46.884 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.511+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1982683763}) 2024-06-11T19:26:46.884 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.884 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913067776}) 2024-06-11T19:26:46.884 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595860534}) 2024-06-11T19:26:46.884 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.884 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913067776}) 2024-06-11T19:26:46.885 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=357869769}) 2024-06-11T19:26:46.885 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.885 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913067776}) 2024-06-11T19:26:46.885 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3893426752}) 2024-06-11T19:26:46.885 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.885 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913067776}) 2024-06-11T19:26:46.885 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=284252107}) 2024-06-11T19:26:46.885 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.885 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913067776}) 2024-06-11T19:26:46.886 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3428955301}) 2024-06-11T19:26:46.886 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.886 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913067776}) 2024-06-11T19:26:46.886 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1063566534}) 2024-06-11T19:26:46.886 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.886 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913067776}) 2024-06-11T19:26:46.886 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2141517508}) 2024-06-11T19:26:46.886 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.887 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1913067776}) 2024-06-11T19:26:46.887 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.512+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3280072818}) 2024-06-11T19:26:46.887 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.887 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353707771}) 2024-06-11T19:26:46.887 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3195785219}) 2024-06-11T19:26:46.887 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.887 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353707771}) 2024-06-11T19:26:46.887 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1648461678}) 2024-06-11T19:26:46.887 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.887 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353707771}) 2024-06-11T19:26:46.888 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3268142823}) 2024-06-11T19:26:46.888 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.888 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353707771}) 2024-06-11T19:26:46.888 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=408231051}) 2024-06-11T19:26:46.888 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.888 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353707771}) 2024-06-11T19:26:46.888 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=476823816}) 2024-06-11T19:26:46.888 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.888 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353707771}) 2024-06-11T19:26:46.889 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=660782228}) 2024-06-11T19:26:46.889 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.889 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353707771}) 2024-06-11T19:26:46.889 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1745000937}) 2024-06-11T19:26:46.889 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.889 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353707771}) 2024-06-11T19:26:46.889 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.514+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2509044059}) 2024-06-11T19:26:46.889 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.889 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946188300}) 2024-06-11T19:26:46.890 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512336998}) 2024-06-11T19:26:46.890 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.890 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946188300}) 2024-06-11T19:26:46.890 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1945038682}) 2024-06-11T19:26:46.890 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.890 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946188300}) 2024-06-11T19:26:46.890 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1423691975}) 2024-06-11T19:26:46.890 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.891 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946188300}) 2024-06-11T19:26:46.891 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=888247667}) 2024-06-11T19:26:46.891 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.891 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946188300}) 2024-06-11T19:26:46.891 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3333375934}) 2024-06-11T19:26:46.891 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.891 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946188300}) 2024-06-11T19:26:46.891 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2301115268}) 2024-06-11T19:26:46.891 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.892 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946188300}) 2024-06-11T19:26:46.892 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4082922091}) 2024-06-11T19:26:46.892 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.892 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946188300}) 2024-06-11T19:26:46.892 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.515+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3191416930}) 2024-06-11T19:26:46.892 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.892 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238754586}) 2024-06-11T19:26:46.892 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086620561}) 2024-06-11T19:26:46.892 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.893 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238754586}) 2024-06-11T19:26:46.893 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149546329}) 2024-06-11T19:26:46.893 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.893 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238754586}) 2024-06-11T19:26:46.893 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=467191657}) 2024-06-11T19:26:46.893 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.893 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238754586}) 2024-06-11T19:26:46.893 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=95730414}) 2024-06-11T19:26:46.893 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.894 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238754586}) 2024-06-11T19:26:46.894 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2965483364}) 2024-06-11T19:26:46.894 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.894 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238754586}) 2024-06-11T19:26:46.894 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3704772324}) 2024-06-11T19:26:46.894 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.894 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238754586}) 2024-06-11T19:26:46.894 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3754265563}) 2024-06-11T19:26:46.895 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.895 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238754586}) 2024-06-11T19:26:46.895 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.516+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1716517886}) 2024-06-11T19:26:46.895 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.517+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.895 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.517+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787004264}) 2024-06-11T19:26:46.895 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.517+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353956902}) 2024-06-11T19:26:46.895 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.517+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.895 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.517+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787004264}) 2024-06-11T19:26:46.896 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.517+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1159376937}) 2024-06-11T19:26:46.896 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.517+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.896 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787004264}) 2024-06-11T19:26:46.896 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=19438422}) 2024-06-11T19:26:46.896 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.896 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787004264}) 2024-06-11T19:26:46.896 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4272941924}) 2024-06-11T19:26:46.896 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.897 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787004264}) 2024-06-11T19:26:46.897 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2086349587}) 2024-06-11T19:26:46.897 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.897 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787004264}) 2024-06-11T19:26:46.897 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2552306341}) 2024-06-11T19:26:46.897 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.897 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787004264}) 2024-06-11T19:26:46.897 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=837068968}) 2024-06-11T19:26:46.898 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.898 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787004264}) 2024-06-11T19:26:46.898 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.518+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3763839631}) 2024-06-11T19:26:46.898 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.898 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186723838}) 2024-06-11T19:26:46.898 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2763575918}) 2024-06-11T19:26:46.898 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.898 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186723838}) 2024-06-11T19:26:46.899 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2841880524}) 2024-06-11T19:26:46.899 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.899 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186723838}) 2024-06-11T19:26:46.899 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=788598293}) 2024-06-11T19:26:46.899 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.899 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186723838}) 2024-06-11T19:26:46.899 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3241030027}) 2024-06-11T19:26:46.899 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.900 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186723838}) 2024-06-11T19:26:46.900 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2507919800}) 2024-06-11T19:26:46.900 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.900 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186723838}) 2024-06-11T19:26:46.900 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1570364091}) 2024-06-11T19:26:46.900 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.900 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186723838}) 2024-06-11T19:26:46.900 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3325870555}) 2024-06-11T19:26:46.901 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.901 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186723838}) 2024-06-11T19:26:46.901 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.519+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3033378760}) 2024-06-11T19:26:46.901 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.520+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.901 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.520+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905449124}) 2024-06-11T19:26:46.901 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.520+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684142386}) 2024-06-11T19:26:46.901 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.520+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.901 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.520+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905449124}) 2024-06-11T19:26:46.902 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.520+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3636039091}) 2024-06-11T19:26:46.902 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.520+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.902 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905449124}) 2024-06-11T19:26:46.902 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2120207328}) 2024-06-11T19:26:46.902 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.902 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905449124}) 2024-06-11T19:26:46.902 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2331700662}) 2024-06-11T19:26:46.903 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.903 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905449124}) 2024-06-11T19:26:46.903 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4229737573}) 2024-06-11T19:26:46.903 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.903 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905449124}) 2024-06-11T19:26:46.903 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1004601587}) 2024-06-11T19:26:46.903 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.903 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905449124}) 2024-06-11T19:26:46.903 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2243417590}) 2024-06-11T19:26:46.904 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.904 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905449124}) 2024-06-11T19:26:46.904 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.521+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2390761264}) 2024-06-11T19:26:46.904 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.904 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532831411}) 2024-06-11T19:26:46.904 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3600968138}) 2024-06-11T19:26:46.904 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.904 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532831411}) 2024-06-11T19:26:46.905 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3903806965}) 2024-06-11T19:26:46.905 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.905 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532831411}) 2024-06-11T19:26:46.905 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=762796492}) 2024-06-11T19:26:46.905 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.905 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532831411}) 2024-06-11T19:26:46.905 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1263091833}) 2024-06-11T19:26:46.905 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.905 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532831411}) 2024-06-11T19:26:46.906 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3174406414}) 2024-06-11T19:26:46.906 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.906 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532831411}) 2024-06-11T19:26:46.906 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=13760670}) 2024-06-11T19:26:46.906 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.906 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532831411}) 2024-06-11T19:26:46.906 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=392007922}) 2024-06-11T19:26:46.906 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.907 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.522+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532831411}) 2024-06-11T19:26:46.907 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.523+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2227002134}) 2024-06-11T19:26:46.907 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.523+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.907 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201327475}) 2024-06-11T19:26:46.907 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489437565}) 2024-06-11T19:26:46.907 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.907 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201327475}) 2024-06-11T19:26:46.907 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3480399165}) 2024-06-11T19:26:46.907 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.908 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201327475}) 2024-06-11T19:26:46.908 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=199492109}) 2024-06-11T19:26:46.908 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.908 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201327475}) 2024-06-11T19:26:46.908 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=968062730}) 2024-06-11T19:26:46.908 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.908 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201327475}) 2024-06-11T19:26:46.908 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1746140378}) 2024-06-11T19:26:46.908 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.909 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201327475}) 2024-06-11T19:26:46.909 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2686874203}) 2024-06-11T19:26:46.909 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.909 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201327475}) 2024-06-11T19:26:46.909 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=124275128}) 2024-06-11T19:26:46.909 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.909 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201327475}) 2024-06-11T19:26:46.909 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.524+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1633468317}) 2024-06-11T19:26:46.909 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.910 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067345923}) 2024-06-11T19:26:46.910 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4280378018}) 2024-06-11T19:26:46.910 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.910 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067345923}) 2024-06-11T19:26:46.910 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443792307}) 2024-06-11T19:26:46.910 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.910 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067345923}) 2024-06-11T19:26:46.910 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3619260817}) 2024-06-11T19:26:46.911 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.911 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067345923}) 2024-06-11T19:26:46.911 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2191503830}) 2024-06-11T19:26:46.911 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.911 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067345923}) 2024-06-11T19:26:46.911 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3881313542}) 2024-06-11T19:26:46.911 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.911 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067345923}) 2024-06-11T19:26:46.912 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1525839430}) 2024-06-11T19:26:46.912 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.912 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067345923}) 2024-06-11T19:26:46.912 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3544309296}) 2024-06-11T19:26:46.912 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.912 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067345923}) 2024-06-11T19:26:46.912 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.525+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2962602176}) 2024-06-11T19:26:46.912 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.912 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105599072}) 2024-06-11T19:26:46.913 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906613355}) 2024-06-11T19:26:46.913 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.913 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105599072}) 2024-06-11T19:26:46.913 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2638312711}) 2024-06-11T19:26:46.913 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.913 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105599072}) 2024-06-11T19:26:46.913 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3919977069}) 2024-06-11T19:26:46.913 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.914 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105599072}) 2024-06-11T19:26:46.914 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=851636301}) 2024-06-11T19:26:46.914 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.914 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105599072}) 2024-06-11T19:26:46.914 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=672315056}) 2024-06-11T19:26:46.914 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.914 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105599072}) 2024-06-11T19:26:46.914 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3270077328}) 2024-06-11T19:26:46.915 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.915 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105599072}) 2024-06-11T19:26:46.915 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3123404102}) 2024-06-11T19:26:46.915 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.915 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105599072}) 2024-06-11T19:26:46.915 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.527+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=722668944}) 2024-06-11T19:26:46.915 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.915 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483256416}) 2024-06-11T19:26:46.915 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824642993}) 2024-06-11T19:26:46.916 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.916 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483256416}) 2024-06-11T19:26:46.916 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=66252470}) 2024-06-11T19:26:46.916 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.916 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483256416}) 2024-06-11T19:26:46.916 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4478989}) 2024-06-11T19:26:46.916 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.916 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483256416}) 2024-06-11T19:26:46.917 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3007119637}) 2024-06-11T19:26:46.917 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.917 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483256416}) 2024-06-11T19:26:46.917 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3434979988}) 2024-06-11T19:26:46.917 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.917 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483256416}) 2024-06-11T19:26:46.917 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3109511930}) 2024-06-11T19:26:46.917 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.918 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483256416}) 2024-06-11T19:26:46.918 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=124985455}) 2024-06-11T19:26:46.918 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.918 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483256416}) 2024-06-11T19:26:46.918 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.528+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2221245981}) 2024-06-11T19:26:46.918 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.918 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555657632}) 2024-06-11T19:26:46.918 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=922056667}) 2024-06-11T19:26:46.919 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.919 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555657632}) 2024-06-11T19:26:46.919 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1910112811}) 2024-06-11T19:26:46.919 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.919 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555657632}) 2024-06-11T19:26:46.919 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2441378626}) 2024-06-11T19:26:46.919 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.919 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555657632}) 2024-06-11T19:26:46.919 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1671589731}) 2024-06-11T19:26:46.920 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.920 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555657632}) 2024-06-11T19:26:46.920 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3185697506}) 2024-06-11T19:26:46.920 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.920 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555657632}) 2024-06-11T19:26:46.920 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1840077759}) 2024-06-11T19:26:46.920 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.920 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555657632}) 2024-06-11T19:26:46.920 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=989838237}) 2024-06-11T19:26:46.921 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.921 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555657632}) 2024-06-11T19:26:46.921 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.530+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3954233151}) 2024-06-11T19:26:46.921 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.921 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185355025}) 2024-06-11T19:26:46.921 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271243936}) 2024-06-11T19:26:46.921 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.921 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185355025}) 2024-06-11T19:26:46.921 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=218965338}) 2024-06-11T19:26:46.922 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.922 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185355025}) 2024-06-11T19:26:46.922 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1744422937}) 2024-06-11T19:26:46.922 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.922 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185355025}) 2024-06-11T19:26:46.922 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1695144770}) 2024-06-11T19:26:46.922 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.922 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185355025}) 2024-06-11T19:26:46.923 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1197273772}) 2024-06-11T19:26:46.923 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.923 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185355025}) 2024-06-11T19:26:46.923 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1749999939}) 2024-06-11T19:26:46.923 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.923 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185355025}) 2024-06-11T19:26:46.923 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3006408410}) 2024-06-11T19:26:46.923 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.923 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185355025}) 2024-06-11T19:26:46.923 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.531+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=419321968}) 2024-06-11T19:26:46.924 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.924 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972201892}) 2024-06-11T19:26:46.924 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4115150928}) 2024-06-11T19:26:46.924 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.924 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972201892}) 2024-06-11T19:26:46.924 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=459956956}) 2024-06-11T19:26:46.924 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.924 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972201892}) 2024-06-11T19:26:46.925 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2048457482}) 2024-06-11T19:26:46.925 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.925 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972201892}) 2024-06-11T19:26:46.925 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=291352924}) 2024-06-11T19:26:46.925 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.925 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972201892}) 2024-06-11T19:26:46.925 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=371233436}) 2024-06-11T19:26:46.925 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.925 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972201892}) 2024-06-11T19:26:46.926 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3378599396}) 2024-06-11T19:26:46.926 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.926 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972201892}) 2024-06-11T19:26:46.926 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2334325320}) 2024-06-11T19:26:46.926 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.926 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972201892}) 2024-06-11T19:26:46.926 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.533+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=954548570}) 2024-06-11T19:26:46.926 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.927 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448530270}) 2024-06-11T19:26:46.927 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235284347}) 2024-06-11T19:26:46.927 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.927 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448530270}) 2024-06-11T19:26:46.927 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1426967733}) 2024-06-11T19:26:46.927 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.927 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448530270}) 2024-06-11T19:26:46.927 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=536061946}) 2024-06-11T19:26:46.928 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.928 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448530270}) 2024-06-11T19:26:46.928 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2629554641}) 2024-06-11T19:26:46.928 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.928 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448530270}) 2024-06-11T19:26:46.928 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.534+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1700232831}) 2024-06-11T19:26:46.928 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.535+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.928 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.535+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448530270}) 2024-06-11T19:26:46.929 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.535+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3888492359}) 2024-06-11T19:26:46.929 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.535+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.929 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.535+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448530270}) 2024-06-11T19:26:46.929 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.535+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1753466068}) 2024-06-11T19:26:46.929 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.535+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.929 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.535+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448530270}) 2024-06-11T19:26:46.929 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.535+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4010583422}) 2024-06-11T19:26:46.929 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.929 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2079433535}) 2024-06-11T19:26:46.930 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684590320}) 2024-06-11T19:26:46.930 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.930 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2079433535}) 2024-06-11T19:26:46.930 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=459281475}) 2024-06-11T19:26:46.930 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.930 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2079433535}) 2024-06-11T19:26:46.930 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1148531952}) 2024-06-11T19:26:46.930 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.931 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2079433535}) 2024-06-11T19:26:46.931 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=75445232}) 2024-06-11T19:26:46.931 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.931 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2079433535}) 2024-06-11T19:26:46.931 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=942843340}) 2024-06-11T19:26:46.931 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.931 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2079433535}) 2024-06-11T19:26:46.931 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1099063721}) 2024-06-11T19:26:46.932 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.932 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2079433535}) 2024-06-11T19:26:46.932 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2355559592}) 2024-06-11T19:26:46.932 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.932 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2079433535}) 2024-06-11T19:26:46.932 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.536+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1663129598}) 2024-06-11T19:26:46.932 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.932 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184021113}) 2024-06-11T19:26:46.933 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624350407}) 2024-06-11T19:26:46.933 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.933 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184021113}) 2024-06-11T19:26:46.933 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1581402920}) 2024-06-11T19:26:46.933 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.933 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184021113}) 2024-06-11T19:26:46.933 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4113841300}) 2024-06-11T19:26:46.933 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.934 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184021113}) 2024-06-11T19:26:46.934 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=656588528}) 2024-06-11T19:26:46.934 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.934 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184021113}) 2024-06-11T19:26:46.934 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3138479382}) 2024-06-11T19:26:46.934 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.934 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184021113}) 2024-06-11T19:26:46.934 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2529806019}) 2024-06-11T19:26:46.935 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.935 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184021113}) 2024-06-11T19:26:46.935 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=35019792}) 2024-06-11T19:26:46.935 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.935 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184021113}) 2024-06-11T19:26:46.935 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.537+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2176558938}) 2024-06-11T19:26:46.935 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.935 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560532770}) 2024-06-11T19:26:46.936 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3778206381}) 2024-06-11T19:26:46.936 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.936 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560532770}) 2024-06-11T19:26:46.936 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1536446365}) 2024-06-11T19:26:46.936 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.936 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560532770}) 2024-06-11T19:26:46.936 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2201161343}) 2024-06-11T19:26:46.936 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.936 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560532770}) 2024-06-11T19:26:46.937 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3159553756}) 2024-06-11T19:26:46.937 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.937 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.538+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560532770}) 2024-06-11T19:26:46.937 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.539+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3932831115}) 2024-06-11T19:26:46.937 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.539+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.937 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.539+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560532770}) 2024-06-11T19:26:46.937 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.539+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3178408538}) 2024-06-11T19:26:46.937 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.539+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.937 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.539+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560532770}) 2024-06-11T19:26:46.938 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.539+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2922893964}) 2024-06-11T19:26:46.938 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.539+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.938 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.539+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560532770}) 2024-06-11T19:26:46.938 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.539+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=970164126}) 2024-06-11T19:26:46.938 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.938 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087141819}) 2024-06-11T19:26:46.938 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3978058145}) 2024-06-11T19:26:46.938 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.939 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087141819}) 2024-06-11T19:26:46.939 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3084191925}) 2024-06-11T19:26:46.939 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.939 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087141819}) 2024-06-11T19:26:46.939 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=872738999}) 2024-06-11T19:26:46.939 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.939 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087141819}) 2024-06-11T19:26:46.939 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4095387109}) 2024-06-11T19:26:46.940 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.940 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087141819}) 2024-06-11T19:26:46.940 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2376785016}) 2024-06-11T19:26:46.940 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.940 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087141819}) 2024-06-11T19:26:46.940 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4140255453}) 2024-06-11T19:26:46.940 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.940 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087141819}) 2024-06-11T19:26:46.940 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2633077}) 2024-06-11T19:26:46.941 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.941 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087141819}) 2024-06-11T19:26:46.941 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.540+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2763882803}) 2024-06-11T19:26:46.941 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.541+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.941 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.541+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195888020}) 2024-06-11T19:26:46.941 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.541+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3581328121}) 2024-06-11T19:26:46.941 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.541+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.941 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.541+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195888020}) 2024-06-11T19:26:46.942 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.541+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2975894360}) 2024-06-11T19:26:46.942 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.942 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195888020}) 2024-06-11T19:26:46.942 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2734970848}) 2024-06-11T19:26:46.942 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.942 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195888020}) 2024-06-11T19:26:46.942 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4279931790}) 2024-06-11T19:26:46.942 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.943 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195888020}) 2024-06-11T19:26:46.943 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1680235524}) 2024-06-11T19:26:46.943 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.943 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195888020}) 2024-06-11T19:26:46.943 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3554271558}) 2024-06-11T19:26:46.943 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.943 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195888020}) 2024-06-11T19:26:46.943 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1558640407}) 2024-06-11T19:26:46.943 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.944 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195888020}) 2024-06-11T19:26:46.944 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.542+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1657816539}) 2024-06-11T19:26:46.944 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.944 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872183328}) 2024-06-11T19:26:46.944 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3644400956}) 2024-06-11T19:26:46.944 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.944 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872183328}) 2024-06-11T19:26:46.944 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2321977210}) 2024-06-11T19:26:46.945 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.945 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872183328}) 2024-06-11T19:26:46.945 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1407349788}) 2024-06-11T19:26:46.945 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.945 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872183328}) 2024-06-11T19:26:46.945 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3541380340}) 2024-06-11T19:26:46.945 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.946 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872183328}) 2024-06-11T19:26:46.946 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=648950464}) 2024-06-11T19:26:46.946 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.946 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872183328}) 2024-06-11T19:26:46.946 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1342442396}) 2024-06-11T19:26:46.946 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.946 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872183328}) 2024-06-11T19:26:46.946 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2868077902}) 2024-06-11T19:26:46.947 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.947 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872183328}) 2024-06-11T19:26:46.947 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.543+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1421311996}) 2024-06-11T19:26:46.947 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.947 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=481593367}) 2024-06-11T19:26:46.947 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4194378393}) 2024-06-11T19:26:46.947 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.947 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=481593367}) 2024-06-11T19:26:46.948 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1042634301}) 2024-06-11T19:26:46.948 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.948 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=481593367}) 2024-06-11T19:26:46.948 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2578063792}) 2024-06-11T19:26:46.948 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.948 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=481593367}) 2024-06-11T19:26:46.948 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=241007400}) 2024-06-11T19:26:46.948 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.948 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=481593367}) 2024-06-11T19:26:46.949 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=297780682}) 2024-06-11T19:26:46.949 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.949 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=481593367}) 2024-06-11T19:26:46.949 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2938109398}) 2024-06-11T19:26:46.949 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.949 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=481593367}) 2024-06-11T19:26:46.949 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=935044089}) 2024-06-11T19:26:46.949 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.950 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=481593367}) 2024-06-11T19:26:46.950 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.544+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3992770098}) 2024-06-11T19:26:46.950 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.545+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.950 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.545+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971847177}) 2024-06-11T19:26:46.950 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.545+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1876894630}) 2024-06-11T19:26:46.950 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.545+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.950 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971847177}) 2024-06-11T19:26:46.950 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=227245424}) 2024-06-11T19:26:46.951 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.951 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971847177}) 2024-06-11T19:26:46.951 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1431602223}) 2024-06-11T19:26:46.951 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.951 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971847177}) 2024-06-11T19:26:46.951 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=422152180}) 2024-06-11T19:26:46.951 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.951 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971847177}) 2024-06-11T19:26:46.952 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3025772532}) 2024-06-11T19:26:46.952 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.952 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971847177}) 2024-06-11T19:26:46.952 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1650387399}) 2024-06-11T19:26:46.952 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.952 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971847177}) 2024-06-11T19:26:46.952 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2523385880}) 2024-06-11T19:26:46.952 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.952 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3971847177}) 2024-06-11T19:26:46.953 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.546+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3644426289}) 2024-06-11T19:26:46.953 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.953 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1738474858}) 2024-06-11T19:26:46.953 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2002693626}) 2024-06-11T19:26:46.953 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.953 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1738474858}) 2024-06-11T19:26:46.953 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1159413417}) 2024-06-11T19:26:46.953 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.954 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1738474858}) 2024-06-11T19:26:46.954 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1472696899}) 2024-06-11T19:26:46.954 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.954 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1738474858}) 2024-06-11T19:26:46.954 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1453134031}) 2024-06-11T19:26:46.954 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.954 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1738474858}) 2024-06-11T19:26:46.954 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4078235841}) 2024-06-11T19:26:46.955 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.955 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1738474858}) 2024-06-11T19:26:46.955 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1471905120}) 2024-06-11T19:26:46.955 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.955 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1738474858}) 2024-06-11T19:26:46.955 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4244219032}) 2024-06-11T19:26:46.955 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.955 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1738474858}) 2024-06-11T19:26:46.956 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.547+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2505047563}) 2024-06-11T19:26:46.956 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.548+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.956 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.548+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244915208}) 2024-06-11T19:26:46.956 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.548+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446523776}) 2024-06-11T19:26:46.956 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.548+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.956 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.548+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244915208}) 2024-06-11T19:26:46.956 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.548+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2000062427}) 2024-06-11T19:26:46.956 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.548+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.957 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.548+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244915208}) 2024-06-11T19:26:46.957 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.548+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2759438818}) 2024-06-11T19:26:46.957 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.957 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244915208}) 2024-06-11T19:26:46.957 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2665040796}) 2024-06-11T19:26:46.957 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.957 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244915208}) 2024-06-11T19:26:46.957 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2119257130}) 2024-06-11T19:26:46.957 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.958 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244915208}) 2024-06-11T19:26:46.958 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=291341069}) 2024-06-11T19:26:46.958 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.958 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244915208}) 2024-06-11T19:26:46.958 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4044748546}) 2024-06-11T19:26:46.958 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.958 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244915208}) 2024-06-11T19:26:46.958 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.549+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=861535843}) 2024-06-11T19:26:46.958 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.959 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4142643075}) 2024-06-11T19:26:46.959 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087718377}) 2024-06-11T19:26:46.959 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.959 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4142643075}) 2024-06-11T19:26:46.959 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3153307132}) 2024-06-11T19:26:46.959 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.959 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4142643075}) 2024-06-11T19:26:46.959 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1211762596}) 2024-06-11T19:26:46.960 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.960 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4142643075}) 2024-06-11T19:26:46.960 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=313826812}) 2024-06-11T19:26:46.960 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.960 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4142643075}) 2024-06-11T19:26:46.960 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3844293033}) 2024-06-11T19:26:46.960 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.960 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4142643075}) 2024-06-11T19:26:46.960 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1083940093}) 2024-06-11T19:26:46.961 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.961 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4142643075}) 2024-06-11T19:26:46.961 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1067306280}) 2024-06-11T19:26:46.961 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.961 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4142643075}) 2024-06-11T19:26:46.961 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.550+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=574977335}) 2024-06-11T19:26:46.961 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.551+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.961 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.551+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149082926}) 2024-06-11T19:26:46.961 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.551+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2282446247}) 2024-06-11T19:26:46.962 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.551+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.962 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.551+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149082926}) 2024-06-11T19:26:46.962 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.551+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1719414811}) 2024-06-11T19:26:46.962 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.551+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.962 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149082926}) 2024-06-11T19:26:46.962 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=382698073}) 2024-06-11T19:26:46.962 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.962 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149082926}) 2024-06-11T19:26:46.963 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2408712014}) 2024-06-11T19:26:46.963 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.963 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149082926}) 2024-06-11T19:26:46.963 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3644290249}) 2024-06-11T19:26:46.963 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.963 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149082926}) 2024-06-11T19:26:46.963 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2399491251}) 2024-06-11T19:26:46.963 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.964 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149082926}) 2024-06-11T19:26:46.964 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3925446890}) 2024-06-11T19:26:46.964 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.964 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149082926}) 2024-06-11T19:26:46.964 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.552+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2734733038}) 2024-06-11T19:26:46.964 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.964 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4286482600}) 2024-06-11T19:26:46.964 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465641171}) 2024-06-11T19:26:46.964 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.965 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4286482600}) 2024-06-11T19:26:46.965 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1156726996}) 2024-06-11T19:26:46.965 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.965 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4286482600}) 2024-06-11T19:26:46.965 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2564306795}) 2024-06-11T19:26:46.965 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.965 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4286482600}) 2024-06-11T19:26:46.965 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1533004212}) 2024-06-11T19:26:46.966 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.966 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4286482600}) 2024-06-11T19:26:46.966 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1242116476}) 2024-06-11T19:26:46.966 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.966 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4286482600}) 2024-06-11T19:26:46.966 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3866766774}) 2024-06-11T19:26:46.966 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.966 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4286482600}) 2024-06-11T19:26:46.967 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=180587896}) 2024-06-11T19:26:46.967 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.967 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4286482600}) 2024-06-11T19:26:46.967 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.553+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=283024138}) 2024-06-11T19:26:46.967 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.967 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217842866}) 2024-06-11T19:26:46.967 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355182532}) 2024-06-11T19:26:46.968 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.968 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217842866}) 2024-06-11T19:26:46.968 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2737907500}) 2024-06-11T19:26:46.968 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.968 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217842866}) 2024-06-11T19:26:46.968 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1567522826}) 2024-06-11T19:26:46.968 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.968 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217842866}) 2024-06-11T19:26:46.968 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=370059014}) 2024-06-11T19:26:46.968 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.554+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.969 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.555+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217842866}) 2024-06-11T19:26:46.969 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.555+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1769759214}) 2024-06-11T19:26:46.969 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.555+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.969 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.555+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217842866}) 2024-06-11T19:26:46.969 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.555+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3181919867}) 2024-06-11T19:26:46.969 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.555+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.969 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.555+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217842866}) 2024-06-11T19:26:46.969 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.555+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3141963773}) 2024-06-11T19:26:46.970 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.555+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.970 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.555+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217842866}) 2024-06-11T19:26:46.970 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.555+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1688264504}) 2024-06-11T19:26:46.970 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.970 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541710935}) 2024-06-11T19:26:46.970 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741608669}) 2024-06-11T19:26:46.970 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.970 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541710935}) 2024-06-11T19:26:46.970 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3909956582}) 2024-06-11T19:26:46.971 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.971 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541710935}) 2024-06-11T19:26:46.971 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1042919715}) 2024-06-11T19:26:46.971 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.971 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541710935}) 2024-06-11T19:26:46.971 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=990915518}) 2024-06-11T19:26:46.971 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.971 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541710935}) 2024-06-11T19:26:46.971 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=783030982}) 2024-06-11T19:26:46.972 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.972 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541710935}) 2024-06-11T19:26:46.972 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1186921457}) 2024-06-11T19:26:46.972 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.972 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541710935}) 2024-06-11T19:26:46.972 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=525289838}) 2024-06-11T19:26:46.972 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.972 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541710935}) 2024-06-11T19:26:46.972 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.556+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=113450650}) 2024-06-11T19:26:46.973 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.973 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255664054}) 2024-06-11T19:26:46.973 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=776376897}) 2024-06-11T19:26:46.973 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.973 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255664054}) 2024-06-11T19:26:46.973 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=93092374}) 2024-06-11T19:26:46.973 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.973 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255664054}) 2024-06-11T19:26:46.974 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3581114953}) 2024-06-11T19:26:46.974 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.974 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255664054}) 2024-06-11T19:26:46.974 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1882514179}) 2024-06-11T19:26:46.974 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.974 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255664054}) 2024-06-11T19:26:46.974 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1604394402}) 2024-06-11T19:26:46.974 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.975 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255664054}) 2024-06-11T19:26:46.975 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2278178224}) 2024-06-11T19:26:46.975 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.975 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255664054}) 2024-06-11T19:26:46.975 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=267189730}) 2024-06-11T19:26:46.975 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.975 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255664054}) 2024-06-11T19:26:46.975 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.557+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1941798439}) 2024-06-11T19:26:46.975 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.976 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1008892844}) 2024-06-11T19:26:46.976 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2189475728}) 2024-06-11T19:26:46.976 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.976 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1008892844}) 2024-06-11T19:26:46.976 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2790549714}) 2024-06-11T19:26:46.976 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.976 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1008892844}) 2024-06-11T19:26:46.976 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2288951257}) 2024-06-11T19:26:46.976 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.977 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1008892844}) 2024-06-11T19:26:46.977 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2267528192}) 2024-06-11T19:26:46.977 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.977 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1008892844}) 2024-06-11T19:26:46.977 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3017097868}) 2024-06-11T19:26:46.977 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.977 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1008892844}) 2024-06-11T19:26:46.977 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1320051996}) 2024-06-11T19:26:46.977 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.978 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1008892844}) 2024-06-11T19:26:46.978 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=818601001}) 2024-06-11T19:26:46.978 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.978 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1008892844}) 2024-06-11T19:26:46.978 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.559+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2336070661}) 2024-06-11T19:26:46.978 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.560+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.978 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.560+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673901652}) 2024-06-11T19:26:46.978 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.560+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2591148930}) 2024-06-11T19:26:46.979 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.560+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.979 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.560+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673901652}) 2024-06-11T19:26:46.979 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.560+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2454056247}) 2024-06-11T19:26:46.979 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.560+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.979 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673901652}) 2024-06-11T19:26:46.979 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3318307673}) 2024-06-11T19:26:46.979 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.979 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673901652}) 2024-06-11T19:26:46.979 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=455152424}) 2024-06-11T19:26:46.979 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.980 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673901652}) 2024-06-11T19:26:46.980 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=32894858}) 2024-06-11T19:26:46.980 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.980 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673901652}) 2024-06-11T19:26:46.980 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2561312785}) 2024-06-11T19:26:46.980 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.980 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673901652}) 2024-06-11T19:26:46.980 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1013939978}) 2024-06-11T19:26:46.980 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.981 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673901652}) 2024-06-11T19:26:46.981 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.561+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2466569636}) 2024-06-11T19:26:46.981 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.981 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555892285}) 2024-06-11T19:26:46.981 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1677187074}) 2024-06-11T19:26:46.981 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.981 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555892285}) 2024-06-11T19:26:46.981 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2880137542}) 2024-06-11T19:26:46.981 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.982 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555892285}) 2024-06-11T19:26:46.982 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=438831829}) 2024-06-11T19:26:46.982 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.982 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555892285}) 2024-06-11T19:26:46.982 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3978311893}) 2024-06-11T19:26:46.982 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.982 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555892285}) 2024-06-11T19:26:46.982 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=193013951}) 2024-06-11T19:26:46.983 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.983 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555892285}) 2024-06-11T19:26:46.983 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2787191747}) 2024-06-11T19:26:46.983 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.983 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555892285}) 2024-06-11T19:26:46.983 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3592604680}) 2024-06-11T19:26:46.983 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.983 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555892285}) 2024-06-11T19:26:46.983 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.562+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=464395131}) 2024-06-11T19:26:46.984 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.984 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610539943}) 2024-06-11T19:26:46.984 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470938540}) 2024-06-11T19:26:46.984 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.984 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610539943}) 2024-06-11T19:26:46.984 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1163836183}) 2024-06-11T19:26:46.984 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.984 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610539943}) 2024-06-11T19:26:46.984 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3184293772}) 2024-06-11T19:26:46.984 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.985 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610539943}) 2024-06-11T19:26:46.985 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1062267929}) 2024-06-11T19:26:46.985 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.985 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610539943}) 2024-06-11T19:26:46.985 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1042731817}) 2024-06-11T19:26:46.985 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.985 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610539943}) 2024-06-11T19:26:46.985 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3827781403}) 2024-06-11T19:26:46.985 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.986 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610539943}) 2024-06-11T19:26:46.986 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.563+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1565651557}) 2024-06-11T19:26:46.986 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.564+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.986 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.564+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610539943}) 2024-06-11T19:26:46.986 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.564+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2548808776}) 2024-06-11T19:26:46.986 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.986 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975607165}) 2024-06-11T19:26:46.986 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479359574}) 2024-06-11T19:26:46.987 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.987 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975607165}) 2024-06-11T19:26:46.987 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2319729901}) 2024-06-11T19:26:46.987 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.987 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975607165}) 2024-06-11T19:26:46.987 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=537789569}) 2024-06-11T19:26:46.987 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.987 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975607165}) 2024-06-11T19:26:46.987 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1424406883}) 2024-06-11T19:26:46.987 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.988 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975607165}) 2024-06-11T19:26:46.988 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=87421426}) 2024-06-11T19:26:46.988 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.988 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975607165}) 2024-06-11T19:26:46.988 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1294044110}) 2024-06-11T19:26:46.988 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.988 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975607165}) 2024-06-11T19:26:46.988 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3286845262}) 2024-06-11T19:26:46.988 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.989 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975607165}) 2024-06-11T19:26:46.989 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.565+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1242427086}) 2024-06-11T19:26:46.989 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.989 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466311144}) 2024-06-11T19:26:46.989 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=327481312}) 2024-06-11T19:26:46.989 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.989 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466311144}) 2024-06-11T19:26:46.989 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3042008941}) 2024-06-11T19:26:46.989 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.990 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466311144}) 2024-06-11T19:26:46.990 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4056554641}) 2024-06-11T19:26:46.990 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.990 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466311144}) 2024-06-11T19:26:46.990 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3171734882}) 2024-06-11T19:26:46.990 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.990 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466311144}) 2024-06-11T19:26:46.990 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1019058790}) 2024-06-11T19:26:46.990 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.991 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466311144}) 2024-06-11T19:26:46.991 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3298900885}) 2024-06-11T19:26:46.991 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.991 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466311144}) 2024-06-11T19:26:46.991 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2797177588}) 2024-06-11T19:26:46.991 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.991 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466311144}) 2024-06-11T19:26:46.991 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.566+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1855260509}) 2024-06-11T19:26:46.991 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.992 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524811122}) 2024-06-11T19:26:46.992 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117263892}) 2024-06-11T19:26:46.992 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.992 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524811122}) 2024-06-11T19:26:46.992 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2490185564}) 2024-06-11T19:26:46.992 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.992 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524811122}) 2024-06-11T19:26:46.992 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1361204257}) 2024-06-11T19:26:46.992 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.993 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524811122}) 2024-06-11T19:26:46.993 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4118478715}) 2024-06-11T19:26:46.993 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.993 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524811122}) 2024-06-11T19:26:46.993 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4280794057}) 2024-06-11T19:26:46.993 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.993 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524811122}) 2024-06-11T19:26:46.993 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4134999251}) 2024-06-11T19:26:46.993 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.994 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524811122}) 2024-06-11T19:26:46.994 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3077056647}) 2024-06-11T19:26:46.994 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.994 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524811122}) 2024-06-11T19:26:46.994 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.567+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=847013367}) 2024-06-11T19:26:46.994 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.994 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3671548623}) 2024-06-11T19:26:46.994 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1051009879}) 2024-06-11T19:26:46.994 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.995 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3671548623}) 2024-06-11T19:26:46.995 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2886780730}) 2024-06-11T19:26:46.995 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.995 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3671548623}) 2024-06-11T19:26:46.995 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1842601596}) 2024-06-11T19:26:46.995 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.995 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3671548623}) 2024-06-11T19:26:46.995 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2251887608}) 2024-06-11T19:26:46.995 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.996 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3671548623}) 2024-06-11T19:26:46.996 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1567074111}) 2024-06-11T19:26:46.996 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.996 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3671548623}) 2024-06-11T19:26:46.996 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1054148868}) 2024-06-11T19:26:46.996 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.996 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3671548623}) 2024-06-11T19:26:46.996 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1056930363}) 2024-06-11T19:26:46.996 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.997 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3671548623}) 2024-06-11T19:26:46.997 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.569+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3207737929}) 2024-06-11T19:26:46.997 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.997 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730475698}) 2024-06-11T19:26:46.997 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360786707}) 2024-06-11T19:26:46.997 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.997 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730475698}) 2024-06-11T19:26:46.997 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2718132107}) 2024-06-11T19:26:46.998 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.998 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730475698}) 2024-06-11T19:26:46.998 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3570512231}) 2024-06-11T19:26:46.998 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.998 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730475698}) 2024-06-11T19:26:46.998 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1679165196}) 2024-06-11T19:26:46.998 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.998 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730475698}) 2024-06-11T19:26:46.999 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3224259104}) 2024-06-11T19:26:46.999 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.999 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730475698}) 2024-06-11T19:26:46.999 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3600559598}) 2024-06-11T19:26:46.999 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.999 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730475698}) 2024-06-11T19:26:46.999 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3242139130}) 2024-06-11T19:26:46.999 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:46.999 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730475698}) 2024-06-11T19:26:47.000 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.570+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=805205820}) 2024-06-11T19:26:47.000 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.571+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.000 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.571+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950043034}) 2024-06-11T19:26:47.000 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.571+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3760543542}) 2024-06-11T19:26:47.000 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.571+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.000 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.571+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950043034}) 2024-06-11T19:26:47.000 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.571+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1578659514}) 2024-06-11T19:26:47.000 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.571+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.000 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.571+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950043034}) 2024-06-11T19:26:47.001 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.571+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3166342152}) 2024-06-11T19:26:47.001 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.001 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950043034}) 2024-06-11T19:26:47.001 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2387224021}) 2024-06-11T19:26:47.001 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.001 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950043034}) 2024-06-11T19:26:47.001 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3109355558}) 2024-06-11T19:26:47.001 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.001 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950043034}) 2024-06-11T19:26:47.002 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3765336549}) 2024-06-11T19:26:47.002 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.002 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950043034}) 2024-06-11T19:26:47.002 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2142178017}) 2024-06-11T19:26:47.002 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.002 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950043034}) 2024-06-11T19:26:47.002 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.572+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=941872667}) 2024-06-11T19:26:47.002 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.003 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434114754}) 2024-06-11T19:26:47.003 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320922260}) 2024-06-11T19:26:47.003 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.003 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434114754}) 2024-06-11T19:26:47.003 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2628340852}) 2024-06-11T19:26:47.003 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.003 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434114754}) 2024-06-11T19:26:47.003 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2982975495}) 2024-06-11T19:26:47.003 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.004 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434114754}) 2024-06-11T19:26:47.004 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3446977301}) 2024-06-11T19:26:47.004 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.004 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434114754}) 2024-06-11T19:26:47.004 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2548320309}) 2024-06-11T19:26:47.004 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.004 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434114754}) 2024-06-11T19:26:47.004 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2314714781}) 2024-06-11T19:26:47.004 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.005 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434114754}) 2024-06-11T19:26:47.005 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1164089594}) 2024-06-11T19:26:47.005 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.005 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434114754}) 2024-06-11T19:26:47.005 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.573+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2751624109}) 2024-06-11T19:26:47.005 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.005 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248571121}) 2024-06-11T19:26:47.005 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2986824157}) 2024-06-11T19:26:47.005 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.006 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248571121}) 2024-06-11T19:26:47.006 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3080567872}) 2024-06-11T19:26:47.006 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.006 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248571121}) 2024-06-11T19:26:47.006 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3864144120}) 2024-06-11T19:26:47.006 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.006 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248571121}) 2024-06-11T19:26:47.006 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2285804788}) 2024-06-11T19:26:47.007 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.007 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248571121}) 2024-06-11T19:26:47.007 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4256008077}) 2024-06-11T19:26:47.007 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.007 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248571121}) 2024-06-11T19:26:47.007 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3764708927}) 2024-06-11T19:26:47.007 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.007 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248571121}) 2024-06-11T19:26:47.007 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2885949488}) 2024-06-11T19:26:47.007 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.008 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248571121}) 2024-06-11T19:26:47.008 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.575+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2988960977}) 2024-06-11T19:26:47.008 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.008 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290886761}) 2024-06-11T19:26:47.008 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4270836473}) 2024-06-11T19:26:47.008 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.008 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290886761}) 2024-06-11T19:26:47.008 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1674869159}) 2024-06-11T19:26:47.008 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.009 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290886761}) 2024-06-11T19:26:47.009 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1628177696}) 2024-06-11T19:26:47.009 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.009 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290886761}) 2024-06-11T19:26:47.009 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1841430066}) 2024-06-11T19:26:47.009 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.009 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290886761}) 2024-06-11T19:26:47.009 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3424010617}) 2024-06-11T19:26:47.009 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.010 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290886761}) 2024-06-11T19:26:47.010 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2540926207}) 2024-06-11T19:26:47.010 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.010 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290886761}) 2024-06-11T19:26:47.010 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=298524909}) 2024-06-11T19:26:47.010 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.010 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290886761}) 2024-06-11T19:26:47.010 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.576+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4148362300}) 2024-06-11T19:26:47.011 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.577+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.011 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1800518849}) 2024-06-11T19:26:47.011 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2351722082}) 2024-06-11T19:26:47.011 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.014 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1800518849}) 2024-06-11T19:26:47.014 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330378840}) 2024-06-11T19:26:47.014 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.014 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1800518849}) 2024-06-11T19:26:47.014 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3493817432}) 2024-06-11T19:26:47.014 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.015 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1800518849}) 2024-06-11T19:26:47.015 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2005104836}) 2024-06-11T19:26:47.015 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.015 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1800518849}) 2024-06-11T19:26:47.015 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=877991909}) 2024-06-11T19:26:47.015 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.015 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1800518849}) 2024-06-11T19:26:47.015 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=459524465}) 2024-06-11T19:26:47.015 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.016 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1800518849}) 2024-06-11T19:26:47.016 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3381586428}) 2024-06-11T19:26:47.016 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.016 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1800518849}) 2024-06-11T19:26:47.016 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.578+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1970113546}) 2024-06-11T19:26:47.016 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.016 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629060569}) 2024-06-11T19:26:47.016 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2015895451}) 2024-06-11T19:26:47.016 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.017 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629060569}) 2024-06-11T19:26:47.017 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3809924388}) 2024-06-11T19:26:47.017 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.017 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629060569}) 2024-06-11T19:26:47.017 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=96476489}) 2024-06-11T19:26:47.017 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.017 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629060569}) 2024-06-11T19:26:47.017 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2040333538}) 2024-06-11T19:26:47.018 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.018 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629060569}) 2024-06-11T19:26:47.018 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=384179629}) 2024-06-11T19:26:47.018 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.018 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629060569}) 2024-06-11T19:26:47.018 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4251523405}) 2024-06-11T19:26:47.018 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.018 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629060569}) 2024-06-11T19:26:47.018 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4288881815}) 2024-06-11T19:26:47.019 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.019 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629060569}) 2024-06-11T19:26:47.019 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.579+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2769908599}) 2024-06-11T19:26:47.019 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.019 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38405616}) 2024-06-11T19:26:47.019 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2146057319}) 2024-06-11T19:26:47.019 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.019 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38405616}) 2024-06-11T19:26:47.020 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3608803669}) 2024-06-11T19:26:47.020 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.020 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38405616}) 2024-06-11T19:26:47.020 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3369955084}) 2024-06-11T19:26:47.020 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.020 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38405616}) 2024-06-11T19:26:47.020 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3760184178}) 2024-06-11T19:26:47.020 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.020 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38405616}) 2024-06-11T19:26:47.021 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1416022130}) 2024-06-11T19:26:47.021 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.021 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38405616}) 2024-06-11T19:26:47.021 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4248746102}) 2024-06-11T19:26:47.021 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.021 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38405616}) 2024-06-11T19:26:47.021 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1523532236}) 2024-06-11T19:26:47.021 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.022 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38405616}) 2024-06-11T19:26:47.022 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.581+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4031204397}) 2024-06-11T19:26:47.022 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.022 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1229308076}) 2024-06-11T19:26:47.022 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3634967908}) 2024-06-11T19:26:47.022 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.022 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1229308076}) 2024-06-11T19:26:47.022 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=975714084}) 2024-06-11T19:26:47.023 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.023 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1229308076}) 2024-06-11T19:26:47.023 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1245578926}) 2024-06-11T19:26:47.023 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.023 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1229308076}) 2024-06-11T19:26:47.023 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3230777716}) 2024-06-11T19:26:47.023 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.023 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1229308076}) 2024-06-11T19:26:47.024 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1308988441}) 2024-06-11T19:26:47.024 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.024 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1229308076}) 2024-06-11T19:26:47.024 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=193392713}) 2024-06-11T19:26:47.024 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.024 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1229308076}) 2024-06-11T19:26:47.024 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3359326352}) 2024-06-11T19:26:47.024 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.024 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1229308076}) 2024-06-11T19:26:47.025 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.582+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1212479876}) 2024-06-11T19:26:47.025 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.025 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454256468}) 2024-06-11T19:26:47.025 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765504675}) 2024-06-11T19:26:47.025 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.025 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454256468}) 2024-06-11T19:26:47.025 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1987885808}) 2024-06-11T19:26:47.025 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.025 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454256468}) 2024-06-11T19:26:47.026 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2912227799}) 2024-06-11T19:26:47.026 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.026 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454256468}) 2024-06-11T19:26:47.026 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2911186574}) 2024-06-11T19:26:47.026 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.026 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454256468}) 2024-06-11T19:26:47.026 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3550645752}) 2024-06-11T19:26:47.026 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.027 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454256468}) 2024-06-11T19:26:47.027 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2425872770}) 2024-06-11T19:26:47.027 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.027 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454256468}) 2024-06-11T19:26:47.027 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3308691309}) 2024-06-11T19:26:47.027 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.027 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454256468}) 2024-06-11T19:26:47.027 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.584+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1088332948}) 2024-06-11T19:26:47.027 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.028 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251273978}) 2024-06-11T19:26:47.028 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3602421899}) 2024-06-11T19:26:47.028 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.028 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251273978}) 2024-06-11T19:26:47.028 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=37567337}) 2024-06-11T19:26:47.028 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.028 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251273978}) 2024-06-11T19:26:47.028 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=574650242}) 2024-06-11T19:26:47.029 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.029 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251273978}) 2024-06-11T19:26:47.029 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3630883972}) 2024-06-11T19:26:47.029 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.029 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251273978}) 2024-06-11T19:26:47.029 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=650795836}) 2024-06-11T19:26:47.029 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.029 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251273978}) 2024-06-11T19:26:47.030 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1452616547}) 2024-06-11T19:26:47.030 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.030 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251273978}) 2024-06-11T19:26:47.030 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3508137299}) 2024-06-11T19:26:47.030 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.030 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251273978}) 2024-06-11T19:26:47.030 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.585+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3260566795}) 2024-06-11T19:26:47.030 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.586+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.031 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.586+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707396702}) 2024-06-11T19:26:47.031 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.586+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307467653}) 2024-06-11T19:26:47.031 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.586+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.031 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.586+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707396702}) 2024-06-11T19:26:47.031 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.586+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3165082156}) 2024-06-11T19:26:47.031 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.586+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.031 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707396702}) 2024-06-11T19:26:47.031 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2120511692}) 2024-06-11T19:26:47.032 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.032 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707396702}) 2024-06-11T19:26:47.032 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3246826781}) 2024-06-11T19:26:47.032 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.032 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707396702}) 2024-06-11T19:26:47.032 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2659667298}) 2024-06-11T19:26:47.032 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.032 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707396702}) 2024-06-11T19:26:47.033 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3524555567}) 2024-06-11T19:26:47.033 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.033 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707396702}) 2024-06-11T19:26:47.033 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2387588461}) 2024-06-11T19:26:47.033 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.033 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707396702}) 2024-06-11T19:26:47.033 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.587+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2471599813}) 2024-06-11T19:26:47.033 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.033 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166188784}) 2024-06-11T19:26:47.034 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=242696125}) 2024-06-11T19:26:47.034 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.034 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166188784}) 2024-06-11T19:26:47.034 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=613677626}) 2024-06-11T19:26:47.034 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.034 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166188784}) 2024-06-11T19:26:47.034 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1152106450}) 2024-06-11T19:26:47.034 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.035 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166188784}) 2024-06-11T19:26:47.035 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1737114481}) 2024-06-11T19:26:47.035 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.035 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166188784}) 2024-06-11T19:26:47.035 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1280268599}) 2024-06-11T19:26:47.035 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.035 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166188784}) 2024-06-11T19:26:47.035 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=925547411}) 2024-06-11T19:26:47.036 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.036 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166188784}) 2024-06-11T19:26:47.036 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1477770325}) 2024-06-11T19:26:47.036 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.036 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166188784}) 2024-06-11T19:26:47.036 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.588+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=692166471}) 2024-06-11T19:26:47.036 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.589+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.036 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.589+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502937234}) 2024-06-11T19:26:47.036 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.589+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1078653319}) 2024-06-11T19:26:47.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.589+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.589+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502937234}) 2024-06-11T19:26:47.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.589+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2379037081}) 2024-06-11T19:26:47.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.589+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.589+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502937234}) 2024-06-11T19:26:47.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.589+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3326953978}) 2024-06-11T19:26:47.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.589+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.037 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502937234}) 2024-06-11T19:26:47.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2140975464}) 2024-06-11T19:26:47.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502937234}) 2024-06-11T19:26:47.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3116244679}) 2024-06-11T19:26:47.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502937234}) 2024-06-11T19:26:47.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=347360397}) 2024-06-11T19:26:47.038 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502937234}) 2024-06-11T19:26:47.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4285171339}) 2024-06-11T19:26:47.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502937234}) 2024-06-11T19:26:47.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.590+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1281647174}) 2024-06-11T19:26:47.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4075233894}) 2024-06-11T19:26:47.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1253533505}) 2024-06-11T19:26:47.039 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.040 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4075233894}) 2024-06-11T19:26:47.040 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=667680308}) 2024-06-11T19:26:47.040 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.040 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4075233894}) 2024-06-11T19:26:47.040 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1420286052}) 2024-06-11T19:26:47.040 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.040 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4075233894}) 2024-06-11T19:26:47.040 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3268906540}) 2024-06-11T19:26:47.041 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.041 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4075233894}) 2024-06-11T19:26:47.041 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1124386941}) 2024-06-11T19:26:47.041 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.041 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4075233894}) 2024-06-11T19:26:47.041 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3124537094}) 2024-06-11T19:26:47.041 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.041 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4075233894}) 2024-06-11T19:26:47.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1392155847}) 2024-06-11T19:26:47.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4075233894}) 2024-06-11T19:26:47.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.591+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2288428943}) 2024-06-11T19:26:47.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=602952736}) 2024-06-11T19:26:47.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=927070712}) 2024-06-11T19:26:47.042 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.043 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=602952736}) 2024-06-11T19:26:47.043 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2812411297}) 2024-06-11T19:26:47.043 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.043 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=602952736}) 2024-06-11T19:26:47.043 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4082749430}) 2024-06-11T19:26:47.043 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.043 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=602952736}) 2024-06-11T19:26:47.043 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3008831879}) 2024-06-11T19:26:47.043 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.044 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=602952736}) 2024-06-11T19:26:47.044 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3669026184}) 2024-06-11T19:26:47.044 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.044 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=602952736}) 2024-06-11T19:26:47.044 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2604509856}) 2024-06-11T19:26:47.044 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.044 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=602952736}) 2024-06-11T19:26:47.044 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1050955935}) 2024-06-11T19:26:47.045 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.045 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=602952736}) 2024-06-11T19:26:47.045 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.592+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3098808732}) 2024-06-11T19:26:47.045 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.593+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.045 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.593+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715926701}) 2024-06-11T19:26:47.045 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.593+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845258648}) 2024-06-11T19:26:47.045 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.593+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.045 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715926701}) 2024-06-11T19:26:47.046 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3293339535}) 2024-06-11T19:26:47.046 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.046 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715926701}) 2024-06-11T19:26:47.046 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3113692582}) 2024-06-11T19:26:47.046 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.046 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715926701}) 2024-06-11T19:26:47.046 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2431464057}) 2024-06-11T19:26:47.047 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.047 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715926701}) 2024-06-11T19:26:47.047 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3798570564}) 2024-06-11T19:26:47.047 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.047 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715926701}) 2024-06-11T19:26:47.047 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1946148649}) 2024-06-11T19:26:47.047 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.047 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715926701}) 2024-06-11T19:26:47.047 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1458320668}) 2024-06-11T19:26:47.048 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.048 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715926701}) 2024-06-11T19:26:47.048 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.594+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1901541721}) 2024-06-11T19:26:47.048 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.048 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278263185}) 2024-06-11T19:26:47.048 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2807556127}) 2024-06-11T19:26:47.048 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.048 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278263185}) 2024-06-11T19:26:47.049 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793012534}) 2024-06-11T19:26:47.049 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.049 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278263185}) 2024-06-11T19:26:47.049 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1334244236}) 2024-06-11T19:26:47.049 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.049 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278263185}) 2024-06-11T19:26:47.049 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1094593659}) 2024-06-11T19:26:47.049 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.050 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278263185}) 2024-06-11T19:26:47.050 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1082849409}) 2024-06-11T19:26:47.050 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.050 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278263185}) 2024-06-11T19:26:47.050 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4182204864}) 2024-06-11T19:26:47.050 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.050 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278263185}) 2024-06-11T19:26:47.050 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1910069254}) 2024-06-11T19:26:47.050 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.051 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278263185}) 2024-06-11T19:26:47.051 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.595+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2491972388}) 2024-06-11T19:26:47.051 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.051 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1385558764}) 2024-06-11T19:26:47.051 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3623030338}) 2024-06-11T19:26:47.051 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.051 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1385558764}) 2024-06-11T19:26:47.051 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2319307979}) 2024-06-11T19:26:47.052 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.052 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1385558764}) 2024-06-11T19:26:47.052 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2887667630}) 2024-06-11T19:26:47.052 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.052 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1385558764}) 2024-06-11T19:26:47.052 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3767561596}) 2024-06-11T19:26:47.052 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.053 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1385558764}) 2024-06-11T19:26:47.053 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=260589713}) 2024-06-11T19:26:47.053 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.053 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1385558764}) 2024-06-11T19:26:47.053 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.596+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=762163411}) 2024-06-11T19:26:47.053 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.597+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.053 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.597+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1385558764}) 2024-06-11T19:26:47.053 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.597+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3047177183}) 2024-06-11T19:26:47.054 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.597+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.054 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.597+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1385558764}) 2024-06-11T19:26:47.054 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.597+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3646381474}) 2024-06-11T19:26:47.054 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.054 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646961024}) 2024-06-11T19:26:47.054 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163541705}) 2024-06-11T19:26:47.054 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.054 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646961024}) 2024-06-11T19:26:47.055 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4144865045}) 2024-06-11T19:26:47.055 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.055 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646961024}) 2024-06-11T19:26:47.055 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=855096062}) 2024-06-11T19:26:47.055 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.055 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646961024}) 2024-06-11T19:26:47.055 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=506359570}) 2024-06-11T19:26:47.055 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.056 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646961024}) 2024-06-11T19:26:47.056 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1474617260}) 2024-06-11T19:26:47.056 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.056 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646961024}) 2024-06-11T19:26:47.056 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3678983327}) 2024-06-11T19:26:47.056 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.056 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646961024}) 2024-06-11T19:26:47.056 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3290212456}) 2024-06-11T19:26:47.056 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.057 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646961024}) 2024-06-11T19:26:47.057 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.598+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1363685358}) 2024-06-11T19:26:47.057 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.057 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925602636}) 2024-06-11T19:26:47.057 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464554504}) 2024-06-11T19:26:47.057 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.057 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925602636}) 2024-06-11T19:26:47.057 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1383071990}) 2024-06-11T19:26:47.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925602636}) 2024-06-11T19:26:47.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=820534619}) 2024-06-11T19:26:47.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925602636}) 2024-06-11T19:26:47.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2663684090}) 2024-06-11T19:26:47.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.058 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925602636}) 2024-06-11T19:26:47.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=850260276}) 2024-06-11T19:26:47.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925602636}) 2024-06-11T19:26:47.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=576976996}) 2024-06-11T19:26:47.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925602636}) 2024-06-11T19:26:47.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2908220665}) 2024-06-11T19:26:47.059 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925602636}) 2024-06-11T19:26:47.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.599+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1246817834}) 2024-06-11T19:26:47.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13} crc {logm=2329943461,mdsmap=2840952586,mgr=2264053585,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=486334433,monmap=506305177,osd_metadata=3768155896,osd_pg_creating=1707938386,osd_snap=962963695}) 2024-06-11T19:26:47.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=9,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=17} crc {logm=1404860083,mdsmap=2840952586,mgr=3884025313,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2141203738,mon_config_key=993607601,monmap=506305176,osd_metadata=300920592,osd_pg_creating=1707938386,osd_snap=3118568041}) 2024-06-11T19:26:47.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13} crc {logm=2329943461,mdsmap=2840952586,mgr=2264053585,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=486334433,monmap=506305177,osd_metadata=3768155896,osd_pg_creating=1707938386,osd_snap=962963695}) 2024-06-11T19:26:47.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14} crc {logm=1883393651,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=3063798029,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1707938386,osd_snap=526457951}) 2024-06-11T19:26:47.060 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13} crc {logm=2329943461,mdsmap=2840952586,mgr=2264053585,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=486334433,monmap=506305177,osd_metadata=3768155896,osd_pg_creating=1707938386,osd_snap=962963695}) 2024-06-11T19:26:47.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=47,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=11} crc {logm=2062029976,mdsmap=2840952586,mgr=1864858522,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=2314039359,monmap=506305176,osd_metadata=3768155896,osd_pg_creating=1707938386,osd_snap=1404624435}) 2024-06-11T19:26:47.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13} crc {logm=2329943461,mdsmap=2840952586,mgr=2264053585,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=486334433,monmap=506305177,osd_metadata=3768155896,osd_pg_creating=1707938386,osd_snap=962963695}) 2024-06-11T19:26:47.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=47,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=13} crc {logm=1807611028,mdsmap=2840952586,mgr=3319824117,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=118980533,mon_config_key=2314039359,monmap=1530261226,osd_metadata=1603283398,osd_pg_creating=1707938386,osd_snap=210294183}) 2024-06-11T19:26:47.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13} crc {logm=2329943461,mdsmap=2840952586,mgr=2264053585,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=486334433,monmap=506305177,osd_metadata=3768155896,osd_pg_creating=1707938386,osd_snap=962963695}) 2024-06-11T19:26:47.061 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=43,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13} crc {logm=590153406,mdsmap=2840952586,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=2314039360,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1707938386,osd_snap=962963695}) 2024-06-11T19:26:47.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13} crc {logm=2329943461,mdsmap=2840952586,mgr=2264053585,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=486334433,monmap=506305177,osd_metadata=3768155896,osd_pg_creating=1707938386,osd_snap=962963695}) 2024-06-11T19:26:47.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=10} crc {logm=4192834541,mdsmap=3764978221,mgr=2102212137,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1707938386,osd_snap=3150342867}) 2024-06-11T19:26:47.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13} crc {logm=2329943461,mdsmap=2840952586,mgr=2264053585,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=486334433,monmap=506305177,osd_metadata=3768155896,osd_pg_creating=1707938386,osd_snap=962963695}) 2024-06-11T19:26:47.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=43,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=15} crc {logm=626682279,mdsmap=2840952586,mgr=2264053585,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=2314039359,monmap=3961624422,osd_metadata=3768155896,osd_pg_creating=1707938386,osd_snap=1286639653}) 2024-06-11T19:26:47.062 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13} crc {logm=2329943461,mdsmap=2840952586,mgr=2264053585,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=486334433,monmap=506305177,osd_metadata=3768155896,osd_pg_creating=1707938386,osd_snap=962963695}) 2024-06-11T19:26:47.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.601+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=46,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=12} crc {logm=3027529046,mdsmap=2840952586,mgr=3180177182,mgr_command_descs=1608769102,mgr_metadata=1826097544,mgrstat=2046151476,mon_config_key=2314039359,monmap=506305176,osd_metadata=1603283398,osd_pg_creating=1707938386,osd_snap=1419695321}) 2024-06-11T19:26:47.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.602+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.602+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=3498120698,osdmap=2295979542}) 2024-06-11T19:26:47.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.602+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=2033934691,osdmap=3048475402}) 2024-06-11T19:26:47.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.602+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.602+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=3498120698,osdmap=2295979542}) 2024-06-11T19:26:47.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.602+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=22,osdmap=78} crc {osd_snap=3663722764,osdmap=669522637}) 2024-06-11T19:26:47.063 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.602+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.602+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=3498120698,osdmap=2295979542}) 2024-06-11T19:26:47.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=2598802764,osdmap=1712663323}) 2024-06-11T19:26:47.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=3498120698,osdmap=2295979542}) 2024-06-11T19:26:47.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=25,osdmap=75} crc {osd_snap=2082570576,osdmap=1687645197}) 2024-06-11T19:26:47.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=3498120698,osdmap=2295979542}) 2024-06-11T19:26:47.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=3385960883,osdmap=2266137486}) 2024-06-11T19:26:47.064 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=3498120698,osdmap=2295979542}) 2024-06-11T19:26:47.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osd_snap=25,osdmap=75} crc {osd_snap=2082570576,osdmap=2078885312}) 2024-06-11T19:26:47.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=3498120698,osdmap=2295979542}) 2024-06-11T19:26:47.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=510494237,osdmap=1951026088}) 2024-06-11T19:26:47.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=24,osdmap=76} crc {osd_snap=3498120698,osdmap=2295979542}) 2024-06-11T19:26:47.065 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.603+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=23,osdmap=77} crc {osd_snap=2738831963,osdmap=1036558748}) 2024-06-11T19:26:47.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3154752528}) 2024-06-11T19:26:47.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3637056356}) 2024-06-11T19:26:47.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3154752528}) 2024-06-11T19:26:47.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2329221545}) 2024-06-11T19:26:47.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.066 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3154752528}) 2024-06-11T19:26:47.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=440366800}) 2024-06-11T19:26:47.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3154752528}) 2024-06-11T19:26:47.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=261249193}) 2024-06-11T19:26:47.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3154752528}) 2024-06-11T19:26:47.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=973767584}) 2024-06-11T19:26:47.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.067 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3154752528}) 2024-06-11T19:26:47.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2162868210}) 2024-06-11T19:26:47.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3154752528}) 2024-06-11T19:26:47.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=420444199}) 2024-06-11T19:26:47.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3154752528}) 2024-06-11T19:26:47.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.604+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3526488773}) 2024-06-11T19:26:47.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.068 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=76} crc {osdmap=736577448}) 2024-06-11T19:26:47.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=75} crc {osdmap=1169362742}) 2024-06-11T19:26:47.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=76} crc {osdmap=736577448}) 2024-06-11T19:26:47.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=75} crc {osdmap=3840729185}) 2024-06-11T19:26:47.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=76} crc {osdmap=736577448}) 2024-06-11T19:26:47.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=76} crc {osdmap=457048597}) 2024-06-11T19:26:47.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.069 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=76} crc {osdmap=736577448}) 2024-06-11T19:26:47.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=77} crc {osdmap=3451550054}) 2024-06-11T19:26:47.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=76} crc {osdmap=736577448}) 2024-06-11T19:26:47.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=77} crc {osdmap=4191852482}) 2024-06-11T19:26:47.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=76} crc {osdmap=736577448}) 2024-06-11T19:26:47.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=76} crc {osdmap=3181114168}) 2024-06-11T19:26:47.070 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=76} crc {osdmap=736577448}) 2024-06-11T19:26:47.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=77} crc {osdmap=1148508980}) 2024-06-11T19:26:47.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-06-11T19:26:47.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=76} crc {osdmap=736577448}) 2024-06-11T19:26:47.071 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-06-11T19:26:46.614+0000 7f13d16e3640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=75} crc {osdmap=1237276601}) 2024-06-11T19:26:47.463 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-06-11T19:26:47.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-11T19:26:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.563+0000 7fd6fb746640 10 monclient: get_monmap_and_config 2024-06-11T19:26:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.563+0000 7fd6fb746640 10 monclient: build_initial_monmap 2024-06-11T19:26:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.563+0000 7fd6fb746640 10 monclient: monmap: 2024-06-11T19:26:47.562 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:47.564319+0000 2024-06-11T19:26:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:47.564319+0000 2024-06-11T19:26:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:47.563 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:47.564 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:47.565 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.563+0000 7fd6fb746640 1 Processor -- start 2024-06-11T19:26:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 1 -- start start 2024-06-11T19:26:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 10 monclient: init 2024-06-11T19:26:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 10 monclient: _add_conns ranks=[3,5,6,10,2,1,7,8,0,9,4] 2024-06-11T19:26:47.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6f41347f0 0x7fd6f409e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 10 monclient(hunting): picked mon.noname-g con 0x7fd6f41347f0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd6f4134bd0 0x7fd6f40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 10 monclient(hunting): picked mon.noname-h con 0x7fd6f4134bd0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd6f4135010 0x7fd6f40a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 10 monclient(hunting): picked mon.noname-d con 0x7fd6f4135010 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f40687f0 con 0x7fd6f41347f0 2024-06-11T19:26:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd6f409d410 con 0x7fd6f4134bd0 2024-06-11T19:26:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd6f409d6f0 con 0x7fd6f4135010 2024-06-11T19:26:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6fb746640 10 monclient(hunting): authenticate will time out at 1817.371660s 2024-06-11T19:26:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f9ce3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd6f4135010 0x7fd6f40a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f9ce3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd6f4135010 0x7fd6f40a3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58902/0 (socket says 172.21.15.98:58902) 2024-06-11T19:26:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f9ce3640 1 -- 172.21.15.98:0/1881126097 learned_addr learned my addr 172.21.15.98:0/1881126097 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:47.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f9ce3640 10 monclient(hunting): get_auth_request con 0x7fd6f4135010 auth_method 0 2024-06-11T19:26:47.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f9ce3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:47.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f9ce3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:47.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f9ce3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:47.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f8ce1640 1 --2- 172.21.15.98:0/1881126097 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd6f4134bd0 0x7fd6f40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:47.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f9ce3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:47.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f9ce3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:47.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f94e2640 1 --2- 172.21.15.98:0/1881126097 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6f41347f0 0x7fd6f409e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:47.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f9ce3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:47.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f8ce1640 10 monclient(hunting): get_auth_request con 0x7fd6f4134bd0 auth_method 0 2024-06-11T19:26:47.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f8ce1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:47.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f8ce1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:47.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f8ce1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:47.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f94e2640 10 monclient(hunting): get_auth_request con 0x7fd6f41347f0 auth_method 0 2024-06-11T19:26:47.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f94e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:47.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f94e2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:47.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.564+0000 7fd6f94e2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:47.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f8ce1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:47.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f8ce1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:47.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f8ce1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:47.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f94e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:47.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f94e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:47.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f94e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:47.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f9ce3640 10 monclient(hunting): handle_auth_done global_id 124449 payload 274 2024-06-11T19:26:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f9ce3640 1 -- 172.21.15.98:0/1881126097 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd6f4134bd0 msgr2=0x7fd6f40a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f9ce3640 1 --2- 172.21.15.98:0/1881126097 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd6f4134bd0 0x7fd6f40a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f9ce3640 1 -- 172.21.15.98:0/1881126097 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6f41347f0 msgr2=0x7fd6f409e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f9ce3640 1 --2- 172.21.15.98:0/1881126097 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6f41347f0 0x7fd6f409e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f9ce3640 10 monclient: _finish_hunting 0 2024-06-11T19:26:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f9ce3640 1 monclient: found mon.noname-d 2024-06-11T19:26:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f9ce3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f9ce3640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f9ce3640 1 -- 172.21.15.98:0/1881126097 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f409d9d0 con 0x7fd6f4135010 2024-06-11T19:26:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6f9ce3640 1 --2- 172.21.15.98:0/1881126097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd6f4135010 0x7fd6f40a3bd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009e80 tx=0x7fd6ec02be00 comp rx=0 tx=0).ready entity=mon.6 client_cookie=9e280860b3de7cd0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6ebfff640 1 -- 172.21.15.98:0/1881126097 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6ec02cc10 con 0x7fd6f4135010 2024-06-11T19:26:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6ebfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.565+0000 7fd6ebfff640 10 monclient: dump: 2024-06-11T19:26:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:47.583 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6ebfff640 10 monclient: _finish_auth 0 2024-06-11T19:26:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6ebfff640 10 monclient: _check_auth_tickets 2024-06-11T19:26:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 5 monclient: authenticate success, global_id 124449 2024-06-11T19:26:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6ebfff640 1 -- 172.21.15.98:0/1881126097 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6ec02cdd0 con 0x7fd6f4135010 2024-06-11T19:26:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6ebfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6ebfff640 1 -- 172.21.15.98:0/1881126097 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6ec034860 con 0x7fd6f4135010 2024-06-11T19:26:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 4 set_mon_vals no callback set 2024-06-11T19:26:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6ebfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6ebfff640 10 monclient: dump: 2024-06-11T19:26:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:47.585 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:47.586 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:47.586 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:47.586 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:47.586 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:47.586 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:47.586 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:47.586 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:47.586 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:47.587 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:47.587 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:47.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 10 monclient: shutdown 2024-06-11T19:26:47.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:47.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 1 -- 172.21.15.98:0/1881126097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd6f4135010 msgr2=0x7fd6f40a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:47.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 1 --2- 172.21.15.98:0/1881126097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd6f4135010 0x7fd6f40a3bd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009e80 tx=0x7fd6ec02be00 comp rx=0 tx=0).stop 2024-06-11T19:26:47.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 1 -- 172.21.15.98:0/1881126097 shutdown_connections 2024-06-11T19:26:47.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 1 --2- 172.21.15.98:0/1881126097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd6f4135010 0x7fd6f40a3bd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 1 --2- 172.21.15.98:0/1881126097 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd6f4134bd0 0x7fd6f40a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 1 --2- 172.21.15.98:0/1881126097 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6f41347f0 0x7fd6f409e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 1 -- 172.21.15.98:0/1881126097 >> 172.21.15.98:0/1881126097 conn(0x7fd6f40a4780 msgr2=0x7fd6f40a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:47.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 1 -- 172.21.15.98:0/1881126097 shutdown_connections 2024-06-11T19:26:47.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.566+0000 7fd6fb746640 1 -- 172.21.15.98:0/1881126097 wait complete. 2024-06-11T19:26:47.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.567+0000 7fd6fb746640 10 monclient: build_initial_monmap 2024-06-11T19:26:47.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.567+0000 7fd6fb746640 10 monclient: monmap: 2024-06-11T19:26:47.588 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:47.588 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:47.588 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:47.589 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:47.590 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:47.590 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:47.590 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:47.590 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:47.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.567+0000 7fd6fb746640 1 librados: starting msgr at 2024-06-11T19:26:47.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.567+0000 7fd6fb746640 1 librados: starting objecter 2024-06-11T19:26:47.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.567+0000 7fd6fb746640 1 Processor -- start 2024-06-11T19:26:47.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.567+0000 7fd6fb746640 1 -- start start 2024-06-11T19:26:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.567+0000 7fd6fb746640 1 librados: setting wanted keys 2024-06-11T19:26:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.567+0000 7fd6fb746640 1 librados: calling monclient init 2024-06-11T19:26:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.567+0000 7fd6fb746640 10 monclient: init 2024-06-11T19:26:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 10 monclient: _add_conns ranks=[1,5,3,8,0,6,7,2,4] 2024-06-11T19:26:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6f41347f0 0x7fd6f41cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 10 monclient(hunting): picked mon.noname-f con 0x7fd6f41347f0 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:26:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd6f4134bd0 0x7fd6f41ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 10 monclient(hunting): picked mon.noname-h con 0x7fd6f4134bd0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:47.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6f4135010 0x7fd6f411ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 10 monclient(hunting): picked mon.noname-g con 0x7fd6f4135010 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:47.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:47.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f4068c40 con 0x7fd6f41347f0 2024-06-11T19:26:47.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:47.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f4068ac0 con 0x7fd6f4135010 2024-06-11T19:26:47.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:47.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7fd6f40a4110 con 0x7fd6f4134bd0 2024-06-11T19:26:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6fb746640 10 monclient(hunting): authenticate will time out at 1817.375175s 2024-06-11T19:26:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f94e2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6f41347f0 0x7fd6f41cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6f4135010 0x7fd6f411ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f8ce1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd6f4134bd0 0x7fd6f41ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f94e2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6f41347f0 0x7fd6f41cdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:48138/0 (socket says 172.21.15.98:48138) 2024-06-11T19:26:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f94e2640 1 -- 172.21.15.98:0/2562496907 learned_addr learned my addr 172.21.15.98:0/2562496907 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:47.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f94e2640 10 monclient(hunting): get_auth_request con 0x7fd6f41347f0 auth_method 0 2024-06-11T19:26:47.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f94e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:47.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f94e2640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:47.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f94e2640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:47.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 10 monclient(hunting): get_auth_request con 0x7fd6f4135010 auth_method 0 2024-06-11T19:26:47.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:47.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:47.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:47.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f8ce1640 10 monclient(hunting): get_auth_request con 0x7fd6f4134bd0 auth_method 0 2024-06-11T19:26:47.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f8ce1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:47.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f8ce1640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:47.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f8ce1640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:47.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:47.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:47.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:47.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f94e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:47.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f94e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f94e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f8ce1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f8ce1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f8ce1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 10 monclient(hunting): handle_auth_done global_id 114546 payload 824 2024-06-11T19:26:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 1 -- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd6f4134bd0 msgr2=0x7fd6f41ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd6f4134bd0 0x7fd6f41ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 1 -- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6f41347f0 msgr2=0x7fd6f41cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:47.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6f41347f0 0x7fd6f41cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 10 monclient: _finish_hunting 0 2024-06-11T19:26:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 1 monclient: found mon.noname-g 2024-06-11T19:26:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:26:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f9ce3640 1 -- 172.21.15.98:0/2562496907 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f4122760 con 0x7fd6f4135010 2024-06-11T19:26:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f94e2640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6f41347f0 0x7fd6f41cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.568+0000 7fd6f8ce1640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd6f4134bd0 0x7fd6f41ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:47.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6f9ce3640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6f4135010 0x7fd6f411ead0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd6f409d8c0 tx=0x7fd6ec004a40 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:47.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 10 client.?.objecter ms_handle_connect 0x7fd6f4135010 2024-06-11T19:26:47.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:47.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 1 -- 172.21.15.98:0/2562496907 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6ec042070 con 0x7fd6f4135010 2024-06-11T19:26:47.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:47.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:26:47.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 10 monclient: dump: 2024-06-11T19:26:47.598 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:47.598 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:47.599 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:47.600 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 10 monclient: _finish_auth 0 2024-06-11T19:26:47.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:26:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 1 -- 172.21.15.98:0/2562496907 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6ec0046c0 con 0x7fd6f4135010 2024-06-11T19:26:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 1 -- 172.21.15.98:0/2562496907 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6ec009510 con 0x7fd6f4135010 2024-06-11T19:26:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6fb746640 5 monclient: authenticate success, global_id 114546 2024-06-11T19:26:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6fb746640 10 monclient: _renew_subs 2024-06-11T19:26:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6fb746640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:26:47.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6fb746640 1 -- 172.21.15.98:0/2562496907 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f41212b0 con 0x7fd6f4135010 2024-06-11T19:26:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6eb7fe640 4 set_mon_vals no callback set 2024-06-11T19:26:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:26:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6e9ffb640 10 monclient: dump: 2024-06-11T19:26:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.602 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:47.603 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:47.604 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6fb746640 10 client.114546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6fb746640 10 monclient: _renew_subs 2024-06-11T19:26:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6fb746640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:26:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6fb746640 1 -- 172.21.15.98:0/2562496907 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f4135fa0 con 0x7fd6f4135010 2024-06-11T19:26:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.569+0000 7fd6fb746640 1 librados: init done 2024-06-11T19:26:47.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6fb746640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6fb746640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6fb746640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:26:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6e9ffb640 1 -- 172.21.15.98:0/2562496907 <== mon.3 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd6ec02cca0 con 0x7fd6f4135010 2024-06-11T19:26:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6fb746640 1 -- 172.21.15.98:0/2562496907 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f40a3510 con 0x7fd6f4135010 2024-06-11T19:26:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6e9ffb640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6d803fdb0 0x7fd6d8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6e9ffb640 1 -- 172.21.15.98:0/2562496907 <== mon.3 v2:172.21.15.117:3301/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7fd6ec0735f0 con 0x7fd6f4135010 2024-06-11T19:26:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6e9ffb640 10 client.114546.objecter ms_dispatch 0x7fd6f4000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:47.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6e9ffb640 3 client.114546.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6e9ffb640 3 client.114546.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6e9ffb640 20 client.114546.objecter dump_active .. 0 homeless 2024-06-11T19:26:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6f94e2640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6d803fdb0 0x7fd6d8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.570+0000 7fd6f94e2640 10 monclient: get_auth_request con 0x7fd6d803fdb0 auth_method 0 2024-06-11T19:26:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.571+0000 7fd6e9ffb640 10 client.114546.objecter ms_handle_connect 0x7fd6d803fdb0 2024-06-11T19:26:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.571+0000 7fd6f94e2640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6d803fdb0 0x7fd6d8042270 secure :-1 s=READY pgs=4330 cs=0 l=1 rev1=1 crypto rx=0x7fd6f00112c0 tx=0x7fd6f0009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.575+0000 7fd6e9ffb640 1 -- 172.21.15.98:0/2562496907 <== mon.3 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd6ec04b070 con 0x7fd6f4135010 2024-06-11T19:26:47.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.575+0000 7fd6e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:47.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.575+0000 7fd6e9ffb640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:47.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.827+0000 7fd6fb746640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-06-11T19:26:47.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.827+0000 7fd6fb746640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-06-11T19:26:47.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.827+0000 7fd6fb746640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.117:3301/0 2024-06-11T19:26:47.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.827+0000 7fd6fb746640 1 -- 172.21.15.98:0/2562496907 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd6f40a4470 con 0x7fd6f4135010 2024-06-11T19:26:47.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.827+0000 7fd6e9ffb640 1 -- 172.21.15.98:0/2562496907 <== mon.3 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fd6ec0333a0 con 0x7fd6f4135010 2024-06-11T19:26:47.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.827+0000 7fd6e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-06-11T19:26:47.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.827+0000 7fd6e9ffb640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:47.827 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":92,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","modified":"2024-06-11T19:08:43.892957Z","created":"2024-06-11T19:08:43.892957Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3301","nonce":0},{"type":"v1","addr":"172.21.15.117:6790","nonce":0}]},"addr":"172.21.15.117:6790/0","public_addr":"172.21.15.117:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3302","nonce":0},{"type":"v1","addr":"172.21.15.117:6791","nonce":0}]},"addr":"172.21.15.117:6791/0","public_addr":"172.21.15.117:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3303","nonce":0},{"type":"v1","addr":"172.21.15.117:6792","nonce":0}]},"addr":"172.21.15.117:6792/0","public_addr":"172.21.15.117:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-11T19:26:47.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 10 librados: watch_flush enter 2024-06-11T19:26:47.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 10 librados: watch_flush exit 2024-06-11T19:26:47.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 20 client.114546.objecter shutdown clearing up homeless session... 2024-06-11T19:26:47.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 10 client.114546.objecter successfully canceled tick 2024-06-11T19:26:47.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 1 -- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6d803fdb0 msgr2=0x7fd6d8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:47.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6d803fdb0 0x7fd6d8042270 secure :-1 s=READY pgs=4330 cs=0 l=1 rev1=1 crypto rx=0x7fd6f00112c0 tx=0x7fd6f0009190 comp rx=0 tx=0).stop 2024-06-11T19:26:47.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 10 monclient: shutdown 2024-06-11T19:26:47.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:47.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 1 -- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6f4135010 msgr2=0x7fd6f411ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:47.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6f4135010 0x7fd6f411ead0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd6f409d8c0 tx=0x7fd6ec004a40 comp rx=0 tx=0).stop 2024-06-11T19:26:47.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 1 -- 172.21.15.98:0/2562496907 shutdown_connections 2024-06-11T19:26:47.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd6d803fdb0 0x7fd6d8042270 unknown :-1 s=CLOSED pgs=4330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd6f4135010 0x7fd6f411ead0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7fd6f4134bd0 0x7fd6f41ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.829+0000 7fd6fb746640 1 --2- 172.21.15.98:0/2562496907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6f41347f0 0x7fd6f41cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.830+0000 7fd6fb746640 1 -- 172.21.15.98:0/2562496907 >> 172.21.15.98:0/2562496907 conn(0x7fd6f40a4780 msgr2=0x7fd6f41be7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:47.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.830+0000 7fd6fb746640 1 -- 172.21.15.98:0/2562496907 shutdown_connections 2024-06-11T19:26:47.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.830+0000 7fd6fb746640 1 -- 172.21.15.98:0/2562496907 wait complete. 2024-06-11T19:26:47.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.830+0000 7fd6fb746640 1 librados: shutdown 2024-06-11T19:26:47.839 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-06-11T19:26:47.839 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-06-11T19:26:47.849 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-06-11T19:26:47.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-11T19:26:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.952+0000 7f0996622640 10 monclient: get_monmap_and_config 2024-06-11T19:26:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.952+0000 7f0996622640 10 monclient: build_initial_monmap 2024-06-11T19:26:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.953+0000 7f0996622640 10 monclient: monmap: 2024-06-11T19:26:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:47.953986+0000 2024-06-11T19:26:47.952 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:47.953986+0000 2024-06-11T19:26:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:47.953 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:47.954 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:47.954 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:47.954 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:47.954 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:47.954 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:47.954 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:47.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.953+0000 7f0996622640 1 Processor -- start 2024-06-11T19:26:47.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.953+0000 7f0996622640 1 -- start start 2024-06-11T19:26:47.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.953+0000 7f0996622640 10 monclient: init 2024-06-11T19:26:47.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:47.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 10 monclient: _add_conns ranks=[1,6,2,10,4,3,7,5,8,9,0] 2024-06-11T19:26:47.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f09900a4920 addr v1:172.21.15.117:6789/0 2024-06-11T19:26:47.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09900a2530 0x7f099013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 10 monclient(hunting): picked mon.noname-d con 0x7f09900a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:47.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f099013e1a0 0x7f0990140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 10 monclient(hunting): picked mon.noname-b con 0x7f099013e1a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:47.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09900a7140 con 0x7f09900a4920 2024-06-11T19:26:47.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f09900a2b50 con 0x7f099013e1a0 2024-06-11T19:26:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f09900a2e30 con 0x7f09900a2530 2024-06-11T19:26:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0996622640 10 monclient(hunting): authenticate will time out at 1817.761234s 2024-06-11T19:26:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f098f7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09900a2530 0x7f099013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f099013e1a0 0x7f0990140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f098f7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09900a2530 0x7f099013dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58912/0 (socket says 172.21.15.98:58912) 2024-06-11T19:26:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f098f7fe640 1 -- 172.21.15.98:0/3429706009 learned_addr learned my addr 172.21.15.98:0/3429706009 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f098f7fe640 10 monclient(hunting): get_auth_request con 0x7f09900a2530 auth_method 0 2024-06-11T19:26:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f098f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f098f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f098f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 10 monclient(hunting): get_auth_request con 0x7f099013e1a0 auth_method 0 2024-06-11T19:26:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f098f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f098f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f098f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 10 monclient(hunting): handle_auth_done global_id 85394 payload 274 2024-06-11T19:26:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 1 -- 172.21.15.98:0/3429706009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09900a2530 msgr2=0x7f099013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 1 --2- 172.21.15.98:0/3429706009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09900a2530 0x7f099013dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 1 -- 172.21.15.98:0/3429706009 >> v1:172.21.15.117:6789/0 conn(0x7f09900a4920 legacy=0x7f09900a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 10 monclient: _finish_hunting 0 2024-06-11T19:26:47.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 1 monclient: found mon.noname-b 2024-06-11T19:26:47.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:47.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:26:47.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f0994bbf640 1 -- 172.21.15.98:0/3429706009 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09900a3110 con 0x7f099013e1a0 2024-06-11T19:26:47.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.954+0000 7f098f7fe640 1 --2- 172.21.15.98:0/3429706009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09900a2530 0x7f099013dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:47.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f0994bbf640 1 --2- 172.21.15.98:0/3429706009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f099013e1a0 0x7f0990140590 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0988009dc0 tx=0x7f098802bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=301f168651840b0a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:47.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 1 -- 172.21.15.98:0/3429706009 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0988039070 con 0x7f099013e1a0 2024-06-11T19:26:47.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:47.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:26:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 10 monclient: dump: 2024-06-11T19:26:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:47.968 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 10 monclient: _finish_auth 0 2024-06-11T19:26:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 10 monclient: _check_auth_tickets 2024-06-11T19:26:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 1 -- 172.21.15.98:0/3429706009 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f098802cd70 con 0x7f099013e1a0 2024-06-11T19:26:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f0996622640 5 monclient: authenticate success, global_id 85394 2024-06-11T19:26:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f0996622640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 1 -- 172.21.15.98:0/3429706009 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09880327b0 con 0x7f099013e1a0 2024-06-11T19:26:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f0996622640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f0996622640 4 set_mon_vals no callback set 2024-06-11T19:26:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:26:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.955+0000 7f098effd640 10 monclient: dump: 2024-06-11T19:26:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:47.971 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:47.971 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 10 monclient: shutdown 2024-06-11T19:26:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 1 -- 172.21.15.98:0/3429706009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f099013e1a0 msgr2=0x7f0990140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 1 --2- 172.21.15.98:0/3429706009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f099013e1a0 0x7f0990140590 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0988009dc0 tx=0x7f098802bd50 comp rx=0 tx=0).stop 2024-06-11T19:26:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 1 -- 172.21.15.98:0/3429706009 shutdown_connections 2024-06-11T19:26:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 1 --2- 172.21.15.98:0/3429706009 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f099013e1a0 0x7f0990140590 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 1 --2- 172.21.15.98:0/3429706009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09900a2530 0x7f099013dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 1 -- 172.21.15.98:0/3429706009 >> 172.21.15.98:0/3429706009 conn(0x7f09900ab850 msgr2=0x7f09900a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 1 -- 172.21.15.98:0/3429706009 shutdown_connections 2024-06-11T19:26:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 1 -- 172.21.15.98:0/3429706009 wait complete. 2024-06-11T19:26:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 10 monclient: build_initial_monmap 2024-06-11T19:26:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 10 monclient: monmap: 2024-06-11T19:26:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:47.974 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:47.975 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 1 librados: starting msgr at 2024-06-11T19:26:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 1 librados: starting objecter 2024-06-11T19:26:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.956+0000 7f0996622640 1 Processor -- start 2024-06-11T19:26:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 1 -- start start 2024-06-11T19:26:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 1 librados: setting wanted keys 2024-06-11T19:26:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 1 librados: calling monclient init 2024-06-11T19:26:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 10 monclient: init 2024-06-11T19:26:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 10 monclient: _add_conns ranks=[4,0,7,3,8,5,1,6,2] 2024-06-11T19:26:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f09900a2530 0x7f09900a1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 10 monclient(hunting): picked mon.noname-c con 0x7f09900a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09900a4920 0x7f09901cd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 10 monclient(hunting): picked mon.noname-a con 0x7f09900a4920 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:26:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f099013e1a0 0x7f09901cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 10 monclient(hunting): picked mon.noname-i con 0x7f099013e1a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0990138e50 con 0x7f09900a4920 2024-06-11T19:26:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f09900a84b0 con 0x7f09900a2530 2024-06-11T19:26:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f09900a6da0 con 0x7f099013e1a0 2024-06-11T19:26:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f0996622640 10 monclient(hunting): authenticate will time out at 1817.764656s 2024-06-11T19:26:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f09900a2530 0x7f09900a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09900a4920 0x7f09901cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f09900a2530 0x7f09900a1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:37238/0 (socket says 172.21.15.98:37238) 2024-06-11T19:26:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098ffff640 1 -- 172.21.15.98:0/1482637259 learned_addr learned my addr 172.21.15.98:0/1482637259 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098ffff640 10 monclient(hunting): get_auth_request con 0x7f09900a2530 auth_method 0 2024-06-11T19:26:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098f7fe640 10 monclient(hunting): get_auth_request con 0x7f09900a4920 auth_method 0 2024-06-11T19:26:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.957+0000 7f098ffff640 10 monclient(hunting): handle_auth_done global_id 134167 payload 824 2024-06-11T19:26:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098ffff640 1 -- 172.21.15.98:0/1482637259 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f099013e1a0 msgr2=0x7f09901cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098ffff640 1 --2- 172.21.15.98:0/1482637259 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f099013e1a0 0x7f09901cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098ffff640 1 -- 172.21.15.98:0/1482637259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09900a4920 msgr2=0x7f09901cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098ffff640 1 --2- 172.21.15.98:0/1482637259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09900a4920 0x7f09901cd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:26:47.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098ffff640 1 monclient: found mon.noname-c 2024-06-11T19:26:47.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:47.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:26:47.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098ffff640 1 -- 172.21.15.98:0/1482637259 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099013da40 con 0x7f09900a2530 2024-06-11T19:26:47.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098ffff640 1 --2- 172.21.15.98:0/1482637259 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f09900a2530 0x7f09900a1340 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f098000c450 tx=0x7f098000c910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:47.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 10 client.?.objecter ms_handle_connect 0x7f09900a2530 2024-06-11T19:26:47.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:47.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 1 -- 172.21.15.98:0/1482637259 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0980001180 con 0x7f09900a2530 2024-06-11T19:26:47.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:47.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:26:47.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 10 monclient: dump: 2024-06-11T19:26:47.982 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:47.982 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:47.982 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.982 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.982 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:47.982 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:47.984 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:47.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 10 monclient: _finish_auth 0 2024-06-11T19:26:47.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 10 monclient: _check_auth_tickets 2024-06-11T19:26:47.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:47.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f0996622640 5 monclient: authenticate success, global_id 134167 2024-06-11T19:26:47.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 1 -- 172.21.15.98:0/1482637259 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0980012630 con 0x7f09900a2530 2024-06-11T19:26:47.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:47.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 1 -- 172.21.15.98:0/1482637259 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0980012c50 con 0x7f09900a2530 2024-06-11T19:26:47.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f0996622640 10 monclient: _renew_subs 2024-06-11T19:26:47.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f0996622640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:47.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f0996622640 1 -- 172.21.15.98:0/1482637259 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09900a08d0 con 0x7f09900a2530 2024-06-11T19:26:47.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:47.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:26:47.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098cff9640 10 monclient: dump: 2024-06-11T19:26:47.985 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:47.985 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:47.985 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.985 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:47.986 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:47.987 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:47.987 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:47.987 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:47.987 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:47.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f098e7fc640 4 set_mon_vals no callback set 2024-06-11T19:26:47.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f0996622640 10 client.134167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:47.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f0996622640 10 monclient: _renew_subs 2024-06-11T19:26:47.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f0996622640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:47.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f0996622640 1 -- 172.21.15.98:0/1482637259 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f09901ce280 con 0x7f09900a2530 2024-06-11T19:26:47.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.958+0000 7f0996622640 1 librados: init done 2024-06-11T19:26:47.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f098cff9640 1 -- 172.21.15.98:0/1482637259 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f09800127e0 con 0x7f09900a2530 2024-06-11T19:26:47.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f098cff9640 1 --2- 172.21.15.98:0/1482637259 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f097c03fde0 0x7f097c0422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:47.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f098cff9640 1 -- 172.21.15.98:0/1482637259 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f09800512b0 con 0x7f09900a2530 2024-06-11T19:26:47.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f098cff9640 10 client.134167.objecter ms_dispatch 0x7f0990000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:47.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f098cff9640 3 client.134167.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:47.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f098cff9640 3 client.134167.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:47.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f0996622640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:47.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f0996622640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:47.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f0996622640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:47.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f0996622640 1 -- 172.21.15.98:0/1482637259 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09900a4d00 con 0x7f09900a2530 2024-06-11T19:26:47.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f098cff9640 20 client.134167.objecter dump_active .. 0 homeless 2024-06-11T19:26:47.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f098f7fe640 1 --2- 172.21.15.98:0/1482637259 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f097c03fde0 0x7f097c0422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:47.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.959+0000 7f098f7fe640 10 monclient: get_auth_request con 0x7f097c03fde0 auth_method 0 2024-06-11T19:26:47.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.960+0000 7f098cff9640 10 client.134167.objecter ms_handle_connect 0x7f097c03fde0 2024-06-11T19:26:47.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.960+0000 7f098f7fe640 1 --2- 172.21.15.98:0/1482637259 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f097c03fde0 0x7f097c0422a0 secure :-1 s=READY pgs=4331 cs=0 l=1 rev1=1 crypto rx=0x7f0984009630 tx=0x7f0984009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:47.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.963+0000 7f098cff9640 1 -- 172.21.15.98:0/1482637259 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f098001a360 con 0x7f09900a2530 2024-06-11T19:26:47.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.963+0000 7f098cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:47.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:47.963+0000 7f098cff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:48.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.142+0000 7f0996622640 1 -- 172.21.15.98:0/1482637259 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0990073130 con 0x7f097c03fde0 2024-06-11T19:26:48.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.145+0000 7f098cff9640 1 -- 172.21.15.98:0/1482637259 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+113897 (secure 0 0 0) 0x7f0990073130 con 0x7f097c03fde0 2024-06-11T19:26:48.144 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:26:48.145 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-06-11T19:26:48.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.146+0000 7f0996622640 10 librados: watch_flush enter 2024-06-11T19:26:48.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 10 librados: watch_flush exit 2024-06-11T19:26:48.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 20 client.134167.objecter shutdown clearing up homeless session... 2024-06-11T19:26:48.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 10 client.134167.objecter successfully canceled tick 2024-06-11T19:26:48.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 -- 172.21.15.98:0/1482637259 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f097c03fde0 msgr2=0x7f097c0422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 --2- 172.21.15.98:0/1482637259 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f097c03fde0 0x7f097c0422a0 secure :-1 s=READY pgs=4331 cs=0 l=1 rev1=1 crypto rx=0x7f0984009630 tx=0x7f0984009240 comp rx=0 tx=0).stop 2024-06-11T19:26:48.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 10 monclient: shutdown 2024-06-11T19:26:48.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:48.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 -- 172.21.15.98:0/1482637259 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f09900a2530 msgr2=0x7f09900a1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 --2- 172.21.15.98:0/1482637259 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f09900a2530 0x7f09900a1340 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f098000c450 tx=0x7f098000c910 comp rx=0 tx=0).stop 2024-06-11T19:26:48.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 -- 172.21.15.98:0/1482637259 shutdown_connections 2024-06-11T19:26:48.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 --2- 172.21.15.98:0/1482637259 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f097c03fde0 0x7f097c0422a0 unknown :-1 s=CLOSED pgs=4331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 --2- 172.21.15.98:0/1482637259 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f099013e1a0 0x7f09901cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 --2- 172.21.15.98:0/1482637259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09900a4920 0x7f09901cd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 --2- 172.21.15.98:0/1482637259 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f09900a2530 0x7f09900a1340 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 -- 172.21.15.98:0/1482637259 >> 172.21.15.98:0/1482637259 conn(0x7f09900ab850 msgr2=0x7f09900a9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:48.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 -- 172.21.15.98:0/1482637259 shutdown_connections 2024-06-11T19:26:48.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 -- 172.21.15.98:0/1482637259 wait complete. 2024-06-11T19:26:48.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.147+0000 7f0996622640 1 librados: shutdown 2024-06-11T19:26:48.156 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1200,"stamp":"2024-06-11T19:26:47.631585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":108,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":471568,"kb_used_data":60832,"kb_used_omap":9,"kb_used_meta":410486,"kb_avail":565759472,"statfs":{"total":579820584960,"available":579337699328,"internally_reserved":0,"allocated":62291968,"data_stored":35204514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":420338336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"7.17","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630466+0000","last_change":"2024-06-11T19:10:36.026367+0000","last_active":"2024-06-11T19:26:21.630466+0000","last_peered":"2024-06-11T19:26:21.630466+0000","last_clean":"2024-06-11T19:26:21.630466+0000","last_became_active":"2024-06-11T19:10:36.005579+0000","last_became_peered":"2024-06-11T19:10:36.005579+0000","last_unstale":"2024-06-11T19:26:21.630466+0000","last_undegraded":"2024-06-11T19:26:21.630466+0000","last_fullsized":"2024-06-11T19:26:21.630466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T05:31:34.753919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630581+0000","last_change":"2024-06-11T19:10:35.961885+0000","last_active":"2024-06-11T19:26:21.630581+0000","last_peered":"2024-06-11T19:26:21.630581+0000","last_clean":"2024-06-11T19:26:21.630581+0000","last_became_active":"2024-06-11T19:10:35.960336+0000","last_became_peered":"2024-06-11T19:10:35.960336+0000","last_unstale":"2024-06-11T19:26:21.630581+0000","last_undegraded":"2024-06-11T19:26:21.630581+0000","last_fullsized":"2024-06-11T19:26:21.630581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:42:55.368948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390705+0000","last_change":"2024-06-11T19:10:36.098514+0000","last_active":"2024-06-11T19:26:21.390705+0000","last_peered":"2024-06-11T19:26:21.390705+0000","last_clean":"2024-06-11T19:26:21.390705+0000","last_became_active":"2024-06-11T19:10:36.087715+0000","last_became_peered":"2024-06-11T19:10:36.087715+0000","last_unstale":"2024-06-11T19:26:21.390705+0000","last_undegraded":"2024-06-11T19:26:21.390705+0000","last_fullsized":"2024-06-11T19:26:21.390705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:25:31.960579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1332,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630521+0000","last_change":"2024-06-11T19:10:36.045515+0000","last_active":"2024-06-11T19:26:21.630521+0000","last_peered":"2024-06-11T19:26:21.630521+0000","last_clean":"2024-06-11T19:26:21.630521+0000","last_became_active":"2024-06-11T19:10:36.036974+0000","last_became_peered":"2024-06-11T19:10:36.036974+0000","last_unstale":"2024-06-11T19:26:21.630521+0000","last_undegraded":"2024-06-11T19:26:21.630521+0000","last_fullsized":"2024-06-11T19:26:21.630521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:35:56.827003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389826+0000","last_change":"2024-06-11T19:10:36.084957+0000","last_active":"2024-06-11T19:26:21.389826+0000","last_peered":"2024-06-11T19:26:21.389826+0000","last_clean":"2024-06-11T19:26:21.389826+0000","last_became_active":"2024-06-11T19:10:36.065872+0000","last_became_peered":"2024-06-11T19:10:36.065872+0000","last_unstale":"2024-06-11T19:26:21.389826+0000","last_undegraded":"2024-06-11T19:26:21.389826+0000","last_fullsized":"2024-06-11T19:26:21.389826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T05:33:25.348773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1329,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630376+0000","last_change":"2024-06-11T19:10:36.077422+0000","last_active":"2024-06-11T19:26:21.630376+0000","last_peered":"2024-06-11T19:26:21.630376+0000","last_clean":"2024-06-11T19:26:21.630376+0000","last_became_active":"2024-06-11T19:10:36.076424+0000","last_became_peered":"2024-06-11T19:10:36.076424+0000","last_unstale":"2024-06-11T19:26:21.630376+0000","last_undegraded":"2024-06-11T19:26:21.630376+0000","last_fullsized":"2024-06-11T19:26:21.630376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:52:42.583531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1333,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630124+0000","last_change":"2024-06-11T19:10:36.037164+0000","last_active":"2024-06-11T19:26:21.630124+0000","last_peered":"2024-06-11T19:26:21.630124+0000","last_clean":"2024-06-11T19:26:21.630124+0000","last_became_active":"2024-06-11T19:10:36.024538+0000","last_became_peered":"2024-06-11T19:10:36.024538+0000","last_unstale":"2024-06-11T19:26:21.630124+0000","last_undegraded":"2024-06-11T19:26:21.630124+0000","last_fullsized":"2024-06-11T19:26:21.630124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:27:41.916846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1333,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630098+0000","last_change":"2024-06-11T19:10:36.087130+0000","last_active":"2024-06-11T19:26:21.630098+0000","last_peered":"2024-06-11T19:26:21.630098+0000","last_clean":"2024-06-11T19:26:21.630098+0000","last_became_active":"2024-06-11T19:10:36.085399+0000","last_became_peered":"2024-06-11T19:10:36.085399+0000","last_unstale":"2024-06-11T19:26:21.630098+0000","last_undegraded":"2024-06-11T19:26:21.630098+0000","last_fullsized":"2024-06-11T19:26:21.630098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:45:20.075258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389693+0000","last_change":"2024-06-11T19:10:35.977870+0000","last_active":"2024-06-11T19:26:21.389693+0000","last_peered":"2024-06-11T19:26:21.389693+0000","last_clean":"2024-06-11T19:26:21.389693+0000","last_became_active":"2024-06-11T19:10:35.976679+0000","last_became_peered":"2024-06-11T19:10:35.976679+0000","last_unstale":"2024-06-11T19:26:21.389693+0000","last_undegraded":"2024-06-11T19:26:21.389693+0000","last_fullsized":"2024-06-11T19:26:21.389693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T06:02:34.810156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389929+0000","last_change":"2024-06-11T19:10:36.004758+0000","last_active":"2024-06-11T19:26:21.389929+0000","last_peered":"2024-06-11T19:26:21.389929+0000","last_clean":"2024-06-11T19:26:21.389929+0000","last_became_active":"2024-06-11T19:10:35.985449+0000","last_became_peered":"2024-06-11T19:10:35.985449+0000","last_unstale":"2024-06-11T19:26:21.389929+0000","last_undegraded":"2024-06-11T19:26:21.389929+0000","last_fullsized":"2024-06-11T19:26:21.389929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T07:01:45.265291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630608+0000","last_change":"2024-06-11T19:10:35.967673+0000","last_active":"2024-06-11T19:26:21.630608+0000","last_peered":"2024-06-11T19:26:21.630608+0000","last_clean":"2024-06-11T19:26:21.630608+0000","last_became_active":"2024-06-11T19:10:35.962677+0000","last_became_peered":"2024-06-11T19:10:35.962677+0000","last_unstale":"2024-06-11T19:26:21.630608+0000","last_undegraded":"2024-06-11T19:26:21.630608+0000","last_fullsized":"2024-06-11T19:26:21.630608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:37:13.485742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1322,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390699+0000","last_change":"2024-06-11T19:10:36.057228+0000","last_active":"2024-06-11T19:26:21.390699+0000","last_peered":"2024-06-11T19:26:21.390699+0000","last_clean":"2024-06-11T19:26:21.390699+0000","last_became_active":"2024-06-11T19:10:36.056338+0000","last_became_peered":"2024-06-11T19:10:36.056338+0000","last_unstale":"2024-06-11T19:26:21.390699+0000","last_undegraded":"2024-06-11T19:26:21.390699+0000","last_fullsized":"2024-06-11T19:26:21.390699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:31:48.434226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1438,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630415+0000","last_change":"2024-06-11T19:09:26.673425+0000","last_active":"2024-06-11T19:26:21.630415+0000","last_peered":"2024-06-11T19:26:21.630415+0000","last_clean":"2024-06-11T19:26:21.630415+0000","last_became_active":"2024-06-11T19:09:26.673120+0000","last_became_peered":"2024-06-11T19:09:26.673120+0000","last_unstale":"2024-06-11T19:26:21.630415+0000","last_undegraded":"2024-06-11T19:26:21.630415+0000","last_fullsized":"2024-06-11T19:26:21.630415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_clean_scrub_stamp":"2024-06-11T19:09:25.660869+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:57:43.705570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630700+0000","last_change":"2024-06-11T19:09:33.725261+0000","last_active":"2024-06-11T19:26:21.630700+0000","last_peered":"2024-06-11T19:26:21.630700+0000","last_clean":"2024-06-11T19:26:21.630700+0000","last_became_active":"2024-06-11T19:09:33.724870+0000","last_became_peered":"2024-06-11T19:09:33.724870+0000","last_unstale":"2024-06-11T19:26:21.630700+0000","last_undegraded":"2024-06-11T19:26:21.630700+0000","last_fullsized":"2024-06-11T19:26:21.630700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:32:51.020718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389982+0000","last_change":"2024-06-11T19:10:36.011730+0000","last_active":"2024-06-11T19:26:21.389982+0000","last_peered":"2024-06-11T19:26:21.389982+0000","last_clean":"2024-06-11T19:26:21.389982+0000","last_became_active":"2024-06-11T19:10:35.988487+0000","last_became_peered":"2024-06-11T19:10:35.988487+0000","last_unstale":"2024-06-11T19:26:21.389982+0000","last_undegraded":"2024-06-11T19:26:21.389982+0000","last_fullsized":"2024-06-11T19:26:21.389982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:48:33.179394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1329,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390443+0000","last_change":"2024-06-11T19:09:33.729213+0000","last_active":"2024-06-11T19:26:21.390443+0000","last_peered":"2024-06-11T19:26:21.390443+0000","last_clean":"2024-06-11T19:26:21.390443+0000","last_became_active":"2024-06-11T19:09:33.728930+0000","last_became_peered":"2024-06-11T19:09:33.728930+0000","last_unstale":"2024-06-11T19:26:21.390443+0000","last_undegraded":"2024-06-11T19:26:21.390443+0000","last_fullsized":"2024-06-11T19:26:21.390443+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:26:38.808169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1322,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390789+0000","last_change":"2024-06-11T19:10:36.088019+0000","last_active":"2024-06-11T19:26:21.390789+0000","last_peered":"2024-06-11T19:26:21.390789+0000","last_clean":"2024-06-11T19:26:21.390789+0000","last_became_active":"2024-06-11T19:10:36.086670+0000","last_became_peered":"2024-06-11T19:10:36.086670+0000","last_unstale":"2024-06-11T19:26:21.390789+0000","last_undegraded":"2024-06-11T19:26:21.390789+0000","last_fullsized":"2024-06-11T19:26:21.390789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T22:06:29.653796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389520+0000","last_change":"2024-06-11T19:09:33.729275+0000","last_active":"2024-06-11T19:26:21.389520+0000","last_peered":"2024-06-11T19:26:21.389520+0000","last_clean":"2024-06-11T19:26:21.389520+0000","last_became_active":"2024-06-11T19:09:33.728948+0000","last_became_peered":"2024-06-11T19:09:33.728948+0000","last_unstale":"2024-06-11T19:26:21.389520+0000","last_undegraded":"2024-06-11T19:26:21.389520+0000","last_fullsized":"2024-06-11T19:26:21.389520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:29:28.908169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1329,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630555+0000","last_change":"2024-06-11T19:10:35.937546+0000","last_active":"2024-06-11T19:26:21.630555+0000","last_peered":"2024-06-11T19:26:21.630555+0000","last_clean":"2024-06-11T19:26:21.630555+0000","last_became_active":"2024-06-11T19:10:35.937161+0000","last_became_peered":"2024-06-11T19:10:35.937161+0000","last_unstale":"2024-06-11T19:26:21.630555+0000","last_undegraded":"2024-06-11T19:26:21.630555+0000","last_fullsized":"2024-06-11T19:26:21.630555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:12:14.668727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1330,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630527+0000","last_change":"2024-06-11T19:09:33.722503+0000","last_active":"2024-06-11T19:26:21.630527+0000","last_peered":"2024-06-11T19:26:21.630527+0000","last_clean":"2024-06-11T19:26:21.630527+0000","last_became_active":"2024-06-11T19:09:33.722180+0000","last_became_peered":"2024-06-11T19:09:33.722180+0000","last_unstale":"2024-06-11T19:26:21.630527+0000","last_undegraded":"2024-06-11T19:26:21.630527+0000","last_fullsized":"2024-06-11T19:26:21.630527+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:28:15.836943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630572+0000","last_change":"2024-06-11T19:10:36.062834+0000","last_active":"2024-06-11T19:26:21.630572+0000","last_peered":"2024-06-11T19:26:21.630572+0000","last_clean":"2024-06-11T19:26:21.630572+0000","last_became_active":"2024-06-11T19:10:36.016575+0000","last_became_peered":"2024-06-11T19:10:36.016575+0000","last_unstale":"2024-06-11T19:26:21.630572+0000","last_undegraded":"2024-06-11T19:26:21.630572+0000","last_fullsized":"2024-06-11T19:26:21.630572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:53:41.228041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1340,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630524+0000","last_change":"2024-06-11T19:09:33.729223+0000","last_active":"2024-06-11T19:26:21.630524+0000","last_peered":"2024-06-11T19:26:21.630524+0000","last_clean":"2024-06-11T19:26:21.630524+0000","last_became_active":"2024-06-11T19:09:33.728897+0000","last_became_peered":"2024-06-11T19:09:33.728897+0000","last_unstale":"2024-06-11T19:26:21.630524+0000","last_undegraded":"2024-06-11T19:26:21.630524+0000","last_fullsized":"2024-06-11T19:26:21.630524+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:49:13.439577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"20'7","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390521+0000","last_change":"2024-06-11T19:10:36.037487+0000","last_active":"2024-06-11T19:26:21.390521+0000","last_peered":"2024-06-11T19:26:21.390521+0000","last_clean":"2024-06-11T19:26:21.390521+0000","last_became_active":"2024-06-11T19:10:36.032874+0000","last_became_peered":"2024-06-11T19:10:36.032874+0000","last_unstale":"2024-06-11T19:26:21.390521+0000","last_undegraded":"2024-06-11T19:26:21.390521+0000","last_fullsized":"2024-06-11T19:26:21.390521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:58:37.764224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630669+0000","last_change":"2024-06-11T19:09:33.720915+0000","last_active":"2024-06-11T19:26:21.630669+0000","last_peered":"2024-06-11T19:26:21.630669+0000","last_clean":"2024-06-11T19:26:21.630669+0000","last_became_active":"2024-06-11T19:09:33.720599+0000","last_became_peered":"2024-06-11T19:09:33.720599+0000","last_unstale":"2024-06-11T19:26:21.630669+0000","last_undegraded":"2024-06-11T19:26:21.630669+0000","last_fullsized":"2024-06-11T19:26:21.630669+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:28:26.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1336,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630711+0000","last_change":"2024-06-11T19:10:35.982779+0000","last_active":"2024-06-11T19:26:21.630711+0000","last_peered":"2024-06-11T19:26:21.630711+0000","last_clean":"2024-06-11T19:26:21.630711+0000","last_became_active":"2024-06-11T19:10:35.976675+0000","last_became_peered":"2024-06-11T19:10:35.976675+0000","last_unstale":"2024-06-11T19:26:21.630711+0000","last_undegraded":"2024-06-11T19:26:21.630711+0000","last_fullsized":"2024-06-11T19:26:21.630711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:39:25.265311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630653+0000","last_change":"2024-06-11T19:09:33.725197+0000","last_active":"2024-06-11T19:26:21.630653+0000","last_peered":"2024-06-11T19:26:21.630653+0000","last_clean":"2024-06-11T19:26:21.630653+0000","last_became_active":"2024-06-11T19:09:33.724912+0000","last_became_peered":"2024-06-11T19:09:33.724912+0000","last_unstale":"2024-06-11T19:26:21.630653+0000","last_undegraded":"2024-06-11T19:26:21.630653+0000","last_fullsized":"2024-06-11T19:26:21.630653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:36:12.941264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389668+0000","last_change":"2024-06-11T19:10:36.090541+0000","last_active":"2024-06-11T19:26:21.389668+0000","last_peered":"2024-06-11T19:26:21.389668+0000","last_clean":"2024-06-11T19:26:21.389668+0000","last_became_active":"2024-06-11T19:10:36.071353+0000","last_became_peered":"2024-06-11T19:10:36.071353+0000","last_unstale":"2024-06-11T19:26:21.389668+0000","last_undegraded":"2024-06-11T19:26:21.389668+0000","last_fullsized":"2024-06-11T19:26:21.389668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:58:46.822805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1341,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630664+0000","last_change":"2024-06-11T19:09:33.725085+0000","last_active":"2024-06-11T19:26:21.630664+0000","last_peered":"2024-06-11T19:26:21.630664+0000","last_clean":"2024-06-11T19:26:21.630664+0000","last_became_active":"2024-06-11T19:09:33.724778+0000","last_became_peered":"2024-06-11T19:09:33.724778+0000","last_unstale":"2024-06-11T19:26:21.630664+0000","last_undegraded":"2024-06-11T19:26:21.630664+0000","last_fullsized":"2024-06-11T19:26:21.630664+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:57:46.868411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389513+0000","last_change":"2024-06-11T19:10:36.060364+0000","last_active":"2024-06-11T19:26:21.389513+0000","last_peered":"2024-06-11T19:26:21.389513+0000","last_clean":"2024-06-11T19:26:21.389513+0000","last_became_active":"2024-06-11T19:10:36.035136+0000","last_became_peered":"2024-06-11T19:10:36.035136+0000","last_unstale":"2024-06-11T19:26:21.389513+0000","last_undegraded":"2024-06-11T19:26:21.389513+0000","last_fullsized":"2024-06-11T19:26:21.389513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:30:08.506219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389506+0000","last_change":"2024-06-11T19:10:36.052997+0000","last_active":"2024-06-11T19:26:21.389506+0000","last_peered":"2024-06-11T19:26:21.389506+0000","last_clean":"2024-06-11T19:26:21.389506+0000","last_became_active":"2024-06-11T19:10:36.044757+0000","last_became_peered":"2024-06-11T19:10:36.044757+0000","last_unstale":"2024-06-11T19:26:21.389506+0000","last_undegraded":"2024-06-11T19:26:21.389506+0000","last_fullsized":"2024-06-11T19:26:21.389506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:11:16.047309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390454+0000","last_change":"2024-06-11T19:10:36.011492+0000","last_active":"2024-06-11T19:26:21.390454+0000","last_peered":"2024-06-11T19:26:21.390454+0000","last_clean":"2024-06-11T19:26:21.390454+0000","last_became_active":"2024-06-11T19:10:36.000792+0000","last_became_peered":"2024-06-11T19:10:36.000792+0000","last_unstale":"2024-06-11T19:26:21.390454+0000","last_undegraded":"2024-06-11T19:26:21.390454+0000","last_fullsized":"2024-06-11T19:26:21.390454+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:42:36.937874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630349+0000","last_change":"2024-06-11T19:10:35.994191+0000","last_active":"2024-06-11T19:26:21.630349+0000","last_peered":"2024-06-11T19:26:21.630349+0000","last_clean":"2024-06-11T19:26:21.630349+0000","last_became_active":"2024-06-11T19:10:35.988056+0000","last_became_peered":"2024-06-11T19:10:35.988056+0000","last_unstale":"2024-06-11T19:26:21.630349+0000","last_undegraded":"2024-06-11T19:26:21.630349+0000","last_fullsized":"2024-06-11T19:26:21.630349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:24:15.498060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1332,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630386+0000","last_change":"2024-06-11T19:10:36.030504+0000","last_active":"2024-06-11T19:26:21.630386+0000","last_peered":"2024-06-11T19:26:21.630386+0000","last_clean":"2024-06-11T19:26:21.630386+0000","last_became_active":"2024-06-11T19:10:36.025371+0000","last_became_peered":"2024-06-11T19:10:36.025371+0000","last_unstale":"2024-06-11T19:26:21.630386+0000","last_undegraded":"2024-06-11T19:26:21.630386+0000","last_fullsized":"2024-06-11T19:26:21.630386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:14:08.888348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390627+0000","last_change":"2024-06-11T19:10:36.091418+0000","last_active":"2024-06-11T19:26:21.390627+0000","last_peered":"2024-06-11T19:26:21.390627+0000","last_clean":"2024-06-11T19:26:21.390627+0000","last_became_active":"2024-06-11T19:10:36.090801+0000","last_became_peered":"2024-06-11T19:10:36.090801+0000","last_unstale":"2024-06-11T19:26:21.390627+0000","last_undegraded":"2024-06-11T19:26:21.390627+0000","last_fullsized":"2024-06-11T19:26:21.390627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:34:25.520333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1332,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630306+0000","last_change":"2024-06-11T19:10:36.063862+0000","last_active":"2024-06-11T19:26:21.630306+0000","last_peered":"2024-06-11T19:26:21.630306+0000","last_clean":"2024-06-11T19:26:21.630306+0000","last_became_active":"2024-06-11T19:10:36.057852+0000","last_became_peered":"2024-06-11T19:10:36.057852+0000","last_unstale":"2024-06-11T19:26:21.630306+0000","last_undegraded":"2024-06-11T19:26:21.630306+0000","last_fullsized":"2024-06-11T19:26:21.630306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:16:37.800621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1336,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389829+0000","last_change":"2024-06-11T19:10:36.056378+0000","last_active":"2024-06-11T19:26:21.389829+0000","last_peered":"2024-06-11T19:26:21.389829+0000","last_clean":"2024-06-11T19:26:21.389829+0000","last_became_active":"2024-06-11T19:10:36.042135+0000","last_became_peered":"2024-06-11T19:10:36.042135+0000","last_unstale":"2024-06-11T19:26:21.389829+0000","last_undegraded":"2024-06-11T19:26:21.389829+0000","last_fullsized":"2024-06-11T19:26:21.389829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:38:55.622181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1322,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630471+0000","last_change":"2024-06-11T19:10:36.026724+0000","last_active":"2024-06-11T19:26:21.630471+0000","last_peered":"2024-06-11T19:26:21.630471+0000","last_clean":"2024-06-11T19:26:21.630471+0000","last_became_active":"2024-06-11T19:10:36.026278+0000","last_became_peered":"2024-06-11T19:10:36.026278+0000","last_unstale":"2024-06-11T19:26:21.630471+0000","last_undegraded":"2024-06-11T19:26:21.630471+0000","last_fullsized":"2024-06-11T19:26:21.630471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:22:26.410474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1328,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630311+0000","last_change":"2024-06-11T19:10:36.034688+0000","last_active":"2024-06-11T19:26:21.630311+0000","last_peered":"2024-06-11T19:26:21.630311+0000","last_clean":"2024-06-11T19:26:21.630311+0000","last_became_active":"2024-06-11T19:10:36.028329+0000","last_became_peered":"2024-06-11T19:10:36.028329+0000","last_unstale":"2024-06-11T19:26:21.630311+0000","last_undegraded":"2024-06-11T19:26:21.630311+0000","last_fullsized":"2024-06-11T19:26:21.630311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T06:59:08.595933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630349+0000","last_change":"2024-06-11T19:10:36.085537+0000","last_active":"2024-06-11T19:26:21.630349+0000","last_peered":"2024-06-11T19:26:21.630349+0000","last_clean":"2024-06-11T19:26:21.630349+0000","last_became_active":"2024-06-11T19:10:36.082897+0000","last_became_peered":"2024-06-11T19:10:36.082897+0000","last_unstale":"2024-06-11T19:26:21.630349+0000","last_undegraded":"2024-06-11T19:26:21.630349+0000","last_fullsized":"2024-06-11T19:26:21.630349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:29:42.550377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390495+0000","last_change":"2024-06-11T19:10:36.017667+0000","last_active":"2024-06-11T19:26:21.390495+0000","last_peered":"2024-06-11T19:26:21.390495+0000","last_clean":"2024-06-11T19:26:21.390495+0000","last_became_active":"2024-06-11T19:10:36.002730+0000","last_became_peered":"2024-06-11T19:10:36.002730+0000","last_unstale":"2024-06-11T19:26:21.390495+0000","last_undegraded":"2024-06-11T19:26:21.390495+0000","last_fullsized":"2024-06-11T19:26:21.390495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:34:23.834164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1328,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630373+0000","last_change":"2024-06-11T19:10:36.076140+0000","last_active":"2024-06-11T19:26:21.630373+0000","last_peered":"2024-06-11T19:26:21.630373+0000","last_clean":"2024-06-11T19:26:21.630373+0000","last_became_active":"2024-06-11T19:10:36.075888+0000","last_became_peered":"2024-06-11T19:10:36.075888+0000","last_unstale":"2024-06-11T19:26:21.630373+0000","last_undegraded":"2024-06-11T19:26:21.630373+0000","last_fullsized":"2024-06-11T19:26:21.630373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:07:12.839020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607763,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77264,"kb_used_data":9968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294576,"statfs":{"total":96636764160,"available":96557645824,"internally_reserved":0,"allocated":10207232,"data_stored":5691113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81956,"kb_used_data":10488,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289884,"statfs":{"total":96636764160,"available":96552841216,"internally_reserved":0,"allocated":10739712,"data_stored":6220041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640467,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86100,"kb_used_data":10484,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94285740,"statfs":{"total":96636764160,"available":96548597760,"internally_reserved":0,"allocated":10735616,"data_stored":6220036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640468,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73112,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298728,"statfs":{"total":96636764160,"available":96561897472,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640467,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67532,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94304308,"statfs":{"total":96636764160,"available":96567611392,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640469,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85604,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286236,"statfs":{"total":96636764160,"available":96549105664,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-11T19:26:48.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-11T19:26:48.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.257+0000 7f10f8822640 10 monclient: get_monmap_and_config 2024-06-11T19:26:48.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.257+0000 7f10f8822640 10 monclient: build_initial_monmap 2024-06-11T19:26:48.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.257+0000 7f10f8822640 10 monclient: monmap: 2024-06-11T19:26:48.257 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:48.257 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.257 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:48.258469+0000 2024-06-11T19:26:48.257 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:48.258469+0000 2024-06-11T19:26:48.257 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:48.257 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.257 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:48.257 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:48.258 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:48.259 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:48.259 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:48.259 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 1 Processor -- start 2024-06-11T19:26:48.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 1 -- start start 2024-06-11T19:26:48.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 10 monclient: init 2024-06-11T19:26:48.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:48.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 10 monclient: _add_conns ranks=[0,7,4,9,10,6,5,8,3,2,1] 2024-06-11T19:26:48.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10f40a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:26:48.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f10f40a2530 0x7f10f413dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 10 monclient(hunting): picked mon.noname-i con 0x7f10f40a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:48.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10f413e1a0 0x7f10f4140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 10 monclient(hunting): picked mon.noname-c con 0x7f10f413e1a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:48.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f40a7140 con 0x7f10f40a4920 2024-06-11T19:26:48.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f10f40a2b50 con 0x7f10f413e1a0 2024-06-11T19:26:48.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f10f40a2e30 con 0x7f10f40a2530 2024-06-11T19:26:48.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:48.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f8822640 10 monclient(hunting): authenticate will time out at 1818.065854s 2024-06-11T19:26:48.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f259c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f10f40a4920 0x7f10f40a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51406/0 (socket says 172.21.15.98:51406) 2024-06-11T19:26:48.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f259c640 1 -- 172.21.15.98:0/1207154889 learned_addr learned my addr 172.21.15.98:0/1207154889 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:26:48.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.258+0000 7f10f2d9d640 1 --2- 172.21.15.98:0/1207154889 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10f413e1a0 0x7f10f4140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 10 monclient(hunting): get_auth_request con 0x7f10f413e1a0 auth_method 0 2024-06-11T19:26:48.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:48.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:48.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:48.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f1d9b640 1 --2- 172.21.15.98:0/1207154889 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f10f40a2530 0x7f10f413dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f1d9b640 10 monclient(hunting): get_auth_request con 0x7f10f40a2530 auth_method 0 2024-06-11T19:26:48.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f1d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f159a640 1 -- 172.21.15.98:0/1207154889 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 347257113 0 0) 0x7f10f40a7140 con 0x7f10f40a4920 2024-06-11T19:26:48.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f159a640 10 monclient(hunting): _negotiate 2024-06-11T19:26:48.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f159a640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f159a640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f159a640 10 monclient(hunting): my global_id is 66447 2024-06-11T19:26:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f159a640 1 -- 172.21.15.98:0/1207154889 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10e0003520 con 0x7f10f40a4920 2024-06-11T19:26:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 10 monclient(hunting): handle_auth_done global_id 134176 payload 274 2024-06-11T19:26:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 1 -- 172.21.15.98:0/1207154889 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f10f40a2530 msgr2=0x7f10f413dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 1 --2- 172.21.15.98:0/1207154889 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f10f40a2530 0x7f10f413dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 1 -- 172.21.15.98:0/1207154889 >> v1:172.21.15.98:6789/0 conn(0x7f10f40a4920 legacy=0x7f10f40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:26:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 1 monclient: found mon.noname-c 2024-06-11T19:26:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:26:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.259+0000 7f10f2d9d640 1 -- 172.21.15.98:0/1207154889 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f40a3110 con 0x7f10f413e1a0 2024-06-11T19:26:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f2d9d640 1 --2- 172.21.15.98:0/1207154889 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10f413e1a0 0x7f10f4140590 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f10e40097d0 tx=0x7f10e402b770 comp rx=0 tx=0).ready entity=mon.4 client_cookie=80be03f7f2f71ba4 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 1 -- 172.21.15.98:0/1207154889 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10e4039070 con 0x7f10f413e1a0 2024-06-11T19:26:48.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:26:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 10 monclient: dump: 2024-06-11T19:26:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:48.274 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:48.275 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:48.276 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 10 monclient: _finish_auth 0 2024-06-11T19:26:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 10 monclient: _check_auth_tickets 2024-06-11T19:26:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 5 monclient: authenticate success, global_id 134176 2024-06-11T19:26:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 1 -- 172.21.15.98:0/1207154889 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10e4030070 con 0x7f10f413e1a0 2024-06-11T19:26:48.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 4 set_mon_vals no callback set 2024-06-11T19:26:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 1 -- 172.21.15.98:0/1207154889 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10e4031db0 con 0x7f10f413e1a0 2024-06-11T19:26:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:26:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f159a640 10 monclient: dump: 2024-06-11T19:26:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.277 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.278 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:48.278 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.278 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:48.278 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:48.278 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:48.278 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:48.278 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:48.278 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:48.279 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:48.279 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:48.279 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:48.279 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 10 monclient: shutdown 2024-06-11T19:26:48.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:48.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 1 -- 172.21.15.98:0/1207154889 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10f413e1a0 msgr2=0x7f10f4140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 1 --2- 172.21.15.98:0/1207154889 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10f413e1a0 0x7f10f4140590 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f10e40097d0 tx=0x7f10e402b770 comp rx=0 tx=0).stop 2024-06-11T19:26:48.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 1 -- 172.21.15.98:0/1207154889 shutdown_connections 2024-06-11T19:26:48.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 1 --2- 172.21.15.98:0/1207154889 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10f413e1a0 0x7f10f4140590 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 1 --2- 172.21.15.98:0/1207154889 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f10f40a2530 0x7f10f413dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 1 -- 172.21.15.98:0/1207154889 >> 172.21.15.98:0/1207154889 conn(0x7f10f40ab850 msgr2=0x7f10f40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:48.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 1 -- 172.21.15.98:0/1207154889 shutdown_connections 2024-06-11T19:26:48.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.260+0000 7f10f8822640 1 -- 172.21.15.98:0/1207154889 wait complete. 2024-06-11T19:26:48.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.261+0000 7f10f8822640 10 monclient: build_initial_monmap 2024-06-11T19:26:48.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.261+0000 7f10f8822640 10 monclient: monmap: 2024-06-11T19:26:48.280 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:48.280 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:48.280 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:48.281 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:48.281 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:48.281 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:48.281 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:48.281 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:48.281 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:48.281 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:48.281 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:48.281 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:48.282 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:48.282 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:48.282 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:48.282 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.261+0000 7f10f8822640 1 librados: starting msgr at 2024-06-11T19:26:48.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.261+0000 7f10f8822640 1 librados: starting objecter 2024-06-11T19:26:48.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.261+0000 7f10f8822640 1 Processor -- start 2024-06-11T19:26:48.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.261+0000 7f10f8822640 1 -- start start 2024-06-11T19:26:48.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.261+0000 7f10f8822640 1 librados: setting wanted keys 2024-06-11T19:26:48.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.261+0000 7f10f8822640 1 librados: calling monclient init 2024-06-11T19:26:48.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.261+0000 7f10f8822640 10 monclient: init 2024-06-11T19:26:48.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:48.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 10 monclient: _add_conns ranks=[8,7,2,4,3,6,0,1,5] 2024-06-11T19:26:48.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10f40a2530 0x7f10f41cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 10 monclient(hunting): picked mon.noname-e con 0x7f10f40a2530 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:48.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f10f40a4920 0x7f10f4136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 10 monclient(hunting): picked mon.noname-i con 0x7f10f40a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:48.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10f413e1a0 0x7f10f41370f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 10 monclient(hunting): picked mon.noname-b con 0x7f10f413e1a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:48.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f10f4138e50 con 0x7f10f413e1a0 2024-06-11T19:26:48.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f10f40a84b0 con 0x7f10f40a4920 2024-06-11T19:26:48.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f10f40a6da0 con 0x7f10f40a2530 2024-06-11T19:26:48.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:48.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f8822640 10 monclient(hunting): authenticate will time out at 1818.069305s 2024-06-11T19:26:48.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10f40a2530 0x7f10f41cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10f40a2530 0x7f10f41cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54466/0 (socket says 172.21.15.98:54466) 2024-06-11T19:26:48.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f2d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10f413e1a0 0x7f10f41370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 1 -- 172.21.15.98:0/710696673 learned_addr learned my addr 172.21.15.98:0/710696673 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:48.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 10 monclient(hunting): get_auth_request con 0x7f10f40a2530 auth_method 0 2024-06-11T19:26:48.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f2d9d640 10 monclient(hunting): get_auth_request con 0x7f10f413e1a0 auth_method 0 2024-06-11T19:26:48.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f2d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:48.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:48.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:48.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:48.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:48.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:48.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f1d9b640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f10f40a4920 0x7f10f4136bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f1d9b640 10 monclient(hunting): get_auth_request con 0x7f10f40a4920 auth_method 0 2024-06-11T19:26:48.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f1d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 10 monclient(hunting): handle_auth_done global_id 124460 payload 824 2024-06-11T19:26:48.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 1 -- 172.21.15.98:0/710696673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f10f40a4920 msgr2=0x7f10f4136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f10f40a4920 0x7f10f4136bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 1 -- 172.21.15.98:0/710696673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10f413e1a0 msgr2=0x7f10f41370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10f413e1a0 0x7f10f41370f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 10 monclient: _finish_hunting 0 2024-06-11T19:26:48.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 1 monclient: found mon.noname-e 2024-06-11T19:26:48.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:48.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:26:48.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 1 -- 172.21.15.98:0/710696673 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f413df10 con 0x7f10f40a2530 2024-06-11T19:26:48.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f2d9d640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10f413e1a0 0x7f10f41370f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:48.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10f259c640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10f40a2530 0x7f10f41cd720 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00cbc0 tx=0x7f10ec00a430 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:48.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10eaffd640 10 client.?.objecter ms_handle_connect 0x7f10f40a2530 2024-06-11T19:26:48.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10eaffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:48.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.262+0000 7f10eaffd640 1 -- 172.21.15.98:0/710696673 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10ec012400 con 0x7f10f40a2530 2024-06-11T19:26:48.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:48.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10eaffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:26:48.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10eaffd640 10 monclient: dump: 2024-06-11T19:26:48.289 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:48.289 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.289 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.289 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.289 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:48.289 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.290 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:48.290 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:48.290 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:48.290 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:48.290 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:48.290 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:48.290 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:48.290 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:48.291 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:48.291 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10eaffd640 10 monclient: _finish_auth 0 2024-06-11T19:26:48.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10eaffd640 10 monclient: _check_auth_tickets 2024-06-11T19:26:48.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:48.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10eaffd640 1 -- 172.21.15.98:0/710696673 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ec0125c0 con 0x7f10f40a2530 2024-06-11T19:26:48.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10f8822640 5 monclient: authenticate success, global_id 124460 2024-06-11T19:26:48.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10f8822640 10 monclient: _renew_subs 2024-06-11T19:26:48.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10f8822640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10f8822640 1 -- 172.21.15.98:0/710696673 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f4123280 con 0x7f10f40a2530 2024-06-11T19:26:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10eaffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10eaffd640 1 -- 172.21.15.98:0/710696673 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10ec012c20 con 0x7f10f40a2530 2024-06-11T19:26:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10eaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10eaffd640 10 monclient: dump: 2024-06-11T19:26:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:48.294 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:48.294 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:48.294 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:48.294 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:48.294 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10f0d99640 4 set_mon_vals no callback set 2024-06-11T19:26:48.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10f8822640 10 client.124460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:48.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10f8822640 10 monclient: _renew_subs 2024-06-11T19:26:48.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10f8822640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10f8822640 1 -- 172.21.15.98:0/710696673 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f4122fa0 con 0x7f10f40a2530 2024-06-11T19:26:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.263+0000 7f10f8822640 1 librados: init done 2024-06-11T19:26:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10eaffd640 1 -- 172.21.15.98:0/710696673 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f10ec0127b0 con 0x7f10f40a2530 2024-06-11T19:26:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10f8822640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10f8822640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10f8822640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10f8822640 1 -- 172.21.15.98:0/710696673 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10f40a4d00 con 0x7f10f40a2530 2024-06-11T19:26:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10eaffd640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f10d403fdb0 0x7f10d4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10eaffd640 1 -- 172.21.15.98:0/710696673 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f10ec051f10 con 0x7f10f40a2530 2024-06-11T19:26:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10eaffd640 10 client.124460.objecter ms_dispatch 0x7f10f4000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10eaffd640 3 client.124460.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10eaffd640 3 client.124460.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10eaffd640 20 client.124460.objecter dump_active .. 0 homeless 2024-06-11T19:26:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10f1d9b640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f10d403fdb0 0x7f10d4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10f1d9b640 10 monclient: get_auth_request con 0x7f10d403fdb0 auth_method 0 2024-06-11T19:26:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10eaffd640 10 client.124460.objecter ms_handle_connect 0x7f10d403fdb0 2024-06-11T19:26:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.264+0000 7f10f1d9b640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f10d403fdb0 0x7f10d4042270 secure :-1 s=READY pgs=4332 cs=0 l=1 rev1=1 crypto rx=0x7f10dc006fd0 tx=0x7f10dc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:48.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.268+0000 7f10eaffd640 1 -- 172.21.15.98:0/710696673 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f10f413df10 con 0x7f10f40a2530 2024-06-11T19:26:48.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.268+0000 7f10eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:48.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.268+0000 7f10eaffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:48.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.448+0000 7f10f8822640 1 -- 172.21.15.98:0/710696673 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f10f4121b10 con 0x7f10d403fdb0 2024-06-11T19:26:48.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.451+0000 7f10eaffd640 1 -- 172.21.15.98:0/710696673 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+113897 (secure 0 0 0) 0x7f10f4121b10 con 0x7f10d403fdb0 2024-06-11T19:26:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:26:48.451 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-06-11T19:26:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 10 librados: watch_flush enter 2024-06-11T19:26:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 10 librados: watch_flush exit 2024-06-11T19:26:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 20 client.124460.objecter shutdown clearing up homeless session... 2024-06-11T19:26:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 10 client.124460.objecter successfully canceled tick 2024-06-11T19:26:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 -- 172.21.15.98:0/710696673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f10d403fdb0 msgr2=0x7f10d4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f10d403fdb0 0x7f10d4042270 secure :-1 s=READY pgs=4332 cs=0 l=1 rev1=1 crypto rx=0x7f10dc006fd0 tx=0x7f10dc008040 comp rx=0 tx=0).stop 2024-06-11T19:26:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 10 monclient: shutdown 2024-06-11T19:26:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 -- 172.21.15.98:0/710696673 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10f40a2530 msgr2=0x7f10f41cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10f40a2530 0x7f10f41cd720 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00cbc0 tx=0x7f10ec00a430 comp rx=0 tx=0).stop 2024-06-11T19:26:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 -- 172.21.15.98:0/710696673 shutdown_connections 2024-06-11T19:26:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f10d403fdb0 0x7f10d4042270 unknown :-1 s=CLOSED pgs=4332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f10f413e1a0 0x7f10f41370f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f10f40a4920 0x7f10f4136bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 --2- 172.21.15.98:0/710696673 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10f40a2530 0x7f10f41cd720 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 -- 172.21.15.98:0/710696673 >> 172.21.15.98:0/710696673 conn(0x7f10f40ab850 msgr2=0x7f10f40a9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 -- 172.21.15.98:0/710696673 shutdown_connections 2024-06-11T19:26:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 -- 172.21.15.98:0/710696673 wait complete. 2024-06-11T19:26:48.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.453+0000 7f10f8822640 1 librados: shutdown 2024-06-11T19:26:48.463 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1200,"stamp":"2024-06-11T19:26:47.631585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":108,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":471568,"kb_used_data":60832,"kb_used_omap":9,"kb_used_meta":410486,"kb_avail":565759472,"statfs":{"total":579820584960,"available":579337699328,"internally_reserved":0,"allocated":62291968,"data_stored":35204514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":420338336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"7.17","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630466+0000","last_change":"2024-06-11T19:10:36.026367+0000","last_active":"2024-06-11T19:26:21.630466+0000","last_peered":"2024-06-11T19:26:21.630466+0000","last_clean":"2024-06-11T19:26:21.630466+0000","last_became_active":"2024-06-11T19:10:36.005579+0000","last_became_peered":"2024-06-11T19:10:36.005579+0000","last_unstale":"2024-06-11T19:26:21.630466+0000","last_undegraded":"2024-06-11T19:26:21.630466+0000","last_fullsized":"2024-06-11T19:26:21.630466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T05:31:34.753919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630581+0000","last_change":"2024-06-11T19:10:35.961885+0000","last_active":"2024-06-11T19:26:21.630581+0000","last_peered":"2024-06-11T19:26:21.630581+0000","last_clean":"2024-06-11T19:26:21.630581+0000","last_became_active":"2024-06-11T19:10:35.960336+0000","last_became_peered":"2024-06-11T19:10:35.960336+0000","last_unstale":"2024-06-11T19:26:21.630581+0000","last_undegraded":"2024-06-11T19:26:21.630581+0000","last_fullsized":"2024-06-11T19:26:21.630581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:42:55.368948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390705+0000","last_change":"2024-06-11T19:10:36.098514+0000","last_active":"2024-06-11T19:26:21.390705+0000","last_peered":"2024-06-11T19:26:21.390705+0000","last_clean":"2024-06-11T19:26:21.390705+0000","last_became_active":"2024-06-11T19:10:36.087715+0000","last_became_peered":"2024-06-11T19:10:36.087715+0000","last_unstale":"2024-06-11T19:26:21.390705+0000","last_undegraded":"2024-06-11T19:26:21.390705+0000","last_fullsized":"2024-06-11T19:26:21.390705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:25:31.960579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1332,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630521+0000","last_change":"2024-06-11T19:10:36.045515+0000","last_active":"2024-06-11T19:26:21.630521+0000","last_peered":"2024-06-11T19:26:21.630521+0000","last_clean":"2024-06-11T19:26:21.630521+0000","last_became_active":"2024-06-11T19:10:36.036974+0000","last_became_peered":"2024-06-11T19:10:36.036974+0000","last_unstale":"2024-06-11T19:26:21.630521+0000","last_undegraded":"2024-06-11T19:26:21.630521+0000","last_fullsized":"2024-06-11T19:26:21.630521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:35:56.827003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389826+0000","last_change":"2024-06-11T19:10:36.084957+0000","last_active":"2024-06-11T19:26:21.389826+0000","last_peered":"2024-06-11T19:26:21.389826+0000","last_clean":"2024-06-11T19:26:21.389826+0000","last_became_active":"2024-06-11T19:10:36.065872+0000","last_became_peered":"2024-06-11T19:10:36.065872+0000","last_unstale":"2024-06-11T19:26:21.389826+0000","last_undegraded":"2024-06-11T19:26:21.389826+0000","last_fullsized":"2024-06-11T19:26:21.389826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T05:33:25.348773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1329,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630376+0000","last_change":"2024-06-11T19:10:36.077422+0000","last_active":"2024-06-11T19:26:21.630376+0000","last_peered":"2024-06-11T19:26:21.630376+0000","last_clean":"2024-06-11T19:26:21.630376+0000","last_became_active":"2024-06-11T19:10:36.076424+0000","last_became_peered":"2024-06-11T19:10:36.076424+0000","last_unstale":"2024-06-11T19:26:21.630376+0000","last_undegraded":"2024-06-11T19:26:21.630376+0000","last_fullsized":"2024-06-11T19:26:21.630376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:52:42.583531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1333,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630124+0000","last_change":"2024-06-11T19:10:36.037164+0000","last_active":"2024-06-11T19:26:21.630124+0000","last_peered":"2024-06-11T19:26:21.630124+0000","last_clean":"2024-06-11T19:26:21.630124+0000","last_became_active":"2024-06-11T19:10:36.024538+0000","last_became_peered":"2024-06-11T19:10:36.024538+0000","last_unstale":"2024-06-11T19:26:21.630124+0000","last_undegraded":"2024-06-11T19:26:21.630124+0000","last_fullsized":"2024-06-11T19:26:21.630124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:27:41.916846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1333,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630098+0000","last_change":"2024-06-11T19:10:36.087130+0000","last_active":"2024-06-11T19:26:21.630098+0000","last_peered":"2024-06-11T19:26:21.630098+0000","last_clean":"2024-06-11T19:26:21.630098+0000","last_became_active":"2024-06-11T19:10:36.085399+0000","last_became_peered":"2024-06-11T19:10:36.085399+0000","last_unstale":"2024-06-11T19:26:21.630098+0000","last_undegraded":"2024-06-11T19:26:21.630098+0000","last_fullsized":"2024-06-11T19:26:21.630098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:45:20.075258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389693+0000","last_change":"2024-06-11T19:10:35.977870+0000","last_active":"2024-06-11T19:26:21.389693+0000","last_peered":"2024-06-11T19:26:21.389693+0000","last_clean":"2024-06-11T19:26:21.389693+0000","last_became_active":"2024-06-11T19:10:35.976679+0000","last_became_peered":"2024-06-11T19:10:35.976679+0000","last_unstale":"2024-06-11T19:26:21.389693+0000","last_undegraded":"2024-06-11T19:26:21.389693+0000","last_fullsized":"2024-06-11T19:26:21.389693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T06:02:34.810156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389929+0000","last_change":"2024-06-11T19:10:36.004758+0000","last_active":"2024-06-11T19:26:21.389929+0000","last_peered":"2024-06-11T19:26:21.389929+0000","last_clean":"2024-06-11T19:26:21.389929+0000","last_became_active":"2024-06-11T19:10:35.985449+0000","last_became_peered":"2024-06-11T19:10:35.985449+0000","last_unstale":"2024-06-11T19:26:21.389929+0000","last_undegraded":"2024-06-11T19:26:21.389929+0000","last_fullsized":"2024-06-11T19:26:21.389929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T07:01:45.265291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630608+0000","last_change":"2024-06-11T19:10:35.967673+0000","last_active":"2024-06-11T19:26:21.630608+0000","last_peered":"2024-06-11T19:26:21.630608+0000","last_clean":"2024-06-11T19:26:21.630608+0000","last_became_active":"2024-06-11T19:10:35.962677+0000","last_became_peered":"2024-06-11T19:10:35.962677+0000","last_unstale":"2024-06-11T19:26:21.630608+0000","last_undegraded":"2024-06-11T19:26:21.630608+0000","last_fullsized":"2024-06-11T19:26:21.630608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:37:13.485742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1322,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390699+0000","last_change":"2024-06-11T19:10:36.057228+0000","last_active":"2024-06-11T19:26:21.390699+0000","last_peered":"2024-06-11T19:26:21.390699+0000","last_clean":"2024-06-11T19:26:21.390699+0000","last_became_active":"2024-06-11T19:10:36.056338+0000","last_became_peered":"2024-06-11T19:10:36.056338+0000","last_unstale":"2024-06-11T19:26:21.390699+0000","last_undegraded":"2024-06-11T19:26:21.390699+0000","last_fullsized":"2024-06-11T19:26:21.390699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:31:48.434226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1438,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630415+0000","last_change":"2024-06-11T19:09:26.673425+0000","last_active":"2024-06-11T19:26:21.630415+0000","last_peered":"2024-06-11T19:26:21.630415+0000","last_clean":"2024-06-11T19:26:21.630415+0000","last_became_active":"2024-06-11T19:09:26.673120+0000","last_became_peered":"2024-06-11T19:09:26.673120+0000","last_unstale":"2024-06-11T19:26:21.630415+0000","last_undegraded":"2024-06-11T19:26:21.630415+0000","last_fullsized":"2024-06-11T19:26:21.630415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_clean_scrub_stamp":"2024-06-11T19:09:25.660869+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:57:43.705570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630700+0000","last_change":"2024-06-11T19:09:33.725261+0000","last_active":"2024-06-11T19:26:21.630700+0000","last_peered":"2024-06-11T19:26:21.630700+0000","last_clean":"2024-06-11T19:26:21.630700+0000","last_became_active":"2024-06-11T19:09:33.724870+0000","last_became_peered":"2024-06-11T19:09:33.724870+0000","last_unstale":"2024-06-11T19:26:21.630700+0000","last_undegraded":"2024-06-11T19:26:21.630700+0000","last_fullsized":"2024-06-11T19:26:21.630700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:32:51.020718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389982+0000","last_change":"2024-06-11T19:10:36.011730+0000","last_active":"2024-06-11T19:26:21.389982+0000","last_peered":"2024-06-11T19:26:21.389982+0000","last_clean":"2024-06-11T19:26:21.389982+0000","last_became_active":"2024-06-11T19:10:35.988487+0000","last_became_peered":"2024-06-11T19:10:35.988487+0000","last_unstale":"2024-06-11T19:26:21.389982+0000","last_undegraded":"2024-06-11T19:26:21.389982+0000","last_fullsized":"2024-06-11T19:26:21.389982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:48:33.179394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1329,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390443+0000","last_change":"2024-06-11T19:09:33.729213+0000","last_active":"2024-06-11T19:26:21.390443+0000","last_peered":"2024-06-11T19:26:21.390443+0000","last_clean":"2024-06-11T19:26:21.390443+0000","last_became_active":"2024-06-11T19:09:33.728930+0000","last_became_peered":"2024-06-11T19:09:33.728930+0000","last_unstale":"2024-06-11T19:26:21.390443+0000","last_undegraded":"2024-06-11T19:26:21.390443+0000","last_fullsized":"2024-06-11T19:26:21.390443+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:26:38.808169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1322,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390789+0000","last_change":"2024-06-11T19:10:36.088019+0000","last_active":"2024-06-11T19:26:21.390789+0000","last_peered":"2024-06-11T19:26:21.390789+0000","last_clean":"2024-06-11T19:26:21.390789+0000","last_became_active":"2024-06-11T19:10:36.086670+0000","last_became_peered":"2024-06-11T19:10:36.086670+0000","last_unstale":"2024-06-11T19:26:21.390789+0000","last_undegraded":"2024-06-11T19:26:21.390789+0000","last_fullsized":"2024-06-11T19:26:21.390789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T22:06:29.653796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389520+0000","last_change":"2024-06-11T19:09:33.729275+0000","last_active":"2024-06-11T19:26:21.389520+0000","last_peered":"2024-06-11T19:26:21.389520+0000","last_clean":"2024-06-11T19:26:21.389520+0000","last_became_active":"2024-06-11T19:09:33.728948+0000","last_became_peered":"2024-06-11T19:09:33.728948+0000","last_unstale":"2024-06-11T19:26:21.389520+0000","last_undegraded":"2024-06-11T19:26:21.389520+0000","last_fullsized":"2024-06-11T19:26:21.389520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:29:28.908169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1329,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630555+0000","last_change":"2024-06-11T19:10:35.937546+0000","last_active":"2024-06-11T19:26:21.630555+0000","last_peered":"2024-06-11T19:26:21.630555+0000","last_clean":"2024-06-11T19:26:21.630555+0000","last_became_active":"2024-06-11T19:10:35.937161+0000","last_became_peered":"2024-06-11T19:10:35.937161+0000","last_unstale":"2024-06-11T19:26:21.630555+0000","last_undegraded":"2024-06-11T19:26:21.630555+0000","last_fullsized":"2024-06-11T19:26:21.630555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:12:14.668727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1330,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630527+0000","last_change":"2024-06-11T19:09:33.722503+0000","last_active":"2024-06-11T19:26:21.630527+0000","last_peered":"2024-06-11T19:26:21.630527+0000","last_clean":"2024-06-11T19:26:21.630527+0000","last_became_active":"2024-06-11T19:09:33.722180+0000","last_became_peered":"2024-06-11T19:09:33.722180+0000","last_unstale":"2024-06-11T19:26:21.630527+0000","last_undegraded":"2024-06-11T19:26:21.630527+0000","last_fullsized":"2024-06-11T19:26:21.630527+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:28:15.836943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630572+0000","last_change":"2024-06-11T19:10:36.062834+0000","last_active":"2024-06-11T19:26:21.630572+0000","last_peered":"2024-06-11T19:26:21.630572+0000","last_clean":"2024-06-11T19:26:21.630572+0000","last_became_active":"2024-06-11T19:10:36.016575+0000","last_became_peered":"2024-06-11T19:10:36.016575+0000","last_unstale":"2024-06-11T19:26:21.630572+0000","last_undegraded":"2024-06-11T19:26:21.630572+0000","last_fullsized":"2024-06-11T19:26:21.630572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:53:41.228041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1340,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630524+0000","last_change":"2024-06-11T19:09:33.729223+0000","last_active":"2024-06-11T19:26:21.630524+0000","last_peered":"2024-06-11T19:26:21.630524+0000","last_clean":"2024-06-11T19:26:21.630524+0000","last_became_active":"2024-06-11T19:09:33.728897+0000","last_became_peered":"2024-06-11T19:09:33.728897+0000","last_unstale":"2024-06-11T19:26:21.630524+0000","last_undegraded":"2024-06-11T19:26:21.630524+0000","last_fullsized":"2024-06-11T19:26:21.630524+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:49:13.439577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"20'7","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390521+0000","last_change":"2024-06-11T19:10:36.037487+0000","last_active":"2024-06-11T19:26:21.390521+0000","last_peered":"2024-06-11T19:26:21.390521+0000","last_clean":"2024-06-11T19:26:21.390521+0000","last_became_active":"2024-06-11T19:10:36.032874+0000","last_became_peered":"2024-06-11T19:10:36.032874+0000","last_unstale":"2024-06-11T19:26:21.390521+0000","last_undegraded":"2024-06-11T19:26:21.390521+0000","last_fullsized":"2024-06-11T19:26:21.390521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:58:37.764224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630669+0000","last_change":"2024-06-11T19:09:33.720915+0000","last_active":"2024-06-11T19:26:21.630669+0000","last_peered":"2024-06-11T19:26:21.630669+0000","last_clean":"2024-06-11T19:26:21.630669+0000","last_became_active":"2024-06-11T19:09:33.720599+0000","last_became_peered":"2024-06-11T19:09:33.720599+0000","last_unstale":"2024-06-11T19:26:21.630669+0000","last_undegraded":"2024-06-11T19:26:21.630669+0000","last_fullsized":"2024-06-11T19:26:21.630669+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:28:26.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1336,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630711+0000","last_change":"2024-06-11T19:10:35.982779+0000","last_active":"2024-06-11T19:26:21.630711+0000","last_peered":"2024-06-11T19:26:21.630711+0000","last_clean":"2024-06-11T19:26:21.630711+0000","last_became_active":"2024-06-11T19:10:35.976675+0000","last_became_peered":"2024-06-11T19:10:35.976675+0000","last_unstale":"2024-06-11T19:26:21.630711+0000","last_undegraded":"2024-06-11T19:26:21.630711+0000","last_fullsized":"2024-06-11T19:26:21.630711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:39:25.265311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630653+0000","last_change":"2024-06-11T19:09:33.725197+0000","last_active":"2024-06-11T19:26:21.630653+0000","last_peered":"2024-06-11T19:26:21.630653+0000","last_clean":"2024-06-11T19:26:21.630653+0000","last_became_active":"2024-06-11T19:09:33.724912+0000","last_became_peered":"2024-06-11T19:09:33.724912+0000","last_unstale":"2024-06-11T19:26:21.630653+0000","last_undegraded":"2024-06-11T19:26:21.630653+0000","last_fullsized":"2024-06-11T19:26:21.630653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:36:12.941264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389668+0000","last_change":"2024-06-11T19:10:36.090541+0000","last_active":"2024-06-11T19:26:21.389668+0000","last_peered":"2024-06-11T19:26:21.389668+0000","last_clean":"2024-06-11T19:26:21.389668+0000","last_became_active":"2024-06-11T19:10:36.071353+0000","last_became_peered":"2024-06-11T19:10:36.071353+0000","last_unstale":"2024-06-11T19:26:21.389668+0000","last_undegraded":"2024-06-11T19:26:21.389668+0000","last_fullsized":"2024-06-11T19:26:21.389668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:58:46.822805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1341,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630664+0000","last_change":"2024-06-11T19:09:33.725085+0000","last_active":"2024-06-11T19:26:21.630664+0000","last_peered":"2024-06-11T19:26:21.630664+0000","last_clean":"2024-06-11T19:26:21.630664+0000","last_became_active":"2024-06-11T19:09:33.724778+0000","last_became_peered":"2024-06-11T19:09:33.724778+0000","last_unstale":"2024-06-11T19:26:21.630664+0000","last_undegraded":"2024-06-11T19:26:21.630664+0000","last_fullsized":"2024-06-11T19:26:21.630664+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:57:46.868411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389513+0000","last_change":"2024-06-11T19:10:36.060364+0000","last_active":"2024-06-11T19:26:21.389513+0000","last_peered":"2024-06-11T19:26:21.389513+0000","last_clean":"2024-06-11T19:26:21.389513+0000","last_became_active":"2024-06-11T19:10:36.035136+0000","last_became_peered":"2024-06-11T19:10:36.035136+0000","last_unstale":"2024-06-11T19:26:21.389513+0000","last_undegraded":"2024-06-11T19:26:21.389513+0000","last_fullsized":"2024-06-11T19:26:21.389513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:30:08.506219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389506+0000","last_change":"2024-06-11T19:10:36.052997+0000","last_active":"2024-06-11T19:26:21.389506+0000","last_peered":"2024-06-11T19:26:21.389506+0000","last_clean":"2024-06-11T19:26:21.389506+0000","last_became_active":"2024-06-11T19:10:36.044757+0000","last_became_peered":"2024-06-11T19:10:36.044757+0000","last_unstale":"2024-06-11T19:26:21.389506+0000","last_undegraded":"2024-06-11T19:26:21.389506+0000","last_fullsized":"2024-06-11T19:26:21.389506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:11:16.047309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390454+0000","last_change":"2024-06-11T19:10:36.011492+0000","last_active":"2024-06-11T19:26:21.390454+0000","last_peered":"2024-06-11T19:26:21.390454+0000","last_clean":"2024-06-11T19:26:21.390454+0000","last_became_active":"2024-06-11T19:10:36.000792+0000","last_became_peered":"2024-06-11T19:10:36.000792+0000","last_unstale":"2024-06-11T19:26:21.390454+0000","last_undegraded":"2024-06-11T19:26:21.390454+0000","last_fullsized":"2024-06-11T19:26:21.390454+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:42:36.937874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630349+0000","last_change":"2024-06-11T19:10:35.994191+0000","last_active":"2024-06-11T19:26:21.630349+0000","last_peered":"2024-06-11T19:26:21.630349+0000","last_clean":"2024-06-11T19:26:21.630349+0000","last_became_active":"2024-06-11T19:10:35.988056+0000","last_became_peered":"2024-06-11T19:10:35.988056+0000","last_unstale":"2024-06-11T19:26:21.630349+0000","last_undegraded":"2024-06-11T19:26:21.630349+0000","last_fullsized":"2024-06-11T19:26:21.630349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:24:15.498060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1332,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630386+0000","last_change":"2024-06-11T19:10:36.030504+0000","last_active":"2024-06-11T19:26:21.630386+0000","last_peered":"2024-06-11T19:26:21.630386+0000","last_clean":"2024-06-11T19:26:21.630386+0000","last_became_active":"2024-06-11T19:10:36.025371+0000","last_became_peered":"2024-06-11T19:10:36.025371+0000","last_unstale":"2024-06-11T19:26:21.630386+0000","last_undegraded":"2024-06-11T19:26:21.630386+0000","last_fullsized":"2024-06-11T19:26:21.630386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:14:08.888348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390627+0000","last_change":"2024-06-11T19:10:36.091418+0000","last_active":"2024-06-11T19:26:21.390627+0000","last_peered":"2024-06-11T19:26:21.390627+0000","last_clean":"2024-06-11T19:26:21.390627+0000","last_became_active":"2024-06-11T19:10:36.090801+0000","last_became_peered":"2024-06-11T19:10:36.090801+0000","last_unstale":"2024-06-11T19:26:21.390627+0000","last_undegraded":"2024-06-11T19:26:21.390627+0000","last_fullsized":"2024-06-11T19:26:21.390627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:34:25.520333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1332,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630306+0000","last_change":"2024-06-11T19:10:36.063862+0000","last_active":"2024-06-11T19:26:21.630306+0000","last_peered":"2024-06-11T19:26:21.630306+0000","last_clean":"2024-06-11T19:26:21.630306+0000","last_became_active":"2024-06-11T19:10:36.057852+0000","last_became_peered":"2024-06-11T19:10:36.057852+0000","last_unstale":"2024-06-11T19:26:21.630306+0000","last_undegraded":"2024-06-11T19:26:21.630306+0000","last_fullsized":"2024-06-11T19:26:21.630306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:16:37.800621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1336,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389829+0000","last_change":"2024-06-11T19:10:36.056378+0000","last_active":"2024-06-11T19:26:21.389829+0000","last_peered":"2024-06-11T19:26:21.389829+0000","last_clean":"2024-06-11T19:26:21.389829+0000","last_became_active":"2024-06-11T19:10:36.042135+0000","last_became_peered":"2024-06-11T19:10:36.042135+0000","last_unstale":"2024-06-11T19:26:21.389829+0000","last_undegraded":"2024-06-11T19:26:21.389829+0000","last_fullsized":"2024-06-11T19:26:21.389829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:38:55.622181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1322,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630471+0000","last_change":"2024-06-11T19:10:36.026724+0000","last_active":"2024-06-11T19:26:21.630471+0000","last_peered":"2024-06-11T19:26:21.630471+0000","last_clean":"2024-06-11T19:26:21.630471+0000","last_became_active":"2024-06-11T19:10:36.026278+0000","last_became_peered":"2024-06-11T19:10:36.026278+0000","last_unstale":"2024-06-11T19:26:21.630471+0000","last_undegraded":"2024-06-11T19:26:21.630471+0000","last_fullsized":"2024-06-11T19:26:21.630471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:22:26.410474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1328,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630311+0000","last_change":"2024-06-11T19:10:36.034688+0000","last_active":"2024-06-11T19:26:21.630311+0000","last_peered":"2024-06-11T19:26:21.630311+0000","last_clean":"2024-06-11T19:26:21.630311+0000","last_became_active":"2024-06-11T19:10:36.028329+0000","last_became_peered":"2024-06-11T19:10:36.028329+0000","last_unstale":"2024-06-11T19:26:21.630311+0000","last_undegraded":"2024-06-11T19:26:21.630311+0000","last_fullsized":"2024-06-11T19:26:21.630311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T06:59:08.595933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630349+0000","last_change":"2024-06-11T19:10:36.085537+0000","last_active":"2024-06-11T19:26:21.630349+0000","last_peered":"2024-06-11T19:26:21.630349+0000","last_clean":"2024-06-11T19:26:21.630349+0000","last_became_active":"2024-06-11T19:10:36.082897+0000","last_became_peered":"2024-06-11T19:10:36.082897+0000","last_unstale":"2024-06-11T19:26:21.630349+0000","last_undegraded":"2024-06-11T19:26:21.630349+0000","last_fullsized":"2024-06-11T19:26:21.630349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:29:42.550377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390495+0000","last_change":"2024-06-11T19:10:36.017667+0000","last_active":"2024-06-11T19:26:21.390495+0000","last_peered":"2024-06-11T19:26:21.390495+0000","last_clean":"2024-06-11T19:26:21.390495+0000","last_became_active":"2024-06-11T19:10:36.002730+0000","last_became_peered":"2024-06-11T19:10:36.002730+0000","last_unstale":"2024-06-11T19:26:21.390495+0000","last_undegraded":"2024-06-11T19:26:21.390495+0000","last_fullsized":"2024-06-11T19:26:21.390495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:34:23.834164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1328,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630373+0000","last_change":"2024-06-11T19:10:36.076140+0000","last_active":"2024-06-11T19:26:21.630373+0000","last_peered":"2024-06-11T19:26:21.630373+0000","last_clean":"2024-06-11T19:26:21.630373+0000","last_became_active":"2024-06-11T19:10:36.075888+0000","last_became_peered":"2024-06-11T19:10:36.075888+0000","last_unstale":"2024-06-11T19:26:21.630373+0000","last_undegraded":"2024-06-11T19:26:21.630373+0000","last_fullsized":"2024-06-11T19:26:21.630373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:07:12.839020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607763,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77264,"kb_used_data":9968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294576,"statfs":{"total":96636764160,"available":96557645824,"internally_reserved":0,"allocated":10207232,"data_stored":5691113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81956,"kb_used_data":10488,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289884,"statfs":{"total":96636764160,"available":96552841216,"internally_reserved":0,"allocated":10739712,"data_stored":6220041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640467,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86100,"kb_used_data":10484,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94285740,"statfs":{"total":96636764160,"available":96548597760,"internally_reserved":0,"allocated":10735616,"data_stored":6220036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640468,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73112,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298728,"statfs":{"total":96636764160,"available":96561897472,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640467,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67532,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94304308,"statfs":{"total":96636764160,"available":96567611392,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640469,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85604,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286236,"statfs":{"total":96636764160,"available":96549105664,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-11T19:26:48.465 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-06-11T19:26:48.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-11T19:26:48.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.563+0000 7f0dc5f40640 10 monclient: get_monmap_and_config 2024-06-11T19:26:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.563+0000 7f0dc5f40640 10 monclient: build_initial_monmap 2024-06-11T19:26:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.563+0000 7f0dc5f40640 10 monclient: monmap: 2024-06-11T19:26:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:48.564852+0000 2024-06-11T19:26:48.563 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:48.564852+0000 2024-06-11T19:26:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:48.564 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:48.565 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.564+0000 7f0dc5f40640 1 Processor -- start 2024-06-11T19:26:48.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.564+0000 7f0dc5f40640 1 -- start start 2024-06-11T19:26:48.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.564+0000 7f0dc5f40640 10 monclient: init 2024-06-11T19:26:48.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:48.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 10 monclient: _add_conns ranks=[5,9,1,7,3,10,2,4,8,6,0] 2024-06-11T19:26:48.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0dc00a4920 0x7f0dc00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 10 monclient(hunting): picked mon.noname-h con 0x7f0dc00a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:48.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0dc00a5240 0x7f0dc00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 10 monclient(hunting): picked mon.noname-a con 0x7f0dc00a5240 addr v2:172.21.15.98:3300/0 2024-06-11T19:26:48.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f0dc00a2530 addr v1:172.21.15.117:6789/0 2024-06-11T19:26:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dc00a7140 con 0x7f0dc00a2530 2024-06-11T19:26:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0dc00a2b50 con 0x7f0dc00a5240 2024-06-11T19:26:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f0dc00a2e30 con 0x7f0dc00a4920 2024-06-11T19:26:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dc5f40640 10 monclient(hunting): authenticate will time out at 1818.372228s 2024-06-11T19:26:48.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0dc00a5240 0x7f0dc00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0dc00a5240 0x7f0dc00a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41870/0 (socket says 172.21.15.98:41870) 2024-06-11T19:26:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 1 -- 172.21.15.98:0/132685040 learned_addr learned my addr 172.21.15.98:0/132685040 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 10 monclient(hunting): get_auth_request con 0x7f0dc00a5240 auth_method 0 2024-06-11T19:26:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbf7fe640 1 --2- 172.21.15.98:0/132685040 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0dc00a4920 0x7f0dc00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbf7fe640 10 monclient(hunting): get_auth_request con 0x7f0dc00a4920 auth_method 0 2024-06-11T19:26:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbf7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 10 monclient(hunting): handle_auth_done global_id 66456 payload 274 2024-06-11T19:26:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 1 -- 172.21.15.98:0/132685040 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0dc00a4920 msgr2=0x7f0dc00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 1 --2- 172.21.15.98:0/132685040 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0dc00a4920 0x7f0dc00a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 1 -- 172.21.15.98:0/132685040 >> v1:172.21.15.117:6789/0 conn(0x7f0dc00a2530 legacy=0x7f0dc00a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 10 monclient: _finish_hunting 0 2024-06-11T19:26:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 1 monclient: found mon.noname-a 2024-06-11T19:26:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:26:48.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.565+0000 7f0dbeffd640 1 -- 172.21.15.98:0/132685040 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc00a3110 con 0x7f0dc00a5240 2024-06-11T19:26:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbeffd640 1 --2- 172.21.15.98:0/132685040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc00a5240 0x7f0dc00a59a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0db4009e50 tx=0x7f0db402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fa31998bb638de server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 1 -- 172.21.15.98:0/132685040 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0db402cbc0 con 0x7f0dc00a5240 2024-06-11T19:26:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:26:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 10 monclient: dump: 2024-06-11T19:26:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.579 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:48.580 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:48.581 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 10 monclient: _finish_auth 0 2024-06-11T19:26:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:26:48.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 1 -- 172.21.15.98:0/132685040 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db402cd80 con 0x7f0dc00a5240 2024-06-11T19:26:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dc5f40640 5 monclient: authenticate success, global_id 66456 2024-06-11T19:26:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dc5f40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dc5f40640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 1 -- 172.21.15.98:0/132685040 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0db4034860 con 0x7f0dc00a5240 2024-06-11T19:26:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:48.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dc5f40640 4 set_mon_vals no callback set 2024-06-11T19:26:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.566+0000 7f0dbe7fc640 10 monclient: dump: 2024-06-11T19:26:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.583 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:48.584 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:48.584 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:48.584 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:48.584 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:48.584 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:48.584 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:48.584 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:48.584 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:48.585 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 10 monclient: shutdown 2024-06-11T19:26:48.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:48.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 1 -- 172.21.15.98:0/132685040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc00a5240 msgr2=0x7f0dc00a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 1 --2- 172.21.15.98:0/132685040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc00a5240 0x7f0dc00a59a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0db4009e50 tx=0x7f0db402bdb0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 1 -- 172.21.15.98:0/132685040 shutdown_connections 2024-06-11T19:26:48.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 1 --2- 172.21.15.98:0/132685040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0dc00a5240 0x7f0dc00a59a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 1 --2- 172.21.15.98:0/132685040 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0dc00a4920 0x7f0dc00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 1 -- 172.21.15.98:0/132685040 >> 172.21.15.98:0/132685040 conn(0x7f0dc00ab850 msgr2=0x7f0dc00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:48.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 1 -- 172.21.15.98:0/132685040 shutdown_connections 2024-06-11T19:26:48.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 1 -- 172.21.15.98:0/132685040 wait complete. 2024-06-11T19:26:48.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 10 monclient: build_initial_monmap 2024-06-11T19:26:48.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 10 monclient: monmap: 2024-06-11T19:26:48.586 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:48.586 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:48.586 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:48.588 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:48.588 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:48.588 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:48.588 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:48.588 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 1 librados: starting msgr at 2024-06-11T19:26:48.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 1 librados: starting objecter 2024-06-11T19:26:48.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.567+0000 7f0dc5f40640 1 Processor -- start 2024-06-11T19:26:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 1 -- start start 2024-06-11T19:26:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 1 librados: setting wanted keys 2024-06-11T19:26:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 1 librados: calling monclient init 2024-06-11T19:26:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 10 monclient: init 2024-06-11T19:26:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 10 monclient: _add_conns ranks=[5,2,3,7,0,6,8,4,1] 2024-06-11T19:26:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0dc00a2530 0x7f0dc01cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 10 monclient(hunting): picked mon.noname-h con 0x7f0dc00a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc00a4920 0x7f0dc01ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 10 monclient(hunting): picked mon.noname-b con 0x7f0dc00a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0dc00a5240 0x7f0dc0124ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 10 monclient(hunting): picked mon.noname-g con 0x7f0dc00a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc00a6da0 con 0x7f0dc00a4920 2024-06-11T19:26:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc0138e50 con 0x7f0dc00a5240 2024-06-11T19:26:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f0dc00a65f0 con 0x7f0dc00a2530 2024-06-11T19:26:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dc5f40640 10 monclient(hunting): authenticate will time out at 1818.375646s 2024-06-11T19:26:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbeffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc00a4920 0x7f0dc01ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbeffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc00a4920 0x7f0dc01ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:43132/0 (socket says 172.21.15.98:43132) 2024-06-11T19:26:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbeffd640 1 -- 172.21.15.98:0/1763484295 learned_addr learned my addr 172.21.15.98:0/1763484295 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbeffd640 10 monclient(hunting): get_auth_request con 0x7f0dc00a4920 auth_method 0 2024-06-11T19:26:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbeffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbf7fe640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0dc00a2530 0x7f0dc01cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbf7fe640 10 monclient(hunting): get_auth_request con 0x7f0dc00a2530 auth_method 0 2024-06-11T19:26:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbffff640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0dc00a5240 0x7f0dc0124ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbf7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.568+0000 7f0dbeffd640 10 monclient(hunting): handle_auth_done global_id 85412 payload 824 2024-06-11T19:26:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbeffd640 1 -- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0dc00a2530 msgr2=0x7f0dc01cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbeffd640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0dc00a2530 0x7f0dc01cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbeffd640 1 -- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0dc00a5240 msgr2=0x7f0dc0124ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbeffd640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0dc00a5240 0x7f0dc0124ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbeffd640 10 monclient: _finish_hunting 0 2024-06-11T19:26:48.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbeffd640 1 monclient: found mon.noname-b 2024-06-11T19:26:48.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:48.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:26:48.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbeffd640 1 -- 172.21.15.98:0/1763484295 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc0126fb0 con 0x7f0dc00a4920 2024-06-11T19:26:48.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbffff640 10 monclient: get_auth_request con 0x7f0dc00a5240 auth_method 0 2024-06-11T19:26:48.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbffff640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0dc00a5240 0x7f0dc0124ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:26:48.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbeffd640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc00a4920 0x7f0dc01ce130 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0db4009f70 tx=0x7f0db4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:48.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 10 client.?.objecter ms_handle_connect 0x7f0dc00a4920 2024-06-11T19:26:48.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:48.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 1 -- 172.21.15.98:0/1763484295 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0db4043020 con 0x7f0dc00a4920 2024-06-11T19:26:48.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:48.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:26:48.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 10 monclient: dump: 2024-06-11T19:26:48.595 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:48.595 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:48.597 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 10 monclient: _finish_auth 0 2024-06-11T19:26:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:26:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 1 -- 172.21.15.98:0/1763484295 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db40046c0 con 0x7f0dc00a4920 2024-06-11T19:26:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dc5f40640 5 monclient: authenticate success, global_id 85412 2024-06-11T19:26:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dc5f40640 10 monclient: _renew_subs 2024-06-11T19:26:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dc5f40640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dc5f40640 1 -- 172.21.15.98:0/1763484295 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc0127670 con 0x7f0dc00a4920 2024-06-11T19:26:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 1 -- 172.21.15.98:0/1763484295 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0db4009510 con 0x7f0dc00a4920 2024-06-11T19:26:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:26:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0da7fff640 10 monclient: dump: 2024-06-11T19:26:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:48.601 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dc5f40640 10 client.85412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dc5f40640 10 monclient: _renew_subs 2024-06-11T19:26:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dbdffb640 4 set_mon_vals no callback set 2024-06-11T19:26:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dc5f40640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dc5f40640 1 -- 172.21.15.98:0/1763484295 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc01387f0 con 0x7f0dc00a4920 2024-06-11T19:26:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.569+0000 7f0dc5f40640 1 librados: init done 2024-06-11T19:26:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0da7fff640 1 -- 172.21.15.98:0/1763484295 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f0db4046020 con 0x7f0dc00a4920 2024-06-11T19:26:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0dc5f40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0dc5f40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0dc5f40640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0dc5f40640 1 -- 172.21.15.98:0/1763484295 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc00aab30 con 0x7f0dc00a4920 2024-06-11T19:26:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0da7fff640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0dac03fdb0 0x7f0dac042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0da7fff640 1 -- 172.21.15.98:0/1763484295 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f0db403a040 con 0x7f0dc00a4920 2024-06-11T19:26:48.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0da7fff640 10 client.85412.objecter ms_dispatch 0x7f0dc0000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0da7fff640 3 client.85412.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0da7fff640 3 client.85412.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0da7fff640 20 client.85412.objecter dump_active .. 0 homeless 2024-06-11T19:26:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0dbf7fe640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0dac03fdb0 0x7f0dac042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.570+0000 7f0dbf7fe640 10 monclient: get_auth_request con 0x7f0dac03fdb0 auth_method 0 2024-06-11T19:26:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.571+0000 7f0da7fff640 10 client.85412.objecter ms_handle_connect 0x7f0dac03fdb0 2024-06-11T19:26:48.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.571+0000 7f0dbf7fe640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0dac03fdb0 0x7f0dac042270 secure :-1 s=READY pgs=4333 cs=0 l=1 rev1=1 crypto rx=0x7f0db0006fd0 tx=0x7f0db0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.575+0000 7f0da7fff640 1 -- 172.21.15.98:0/1763484295 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0dc00aab30 con 0x7f0dc00a4920 2024-06-11T19:26:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.575+0000 7f0da7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.575+0000 7f0da7fff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:48.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.753+0000 7f0dc5f40640 1 -- 172.21.15.98:0/1763484295 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0dc0128560 con 0x7f0dac03fdb0 2024-06-11T19:26:48.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.756+0000 7f0da7fff640 1 -- 172.21.15.98:0/1763484295 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+113897 (secure 0 0 0) 0x7f0db000b200 con 0x7f0dac03fdb0 2024-06-11T19:26:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:26:48.756 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-06-11T19:26:48.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.757+0000 7f0dc5f40640 10 librados: watch_flush enter 2024-06-11T19:26:48.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 10 librados: watch_flush exit 2024-06-11T19:26:48.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 20 client.85412.objecter shutdown clearing up homeless session... 2024-06-11T19:26:48.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 10 client.85412.objecter successfully canceled tick 2024-06-11T19:26:48.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 -- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0dac03fdb0 msgr2=0x7f0dac042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0dac03fdb0 0x7f0dac042270 secure :-1 s=READY pgs=4333 cs=0 l=1 rev1=1 crypto rx=0x7f0db0006fd0 tx=0x7f0db0008040 comp rx=0 tx=0).stop 2024-06-11T19:26:48.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 10 monclient: shutdown 2024-06-11T19:26:48.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:48.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 -- 172.21.15.98:0/1763484295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc00a4920 msgr2=0x7f0dc01ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc00a4920 0x7f0dc01ce130 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0db4009f70 tx=0x7f0db4004a40 comp rx=0 tx=0).stop 2024-06-11T19:26:48.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 -- 172.21.15.98:0/1763484295 shutdown_connections 2024-06-11T19:26:48.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f0dac03fdb0 0x7f0dac042270 unknown :-1 s=CLOSED pgs=4333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0dc00a5240 0x7f0dc0124ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0dc00a4920 0x7f0dc01ce130 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 --2- 172.21.15.98:0/1763484295 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f0dc00a2530 0x7f0dc01cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 -- 172.21.15.98:0/1763484295 >> 172.21.15.98:0/1763484295 conn(0x7f0dc00ab850 msgr2=0x7f0dc009c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:48.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 -- 172.21.15.98:0/1763484295 shutdown_connections 2024-06-11T19:26:48.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 -- 172.21.15.98:0/1763484295 wait complete. 2024-06-11T19:26:48.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.758+0000 7f0dc5f40640 1 librados: shutdown 2024-06-11T19:26:48.768 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1200,"stamp":"2024-06-11T19:26:47.631585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":108,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":471568,"kb_used_data":60832,"kb_used_omap":9,"kb_used_meta":410486,"kb_avail":565759472,"statfs":{"total":579820584960,"available":579337699328,"internally_reserved":0,"allocated":62291968,"data_stored":35204514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":420338336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"7.17","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630466+0000","last_change":"2024-06-11T19:10:36.026367+0000","last_active":"2024-06-11T19:26:21.630466+0000","last_peered":"2024-06-11T19:26:21.630466+0000","last_clean":"2024-06-11T19:26:21.630466+0000","last_became_active":"2024-06-11T19:10:36.005579+0000","last_became_peered":"2024-06-11T19:10:36.005579+0000","last_unstale":"2024-06-11T19:26:21.630466+0000","last_undegraded":"2024-06-11T19:26:21.630466+0000","last_fullsized":"2024-06-11T19:26:21.630466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T05:31:34.753919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630581+0000","last_change":"2024-06-11T19:10:35.961885+0000","last_active":"2024-06-11T19:26:21.630581+0000","last_peered":"2024-06-11T19:26:21.630581+0000","last_clean":"2024-06-11T19:26:21.630581+0000","last_became_active":"2024-06-11T19:10:35.960336+0000","last_became_peered":"2024-06-11T19:10:35.960336+0000","last_unstale":"2024-06-11T19:26:21.630581+0000","last_undegraded":"2024-06-11T19:26:21.630581+0000","last_fullsized":"2024-06-11T19:26:21.630581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:42:55.368948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390705+0000","last_change":"2024-06-11T19:10:36.098514+0000","last_active":"2024-06-11T19:26:21.390705+0000","last_peered":"2024-06-11T19:26:21.390705+0000","last_clean":"2024-06-11T19:26:21.390705+0000","last_became_active":"2024-06-11T19:10:36.087715+0000","last_became_peered":"2024-06-11T19:10:36.087715+0000","last_unstale":"2024-06-11T19:26:21.390705+0000","last_undegraded":"2024-06-11T19:26:21.390705+0000","last_fullsized":"2024-06-11T19:26:21.390705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:25:31.960579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1332,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630521+0000","last_change":"2024-06-11T19:10:36.045515+0000","last_active":"2024-06-11T19:26:21.630521+0000","last_peered":"2024-06-11T19:26:21.630521+0000","last_clean":"2024-06-11T19:26:21.630521+0000","last_became_active":"2024-06-11T19:10:36.036974+0000","last_became_peered":"2024-06-11T19:10:36.036974+0000","last_unstale":"2024-06-11T19:26:21.630521+0000","last_undegraded":"2024-06-11T19:26:21.630521+0000","last_fullsized":"2024-06-11T19:26:21.630521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:35:56.827003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389826+0000","last_change":"2024-06-11T19:10:36.084957+0000","last_active":"2024-06-11T19:26:21.389826+0000","last_peered":"2024-06-11T19:26:21.389826+0000","last_clean":"2024-06-11T19:26:21.389826+0000","last_became_active":"2024-06-11T19:10:36.065872+0000","last_became_peered":"2024-06-11T19:10:36.065872+0000","last_unstale":"2024-06-11T19:26:21.389826+0000","last_undegraded":"2024-06-11T19:26:21.389826+0000","last_fullsized":"2024-06-11T19:26:21.389826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T05:33:25.348773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1329,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630376+0000","last_change":"2024-06-11T19:10:36.077422+0000","last_active":"2024-06-11T19:26:21.630376+0000","last_peered":"2024-06-11T19:26:21.630376+0000","last_clean":"2024-06-11T19:26:21.630376+0000","last_became_active":"2024-06-11T19:10:36.076424+0000","last_became_peered":"2024-06-11T19:10:36.076424+0000","last_unstale":"2024-06-11T19:26:21.630376+0000","last_undegraded":"2024-06-11T19:26:21.630376+0000","last_fullsized":"2024-06-11T19:26:21.630376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:52:42.583531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1333,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630124+0000","last_change":"2024-06-11T19:10:36.037164+0000","last_active":"2024-06-11T19:26:21.630124+0000","last_peered":"2024-06-11T19:26:21.630124+0000","last_clean":"2024-06-11T19:26:21.630124+0000","last_became_active":"2024-06-11T19:10:36.024538+0000","last_became_peered":"2024-06-11T19:10:36.024538+0000","last_unstale":"2024-06-11T19:26:21.630124+0000","last_undegraded":"2024-06-11T19:26:21.630124+0000","last_fullsized":"2024-06-11T19:26:21.630124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:27:41.916846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1333,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630098+0000","last_change":"2024-06-11T19:10:36.087130+0000","last_active":"2024-06-11T19:26:21.630098+0000","last_peered":"2024-06-11T19:26:21.630098+0000","last_clean":"2024-06-11T19:26:21.630098+0000","last_became_active":"2024-06-11T19:10:36.085399+0000","last_became_peered":"2024-06-11T19:10:36.085399+0000","last_unstale":"2024-06-11T19:26:21.630098+0000","last_undegraded":"2024-06-11T19:26:21.630098+0000","last_fullsized":"2024-06-11T19:26:21.630098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:45:20.075258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389693+0000","last_change":"2024-06-11T19:10:35.977870+0000","last_active":"2024-06-11T19:26:21.389693+0000","last_peered":"2024-06-11T19:26:21.389693+0000","last_clean":"2024-06-11T19:26:21.389693+0000","last_became_active":"2024-06-11T19:10:35.976679+0000","last_became_peered":"2024-06-11T19:10:35.976679+0000","last_unstale":"2024-06-11T19:26:21.389693+0000","last_undegraded":"2024-06-11T19:26:21.389693+0000","last_fullsized":"2024-06-11T19:26:21.389693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T06:02:34.810156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389929+0000","last_change":"2024-06-11T19:10:36.004758+0000","last_active":"2024-06-11T19:26:21.389929+0000","last_peered":"2024-06-11T19:26:21.389929+0000","last_clean":"2024-06-11T19:26:21.389929+0000","last_became_active":"2024-06-11T19:10:35.985449+0000","last_became_peered":"2024-06-11T19:10:35.985449+0000","last_unstale":"2024-06-11T19:26:21.389929+0000","last_undegraded":"2024-06-11T19:26:21.389929+0000","last_fullsized":"2024-06-11T19:26:21.389929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T07:01:45.265291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630608+0000","last_change":"2024-06-11T19:10:35.967673+0000","last_active":"2024-06-11T19:26:21.630608+0000","last_peered":"2024-06-11T19:26:21.630608+0000","last_clean":"2024-06-11T19:26:21.630608+0000","last_became_active":"2024-06-11T19:10:35.962677+0000","last_became_peered":"2024-06-11T19:10:35.962677+0000","last_unstale":"2024-06-11T19:26:21.630608+0000","last_undegraded":"2024-06-11T19:26:21.630608+0000","last_fullsized":"2024-06-11T19:26:21.630608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:37:13.485742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1322,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390699+0000","last_change":"2024-06-11T19:10:36.057228+0000","last_active":"2024-06-11T19:26:21.390699+0000","last_peered":"2024-06-11T19:26:21.390699+0000","last_clean":"2024-06-11T19:26:21.390699+0000","last_became_active":"2024-06-11T19:10:36.056338+0000","last_became_peered":"2024-06-11T19:10:36.056338+0000","last_unstale":"2024-06-11T19:26:21.390699+0000","last_undegraded":"2024-06-11T19:26:21.390699+0000","last_fullsized":"2024-06-11T19:26:21.390699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:31:48.434226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1438,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630415+0000","last_change":"2024-06-11T19:09:26.673425+0000","last_active":"2024-06-11T19:26:21.630415+0000","last_peered":"2024-06-11T19:26:21.630415+0000","last_clean":"2024-06-11T19:26:21.630415+0000","last_became_active":"2024-06-11T19:09:26.673120+0000","last_became_peered":"2024-06-11T19:09:26.673120+0000","last_unstale":"2024-06-11T19:26:21.630415+0000","last_undegraded":"2024-06-11T19:26:21.630415+0000","last_fullsized":"2024-06-11T19:26:21.630415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_clean_scrub_stamp":"2024-06-11T19:09:25.660869+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:57:43.705570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630700+0000","last_change":"2024-06-11T19:09:33.725261+0000","last_active":"2024-06-11T19:26:21.630700+0000","last_peered":"2024-06-11T19:26:21.630700+0000","last_clean":"2024-06-11T19:26:21.630700+0000","last_became_active":"2024-06-11T19:09:33.724870+0000","last_became_peered":"2024-06-11T19:09:33.724870+0000","last_unstale":"2024-06-11T19:26:21.630700+0000","last_undegraded":"2024-06-11T19:26:21.630700+0000","last_fullsized":"2024-06-11T19:26:21.630700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:32:51.020718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389982+0000","last_change":"2024-06-11T19:10:36.011730+0000","last_active":"2024-06-11T19:26:21.389982+0000","last_peered":"2024-06-11T19:26:21.389982+0000","last_clean":"2024-06-11T19:26:21.389982+0000","last_became_active":"2024-06-11T19:10:35.988487+0000","last_became_peered":"2024-06-11T19:10:35.988487+0000","last_unstale":"2024-06-11T19:26:21.389982+0000","last_undegraded":"2024-06-11T19:26:21.389982+0000","last_fullsized":"2024-06-11T19:26:21.389982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:48:33.179394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1329,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390443+0000","last_change":"2024-06-11T19:09:33.729213+0000","last_active":"2024-06-11T19:26:21.390443+0000","last_peered":"2024-06-11T19:26:21.390443+0000","last_clean":"2024-06-11T19:26:21.390443+0000","last_became_active":"2024-06-11T19:09:33.728930+0000","last_became_peered":"2024-06-11T19:09:33.728930+0000","last_unstale":"2024-06-11T19:26:21.390443+0000","last_undegraded":"2024-06-11T19:26:21.390443+0000","last_fullsized":"2024-06-11T19:26:21.390443+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:26:38.808169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1322,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390789+0000","last_change":"2024-06-11T19:10:36.088019+0000","last_active":"2024-06-11T19:26:21.390789+0000","last_peered":"2024-06-11T19:26:21.390789+0000","last_clean":"2024-06-11T19:26:21.390789+0000","last_became_active":"2024-06-11T19:10:36.086670+0000","last_became_peered":"2024-06-11T19:10:36.086670+0000","last_unstale":"2024-06-11T19:26:21.390789+0000","last_undegraded":"2024-06-11T19:26:21.390789+0000","last_fullsized":"2024-06-11T19:26:21.390789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T22:06:29.653796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389520+0000","last_change":"2024-06-11T19:09:33.729275+0000","last_active":"2024-06-11T19:26:21.389520+0000","last_peered":"2024-06-11T19:26:21.389520+0000","last_clean":"2024-06-11T19:26:21.389520+0000","last_became_active":"2024-06-11T19:09:33.728948+0000","last_became_peered":"2024-06-11T19:09:33.728948+0000","last_unstale":"2024-06-11T19:26:21.389520+0000","last_undegraded":"2024-06-11T19:26:21.389520+0000","last_fullsized":"2024-06-11T19:26:21.389520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:29:28.908169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1329,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630555+0000","last_change":"2024-06-11T19:10:35.937546+0000","last_active":"2024-06-11T19:26:21.630555+0000","last_peered":"2024-06-11T19:26:21.630555+0000","last_clean":"2024-06-11T19:26:21.630555+0000","last_became_active":"2024-06-11T19:10:35.937161+0000","last_became_peered":"2024-06-11T19:10:35.937161+0000","last_unstale":"2024-06-11T19:26:21.630555+0000","last_undegraded":"2024-06-11T19:26:21.630555+0000","last_fullsized":"2024-06-11T19:26:21.630555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:12:14.668727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1330,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630527+0000","last_change":"2024-06-11T19:09:33.722503+0000","last_active":"2024-06-11T19:26:21.630527+0000","last_peered":"2024-06-11T19:26:21.630527+0000","last_clean":"2024-06-11T19:26:21.630527+0000","last_became_active":"2024-06-11T19:09:33.722180+0000","last_became_peered":"2024-06-11T19:09:33.722180+0000","last_unstale":"2024-06-11T19:26:21.630527+0000","last_undegraded":"2024-06-11T19:26:21.630527+0000","last_fullsized":"2024-06-11T19:26:21.630527+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:28:15.836943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630572+0000","last_change":"2024-06-11T19:10:36.062834+0000","last_active":"2024-06-11T19:26:21.630572+0000","last_peered":"2024-06-11T19:26:21.630572+0000","last_clean":"2024-06-11T19:26:21.630572+0000","last_became_active":"2024-06-11T19:10:36.016575+0000","last_became_peered":"2024-06-11T19:10:36.016575+0000","last_unstale":"2024-06-11T19:26:21.630572+0000","last_undegraded":"2024-06-11T19:26:21.630572+0000","last_fullsized":"2024-06-11T19:26:21.630572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:53:41.228041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1340,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630524+0000","last_change":"2024-06-11T19:09:33.729223+0000","last_active":"2024-06-11T19:26:21.630524+0000","last_peered":"2024-06-11T19:26:21.630524+0000","last_clean":"2024-06-11T19:26:21.630524+0000","last_became_active":"2024-06-11T19:09:33.728897+0000","last_became_peered":"2024-06-11T19:09:33.728897+0000","last_unstale":"2024-06-11T19:26:21.630524+0000","last_undegraded":"2024-06-11T19:26:21.630524+0000","last_fullsized":"2024-06-11T19:26:21.630524+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:49:13.439577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"20'7","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390521+0000","last_change":"2024-06-11T19:10:36.037487+0000","last_active":"2024-06-11T19:26:21.390521+0000","last_peered":"2024-06-11T19:26:21.390521+0000","last_clean":"2024-06-11T19:26:21.390521+0000","last_became_active":"2024-06-11T19:10:36.032874+0000","last_became_peered":"2024-06-11T19:10:36.032874+0000","last_unstale":"2024-06-11T19:26:21.390521+0000","last_undegraded":"2024-06-11T19:26:21.390521+0000","last_fullsized":"2024-06-11T19:26:21.390521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:58:37.764224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630669+0000","last_change":"2024-06-11T19:09:33.720915+0000","last_active":"2024-06-11T19:26:21.630669+0000","last_peered":"2024-06-11T19:26:21.630669+0000","last_clean":"2024-06-11T19:26:21.630669+0000","last_became_active":"2024-06-11T19:09:33.720599+0000","last_became_peered":"2024-06-11T19:09:33.720599+0000","last_unstale":"2024-06-11T19:26:21.630669+0000","last_undegraded":"2024-06-11T19:26:21.630669+0000","last_fullsized":"2024-06-11T19:26:21.630669+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:28:26.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1336,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630711+0000","last_change":"2024-06-11T19:10:35.982779+0000","last_active":"2024-06-11T19:26:21.630711+0000","last_peered":"2024-06-11T19:26:21.630711+0000","last_clean":"2024-06-11T19:26:21.630711+0000","last_became_active":"2024-06-11T19:10:35.976675+0000","last_became_peered":"2024-06-11T19:10:35.976675+0000","last_unstale":"2024-06-11T19:26:21.630711+0000","last_undegraded":"2024-06-11T19:26:21.630711+0000","last_fullsized":"2024-06-11T19:26:21.630711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:39:25.265311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630653+0000","last_change":"2024-06-11T19:09:33.725197+0000","last_active":"2024-06-11T19:26:21.630653+0000","last_peered":"2024-06-11T19:26:21.630653+0000","last_clean":"2024-06-11T19:26:21.630653+0000","last_became_active":"2024-06-11T19:09:33.724912+0000","last_became_peered":"2024-06-11T19:09:33.724912+0000","last_unstale":"2024-06-11T19:26:21.630653+0000","last_undegraded":"2024-06-11T19:26:21.630653+0000","last_fullsized":"2024-06-11T19:26:21.630653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:36:12.941264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389668+0000","last_change":"2024-06-11T19:10:36.090541+0000","last_active":"2024-06-11T19:26:21.389668+0000","last_peered":"2024-06-11T19:26:21.389668+0000","last_clean":"2024-06-11T19:26:21.389668+0000","last_became_active":"2024-06-11T19:10:36.071353+0000","last_became_peered":"2024-06-11T19:10:36.071353+0000","last_unstale":"2024-06-11T19:26:21.389668+0000","last_undegraded":"2024-06-11T19:26:21.389668+0000","last_fullsized":"2024-06-11T19:26:21.389668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:58:46.822805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1341,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630664+0000","last_change":"2024-06-11T19:09:33.725085+0000","last_active":"2024-06-11T19:26:21.630664+0000","last_peered":"2024-06-11T19:26:21.630664+0000","last_clean":"2024-06-11T19:26:21.630664+0000","last_became_active":"2024-06-11T19:09:33.724778+0000","last_became_peered":"2024-06-11T19:09:33.724778+0000","last_unstale":"2024-06-11T19:26:21.630664+0000","last_undegraded":"2024-06-11T19:26:21.630664+0000","last_fullsized":"2024-06-11T19:26:21.630664+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:57:46.868411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389513+0000","last_change":"2024-06-11T19:10:36.060364+0000","last_active":"2024-06-11T19:26:21.389513+0000","last_peered":"2024-06-11T19:26:21.389513+0000","last_clean":"2024-06-11T19:26:21.389513+0000","last_became_active":"2024-06-11T19:10:36.035136+0000","last_became_peered":"2024-06-11T19:10:36.035136+0000","last_unstale":"2024-06-11T19:26:21.389513+0000","last_undegraded":"2024-06-11T19:26:21.389513+0000","last_fullsized":"2024-06-11T19:26:21.389513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:30:08.506219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389506+0000","last_change":"2024-06-11T19:10:36.052997+0000","last_active":"2024-06-11T19:26:21.389506+0000","last_peered":"2024-06-11T19:26:21.389506+0000","last_clean":"2024-06-11T19:26:21.389506+0000","last_became_active":"2024-06-11T19:10:36.044757+0000","last_became_peered":"2024-06-11T19:10:36.044757+0000","last_unstale":"2024-06-11T19:26:21.389506+0000","last_undegraded":"2024-06-11T19:26:21.389506+0000","last_fullsized":"2024-06-11T19:26:21.389506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:11:16.047309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390454+0000","last_change":"2024-06-11T19:10:36.011492+0000","last_active":"2024-06-11T19:26:21.390454+0000","last_peered":"2024-06-11T19:26:21.390454+0000","last_clean":"2024-06-11T19:26:21.390454+0000","last_became_active":"2024-06-11T19:10:36.000792+0000","last_became_peered":"2024-06-11T19:10:36.000792+0000","last_unstale":"2024-06-11T19:26:21.390454+0000","last_undegraded":"2024-06-11T19:26:21.390454+0000","last_fullsized":"2024-06-11T19:26:21.390454+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:42:36.937874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630349+0000","last_change":"2024-06-11T19:10:35.994191+0000","last_active":"2024-06-11T19:26:21.630349+0000","last_peered":"2024-06-11T19:26:21.630349+0000","last_clean":"2024-06-11T19:26:21.630349+0000","last_became_active":"2024-06-11T19:10:35.988056+0000","last_became_peered":"2024-06-11T19:10:35.988056+0000","last_unstale":"2024-06-11T19:26:21.630349+0000","last_undegraded":"2024-06-11T19:26:21.630349+0000","last_fullsized":"2024-06-11T19:26:21.630349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:24:15.498060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1332,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630386+0000","last_change":"2024-06-11T19:10:36.030504+0000","last_active":"2024-06-11T19:26:21.630386+0000","last_peered":"2024-06-11T19:26:21.630386+0000","last_clean":"2024-06-11T19:26:21.630386+0000","last_became_active":"2024-06-11T19:10:36.025371+0000","last_became_peered":"2024-06-11T19:10:36.025371+0000","last_unstale":"2024-06-11T19:26:21.630386+0000","last_undegraded":"2024-06-11T19:26:21.630386+0000","last_fullsized":"2024-06-11T19:26:21.630386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:14:08.888348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390627+0000","last_change":"2024-06-11T19:10:36.091418+0000","last_active":"2024-06-11T19:26:21.390627+0000","last_peered":"2024-06-11T19:26:21.390627+0000","last_clean":"2024-06-11T19:26:21.390627+0000","last_became_active":"2024-06-11T19:10:36.090801+0000","last_became_peered":"2024-06-11T19:10:36.090801+0000","last_unstale":"2024-06-11T19:26:21.390627+0000","last_undegraded":"2024-06-11T19:26:21.390627+0000","last_fullsized":"2024-06-11T19:26:21.390627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:34:25.520333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1332,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630306+0000","last_change":"2024-06-11T19:10:36.063862+0000","last_active":"2024-06-11T19:26:21.630306+0000","last_peered":"2024-06-11T19:26:21.630306+0000","last_clean":"2024-06-11T19:26:21.630306+0000","last_became_active":"2024-06-11T19:10:36.057852+0000","last_became_peered":"2024-06-11T19:10:36.057852+0000","last_unstale":"2024-06-11T19:26:21.630306+0000","last_undegraded":"2024-06-11T19:26:21.630306+0000","last_fullsized":"2024-06-11T19:26:21.630306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:16:37.800621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1336,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389829+0000","last_change":"2024-06-11T19:10:36.056378+0000","last_active":"2024-06-11T19:26:21.389829+0000","last_peered":"2024-06-11T19:26:21.389829+0000","last_clean":"2024-06-11T19:26:21.389829+0000","last_became_active":"2024-06-11T19:10:36.042135+0000","last_became_peered":"2024-06-11T19:10:36.042135+0000","last_unstale":"2024-06-11T19:26:21.389829+0000","last_undegraded":"2024-06-11T19:26:21.389829+0000","last_fullsized":"2024-06-11T19:26:21.389829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:38:55.622181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1322,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630471+0000","last_change":"2024-06-11T19:10:36.026724+0000","last_active":"2024-06-11T19:26:21.630471+0000","last_peered":"2024-06-11T19:26:21.630471+0000","last_clean":"2024-06-11T19:26:21.630471+0000","last_became_active":"2024-06-11T19:10:36.026278+0000","last_became_peered":"2024-06-11T19:10:36.026278+0000","last_unstale":"2024-06-11T19:26:21.630471+0000","last_undegraded":"2024-06-11T19:26:21.630471+0000","last_fullsized":"2024-06-11T19:26:21.630471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:22:26.410474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1328,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630311+0000","last_change":"2024-06-11T19:10:36.034688+0000","last_active":"2024-06-11T19:26:21.630311+0000","last_peered":"2024-06-11T19:26:21.630311+0000","last_clean":"2024-06-11T19:26:21.630311+0000","last_became_active":"2024-06-11T19:10:36.028329+0000","last_became_peered":"2024-06-11T19:10:36.028329+0000","last_unstale":"2024-06-11T19:26:21.630311+0000","last_undegraded":"2024-06-11T19:26:21.630311+0000","last_fullsized":"2024-06-11T19:26:21.630311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T06:59:08.595933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630349+0000","last_change":"2024-06-11T19:10:36.085537+0000","last_active":"2024-06-11T19:26:21.630349+0000","last_peered":"2024-06-11T19:26:21.630349+0000","last_clean":"2024-06-11T19:26:21.630349+0000","last_became_active":"2024-06-11T19:10:36.082897+0000","last_became_peered":"2024-06-11T19:10:36.082897+0000","last_unstale":"2024-06-11T19:26:21.630349+0000","last_undegraded":"2024-06-11T19:26:21.630349+0000","last_fullsized":"2024-06-11T19:26:21.630349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:29:42.550377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390495+0000","last_change":"2024-06-11T19:10:36.017667+0000","last_active":"2024-06-11T19:26:21.390495+0000","last_peered":"2024-06-11T19:26:21.390495+0000","last_clean":"2024-06-11T19:26:21.390495+0000","last_became_active":"2024-06-11T19:10:36.002730+0000","last_became_peered":"2024-06-11T19:10:36.002730+0000","last_unstale":"2024-06-11T19:26:21.390495+0000","last_undegraded":"2024-06-11T19:26:21.390495+0000","last_fullsized":"2024-06-11T19:26:21.390495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:34:23.834164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1328,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630373+0000","last_change":"2024-06-11T19:10:36.076140+0000","last_active":"2024-06-11T19:26:21.630373+0000","last_peered":"2024-06-11T19:26:21.630373+0000","last_clean":"2024-06-11T19:26:21.630373+0000","last_became_active":"2024-06-11T19:10:36.075888+0000","last_became_peered":"2024-06-11T19:10:36.075888+0000","last_unstale":"2024-06-11T19:26:21.630373+0000","last_undegraded":"2024-06-11T19:26:21.630373+0000","last_fullsized":"2024-06-11T19:26:21.630373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:07:12.839020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607763,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77264,"kb_used_data":9968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294576,"statfs":{"total":96636764160,"available":96557645824,"internally_reserved":0,"allocated":10207232,"data_stored":5691113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81956,"kb_used_data":10488,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289884,"statfs":{"total":96636764160,"available":96552841216,"internally_reserved":0,"allocated":10739712,"data_stored":6220041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640467,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86100,"kb_used_data":10484,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94285740,"statfs":{"total":96636764160,"available":96548597760,"internally_reserved":0,"allocated":10735616,"data_stored":6220036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640468,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73112,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298728,"statfs":{"total":96636764160,"available":96561897472,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640467,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67532,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94304308,"statfs":{"total":96636764160,"available":96567611392,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640469,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85604,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286236,"statfs":{"total":96636764160,"available":96549105664,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-11T19:26:48.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-11T19:26:48.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.868+0000 7fd7bc6f9640 10 monclient: get_monmap_and_config 2024-06-11T19:26:48.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.868+0000 7fd7bc6f9640 10 monclient: build_initial_monmap 2024-06-11T19:26:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.868+0000 7fd7bc6f9640 10 monclient: monmap: 2024-06-11T19:26:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:48.869467+0000 2024-06-11T19:26:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:48.869467+0000 2024-06-11T19:26:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.868 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:48.869 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:48.869 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:48.869 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:48.869 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:48.869 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:48.869 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:48.869 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:48.869 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:48.870 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:48.870 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:48.870 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 1 Processor -- start 2024-06-11T19:26:48.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 1 -- start start 2024-06-11T19:26:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 10 monclient: init 2024-06-11T19:26:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 10 monclient: _add_conns ranks=[10,4,3,6,7,0,8,9,2,5,1] 2024-06-11T19:26:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd7b40a4920 0x7fd7b40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 10 monclient(hunting): picked mon.noname-f con 0x7fd7b40a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:26:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a5240 0x7fd7b40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b40a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7b40a2530 0x7fd7b40a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 10 monclient(hunting): picked mon.noname-g con 0x7fd7b40a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7fd7b40a7140 con 0x7fd7b40a4920 2024-06-11T19:26:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b40a2b50 con 0x7fd7b40a2530 2024-06-11T19:26:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd7b40a2e30 con 0x7fd7b40a5240 2024-06-11T19:26:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:48.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bc6f9640 10 monclient(hunting): authenticate will time out at 1818.676685s 2024-06-11T19:26:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7b9c94640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a5240 0x7fd7b40a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7b9c94640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a5240 0x7fd7b40a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:37256/0 (socket says 172.21.15.98:37256) 2024-06-11T19:26:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7b9c94640 1 -- 172.21.15.98:0/3625060413 learned_addr learned my addr 172.21.15.98:0/3625060413 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7b9c94640 10 monclient(hunting): get_auth_request con 0x7fd7b40a5240 auth_method 0 2024-06-11T19:26:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7b9c94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7b9c94640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7b9c94640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7ba495640 1 --2- 172.21.15.98:0/3625060413 >> v2:172.21.15.117:3300/0 conn(0x7fd7b40a4920 0x7fd7b40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7ba495640 10 monclient(hunting): get_auth_request con 0x7fd7b40a4920 auth_method 0 2024-06-11T19:26:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7ba495640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7ba495640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7ba495640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7b9c94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7b9c94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7b9c94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:48.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bac96640 1 --2- 172.21.15.98:0/3625060413 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7b40a2530 0x7fd7b40a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bac96640 10 monclient(hunting): get_auth_request con 0x7fd7b40a2530 auth_method 0 2024-06-11T19:26:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bac96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bac96640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.869+0000 7fd7bac96640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7ba495640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7ba495640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7ba495640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7bac96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:48.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7bac96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7bac96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7b9c94640 10 monclient(hunting): handle_auth_done global_id 134185 payload 274 2024-06-11T19:26:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7b9c94640 1 -- 172.21.15.98:0/3625060413 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7b40a2530 msgr2=0x7fd7b40a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7b9c94640 1 --2- 172.21.15.98:0/3625060413 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7b40a2530 0x7fd7b40a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7b9c94640 1 -- 172.21.15.98:0/3625060413 >> v2:172.21.15.117:3300/0 conn(0x7fd7b40a4920 msgr2=0x7fd7b40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7b9c94640 1 --2- 172.21.15.98:0/3625060413 >> v2:172.21.15.117:3300/0 conn(0x7fd7b40a4920 0x7fd7b40a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7b9c94640 10 monclient: _finish_hunting 0 2024-06-11T19:26:48.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7b9c94640 1 monclient: found mon.noname-c 2024-06-11T19:26:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7b9c94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7b9c94640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:26:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7b9c94640 1 -- 172.21.15.98:0/3625060413 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b40a3110 con 0x7fd7b40a5240 2024-06-11T19:26:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.870+0000 7fd7b9c94640 1 --2- 172.21.15.98:0/3625060413 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a5240 0x7fd7b40a59a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8009870 tx=0x7fd7a802b7c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=b10495cf03fefd9a server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 1 -- 172.21.15.98:0/3625060413 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7a8038070 con 0x7fd7b40a5240 2024-06-11T19:26:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:26:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 10 monclient: dump: 2024-06-11T19:26:48.885 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:48.886 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:48.887 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 10 monclient: _finish_auth 0 2024-06-11T19:26:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 10 monclient: _check_auth_tickets 2024-06-11T19:26:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 5 monclient: authenticate success, global_id 134185 2024-06-11T19:26:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 1 -- 172.21.15.98:0/3625060413 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a8031990 con 0x7fd7b40a5240 2024-06-11T19:26:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:48.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 4 set_mon_vals no callback set 2024-06-11T19:26:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 1 -- 172.21.15.98:0/3625060413 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7a80323b0 con 0x7fd7b40a5240 2024-06-11T19:26:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:26:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7b9493640 10 monclient: dump: 2024-06-11T19:26:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.889 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:48.890 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:48.891 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 10 monclient: shutdown 2024-06-11T19:26:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3625060413 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a5240 msgr2=0x7fd7b40a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 1 --2- 172.21.15.98:0/3625060413 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a5240 0x7fd7b40a59a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8009870 tx=0x7fd7a802b7c0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3625060413 shutdown_connections 2024-06-11T19:26:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 1 --2- 172.21.15.98:0/3625060413 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7b40a2530 0x7fd7b40a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 1 --2- 172.21.15.98:0/3625060413 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a5240 0x7fd7b40a59a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 1 --2- 172.21.15.98:0/3625060413 >> v2:172.21.15.117:3300/0 conn(0x7fd7b40a4920 0x7fd7b40a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3625060413 >> 172.21.15.98:0/3625060413 conn(0x7fd7b40ab850 msgr2=0x7fd7b40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3625060413 shutdown_connections 2024-06-11T19:26:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.871+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3625060413 wait complete. 2024-06-11T19:26:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient: build_initial_monmap 2024-06-11T19:26:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient: monmap: 2024-06-11T19:26:48.892 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:48.894 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 1 librados: starting msgr at 2024-06-11T19:26:48.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 1 librados: starting objecter 2024-06-11T19:26:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 1 Processor -- start 2024-06-11T19:26:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 1 -- start start 2024-06-11T19:26:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 1 librados: setting wanted keys 2024-06-11T19:26:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 1 librados: calling monclient init 2024-06-11T19:26:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient: init 2024-06-11T19:26:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient: _add_conns ranks=[4,8,6,3,2,0,7,1,5] 2024-06-11T19:26:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a2530 0x7fd7b41cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b40a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd7b40a4920 0x7fd7b41cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient(hunting): picked mon.noname-e con 0x7fd7b40a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7b40a5240 0x7fd7b4125020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient(hunting): picked mon.noname-d con 0x7fd7b40a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd7b40a6da0 con 0x7fd7b40a2530 2024-06-11T19:26:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd7b4138e50 con 0x7fd7b40a5240 2024-06-11T19:26:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd7b40a6420 con 0x7fd7b40a4920 2024-06-11T19:26:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.872+0000 7fd7bc6f9640 10 monclient(hunting): authenticate will time out at 1818.679974s 2024-06-11T19:26:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a2530 0x7fd7b41cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b9c94640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd7b40a4920 0x7fd7b41cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a2530 0x7fd7b41cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:37264/0 (socket says 172.21.15.98:37264) 2024-06-11T19:26:48.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b9c94640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd7b40a4920 0x7fd7b41cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54478/0 (socket says 172.21.15.98:54478) 2024-06-11T19:26:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 1 -- 172.21.15.98:0/3930141133 learned_addr learned my addr 172.21.15.98:0/3930141133 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 10 monclient(hunting): get_auth_request con 0x7fd7b40a2530 auth_method 0 2024-06-11T19:26:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bac96640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7b40a5240 0x7fd7b4125020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b9c94640 10 monclient(hunting): get_auth_request con 0x7fd7b40a4920 auth_method 0 2024-06-11T19:26:48.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b9c94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b9c94640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b9c94640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bac96640 10 monclient(hunting): get_auth_request con 0x7fd7b40a5240 auth_method 0 2024-06-11T19:26:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bac96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bac96640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bac96640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:48.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bac96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bac96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bac96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b9c94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b9c94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b9c94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 10 monclient(hunting): handle_auth_done global_id 134194 payload 824 2024-06-11T19:26:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 1 -- 172.21.15.98:0/3930141133 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd7b40a4920 msgr2=0x7fd7b41cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd7b40a4920 0x7fd7b41cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 1 -- 172.21.15.98:0/3930141133 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7b40a5240 msgr2=0x7fd7b4125020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7b40a5240 0x7fd7b4125020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 10 monclient: _finish_hunting 0 2024-06-11T19:26:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 1 monclient: found mon.noname-c 2024-06-11T19:26:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:26:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 1 -- 172.21.15.98:0/3930141133 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b4128a20 con 0x7fd7b40a2530 2024-06-11T19:26:48.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bac96640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7b40a5240 0x7fd7b4125020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7ba495640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a2530 0x7fd7b41cdaa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00e9b0 tx=0x7fd7ac00ee70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 10 client.?.objecter ms_handle_connect 0x7fd7b40a2530 2024-06-11T19:26:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 1 -- 172.21.15.98:0/3930141133 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7ac017070 con 0x7fd7b40a2530 2024-06-11T19:26:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:26:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 10 monclient: dump: 2024-06-11T19:26:48.902 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:48.903 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:48.904 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 10 monclient: _finish_auth 0 2024-06-11T19:26:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:26:48.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 1 -- 172.21.15.98:0/3930141133 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ac00ce10 con 0x7fd7b40a2530 2024-06-11T19:26:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bc6f9640 5 monclient: authenticate success, global_id 134194 2024-06-11T19:26:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bc6f9640 10 monclient: _renew_subs 2024-06-11T19:26:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bc6f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3930141133 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b4127190 con 0x7fd7b40a2530 2024-06-11T19:26:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 1 -- 172.21.15.98:0/3930141133 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7ac0138e0 con 0x7fd7b40a2530 2024-06-11T19:26:48.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:48.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:26:48.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b2ffd640 10 monclient: dump: 2024-06-11T19:26:48.906 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:48.906 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:48.906 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.906 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:48.906 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:48.906 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:48.907 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:48.907 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:48.907 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:48.907 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:48.907 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:48.907 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:48.907 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:48.907 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:48.907 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:48.908 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:48.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bc6f9640 10 client.134194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:48.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bc6f9640 10 monclient: _renew_subs 2024-06-11T19:26:48.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7b8c92640 4 set_mon_vals no callback set 2024-06-11T19:26:48.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bc6f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:48.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3930141133 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b4128d00 con 0x7fd7b40a2530 2024-06-11T19:26:48.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.873+0000 7fd7bc6f9640 1 librados: init done 2024-06-11T19:26:48.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.874+0000 7fd7b2ffd640 1 -- 172.21.15.98:0/3930141133 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7fd7ac01a070 con 0x7fd7b40a2530 2024-06-11T19:26:48.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.874+0000 7fd7bc6f9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:48.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.874+0000 7fd7bc6f9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:48.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.874+0000 7fd7bc6f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:48.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.874+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3930141133 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7b41387f0 con 0x7fd7b40a2530 2024-06-11T19:26:48.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.874+0000 7fd7b2ffd640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd7a003fdb0 0x7fd7a0042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:48.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.874+0000 7fd7b2ffd640 1 -- 172.21.15.98:0/3930141133 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7fd7ac00f040 con 0x7fd7b40a2530 2024-06-11T19:26:48.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.874+0000 7fd7b2ffd640 10 client.134194.objecter ms_dispatch 0x7fd7b4000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:48.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.874+0000 7fd7b2ffd640 3 client.134194.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:48.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.874+0000 7fd7b2ffd640 3 client.134194.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:48.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.874+0000 7fd7b2ffd640 20 client.134194.objecter dump_active .. 0 homeless 2024-06-11T19:26:48.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.875+0000 7fd7b9c94640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd7a003fdb0 0x7fd7a0042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:48.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.875+0000 7fd7b9c94640 10 monclient: get_auth_request con 0x7fd7a003fdb0 auth_method 0 2024-06-11T19:26:48.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.875+0000 7fd7b2ffd640 10 client.134194.objecter ms_handle_connect 0x7fd7a003fdb0 2024-06-11T19:26:48.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.875+0000 7fd7b9c94640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd7a003fdb0 0x7fd7a0042270 secure :-1 s=READY pgs=4334 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8035000 tx=0x7fd7a8030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:48.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.879+0000 7fd7b2ffd640 1 -- 172.21.15.98:0/3930141133 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd7ac021070 con 0x7fd7b40a2530 2024-06-11T19:26:48.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.879+0000 7fd7b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:48.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:48.879+0000 7fd7b2ffd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.065+0000 7fd7bc6f9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:26:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.065+0000 7fd7bc6f9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:26:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.065+0000 7fd7bc6f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:49.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.065+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3930141133 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7b40a6720 con 0x7fd7b40a2530 2024-06-11T19:26:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.065+0000 7fd7b2ffd640 1 -- 172.21.15.98:0/3930141133 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v685) ==== 74+0+15372 (secure 0 0 0) 0x7fd7ac013650 con 0x7fd7b40a2530 2024-06-11T19:26:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.065+0000 7fd7b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-11T19:26:49.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.065+0000 7fd7b2ffd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:26:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:26:49.066 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":685,"fsid":"9daa5251-9ae7-4a07-8f69-80112d2163e1","created":"2024-06-11T19:09:15.016586+0000","modified":"2024-06-11T19:26:21.383262+0000","last_up_change":"2024-06-11T19:09:24.656699+0000","last_in_change":"2024-06-11T19:09:16.662340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":100,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":340,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-11T19:09:25.381456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-11T19:09:31.785910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-06-11T19:10:34.142046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ef1075e3-3e1b-4904-b14b-90751df6c38c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":680,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6811","nonce":4087730304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6815","nonce":4087730304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6823","nonce":4087730304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":4087730304},{"type":"v1","addr":"172.21.15.98:6819","nonce":4087730304}]},"public_addr":"172.21.15.98:6811/4087730304","cluster_addr":"172.21.15.98:6815/4087730304","heartbeat_back_addr":"172.21.15.98:6823/4087730304","heartbeat_front_addr":"172.21.15.98:6819/4087730304","state":["exists","up"]},{"osd":1,"uuid":"985893e9-bd60-46fc-b913-7502ff73628d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":680,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6801","nonce":3886067876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6803","nonce":3886067876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6807","nonce":3886067876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3886067876},{"type":"v1","addr":"172.21.15.98:6805","nonce":3886067876}]},"public_addr":"172.21.15.98:6801/3886067876","cluster_addr":"172.21.15.98:6803/3886067876","heartbeat_back_addr":"172.21.15.98:6807/3886067876","heartbeat_front_addr":"172.21.15.98:6805/3886067876","state":["exists","up"]},{"osd":2,"uuid":"2b0b4d24-8009-4cec-80e4-c0fa2d705879","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":680,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6810","nonce":3751501255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6814","nonce":3751501255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6822","nonce":3751501255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3751501255},{"type":"v1","addr":"172.21.15.98:6818","nonce":3751501255}]},"public_addr":"172.21.15.98:6810/3751501255","cluster_addr":"172.21.15.98:6814/3751501255","heartbeat_back_addr":"172.21.15.98:6822/3751501255","heartbeat_front_addr":"172.21.15.98:6818/3751501255","state":["exists","up"]},{"osd":3,"uuid":"73793608-9e04-4837-bd2b-1b130a4e88d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":680,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6809","nonce":2561136926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6811","nonce":2561136926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6815","nonce":2561136926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2561136926},{"type":"v1","addr":"172.21.15.117:6813","nonce":2561136926}]},"public_addr":"172.21.15.117:6809/2561136926","cluster_addr":"172.21.15.117:6811/2561136926","heartbeat_back_addr":"172.21.15.117:6815/2561136926","heartbeat_front_addr":"172.21.15.117:6813/2561136926","state":["exists","up"]},{"osd":4,"uuid":"60e657ae-e2a2-471e-b998-fba98d9c94e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":680,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6801","nonce":3877218228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6803","nonce":3877218228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6807","nonce":3877218228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3877218228},{"type":"v1","addr":"172.21.15.117:6805","nonce":3877218228}]},"public_addr":"172.21.15.117:6801/3877218228","cluster_addr":"172.21.15.117:6803/3877218228","heartbeat_back_addr":"172.21.15.117:6807/3877218228","heartbeat_front_addr":"172.21.15.117:6805/3877218228","state":["exists","up"]},{"osd":5,"uuid":"dae66a65-2ca2-4f8a-98e3-52b190abee7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":680,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6817","nonce":987023138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6819","nonce":987023138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6823","nonce":987023138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":987023138},{"type":"v1","addr":"172.21.15.117:6821","nonce":987023138}]},"public_addr":"172.21.15.117:6817/987023138","cluster_addr":"172.21.15.117:6819/987023138","heartbeat_back_addr":"172.21.15.117:6823/987023138","heartbeat_front_addr":"172.21.15.117:6821/987023138","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.744223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.800796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.792849+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.854633+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.844451+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-11T19:09:21.763441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":2,"length":5}]},{"pool":14,"snaps":[{"begin":2,"length":3}]},{"pool":34,"snaps":[{"begin":1,"length":1}]},{"pool":66,"snaps":[{"begin":1,"length":1}]},{"pool":94,"snaps":[{"begin":4,"length":1},{"begin":7,"length":2}]},{"pool":103,"snaps":[{"begin":1,"length":1}]},{"pool":122,"snaps":[{"begin":1,"length":2}]},{"pool":137,"snaps":[{"begin":3,"length":1}]},{"pool":144,"snaps":[{"begin":1,"length":4}]},{"pool":151,"snaps":[{"begin":2,"length":1}]},{"pool":152,"snaps":[{"begin":2,"length":2}]},{"pool":162,"snaps":[{"begin":2,"length":3}]},{"pool":167,"snaps":[{"begin":3,"length":4}]},{"pool":170,"snaps":[{"begin":2,"length":4}]},{"pool":172,"snaps":[{"begin":2,"length":2}]},{"pool":181,"snaps":[{"begin":3,"length":1}]},{"pool":182,"snaps":[{"begin":2,"length":1}]},{"pool":187,"snaps":[{"begin":1,"length":4}]},{"pool":193,"snaps":[{"begin":7,"length":1}]},{"pool":201,"snaps":[{"begin":2,"length":1},{"begin":4,"length":4}]},{"pool":222,"snaps":[{"begin":8,"length":1}]},{"pool":233,"snaps":[{"begin":10,"length":1}]},{"pool":241,"snaps":[{"begin":11,"length":2}]},{"pool":255,"snaps":[{"begin":14,"length":1}]},{"pool":321,"snaps":[{"begin":2,"length":1}]},{"pool":322,"snaps":[{"begin":3,"length":1}]},{"pool":325,"snaps":[{"begin":4,"length":1}]},{"pool":329,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-11T19:26:49.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 10 librados: watch_flush enter 2024-06-11T19:26:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 10 librados: watch_flush exit 2024-06-11T19:26:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 20 client.134194.objecter shutdown clearing up homeless session... 2024-06-11T19:26:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 10 client.134194.objecter successfully canceled tick 2024-06-11T19:26:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3930141133 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd7a003fdb0 msgr2=0x7fd7a0042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd7a003fdb0 0x7fd7a0042270 secure :-1 s=READY pgs=4334 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8035000 tx=0x7fd7a8030000 comp rx=0 tx=0).stop 2024-06-11T19:26:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 10 monclient: shutdown 2024-06-11T19:26:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:49.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3930141133 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a2530 msgr2=0x7fd7b41cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a2530 0x7fd7b41cdaa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00e9b0 tx=0x7fd7ac00ee70 comp rx=0 tx=0).stop 2024-06-11T19:26:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3930141133 shutdown_connections 2024-06-11T19:26:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7fd7a003fdb0 0x7fd7a0042270 unknown :-1 s=CLOSED pgs=4334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd7b40a5240 0x7fd7b4125020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd7b40a4920 0x7fd7b41cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 --2- 172.21.15.98:0/3930141133 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd7b40a2530 0x7fd7b41cdaa0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3930141133 >> 172.21.15.98:0/3930141133 conn(0x7fd7b40ab850 msgr2=0x7fd7b409c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:49.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3930141133 shutdown_connections 2024-06-11T19:26:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 -- 172.21.15.98:0/3930141133 wait complete. 2024-06-11T19:26:49.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:49.068+0000 7fd7bc6f9640 1 librados: shutdown 2024-06-11T19:26:50.078 INFO:tasks.ceph:Scrubbing osd.0 2024-06-11T19:26:50.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-06-11T19:26:50.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.193+0000 7faa7f1ec640 10 monclient: get_monmap_and_config 2024-06-11T19:26:50.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.193+0000 7faa7f1ec640 10 monclient: build_initial_monmap 2024-06-11T19:26:50.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.193+0000 7faa7f1ec640 10 monclient: monmap: 2024-06-11T19:26:50.192 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:50.192 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.192 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:50.194089+0000 2024-06-11T19:26:50.193 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:50.194089+0000 2024-06-11T19:26:50.193 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:50.193 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.193 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:50.193 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:50.193 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:50.193 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:50.193 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:50.194 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:50.194 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:50.194 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:50.194 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:50.194 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:50.194 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:50.194 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.193+0000 7faa7f1ec640 1 Processor -- start 2024-06-11T19:26:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.193+0000 7faa7f1ec640 1 -- start start 2024-06-11T19:26:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.193+0000 7faa7f1ec640 10 monclient: init 2024-06-11T19:26:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 10 monclient: _add_conns ranks=[3,5,1,0,8,6,9,2,10,4,7] 2024-06-11T19:26:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa7806c780 0x7faa780a1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 10 monclient(hunting): picked mon.noname-g con 0x7faa7806c780 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7faa780a1c80 0x7faa780a3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 10 monclient(hunting): picked mon.noname-h con 0x7faa780a1c80 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7faa780a3c70 addr v1:172.21.15.117:6789/0 2024-06-11T19:26:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa7809d450 con 0x7faa780a3c70 2024-06-11T19:26:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7faa7806cb60 con 0x7faa7806c780 2024-06-11T19:26:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7faa78135ef0 con 0x7faa780a1c80 2024-06-11T19:26:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7f1ec640 10 monclient(hunting): authenticate will time out at 1820.001409s 2024-06-11T19:26:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7cf88640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa7806c780 0x7faa780a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa77fff640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7faa780a1c80 0x7faa780a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7d789640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7faa780a3c70 0x7faa7813c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:60706/0 (socket says 172.21.15.98:60706) 2024-06-11T19:26:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7d789640 1 -- 172.21.15.98:0/2411505067 learned_addr learned my addr 172.21.15.98:0/2411505067 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:26:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa77fff640 10 monclient(hunting): get_auth_request con 0x7faa780a1c80 auth_method 0 2024-06-11T19:26:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa77fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa77fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7cf88640 10 monclient(hunting): get_auth_request con 0x7faa7806c780 auth_method 0 2024-06-11T19:26:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7cf88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7cf88640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7cf88640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7cf88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7cf88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.194+0000 7faa7cf88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa7cf88640 10 monclient(hunting): handle_auth_done global_id 114564 payload 274 2024-06-11T19:26:50.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa7cf88640 1 -- 172.21.15.98:0/2411505067 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7faa780a1c80 msgr2=0x7faa780a3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:50.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa7cf88640 1 --2- 172.21.15.98:0/2411505067 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7faa780a1c80 0x7faa780a3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa777fe640 1 -- 172.21.15.98:0/2411505067 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 607843012 0 0) 0x7faa7809d450 con 0x7faa780a3c70 2024-06-11T19:26:50.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa7cf88640 1 -- 172.21.15.98:0/2411505067 >> v1:172.21.15.117:6789/0 conn(0x7faa780a3c70 legacy=0x7faa7813c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa7cf88640 10 monclient: _finish_hunting 0 2024-06-11T19:26:50.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa7cf88640 1 monclient: found mon.noname-g 2024-06-11T19:26:50.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa7cf88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:50.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa7cf88640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.117:3301/0 2024-06-11T19:26:50.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa7cf88640 1 -- 172.21.15.98:0/2411505067 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7809e700 con 0x7faa7806c780 2024-06-11T19:26:50.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa77fff640 1 --2- 172.21.15.98:0/2411505067 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7faa780a1c80 0x7faa780a3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:50.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa777fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:26:50.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.195+0000 7faa7cf88640 1 --2- 172.21.15.98:0/2411505067 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa7806c780 0x7faa780a1740 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faa700097d0 tx=0x7faa7002b770 comp rx=0 tx=0).ready entity=mon.3 client_cookie=b0b2ecb822dd57a0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 1 -- 172.21.15.98:0/2411505067 <== mon.3 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faa70039070 con 0x7faa7806c780 2024-06-11T19:26:50.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-06-11T19:26:50.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 10 monclient: dump: 2024-06-11T19:26:50.211 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.211 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.211 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.211 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.211 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.211 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.211 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.212 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.212 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.212 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.212 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.212 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.212 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.212 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.212 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.212 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 10 monclient: _finish_auth 0 2024-06-11T19:26:50.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 10 monclient: _check_auth_tickets 2024-06-11T19:26:50.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:50.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 5 monclient: authenticate success, global_id 114564 2024-06-11T19:26:50.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:50.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 1 -- 172.21.15.98:0/2411505067 <== mon.3 v2:172.21.15.117:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa70030070 con 0x7faa7806c780 2024-06-11T19:26:50.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:50.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:50.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 4 set_mon_vals no callback set 2024-06-11T19:26:50.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 1 -- 172.21.15.98:0/2411505067 <== mon.3 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faa70031db0 con 0x7faa7806c780 2024-06-11T19:26:50.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-06-11T19:26:50.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa777fe640 10 monclient: dump: 2024-06-11T19:26:50.214 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.214 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.214 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.216 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 10 monclient: shutdown 2024-06-11T19:26:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 1 -- 172.21.15.98:0/2411505067 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa7806c780 msgr2=0x7faa780a1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 1 --2- 172.21.15.98:0/2411505067 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa7806c780 0x7faa780a1740 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faa700097d0 tx=0x7faa7002b770 comp rx=0 tx=0).stop 2024-06-11T19:26:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 1 -- 172.21.15.98:0/2411505067 shutdown_connections 2024-06-11T19:26:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 1 --2- 172.21.15.98:0/2411505067 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7faa780a1c80 0x7faa780a3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 1 --2- 172.21.15.98:0/2411505067 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa7806c780 0x7faa780a1740 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 1 -- 172.21.15.98:0/2411505067 >> 172.21.15.98:0/2411505067 conn(0x7faa780a4f00 msgr2=0x7faa7806bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 1 -- 172.21.15.98:0/2411505067 shutdown_connections 2024-06-11T19:26:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.196+0000 7faa7f1ec640 1 -- 172.21.15.98:0/2411505067 wait complete. 2024-06-11T19:26:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.197+0000 7faa7f1ec640 10 monclient: build_initial_monmap 2024-06-11T19:26:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.197+0000 7faa7f1ec640 10 monclient: monmap: 2024-06-11T19:26:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:50.219 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.197+0000 7faa7f1ec640 1 librados: starting msgr at 2024-06-11T19:26:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.197+0000 7faa7f1ec640 1 librados: starting objecter 2024-06-11T19:26:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.197+0000 7faa7f1ec640 1 Processor -- start 2024-06-11T19:26:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.197+0000 7faa7f1ec640 1 -- start start 2024-06-11T19:26:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.197+0000 7faa7f1ec640 1 librados: setting wanted keys 2024-06-11T19:26:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.197+0000 7faa7f1ec640 1 librados: calling monclient init 2024-06-11T19:26:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.197+0000 7faa7f1ec640 10 monclient: init 2024-06-11T19:26:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 10 monclient: _add_conns ranks=[3,8,7,2,1,5,0,6,4] 2024-06-11T19:26:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa7806c780 0x7faa781cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 10 monclient(hunting): picked mon.noname-g con 0x7faa7806c780 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faa780a1c80 0x7faa781cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 10 monclient(hunting): picked mon.noname-e con 0x7faa780a1c80 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7faa780a3c70 0x7faa781cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 10 monclient(hunting): picked mon.noname-i con 0x7faa780a3c70 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7faa7809ecc0 con 0x7faa7806c780 2024-06-11T19:26:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7faa78050490 con 0x7faa780a3c70 2024-06-11T19:26:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7faa78135be0 con 0x7faa780a1c80 2024-06-11T19:26:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7f1ec640 10 monclient(hunting): authenticate will time out at 1820.005274s 2024-06-11T19:26:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faa780a1c80 0x7faa781cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faa780a1c80 0x7faa781cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54492/0 (socket says 172.21.15.98:54492) 2024-06-11T19:26:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 1 -- 172.21.15.98:0/1795839504 learned_addr learned my addr 172.21.15.98:0/1795839504 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 10 monclient(hunting): get_auth_request con 0x7faa780a1c80 auth_method 0 2024-06-11T19:26:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7cf88640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa7806c780 0x7faa781cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7d789640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7faa780a3c70 0x7faa781cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7cf88640 10 monclient(hunting): get_auth_request con 0x7faa7806c780 auth_method 0 2024-06-11T19:26:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7cf88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7cf88640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7cf88640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7d789640 10 monclient(hunting): get_auth_request con 0x7faa780a3c70 auth_method 0 2024-06-11T19:26:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7d789640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7d789640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7d789640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7cf88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7cf88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7cf88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7d789640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7d789640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa7d789640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 10 monclient(hunting): handle_auth_done global_id 124478 payload 824 2024-06-11T19:26:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 1 -- 172.21.15.98:0/1795839504 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7faa780a3c70 msgr2=0x7faa781cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7faa780a3c70 0x7faa781cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 1 -- 172.21.15.98:0/1795839504 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa7806c780 msgr2=0x7faa781cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa7806c780 0x7faa781cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 10 monclient: _finish_hunting 0 2024-06-11T19:26:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 1 monclient: found mon.noname-e 2024-06-11T19:26:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:26:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 1 -- 172.21.15.98:0/1795839504 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa78128950 con 0x7faa780a1c80 2024-06-11T19:26:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa77fff640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faa780a1c80 0x7faa781cd9b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faa68009f60 tx=0x7faa68009f90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa757fa640 10 client.?.objecter ms_handle_connect 0x7faa780a1c80 2024-06-11T19:26:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.198+0000 7faa757fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 1 -- 172.21.15.98:0/1795839504 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faa68018020 con 0x7faa780a1c80 2024-06-11T19:26:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:26:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 10 monclient: dump: 2024-06-11T19:26:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.229 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 10 monclient: _finish_auth 0 2024-06-11T19:26:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 10 monclient: _check_auth_tickets 2024-06-11T19:26:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 1 -- 172.21.15.98:0/1795839504 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa680148f0 con 0x7faa780a1c80 2024-06-11T19:26:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 5 monclient: authenticate success, global_id 124478 2024-06-11T19:26:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 10 monclient: _renew_subs 2024-06-11T19:26:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 1 -- 172.21.15.98:0/1795839504 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa78128f10 con 0x7faa780a1c80 2024-06-11T19:26:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 1 -- 172.21.15.98:0/1795839504 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faa680133e0 con 0x7faa780a1c80 2024-06-11T19:26:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa76ffd640 4 set_mon_vals no callback set 2024-06-11T19:26:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 10 client.124478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 10 monclient: dump: 2024-06-11T19:26:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.233 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 10 monclient: _renew_subs 2024-06-11T19:26:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 1 -- 172.21.15.98:0/1795839504 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7faa780a6dc0 con 0x7faa780a1c80 2024-06-11T19:26:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 1 librados: init done 2024-06-11T19:26:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 10 client.124478.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 15 client.124478.objecter _session_command_op_assign -1 1 2024-06-11T19:26:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 20 client.124478.objecter _get_session osd=-1 returning homeless 2024-06-11T19:26:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 10 client.124478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa7f1ec640 1 -- 172.21.15.98:0/1795839504 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faa78068880 con 0x7faa780a1c80 2024-06-11T19:26:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.199+0000 7faa757fa640 1 -- 172.21.15.98:0/1795839504 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7faa68005d10 con 0x7faa780a1c80 2024-06-11T19:26:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7faa5803fdb0 0x7faa58042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 1 -- 172.21.15.98:0/1795839504 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7faa68051fd0 con 0x7faa780a1c80 2024-06-11T19:26:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 10 client.124478.objecter ms_dispatch 0x7faa78000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 3 client.124478.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 3 client.124478.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 10 client.124478.objecter checking command 1 2024-06-11T19:26:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] conn(0x7faa580475f0 0x7faa58049a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 20 client.124478.objecter _get_session s=0x7faa58046f20 osd=0 3 2024-06-11T19:26:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 20 client.124478.objecter put_session s=0x7faa58046f20 osd=0 3 2024-06-11T19:26:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 15 client.124478.objecter _session_command_op_remove -1 1 2024-06-11T19:26:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 20 client.124478.objecter _get_session s=0x7faa58046f20 osd=0 3 2024-06-11T19:26:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 20 client.124478.objecter get_session s=0x7faa58046f20 osd=0 3 2024-06-11T19:26:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 15 client.124478.objecter _session_command_op_assign 0 1 2024-06-11T19:26:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 20 client.124478.objecter put_session s=0x7faa58046f20 osd=0 4 2024-06-11T19:26:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 10 client.124478.objecter _send_command 1 2024-06-11T19:26:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 1 -- 172.21.15.98:0/1795839504 --> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa6800be50 con 0x7faa580475f0 2024-06-11T19:26:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 20 client.124478.objecter dump_active .. 0 homeless 2024-06-11T19:26:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 1 -- 172.21.15.98:0/1795839504 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_get_version_reply(handle=1 version=685) ==== 24+0+0 (secure 0 0 0) 0x7faa68059050 con 0x7faa780a1c80 2024-06-11T19:26:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 10 monclient: handle_get_version_reply finishing 1 version 685 2024-06-11T19:26:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa7cf88640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7faa5803fdb0 0x7faa58042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa7cf88640 10 monclient: get_auth_request con 0x7faa5803fdb0 auth_method 0 2024-06-11T19:26:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa7d789640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] conn(0x7faa580475f0 0x7faa58049a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa7d789640 10 monclient: get_auth_request con 0x7faa580475f0 auth_method 0 2024-06-11T19:26:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 10 client.124478.objecter ms_handle_connect 0x7faa580475f0 2024-06-11T19:26:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa7d789640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] conn(0x7faa580475f0 0x7faa58049a90 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa757fa640 10 client.124478.objecter ms_handle_connect 0x7faa5803fdb0 2024-06-11T19:26:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.200+0000 7faa7cf88640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7faa5803fdb0 0x7faa58042270 secure :-1 s=READY pgs=4335 cs=0 l=1 rev1=1 crypto rx=0x7faa700097d0 tx=0x7faa70008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.201+0000 7faa757fa640 1 -- 172.21.15.98:0/1795839504 <== osd.0 v2:172.21.15.98:6809/4087730304 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faa6800be50 con 0x7faa580475f0 2024-06-11T19:26:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.201+0000 7faa757fa640 10 client.124478.objecter ms_dispatch 0x7faa78000b90 command_reply(tid 1: 0 ) 2024-06-11T19:26:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.201+0000 7faa757fa640 10 client.124478.objecter _finish_command 1 = system:0 2024-06-11T19:26:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.201+0000 7faa757fa640 20 client.124478.objecter put_session s=0x7faa58046f20 osd=0 4 2024-06-11T19:26:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.201+0000 7faa757fa640 15 client.124478.objecter _session_command_op_remove 0 1 2024-06-11T19:26:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 10 client.124478.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-11T19:26:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 15 client.124478.objecter _session_command_op_assign -1 2 2024-06-11T19:26:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 20 client.124478.objecter _get_session s=0x7faa58046f20 osd=0 3 2024-06-11T19:26:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 20 client.124478.objecter put_session s=0x7faa58046f20 osd=0 3 2024-06-11T19:26:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 20 client.124478.objecter _get_session s=0x7faa58046f20 osd=0 3 2024-06-11T19:26:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 15 client.124478.objecter _session_command_op_remove -1 2 2024-06-11T19:26:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 20 client.124478.objecter get_session s=0x7faa58046f20 osd=0 3 2024-06-11T19:26:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 15 client.124478.objecter _session_command_op_assign 0 2 2024-06-11T19:26:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 20 client.124478.objecter put_session s=0x7faa58046f20 osd=0 4 2024-06-11T19:26:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 10 client.124478.objecter _send_command 2 2024-06-11T19:26:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 1 -- 172.21.15.98:0/1795839504 --> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7faa780a2060 con 0x7faa580475f0 2024-06-11T19:26:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa757fa640 1 -- 172.21.15.98:0/1795839504 <== osd.0 v2:172.21.15.98:6809/4087730304 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7faa780a2060 con 0x7faa580475f0 2024-06-11T19:26:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa757fa640 10 client.124478.objecter ms_dispatch 0x7faa78000b90 command_reply(tid 2: 0 ) 2024-06-11T19:26:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa757fa640 10 client.124478.objecter _finish_command 2 = system:0 2024-06-11T19:26:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa757fa640 20 client.124478.objecter put_session s=0x7faa58046f20 osd=0 4 2024-06-11T19:26:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa757fa640 15 client.124478.objecter _session_command_op_remove 0 2 2024-06-11T19:26:50.241 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:50.241 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-11T19:26:50.241 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 10 librados: watch_flush enter 2024-06-11T19:26:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 10 librados: watch_flush exit 2024-06-11T19:26:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 10 client.124478.objecter close_session for osd.0 2024-06-11T19:26:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 1 -- 172.21.15.98:0/1795839504 >> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] conn(0x7faa580475f0 msgr2=0x7faa58049a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.98:6809/4087730304,v1:172.21.15.98:6811/4087730304] conn(0x7faa580475f0 0x7faa58049a90 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 20 client.124478.objecter put_session s=0x7faa58046f20 osd=0 1 2024-06-11T19:26:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 20 client.124478.objecter shutdown clearing up homeless session... 2024-06-11T19:26:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.214+0000 7faa7f1ec640 10 client.124478.objecter successfully canceled tick 2024-06-11T19:26:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.215+0000 7faa7f1ec640 1 -- 172.21.15.98:0/1795839504 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7faa5803fdb0 msgr2=0x7faa58042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.215+0000 7faa7f1ec640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7faa5803fdb0 0x7faa58042270 secure :-1 s=READY pgs=4335 cs=0 l=1 rev1=1 crypto rx=0x7faa700097d0 tx=0x7faa70008030 comp rx=0 tx=0).stop 2024-06-11T19:26:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.215+0000 7faa7f1ec640 10 monclient: shutdown 2024-06-11T19:26:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.215+0000 7faa7f1ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.215+0000 7faa7f1ec640 1 -- 172.21.15.98:0/1795839504 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faa780a1c80 msgr2=0x7faa781cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.215+0000 7faa7f1ec640 1 --2- 172.21.15.98:0/1795839504 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7faa780a1c80 0x7faa781cd9b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faa68009f60 tx=0x7faa68009f90 comp rx=0 tx=0).stop 2024-06-11T19:26:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.215+0000 7faa7d789640 1 -- 172.21.15.98:0/1795839504 reap_dead start 2024-06-11T19:26:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.215+0000 7faa7f1ec640 1 -- 172.21.15.98:0/1795839504 shutdown_connections 2024-06-11T19:26:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.215+0000 7faa7f1ec640 1 -- 172.21.15.98:0/1795839504 >> 172.21.15.98:0/1795839504 conn(0x7faa780a4f00 msgr2=0x7faa780a6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.215+0000 7faa7f1ec640 1 -- 172.21.15.98:0/1795839504 shutdown_connections 2024-06-11T19:26:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.215+0000 7faa7f1ec640 1 -- 172.21.15.98:0/1795839504 wait complete. 2024-06-11T19:26:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.215+0000 7faa7f1ec640 1 librados: shutdown 2024-06-11T19:26:50.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-06-11T19:26:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.351+0000 7f7a4ffec640 10 monclient: get_monmap_and_config 2024-06-11T19:26:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.351+0000 7f7a4ffec640 10 monclient: build_initial_monmap 2024-06-11T19:26:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.352+0000 7f7a4ffec640 10 monclient: monmap: 2024-06-11T19:26:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.351 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:50.353037+0000 2024-06-11T19:26:50.352 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:50.353037+0000 2024-06-11T19:26:50.352 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:50.352 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.352 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:50.352 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:50.352 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:50.352 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:50.353 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:50.353 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:50.353 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:50.353 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:50.353 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:50.353 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:50.353 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:50.353 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.352+0000 7f7a4ffec640 1 Processor -- start 2024-06-11T19:26:50.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.352+0000 7f7a4ffec640 1 -- start start 2024-06-11T19:26:50.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.352+0000 7f7a4ffec640 10 monclient: init 2024-06-11T19:26:50.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:50.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 10 monclient: _add_conns ranks=[5,10,7,1,8,9,0,3,6,2,4] 2024-06-11T19:26:50.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7a480a4920 0x7f7a480a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 10 monclient(hunting): picked mon.noname-h con 0x7f7a480a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:50.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7a480a5240 0x7f7a480a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 10 monclient(hunting): picked mon.noname-f con 0x7f7a480a5240 addr v2:172.21.15.117:3300/0 2024-06-11T19:26:50.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7a480a2530 0x7f7a480a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 10 monclient(hunting): picked mon.noname-i con 0x7f7a480a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:50.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f7a480a7140 con 0x7f7a480a5240 2024-06-11T19:26:50.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f7a480a2b50 con 0x7f7a480a4920 2024-06-11T19:26:50.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f7a480a2e30 con 0x7f7a480a2530 2024-06-11T19:26:50.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:50.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4ffec640 10 monclient(hunting): authenticate will time out at 1820.160392s 2024-06-11T19:26:50.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7a480a5240 0x7f7a480a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4dd88640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7a480a4920 0x7f7a480a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4e589640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7a480a2530 0x7f7a480a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7a480a5240 0x7f7a480a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.98:48168/0 (socket says 172.21.15.98:48168) 2024-06-11T19:26:50.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 1 -- 172.21.15.98:0/1049481114 learned_addr learned my addr 172.21.15.98:0/1049481114 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:50.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 10 monclient(hunting): get_auth_request con 0x7f7a480a5240 auth_method 0 2024-06-11T19:26:50.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4dd88640 10 monclient(hunting): get_auth_request con 0x7f7a480a4920 auth_method 0 2024-06-11T19:26:50.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4dd88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4dd88640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4dd88640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4e589640 10 monclient(hunting): get_auth_request con 0x7f7a480a2530 auth_method 0 2024-06-11T19:26:50.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4e589640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4e589640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4e589640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4dd88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4dd88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4dd88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4e589640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4e589640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4e589640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 10 monclient(hunting): handle_auth_done global_id 124453 payload 274 2024-06-11T19:26:50.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 1 -- 172.21.15.98:0/1049481114 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7a480a2530 msgr2=0x7f7a480a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:50.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 1 --2- 172.21.15.98:0/1049481114 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7a480a2530 0x7f7a480a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 1 -- 172.21.15.98:0/1049481114 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7a480a4920 msgr2=0x7f7a480a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:50.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 1 --2- 172.21.15.98:0/1049481114 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7a480a4920 0x7f7a480a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 10 monclient: _finish_hunting 0 2024-06-11T19:26:50.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 1 monclient: found mon.noname-f 2024-06-11T19:26:50.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:50.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:26:50.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.353+0000 7f7a4d587640 1 -- 172.21.15.98:0/1049481114 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a480a3110 con 0x7f7a480a5240 2024-06-11T19:26:50.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4d587640 1 --2- 172.21.15.98:0/1049481114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a480a5240 0x7f7a480a59a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009870 tx=0x7f7a3802b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5356f74f17b245eb server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 1 -- 172.21.15.98:0/1049481114 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a38038070 con 0x7f7a480a5240 2024-06-11T19:26:50.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:26:50.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 10 monclient: dump: 2024-06-11T19:26:50.371 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.371 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.372 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.372 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.372 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.372 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.372 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.372 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.372 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.373 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.373 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.373 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.373 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.373 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.373 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.373 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 10 monclient: _finish_auth 0 2024-06-11T19:26:50.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 10 monclient: _check_auth_tickets 2024-06-11T19:26:50.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:50.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4ffec640 5 monclient: authenticate success, global_id 124453 2024-06-11T19:26:50.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 1 -- 172.21.15.98:0/1049481114 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a38031990 con 0x7f7a480a5240 2024-06-11T19:26:50.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4ffec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:50.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:50.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4ffec640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:50.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 1 -- 172.21.15.98:0/1049481114 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a380323b0 con 0x7f7a480a5240 2024-06-11T19:26:50.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4ffec640 4 set_mon_vals no callback set 2024-06-11T19:26:50.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:26:50.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.354+0000 7f7a4cd86640 10 monclient: dump: 2024-06-11T19:26:50.375 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.375 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.376 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.376 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.376 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.376 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.376 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.376 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.376 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.377 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.355+0000 7f7a4ffec640 10 monclient: shutdown 2024-06-11T19:26:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.355+0000 7f7a4ffec640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.355+0000 7f7a4ffec640 1 -- 172.21.15.98:0/1049481114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a480a5240 msgr2=0x7f7a480a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.355+0000 7f7a4ffec640 1 --2- 172.21.15.98:0/1049481114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a480a5240 0x7f7a480a59a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009870 tx=0x7f7a3802b7c0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.355+0000 7f7a4ffec640 1 -- 172.21.15.98:0/1049481114 shutdown_connections 2024-06-11T19:26:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.355+0000 7f7a4ffec640 1 --2- 172.21.15.98:0/1049481114 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7a480a2530 0x7f7a480a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.355+0000 7f7a4ffec640 1 --2- 172.21.15.98:0/1049481114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a480a5240 0x7f7a480a59a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.355+0000 7f7a4ffec640 1 --2- 172.21.15.98:0/1049481114 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7a480a4920 0x7f7a480a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.355+0000 7f7a4ffec640 1 -- 172.21.15.98:0/1049481114 >> 172.21.15.98:0/1049481114 conn(0x7f7a480ab850 msgr2=0x7f7a480a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:50.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.355+0000 7f7a4ffec640 1 -- 172.21.15.98:0/1049481114 shutdown_connections 2024-06-11T19:26:50.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.355+0000 7f7a4ffec640 1 -- 172.21.15.98:0/1049481114 wait complete. 2024-06-11T19:26:50.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient: build_initial_monmap 2024-06-11T19:26:50.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient: monmap: 2024-06-11T19:26:50.379 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:50.379 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:50.380 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:50.381 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:50.381 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 1 librados: starting msgr at 2024-06-11T19:26:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 1 librados: starting objecter 2024-06-11T19:26:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 1 Processor -- start 2024-06-11T19:26:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 1 -- start start 2024-06-11T19:26:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 1 librados: setting wanted keys 2024-06-11T19:26:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 1 librados: calling monclient init 2024-06-11T19:26:50.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient: init 2024-06-11T19:26:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient: _add_conns ranks=[0,5,4,3,1,6,2,8,7] 2024-06-11T19:26:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a480a2530 0x7f7a481cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient(hunting): picked mon.noname-a con 0x7f7a480a2530 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:26:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7a480a4920 0x7f7a481ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient(hunting): picked mon.noname-h con 0x7f7a480a4920 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a480a5240 0x7f7a48122fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient(hunting): picked mon.noname-c con 0x7f7a480a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a480a6da0 con 0x7f7a480a2530 2024-06-11T19:26:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7a48138e50 con 0x7f7a480a5240 2024-06-11T19:26:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f7a480a6420 con 0x7f7a480a4920 2024-06-11T19:26:50.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.356+0000 7f7a4ffec640 10 monclient(hunting): authenticate will time out at 1820.163892s 2024-06-11T19:26:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a480a2530 0x7f7a481cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4e589640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a480a5240 0x7f7a48122fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a480a2530 0x7f7a481cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41876/0 (socket says 172.21.15.98:41876) 2024-06-11T19:26:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4e589640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a480a5240 0x7f7a48122fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:37278/0 (socket says 172.21.15.98:37278) 2024-06-11T19:26:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 1 -- 172.21.15.98:0/3657374873 learned_addr learned my addr 172.21.15.98:0/3657374873 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:50.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 10 monclient(hunting): get_auth_request con 0x7f7a480a2530 auth_method 0 2024-06-11T19:26:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4e589640 10 monclient(hunting): get_auth_request con 0x7f7a480a5240 auth_method 0 2024-06-11T19:26:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4e589640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4e589640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4e589640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4d587640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7a480a4920 0x7f7a481ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4d587640 10 monclient(hunting): get_auth_request con 0x7f7a480a4920 auth_method 0 2024-06-11T19:26:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4d587640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4d587640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4d587640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4e589640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4e589640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4e589640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4d587640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4d587640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4d587640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 10 monclient(hunting): handle_auth_done global_id 66465 payload 824 2024-06-11T19:26:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 1 -- 172.21.15.98:0/3657374873 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7a480a4920 msgr2=0x7f7a481ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7a480a4920 0x7f7a481ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 1 -- 172.21.15.98:0/3657374873 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a480a5240 msgr2=0x7f7a48122fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a480a5240 0x7f7a48122fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 10 monclient: _finish_hunting 0 2024-06-11T19:26:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 1 monclient: found mon.noname-a 2024-06-11T19:26:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:50.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:26:50.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 1 -- 172.21.15.98:0/3657374873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a48125710 con 0x7f7a480a2530 2024-06-11T19:26:50.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4d587640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7a480a4920 0x7f7a481ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:50.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4e589640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a480a5240 0x7f7a48122fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:50.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4dd88640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a480a2530 0x7f7a481cdcf0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400efc0 tx=0x7f7a4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 10 client.?.objecter ms_handle_connect 0x7f7a480a2530 2024-06-11T19:26:50.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 1 -- 172.21.15.98:0/3657374873 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a44018020 con 0x7f7a480a2530 2024-06-11T19:26:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:26:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 10 monclient: dump: 2024-06-11T19:26:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.392 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.393 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 10 monclient: _finish_auth 0 2024-06-11T19:26:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:26:50.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:50.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4ffec640 5 monclient: authenticate success, global_id 66465 2024-06-11T19:26:50.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4ffec640 10 monclient: _renew_subs 2024-06-11T19:26:50.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4ffec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:50.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4ffec640 1 -- 172.21.15.98:0/3657374873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a481268e0 con 0x7f7a480a2530 2024-06-11T19:26:50.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4ffec640 10 client.66465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:50.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4ffec640 10 monclient: _renew_subs 2024-06-11T19:26:50.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4ffec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:50.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4ffec640 1 -- 172.21.15.98:0/3657374873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a480a91a0 con 0x7f7a480a2530 2024-06-11T19:26:50.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a4ffec640 1 librados: init done 2024-06-11T19:26:50.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 1 -- 172.21.15.98:0/3657374873 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a4400cd50 con 0x7f7a480a2530 2024-06-11T19:26:50.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:50.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 1 -- 172.21.15.98:0/3657374873 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a44003e00 con 0x7f7a480a2530 2024-06-11T19:26:50.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:50.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3ffff640 4 set_mon_vals no callback set 2024-06-11T19:26:50.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.357+0000 7f7a3e7fc640 10 monclient: dump: 2024-06-11T19:26:50.396 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.396 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.396 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.396 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.396 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.397 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.397 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.397 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.397 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.397 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.397 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.397 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.398 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.358+0000 7f7a3e7fc640 1 -- 172.21.15.98:0/3657374873 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f7a44005d10 con 0x7f7a480a2530 2024-06-11T19:26:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.358+0000 7f7a4ffec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:50.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.358+0000 7f7a4ffec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.358+0000 7f7a4ffec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.358+0000 7f7a4ffec640 1 -- 172.21.15.98:0/3657374873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a30005080 con 0x7f7a480a2530 2024-06-11T19:26:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.358+0000 7f7a3e7fc640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7a2c03fdb0 0x7f7a2c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.358+0000 7f7a3e7fc640 1 -- 172.21.15.98:0/3657374873 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f7a44051ea0 con 0x7f7a480a2530 2024-06-11T19:26:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.358+0000 7f7a3e7fc640 10 client.66465.objecter ms_dispatch 0x7f7a48000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.358+0000 7f7a3e7fc640 3 client.66465.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:50.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.358+0000 7f7a3e7fc640 3 client.66465.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.359+0000 7f7a4d587640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7a2c03fdb0 0x7f7a2c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.359+0000 7f7a3e7fc640 20 client.66465.objecter dump_active .. 0 homeless 2024-06-11T19:26:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.359+0000 7f7a4d587640 10 monclient: get_auth_request con 0x7f7a2c03fdb0 auth_method 0 2024-06-11T19:26:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.359+0000 7f7a3e7fc640 10 client.66465.objecter ms_handle_connect 0x7f7a2c03fdb0 2024-06-11T19:26:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.359+0000 7f7a4d587640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7a2c03fdb0 0x7f7a2c042270 secure :-1 s=READY pgs=4336 cs=0 l=1 rev1=1 crypto rx=0x7f7a38035000 tx=0x7f7a38030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.363+0000 7f7a3e7fc640 1 -- 172.21.15.98:0/3657374873 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7a44010040 con 0x7f7a480a2530 2024-06-11T19:26:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.363+0000 7f7a3e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.363+0000 7f7a3e7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:50.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.546+0000 7f7a4ffec640 1 -- 172.21.15.98:0/3657374873 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f7a30000bc0 con 0x7f7a2c03fdb0 2024-06-11T19:26:50.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.547+0000 7f7a3e7fc640 1 -- 172.21.15.98:0/3657374873 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7a30000bc0 con 0x7f7a2c03fdb0 2024-06-11T19:26:50.546 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 0 to deep-scrub 2024-06-11T19:26:50.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.548+0000 7f7a4ffec640 10 librados: watch_flush enter 2024-06-11T19:26:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.548+0000 7f7a4ffec640 10 librados: watch_flush exit 2024-06-11T19:26:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.548+0000 7f7a4ffec640 20 client.66465.objecter shutdown clearing up homeless session... 2024-06-11T19:26:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.548+0000 7f7a4ffec640 10 client.66465.objecter successfully canceled tick 2024-06-11T19:26:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.548+0000 7f7a4ffec640 1 -- 172.21.15.98:0/3657374873 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7a2c03fdb0 msgr2=0x7f7a2c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.548+0000 7f7a4ffec640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7a2c03fdb0 0x7f7a2c042270 secure :-1 s=READY pgs=4336 cs=0 l=1 rev1=1 crypto rx=0x7f7a38035000 tx=0x7f7a38030000 comp rx=0 tx=0).stop 2024-06-11T19:26:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.548+0000 7f7a4ffec640 10 monclient: shutdown 2024-06-11T19:26:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.548+0000 7f7a4ffec640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:50.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.548+0000 7f7a4ffec640 1 -- 172.21.15.98:0/3657374873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a480a2530 msgr2=0x7f7a481cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.548+0000 7f7a4ffec640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a480a2530 0x7f7a481cdcf0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400efc0 tx=0x7f7a4400c490 comp rx=0 tx=0).stop 2024-06-11T19:26:50.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.549+0000 7f7a4ffec640 1 -- 172.21.15.98:0/3657374873 shutdown_connections 2024-06-11T19:26:50.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.549+0000 7f7a4ffec640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7a2c03fdb0 0x7f7a2c042270 unknown :-1 s=CLOSED pgs=4336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.549+0000 7f7a4ffec640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a480a5240 0x7f7a48122fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.549+0000 7f7a4ffec640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f7a480a4920 0x7f7a481ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.549+0000 7f7a4ffec640 1 --2- 172.21.15.98:0/3657374873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a480a2530 0x7f7a481cdcf0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.549+0000 7f7a4ffec640 1 -- 172.21.15.98:0/3657374873 >> 172.21.15.98:0/3657374873 conn(0x7f7a480ab850 msgr2=0x7f7a481be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:50.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.549+0000 7f7a4ffec640 1 -- 172.21.15.98:0/3657374873 shutdown_connections 2024-06-11T19:26:50.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.549+0000 7f7a4ffec640 1 -- 172.21.15.98:0/3657374873 wait complete. 2024-06-11T19:26:50.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.549+0000 7f7a4ffec640 1 librados: shutdown 2024-06-11T19:26:50.558 INFO:tasks.ceph:Scrubbing osd.1 2024-06-11T19:26:50.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-06-11T19:26:50.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.661+0000 7f436a449640 10 monclient: get_monmap_and_config 2024-06-11T19:26:50.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.661+0000 7f436a449640 10 monclient: build_initial_monmap 2024-06-11T19:26:50.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.661+0000 7f436a449640 10 monclient: monmap: 2024-06-11T19:26:50.661 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:50.661 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.661 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:50.662633+0000 2024-06-11T19:26:50.661 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:50.662633+0000 2024-06-11T19:26:50.661 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:50.661 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:50.663 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:50.663 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:50.663 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:50.663 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 1 Processor -- start 2024-06-11T19:26:50.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 1 -- start start 2024-06-11T19:26:50.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 10 monclient: init 2024-06-11T19:26:50.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:50.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 10 monclient: _add_conns ranks=[7,10,0,5,9,2,6,4,8,1,3] 2024-06-11T19:26:50.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4364135b60 0x7f4364135f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 10 monclient(hunting): picked mon.noname-i con 0x7f4364135b60 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:50.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f43640a5a10 0x7f43640a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 10 monclient(hunting): picked mon.noname-f con 0x7f43640a5a10 addr v2:172.21.15.117:3300/0 2024-06-11T19:26:50.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43640a6350 addr v1:172.21.15.98:6789/0 2024-06-11T19:26:50.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43640a8580 con 0x7f43640a6350 2024-06-11T19:26:50.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f43640a2c60 con 0x7f43640a5a10 2024-06-11T19:26:50.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f43640a2f40 con 0x7f4364135b60 2024-06-11T19:26:50.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:50.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.662+0000 7f436a449640 10 monclient(hunting): authenticate will time out at 1820.469955s 2024-06-11T19:26:50.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43689e6640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f43640a6350 0x7f436413fce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51416/0 (socket says 172.21.15.98:51416) 2024-06-11T19:26:50.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43689e6640 1 -- 172.21.15.98:0/974054270 learned_addr learned my addr 172.21.15.98:0/974054270 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:26:50.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 1 --2- 172.21.15.98:0/974054270 >> v2:172.21.15.117:3300/0 conn(0x7f43640a5a10 0x7f43640a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 10 monclient(hunting): get_auth_request con 0x7f43640a5a10 auth_method 0 2024-06-11T19:26:50.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4363fff640 1 --2- 172.21.15.98:0/974054270 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4364135b60 0x7f4364135f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4363fff640 10 monclient(hunting): get_auth_request con 0x7f4364135b60 auth_method 0 2024-06-11T19:26:50.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4363fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4363fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4363fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4363fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4363fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4363fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4362ffd640 1 -- 172.21.15.98:0/974054270 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 513644614 0 0) 0x7f43640a8580 con 0x7f43640a6350 2024-06-11T19:26:50.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4362ffd640 10 monclient(hunting): _negotiate 2024-06-11T19:26:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4362ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4362ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4362ffd640 10 monclient(hunting): my global_id is 66474 2024-06-11T19:26:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4362ffd640 1 -- 172.21.15.98:0/974054270 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f434c003520 con 0x7f43640a6350 2024-06-11T19:26:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 10 monclient(hunting): handle_auth_done global_id 124462 payload 274 2024-06-11T19:26:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 1 -- 172.21.15.98:0/974054270 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4364135b60 msgr2=0x7f4364135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 1 --2- 172.21.15.98:0/974054270 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4364135b60 0x7f4364135f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 1 -- 172.21.15.98:0/974054270 >> v1:172.21.15.98:6789/0 conn(0x7f43640a6350 legacy=0x7f436413fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 10 monclient: _finish_hunting 0 2024-06-11T19:26:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 1 monclient: found mon.noname-f 2024-06-11T19:26:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.117:3300/0 2024-06-11T19:26:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f43637fe640 1 -- 172.21.15.98:0/974054270 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43640a3220 con 0x7f43640a5a10 2024-06-11T19:26:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.663+0000 7f4363fff640 1 --2- 172.21.15.98:0/974054270 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4364135b60 0x7f4364135f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f43637fe640 1 --2- 172.21.15.98:0/974054270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43640a5a10 0x7f43640a5e10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f43580097a0 tx=0x7f435802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eba82a7d2c5e02fe server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 1 -- 172.21.15.98:0/974054270 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4358038070 con 0x7f43640a5a10 2024-06-11T19:26:50.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-06-11T19:26:50.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 10 monclient: dump: 2024-06-11T19:26:50.679 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.679 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.679 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.679 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.679 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.680 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.680 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.680 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.680 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.680 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.680 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.680 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.681 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.681 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.681 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.681 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 10 monclient: _finish_auth 0 2024-06-11T19:26:50.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:26:50.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:50.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f436a449640 5 monclient: authenticate success, global_id 124462 2024-06-11T19:26:50.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 1 -- 172.21.15.98:0/974054270 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4358031970 con 0x7f43640a5a10 2024-06-11T19:26:50.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:50.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f436a449640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:50.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 1 -- 172.21.15.98:0/974054270 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4358032390 con 0x7f43640a5a10 2024-06-11T19:26:50.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f436a449640 4 set_mon_vals no callback set 2024-06-11T19:26:50.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-06-11T19:26:50.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.664+0000 7f4362ffd640 10 monclient: dump: 2024-06-11T19:26:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.683 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.685 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 10 monclient: shutdown 2024-06-11T19:26:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 1 -- 172.21.15.98:0/974054270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43640a5a10 msgr2=0x7f43640a5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 1 --2- 172.21.15.98:0/974054270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43640a5a10 0x7f43640a5e10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f43580097a0 tx=0x7f435802b7a0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 1 -- 172.21.15.98:0/974054270 shutdown_connections 2024-06-11T19:26:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 1 --2- 172.21.15.98:0/974054270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43640a5a10 0x7f43640a5e10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 1 --2- 172.21.15.98:0/974054270 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4364135b60 0x7f4364135f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 1 -- 172.21.15.98:0/974054270 >> 172.21.15.98:0/974054270 conn(0x7f43640aba20 msgr2=0x7f43640a7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 1 -- 172.21.15.98:0/974054270 shutdown_connections 2024-06-11T19:26:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 1 -- 172.21.15.98:0/974054270 wait complete. 2024-06-11T19:26:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 10 monclient: build_initial_monmap 2024-06-11T19:26:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 10 monclient: monmap: 2024-06-11T19:26:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:50.687 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:50.688 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 1 librados: starting msgr at 2024-06-11T19:26:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 1 librados: starting objecter 2024-06-11T19:26:50.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.665+0000 7f436a449640 1 Processor -- start 2024-06-11T19:26:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 1 -- start start 2024-06-11T19:26:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 1 librados: setting wanted keys 2024-06-11T19:26:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 1 librados: calling monclient init 2024-06-11T19:26:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 10 monclient: init 2024-06-11T19:26:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 10 monclient: _add_conns ranks=[2,5,6,1,4,8,3,7,0] 2024-06-11T19:26:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43640a5a10 0x7f43641cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 10 monclient(hunting): picked mon.noname-b con 0x7f43640a5a10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:50.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f43640a6350 0x7f4364134990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 10 monclient(hunting): picked mon.noname-h con 0x7f43640a6350 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:50.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4364135b60 0x7f436409d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 10 monclient(hunting): picked mon.noname-d con 0x7f4364135b60 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:50.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f43640a67a0 con 0x7f43640a5a10 2024-06-11T19:26:50.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f4364050490 con 0x7f43640a6350 2024-06-11T19:26:50.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f43640a7290 con 0x7f4364135b60 2024-06-11T19:26:50.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:50.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f436a449640 10 monclient(hunting): authenticate will time out at 1820.473610s 2024-06-11T19:26:50.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f4363fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43640a5a10 0x7f43641cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f4363fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43640a5a10 0x7f43641cd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:43140/0 (socket says 172.21.15.98:43140) 2024-06-11T19:26:50.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f4363fff640 1 -- 172.21.15.98:0/4166812730 learned_addr learned my addr 172.21.15.98:0/4166812730 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:50.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f4363fff640 10 monclient(hunting): get_auth_request con 0x7f43640a5a10 auth_method 0 2024-06-11T19:26:50.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f4363fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f4363fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f4363fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43689e6640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4364135b60 0x7f436409d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43689e6640 10 monclient(hunting): get_auth_request con 0x7f4364135b60 auth_method 0 2024-06-11T19:26:50.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43689e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43689e6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43689e6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f4363fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f4363fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f4363fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43689e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43689e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43637fe640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f43640a6350 0x7f4364134990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43689e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43637fe640 10 monclient(hunting): get_auth_request con 0x7f43640a6350 auth_method 0 2024-06-11T19:26:50.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43637fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43637fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f43637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.666+0000 7f4363fff640 10 monclient(hunting): handle_auth_done global_id 85421 payload 824 2024-06-11T19:26:50.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4363fff640 1 -- 172.21.15.98:0/4166812730 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4364135b60 msgr2=0x7f436409d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4363fff640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4364135b60 0x7f436409d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4363fff640 1 -- 172.21.15.98:0/4166812730 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f43640a6350 msgr2=0x7f4364134990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4363fff640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f43640a6350 0x7f4364134990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4363fff640 10 monclient: _finish_hunting 0 2024-06-11T19:26:50.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4363fff640 1 monclient: found mon.noname-b 2024-06-11T19:26:50.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4363fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:50.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4363fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:26:50.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4363fff640 1 -- 172.21.15.98:0/4166812730 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4364122070 con 0x7f43640a5a10 2024-06-11T19:26:50.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f43689e6640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4364135b60 0x7f436409d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:50.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4363fff640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43640a5a10 0x7f43641cd5c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f435400e960 tx=0x7f435400ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 10 client.?.objecter ms_handle_connect 0x7f43640a5a10 2024-06-11T19:26:50.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:50.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 1 -- 172.21.15.98:0/4166812730 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4354017070 con 0x7f43640a5a10 2024-06-11T19:26:50.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:26:50.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 10 monclient: dump: 2024-06-11T19:26:50.697 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.697 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.697 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.697 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.697 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.697 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.697 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.697 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.698 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 10 monclient: _finish_auth 0 2024-06-11T19:26:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 10 monclient: _check_auth_tickets 2024-06-11T19:26:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 1 -- 172.21.15.98:0/4166812730 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f435400ce70 con 0x7f43640a5a10 2024-06-11T19:26:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f436a449640 5 monclient: authenticate success, global_id 85421 2024-06-11T19:26:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f436a449640 10 monclient: _renew_subs 2024-06-11T19:26:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f436a449640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f436a449640 1 -- 172.21.15.98:0/4166812730 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4364122350 con 0x7f43640a5a10 2024-06-11T19:26:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:50.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 1 -- 172.21.15.98:0/4166812730 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43540138e0 con 0x7f43640a5a10 2024-06-11T19:26:50.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:26:50.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f4360ff9640 10 monclient: dump: 2024-06-11T19:26:50.700 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.700 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.700 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.700 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.701 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.701 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.701 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.701 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.701 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.701 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.701 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.701 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.701 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.702 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.702 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.702 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f436a449640 10 client.85421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:50.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f436a449640 10 monclient: _renew_subs 2024-06-11T19:26:50.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f43627fc640 4 set_mon_vals no callback set 2024-06-11T19:26:50.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f436a449640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:50.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f436a449640 1 -- 172.21.15.98:0/4166812730 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f43640a0f70 con 0x7f43640a5a10 2024-06-11T19:26:50.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.667+0000 7f436a449640 1 librados: init done 2024-06-11T19:26:50.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f436a449640 10 client.85421.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:50.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f436a449640 15 client.85421.objecter _session_command_op_assign -1 1 2024-06-11T19:26:50.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f436a449640 20 client.85421.objecter _get_session osd=-1 returning homeless 2024-06-11T19:26:50.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f436a449640 10 client.85421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:50.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f436a449640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:50.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f436a449640 1 -- 172.21.15.98:0/4166812730 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4364073250 con 0x7f43640a5a10 2024-06-11T19:26:50.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 1 -- 172.21.15.98:0/4166812730 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f435401a070 con 0x7f43640a5a10 2024-06-11T19:26:50.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f435003fdb0 0x7f4350042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 1 -- 172.21.15.98:0/4166812730 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f435400f040 con 0x7f43640a5a10 2024-06-11T19:26:50.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 10 client.85421.objecter ms_dispatch 0x7f4364000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:50.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 3 client.85421.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:50.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 3 client.85421.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:50.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 10 client.85421.objecter checking command 1 2024-06-11T19:26:50.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] conn(0x7f43500475f0 0x7f4350049a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 20 client.85421.objecter _get_session s=0x7f4350046f20 osd=1 3 2024-06-11T19:26:50.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 20 client.85421.objecter put_session s=0x7f4350046f20 osd=1 3 2024-06-11T19:26:50.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 15 client.85421.objecter _session_command_op_remove -1 1 2024-06-11T19:26:50.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 20 client.85421.objecter _get_session s=0x7f4350046f20 osd=1 3 2024-06-11T19:26:50.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 20 client.85421.objecter get_session s=0x7f4350046f20 osd=1 3 2024-06-11T19:26:50.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 15 client.85421.objecter _session_command_op_assign 1 1 2024-06-11T19:26:50.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 20 client.85421.objecter put_session s=0x7f4350046f20 osd=1 4 2024-06-11T19:26:50.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 10 client.85421.objecter _send_command 1 2024-06-11T19:26:50.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 1 -- 172.21.15.98:0/4166812730 --> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f435004a160 con 0x7f43500475f0 2024-06-11T19:26:50.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 20 client.85421.objecter dump_active .. 0 homeless 2024-06-11T19:26:50.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 1 -- 172.21.15.98:0/4166812730 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_get_version_reply(handle=1 version=685) ==== 24+0+0 (secure 0 0 0) 0x7f4354058050 con 0x7f43640a5a10 2024-06-11T19:26:50.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 10 monclient: handle_get_version_reply finishing 1 version 685 2024-06-11T19:26:50.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f43637fe640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f435003fdb0 0x7f4350042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f43637fe640 10 monclient: get_auth_request con 0x7f435003fdb0 auth_method 0 2024-06-11T19:26:50.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f43689e6640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] conn(0x7f43500475f0 0x7f4350049a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f43689e6640 10 monclient: get_auth_request con 0x7f43500475f0 auth_method 0 2024-06-11T19:26:50.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f43689e6640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] conn(0x7f43500475f0 0x7f4350049a90 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.668+0000 7f4360ff9640 10 client.85421.objecter ms_handle_connect 0x7f43500475f0 2024-06-11T19:26:50.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.669+0000 7f4360ff9640 10 client.85421.objecter ms_handle_connect 0x7f435003fdb0 2024-06-11T19:26:50.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.669+0000 7f43637fe640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f435003fdb0 0x7f4350042270 secure :-1 s=READY pgs=4337 cs=0 l=1 rev1=1 crypto rx=0x7f4358035000 tx=0x7f4358008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.669+0000 7f4360ff9640 1 -- 172.21.15.98:0/4166812730 <== osd.1 v2:172.21.15.98:6800/3886067876 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f435004a160 con 0x7f43500475f0 2024-06-11T19:26:50.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.669+0000 7f4360ff9640 10 client.85421.objecter ms_dispatch 0x7f4364000b90 command_reply(tid 1: 0 ) 2024-06-11T19:26:50.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.669+0000 7f4360ff9640 10 client.85421.objecter _finish_command 1 = system:0 2024-06-11T19:26:50.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.669+0000 7f4360ff9640 20 client.85421.objecter put_session s=0x7f4350046f20 osd=1 4 2024-06-11T19:26:50.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.669+0000 7f4360ff9640 15 client.85421.objecter _session_command_op_remove 1 1 2024-06-11T19:26:50.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 10 client.85421.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-11T19:26:50.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 15 client.85421.objecter _session_command_op_assign -1 2 2024-06-11T19:26:50.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 20 client.85421.objecter _get_session s=0x7f4350046f20 osd=1 3 2024-06-11T19:26:50.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 20 client.85421.objecter put_session s=0x7f4350046f20 osd=1 3 2024-06-11T19:26:50.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 20 client.85421.objecter _get_session s=0x7f4350046f20 osd=1 3 2024-06-11T19:26:50.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 15 client.85421.objecter _session_command_op_remove -1 2 2024-06-11T19:26:50.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 20 client.85421.objecter get_session s=0x7f4350046f20 osd=1 3 2024-06-11T19:26:50.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 15 client.85421.objecter _session_command_op_assign 1 2 2024-06-11T19:26:50.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 20 client.85421.objecter put_session s=0x7f4350046f20 osd=1 4 2024-06-11T19:26:50.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 10 client.85421.objecter _send_command 2 2024-06-11T19:26:50.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 1 -- 172.21.15.98:0/4166812730 --> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f43640a53a0 con 0x7f43500475f0 2024-06-11T19:26:50.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f4360ff9640 1 -- 172.21.15.98:0/4166812730 <== osd.1 v2:172.21.15.98:6800/3886067876 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f43640a53a0 con 0x7f43500475f0 2024-06-11T19:26:50.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f4360ff9640 10 client.85421.objecter ms_dispatch 0x7f4364000b90 command_reply(tid 2: 0 ) 2024-06-11T19:26:50.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f4360ff9640 10 client.85421.objecter _finish_command 2 = system:0 2024-06-11T19:26:50.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f4360ff9640 20 client.85421.objecter put_session s=0x7f4350046f20 osd=1 4 2024-06-11T19:26:50.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f4360ff9640 15 client.85421.objecter _session_command_op_remove 1 2 2024-06-11T19:26:50.711 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:50.711 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-11T19:26:50.711 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 10 librados: watch_flush enter 2024-06-11T19:26:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 10 librados: watch_flush exit 2024-06-11T19:26:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 10 client.85421.objecter close_session for osd.1 2024-06-11T19:26:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 1 -- 172.21.15.98:0/4166812730 >> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] conn(0x7f43500475f0 msgr2=0x7f4350049a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.98:6800/3886067876,v1:172.21.15.98:6801/3886067876] conn(0x7f43500475f0 0x7f4350049a90 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 20 client.85421.objecter put_session s=0x7f4350046f20 osd=1 1 2024-06-11T19:26:50.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 20 client.85421.objecter shutdown clearing up homeless session... 2024-06-11T19:26:50.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.682+0000 7f436a449640 10 client.85421.objecter successfully canceled tick 2024-06-11T19:26:50.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.683+0000 7f436a449640 1 -- 172.21.15.98:0/4166812730 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f435003fdb0 msgr2=0x7f4350042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.683+0000 7f436a449640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f435003fdb0 0x7f4350042270 secure :-1 s=READY pgs=4337 cs=0 l=1 rev1=1 crypto rx=0x7f4358035000 tx=0x7f4358008030 comp rx=0 tx=0).stop 2024-06-11T19:26:50.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.683+0000 7f436a449640 10 monclient: shutdown 2024-06-11T19:26:50.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.683+0000 7f436a449640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:50.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.683+0000 7f436a449640 1 -- 172.21.15.98:0/4166812730 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43640a5a10 msgr2=0x7f43641cd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.683+0000 7f436a449640 1 --2- 172.21.15.98:0/4166812730 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f43640a5a10 0x7f43641cd5c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f435400e960 tx=0x7f435400ee20 comp rx=0 tx=0).stop 2024-06-11T19:26:50.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.683+0000 7f43689e6640 1 -- 172.21.15.98:0/4166812730 reap_dead start 2024-06-11T19:26:50.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.683+0000 7f436a449640 1 -- 172.21.15.98:0/4166812730 shutdown_connections 2024-06-11T19:26:50.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.683+0000 7f436a449640 1 -- 172.21.15.98:0/4166812730 >> 172.21.15.98:0/4166812730 conn(0x7f43640aba20 msgr2=0x7f43640a9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:50.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.683+0000 7f436a449640 1 -- 172.21.15.98:0/4166812730 shutdown_connections 2024-06-11T19:26:50.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.683+0000 7f436a449640 1 -- 172.21.15.98:0/4166812730 wait complete. 2024-06-11T19:26:50.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.683+0000 7f436a449640 1 librados: shutdown 2024-06-11T19:26:50.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-06-11T19:26:50.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.815+0000 7ff7588da640 10 monclient: get_monmap_and_config 2024-06-11T19:26:50.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.815+0000 7ff7588da640 10 monclient: build_initial_monmap 2024-06-11T19:26:50.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.815+0000 7ff7588da640 10 monclient: monmap: 2024-06-11T19:26:50.815 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:50.815 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.815 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:50.816269+0000 2024-06-11T19:26:50.815 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:50.816269+0000 2024-06-11T19:26:50.815 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:50.815 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.815 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:50.816 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:50.816 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:50.816 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:50.816 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:50.816 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:50.816 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:50.816 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:50.816 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:50.817 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:50.817 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:50.817 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.815+0000 7ff7588da640 1 Processor -- start 2024-06-11T19:26:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.815+0000 7ff7588da640 1 -- start start 2024-06-11T19:26:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.815+0000 7ff7588da640 10 monclient: init 2024-06-11T19:26:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 10 monclient: _add_conns ranks=[2,7,4,5,8,10,3,1,9,6,0] 2024-06-11T19:26:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7540a4920 0x7ff7540a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 10 monclient(hunting): picked mon.noname-b con 0x7ff7540a4920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7540a5240 0x7ff7540a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 10 monclient(hunting): picked mon.noname-i con 0x7ff7540a5240 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7540a2530 0x7ff7540a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 10 monclient(hunting): picked mon.noname-c con 0x7ff7540a2530 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff7540a7140 con 0x7ff7540a4920 2024-06-11T19:26:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff7540a2b50 con 0x7ff7540a2530 2024-06-11T19:26:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff7540a2e30 con 0x7ff7540a5240 2024-06-11T19:26:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff7588da640 10 monclient(hunting): authenticate will time out at 1820.623571s 2024-06-11T19:26:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7540a2530 0x7ff7540a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff75259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7540a4920 0x7ff7540a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7540a2530 0x7ff7540a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:37282/0 (socket says 172.21.15.98:37282) 2024-06-11T19:26:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 1 -- 172.21.15.98:0/3955830335 learned_addr learned my addr 172.21.15.98:0/3955830335 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff75259c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7540a4920 0x7ff7540a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:43148/0 (socket says 172.21.15.98:43148) 2024-06-11T19:26:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 10 monclient(hunting): get_auth_request con 0x7ff7540a2530 auth_method 0 2024-06-11T19:26:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff75259c640 10 monclient(hunting): get_auth_request con 0x7ff7540a4920 auth_method 0 2024-06-11T19:26:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff75259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff75259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff75259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff751d9b640 1 --2- 172.21.15.98:0/3955830335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7540a5240 0x7ff7540a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff75259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff75259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff75259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff751d9b640 10 monclient(hunting): get_auth_request con 0x7ff7540a5240 auth_method 0 2024-06-11T19:26:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff751d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff751d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff751d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff751d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff751d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff751d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 10 monclient(hunting): handle_auth_done global_id 134212 payload 274 2024-06-11T19:26:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 1 -- 172.21.15.98:0/3955830335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7540a5240 msgr2=0x7ff7540a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 1 --2- 172.21.15.98:0/3955830335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7540a5240 0x7ff7540a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 1 -- 172.21.15.98:0/3955830335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7540a4920 msgr2=0x7ff7540a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.816+0000 7ff752d9d640 1 --2- 172.21.15.98:0/3955830335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7540a4920 0x7ff7540a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff752d9d640 10 monclient: _finish_hunting 0 2024-06-11T19:26:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff752d9d640 1 monclient: found mon.noname-c 2024-06-11T19:26:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff752d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff752d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:26:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff752d9d640 1 -- 172.21.15.98:0/3955830335 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7540a3110 con 0x7ff7540a2530 2024-06-11T19:26:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75259c640 1 --2- 172.21.15.98:0/3955830335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7540a4920 0x7ff7540a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff752d9d640 1 --2- 172.21.15.98:0/3955830335 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7540a2530 0x7ff7540a5ee0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009e80 tx=0x7ff74c02bd70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=6b54a06b548f17ea server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 1 -- 172.21.15.98:0/3955830335 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff74c02cb80 con 0x7ff7540a2530 2024-06-11T19:26:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:26:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 10 monclient: dump: 2024-06-11T19:26:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.835 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 10 monclient: _finish_auth 0 2024-06-11T19:26:50.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 10 monclient: _check_auth_tickets 2024-06-11T19:26:50.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:50.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff7588da640 5 monclient: authenticate success, global_id 134212 2024-06-11T19:26:50.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 1 -- 172.21.15.98:0/3955830335 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff74c02cd40 con 0x7ff7540a2530 2024-06-11T19:26:50.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff7588da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:50.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff7588da640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 1 -- 172.21.15.98:0/3955830335 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff74c034860 con 0x7ff7540a2530 2024-06-11T19:26:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff7588da640 4 set_mon_vals no callback set 2024-06-11T19:26:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:26:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.817+0000 7ff75159a640 10 monclient: dump: 2024-06-11T19:26:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.838 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.818+0000 7ff7588da640 10 monclient: shutdown 2024-06-11T19:26:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.818+0000 7ff7588da640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.818+0000 7ff7588da640 1 -- 172.21.15.98:0/3955830335 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7540a2530 msgr2=0x7ff7540a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.818+0000 7ff7588da640 1 --2- 172.21.15.98:0/3955830335 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7540a2530 0x7ff7540a5ee0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009e80 tx=0x7ff74c02bd70 comp rx=0 tx=0).stop 2024-06-11T19:26:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.818+0000 7ff7588da640 1 -- 172.21.15.98:0/3955830335 shutdown_connections 2024-06-11T19:26:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.818+0000 7ff7588da640 1 --2- 172.21.15.98:0/3955830335 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff7540a2530 0x7ff7540a5ee0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.818+0000 7ff7588da640 1 --2- 172.21.15.98:0/3955830335 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7540a5240 0x7ff7540a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.818+0000 7ff7588da640 1 --2- 172.21.15.98:0/3955830335 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff7540a4920 0x7ff7540a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.818+0000 7ff7588da640 1 -- 172.21.15.98:0/3955830335 >> 172.21.15.98:0/3955830335 conn(0x7ff7540ab850 msgr2=0x7ff7540a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.818+0000 7ff7588da640 1 -- 172.21.15.98:0/3955830335 shutdown_connections 2024-06-11T19:26:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.818+0000 7ff7588da640 1 -- 172.21.15.98:0/3955830335 wait complete. 2024-06-11T19:26:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 10 monclient: build_initial_monmap 2024-06-11T19:26:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 10 monclient: monmap: 2024-06-11T19:26:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:50.841 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 1 librados: starting msgr at 2024-06-11T19:26:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 1 librados: starting objecter 2024-06-11T19:26:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 1 Processor -- start 2024-06-11T19:26:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 1 -- start start 2024-06-11T19:26:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 1 librados: setting wanted keys 2024-06-11T19:26:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 1 librados: calling monclient init 2024-06-11T19:26:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 10 monclient: init 2024-06-11T19:26:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 10 monclient: _add_conns ranks=[7,8,1,4,0,6,3,2,5] 2024-06-11T19:26:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7540a2530 0x7ff7541cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 10 monclient(hunting): picked mon.noname-i con 0x7ff7540a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.819+0000 7ff7588da640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff7540a4920 0x7ff7541ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 10 monclient(hunting): picked mon.noname-e con 0x7ff7540a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7540a5240 0x7ff754121ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 10 monclient(hunting): picked mon.noname-f con 0x7ff7540a5240 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:26:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff7540a6da0 con 0x7ff7540a5240 2024-06-11T19:26:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff754138e50 con 0x7ff7540a2530 2024-06-11T19:26:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff7540a6420 con 0x7ff7540a4920 2024-06-11T19:26:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 10 monclient(hunting): authenticate will time out at 1820.627047s 2024-06-11T19:26:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff7540a4920 0x7ff7541ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff7540a4920 0x7ff7541ce0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54504/0 (socket says 172.21.15.98:54504) 2024-06-11T19:26:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 1 -- 172.21.15.98:0/2119712395 learned_addr learned my addr 172.21.15.98:0/2119712395 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 10 monclient(hunting): get_auth_request con 0x7ff7540a4920 auth_method 0 2024-06-11T19:26:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff75259c640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7540a2530 0x7ff7541cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff75259c640 10 monclient(hunting): get_auth_request con 0x7ff7540a2530 auth_method 0 2024-06-11T19:26:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff75259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff75259c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff75259c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff752d9d640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7540a5240 0x7ff754121ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff752d9d640 10 monclient(hunting): get_auth_request con 0x7ff7540a5240 auth_method 0 2024-06-11T19:26:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff752d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff752d9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff752d9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff75259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff75259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff75259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 10 monclient(hunting): handle_auth_done global_id 124487 payload 824 2024-06-11T19:26:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 1 -- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7540a2530 msgr2=0x7ff7541cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7540a2530 0x7ff7541cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 1 -- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7540a5240 msgr2=0x7ff754121ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7540a5240 0x7ff754121ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 10 monclient: _finish_hunting 0 2024-06-11T19:26:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 1 monclient: found mon.noname-e 2024-06-11T19:26:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:26:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 1 -- 172.21.15.98:0/2119712395 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7541258d0 con 0x7ff7540a4920 2024-06-11T19:26:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff752d9d640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7540a5240 0x7ff754121ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:26:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff751d9b640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff7540a4920 0x7ff7541ce0e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00e8a0 tx=0x7ff73c00ed60 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 10 client.?.objecter ms_handle_connect 0x7ff7540a4920 2024-06-11T19:26:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 1 -- 172.21.15.98:0/2119712395 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff73c017070 con 0x7ff7540a4920 2024-06-11T19:26:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:26:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 10 monclient: dump: 2024-06-11T19:26:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.851 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 10 monclient: _finish_auth 0 2024-06-11T19:26:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 10 monclient: _check_auth_tickets 2024-06-11T19:26:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 1 -- 172.21.15.98:0/2119712395 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff73c00ee90 con 0x7ff7540a4920 2024-06-11T19:26:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 5 monclient: authenticate success, global_id 124487 2024-06-11T19:26:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 10 monclient: _renew_subs 2024-06-11T19:26:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 1 -- 172.21.15.98:0/2119712395 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff754124690 con 0x7ff7540a4920 2024-06-11T19:26:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 1 -- 172.21.15.98:0/2119712395 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff73c00cdb0 con 0x7ff7540a4920 2024-06-11T19:26:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff74affd640 10 monclient: dump: 2024-06-11T19:26:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:50.854 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff750d99640 4 set_mon_vals no callback set 2024-06-11T19:26:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 10 client.124487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 10 monclient: _renew_subs 2024-06-11T19:26:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 1 -- 172.21.15.98:0/2119712395 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7540aa9a0 con 0x7ff7540a4920 2024-06-11T19:26:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.820+0000 7ff7588da640 1 librados: init done 2024-06-11T19:26:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.821+0000 7ff74affd640 1 -- 172.21.15.98:0/2119712395 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff73c01a070 con 0x7ff7540a4920 2024-06-11T19:26:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.821+0000 7ff74affd640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff73403fdb0 0x7ff734042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.821+0000 7ff74affd640 1 -- 172.21.15.98:0/2119712395 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7ff73c00f040 con 0x7ff7540a4920 2024-06-11T19:26:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.821+0000 7ff74affd640 10 client.124487.objecter ms_dispatch 0x7ff754000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.821+0000 7ff74affd640 3 client.124487.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.821+0000 7ff74affd640 3 client.124487.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.821+0000 7ff7588da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.821+0000 7ff7588da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.821+0000 7ff7588da640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.821+0000 7ff7588da640 1 -- 172.21.15.98:0/2119712395 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7540a6720 con 0x7ff7540a4920 2024-06-11T19:26:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.821+0000 7ff74affd640 20 client.124487.objecter dump_active .. 0 homeless 2024-06-11T19:26:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.822+0000 7ff75259c640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff73403fdb0 0x7ff734042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.822+0000 7ff75259c640 10 monclient: get_auth_request con 0x7ff73403fdb0 auth_method 0 2024-06-11T19:26:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.822+0000 7ff74affd640 10 client.124487.objecter ms_handle_connect 0x7ff73403fdb0 2024-06-11T19:26:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.822+0000 7ff75259c640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff73403fdb0 0x7ff734042270 secure :-1 s=READY pgs=4338 cs=0 l=1 rev1=1 crypto rx=0x7ff744009560 tx=0x7ff744009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.826+0000 7ff74affd640 1 -- 172.21.15.98:0/2119712395 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff73c01a360 con 0x7ff7540a4920 2024-06-11T19:26:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.826+0000 7ff74affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:50.826+0000 7ff74affd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.010+0000 7ff7588da640 1 -- 172.21.15.98:0/2119712395 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7ff7540008d0 con 0x7ff73403fdb0 2024-06-11T19:26:51.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.011+0000 7ff74affd640 1 -- 172.21.15.98:0/2119712395 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff7540008d0 con 0x7ff73403fdb0 2024-06-11T19:26:51.010 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 1 to deep-scrub 2024-06-11T19:26:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 10 librados: watch_flush enter 2024-06-11T19:26:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 10 librados: watch_flush exit 2024-06-11T19:26:51.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 20 client.124487.objecter shutdown clearing up homeless session... 2024-06-11T19:26:51.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 10 client.124487.objecter successfully canceled tick 2024-06-11T19:26:51.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 -- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff73403fdb0 msgr2=0x7ff734042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff73403fdb0 0x7ff734042270 secure :-1 s=READY pgs=4338 cs=0 l=1 rev1=1 crypto rx=0x7ff744009560 tx=0x7ff744009190 comp rx=0 tx=0).stop 2024-06-11T19:26:51.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 10 monclient: shutdown 2024-06-11T19:26:51.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:51.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 -- 172.21.15.98:0/2119712395 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff7540a4920 msgr2=0x7ff7541ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff7540a4920 0x7ff7541ce0e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00e8a0 tx=0x7ff73c00ed60 comp rx=0 tx=0).stop 2024-06-11T19:26:51.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 -- 172.21.15.98:0/2119712395 shutdown_connections 2024-06-11T19:26:51.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff73403fdb0 0x7ff734042270 unknown :-1 s=CLOSED pgs=4338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7540a5240 0x7ff754121ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff7540a4920 0x7ff7541ce0e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 --2- 172.21.15.98:0/2119712395 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff7540a2530 0x7ff7541cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 -- 172.21.15.98:0/2119712395 >> 172.21.15.98:0/2119712395 conn(0x7ff7540ab850 msgr2=0x7ff7540a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:51.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 -- 172.21.15.98:0/2119712395 shutdown_connections 2024-06-11T19:26:51.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 -- 172.21.15.98:0/2119712395 wait complete. 2024-06-11T19:26:51.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.013+0000 7ff7588da640 1 librados: shutdown 2024-06-11T19:26:51.023 INFO:tasks.ceph:Scrubbing osd.2 2024-06-11T19:26:51.023 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-06-11T19:26:51.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.126+0000 7ff605d8c640 10 monclient: get_monmap_and_config 2024-06-11T19:26:51.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.126+0000 7ff605d8c640 10 monclient: build_initial_monmap 2024-06-11T19:26:51.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.126+0000 7ff605d8c640 10 monclient: monmap: 2024-06-11T19:26:51.125 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:51.125 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.126 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:51.127170+0000 2024-06-11T19:26:51.126 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:51.127170+0000 2024-06-11T19:26:51.126 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:51.126 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.126 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:51.126 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:51.126 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:51.126 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:51.127 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:51.127 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:51.127 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:51.127 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:51.127 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:51.127 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:51.127 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:51.127 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.126+0000 7ff605d8c640 1 Processor -- start 2024-06-11T19:26:51.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.126+0000 7ff605d8c640 1 -- start start 2024-06-11T19:26:51.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 10 monclient: init 2024-06-11T19:26:51.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:51.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 10 monclient: _add_conns ranks=[10,7,1,9,5,4,8,0,2,6,3] 2024-06-11T19:26:51.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff60009d990 0x7ff60009dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 10 monclient(hunting): picked mon.noname-f con 0x7ff60009d990 addr v2:172.21.15.117:3300/0 2024-06-11T19:26:51.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff600138d60 0x7ff6000a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 10 monclient(hunting): picked mon.noname-i con 0x7ff600138d60 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:51.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff600139160 addr v1:172.21.15.117:6789/0 2024-06-11T19:26:51.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff600076c80 con 0x7ff600139160 2024-06-11T19:26:51.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7ff600073c90 con 0x7ff60009d990 2024-06-11T19:26:51.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7ff600076a30 con 0x7ff600138d60 2024-06-11T19:26:51.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:51.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff605d8c640 10 monclient(hunting): authenticate will time out at 1820.934481s 2024-06-11T19:26:51.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff600138d60 0x7ff6000a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5ff7fe640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff60009d990 0x7ff60009dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff600138d60 0x7ff6000a2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3303/0 says I am v2:172.21.15.98:38926/0 (socket says 172.21.15.98:38926) 2024-06-11T19:26:51.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 1 -- 172.21.15.98:0/4193378078 learned_addr learned my addr 172.21.15.98:0/4193378078 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 10 monclient(hunting): get_auth_request con 0x7ff600138d60 auth_method 0 2024-06-11T19:26:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5fffff640 1 --1- 172.21.15.98:0/4193378078 >> v1:172.21.15.117:6789/0 conn(0x7ff600139160 0x7ff6000a3380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.98:60714/0 (socket says 172.21.15.98:60714) 2024-06-11T19:26:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5ff7fe640 10 monclient(hunting): get_auth_request con 0x7ff60009d990 auth_method 0 2024-06-11T19:26:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5ff7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 10 monclient(hunting): handle_auth_done global_id 134224 payload 274 2024-06-11T19:26:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 1 -- 172.21.15.98:0/4193378078 >> v2:172.21.15.117:3300/0 conn(0x7ff60009d990 msgr2=0x7ff60009dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:51.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 1 --2- 172.21.15.98:0/4193378078 >> v2:172.21.15.117:3300/0 conn(0x7ff60009d990 0x7ff60009dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 1 -- 172.21.15.98:0/4193378078 >> v1:172.21.15.117:6789/0 conn(0x7ff600139160 legacy=0x7ff6000a3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 10 monclient: _finish_hunting 0 2024-06-11T19:26:51.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 1 monclient: found mon.noname-i 2024-06-11T19:26:51.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:51.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.117:3303/0 2024-06-11T19:26:51.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.127+0000 7ff5feffd640 1 -- 172.21.15.98:0/4193378078 --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60009b870 con 0x7ff600138d60 2024-06-11T19:26:51.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5feffd640 1 --2- 172.21.15.98:0/4193378078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff600138d60 0x7ff6000a2e40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009870 tx=0x7ff5f402b860 comp rx=0 tx=0).ready entity=mon.7 client_cookie=f58fa4517c9f0214 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 1 -- 172.21.15.98:0/4193378078 <== mon.7 v2:172.21.15.117:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff5f4031a30 con 0x7ff600138d60 2024-06-11T19:26:51.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-06-11T19:26:51.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 10 monclient: dump: 2024-06-11T19:26:51.141 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.141 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.141 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.141 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.141 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.141 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.141 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.142 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.142 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.142 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.142 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.142 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.142 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.142 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.142 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.143 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 10 monclient: _finish_auth 0 2024-06-11T19:26:51.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:26:51.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:51.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff605d8c640 5 monclient: authenticate success, global_id 134224 2024-06-11T19:26:51.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff605d8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:51.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 1 -- 172.21.15.98:0/4193378078 <== mon.7 v2:172.21.15.117:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5f4037070 con 0x7ff600138d60 2024-06-11T19:26:51.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:51.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff605d8c640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:51.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff605d8c640 4 set_mon_vals no callback set 2024-06-11T19:26:51.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 1 -- 172.21.15.98:0/4193378078 <== mon.7 v2:172.21.15.117:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff5f4031c30 con 0x7ff600138d60 2024-06-11T19:26:51.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-06-11T19:26:51.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.128+0000 7ff5fe7fc640 10 monclient: dump: 2024-06-11T19:26:51.144 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.144 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.144 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.145 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.145 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.145 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.145 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.145 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.145 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.145 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.145 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.145 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.146 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.146 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.146 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.146 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.129+0000 7ff605d8c640 10 monclient: shutdown 2024-06-11T19:26:51.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.129+0000 7ff605d8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:51.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.129+0000 7ff605d8c640 1 -- 172.21.15.98:0/4193378078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff600138d60 msgr2=0x7ff6000a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.129+0000 7ff605d8c640 1 --2- 172.21.15.98:0/4193378078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff600138d60 0x7ff6000a2e40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009870 tx=0x7ff5f402b860 comp rx=0 tx=0).stop 2024-06-11T19:26:51.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.129+0000 7ff605d8c640 1 -- 172.21.15.98:0/4193378078 shutdown_connections 2024-06-11T19:26:51.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.129+0000 7ff605d8c640 1 --2- 172.21.15.98:0/4193378078 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7ff600138d60 0x7ff6000a2e40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.129+0000 7ff605d8c640 1 --2- 172.21.15.98:0/4193378078 >> v2:172.21.15.117:3300/0 conn(0x7ff60009d990 0x7ff60009dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.129+0000 7ff605d8c640 1 -- 172.21.15.98:0/4193378078 >> 172.21.15.98:0/4193378078 conn(0x7ff6000a3f30 msgr2=0x7ff6000a4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:51.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.129+0000 7ff605d8c640 1 -- 172.21.15.98:0/4193378078 shutdown_connections 2024-06-11T19:26:51.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.129+0000 7ff605d8c640 1 -- 172.21.15.98:0/4193378078 wait complete. 2024-06-11T19:26:51.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.129+0000 7ff605d8c640 10 monclient: build_initial_monmap 2024-06-11T19:26:51.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.130+0000 7ff605d8c640 10 monclient: monmap: 2024-06-11T19:26:51.147 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:51.148 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:51.148 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:51.148 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:51.148 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:51.148 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:51.148 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:51.148 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:51.148 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:51.148 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:51.149 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:51.149 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:51.149 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:51.149 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:51.149 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:51.149 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.130+0000 7ff605d8c640 1 librados: starting msgr at 2024-06-11T19:26:51.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.130+0000 7ff605d8c640 1 librados: starting objecter 2024-06-11T19:26:51.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.130+0000 7ff605d8c640 1 Processor -- start 2024-06-11T19:26:51.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.130+0000 7ff605d8c640 1 -- start start 2024-06-11T19:26:51.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.130+0000 7ff605d8c640 1 librados: setting wanted keys 2024-06-11T19:26:51.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.130+0000 7ff605d8c640 1 librados: calling monclient init 2024-06-11T19:26:51.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.130+0000 7ff605d8c640 10 monclient: init 2024-06-11T19:26:51.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.130+0000 7ff605d8c640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:51.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.130+0000 7ff605d8c640 10 monclient: _add_conns ranks=[5,4,2,6,7,8,3,0,1] 2024-06-11T19:26:51.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff60009d990 0x7ff600133950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 monclient(hunting): picked mon.noname-h con 0x7ff60009d990 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:51.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff600138d60 0x7ff6001cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 monclient(hunting): picked mon.noname-c con 0x7ff600138d60 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:51.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff600139160 0x7ff6001ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 monclient(hunting): picked mon.noname-b con 0x7ff600139160 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:51.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff6000a3a90 con 0x7ff600139160 2024-06-11T19:26:51.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff600050490 con 0x7ff600138d60 2024-06-11T19:26:51.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7ff6000a3c10 con 0x7ff60009d990 2024-06-11T19:26:51.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:51.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 monclient(hunting): authenticate will time out at 1820.938077s 2024-06-11T19:26:51.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5fffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff600139160 0x7ff6001ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5fffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff600139160 0x7ff6001ce0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:43152/0 (socket says 172.21.15.98:43152) 2024-06-11T19:26:51.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5fffff640 1 -- 172.21.15.98:0/1482532283 learned_addr learned my addr 172.21.15.98:0/1482532283 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:51.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5fffff640 10 monclient(hunting): get_auth_request con 0x7ff600139160 auth_method 0 2024-06-11T19:26:51.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5fffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5fffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff600138d60 0x7ff6001cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 10 monclient(hunting): get_auth_request con 0x7ff600138d60 auth_method 0 2024-06-11T19:26:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5ff7fe640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff60009d990 0x7ff600133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5ff7fe640 10 monclient(hunting): get_auth_request con 0x7ff60009d990 auth_method 0 2024-06-11T19:26:51.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5ff7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 10 monclient(hunting): handle_auth_done global_id 134221 payload 824 2024-06-11T19:26:51.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 1 -- 172.21.15.98:0/1482532283 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff60009d990 msgr2=0x7ff600133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7ff60009d990 0x7ff600133950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 1 -- 172.21.15.98:0/1482532283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff600139160 msgr2=0x7ff6001ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff600139160 0x7ff6001ce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 10 monclient: _finish_hunting 0 2024-06-11T19:26:51.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 1 monclient: found mon.noname-c 2024-06-11T19:26:51.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:51.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:26:51.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 1 -- 172.21.15.98:0/1482532283 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6001336e0 con 0x7ff600138d60 2024-06-11T19:26:51.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5fffff640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff600139160 0x7ff6001ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:51.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5feffd640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff600138d60 0x7ff6001cdb80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4033000 tx=0x7ff5f40046c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 10 client.?.objecter ms_handle_connect 0x7ff600138d60 2024-06-11T19:26:51.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:51.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 1 -- 172.21.15.98:0/1482532283 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff5f4041070 con 0x7ff600138d60 2024-06-11T19:26:51.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:26:51.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 10 monclient: dump: 2024-06-11T19:26:51.157 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.157 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.157 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.157 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.157 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.157 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.158 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.158 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.158 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.158 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.158 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.158 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.158 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.158 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.159 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.159 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 10 monclient: _finish_auth 0 2024-06-11T19:26:51.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 10 monclient: _check_auth_tickets 2024-06-11T19:26:51.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:51.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 1 -- 172.21.15.98:0/1482532283 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5f4003610 con 0x7ff600138d60 2024-06-11T19:26:51.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 5 monclient: authenticate success, global_id 134221 2024-06-11T19:26:51.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:51.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 monclient: _renew_subs 2024-06-11T19:26:51.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 1 -- 172.21.15.98:0/1482532283 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff5f4003c30 con 0x7ff600138d60 2024-06-11T19:26:51.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:51.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 1 -- 172.21.15.98:0/1482532283 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff600134eb0 con 0x7ff600138d60 2024-06-11T19:26:51.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:26:51.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5e7fff640 10 monclient: dump: 2024-06-11T19:26:51.160 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.160 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.160 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.160 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.161 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.161 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.161 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.161 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.161 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.161 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.161 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.161 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.161 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.162 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.162 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.162 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff5fdffb640 4 set_mon_vals no callback set 2024-06-11T19:26:51.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 client.134221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:51.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 monclient: _renew_subs 2024-06-11T19:26:51.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:51.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 1 -- 172.21.15.98:0/1482532283 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ff600135470 con 0x7ff600138d60 2024-06-11T19:26:51.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.131+0000 7ff605d8c640 1 librados: init done 2024-06-11T19:26:51.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.132+0000 7ff5e7fff640 1 -- 172.21.15.98:0/1482532283 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7ff5f404c060 con 0x7ff600138d60 2024-06-11T19:26:51.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.132+0000 7ff5e7fff640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff5ec03fdb0 0x7ff5ec042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.132+0000 7ff605d8c640 10 client.134221.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:51.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.132+0000 7ff605d8c640 15 client.134221.objecter _session_command_op_assign -1 1 2024-06-11T19:26:51.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.132+0000 7ff605d8c640 20 client.134221.objecter _get_session osd=-1 returning homeless 2024-06-11T19:26:51.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.132+0000 7ff605d8c640 10 client.134221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:51.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.132+0000 7ff605d8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:51.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.132+0000 7ff605d8c640 1 -- 172.21.15.98:0/1482532283 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff600134040 con 0x7ff600138d60 2024-06-11T19:26:51.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.132+0000 7ff5e7fff640 1 -- 172.21.15.98:0/1482532283 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7ff5f40729f0 con 0x7ff600138d60 2024-06-11T19:26:51.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.132+0000 7ff5e7fff640 10 client.134221.objecter ms_dispatch 0x7ff600000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:51.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.132+0000 7ff5e7fff640 3 client.134221.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:51.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.132+0000 7ff5e7fff640 3 client.134221.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:51.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 10 client.134221.objecter checking command 1 2024-06-11T19:26:51.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x7ff5ec0475e0 0x7ff5ec049a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 20 client.134221.objecter _get_session s=0x7ff5ec046f10 osd=2 3 2024-06-11T19:26:51.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 20 client.134221.objecter put_session s=0x7ff5ec046f10 osd=2 3 2024-06-11T19:26:51.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 15 client.134221.objecter _session_command_op_remove -1 1 2024-06-11T19:26:51.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 20 client.134221.objecter _get_session s=0x7ff5ec046f10 osd=2 3 2024-06-11T19:26:51.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 20 client.134221.objecter get_session s=0x7ff5ec046f10 osd=2 3 2024-06-11T19:26:51.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 15 client.134221.objecter _session_command_op_assign 2 1 2024-06-11T19:26:51.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 20 client.134221.objecter put_session s=0x7ff5ec046f10 osd=2 4 2024-06-11T19:26:51.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 10 client.134221.objecter _send_command 1 2024-06-11T19:26:51.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 1 -- 172.21.15.98:0/1482532283 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5ec04a150 con 0x7ff5ec0475e0 2024-06-11T19:26:51.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 20 client.134221.objecter dump_active .. 0 homeless 2024-06-11T19:26:51.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 1 -- 172.21.15.98:0/1482532283 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_get_version_reply(handle=1 version=685) ==== 24+0+0 (secure 0 0 0) 0x7ff5f4031890 con 0x7ff600138d60 2024-06-11T19:26:51.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 10 monclient: handle_get_version_reply finishing 1 version 685 2024-06-11T19:26:51.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5ff7fe640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff5ec03fdb0 0x7ff5ec042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5ff7fe640 10 monclient: get_auth_request con 0x7ff5ec03fdb0 auth_method 0 2024-06-11T19:26:51.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5fffff640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x7ff5ec0475e0 0x7ff5ec049a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5fffff640 10 monclient: get_auth_request con 0x7ff5ec0475e0 auth_method 0 2024-06-11T19:26:51.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 10 client.134221.objecter ms_handle_connect 0x7ff5ec0475e0 2024-06-11T19:26:51.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5fffff640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x7ff5ec0475e0 0x7ff5ec049a80 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5e7fff640 10 client.134221.objecter ms_handle_connect 0x7ff5ec03fdb0 2024-06-11T19:26:51.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.133+0000 7ff5ff7fe640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff5ec03fdb0 0x7ff5ec042270 secure :-1 s=READY pgs=4339 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009ad0 tx=0x7ff5f0008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.134+0000 7ff5e7fff640 1 -- 172.21.15.98:0/1482532283 <== osd.2 v2:172.21.15.98:6808/3751501255 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5ec04a150 con 0x7ff5ec0475e0 2024-06-11T19:26:51.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.134+0000 7ff5e7fff640 10 client.134221.objecter ms_dispatch 0x7ff600000b90 command_reply(tid 1: 0 ) 2024-06-11T19:26:51.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.134+0000 7ff5e7fff640 10 client.134221.objecter _finish_command 1 = system:0 2024-06-11T19:26:51.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.134+0000 7ff5e7fff640 20 client.134221.objecter put_session s=0x7ff5ec046f10 osd=2 4 2024-06-11T19:26:51.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.134+0000 7ff5e7fff640 15 client.134221.objecter _session_command_op_remove 2 1 2024-06-11T19:26:51.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 10 client.134221.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-11T19:26:51.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 15 client.134221.objecter _session_command_op_assign -1 2 2024-06-11T19:26:51.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 20 client.134221.objecter _get_session s=0x7ff5ec046f10 osd=2 3 2024-06-11T19:26:51.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 20 client.134221.objecter put_session s=0x7ff5ec046f10 osd=2 3 2024-06-11T19:26:51.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 20 client.134221.objecter _get_session s=0x7ff5ec046f10 osd=2 3 2024-06-11T19:26:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 15 client.134221.objecter _session_command_op_remove -1 2 2024-06-11T19:26:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 20 client.134221.objecter get_session s=0x7ff5ec046f10 osd=2 3 2024-06-11T19:26:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 15 client.134221.objecter _session_command_op_assign 2 2 2024-06-11T19:26:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 20 client.134221.objecter put_session s=0x7ff5ec046f10 osd=2 4 2024-06-11T19:26:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 10 client.134221.objecter _send_command 2 2024-06-11T19:26:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 1 -- 172.21.15.98:0/1482532283 --> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff60009dd70 con 0x7ff5ec0475e0 2024-06-11T19:26:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff5e7fff640 1 -- 172.21.15.98:0/1482532283 <== osd.2 v2:172.21.15.98:6808/3751501255 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff60009dd70 con 0x7ff5ec0475e0 2024-06-11T19:26:51.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff5e7fff640 10 client.134221.objecter ms_dispatch 0x7ff600000b90 command_reply(tid 2: 0 ) 2024-06-11T19:26:51.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff5e7fff640 10 client.134221.objecter _finish_command 2 = system:0 2024-06-11T19:26:51.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff5e7fff640 20 client.134221.objecter put_session s=0x7ff5ec046f10 osd=2 4 2024-06-11T19:26:51.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff5e7fff640 15 client.134221.objecter _session_command_op_remove 2 2 2024-06-11T19:26:51.170 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:51.170 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-11T19:26:51.170 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:51.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 10 librados: watch_flush enter 2024-06-11T19:26:51.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 10 librados: watch_flush exit 2024-06-11T19:26:51.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 10 client.134221.objecter close_session for osd.2 2024-06-11T19:26:51.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 1 -- 172.21.15.98:0/1482532283 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x7ff5ec0475e0 msgr2=0x7ff5ec049a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.98:6808/3751501255,v1:172.21.15.98:6810/3751501255] conn(0x7ff5ec0475e0 0x7ff5ec049a80 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 20 client.134221.objecter put_session s=0x7ff5ec046f10 osd=2 1 2024-06-11T19:26:51.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 20 client.134221.objecter shutdown clearing up homeless session... 2024-06-11T19:26:51.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 10 client.134221.objecter successfully canceled tick 2024-06-11T19:26:51.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 1 -- 172.21.15.98:0/1482532283 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff5ec03fdb0 msgr2=0x7ff5ec042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7ff5ec03fdb0 0x7ff5ec042270 secure :-1 s=READY pgs=4339 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009ad0 tx=0x7ff5f0008040 comp rx=0 tx=0).stop 2024-06-11T19:26:51.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 10 monclient: shutdown 2024-06-11T19:26:51.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:51.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 1 -- 172.21.15.98:0/1482532283 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff600138d60 msgr2=0x7ff6001cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 1 --2- 172.21.15.98:0/1482532283 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff600138d60 0x7ff6001cdb80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4033000 tx=0x7ff5f40046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff5fffff640 1 -- 172.21.15.98:0/1482532283 reap_dead start 2024-06-11T19:26:51.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 1 -- 172.21.15.98:0/1482532283 shutdown_connections 2024-06-11T19:26:51.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 1 -- 172.21.15.98:0/1482532283 >> 172.21.15.98:0/1482532283 conn(0x7ff6000a3f30 msgr2=0x7ff6000a5330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:51.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 1 -- 172.21.15.98:0/1482532283 shutdown_connections 2024-06-11T19:26:51.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 1 -- 172.21.15.98:0/1482532283 wait complete. 2024-06-11T19:26:51.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.147+0000 7ff605d8c640 1 librados: shutdown 2024-06-11T19:26:51.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-06-11T19:26:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.274+0000 7f58a990b640 10 monclient: get_monmap_and_config 2024-06-11T19:26:51.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.274+0000 7f58a990b640 10 monclient: build_initial_monmap 2024-06-11T19:26:51.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.275+0000 7f58a990b640 10 monclient: monmap: 2024-06-11T19:26:51.274 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:51.274 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.274 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:51.275989+0000 2024-06-11T19:26:51.274 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:51.275989+0000 2024-06-11T19:26:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:51.276 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.275+0000 7f58a990b640 1 Processor -- start 2024-06-11T19:26:51.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.275+0000 7f58a990b640 1 -- start start 2024-06-11T19:26:51.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.275+0000 7f58a990b640 10 monclient: init 2024-06-11T19:26:51.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:51.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 10 monclient: _add_conns ranks=[7,8,6,0,5,10,2,9,1,3,4] 2024-06-11T19:26:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58a40a4920 0x7f58a40a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 10 monclient(hunting): picked mon.noname-i con 0x7f58a40a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f58a40a5240 0x7f58a40a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 10 monclient(hunting): picked mon.noname-e con 0x7f58a40a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 0x7f58a40a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 10 monclient(hunting): picked mon.noname-d con 0x7f58a40a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f58a40a7140 con 0x7f58a40a2530 2024-06-11T19:26:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f58a40a2b50 con 0x7f58a40a4920 2024-06-11T19:26:51.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f58a40a2e30 con 0x7f58a40a5240 2024-06-11T19:26:51.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:51.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a990b640 10 monclient(hunting): authenticate will time out at 1821.083254s 2024-06-11T19:26:51.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a27fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f58a40a5240 0x7f58a40a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 0x7f58a40a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a27fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f58a40a5240 0x7f58a40a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54508/0 (socket says 172.21.15.98:54508) 2024-06-11T19:26:51.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a27fc640 1 -- 172.21.15.98:0/4069431529 learned_addr learned my addr 172.21.15.98:0/4069431529 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:51.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a27fc640 10 monclient(hunting): get_auth_request con 0x7f58a40a5240 auth_method 0 2024-06-11T19:26:51.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 10 monclient(hunting): get_auth_request con 0x7f58a40a2530 auth_method 0 2024-06-11T19:26:51.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a2ffd640 1 --2- 172.21.15.98:0/4069431529 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58a40a4920 0x7f58a40a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a2ffd640 10 monclient(hunting): get_auth_request con 0x7f58a40a4920 auth_method 0 2024-06-11T19:26:51.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 10 monclient(hunting): handle_auth_done global_id 124485 payload 274 2024-06-11T19:26:51.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 1 -- 172.21.15.98:0/4069431529 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f58a40a5240 msgr2=0x7f58a40a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:51.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 1 --2- 172.21.15.98:0/4069431529 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f58a40a5240 0x7f58a40a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 1 -- 172.21.15.98:0/4069431529 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58a40a4920 msgr2=0x7f58a40a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:51.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 1 --2- 172.21.15.98:0/4069431529 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58a40a4920 0x7f58a40a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 10 monclient: _finish_hunting 0 2024-06-11T19:26:51.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 1 monclient: found mon.noname-d 2024-06-11T19:26:51.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:51.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:51.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a37fe640 1 -- 172.21.15.98:0/4069431529 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a40a3110 con 0x7f58a40a2530 2024-06-11T19:26:51.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.276+0000 7f58a27fc640 1 --2- 172.21.15.98:0/4069431529 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f58a40a5240 0x7f58a40a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:51.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a37fe640 1 --2- 172.21.15.98:0/4069431529 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 0x7f58a40a5ee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f589c009dc0 tx=0x7f589c02bcc0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=ce046a05cf8b1b8c server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 1 -- 172.21.15.98:0/4069431529 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f589c039070 con 0x7f58a40a2530 2024-06-11T19:26:51.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:51.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 10 monclient: dump: 2024-06-11T19:26:51.291 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.291 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.291 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.291 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.291 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.291 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.291 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.292 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.292 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.292 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.292 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.292 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.292 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.292 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.292 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.292 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 10 monclient: _finish_auth 0 2024-06-11T19:26:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:26:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a990b640 5 monclient: authenticate success, global_id 124485 2024-06-11T19:26:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 1 -- 172.21.15.98:0/4069431529 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f589c02cce0 con 0x7f58a40a2530 2024-06-11T19:26:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a990b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a990b640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 1 -- 172.21.15.98:0/4069431529 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f589c0327b0 con 0x7f58a40a2530 2024-06-11T19:26:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a990b640 4 set_mon_vals no callback set 2024-06-11T19:26:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.277+0000 7f58a1ffb640 10 monclient: dump: 2024-06-11T19:26:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.296 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 10 monclient: shutdown 2024-06-11T19:26:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 1 -- 172.21.15.98:0/4069431529 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 msgr2=0x7f58a40a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 1 --2- 172.21.15.98:0/4069431529 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 0x7f58a40a5ee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f589c009dc0 tx=0x7f589c02bcc0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 1 -- 172.21.15.98:0/4069431529 shutdown_connections 2024-06-11T19:26:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 1 --2- 172.21.15.98:0/4069431529 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f58a40a5240 0x7f58a40a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 1 --2- 172.21.15.98:0/4069431529 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 0x7f58a40a5ee0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 1 --2- 172.21.15.98:0/4069431529 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58a40a4920 0x7f58a40a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 1 -- 172.21.15.98:0/4069431529 >> 172.21.15.98:0/4069431529 conn(0x7f58a40ab850 msgr2=0x7f58a40a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 1 -- 172.21.15.98:0/4069431529 shutdown_connections 2024-06-11T19:26:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 1 -- 172.21.15.98:0/4069431529 wait complete. 2024-06-11T19:26:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 10 monclient: build_initial_monmap 2024-06-11T19:26:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 10 monclient: monmap: 2024-06-11T19:26:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:51.298 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:51.298 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:51.298 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:51.298 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:51.298 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:51.298 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:51.298 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:51.298 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:51.299 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:51.299 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:51.299 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:51.299 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:51.299 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:51.299 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:51.299 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 1 librados: starting msgr at 2024-06-11T19:26:51.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.278+0000 7f58a990b640 1 librados: starting objecter 2024-06-11T19:26:51.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 1 Processor -- start 2024-06-11T19:26:51.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 1 -- start start 2024-06-11T19:26:51.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 1 librados: setting wanted keys 2024-06-11T19:26:51.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 1 librados: calling monclient init 2024-06-11T19:26:51.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 10 monclient: init 2024-06-11T19:26:51.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:51.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 10 monclient: _add_conns ranks=[6,7,3,1,2,5,4,8,0] 2024-06-11T19:26:51.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 0x7f58a41cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 10 monclient(hunting): picked mon.noname-d con 0x7f58a40a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:51.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58a40a4920 0x7f58a41ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 10 monclient(hunting): picked mon.noname-i con 0x7f58a40a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:51.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58a40a5240 0x7f58a4121f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 10 monclient(hunting): picked mon.noname-g con 0x7f58a40a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:51.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f58a40a6da0 con 0x7f58a40a5240 2024-06-11T19:26:51.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f58a4138e50 con 0x7f58a40a2530 2024-06-11T19:26:51.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f58a40a6420 con 0x7f58a40a4920 2024-06-11T19:26:51.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:51.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a990b640 10 monclient(hunting): authenticate will time out at 1821.086795s 2024-06-11T19:26:51.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a2ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 0x7f58a41cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a2ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 0x7f58a41cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58934/0 (socket says 172.21.15.98:58934) 2024-06-11T19:26:51.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a2ffd640 1 -- 172.21.15.98:0/1188434283 learned_addr learned my addr 172.21.15.98:0/1188434283 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:51.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a2ffd640 10 monclient(hunting): get_auth_request con 0x7f58a40a2530 auth_method 0 2024-06-11T19:26:51.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.279+0000 7f58a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a27fc640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58a40a4920 0x7f58a41ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a37fe640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58a40a5240 0x7f58a4121f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a27fc640 10 monclient(hunting): get_auth_request con 0x7f58a40a4920 auth_method 0 2024-06-11T19:26:51.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a27fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a37fe640 10 monclient(hunting): get_auth_request con 0x7f58a40a5240 auth_method 0 2024-06-11T19:26:51.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a37fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a2ffd640 10 monclient(hunting): handle_auth_done global_id 124494 payload 824 2024-06-11T19:26:51.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a2ffd640 1 -- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58a40a4920 msgr2=0x7f58a41ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a2ffd640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58a40a4920 0x7f58a41ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a2ffd640 1 -- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58a40a5240 msgr2=0x7f58a4121f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a2ffd640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58a40a5240 0x7f58a4121f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a2ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:26:51.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a2ffd640 1 monclient: found mon.noname-d 2024-06-11T19:26:51.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:51.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a2ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a2ffd640 1 -- 172.21.15.98:0/1188434283 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a4124740 con 0x7f58a40a2530 2024-06-11T19:26:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a2ffd640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 0x7f58a41cdad0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f589400d800 tx=0x7f589400dcc0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 10 client.?.objecter ms_handle_connect 0x7f58a40a2530 2024-06-11T19:26:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 1 -- 172.21.15.98:0/1188434283 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f589400bb30 con 0x7f58a40a2530 2024-06-11T19:26:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 10 monclient: dump: 2024-06-11T19:26:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.308 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.308 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.308 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.308 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.308 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.308 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.308 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 10 monclient: _finish_auth 0 2024-06-11T19:26:51.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:26:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 1 -- 172.21.15.98:0/1188434283 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5894019070 con 0x7f58a40a2530 2024-06-11T19:26:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a990b640 5 monclient: authenticate success, global_id 124494 2024-06-11T19:26:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 1 -- 172.21.15.98:0/1188434283 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f589400bd10 con 0x7f58a40a2530 2024-06-11T19:26:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a990b640 10 monclient: _renew_subs 2024-06-11T19:26:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a990b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a990b640 1 -- 172.21.15.98:0/1188434283 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a4125ed0 con 0x7f58a40a2530 2024-06-11T19:26:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a17fa640 4 set_mon_vals no callback set 2024-06-11T19:26:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f588bfff640 10 monclient: dump: 2024-06-11T19:26:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.312 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a990b640 10 client.124494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a990b640 10 monclient: _renew_subs 2024-06-11T19:26:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a990b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a990b640 1 -- 172.21.15.98:0/1188434283 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a41387f0 con 0x7f58a40a2530 2024-06-11T19:26:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.280+0000 7f58a990b640 1 librados: init done 2024-06-11T19:26:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f588bfff640 1 -- 172.21.15.98:0/1188434283 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f589400b6c0 con 0x7f58a40a2530 2024-06-11T19:26:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f58a990b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f58a990b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f58a990b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f58a990b640 1 -- 172.21.15.98:0/1188434283 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a40aac70 con 0x7f58a40a2530 2024-06-11T19:26:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f588bfff640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f589003fde0 0x7f58900422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f588bfff640 1 -- 172.21.15.98:0/1188434283 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f589400f040 con 0x7f58a40a2530 2024-06-11T19:26:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f588bfff640 10 client.124494.objecter ms_dispatch 0x7f58a4000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f588bfff640 3 client.124494.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:51.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f588bfff640 3 client.124494.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f588bfff640 20 client.124494.objecter dump_active .. 0 homeless 2024-06-11T19:26:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f58a27fc640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f589003fde0 0x7f58900422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.281+0000 7f58a27fc640 10 monclient: get_auth_request con 0x7f589003fde0 auth_method 0 2024-06-11T19:26:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.282+0000 7f588bfff640 10 client.124494.objecter ms_handle_connect 0x7f589003fde0 2024-06-11T19:26:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.282+0000 7f58a27fc640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f589003fde0 0x7f58900422a0 secure :-1 s=READY pgs=4340 cs=0 l=1 rev1=1 crypto rx=0x7f58a4123670 tx=0x7f5898008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.286+0000 7f588bfff640 1 -- 172.21.15.98:0/1188434283 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5894022020 con 0x7f58a40a2530 2024-06-11T19:26:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.286+0000 7f588bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.286+0000 7f588bfff640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.470+0000 7f58a990b640 1 -- 172.21.15.98:0/1188434283 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f58a40a66f0 con 0x7f589003fde0 2024-06-11T19:26:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.470+0000 7f588bfff640 1 -- 172.21.15.98:0/1188434283 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f58a40a66f0 con 0x7f589003fde0 2024-06-11T19:26:51.469 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 2 to deep-scrub 2024-06-11T19:26:51.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.472+0000 7f58a990b640 10 librados: watch_flush enter 2024-06-11T19:26:51.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.472+0000 7f58a990b640 10 librados: watch_flush exit 2024-06-11T19:26:51.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.472+0000 7f58a990b640 20 client.124494.objecter shutdown clearing up homeless session... 2024-06-11T19:26:51.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.472+0000 7f58a990b640 10 client.124494.objecter successfully canceled tick 2024-06-11T19:26:51.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 -- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f589003fde0 msgr2=0x7f58900422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f589003fde0 0x7f58900422a0 secure :-1 s=READY pgs=4340 cs=0 l=1 rev1=1 crypto rx=0x7f58a4123670 tx=0x7f5898008040 comp rx=0 tx=0).stop 2024-06-11T19:26:51.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 10 monclient: shutdown 2024-06-11T19:26:51.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:51.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 -- 172.21.15.98:0/1188434283 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 msgr2=0x7f58a41cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 0x7f58a41cdad0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f589400d800 tx=0x7f589400dcc0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 -- 172.21.15.98:0/1188434283 shutdown_connections 2024-06-11T19:26:51.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f589003fde0 0x7f58900422a0 unknown :-1 s=CLOSED pgs=4340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58a40a5240 0x7f58a4121f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f58a40a4920 0x7f58a41ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 --2- 172.21.15.98:0/1188434283 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f58a40a2530 0x7f58a41cdad0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 -- 172.21.15.98:0/1188434283 >> 172.21.15.98:0/1188434283 conn(0x7f58a40ab850 msgr2=0x7f58a40a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 -- 172.21.15.98:0/1188434283 shutdown_connections 2024-06-11T19:26:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 -- 172.21.15.98:0/1188434283 wait complete. 2024-06-11T19:26:51.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.473+0000 7f58a990b640 1 librados: shutdown 2024-06-11T19:26:51.482 INFO:tasks.ceph:Scrubbing osd.3 2024-06-11T19:26:51.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-06-11T19:26:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.585+0000 7f4e108ff640 10 monclient: get_monmap_and_config 2024-06-11T19:26:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.585+0000 7f4e108ff640 10 monclient: build_initial_monmap 2024-06-11T19:26:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.585+0000 7f4e108ff640 10 monclient: monmap: 2024-06-11T19:26:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:51.586675+0000 2024-06-11T19:26:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:51.586675+0000 2024-06-11T19:26:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:51.587 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:51.587 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 1 Processor -- start 2024-06-11T19:26:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 1 -- start start 2024-06-11T19:26:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 10 monclient: init 2024-06-11T19:26:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 10 monclient: _add_conns ranks=[6,3,7,1,8,9,4,2,0,10,5] 2024-06-11T19:26:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 0x7f4e0c135f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 10 monclient(hunting): picked mon.noname-d con 0x7f4e0c135b60 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e0c0a5a10 0x7f4e0c0a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 10 monclient(hunting): picked mon.noname-g con 0x7f4e0c0a5a10 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e0c0a6350 0x7f4e0c13fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 10 monclient(hunting): picked mon.noname-i con 0x7f4e0c0a6350 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f4e0c0a8580 con 0x7f4e0c0a5a10 2024-06-11T19:26:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4e0c0a2c60 con 0x7f4e0c135b60 2024-06-11T19:26:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f4e0c0a2f40 con 0x7f4e0c0a6350 2024-06-11T19:26:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.586+0000 7f4e108ff640 10 monclient(hunting): authenticate will time out at 1821.393924s 2024-06-11T19:26:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 0x7f4e0c135f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 0x7f4e0c135f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58944/0 (socket says 172.21.15.98:58944) 2024-06-11T19:26:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 1 -- 172.21.15.98:0/364676209 learned_addr learned my addr 172.21.15.98:0/364676209 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 10 monclient(hunting): get_auth_request con 0x7f4e0c135b60 auth_method 0 2024-06-11T19:26:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e09d9b640 1 --2- 172.21.15.98:0/364676209 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e0c0a5a10 0x7f4e0c0a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0ad9d640 1 --2- 172.21.15.98:0/364676209 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e0c0a6350 0x7f4e0c13fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0ad9d640 10 monclient(hunting): get_auth_request con 0x7f4e0c0a6350 auth_method 0 2024-06-11T19:26:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e09d9b640 10 monclient(hunting): get_auth_request con 0x7f4e0c0a5a10 auth_method 0 2024-06-11T19:26:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e09d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 10 monclient(hunting): handle_auth_done global_id 124503 payload 274 2024-06-11T19:26:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 1 -- 172.21.15.98:0/364676209 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e0c0a6350 msgr2=0x7f4e0c13fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 1 --2- 172.21.15.98:0/364676209 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e0c0a6350 0x7f4e0c13fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 1 -- 172.21.15.98:0/364676209 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e0c0a5a10 msgr2=0x7f4e0c0a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:51.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 1 --2- 172.21.15.98:0/364676209 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e0c0a5a10 0x7f4e0c0a5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 10 monclient: _finish_hunting 0 2024-06-11T19:26:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 1 monclient: found mon.noname-d 2024-06-11T19:26:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0a59c640 1 -- 172.21.15.98:0/364676209 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0c0a3220 con 0x7f4e0c135b60 2024-06-11T19:26:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e09d9b640 1 --2- 172.21.15.98:0/364676209 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e0c0a5a10 0x7f4e0c0a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:26:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.587+0000 7f4e0ad9d640 1 --2- 172.21.15.98:0/364676209 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e0c0a6350 0x7f4e0c13fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:26:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0a59c640 1 --2- 172.21.15.98:0/364676209 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 0x7f4e0c135f40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0098a0 tx=0x7f4dfc02b7a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=de84669712c8a308 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 1 -- 172.21.15.98:0/364676209 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4dfc038070 con 0x7f4e0c135b60 2024-06-11T19:26:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 10 monclient: dump: 2024-06-11T19:26:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.602 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.603 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.604 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 10 monclient: _finish_auth 0 2024-06-11T19:26:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 10 monclient: _check_auth_tickets 2024-06-11T19:26:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 5 monclient: authenticate success, global_id 124503 2024-06-11T19:26:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 1 -- 172.21.15.98:0/364676209 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dfc031970 con 0x7f4e0c135b60 2024-06-11T19:26:51.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:51.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 1 -- 172.21.15.98:0/364676209 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4dfc031ba0 con 0x7f4e0c135b60 2024-06-11T19:26:51.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:51.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 4 set_mon_vals no callback set 2024-06-11T19:26:51.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:51.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e0959a640 10 monclient: dump: 2024-06-11T19:26:51.605 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.605 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.606 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.606 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.606 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.606 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.606 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.606 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.606 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.606 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.607 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 10 monclient: shutdown 2024-06-11T19:26:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:51.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 1 -- 172.21.15.98:0/364676209 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 msgr2=0x7f4e0c135f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 1 --2- 172.21.15.98:0/364676209 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 0x7f4e0c135f40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0098a0 tx=0x7f4dfc02b7a0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 1 -- 172.21.15.98:0/364676209 shutdown_connections 2024-06-11T19:26:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 1 --2- 172.21.15.98:0/364676209 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e0c0a5a10 0x7f4e0c0a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 1 --2- 172.21.15.98:0/364676209 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f4e0c0a6350 0x7f4e0c13fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 1 --2- 172.21.15.98:0/364676209 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 0x7f4e0c135f40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 1 -- 172.21.15.98:0/364676209 >> 172.21.15.98:0/364676209 conn(0x7f4e0c0aba20 msgr2=0x7f4e0c0a7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 1 -- 172.21.15.98:0/364676209 shutdown_connections 2024-06-11T19:26:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.588+0000 7f4e108ff640 1 -- 172.21.15.98:0/364676209 wait complete. 2024-06-11T19:26:51.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.589+0000 7f4e108ff640 10 monclient: build_initial_monmap 2024-06-11T19:26:51.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.589+0000 7f4e108ff640 10 monclient: monmap: 2024-06-11T19:26:51.609 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:51.609 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:51.609 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:51.609 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:51.609 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:51.609 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:51.609 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:51.609 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:51.610 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:51.610 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:51.610 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:51.610 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:51.610 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:51.610 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:51.610 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:51.610 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.589+0000 7f4e108ff640 1 librados: starting msgr at 2024-06-11T19:26:51.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.589+0000 7f4e108ff640 1 librados: starting objecter 2024-06-11T19:26:51.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.589+0000 7f4e108ff640 1 Processor -- start 2024-06-11T19:26:51.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 1 -- start start 2024-06-11T19:26:51.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 1 librados: setting wanted keys 2024-06-11T19:26:51.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 1 librados: calling monclient init 2024-06-11T19:26:51.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 10 monclient: init 2024-06-11T19:26:51.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:51.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 10 monclient: _add_conns ranks=[2,5,6,0,1,3,8,4,7] 2024-06-11T19:26:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e0c0a5a10 0x7f4e0c1cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 10 monclient(hunting): picked mon.noname-b con 0x7f4e0c0a5a10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e0c0a6350 0x7f4e0c134c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 10 monclient(hunting): picked mon.noname-h con 0x7f4e0c0a6350 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 0x7f4e0c135170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 10 monclient(hunting): picked mon.noname-d con 0x7f4e0c135b60 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4e0c0a6f50 con 0x7f4e0c0a5a10 2024-06-11T19:26:51.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f4e0c050490 con 0x7f4e0c0a6350 2024-06-11T19:26:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4e0c0a7290 con 0x7f4e0c135b60 2024-06-11T19:26:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e108ff640 10 monclient(hunting): authenticate will time out at 1821.397499s 2024-06-11T19:26:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0a59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e0c0a5a10 0x7f4e0c1cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 0x7f4e0c135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 0x7f4e0c135170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58960/0 (socket says 172.21.15.98:58960) 2024-06-11T19:26:51.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0a59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e0c0a5a10 0x7f4e0c1cd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:43154/0 (socket says 172.21.15.98:43154) 2024-06-11T19:26:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 1 -- 172.21.15.98:0/2343335171 learned_addr learned my addr 172.21.15.98:0/2343335171 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 10 monclient(hunting): get_auth_request con 0x7f4e0c135b60 auth_method 0 2024-06-11T19:26:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0a59c640 10 monclient(hunting): get_auth_request con 0x7f4e0c0a5a10 auth_method 0 2024-06-11T19:26:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0a59c640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e09d9b640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e0c0a6350 0x7f4e0c134c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e09d9b640 10 monclient(hunting): get_auth_request con 0x7f4e0c0a6350 auth_method 0 2024-06-11T19:26:51.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e09d9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 10 monclient(hunting): handle_auth_done global_id 124512 payload 824 2024-06-11T19:26:51.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 1 -- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e0c0a6350 msgr2=0x7f4e0c134c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f4e0c0a6350 0x7f4e0c134c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 1 -- 172.21.15.98:0/2343335171 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e0c0a5a10 msgr2=0x7f4e0c1cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e0c0a5a10 0x7f4e0c1cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 10 monclient: _finish_hunting 0 2024-06-11T19:26:51.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 1 monclient: found mon.noname-d 2024-06-11T19:26:51.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:51.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:51.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0ad9d640 1 -- 172.21.15.98:0/2343335171 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0c1cdc10 con 0x7f4e0c135b60 2024-06-11T19:26:51.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.590+0000 7f4e0a59c640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e0c0a5a10 0x7f4e0c1cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:51.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4e0ad9d640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 0x7f4e0c135170 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4e04009fd0 tx=0x7f4e0400d560 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 10 client.?.objecter ms_handle_connect 0x7f4e0c135b60 2024-06-11T19:26:51.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 1 -- 172.21.15.98:0/2343335171 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4e0401a070 con 0x7f4e0c135b60 2024-06-11T19:26:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 10 monclient: dump: 2024-06-11T19:26:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.618 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.619 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.620 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.620 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.620 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.620 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 10 monclient: _finish_auth 0 2024-06-11T19:26:51.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 10 monclient: _check_auth_tickets 2024-06-11T19:26:51.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:51.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 1 -- 172.21.15.98:0/2343335171 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e04003e00 con 0x7f4e0c135b60 2024-06-11T19:26:51.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4e108ff640 5 monclient: authenticate success, global_id 124512 2024-06-11T19:26:51.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4e108ff640 10 monclient: _renew_subs 2024-06-11T19:26:51.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4e108ff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:51.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4e108ff640 1 -- 172.21.15.98:0/2343335171 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0c0a4470 con 0x7f4e0c135b60 2024-06-11T19:26:51.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:51.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 1 -- 172.21.15.98:0/2343335171 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4e040044a0 con 0x7f4e0c135b60 2024-06-11T19:26:51.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:51.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4dfaffd640 10 monclient: dump: 2024-06-11T19:26:51.621 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.622 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.622 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.622 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.622 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.622 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.622 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.622 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.622 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.623 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.623 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.623 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.623 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.623 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.623 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.623 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4e08d99640 4 set_mon_vals no callback set 2024-06-11T19:26:51.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4e108ff640 10 client.124512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:51.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4e108ff640 10 monclient: _renew_subs 2024-06-11T19:26:51.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4e108ff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:51.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4e108ff640 1 -- 172.21.15.98:0/2343335171 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e0c1bbeb0 con 0x7f4e0c135b60 2024-06-11T19:26:51.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.591+0000 7f4e108ff640 1 librados: init done 2024-06-11T19:26:51.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4e108ff640 10 client.124512.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:51.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4e108ff640 15 client.124512.objecter _session_command_op_assign -1 1 2024-06-11T19:26:51.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4e108ff640 20 client.124512.objecter _get_session osd=-1 returning homeless 2024-06-11T19:26:51.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4e108ff640 10 client.124512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:51.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4e108ff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:26:51.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 1 -- 172.21.15.98:0/2343335171 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f4e04004700 con 0x7f4e0c135b60 2024-06-11T19:26:51.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4e108ff640 1 -- 172.21.15.98:0/2343335171 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4e0c0a1410 con 0x7f4e0c135b60 2024-06-11T19:26:51.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4dec03fdb0 0x7f4dec042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 1 -- 172.21.15.98:0/2343335171 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f4e040519b0 con 0x7f4e0c135b60 2024-06-11T19:26:51.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 10 client.124512.objecter ms_dispatch 0x7f4e0c000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:51.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 3 client.124512.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:51.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 3 client.124512.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:51.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 10 client.124512.objecter checking command 1 2024-06-11T19:26:51.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] conn(0x7f4dec0475f0 0x7f4dec049a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 20 client.124512.objecter _get_session s=0x7f4dec046f20 osd=3 3 2024-06-11T19:26:51.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 20 client.124512.objecter put_session s=0x7f4dec046f20 osd=3 3 2024-06-11T19:26:51.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 15 client.124512.objecter _session_command_op_remove -1 1 2024-06-11T19:26:51.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 20 client.124512.objecter _get_session s=0x7f4dec046f20 osd=3 3 2024-06-11T19:26:51.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 20 client.124512.objecter get_session s=0x7f4dec046f20 osd=3 3 2024-06-11T19:26:51.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 15 client.124512.objecter _session_command_op_assign 3 1 2024-06-11T19:26:51.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 20 client.124512.objecter put_session s=0x7f4dec046f20 osd=3 4 2024-06-11T19:26:51.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 10 client.124512.objecter _send_command 1 2024-06-11T19:26:51.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 1 -- 172.21.15.98:0/2343335171 --> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4dec04a160 con 0x7f4dec0475f0 2024-06-11T19:26:51.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 20 client.124512.objecter dump_active .. 0 homeless 2024-06-11T19:26:51.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 1 -- 172.21.15.98:0/2343335171 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_get_version_reply(handle=1 version=685) ==== 24+0+0 (secure 0 0 0) 0x7f4e040562c0 con 0x7f4e0c135b60 2024-06-11T19:26:51.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4dfaffd640 10 monclient: handle_get_version_reply finishing 1 version 685 2024-06-11T19:26:51.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4e0a59c640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4dec03fdb0 0x7f4dec042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4e0a59c640 10 monclient: get_auth_request con 0x7f4dec03fdb0 auth_method 0 2024-06-11T19:26:51.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4e09d9b640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] conn(0x7f4dec0475f0 0x7f4dec049a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.592+0000 7f4e09d9b640 10 monclient: get_auth_request con 0x7f4dec0475f0 auth_method 0 2024-06-11T19:26:51.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.593+0000 7f4dfaffd640 10 client.124512.objecter ms_handle_connect 0x7f4dec03fdb0 2024-06-11T19:26:51.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.593+0000 7f4dfaffd640 10 client.124512.objecter ms_handle_connect 0x7f4dec0475f0 2024-06-11T19:26:51.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.593+0000 7f4e0a59c640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4dec03fdb0 0x7f4dec042270 secure :-1 s=READY pgs=4341 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc035000 tx=0x7f4dfc008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.593+0000 7f4e09d9b640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] conn(0x7f4dec0475f0 0x7f4dec049a90 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.594+0000 7f4dfaffd640 1 -- 172.21.15.98:0/2343335171 <== osd.3 v2:172.21.15.117:6808/2561136926 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4dec04a160 con 0x7f4dec0475f0 2024-06-11T19:26:51.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.594+0000 7f4dfaffd640 10 client.124512.objecter ms_dispatch 0x7f4e0c000b90 command_reply(tid 1: 0 ) 2024-06-11T19:26:51.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.594+0000 7f4dfaffd640 10 client.124512.objecter _finish_command 1 = system:0 2024-06-11T19:26:51.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.594+0000 7f4dfaffd640 20 client.124512.objecter put_session s=0x7f4dec046f20 osd=3 4 2024-06-11T19:26:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.594+0000 7f4dfaffd640 15 client.124512.objecter _session_command_op_remove 3 1 2024-06-11T19:26:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.607+0000 7f4e108ff640 10 client.124512.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-11T19:26:51.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.607+0000 7f4e108ff640 15 client.124512.objecter _session_command_op_assign -1 2 2024-06-11T19:26:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.607+0000 7f4e108ff640 20 client.124512.objecter _get_session s=0x7f4dec046f20 osd=3 3 2024-06-11T19:26:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.607+0000 7f4e108ff640 20 client.124512.objecter put_session s=0x7f4dec046f20 osd=3 3 2024-06-11T19:26:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.607+0000 7f4e108ff640 20 client.124512.objecter _get_session s=0x7f4dec046f20 osd=3 3 2024-06-11T19:26:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.607+0000 7f4e108ff640 15 client.124512.objecter _session_command_op_remove -1 2 2024-06-11T19:26:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.607+0000 7f4e108ff640 20 client.124512.objecter get_session s=0x7f4dec046f20 osd=3 3 2024-06-11T19:26:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.607+0000 7f4e108ff640 15 client.124512.objecter _session_command_op_assign 3 2 2024-06-11T19:26:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.607+0000 7f4e108ff640 20 client.124512.objecter put_session s=0x7f4dec046f20 osd=3 4 2024-06-11T19:26:51.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.607+0000 7f4e108ff640 10 client.124512.objecter _send_command 2 2024-06-11T19:26:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.607+0000 7f4e108ff640 1 -- 172.21.15.98:0/2343335171 --> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4e0c0a0ee0 con 0x7f4dec0475f0 2024-06-11T19:26:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4dfaffd640 1 -- 172.21.15.98:0/2343335171 <== osd.3 v2:172.21.15.117:6808/2561136926 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4e0000c9d0 con 0x7f4dec0475f0 2024-06-11T19:26:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4dfaffd640 10 client.124512.objecter ms_dispatch 0x7f4e0c000b90 command_reply(tid 2: 0 ) 2024-06-11T19:26:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4dfaffd640 10 client.124512.objecter _finish_command 2 = system:0 2024-06-11T19:26:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4dfaffd640 20 client.124512.objecter put_session s=0x7f4dec046f20 osd=3 4 2024-06-11T19:26:51.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4dfaffd640 15 client.124512.objecter _session_command_op_remove 3 2 2024-06-11T19:26:51.631 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:51.632 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-11T19:26:51.632 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:51.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 10 librados: watch_flush enter 2024-06-11T19:26:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 10 librados: watch_flush exit 2024-06-11T19:26:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 10 client.124512.objecter close_session for osd.3 2024-06-11T19:26:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 1 -- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] conn(0x7f4dec0475f0 msgr2=0x7f4dec049a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:6808/2561136926,v1:172.21.15.117:6809/2561136926] conn(0x7f4dec0475f0 0x7f4dec049a90 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 20 client.124512.objecter put_session s=0x7f4dec046f20 osd=3 1 2024-06-11T19:26:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 20 client.124512.objecter shutdown clearing up homeless session... 2024-06-11T19:26:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 10 client.124512.objecter successfully canceled tick 2024-06-11T19:26:51.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 1 -- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4dec03fdb0 msgr2=0x7f4dec042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f4dec03fdb0 0x7f4dec042270 secure :-1 s=READY pgs=4341 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc035000 tx=0x7f4dfc008030 comp rx=0 tx=0).stop 2024-06-11T19:26:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 10 monclient: shutdown 2024-06-11T19:26:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 1 -- 172.21.15.98:0/2343335171 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 msgr2=0x7f4e0c135170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 1 --2- 172.21.15.98:0/2343335171 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e0c135b60 0x7f4e0c135170 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4e04009fd0 tx=0x7f4e0400d560 comp rx=0 tx=0).stop 2024-06-11T19:26:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e0ad9d640 1 -- 172.21.15.98:0/2343335171 reap_dead start 2024-06-11T19:26:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 1 -- 172.21.15.98:0/2343335171 shutdown_connections 2024-06-11T19:26:51.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 1 -- 172.21.15.98:0/2343335171 >> 172.21.15.98:0/2343335171 conn(0x7f4e0c0aba20 msgr2=0x7f4e0c1c1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:51.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 1 -- 172.21.15.98:0/2343335171 shutdown_connections 2024-06-11T19:26:51.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 1 -- 172.21.15.98:0/2343335171 wait complete. 2024-06-11T19:26:51.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.608+0000 7f4e108ff640 1 librados: shutdown 2024-06-11T19:26:51.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-06-11T19:26:51.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.736+0000 7f47dbf8c640 10 monclient: get_monmap_and_config 2024-06-11T19:26:51.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.736+0000 7f47dbf8c640 10 monclient: build_initial_monmap 2024-06-11T19:26:51.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.736+0000 7f47dbf8c640 10 monclient: monmap: 2024-06-11T19:26:51.735 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:51.736 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.736 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:51.737196+0000 2024-06-11T19:26:51.736 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:51.737196+0000 2024-06-11T19:26:51.736 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:51.736 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.736 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:51.736 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:51.736 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:51.736 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:51.737 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:51.737 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:51.737 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:51.737 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:51.737 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:51.737 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:51.737 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:51.737 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.736+0000 7f47dbf8c640 1 Processor -- start 2024-06-11T19:26:51.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.736+0000 7f47dbf8c640 1 -- start start 2024-06-11T19:26:51.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.736+0000 7f47dbf8c640 10 monclient: init 2024-06-11T19:26:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 10 monclient: _add_conns ranks=[1,5,8,10,3,0,6,4,9,7,2] 2024-06-11T19:26:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f47d41347f0 addr v1:172.21.15.117:6789/0 2024-06-11T19:26:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d409e6b0 0x7f47d413dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 10 monclient(hunting): picked mon.noname-h con 0x7f47d409e6b0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f47d413e210 0x7f47d4140600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 10 monclient(hunting): picked mon.noname-e con 0x7f47d413e210 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d409d410 con 0x7f47d41347f0 2024-06-11T19:26:51.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f47d40687f0 con 0x7f47d409e6b0 2024-06-11T19:26:51.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f47d409eaf0 con 0x7f47d413e210 2024-06-11T19:26:51.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:51.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47dbf8c640 10 monclient(hunting): authenticate will time out at 1821.544431s 2024-06-11T19:26:51.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f47d413e210 0x7f47d4140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f47d413e210 0x7f47d4140600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54518/0 (socket says 172.21.15.98:54518) 2024-06-11T19:26:51.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 1 -- 172.21.15.98:0/2701531972 learned_addr learned my addr 172.21.15.98:0/2701531972 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 10 monclient(hunting): get_auth_request con 0x7f47d413e210 auth_method 0 2024-06-11T19:26:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47d9527640 1 --2- 172.21.15.98:0/2701531972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d409e6b0 0x7f47d413dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47d9527640 10 monclient(hunting): get_auth_request con 0x7f47d409e6b0 auth_method 0 2024-06-11T19:26:51.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47d9527640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47d9527640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47d9527640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 10 monclient(hunting): handle_auth_done global_id 124505 payload 274 2024-06-11T19:26:51.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 1 -- 172.21.15.98:0/2701531972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d409e6b0 msgr2=0x7f47d413dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:51.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 1 --2- 172.21.15.98:0/2701531972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d409e6b0 0x7f47d413dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 1 -- 172.21.15.98:0/2701531972 >> v1:172.21.15.117:6789/0 conn(0x7f47d41347f0 legacy=0x7f47d4134bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 10 monclient: _finish_hunting 0 2024-06-11T19:26:51.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 1 monclient: found mon.noname-e 2024-06-11T19:26:51.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:51.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:26:51.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47da529640 1 -- 172.21.15.98:0/2701531972 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d409d6f0 con 0x7f47d413e210 2024-06-11T19:26:51.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.737+0000 7f47d9527640 1 --2- 172.21.15.98:0/2701531972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d409e6b0 0x7f47d413dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:26:51.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47da529640 1 --2- 172.21.15.98:0/2701531972 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f47d413e210 0x7f47d4140600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009e80 tx=0x7f47cc02be10 comp rx=0 tx=0).ready entity=mon.8 client_cookie=800a463dc4ec1528 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 1 -- 172.21.15.98:0/2701531972 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47cc02cc20 con 0x7f47d413e210 2024-06-11T19:26:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:26:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 10 monclient: dump: 2024-06-11T19:26:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.751 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.752 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.752 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.752 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.752 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.752 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.752 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.752 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.752 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.753 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.753 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 10 monclient: _finish_auth 0 2024-06-11T19:26:51.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 10 monclient: _check_auth_tickets 2024-06-11T19:26:51.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:51.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47dbf8c640 5 monclient: authenticate success, global_id 124505 2024-06-11T19:26:51.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 1 -- 172.21.15.98:0/2701531972 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47cc02cde0 con 0x7f47d413e210 2024-06-11T19:26:51.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47dbf8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:51.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:51.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47dbf8c640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:51.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47dbf8c640 4 set_mon_vals no callback set 2024-06-11T19:26:51.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 1 -- 172.21.15.98:0/2701531972 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47cc034860 con 0x7f47d413e210 2024-06-11T19:26:51.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:51.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.738+0000 7f47d8d26640 10 monclient: dump: 2024-06-11T19:26:51.754 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.754 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.755 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.755 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.755 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.755 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.755 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.755 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.755 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.755 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.756 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.739+0000 7f47dbf8c640 10 monclient: shutdown 2024-06-11T19:26:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.739+0000 7f47dbf8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:51.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.739+0000 7f47dbf8c640 1 -- 172.21.15.98:0/2701531972 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f47d413e210 msgr2=0x7f47d4140600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.739+0000 7f47dbf8c640 1 --2- 172.21.15.98:0/2701531972 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f47d413e210 0x7f47d4140600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009e80 tx=0x7f47cc02be10 comp rx=0 tx=0).stop 2024-06-11T19:26:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.739+0000 7f47dbf8c640 1 -- 172.21.15.98:0/2701531972 shutdown_connections 2024-06-11T19:26:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.739+0000 7f47dbf8c640 1 --2- 172.21.15.98:0/2701531972 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f47d413e210 0x7f47d4140600 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.739+0000 7f47dbf8c640 1 --2- 172.21.15.98:0/2701531972 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d409e6b0 0x7f47d413dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.739+0000 7f47dbf8c640 1 -- 172.21.15.98:0/2701531972 >> 172.21.15.98:0/2701531972 conn(0x7f47d40a4780 msgr2=0x7f47d40a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.739+0000 7f47dbf8c640 1 -- 172.21.15.98:0/2701531972 shutdown_connections 2024-06-11T19:26:51.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.739+0000 7f47dbf8c640 1 -- 172.21.15.98:0/2701531972 wait complete. 2024-06-11T19:26:51.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.739+0000 7f47dbf8c640 10 monclient: build_initial_monmap 2024-06-11T19:26:51.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.739+0000 7f47dbf8c640 10 monclient: monmap: 2024-06-11T19:26:51.758 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:51.758 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:51.758 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:51.758 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:51.759 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:51.759 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:51.759 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:51.759 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:51.759 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:51.759 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:51.759 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:51.759 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:51.759 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:51.760 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:51.760 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:51.760 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 1 librados: starting msgr at 2024-06-11T19:26:51.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 1 librados: starting objecter 2024-06-11T19:26:51.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 1 Processor -- start 2024-06-11T19:26:51.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 1 -- start start 2024-06-11T19:26:51.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 1 librados: setting wanted keys 2024-06-11T19:26:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 1 librados: calling monclient init 2024-06-11T19:26:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 10 monclient: init 2024-06-11T19:26:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 10 monclient: _add_conns ranks=[5,3,2,4,8,7,1,0,6] 2024-06-11T19:26:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d409e6b0 0x7f47d41ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 10 monclient(hunting): picked mon.noname-h con 0x7f47d409e6b0 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47d41347f0 0x7f47d41ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 10 monclient(hunting): picked mon.noname-g con 0x7f47d41347f0 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d413e210 0x7f47d41cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 10 monclient(hunting): picked mon.noname-b con 0x7f47d413e210 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f47d40a56f0 con 0x7f47d413e210 2024-06-11T19:26:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f47d40a5570 con 0x7f47d41347f0 2024-06-11T19:26:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f47d4068ac0 con 0x7f47d409e6b0 2024-06-11T19:26:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47dbf8c640 10 monclient(hunting): authenticate will time out at 1821.547827s 2024-06-11T19:26:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47da529640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d413e210 0x7f47d41cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47da529640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d413e210 0x7f47d41cee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:43170/0 (socket says 172.21.15.98:43170) 2024-06-11T19:26:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47da529640 1 -- 172.21.15.98:0/72065969 learned_addr learned my addr 172.21.15.98:0/72065969 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47da529640 10 monclient(hunting): get_auth_request con 0x7f47d413e210 auth_method 0 2024-06-11T19:26:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47da529640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47da529640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.740+0000 7f47da529640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9d28640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d409e6b0 0x7f47d41ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9527640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47d41347f0 0x7f47d41ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9d28640 10 monclient(hunting): get_auth_request con 0x7f47d409e6b0 auth_method 0 2024-06-11T19:26:51.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9d28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9d28640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9d28640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9527640 10 monclient(hunting): get_auth_request con 0x7f47d41347f0 auth_method 0 2024-06-11T19:26:51.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9527640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9527640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9527640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9d28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9d28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9d28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9527640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:51.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9527640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:51.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47d9527640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:51.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 10 monclient(hunting): handle_auth_done global_id 85457 payload 824 2024-06-11T19:26:51.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 1 -- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d409e6b0 msgr2=0x7f47d41ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d409e6b0 0x7f47d41ce3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 1 -- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47d41347f0 msgr2=0x7f47d41ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47d41347f0 0x7f47d41ce8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 10 monclient: _finish_hunting 0 2024-06-11T19:26:51.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 1 monclient: found mon.noname-b 2024-06-11T19:26:51.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:51.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:26:51.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 1 -- 172.21.15.98:0/72065969 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d411f560 con 0x7f47d413e210 2024-06-11T19:26:51.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47da529640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d413e210 0x7f47d41cee30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f47cc036040 tx=0x7f47cc004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 10 client.?.objecter ms_handle_connect 0x7f47d413e210 2024-06-11T19:26:51.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:51.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 1 -- 172.21.15.98:0/72065969 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47cc044020 con 0x7f47d413e210 2024-06-11T19:26:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:26:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 10 monclient: dump: 2024-06-11T19:26:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.769 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.769 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.769 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.769 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.769 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.769 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.769 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.769 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.769 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.770 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 10 monclient: _finish_auth 0 2024-06-11T19:26:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 10 monclient: _check_auth_tickets 2024-06-11T19:26:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 1 -- 172.21.15.98:0/72065969 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47cc034cb0 con 0x7f47d413e210 2024-06-11T19:26:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47dbf8c640 5 monclient: authenticate success, global_id 85457 2024-06-11T19:26:51.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47dbf8c640 10 monclient: _renew_subs 2024-06-11T19:26:51.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47dbf8c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:51.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47dbf8c640 1 -- 172.21.15.98:0/72065969 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d413dde0 con 0x7f47d413e210 2024-06-11T19:26:51.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:51.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 1 -- 172.21.15.98:0/72065969 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47cc009510 con 0x7f47d413e210 2024-06-11T19:26:51.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:51.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:26:51.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47ca7fc640 10 monclient: dump: 2024-06-11T19:26:51.771 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:51.772 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:51.772 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.772 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:51.772 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:51.772 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:51.772 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:51.772 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:51.772 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:51.772 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:51.773 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:51.773 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:51.773 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:51.773 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:51.773 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:51.773 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:51.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47cbfff640 4 set_mon_vals no callback set 2024-06-11T19:26:51.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47dbf8c640 10 client.85457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:51.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47dbf8c640 10 monclient: _renew_subs 2024-06-11T19:26:51.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47dbf8c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:51.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47dbf8c640 1 -- 172.21.15.98:0/72065969 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d41393e0 con 0x7f47d413e210 2024-06-11T19:26:51.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.741+0000 7f47dbf8c640 1 librados: init done 2024-06-11T19:26:51.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.742+0000 7f47dbf8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:51.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.742+0000 7f47dbf8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:51.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.742+0000 7f47dbf8c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:51.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.742+0000 7f47dbf8c640 1 -- 172.21.15.98:0/72065969 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d409c9e0 con 0x7f47d413e210 2024-06-11T19:26:51.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.742+0000 7f47ca7fc640 1 -- 172.21.15.98:0/72065969 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f47cc047020 con 0x7f47d413e210 2024-06-11T19:26:51.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.742+0000 7f47ca7fc640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f47b803fdb0 0x7f47b8042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:51.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.743+0000 7f47ca7fc640 1 -- 172.21.15.98:0/72065969 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f47cc073e30 con 0x7f47d413e210 2024-06-11T19:26:51.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.743+0000 7f47ca7fc640 10 client.85457.objecter ms_dispatch 0x7f47d4000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:51.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.743+0000 7f47ca7fc640 3 client.85457.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:51.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.743+0000 7f47ca7fc640 3 client.85457.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:51.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.743+0000 7f47ca7fc640 20 client.85457.objecter dump_active .. 0 homeless 2024-06-11T19:26:51.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.743+0000 7f47d9d28640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f47b803fdb0 0x7f47b8042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:51.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.743+0000 7f47d9d28640 10 monclient: get_auth_request con 0x7f47b803fdb0 auth_method 0 2024-06-11T19:26:51.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.743+0000 7f47ca7fc640 10 client.85457.objecter ms_handle_connect 0x7f47b803fdb0 2024-06-11T19:26:51.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.743+0000 7f47d9d28640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f47b803fdb0 0x7f47b8042270 secure :-1 s=READY pgs=4342 cs=0 l=1 rev1=1 crypto rx=0x7f47d00017c0 tx=0x7f47d0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:51.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.747+0000 7f47ca7fc640 1 -- 172.21.15.98:0/72065969 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f47d409c9e0 con 0x7f47d413e210 2024-06-11T19:26:51.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.747+0000 7f47ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:51.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.747+0000 7f47ca7fc640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:51.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.928+0000 7f47dbf8c640 1 -- 172.21.15.98:0/72065969 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f47d4134bd0 con 0x7f47b803fdb0 2024-06-11T19:26:51.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.929+0000 7f47ca7fc640 1 -- 172.21.15.98:0/72065969 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f47d4134bd0 con 0x7f47b803fdb0 2024-06-11T19:26:51.928 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 3 to deep-scrub 2024-06-11T19:26:51.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.931+0000 7f47dbf8c640 10 librados: watch_flush enter 2024-06-11T19:26:51.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.931+0000 7f47dbf8c640 10 librados: watch_flush exit 2024-06-11T19:26:51.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.931+0000 7f47dbf8c640 20 client.85457.objecter shutdown clearing up homeless session... 2024-06-11T19:26:51.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.931+0000 7f47dbf8c640 10 client.85457.objecter successfully canceled tick 2024-06-11T19:26:51.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 1 -- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f47b803fdb0 msgr2=0x7f47b8042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f47b803fdb0 0x7f47b8042270 secure :-1 s=READY pgs=4342 cs=0 l=1 rev1=1 crypto rx=0x7f47d00017c0 tx=0x7f47d0009040 comp rx=0 tx=0).stop 2024-06-11T19:26:51.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 10 monclient: shutdown 2024-06-11T19:26:51.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:51.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 1 -- 172.21.15.98:0/72065969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d413e210 msgr2=0x7f47d41cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:51.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d413e210 0x7f47d41cee30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f47cc036040 tx=0x7f47cc004ab0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 1 -- 172.21.15.98:0/72065969 shutdown_connections 2024-06-11T19:26:51.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f47b803fdb0 0x7f47b8042270 unknown :-1 s=CLOSED pgs=4342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f47d413e210 0x7f47d41cee30 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47d41347f0 0x7f47d41ce8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 1 --2- 172.21.15.98:0/72065969 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f47d409e6b0 0x7f47d41ce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:51.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 1 -- 172.21.15.98:0/72065969 >> 172.21.15.98:0/72065969 conn(0x7f47d40a4780 msgr2=0x7f47d4137930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:51.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 1 -- 172.21.15.98:0/72065969 shutdown_connections 2024-06-11T19:26:51.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.933+0000 7f47dbf8c640 1 -- 172.21.15.98:0/72065969 wait complete. 2024-06-11T19:26:51.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:51.934+0000 7f47dbf8c640 1 librados: shutdown 2024-06-11T19:26:51.944 INFO:tasks.ceph:Scrubbing osd.4 2024-06-11T19:26:51.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-06-11T19:26:52.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.048+0000 7f2949c20640 10 monclient: get_monmap_and_config 2024-06-11T19:26:52.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.048+0000 7f2949c20640 10 monclient: build_initial_monmap 2024-06-11T19:26:52.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.048+0000 7f2949c20640 10 monclient: monmap: 2024-06-11T19:26:52.048 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:52.048 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.048 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:52.049745+0000 2024-06-11T19:26:52.048 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:52.049745+0000 2024-06-11T19:26:52.048 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:52.048 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.048 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:52.049 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:52.049 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:52.049 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:52.049 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:52.049 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:52.049 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:52.049 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:52.049 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:52.050 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:52.050 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:52.050 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.049+0000 7f2949c20640 1 Processor -- start 2024-06-11T19:26:52.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.049+0000 7f2949c20640 1 -- start start 2024-06-11T19:26:52.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.049+0000 7f2949c20640 10 monclient: init 2024-06-11T19:26:52.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.049+0000 7f2949c20640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:52.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.049+0000 7f2949c20640 10 monclient: _add_conns ranks=[4,0,2,10,8,9,7,3,1,5,6] 2024-06-11T19:26:52.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.049+0000 7f2949c20640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2944135b60 0x7f2944135f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2949c20640 10 monclient(hunting): picked mon.noname-c con 0x7f2944135b60 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2949c20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29440a5a10 addr v1:172.21.15.98:6789/0 2024-06-11T19:26:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2949c20640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29440a6520 0x7f2944140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2949c20640 10 monclient(hunting): picked mon.noname-b con 0x7f29440a6520 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2949c20640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29440a8580 con 0x7f29440a5a10 2024-06-11T19:26:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2949c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2949c20640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f29440a2c60 con 0x7f29440a6520 2024-06-11T19:26:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2949c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2949c20640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f29440a2f40 con 0x7f2944135b60 2024-06-11T19:26:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2949c20640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2949c20640 10 monclient(hunting): authenticate will time out at 1821.857065s 2024-06-11T19:26:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2944135b60 0x7f2944135f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2942ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f29440a5a10 0x7f29440a5e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51428/0 (socket says 172.21.15.98:51428) 2024-06-11T19:26:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2942ffd640 1 -- 172.21.15.98:0/3645599996 learned_addr learned my addr 172.21.15.98:0/3645599996 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-06-11T19:26:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2943fff640 1 --2- 172.21.15.98:0/3645599996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29440a6520 0x7f2944140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 10 monclient(hunting): get_auth_request con 0x7f2944135b60 auth_method 0 2024-06-11T19:26:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2943fff640 10 monclient(hunting): get_auth_request con 0x7f29440a6520 auth_method 0 2024-06-11T19:26:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2943fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2943fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2943fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2943fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2943fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2943fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 10 monclient(hunting): handle_auth_done global_id 134230 payload 274 2024-06-11T19:26:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 1 -- 172.21.15.98:0/3645599996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29440a6520 msgr2=0x7f2944140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 1 --2- 172.21.15.98:0/3645599996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29440a6520 0x7f2944140cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 1 -- 172.21.15.98:0/3645599996 >> v1:172.21.15.98:6789/0 conn(0x7f29440a5a10 legacy=0x7f29440a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 10 monclient: _finish_hunting 0 2024-06-11T19:26:52.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 1 monclient: found mon.noname-c 2024-06-11T19:26:52.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:52.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:26:52.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29427fc640 1 -- 172.21.15.98:0/3645599996 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317413818 0 0) 0x7f29440a8580 con 0x7f29440a5a10 2024-06-11T19:26:52.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29437fe640 1 -- 172.21.15.98:0/3645599996 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29440a3220 con 0x7f2944135b60 2024-06-11T19:26:52.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f2943fff640 1 --2- 172.21.15.98:0/3645599996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29440a6520 0x7f2944140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:52.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.050+0000 7f29427fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:26:52.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29437fe640 1 --2- 172.21.15.98:0/3645599996 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2944135b60 0x7f2944135f40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f29340097d0 tx=0x7f293402b770 comp rx=0 tx=0).ready entity=mon.4 client_cookie=20ef188edeaea9c5 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 1 -- 172.21.15.98:0/3645599996 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2934039070 con 0x7f2944135b60 2024-06-11T19:26:52.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:26:52.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 10 monclient: dump: 2024-06-11T19:26:52.065 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.065 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.065 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.065 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.065 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.065 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.066 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.066 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.066 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.066 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.066 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.066 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.066 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.066 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.066 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.067 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 10 monclient: _finish_auth 0 2024-06-11T19:26:52.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 10 monclient: _check_auth_tickets 2024-06-11T19:26:52.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:52.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f2949c20640 5 monclient: authenticate success, global_id 134230 2024-06-11T19:26:52.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 1 -- 172.21.15.98:0/3645599996 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2934030070 con 0x7f2944135b60 2024-06-11T19:26:52.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:52.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f2949c20640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:52.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f2949c20640 4 set_mon_vals no callback set 2024-06-11T19:26:52.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 1 -- 172.21.15.98:0/3645599996 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2934031db0 con 0x7f2944135b60 2024-06-11T19:26:52.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:26:52.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f29427fc640 10 monclient: dump: 2024-06-11T19:26:52.068 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.068 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.068 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.068 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.069 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.069 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.069 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.069 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.069 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.069 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.069 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.069 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.069 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.070 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.070 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.070 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f2949c20640 10 monclient: shutdown 2024-06-11T19:26:52.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f2949c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:52.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f2949c20640 1 -- 172.21.15.98:0/3645599996 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2944135b60 msgr2=0x7f2944135f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f2949c20640 1 --2- 172.21.15.98:0/3645599996 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2944135b60 0x7f2944135f40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f29340097d0 tx=0x7f293402b770 comp rx=0 tx=0).stop 2024-06-11T19:26:52.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f2949c20640 1 -- 172.21.15.98:0/3645599996 shutdown_connections 2024-06-11T19:26:52.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f2949c20640 1 --2- 172.21.15.98:0/3645599996 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29440a6520 0x7f2944140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f2949c20640 1 --2- 172.21.15.98:0/3645599996 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2944135b60 0x7f2944135f40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.051+0000 7f2949c20640 1 -- 172.21.15.98:0/3645599996 >> 172.21.15.98:0/3645599996 conn(0x7f29440aba20 msgr2=0x7f29440a7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.052+0000 7f2949c20640 1 -- 172.21.15.98:0/3645599996 shutdown_connections 2024-06-11T19:26:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.052+0000 7f2949c20640 1 -- 172.21.15.98:0/3645599996 wait complete. 2024-06-11T19:26:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.052+0000 7f2949c20640 10 monclient: build_initial_monmap 2024-06-11T19:26:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.052+0000 7f2949c20640 10 monclient: monmap: 2024-06-11T19:26:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:52.072 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:52.072 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:52.072 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:52.072 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:52.072 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:52.072 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:52.072 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:52.072 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:52.072 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:52.073 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:52.073 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:52.073 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:52.073 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.052+0000 7f2949c20640 1 librados: starting msgr at 2024-06-11T19:26:52.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.052+0000 7f2949c20640 1 librados: starting objecter 2024-06-11T19:26:52.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.052+0000 7f2949c20640 1 Processor -- start 2024-06-11T19:26:52.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 1 -- start start 2024-06-11T19:26:52.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 1 librados: setting wanted keys 2024-06-11T19:26:52.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 1 librados: calling monclient init 2024-06-11T19:26:52.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 10 monclient: init 2024-06-11T19:26:52.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:52.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 10 monclient: _add_conns ranks=[1,7,0,3,5,2,4,8,6] 2024-06-11T19:26:52.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29440a5a10 0x7f29441cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 10 monclient(hunting): picked mon.noname-f con 0x7f29440a5a10 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:26:52.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f29440a6520 0x7f2944134e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 10 monclient(hunting): picked mon.noname-i con 0x7f29440a6520 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:52.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2944135b60 0x7f294409dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 10 monclient(hunting): picked mon.noname-a con 0x7f2944135b60 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:26:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f29440a7290 con 0x7f2944135b60 2024-06-11T19:26:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2944050490 con 0x7f29440a5a10 2024-06-11T19:26:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f29440a53a0 con 0x7f29440a6520 2024-06-11T19:26:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:52.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2949c20640 10 monclient(hunting): authenticate will time out at 1821.860601s 2024-06-11T19:26:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2944135b60 0x7f294409dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2944135b60 0x7f294409dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41880/0 (socket says 172.21.15.98:41880) 2024-06-11T19:26:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 1 -- 172.21.15.98:0/1261059529 learned_addr learned my addr 172.21.15.98:0/1261059529 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 10 monclient(hunting): get_auth_request con 0x7f2944135b60 auth_method 0 2024-06-11T19:26:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f29437fe640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29440a5a10 0x7f29441cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2942ffd640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f29440a6520 0x7f2944134e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f29437fe640 10 monclient(hunting): get_auth_request con 0x7f29440a5a10 auth_method 0 2024-06-11T19:26:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f29437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f29437fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f29437fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2942ffd640 10 monclient(hunting): get_auth_request con 0x7f29440a6520 auth_method 0 2024-06-11T19:26:52.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2942ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2942ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 10 monclient(hunting): handle_auth_done global_id 66492 payload 824 2024-06-11T19:26:52.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 1 -- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f29440a6520 msgr2=0x7f2944134e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f29440a6520 0x7f2944134e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 1 -- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29440a5a10 msgr2=0x7f29441cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29440a5a10 0x7f29441cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 10 monclient: _finish_hunting 0 2024-06-11T19:26:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.053+0000 7f2943fff640 1 monclient: found mon.noname-a 2024-06-11T19:26:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2943fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2943fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:26:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2943fff640 1 -- 172.21.15.98:0/1261059529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29440a13b0 con 0x7f2944135b60 2024-06-11T19:26:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f29437fe640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29440a5a10 0x7f29441cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:26:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2943fff640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2944135b60 0x7f294409dbf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f293c00efc0 tx=0x7f293c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 10 client.?.objecter ms_handle_connect 0x7f2944135b60 2024-06-11T19:26:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 1 -- 172.21.15.98:0/1261059529 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f293c018030 con 0x7f2944135b60 2024-06-11T19:26:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:26:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 10 monclient: dump: 2024-06-11T19:26:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.082 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.082 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 10 monclient: _finish_auth 0 2024-06-11T19:26:52.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 10 monclient: _check_auth_tickets 2024-06-11T19:26:52.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:52.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 1 -- 172.21.15.98:0/1261059529 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f293c00cde0 con 0x7f2944135b60 2024-06-11T19:26:52.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:52.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 1 -- 172.21.15.98:0/1261059529 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f293c014d60 con 0x7f2944135b60 2024-06-11T19:26:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f292bfff640 10 monclient: dump: 2024-06-11T19:26:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.084 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.084 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.084 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.084 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.084 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.084 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.084 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.084 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.084 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.085 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2941ffb640 4 set_mon_vals no callback set 2024-06-11T19:26:52.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2949c20640 5 monclient: authenticate success, global_id 66492 2024-06-11T19:26:52.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2949c20640 10 monclient: _renew_subs 2024-06-11T19:26:52.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2949c20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:52.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2949c20640 1 -- 172.21.15.98:0/1261059529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2944122510 con 0x7f2944135b60 2024-06-11T19:26:52.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2949c20640 10 client.66492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:52.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2949c20640 10 monclient: _renew_subs 2024-06-11T19:26:52.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2949c20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2949c20640 1 -- 172.21.15.98:0/1261059529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29440a0ea0 con 0x7f2944135b60 2024-06-11T19:26:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.054+0000 7f2949c20640 1 librados: init done 2024-06-11T19:26:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f2949c20640 10 client.66492.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f2949c20640 15 client.66492.objecter _session_command_op_assign -1 1 2024-06-11T19:26:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f2949c20640 20 client.66492.objecter _get_session osd=-1 returning homeless 2024-06-11T19:26:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f2949c20640 10 client.66492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f2949c20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f2949c20640 1 -- 172.21.15.98:0/1261059529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f29440a10d0 con 0x7f2944135b60 2024-06-11T19:26:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 1 -- 172.21.15.98:0/1261059529 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f293c005d10 con 0x7f2944135b60 2024-06-11T19:26:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f293003fe10 0x7f29300422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 1 -- 172.21.15.98:0/1261059529 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f293c019070 con 0x7f2944135b60 2024-06-11T19:26:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 10 client.66492.objecter ms_dispatch 0x7f2944000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 3 client.66492.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 3 client.66492.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 10 client.66492.objecter checking command 1 2024-06-11T19:26:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] conn(0x7f2930047650 0x7f2930049af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 20 client.66492.objecter _get_session s=0x7f2930046f80 osd=4 3 2024-06-11T19:26:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 20 client.66492.objecter put_session s=0x7f2930046f80 osd=4 3 2024-06-11T19:26:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 15 client.66492.objecter _session_command_op_remove -1 1 2024-06-11T19:26:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 20 client.66492.objecter _get_session s=0x7f2930046f80 osd=4 3 2024-06-11T19:26:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 20 client.66492.objecter get_session s=0x7f2930046f80 osd=4 3 2024-06-11T19:26:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 15 client.66492.objecter _session_command_op_assign 4 1 2024-06-11T19:26:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 20 client.66492.objecter put_session s=0x7f2930046f80 osd=4 4 2024-06-11T19:26:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 10 client.66492.objecter _send_command 1 2024-06-11T19:26:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 1 -- 172.21.15.98:0/1261059529 --> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f293c012e50 con 0x7f2930047650 2024-06-11T19:26:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 20 client.66492.objecter dump_active .. 0 homeless 2024-06-11T19:26:52.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 1 -- 172.21.15.98:0/1261059529 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=685) ==== 24+0+0 (secure 0 0 0) 0x7f293c0542e0 con 0x7f2944135b60 2024-06-11T19:26:52.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f292bfff640 10 monclient: handle_get_version_reply finishing 1 version 685 2024-06-11T19:26:52.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f29437fe640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f293003fe10 0x7f29300422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.055+0000 7f29437fe640 10 monclient: get_auth_request con 0x7f293003fe10 auth_method 0 2024-06-11T19:26:52.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.056+0000 7f2942ffd640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] conn(0x7f2930047650 0x7f2930049af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.056+0000 7f2942ffd640 10 monclient: get_auth_request con 0x7f2930047650 auth_method 0 2024-06-11T19:26:52.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.056+0000 7f292bfff640 10 client.66492.objecter ms_handle_connect 0x7f293003fe10 2024-06-11T19:26:52.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.056+0000 7f29437fe640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f293003fe10 0x7f29300422d0 secure :-1 s=READY pgs=4343 cs=0 l=1 rev1=1 crypto rx=0x7f2934036000 tx=0x7f2934008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.056+0000 7f292bfff640 10 client.66492.objecter ms_handle_connect 0x7f2930047650 2024-06-11T19:26:52.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.056+0000 7f2942ffd640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] conn(0x7f2930047650 0x7f2930049af0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.057+0000 7f292bfff640 1 -- 172.21.15.98:0/1261059529 <== osd.4 v2:172.21.15.117:6800/3877218228 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f293c012e50 con 0x7f2930047650 2024-06-11T19:26:52.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.057+0000 7f292bfff640 10 client.66492.objecter ms_dispatch 0x7f2944000b90 command_reply(tid 1: 0 ) 2024-06-11T19:26:52.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.057+0000 7f292bfff640 10 client.66492.objecter _finish_command 1 = system:0 2024-06-11T19:26:52.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.057+0000 7f292bfff640 20 client.66492.objecter put_session s=0x7f2930046f80 osd=4 4 2024-06-11T19:26:52.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.057+0000 7f292bfff640 15 client.66492.objecter _session_command_op_remove 4 1 2024-06-11T19:26:52.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 10 client.66492.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-11T19:26:52.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 15 client.66492.objecter _session_command_op_assign -1 2 2024-06-11T19:26:52.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 20 client.66492.objecter _get_session s=0x7f2930046f80 osd=4 3 2024-06-11T19:26:52.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 20 client.66492.objecter put_session s=0x7f2930046f80 osd=4 3 2024-06-11T19:26:52.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 20 client.66492.objecter _get_session s=0x7f2930046f80 osd=4 3 2024-06-11T19:26:52.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 15 client.66492.objecter _session_command_op_remove -1 2 2024-06-11T19:26:52.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 20 client.66492.objecter get_session s=0x7f2930046f80 osd=4 3 2024-06-11T19:26:52.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 15 client.66492.objecter _session_command_op_assign 4 2 2024-06-11T19:26:52.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 20 client.66492.objecter put_session s=0x7f2930046f80 osd=4 4 2024-06-11T19:26:52.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 10 client.66492.objecter _send_command 2 2024-06-11T19:26:52.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 1 -- 172.21.15.98:0/1261059529 --> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f29440732d0 con 0x7f2930047650 2024-06-11T19:26:52.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f292bfff640 1 -- 172.21.15.98:0/1261059529 <== osd.4 v2:172.21.15.117:6800/3877218228 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f29440732d0 con 0x7f2930047650 2024-06-11T19:26:52.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f292bfff640 10 client.66492.objecter ms_dispatch 0x7f2944000b90 command_reply(tid 2: 0 ) 2024-06-11T19:26:52.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f292bfff640 10 client.66492.objecter _finish_command 2 = system:0 2024-06-11T19:26:52.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f292bfff640 20 client.66492.objecter put_session s=0x7f2930046f80 osd=4 4 2024-06-11T19:26:52.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f292bfff640 15 client.66492.objecter _session_command_op_remove 4 2 2024-06-11T19:26:52.093 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:52.093 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-11T19:26:52.093 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:52.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 10 librados: watch_flush enter 2024-06-11T19:26:52.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 10 librados: watch_flush exit 2024-06-11T19:26:52.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 10 client.66492.objecter close_session for osd.4 2024-06-11T19:26:52.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 1 -- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] conn(0x7f2930047650 msgr2=0x7f2930049af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:6800/3877218228,v1:172.21.15.117:6801/3877218228] conn(0x7f2930047650 0x7f2930049af0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 20 client.66492.objecter put_session s=0x7f2930046f80 osd=4 1 2024-06-11T19:26:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 20 client.66492.objecter shutdown clearing up homeless session... 2024-06-11T19:26:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 10 client.66492.objecter successfully canceled tick 2024-06-11T19:26:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 1 -- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f293003fe10 msgr2=0x7f29300422d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f293003fe10 0x7f29300422d0 secure :-1 s=READY pgs=4343 cs=0 l=1 rev1=1 crypto rx=0x7f2934036000 tx=0x7f2934008030 comp rx=0 tx=0).stop 2024-06-11T19:26:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 10 monclient: shutdown 2024-06-11T19:26:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:52.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 1 -- 172.21.15.98:0/1261059529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2944135b60 msgr2=0x7f294409dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 1 --2- 172.21.15.98:0/1261059529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2944135b60 0x7f294409dbf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f293c00efc0 tx=0x7f293c009f90 comp rx=0 tx=0).stop 2024-06-11T19:26:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2943fff640 1 -- 172.21.15.98:0/1261059529 reap_dead start 2024-06-11T19:26:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 1 -- 172.21.15.98:0/1261059529 shutdown_connections 2024-06-11T19:26:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 1 -- 172.21.15.98:0/1261059529 >> 172.21.15.98:0/1261059529 conn(0x7f29440aba20 msgr2=0x7f29440a9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.070+0000 7f2949c20640 1 -- 172.21.15.98:0/1261059529 shutdown_connections 2024-06-11T19:26:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.071+0000 7f2949c20640 1 -- 172.21.15.98:0/1261059529 wait complete. 2024-06-11T19:26:52.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.071+0000 7f2949c20640 1 librados: shutdown 2024-06-11T19:26:52.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-06-11T19:26:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.198+0000 7f308ed2a640 10 monclient: get_monmap_and_config 2024-06-11T19:26:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.198+0000 7f308ed2a640 10 monclient: build_initial_monmap 2024-06-11T19:26:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.198+0000 7f308ed2a640 10 monclient: monmap: 2024-06-11T19:26:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:52.199165+0000 2024-06-11T19:26:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:52.199165+0000 2024-06-11T19:26:52.198 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:52.198 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.198 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:52.198 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:52.198 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:52.198 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:52.198 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:52.198 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:52.198 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:52.199 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:52.199 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:52.199 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:52.199 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:52.199 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.198+0000 7f308ed2a640 1 Processor -- start 2024-06-11T19:26:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.198+0000 7f308ed2a640 1 -- start start 2024-06-11T19:26:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.198+0000 7f308ed2a640 10 monclient: init 2024-06-11T19:26:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 10 monclient: _add_conns ranks=[7,5,6,0,3,8,1,10,2,4,9] 2024-06-11T19:26:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f30880a52a0 0x7f30880a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 10 monclient(hunting): picked mon.noname-i con 0x7f30880a52a0 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f308809d520 0x7f30880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 10 monclient(hunting): picked mon.noname-h con 0x7f308809d520 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:26:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3088073830 0x7f30880a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 10 monclient(hunting): picked mon.noname-d con 0x7f3088073830 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f308809daa0 con 0x7f308809d520 2024-06-11T19:26:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f308809d2e0 con 0x7f3088073830 2024-06-11T19:26:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f30880a5680 con 0x7f30880a52a0 2024-06-11T19:26:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308ed2a640 10 monclient(hunting): authenticate will time out at 1822.006504s 2024-06-11T19:26:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3088073830 0x7f30880a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3088073830 0x7f30880a3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58962/0 (socket says 172.21.15.98:58962) 2024-06-11T19:26:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 1 -- 172.21.15.98:0/616419601 learned_addr learned my addr 172.21.15.98:0/616419601 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 10 monclient(hunting): get_auth_request con 0x7f3088073830 auth_method 0 2024-06-11T19:26:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308cac6640 1 --2- 172.21.15.98:0/616419601 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f30880a52a0 0x7f30880a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f3087fff640 1 --2- 172.21.15.98:0/616419601 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f308809d520 0x7f30880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308cac6640 10 monclient(hunting): get_auth_request con 0x7f30880a52a0 auth_method 0 2024-06-11T19:26:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308cac6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308cac6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308cac6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 10 monclient(hunting): handle_auth_done global_id 124521 payload 274 2024-06-11T19:26:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 1 -- 172.21.15.98:0/616419601 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f30880a52a0 msgr2=0x7f30880a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 1 --2- 172.21.15.98:0/616419601 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f30880a52a0 0x7f30880a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 1 -- 172.21.15.98:0/616419601 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f308809d520 msgr2=0x7f30880a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 1 --2- 172.21.15.98:0/616419601 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f308809d520 0x7f30880a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 10 monclient: _finish_hunting 0 2024-06-11T19:26:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 1 monclient: found mon.noname-d 2024-06-11T19:26:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:26:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f308d2c7640 1 -- 172.21.15.98:0/616419601 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f308809e9d0 con 0x7f3088073830 2024-06-11T19:26:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f3087fff640 10 monclient: get_auth_request con 0x7f308809d520 auth_method 0 2024-06-11T19:26:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.199+0000 7f3087fff640 1 --2- 172.21.15.98:0/616419601 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f308809d520 0x7f30880a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:26:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f308d2c7640 1 --2- 172.21.15.98:0/616419601 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3088073830 0x7f30880a3bd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3080009e80 tx=0x7f308002bd70 comp rx=0 tx=0).ready entity=mon.6 client_cookie=cf06f59d67b5fce0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 1 -- 172.21.15.98:0/616419601 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f308002cb80 con 0x7f3088073830 2024-06-11T19:26:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:26:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 10 monclient: dump: 2024-06-11T19:26:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.214 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.214 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.214 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 10 monclient: _finish_auth 0 2024-06-11T19:26:52.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 10 monclient: _check_auth_tickets 2024-06-11T19:26:52.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:52.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 1 -- 172.21.15.98:0/616419601 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f308002cd40 con 0x7f3088073830 2024-06-11T19:26:52.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f308ed2a640 5 monclient: authenticate success, global_id 124521 2024-06-11T19:26:52.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f308ed2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f308ed2a640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 1 -- 172.21.15.98:0/616419601 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3080034860 con 0x7f3088073830 2024-06-11T19:26:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f308ed2a640 4 set_mon_vals no callback set 2024-06-11T19:26:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:26:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.200+0000 7f30877fe640 10 monclient: dump: 2024-06-11T19:26:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.217 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 10 monclient: shutdown 2024-06-11T19:26:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 1 -- 172.21.15.98:0/616419601 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3088073830 msgr2=0x7f30880a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 1 --2- 172.21.15.98:0/616419601 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3088073830 0x7f30880a3bd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3080009e80 tx=0x7f308002bd70 comp rx=0 tx=0).stop 2024-06-11T19:26:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 1 -- 172.21.15.98:0/616419601 shutdown_connections 2024-06-11T19:26:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 1 --2- 172.21.15.98:0/616419601 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f308809d520 0x7f30880a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 1 --2- 172.21.15.98:0/616419601 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3088073830 0x7f30880a3bd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 1 --2- 172.21.15.98:0/616419601 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f30880a52a0 0x7f30880a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 1 -- 172.21.15.98:0/616419601 >> 172.21.15.98:0/616419601 conn(0x7f30880a4780 msgr2=0x7f30880a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 1 -- 172.21.15.98:0/616419601 shutdown_connections 2024-06-11T19:26:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 1 -- 172.21.15.98:0/616419601 wait complete. 2024-06-11T19:26:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 10 monclient: build_initial_monmap 2024-06-11T19:26:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.201+0000 7f308ed2a640 10 monclient: monmap: 2024-06-11T19:26:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:52.220 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.202+0000 7f308ed2a640 1 librados: starting msgr at 2024-06-11T19:26:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.202+0000 7f308ed2a640 1 librados: starting objecter 2024-06-11T19:26:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.202+0000 7f308ed2a640 1 Processor -- start 2024-06-11T19:26:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.202+0000 7f308ed2a640 1 -- start start 2024-06-11T19:26:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.202+0000 7f308ed2a640 1 librados: setting wanted keys 2024-06-11T19:26:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.202+0000 7f308ed2a640 1 librados: calling monclient init 2024-06-11T19:26:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.202+0000 7f308ed2a640 10 monclient: init 2024-06-11T19:26:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.202+0000 7f308ed2a640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.202+0000 7f308ed2a640 10 monclient: _add_conns ranks=[8,1,2,7,3,4,6,5,0] 2024-06-11T19:26:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.202+0000 7f308ed2a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3088073830 0x7f3088067ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.202+0000 7f308ed2a640 10 monclient(hunting): picked mon.noname-e con 0x7f3088073830 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f308809d520 0x7f3088068220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 10 monclient(hunting): picked mon.noname-f con 0x7f308809d520 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:26:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30880a52a0 0x7f3088068760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 10 monclient(hunting): picked mon.noname-b con 0x7f30880a52a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f30880a1fd0 con 0x7f308809d520 2024-06-11T19:26:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f30880a1e50 con 0x7f30880a52a0 2024-06-11T19:26:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f30880a2150 con 0x7f3088073830 2024-06-11T19:26:52.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:52.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 10 monclient(hunting): authenticate will time out at 1822.010065s 2024-06-11T19:26:52.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3088073830 0x7f3088067ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3088073830 0x7f3088067ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54534/0 (socket says 172.21.15.98:54534) 2024-06-11T19:26:52.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 1 -- 172.21.15.98:0/250346113 learned_addr learned my addr 172.21.15.98:0/250346113 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:52.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 10 monclient(hunting): get_auth_request con 0x7f3088073830 auth_method 0 2024-06-11T19:26:52.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308d2c7640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30880a52a0 0x7f3088068760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308d2c7640 10 monclient(hunting): get_auth_request con 0x7f30880a52a0 auth_method 0 2024-06-11T19:26:52.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308d2c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308d2c7640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308d2c7640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f3087fff640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f308809d520 0x7f3088068220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f3087fff640 10 monclient(hunting): get_auth_request con 0x7f308809d520 auth_method 0 2024-06-11T19:26:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f3087fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f3087fff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f3087fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308d2c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308d2c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308d2c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f3087fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f3087fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f3087fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 10 monclient(hunting): handle_auth_done global_id 124514 payload 824 2024-06-11T19:26:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 1 -- 172.21.15.98:0/250346113 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30880a52a0 msgr2=0x7f3088068760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30880a52a0 0x7f3088068760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 1 -- 172.21.15.98:0/250346113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f308809d520 msgr2=0x7f3088068220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f308809d520 0x7f3088068220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 10 monclient: _finish_hunting 0 2024-06-11T19:26:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 1 monclient: found mon.noname-e 2024-06-11T19:26:52.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:26:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 1 -- 172.21.15.98:0/250346113 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30880a4120 con 0x7f3088073830 2024-06-11T19:26:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308cac6640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3088073830 0x7f3088067ce0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f307800d5f0 tx=0x7f307800dab0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 10 client.?.objecter ms_handle_connect 0x7f3088073830 2024-06-11T19:26:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 1 -- 172.21.15.98:0/250346113 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3078012a60 con 0x7f3088073830 2024-06-11T19:26:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:26:52.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 10 monclient: dump: 2024-06-11T19:26:52.228 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.228 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.228 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.228 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.228 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.228 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.228 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.228 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.228 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.228 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.229 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.229 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.229 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.229 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.229 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.229 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 10 monclient: _finish_auth 0 2024-06-11T19:26:52.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 10 monclient: _check_auth_tickets 2024-06-11T19:26:52.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:52.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 5 monclient: authenticate success, global_id 124514 2024-06-11T19:26:52.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 1 -- 172.21.15.98:0/250346113 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3078019070 con 0x7f3088073830 2024-06-11T19:26:52.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:52.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 1 -- 172.21.15.98:0/250346113 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3078012c40 con 0x7f3088073830 2024-06-11T19:26:52.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 10 monclient: _renew_subs 2024-06-11T19:26:52.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:52.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 1 -- 172.21.15.98:0/250346113 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3088135080 con 0x7f3088073830 2024-06-11T19:26:52.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f3086ffd640 4 set_mon_vals no callback set 2024-06-11T19:26:52.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:52.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f30857fa640 10 monclient: dump: 2024-06-11T19:26:52.231 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.231 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.231 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.231 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.231 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.231 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.231 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.232 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.232 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.232 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.232 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.232 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.232 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.232 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.232 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.232 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 10 client.124514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:52.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 10 monclient: _renew_subs 2024-06-11T19:26:52.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:52.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 1 -- 172.21.15.98:0/250346113 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f3088121470 con 0x7f3088073830 2024-06-11T19:26:52.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.203+0000 7f308ed2a640 1 librados: init done 2024-06-11T19:26:52.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.204+0000 7f30857fa640 1 -- 172.21.15.98:0/250346113 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f30780125f0 con 0x7f3088073830 2024-06-11T19:26:52.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.204+0000 7f30857fa640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f307403fde0 0x7f30740422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.204+0000 7f30857fa640 1 -- 172.21.15.98:0/250346113 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f307800f040 con 0x7f3088073830 2024-06-11T19:26:52.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.204+0000 7f30857fa640 10 client.124514.objecter ms_dispatch 0x7f3088000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:52.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.204+0000 7f30857fa640 3 client.124514.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:52.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.204+0000 7f30857fa640 3 client.124514.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:52.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.204+0000 7f308ed2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:52.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.204+0000 7f308ed2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:52.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.204+0000 7f308ed2a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-06-11T19:26:52.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.204+0000 7f308ed2a640 1 -- 172.21.15.98:0/250346113 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3088122520 con 0x7f3088073830 2024-06-11T19:26:52.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.204+0000 7f30857fa640 20 client.124514.objecter dump_active .. 0 homeless 2024-06-11T19:26:52.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.204+0000 7f3087fff640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f307403fde0 0x7f30740422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.205+0000 7f3087fff640 10 monclient: get_auth_request con 0x7f307403fde0 auth_method 0 2024-06-11T19:26:52.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.205+0000 7f30857fa640 10 client.124514.objecter ms_handle_connect 0x7f307403fde0 2024-06-11T19:26:52.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.205+0000 7f3087fff640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f307403fde0 0x7f30740422a0 secure :-1 s=READY pgs=4344 cs=0 l=1 rev1=1 crypto rx=0x7f30880a8d40 tx=0x7f307c008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.209+0000 7f30857fa640 1 -- 172.21.15.98:0/250346113 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3078012de0 con 0x7f3088073830 2024-06-11T19:26:52.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.209+0000 7f30857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:52.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.209+0000 7f30857fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:52.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.390+0000 7f308ed2a640 1 -- 172.21.15.98:0/250346113 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f308806d540 con 0x7f307403fde0 2024-06-11T19:26:52.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.391+0000 7f30857fa640 1 -- 172.21.15.98:0/250346113 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f308806d540 con 0x7f307403fde0 2024-06-11T19:26:52.390 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 4 to deep-scrub 2024-06-11T19:26:52.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 10 librados: watch_flush enter 2024-06-11T19:26:52.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 10 librados: watch_flush exit 2024-06-11T19:26:52.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 20 client.124514.objecter shutdown clearing up homeless session... 2024-06-11T19:26:52.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 10 client.124514.objecter successfully canceled tick 2024-06-11T19:26:52.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 -- 172.21.15.98:0/250346113 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f307403fde0 msgr2=0x7f30740422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f307403fde0 0x7f30740422a0 secure :-1 s=READY pgs=4344 cs=0 l=1 rev1=1 crypto rx=0x7f30880a8d40 tx=0x7f307c008040 comp rx=0 tx=0).stop 2024-06-11T19:26:52.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 10 monclient: shutdown 2024-06-11T19:26:52.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:52.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 -- 172.21.15.98:0/250346113 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3088073830 msgr2=0x7f3088067ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3088073830 0x7f3088067ce0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f307800d5f0 tx=0x7f307800dab0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 -- 172.21.15.98:0/250346113 shutdown_connections 2024-06-11T19:26:52.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f307403fde0 0x7f30740422a0 unknown :-1 s=CLOSED pgs=4344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f30880a52a0 0x7f3088068760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f308809d520 0x7f3088068220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 --2- 172.21.15.98:0/250346113 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3088073830 0x7f3088067ce0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 -- 172.21.15.98:0/250346113 >> 172.21.15.98:0/250346113 conn(0x7f30880a4780 msgr2=0x7f30881bcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:52.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 -- 172.21.15.98:0/250346113 shutdown_connections 2024-06-11T19:26:52.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 -- 172.21.15.98:0/250346113 wait complete. 2024-06-11T19:26:52.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.393+0000 7f308ed2a640 1 librados: shutdown 2024-06-11T19:26:52.403 INFO:tasks.ceph:Scrubbing osd.5 2024-06-11T19:26:52.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-06-11T19:26:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.506+0000 7f7d88385640 10 monclient: get_monmap_and_config 2024-06-11T19:26:52.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.506+0000 7f7d88385640 10 monclient: build_initial_monmap 2024-06-11T19:26:52.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.506+0000 7f7d88385640 10 monclient: monmap: 2024-06-11T19:26:52.506 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:52.506 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.506 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:52.507805+0000 2024-06-11T19:26:52.506 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:52.507805+0000 2024-06-11T19:26:52.506 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:52.508 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:52.508 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:52.508 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:52.508 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:52.508 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.507+0000 7f7d88385640 1 Processor -- start 2024-06-11T19:26:52.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.507+0000 7f7d88385640 1 -- start start 2024-06-11T19:26:52.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.507+0000 7f7d88385640 10 monclient: init 2024-06-11T19:26:52.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.507+0000 7f7d88385640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:52.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.507+0000 7f7d88385640 10 monclient: _add_conns ranks=[8,6,0,7,2,3,5,10,1,9,4] 2024-06-11T19:26:52.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.507+0000 7f7d88385640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d80135b60 0x7f7d80135f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.507+0000 7f7d88385640 10 monclient(hunting): picked mon.noname-e con 0x7f7d80135b60 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:52.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d88385640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7d800a5a10 0x7f7d800a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d88385640 10 monclient(hunting): picked mon.noname-d con 0x7f7d800a5a10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:52.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d88385640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d800a6350 addr v1:172.21.15.98:6789/0 2024-06-11T19:26:52.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d88385640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d800a6800 con 0x7f7d800a6350 2024-06-11T19:26:52.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d88385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d88385640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f7d800a8580 con 0x7f7d800a5a10 2024-06-11T19:26:52.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d88385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d88385640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f7d800a2c60 con 0x7f7d80135b60 2024-06-11T19:26:52.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d88385640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:52.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d88385640 10 monclient(hunting): authenticate will time out at 1822.315050s 2024-06-11T19:26:52.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d85920640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7d800a5a10 0x7f7d800a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d80135b60 0x7f7d80135f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d85920640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7d800a5a10 0x7f7d800a5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58976/0 (socket says 172.21.15.98:58976) 2024-06-11T19:26:52.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d80135b60 0x7f7d80135f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54550/0 (socket says 172.21.15.98:54550) 2024-06-11T19:26:52.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d85920640 1 -- 172.21.15.98:0/4117851161 learned_addr learned my addr 172.21.15.98:0/4117851161 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:52.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86922640 1 --1- 172.21.15.98:0/4117851161 >> v1:172.21.15.98:6789/0 conn(0x7f7d800a6350 0x7f7d8013fce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51442/0 (socket says 172.21.15.98:51442) 2024-06-11T19:26:52.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d85920640 10 monclient(hunting): get_auth_request con 0x7f7d800a5a10 auth_method 0 2024-06-11T19:26:52.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d85920640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d85920640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d85920640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 10 monclient(hunting): get_auth_request con 0x7f7d80135b60 auth_method 0 2024-06-11T19:26:52.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d85920640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d85920640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d85920640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 10 monclient(hunting): handle_auth_done global_id 124523 payload 274 2024-06-11T19:26:52.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d8511f640 1 -- 172.21.15.98:0/4117851161 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3294959517 0 0) 0x7f7d800a6800 con 0x7f7d800a6350 2024-06-11T19:26:52.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 1 -- 172.21.15.98:0/4117851161 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7d800a5a10 msgr2=0x7f7d800a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 1 --2- 172.21.15.98:0/4117851161 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7d800a5a10 0x7f7d800a5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 1 -- 172.21.15.98:0/4117851161 >> v1:172.21.15.98:6789/0 conn(0x7f7d800a6350 legacy=0x7f7d8013fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 10 monclient: _finish_hunting 0 2024-06-11T19:26:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 1 monclient: found mon.noname-e 2024-06-11T19:26:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:26:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d86121640 1 -- 172.21.15.98:0/4117851161 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d800a2f40 con 0x7f7d80135b60 2024-06-11T19:26:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.508+0000 7f7d8511f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-06-11T19:26:52.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d86121640 1 --2- 172.21.15.98:0/4117851161 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d80135b60 0x7f7d80135f40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009dc0 tx=0x7f7d7802bd50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=5e0bed20c9cbd59b server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 1 -- 172.21.15.98:0/4117851161 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d78039070 con 0x7f7d80135b60 2024-06-11T19:26:52.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:26:52.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 10 monclient: dump: 2024-06-11T19:26:52.523 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.523 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.523 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.523 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.524 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.524 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.524 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.524 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.524 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.524 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.524 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.524 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.525 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.525 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.525 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.525 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 10 monclient: _finish_auth 0 2024-06-11T19:26:52.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 10 monclient: _check_auth_tickets 2024-06-11T19:26:52.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:52.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d88385640 5 monclient: authenticate success, global_id 124523 2024-06-11T19:26:52.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 1 -- 172.21.15.98:0/4117851161 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d7802cd70 con 0x7f7d80135b60 2024-06-11T19:26:52.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d88385640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:52.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:52.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d88385640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:52.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d88385640 4 set_mon_vals no callback set 2024-06-11T19:26:52.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 1 -- 172.21.15.98:0/4117851161 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d780327b0 con 0x7f7d80135b60 2024-06-11T19:26:52.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:52.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d8511f640 10 monclient: dump: 2024-06-11T19:26:52.527 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.527 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.527 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.527 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.527 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.527 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.527 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.527 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.528 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.528 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.528 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.528 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.528 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.528 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.528 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.528 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d88385640 10 monclient: shutdown 2024-06-11T19:26:52.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d88385640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:52.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d88385640 1 -- 172.21.15.98:0/4117851161 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d80135b60 msgr2=0x7f7d80135f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.509+0000 7f7d88385640 1 --2- 172.21.15.98:0/4117851161 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d80135b60 0x7f7d80135f40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009dc0 tx=0x7f7d7802bd50 comp rx=0 tx=0).stop 2024-06-11T19:26:52.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.510+0000 7f7d88385640 1 -- 172.21.15.98:0/4117851161 shutdown_connections 2024-06-11T19:26:52.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.510+0000 7f7d88385640 1 --2- 172.21.15.98:0/4117851161 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7d800a5a10 0x7f7d800a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.510+0000 7f7d88385640 1 --2- 172.21.15.98:0/4117851161 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7d80135b60 0x7f7d80135f40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.510+0000 7f7d88385640 1 -- 172.21.15.98:0/4117851161 >> 172.21.15.98:0/4117851161 conn(0x7f7d800aba20 msgr2=0x7f7d800a7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:52.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.510+0000 7f7d88385640 1 -- 172.21.15.98:0/4117851161 shutdown_connections 2024-06-11T19:26:52.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.510+0000 7f7d88385640 1 -- 172.21.15.98:0/4117851161 wait complete. 2024-06-11T19:26:52.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.510+0000 7f7d88385640 10 monclient: build_initial_monmap 2024-06-11T19:26:52.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.510+0000 7f7d88385640 10 monclient: monmap: 2024-06-11T19:26:52.530 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:52.530 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:52.530 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:52.531 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:52.531 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:52.531 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:52.531 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:52.531 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:52.531 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:52.531 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:52.531 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:52.532 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:52.532 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:52.532 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:52.532 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:52.532 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.510+0000 7f7d88385640 1 librados: starting msgr at 2024-06-11T19:26:52.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.510+0000 7f7d88385640 1 librados: starting objecter 2024-06-11T19:26:52.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.510+0000 7f7d88385640 1 Processor -- start 2024-06-11T19:26:52.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 1 -- start start 2024-06-11T19:26:52.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 1 librados: setting wanted keys 2024-06-11T19:26:52.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 1 librados: calling monclient init 2024-06-11T19:26:52.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 10 monclient: init 2024-06-11T19:26:52.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:52.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 10 monclient: _add_conns ranks=[2,7,0,3,8,6,1,4,5] 2024-06-11T19:26:52.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d800a5a10 0x7f7d801cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 10 monclient(hunting): picked mon.noname-b con 0x7f7d800a5a10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:52.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7d800a6350 0x7f7d80134b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 10 monclient(hunting): picked mon.noname-i con 0x7f7d800a6350 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:52.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d80135b60 0x7f7d801350c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 10 monclient(hunting): picked mon.noname-a con 0x7f7d80135b60 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-06-11T19:26:52.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d800a6f50 con 0x7f7d80135b60 2024-06-11T19:26:52.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7d80050490 con 0x7f7d800a5a10 2024-06-11T19:26:52.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f7d800a7290 con 0x7f7d800a6350 2024-06-11T19:26:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d88385640 10 monclient(hunting): authenticate will time out at 1822.318433s 2024-06-11T19:26:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86121640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d800a5a10 0x7f7d801cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d80135b60 0x7f7d801350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86121640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d800a5a10 0x7f7d801cd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:43188/0 (socket says 172.21.15.98:43188) 2024-06-11T19:26:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86121640 1 -- 172.21.15.98:0/1985384734 learned_addr learned my addr 172.21.15.98:0/1985384734 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d80135b60 0x7f7d801350c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41892/0 (socket says 172.21.15.98:41892) 2024-06-11T19:26:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86121640 10 monclient(hunting): get_auth_request con 0x7f7d800a5a10 auth_method 0 2024-06-11T19:26:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86121640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86121640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86121640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 10 monclient(hunting): get_auth_request con 0x7f7d80135b60 auth_method 0 2024-06-11T19:26:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86121640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86121640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86121640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 10 monclient(hunting): handle_auth_done global_id 66510 payload 824 2024-06-11T19:26:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d85920640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7d800a6350 0x7f7d80134b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 1 -- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7d800a6350 msgr2=0x7f7d80134b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7d800a6350 0x7f7d80134b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 1 -- 172.21.15.98:0/1985384734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d800a5a10 msgr2=0x7f7d801cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d800a5a10 0x7f7d801cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 10 monclient: _finish_hunting 0 2024-06-11T19:26:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 1 monclient: found mon.noname-a 2024-06-11T19:26:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:26:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 1 -- 172.21.15.98:0/1985384734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d80122540 con 0x7f7d80135b60 2024-06-11T19:26:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d85920640 10 monclient: get_auth_request con 0x7f7d800a6350 auth_method 0 2024-06-11T19:26:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d85920640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f7d800a6350 0x7f7d80134b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:26:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86121640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7d800a5a10 0x7f7d801cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d86922640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d80135b60 0x7f7d801350c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c00c370 tx=0x7f7d7c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d76ffd640 10 client.?.objecter ms_handle_connect 0x7f7d80135b60 2024-06-11T19:26:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.511+0000 7f7d76ffd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 1 -- 172.21.15.98:0/1985384734 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d7c0047e0 con 0x7f7d80135b60 2024-06-11T19:26:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:26:52.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 10 monclient: dump: 2024-06-11T19:26:52.540 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.540 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.540 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.540 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.540 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.540 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.540 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.540 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.541 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.541 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.541 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.541 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.541 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.541 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.541 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.541 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 10 monclient: _finish_auth 0 2024-06-11T19:26:52.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 10 monclient: _check_auth_tickets 2024-06-11T19:26:52.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:52.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 5 monclient: authenticate success, global_id 66510 2024-06-11T19:26:52.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 1 -- 172.21.15.98:0/1985384734 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d7c0049c0 con 0x7f7d80135b60 2024-06-11T19:26:52.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 10 monclient: _renew_subs 2024-06-11T19:26:52.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:52.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 1 -- 172.21.15.98:0/1985384734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d800a0ec0 con 0x7f7d80135b60 2024-06-11T19:26:52.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:52.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 1 -- 172.21.15.98:0/1985384734 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d7c0133c0 con 0x7f7d80135b60 2024-06-11T19:26:52.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:52.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 10 client.66510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:52.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d8491e640 4 set_mon_vals no callback set 2024-06-11T19:26:52.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d76ffd640 10 monclient: dump: 2024-06-11T19:26:52.543 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.543 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.543 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.544 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.544 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.544 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.544 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.544 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.544 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.544 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.544 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.545 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.545 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.545 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.545 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.545 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 10 monclient: _renew_subs 2024-06-11T19:26:52.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:52.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 1 -- 172.21.15.98:0/1985384734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d800a1410 con 0x7f7d80135b60 2024-06-11T19:26:52.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 1 librados: init done 2024-06-11T19:26:52.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 10 client.66510.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:52.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 15 client.66510.objecter _session_command_op_assign -1 1 2024-06-11T19:26:52.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 20 client.66510.objecter _get_session osd=-1 returning homeless 2024-06-11T19:26:52.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 10 client.66510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:52.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-06-11T19:26:52.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.512+0000 7f7d88385640 1 -- 172.21.15.98:0/1985384734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7d801cda00 con 0x7f7d80135b60 2024-06-11T19:26:52.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 1 -- 172.21.15.98:0/1985384734 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f7d7c011070 con 0x7f7d80135b60 2024-06-11T19:26:52.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7d6403fde0 0x7f7d640422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 1 -- 172.21.15.98:0/1985384734 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f7d7c051270 con 0x7f7d80135b60 2024-06-11T19:26:52.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 10 client.66510.objecter ms_dispatch 0x7f7d80000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:52.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 3 client.66510.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:52.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 3 client.66510.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:52.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 10 client.66510.objecter checking command 1 2024-06-11T19:26:52.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] conn(0x7f7d64047620 0x7f7d64049ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 20 client.66510.objecter _get_session s=0x7f7d64046f50 osd=5 3 2024-06-11T19:26:52.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 20 client.66510.objecter put_session s=0x7f7d64046f50 osd=5 3 2024-06-11T19:26:52.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 15 client.66510.objecter _session_command_op_remove -1 1 2024-06-11T19:26:52.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 20 client.66510.objecter _get_session s=0x7f7d64046f50 osd=5 3 2024-06-11T19:26:52.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 20 client.66510.objecter get_session s=0x7f7d64046f50 osd=5 3 2024-06-11T19:26:52.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 15 client.66510.objecter _session_command_op_assign 5 1 2024-06-11T19:26:52.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 20 client.66510.objecter put_session s=0x7f7d64046f50 osd=5 4 2024-06-11T19:26:52.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 10 client.66510.objecter _send_command 1 2024-06-11T19:26:52.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 1 -- 172.21.15.98:0/1985384734 --> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d7c011e70 con 0x7f7d64047620 2024-06-11T19:26:52.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 20 client.66510.objecter dump_active .. 0 homeless 2024-06-11T19:26:52.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 1 -- 172.21.15.98:0/1985384734 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=685) ==== 24+0+0 (secure 0 0 0) 0x7f7d7c058050 con 0x7f7d80135b60 2024-06-11T19:26:52.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d76ffd640 10 monclient: handle_get_version_reply finishing 1 version 685 2024-06-11T19:26:52.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d86121640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7d6403fde0 0x7f7d640422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d86121640 10 monclient: get_auth_request con 0x7f7d6403fde0 auth_method 0 2024-06-11T19:26:52.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d85920640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] conn(0x7f7d64047620 0x7f7d64049ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.513+0000 7f7d85920640 10 monclient: get_auth_request con 0x7f7d64047620 auth_method 0 2024-06-11T19:26:52.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.514+0000 7f7d76ffd640 10 client.66510.objecter ms_handle_connect 0x7f7d6403fde0 2024-06-11T19:26:52.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.514+0000 7f7d86121640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7d6403fde0 0x7f7d640422a0 secure :-1 s=READY pgs=4345 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000f80 tx=0x7f7d78030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.514+0000 7f7d76ffd640 10 client.66510.objecter ms_handle_connect 0x7f7d64047620 2024-06-11T19:26:52.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.514+0000 7f7d85920640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] conn(0x7f7d64047620 0x7f7d64049ac0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.514+0000 7f7d76ffd640 1 -- 172.21.15.98:0/1985384734 <== osd.5 v2:172.21.15.117:6816/987023138 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d7c011e70 con 0x7f7d64047620 2024-06-11T19:26:52.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.514+0000 7f7d76ffd640 10 client.66510.objecter ms_dispatch 0x7f7d80000b90 command_reply(tid 1: 0 ) 2024-06-11T19:26:52.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.514+0000 7f7d76ffd640 10 client.66510.objecter _finish_command 1 = system:0 2024-06-11T19:26:52.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.514+0000 7f7d76ffd640 20 client.66510.objecter put_session s=0x7f7d64046f50 osd=5 4 2024-06-11T19:26:52.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.514+0000 7f7d76ffd640 15 client.66510.objecter _session_command_op_remove 5 1 2024-06-11T19:26:52.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.527+0000 7f7d88385640 10 client.66510.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-11T19:26:52.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.527+0000 7f7d88385640 15 client.66510.objecter _session_command_op_assign -1 2 2024-06-11T19:26:52.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.527+0000 7f7d88385640 20 client.66510.objecter _get_session s=0x7f7d64046f50 osd=5 3 2024-06-11T19:26:52.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.527+0000 7f7d88385640 20 client.66510.objecter put_session s=0x7f7d64046f50 osd=5 3 2024-06-11T19:26:52.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.527+0000 7f7d88385640 20 client.66510.objecter _get_session s=0x7f7d64046f50 osd=5 3 2024-06-11T19:26:52.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.527+0000 7f7d88385640 15 client.66510.objecter _session_command_op_remove -1 2 2024-06-11T19:26:52.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.527+0000 7f7d88385640 20 client.66510.objecter get_session s=0x7f7d64046f50 osd=5 3 2024-06-11T19:26:52.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.527+0000 7f7d88385640 15 client.66510.objecter _session_command_op_assign 5 2 2024-06-11T19:26:52.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.527+0000 7f7d88385640 20 client.66510.objecter put_session s=0x7f7d64046f50 osd=5 4 2024-06-11T19:26:52.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.527+0000 7f7d88385640 10 client.66510.objecter _send_command 2 2024-06-11T19:26:52.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.527+0000 7f7d88385640 1 -- 172.21.15.98:0/1985384734 --> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7d800a53a0 con 0x7f7d64047620 2024-06-11T19:26:52.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d76ffd640 1 -- 172.21.15.98:0/1985384734 <== osd.5 v2:172.21.15.117:6816/987023138 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7d800a53a0 con 0x7f7d64047620 2024-06-11T19:26:52.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d76ffd640 10 client.66510.objecter ms_dispatch 0x7f7d80000b90 command_reply(tid 2: 0 ) 2024-06-11T19:26:52.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d76ffd640 10 client.66510.objecter _finish_command 2 = system:0 2024-06-11T19:26:52.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d76ffd640 20 client.66510.objecter put_session s=0x7f7d64046f50 osd=5 4 2024-06-11T19:26:52.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d76ffd640 15 client.66510.objecter _session_command_op_remove 5 2 2024-06-11T19:26:52.553 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-06-11T19:26:52.553 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-11T19:26:52.553 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-06-11T19:26:52.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d88385640 10 librados: watch_flush enter 2024-06-11T19:26:52.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d88385640 10 librados: watch_flush exit 2024-06-11T19:26:52.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d88385640 10 client.66510.objecter close_session for osd.5 2024-06-11T19:26:52.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d88385640 1 -- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] conn(0x7f7d64047620 msgr2=0x7f7d64049ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d88385640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:6816/987023138,v1:172.21.15.117:6817/987023138] conn(0x7f7d64047620 0x7f7d64049ac0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d88385640 20 client.66510.objecter put_session s=0x7f7d64046f50 osd=5 1 2024-06-11T19:26:52.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d88385640 20 client.66510.objecter shutdown clearing up homeless session... 2024-06-11T19:26:52.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d88385640 10 client.66510.objecter successfully canceled tick 2024-06-11T19:26:52.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d88385640 1 -- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7d6403fde0 msgr2=0x7f7d640422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.528+0000 7f7d88385640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f7d6403fde0 0x7f7d640422a0 secure :-1 s=READY pgs=4345 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000f80 tx=0x7f7d78030000 comp rx=0 tx=0).stop 2024-06-11T19:26:52.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.529+0000 7f7d88385640 10 monclient: shutdown 2024-06-11T19:26:52.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.529+0000 7f7d88385640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:52.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.529+0000 7f7d88385640 1 -- 172.21.15.98:0/1985384734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d80135b60 msgr2=0x7f7d801350c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.529+0000 7f7d88385640 1 --2- 172.21.15.98:0/1985384734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d80135b60 0x7f7d801350c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c00c370 tx=0x7f7d7c007f90 comp rx=0 tx=0).stop 2024-06-11T19:26:52.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.529+0000 7f7d86922640 1 -- 172.21.15.98:0/1985384734 reap_dead start 2024-06-11T19:26:52.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.529+0000 7f7d88385640 1 -- 172.21.15.98:0/1985384734 shutdown_connections 2024-06-11T19:26:52.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.529+0000 7f7d88385640 1 -- 172.21.15.98:0/1985384734 >> 172.21.15.98:0/1985384734 conn(0x7f7d800aba20 msgr2=0x7f7d801c0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:52.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.529+0000 7f7d88385640 1 -- 172.21.15.98:0/1985384734 shutdown_connections 2024-06-11T19:26:52.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.529+0000 7f7d88385640 1 -- 172.21.15.98:0/1985384734 wait complete. 2024-06-11T19:26:52.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.529+0000 7f7d88385640 1 librados: shutdown 2024-06-11T19:26:52.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-06-11T19:26:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.658+0000 7f2ba633f640 10 monclient: get_monmap_and_config 2024-06-11T19:26:52.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.658+0000 7f2ba633f640 10 monclient: build_initial_monmap 2024-06-11T19:26:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.658+0000 7f2ba633f640 10 monclient: monmap: 2024-06-11T19:26:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:52.659489+0000 2024-06-11T19:26:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:52.659489+0000 2024-06-11T19:26:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.658 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:52.659 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:52.660 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 1 Processor -- start 2024-06-11T19:26:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 1 -- start start 2024-06-11T19:26:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 10 monclient: init 2024-06-11T19:26:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 10 monclient: _add_conns ranks=[3,8,9,4,10,0,2,6,1,5,7] 2024-06-11T19:26:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ba00a4920 0x7f2ba00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 10 monclient(hunting): picked mon.noname-g con 0x7f2ba00a4920 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ba00a5240 0x7f2ba00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 10 monclient(hunting): picked mon.noname-e con 0x7f2ba00a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2ba00a2530 0x7f2ba00a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 10 monclient(hunting): picked mon.noname-a con 0x7f2ba00a2530 addr v2:172.21.15.98:3300/0 2024-06-11T19:26:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2ba00a7140 con 0x7f2ba00a2530 2024-06-11T19:26:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f2ba00a2b50 con 0x7f2ba00a4920 2024-06-11T19:26:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2ba00a2e30 con 0x7f2ba00a5240 2024-06-11T19:26:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba633f640 10 monclient(hunting): authenticate will time out at 1822.466830s 2024-06-11T19:26:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba48dc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2ba00a2530 0x7f2ba00a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2b9f7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ba00a5240 0x7f2ba00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2b9f7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ba00a5240 0x7f2ba00a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:54558/0 (socket says 172.21.15.98:54558) 2024-06-11T19:26:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba48dc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2ba00a2530 0x7f2ba00a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41906/0 (socket says 172.21.15.98:41906) 2024-06-11T19:26:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2b9f7fe640 1 -- 172.21.15.98:0/3219971598 learned_addr learned my addr 172.21.15.98:0/3219971598 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba48dc640 10 monclient(hunting): get_auth_request con 0x7f2ba00a2530 auth_method 0 2024-06-11T19:26:52.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba48dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba48dc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2ba48dc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2b9f7fe640 10 monclient(hunting): get_auth_request con 0x7f2ba00a5240 auth_method 0 2024-06-11T19:26:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.659+0000 7f2b9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2b9f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2b9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2b9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2b9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2b9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2b9ffff640 1 --2- 172.21.15.98:0/3219971598 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ba00a4920 0x7f2ba00a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 10 monclient(hunting): handle_auth_done global_id 66519 payload 274 2024-06-11T19:26:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 1 -- 172.21.15.98:0/3219971598 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ba00a5240 msgr2=0x7f2ba00a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 1 --2- 172.21.15.98:0/3219971598 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ba00a5240 0x7f2ba00a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 1 -- 172.21.15.98:0/3219971598 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ba00a4920 msgr2=0x7f2ba00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:52.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 1 --2- 172.21.15.98:0/3219971598 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ba00a4920 0x7f2ba00a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 10 monclient: _finish_hunting 0 2024-06-11T19:26:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 1 monclient: found mon.noname-a 2024-06-11T19:26:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-06-11T19:26:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 1 -- 172.21.15.98:0/3219971598 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba00a3110 con 0x7f2ba00a2530 2024-06-11T19:26:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.660+0000 7f2ba48dc640 1 --2- 172.21.15.98:0/3219971598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba00a2530 0x7f2ba00a5ee0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009d90 tx=0x7f2b9802bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdc43a5044ac3fd server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 1 -- 172.21.15.98:0/3219971598 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b98039070 con 0x7f2ba00a2530 2024-06-11T19:26:52.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-11T19:26:52.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 10 monclient: dump: 2024-06-11T19:26:52.675 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.675 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.675 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.675 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.675 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.675 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.675 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.676 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.676 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.676 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.676 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.676 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.676 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.676 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.676 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.677 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 10 monclient: _finish_auth 0 2024-06-11T19:26:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 10 monclient: _check_auth_tickets 2024-06-11T19:26:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 1 -- 172.21.15.98:0/3219971598 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b9802cd00 con 0x7f2ba00a2530 2024-06-11T19:26:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 5 monclient: authenticate success, global_id 66519 2024-06-11T19:26:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:52.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 1 -- 172.21.15.98:0/3219971598 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b980327b0 con 0x7f2ba00a2530 2024-06-11T19:26:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 4 set_mon_vals no callback set 2024-06-11T19:26:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-11T19:26:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2b9effd640 10 monclient: dump: 2024-06-11T19:26:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.678 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.679 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.680 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.680 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.680 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.680 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 10 monclient: shutdown 2024-06-11T19:26:52.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:52.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 1 -- 172.21.15.98:0/3219971598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba00a2530 msgr2=0x7f2ba00a5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 1 --2- 172.21.15.98:0/3219971598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba00a2530 0x7f2ba00a5ee0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009d90 tx=0x7f2b9802bce0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 1 -- 172.21.15.98:0/3219971598 shutdown_connections 2024-06-11T19:26:52.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 1 --2- 172.21.15.98:0/3219971598 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ba00a5240 0x7f2ba00a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 1 --2- 172.21.15.98:0/3219971598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ba00a2530 0x7f2ba00a5ee0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 1 --2- 172.21.15.98:0/3219971598 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ba00a4920 0x7f2ba00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 1 -- 172.21.15.98:0/3219971598 >> 172.21.15.98:0/3219971598 conn(0x7f2ba00ab850 msgr2=0x7f2ba00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:52.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 1 -- 172.21.15.98:0/3219971598 shutdown_connections 2024-06-11T19:26:52.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.661+0000 7f2ba633f640 1 -- 172.21.15.98:0/3219971598 wait complete. 2024-06-11T19:26:52.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.662+0000 7f2ba633f640 10 monclient: build_initial_monmap 2024-06-11T19:26:52.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.662+0000 7f2ba633f640 10 monclient: monmap: 2024-06-11T19:26:52.681 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:52.682 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:52.682 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:52.682 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:52.682 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:52.682 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:52.682 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:52.682 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:52.682 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:52.683 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:52.683 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:52.683 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:52.683 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:52.683 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:52.683 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:52.683 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.662+0000 7f2ba633f640 1 librados: starting msgr at 2024-06-11T19:26:52.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.662+0000 7f2ba633f640 1 librados: starting objecter 2024-06-11T19:26:52.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.662+0000 7f2ba633f640 1 Processor -- start 2024-06-11T19:26:52.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.662+0000 7f2ba633f640 1 -- start start 2024-06-11T19:26:52.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.662+0000 7f2ba633f640 1 librados: setting wanted keys 2024-06-11T19:26:52.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.662+0000 7f2ba633f640 1 librados: calling monclient init 2024-06-11T19:26:52.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.662+0000 7f2ba633f640 10 monclient: init 2024-06-11T19:26:52.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:52.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 10 monclient: _add_conns ranks=[3,4,6,0,1,8,5,2,7] 2024-06-11T19:26:52.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ba00a2530 0x7f2ba01cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 10 monclient(hunting): picked mon.noname-g con 0x7f2ba00a2530 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:52.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ba00a4920 0x7f2ba01251e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 10 monclient(hunting): picked mon.noname-c con 0x7f2ba00a4920 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:52.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ba00a5240 0x7f2ba0125720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 10 monclient(hunting): picked mon.noname-d con 0x7f2ba00a5240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:26:52.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f2ba00a6da0 con 0x7f2ba00a2530 2024-06-11T19:26:52.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2ba0138e50 con 0x7f2ba00a4920 2024-06-11T19:26:52.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f2ba00a6420 con 0x7f2ba00a5240 2024-06-11T19:26:52.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:52.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba633f640 10 monclient(hunting): authenticate will time out at 1822.470352s 2024-06-11T19:26:52.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ba00a4920 0x7f2ba01251e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba48dc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ba00a5240 0x7f2ba0125720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ba00a4920 0x7f2ba01251e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:37302/0 (socket says 172.21.15.98:37302) 2024-06-11T19:26:52.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 1 -- 172.21.15.98:0/926781318 learned_addr learned my addr 172.21.15.98:0/926781318 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:52.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 10 monclient(hunting): get_auth_request con 0x7f2ba00a4920 auth_method 0 2024-06-11T19:26:52.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba48dc640 10 monclient(hunting): get_auth_request con 0x7f2ba00a5240 auth_method 0 2024-06-11T19:26:52.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba48dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba48dc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba48dc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba48dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba48dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9ffff640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ba00a2530 0x7f2ba01cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba48dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9ffff640 10 monclient(hunting): get_auth_request con 0x7f2ba00a2530 auth_method 0 2024-06-11T19:26:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 10 monclient(hunting): handle_auth_done global_id 134239 payload 824 2024-06-11T19:26:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 1 -- 172.21.15.98:0/926781318 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ba00a5240 msgr2=0x7f2ba0125720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ba00a5240 0x7f2ba0125720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 1 -- 172.21.15.98:0/926781318 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ba00a2530 msgr2=0x7f2ba01cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ba00a2530 0x7f2ba01cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 10 monclient: _finish_hunting 0 2024-06-11T19:26:52.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 1 monclient: found mon.noname-c 2024-06-11T19:26:52.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:52.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-06-11T19:26:52.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 1 -- 172.21.15.98:0/926781318 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba0128ec0 con 0x7f2ba00a4920 2024-06-11T19:26:52.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2ba48dc640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ba00a5240 0x7f2ba0125720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:52.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9f7fe640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ba00a4920 0x7f2ba01251e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400e840 tx=0x7f2b9400ed00 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9cff9640 10 client.?.objecter ms_handle_connect 0x7f2ba00a4920 2024-06-11T19:26:52.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.663+0000 7f2b9cff9640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:52.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 1 -- 172.21.15.98:0/926781318 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b94017070 con 0x7f2ba00a4920 2024-06-11T19:26:52.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-11T19:26:52.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 10 monclient: dump: 2024-06-11T19:26:52.694 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.695 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.695 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.695 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.695 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.695 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.695 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.695 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.695 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.696 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.696 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.696 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.696 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.696 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.696 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.696 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 10 monclient: _finish_auth 0 2024-06-11T19:26:52.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 10 monclient: _check_auth_tickets 2024-06-11T19:26:52.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:52.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 1 -- 172.21.15.98:0/926781318 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b9400cdb0 con 0x7f2ba00a4920 2024-06-11T19:26:52.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2ba633f640 5 monclient: authenticate success, global_id 134239 2024-06-11T19:26:52.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2ba633f640 10 monclient: _renew_subs 2024-06-11T19:26:52.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2ba633f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:52.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2ba633f640 1 -- 172.21.15.98:0/926781318 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba0127350 con 0x7f2ba00a4920 2024-06-11T19:26:52.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:52.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 1 -- 172.21.15.98:0/926781318 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b940138e0 con 0x7f2ba00a4920 2024-06-11T19:26:52.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-11T19:26:52.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9cff9640 10 monclient: dump: 2024-06-11T19:26:52.698 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.698 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.698 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.698 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.698 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.698 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.699 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.699 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.699 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.699 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.699 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.699 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.699 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.699 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.700 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2ba633f640 10 client.134239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2b9e7fc640 4 set_mon_vals no callback set 2024-06-11T19:26:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2ba633f640 10 monclient: _renew_subs 2024-06-11T19:26:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2ba633f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2ba633f640 1 -- 172.21.15.98:0/926781318 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba00a9030 con 0x7f2ba00a4920 2024-06-11T19:26:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.664+0000 7f2ba633f640 1 librados: init done 2024-06-11T19:26:52.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2b9cff9640 1 -- 172.21.15.98:0/926781318 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f2b9401a070 con 0x7f2ba00a4920 2024-06-11T19:26:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2ba633f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2ba633f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2ba633f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-06-11T19:26:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2ba633f640 1 -- 172.21.15.98:0/926781318 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ba00aa760 con 0x7f2ba00a4920 2024-06-11T19:26:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2b9cff9640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2b8c03fdb0 0x7f2b8c042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2b9cff9640 1 -- 172.21.15.98:0/926781318 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f2b9400f040 con 0x7f2ba00a4920 2024-06-11T19:26:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2b9cff9640 10 client.134239.objecter ms_dispatch 0x7f2ba0000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:52.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2b9cff9640 3 client.134239.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2b9cff9640 3 client.134239.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2b9cff9640 20 client.134239.objecter dump_active .. 0 homeless 2024-06-11T19:26:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2b9ffff640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2b8c03fdb0 0x7f2b8c042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.665+0000 7f2b9ffff640 10 monclient: get_auth_request con 0x7f2b8c03fdb0 auth_method 0 2024-06-11T19:26:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.666+0000 7f2b9cff9640 10 client.134239.objecter ms_handle_connect 0x7f2b8c03fdb0 2024-06-11T19:26:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.666+0000 7f2b9ffff640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2b8c03fdb0 0x7f2b8c042270 secure :-1 s=READY pgs=4346 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000aac0 tx=0x7f2b90008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.669+0000 7f2b9cff9640 1 -- 172.21.15.98:0/926781318 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2ba00aa760 con 0x7f2ba00a4920 2024-06-11T19:26:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.669+0000 7f2b9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:52.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.669+0000 7f2b9cff9640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:52.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.853+0000 7f2ba633f640 1 -- 172.21.15.98:0/926781318 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f2ba01387f0 con 0x7f2b8c03fdb0 2024-06-11T19:26:52.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.853+0000 7f2b9cff9640 1 -- 172.21.15.98:0/926781318 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2ba01387f0 con 0x7f2b8c03fdb0 2024-06-11T19:26:52.853 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 5 to deep-scrub 2024-06-11T19:26:52.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 10 librados: watch_flush enter 2024-06-11T19:26:52.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 10 librados: watch_flush exit 2024-06-11T19:26:52.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 20 client.134239.objecter shutdown clearing up homeless session... 2024-06-11T19:26:52.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 10 client.134239.objecter successfully canceled tick 2024-06-11T19:26:52.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 1 -- 172.21.15.98:0/926781318 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2b8c03fdb0 msgr2=0x7f2b8c042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2b8c03fdb0 0x7f2b8c042270 secure :-1 s=READY pgs=4346 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000aac0 tx=0x7f2b90008040 comp rx=0 tx=0).stop 2024-06-11T19:26:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 10 monclient: shutdown 2024-06-11T19:26:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 1 -- 172.21.15.98:0/926781318 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ba00a4920 msgr2=0x7f2ba01251e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ba00a4920 0x7f2ba01251e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400e840 tx=0x7f2b9400ed00 comp rx=0 tx=0).stop 2024-06-11T19:26:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 1 -- 172.21.15.98:0/926781318 shutdown_connections 2024-06-11T19:26:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f2b8c03fdb0 0x7f2b8c042270 unknown :-1 s=CLOSED pgs=4346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ba00a5240 0x7f2ba0125720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ba00a4920 0x7f2ba01251e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 1 --2- 172.21.15.98:0/926781318 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ba00a2530 0x7f2ba01cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 1 -- 172.21.15.98:0/926781318 >> 172.21.15.98:0/926781318 conn(0x7f2ba00ab850 msgr2=0x7f2ba009c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 1 -- 172.21.15.98:0/926781318 shutdown_connections 2024-06-11T19:26:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.855+0000 7f2ba633f640 1 -- 172.21.15.98:0/926781318 wait complete. 2024-06-11T19:26:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.856+0000 7f2ba633f640 1 librados: shutdown 2024-06-11T19:26:52.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-11T19:26:52.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.965+0000 7f3efd3cd640 10 monclient: get_monmap_and_config 2024-06-11T19:26:52.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.966+0000 7f3efd3cd640 10 monclient: build_initial_monmap 2024-06-11T19:26:52.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.966+0000 7f3efd3cd640 10 monclient: monmap: 2024-06-11T19:26:52.965 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:52.965 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.965 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:26:52.967047+0000 2024-06-11T19:26:52.965 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:26:52.967047+0000 2024-06-11T19:26:52.965 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:52.966 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.966 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:26:52.966 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:26:52.966 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:52.966 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:52.966 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:52.966 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:52.966 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:52.967 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:52.967 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:52.967 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:26:52.967 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:26:52.967 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.966+0000 7f3efd3cd640 1 Processor -- start 2024-06-11T19:26:52.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.966+0000 7f3efd3cd640 1 -- start start 2024-06-11T19:26:52.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.966+0000 7f3efd3cd640 10 monclient: init 2024-06-11T19:26:52.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:52.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 10 monclient: _add_conns ranks=[8,4,7,1,9,10,0,6,3,5,2] 2024-06-11T19:26:52.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ef80a4920 0x7f3ef80a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 10 monclient(hunting): picked mon.noname-e con 0x7f3ef80a4920 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:26:52.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3ef80a5240 0x7f3ef80a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 10 monclient(hunting): picked mon.noname-c con 0x7f3ef80a5240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:26:52.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3ef80a2530 0x7f3ef80a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 10 monclient(hunting): picked mon.noname-i con 0x7f3ef80a2530 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:26:52.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3ef80a7140 con 0x7f3ef80a5240 2024-06-11T19:26:52.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f3ef80a2b50 con 0x7f3ef80a2530 2024-06-11T19:26:52.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3ef80a2e30 con 0x7f3ef80a4920 2024-06-11T19:26:52.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3efd3cd640 10 monclient(hunting): authenticate will time out at 1822.774236s 2024-06-11T19:26:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ef80a4920 0x7f3ef80a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef67fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3ef80a5240 0x7f3ef80a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ef80a4920 0x7f3ef80a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:33510/0 (socket says 172.21.15.98:33510) 2024-06-11T19:26:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 1 -- 172.21.15.98:0/1427766451 learned_addr learned my addr 172.21.15.98:0/1427766451 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef67fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3ef80a5240 0x7f3ef80a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34082/0 (socket says 172.21.15.98:34082) 2024-06-11T19:26:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 10 monclient(hunting): get_auth_request con 0x7f3ef80a4920 auth_method 0 2024-06-11T19:26:52.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef67fc640 10 monclient(hunting): get_auth_request con 0x7f3ef80a5240 auth_method 0 2024-06-11T19:26:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef67fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 10 monclient(hunting): handle_auth_done global_id 124541 payload 274 2024-06-11T19:26:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef77fe640 1 --2- 172.21.15.98:0/1427766451 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3ef80a2530 0x7f3ef80a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 1 -- 172.21.15.98:0/1427766451 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3ef80a2530 msgr2=0x7f3ef80a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 1 --2- 172.21.15.98:0/1427766451 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3ef80a2530 0x7f3ef80a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 1 -- 172.21.15.98:0/1427766451 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3ef80a5240 msgr2=0x7f3ef80a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:26:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 1 --2- 172.21.15.98:0/1427766451 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3ef80a5240 0x7f3ef80a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:26:52.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 1 monclient: found mon.noname-e 2024-06-11T19:26:52.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:52.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:26:52.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef6ffd640 1 -- 172.21.15.98:0/1427766451 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef80a3110 con 0x7f3ef80a4920 2024-06-11T19:26:52.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef77fe640 10 monclient: get_auth_request con 0x7f3ef80a2530 auth_method 0 2024-06-11T19:26:52.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef67fc640 1 --2- 172.21.15.98:0/1427766451 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3ef80a5240 0x7f3ef80a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:26:52.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.967+0000 7f3ef77fe640 1 --2- 172.21.15.98:0/1427766451 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3ef80a2530 0x7f3ef80a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:26:52.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef6ffd640 1 --2- 172.21.15.98:0/1427766451 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ef80a4920 0x7f3ef80a4d00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80098a0 tx=0x7f3ee802b7a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=c7b03209e28cf236 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 1 -- 172.21.15.98:0/1427766451 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ee8038070 con 0x7f3ef80a4920 2024-06-11T19:26:52.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:26:52.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 10 monclient: dump: 2024-06-11T19:26:52.982 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.982 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.982 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.982 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.982 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.983 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.983 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.983 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.983 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.983 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.983 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.983 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.983 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.983 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.984 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.984 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 10 monclient: _finish_auth 0 2024-06-11T19:26:52.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 10 monclient: _check_auth_tickets 2024-06-11T19:26:52.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:52.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 1 -- 172.21.15.98:0/1427766451 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ee8031970 con 0x7f3ef80a4920 2024-06-11T19:26:52.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3efd3cd640 5 monclient: authenticate success, global_id 124541 2024-06-11T19:26:52.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3efd3cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:26:52.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:52.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 1 -- 172.21.15.98:0/1427766451 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ee8031ba0 con 0x7f3ef80a4920 2024-06-11T19:26:52.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3efd3cd640 10 monclient: get_monmap_and_config success 2024-06-11T19:26:52.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3efd3cd640 4 set_mon_vals no callback set 2024-06-11T19:26:52.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:26:52.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.968+0000 7f3ef5ffb640 10 monclient: dump: 2024-06-11T19:26:52.985 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.985 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.986 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:52.987 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 10 monclient: shutdown 2024-06-11T19:26:52.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1427766451 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ef80a4920 msgr2=0x7f3ef80a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 1 --2- 172.21.15.98:0/1427766451 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ef80a4920 0x7f3ef80a4d00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80098a0 tx=0x7f3ee802b7a0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1427766451 shutdown_connections 2024-06-11T19:26:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 1 --2- 172.21.15.98:0/1427766451 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f3ef80a2530 0x7f3ef80a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 1 --2- 172.21.15.98:0/1427766451 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3ef80a5240 0x7f3ef80a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 1 --2- 172.21.15.98:0/1427766451 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3ef80a4920 0x7f3ef80a4d00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1427766451 >> 172.21.15.98:0/1427766451 conn(0x7f3ef80ab850 msgr2=0x7f3ef80a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:52.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1427766451 shutdown_connections 2024-06-11T19:26:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1427766451 wait complete. 2024-06-11T19:26:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 10 monclient: build_initial_monmap 2024-06-11T19:26:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 10 monclient: monmap: 2024-06-11T19:26:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:26:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:26:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:26:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:26:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:26:52.989 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:26:52.990 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:26:52.990 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:26:52.990 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:26:52.990 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:26:52.990 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:26:52.990 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:26:52.990 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:26:52.990 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:26:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:26:52.991 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 1 librados: starting msgr at 2024-06-11T19:26:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.969+0000 7f3efd3cd640 1 librados: starting objecter 2024-06-11T19:26:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 1 Processor -- start 2024-06-11T19:26:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 1 -- start start 2024-06-11T19:26:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 1 librados: setting wanted keys 2024-06-11T19:26:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 1 librados: calling monclient init 2024-06-11T19:26:52.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 10 monclient: init 2024-06-11T19:26:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 10 monclient: _reopen_session rank -1 2024-06-11T19:26:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 10 monclient: _add_conns ranks=[2,1,3,5,0,6,4,8,7] 2024-06-11T19:26:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ef80a2530 0x7f3ef81cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 10 monclient(hunting): picked mon.noname-b con 0x7f3ef80a2530 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-06-11T19:26:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ef80a4920 0x7f3ef81ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 10 monclient(hunting): picked mon.noname-f con 0x7f3ef80a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:26:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ef80a5240 0x7f3ef8122d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:52.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 10 monclient(hunting): picked mon.noname-g con 0x7f3ef80a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:26:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef80a6da0 con 0x7f3ef80a4920 2024-06-11T19:26:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef8138e50 con 0x7f3ef80a2530 2024-06-11T19:26:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 10 monclient(hunting): start opening mon connection 2024-06-11T19:26:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef80a6420 con 0x7f3ef80a5240 2024-06-11T19:26:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 10 monclient(hunting): _renew_subs 2024-06-11T19:26:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3efd3cd640 10 monclient(hunting): authenticate will time out at 1822.777641s 2024-06-11T19:26:52.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ef80a2530 0x7f3ef81cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ef80a2530 0x7f3ef81cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:56616/0 (socket says 172.21.15.98:56616) 2024-06-11T19:26:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 1 -- 172.21.15.98:0/1012557435 learned_addr learned my addr 172.21.15.98:0/1012557435 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:26:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 10 monclient(hunting): get_auth_request con 0x7f3ef80a2530 auth_method 0 2024-06-11T19:26:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:26:52.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:26:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:26:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef67fc640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ef80a4920 0x7f3ef81ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef77fe640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ef80a5240 0x7f3ef8122d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef67fc640 10 monclient(hunting): get_auth_request con 0x7f3ef80a4920 auth_method 0 2024-06-11T19:26:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef67fc640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef77fe640 10 monclient(hunting): get_auth_request con 0x7f3ef80a5240 auth_method 0 2024-06-11T19:26:52.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:26:52.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef77fe640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:26:52.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:26:52.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 10 monclient(hunting): handle_auth_done global_id 85493 payload 824 2024-06-11T19:26:52.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 1 -- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ef80a5240 msgr2=0x7f3ef8122d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ef80a5240 0x7f3ef8122d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 1 -- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ef80a4920 msgr2=0x7f3ef81ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:52.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.970+0000 7f3ef6ffd640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ef80a4920 0x7f3ef81ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:52.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3ef6ffd640 10 monclient: _finish_hunting 0 2024-06-11T19:26:52.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3ef6ffd640 1 monclient: found mon.noname-b 2024-06-11T19:26:52.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3ef6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:26:52.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3ef6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-06-11T19:26:52.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3ef6ffd640 1 -- 172.21.15.98:0/1012557435 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef80a61f0 con 0x7f3ef80a2530 2024-06-11T19:26:52.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3ef77fe640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ef80a5240 0x7f3ef8122d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:26:52.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3ef67fc640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ef80a4920 0x7f3ef81ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:26:52.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3ef6ffd640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ef80a2530 0x7f3ef81cdb40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8000c00 tx=0x7f3ee80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:52.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 10 client.?.objecter ms_handle_connect 0x7f3ef80a2530 2024-06-11T19:26:52.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 10 client.?.objecter resend_mon_ops 2024-06-11T19:26:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 1 -- 172.21.15.98:0/1012557435 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ee8042020 con 0x7f3ef80a2530 2024-06-11T19:26:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-11T19:26:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 10 monclient: dump: 2024-06-11T19:26:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.998 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:53.000 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 10 monclient: _finish_auth 0 2024-06-11T19:26:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:26:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:26:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 1 -- 172.21.15.98:0/1012557435 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ee8031ea0 con 0x7f3ef80a2530 2024-06-11T19:26:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3efd3cd640 5 monclient: authenticate success, global_id 85493 2024-06-11T19:26:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3efd3cd640 10 monclient: _renew_subs 2024-06-11T19:26:53.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3efd3cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1012557435 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef8126ce0 con 0x7f3ef80a2530 2024-06-11T19:26:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:26:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 1 -- 172.21.15.98:0/1012557435 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ee8038040 con 0x7f3ef80a2530 2024-06-11T19:26:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:26:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-11T19:26:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3edf7fe640 10 monclient: dump: 2024-06-11T19:26:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:26:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:26:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:53.002 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:26:53.002 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:26:53.002 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:26:53.002 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:26:53.002 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:26:53.002 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:26:53.002 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:26:53.002 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:26:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:26:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:26:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:26:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:26:53.003 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:26:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3efd3cd640 10 client.85493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:26:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3efd3cd640 10 monclient: _renew_subs 2024-06-11T19:26:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3efd3cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1012557435 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef8126a00 con 0x7f3ef80a2530 2024-06-11T19:26:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3ef57fa640 4 set_mon_vals no callback set 2024-06-11T19:26:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.971+0000 7f3efd3cd640 1 librados: init done 2024-06-11T19:26:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3edf7fe640 1 -- 172.21.15.98:0/1012557435 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f3ee8045020 con 0x7f3ef80a2530 2024-06-11T19:26:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3efd3cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3efd3cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3efd3cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-06-11T19:26:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1012557435 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ef80a66f0 con 0x7f3ef80a2530 2024-06-11T19:26:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3edf7fe640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3ee403fdb0 0x7f3ee4042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:26:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3edf7fe640 1 -- 172.21.15.98:0/1012557435 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f3ee8073080 con 0x7f3ef80a2530 2024-06-11T19:26:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3edf7fe640 10 client.85493.objecter ms_dispatch 0x7f3ef8000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:26:53.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3edf7fe640 3 client.85493.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:26:53.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3edf7fe640 3 client.85493.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:26:53.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3edf7fe640 20 client.85493.objecter dump_active .. 0 homeless 2024-06-11T19:26:53.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3ef67fc640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3ee403fdb0 0x7f3ee4042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:26:53.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.972+0000 7f3ef67fc640 10 monclient: get_auth_request con 0x7f3ee403fdb0 auth_method 0 2024-06-11T19:26:53.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.973+0000 7f3edf7fe640 10 client.85493.objecter ms_handle_connect 0x7f3ee403fdb0 2024-06-11T19:26:53.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.973+0000 7f3ef67fc640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3ee403fdb0 0x7f3ee4042270 secure :-1 s=READY pgs=4347 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8124480 tx=0x7f3eec008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:26:53.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.977+0000 7f3edf7fe640 1 -- 172.21.15.98:0/1012557435 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3ee804c030 con 0x7f3ef80a2530 2024-06-11T19:26:53.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.977+0000 7f3edf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:26:53.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:52.977+0000 7f3edf7fe640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:26:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.154+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1012557435 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3ef8125890 con 0x7f3ee403fdb0 2024-06-11T19:26:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.157+0000 7f3edf7fe640 1 -- 172.21.15.98:0/1012557435 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+113897 (secure 0 0 0) 0x7f3ef8125890 con 0x7f3ee403fdb0 2024-06-11T19:26:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:26:53.157 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-06-11T19:26:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.158+0000 7f3efd3cd640 10 librados: watch_flush enter 2024-06-11T19:26:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.158+0000 7f3efd3cd640 10 librados: watch_flush exit 2024-06-11T19:26:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.158+0000 7f3efd3cd640 20 client.85493.objecter shutdown clearing up homeless session... 2024-06-11T19:26:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.158+0000 7f3efd3cd640 10 client.85493.objecter successfully canceled tick 2024-06-11T19:26:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3ee403fdb0 msgr2=0x7f3ee4042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3ee403fdb0 0x7f3ee4042270 secure :-1 s=READY pgs=4347 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8124480 tx=0x7f3eec008040 comp rx=0 tx=0).stop 2024-06-11T19:26:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 10 monclient: shutdown 2024-06-11T19:26:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:26:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1012557435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ef80a2530 msgr2=0x7f3ef81cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:26:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ef80a2530 0x7f3ef81cdb40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8000c00 tx=0x7f3ee80046c0 comp rx=0 tx=0).stop 2024-06-11T19:26:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1012557435 shutdown_connections 2024-06-11T19:26:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f3ee403fdb0 0x7f3ee4042270 unknown :-1 s=CLOSED pgs=4347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ef80a5240 0x7f3ef8122d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ef80a4920 0x7f3ef81ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 --2- 172.21.15.98:0/1012557435 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3ef80a2530 0x7f3ef81cdb40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:26:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1012557435 >> 172.21.15.98:0/1012557435 conn(0x7f3ef80ab850 msgr2=0x7f3ef81bff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:26:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1012557435 shutdown_connections 2024-06-11T19:26:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 -- 172.21.15.98:0/1012557435 wait complete. 2024-06-11T19:26:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:26:53.159+0000 7f3efd3cd640 1 librados: shutdown 2024-06-11T19:26:53.168 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1202,"stamp":"2024-06-11T19:26:51.632766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":108,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":471568,"kb_used_data":60832,"kb_used_omap":9,"kb_used_meta":410486,"kb_avail":565759472,"statfs":{"total":579820584960,"available":579337699328,"internally_reserved":0,"allocated":62291968,"data_stored":35204514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":420338336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002947"},"pg_stats":[{"pgid":"7.17","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630466+0000","last_change":"2024-06-11T19:10:36.026367+0000","last_active":"2024-06-11T19:26:21.630466+0000","last_peered":"2024-06-11T19:26:21.630466+0000","last_clean":"2024-06-11T19:26:21.630466+0000","last_became_active":"2024-06-11T19:10:36.005579+0000","last_became_peered":"2024-06-11T19:10:36.005579+0000","last_unstale":"2024-06-11T19:26:21.630466+0000","last_undegraded":"2024-06-11T19:26:21.630466+0000","last_fullsized":"2024-06-11T19:26:21.630466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T05:31:34.753919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630581+0000","last_change":"2024-06-11T19:10:35.961885+0000","last_active":"2024-06-11T19:26:21.630581+0000","last_peered":"2024-06-11T19:26:21.630581+0000","last_clean":"2024-06-11T19:26:21.630581+0000","last_became_active":"2024-06-11T19:10:35.960336+0000","last_became_peered":"2024-06-11T19:10:35.960336+0000","last_unstale":"2024-06-11T19:26:21.630581+0000","last_undegraded":"2024-06-11T19:26:21.630581+0000","last_fullsized":"2024-06-11T19:26:21.630581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:42:55.368948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390705+0000","last_change":"2024-06-11T19:10:36.098514+0000","last_active":"2024-06-11T19:26:21.390705+0000","last_peered":"2024-06-11T19:26:21.390705+0000","last_clean":"2024-06-11T19:26:21.390705+0000","last_became_active":"2024-06-11T19:10:36.087715+0000","last_became_peered":"2024-06-11T19:10:36.087715+0000","last_unstale":"2024-06-11T19:26:21.390705+0000","last_undegraded":"2024-06-11T19:26:21.390705+0000","last_fullsized":"2024-06-11T19:26:21.390705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:25:31.960579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1332,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630521+0000","last_change":"2024-06-11T19:10:36.045515+0000","last_active":"2024-06-11T19:26:21.630521+0000","last_peered":"2024-06-11T19:26:21.630521+0000","last_clean":"2024-06-11T19:26:21.630521+0000","last_became_active":"2024-06-11T19:10:36.036974+0000","last_became_peered":"2024-06-11T19:10:36.036974+0000","last_unstale":"2024-06-11T19:26:21.630521+0000","last_undegraded":"2024-06-11T19:26:21.630521+0000","last_fullsized":"2024-06-11T19:26:21.630521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:35:56.827003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389826+0000","last_change":"2024-06-11T19:10:36.084957+0000","last_active":"2024-06-11T19:26:21.389826+0000","last_peered":"2024-06-11T19:26:21.389826+0000","last_clean":"2024-06-11T19:26:21.389826+0000","last_became_active":"2024-06-11T19:10:36.065872+0000","last_became_peered":"2024-06-11T19:10:36.065872+0000","last_unstale":"2024-06-11T19:26:21.389826+0000","last_undegraded":"2024-06-11T19:26:21.389826+0000","last_fullsized":"2024-06-11T19:26:21.389826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T05:33:25.348773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1329,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630376+0000","last_change":"2024-06-11T19:10:36.077422+0000","last_active":"2024-06-11T19:26:21.630376+0000","last_peered":"2024-06-11T19:26:21.630376+0000","last_clean":"2024-06-11T19:26:21.630376+0000","last_became_active":"2024-06-11T19:10:36.076424+0000","last_became_peered":"2024-06-11T19:10:36.076424+0000","last_unstale":"2024-06-11T19:26:21.630376+0000","last_undegraded":"2024-06-11T19:26:21.630376+0000","last_fullsized":"2024-06-11T19:26:21.630376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:52:42.583531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1333,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630124+0000","last_change":"2024-06-11T19:10:36.037164+0000","last_active":"2024-06-11T19:26:21.630124+0000","last_peered":"2024-06-11T19:26:21.630124+0000","last_clean":"2024-06-11T19:26:21.630124+0000","last_became_active":"2024-06-11T19:10:36.024538+0000","last_became_peered":"2024-06-11T19:10:36.024538+0000","last_unstale":"2024-06-11T19:26:21.630124+0000","last_undegraded":"2024-06-11T19:26:21.630124+0000","last_fullsized":"2024-06-11T19:26:21.630124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:27:41.916846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1333,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630098+0000","last_change":"2024-06-11T19:10:36.087130+0000","last_active":"2024-06-11T19:26:21.630098+0000","last_peered":"2024-06-11T19:26:21.630098+0000","last_clean":"2024-06-11T19:26:21.630098+0000","last_became_active":"2024-06-11T19:10:36.085399+0000","last_became_peered":"2024-06-11T19:10:36.085399+0000","last_unstale":"2024-06-11T19:26:21.630098+0000","last_undegraded":"2024-06-11T19:26:21.630098+0000","last_fullsized":"2024-06-11T19:26:21.630098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:45:20.075258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389693+0000","last_change":"2024-06-11T19:10:35.977870+0000","last_active":"2024-06-11T19:26:21.389693+0000","last_peered":"2024-06-11T19:26:21.389693+0000","last_clean":"2024-06-11T19:26:21.389693+0000","last_became_active":"2024-06-11T19:10:35.976679+0000","last_became_peered":"2024-06-11T19:10:35.976679+0000","last_unstale":"2024-06-11T19:26:21.389693+0000","last_undegraded":"2024-06-11T19:26:21.389693+0000","last_fullsized":"2024-06-11T19:26:21.389693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T06:02:34.810156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389929+0000","last_change":"2024-06-11T19:10:36.004758+0000","last_active":"2024-06-11T19:26:21.389929+0000","last_peered":"2024-06-11T19:26:21.389929+0000","last_clean":"2024-06-11T19:26:21.389929+0000","last_became_active":"2024-06-11T19:10:35.985449+0000","last_became_peered":"2024-06-11T19:10:35.985449+0000","last_unstale":"2024-06-11T19:26:21.389929+0000","last_undegraded":"2024-06-11T19:26:21.389929+0000","last_fullsized":"2024-06-11T19:26:21.389929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T07:01:45.265291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630608+0000","last_change":"2024-06-11T19:10:35.967673+0000","last_active":"2024-06-11T19:26:21.630608+0000","last_peered":"2024-06-11T19:26:21.630608+0000","last_clean":"2024-06-11T19:26:21.630608+0000","last_became_active":"2024-06-11T19:10:35.962677+0000","last_became_peered":"2024-06-11T19:10:35.962677+0000","last_unstale":"2024-06-11T19:26:21.630608+0000","last_undegraded":"2024-06-11T19:26:21.630608+0000","last_fullsized":"2024-06-11T19:26:21.630608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:37:13.485742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1322,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390699+0000","last_change":"2024-06-11T19:10:36.057228+0000","last_active":"2024-06-11T19:26:21.390699+0000","last_peered":"2024-06-11T19:26:21.390699+0000","last_clean":"2024-06-11T19:26:21.390699+0000","last_became_active":"2024-06-11T19:10:36.056338+0000","last_became_peered":"2024-06-11T19:10:36.056338+0000","last_unstale":"2024-06-11T19:26:21.390699+0000","last_undegraded":"2024-06-11T19:26:21.390699+0000","last_fullsized":"2024-06-11T19:26:21.390699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:31:48.434226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1438,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630415+0000","last_change":"2024-06-11T19:09:26.673425+0000","last_active":"2024-06-11T19:26:21.630415+0000","last_peered":"2024-06-11T19:26:21.630415+0000","last_clean":"2024-06-11T19:26:21.630415+0000","last_became_active":"2024-06-11T19:09:26.673120+0000","last_became_peered":"2024-06-11T19:09:26.673120+0000","last_unstale":"2024-06-11T19:26:21.630415+0000","last_undegraded":"2024-06-11T19:26:21.630415+0000","last_fullsized":"2024-06-11T19:26:21.630415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:25.660869+0000","last_clean_scrub_stamp":"2024-06-11T19:09:25.660869+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:57:43.705570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630700+0000","last_change":"2024-06-11T19:09:33.725261+0000","last_active":"2024-06-11T19:26:21.630700+0000","last_peered":"2024-06-11T19:26:21.630700+0000","last_clean":"2024-06-11T19:26:21.630700+0000","last_became_active":"2024-06-11T19:09:33.724870+0000","last_became_peered":"2024-06-11T19:09:33.724870+0000","last_unstale":"2024-06-11T19:26:21.630700+0000","last_undegraded":"2024-06-11T19:26:21.630700+0000","last_fullsized":"2024-06-11T19:26:21.630700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:32:51.020718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389982+0000","last_change":"2024-06-11T19:10:36.011730+0000","last_active":"2024-06-11T19:26:21.389982+0000","last_peered":"2024-06-11T19:26:21.389982+0000","last_clean":"2024-06-11T19:26:21.389982+0000","last_became_active":"2024-06-11T19:10:35.988487+0000","last_became_peered":"2024-06-11T19:10:35.988487+0000","last_unstale":"2024-06-11T19:26:21.389982+0000","last_undegraded":"2024-06-11T19:26:21.389982+0000","last_fullsized":"2024-06-11T19:26:21.389982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:48:33.179394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1329,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390443+0000","last_change":"2024-06-11T19:09:33.729213+0000","last_active":"2024-06-11T19:26:21.390443+0000","last_peered":"2024-06-11T19:26:21.390443+0000","last_clean":"2024-06-11T19:26:21.390443+0000","last_became_active":"2024-06-11T19:09:33.728930+0000","last_became_peered":"2024-06-11T19:09:33.728930+0000","last_unstale":"2024-06-11T19:26:21.390443+0000","last_undegraded":"2024-06-11T19:26:21.390443+0000","last_fullsized":"2024-06-11T19:26:21.390443+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:26:38.808169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1322,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390789+0000","last_change":"2024-06-11T19:10:36.088019+0000","last_active":"2024-06-11T19:26:21.390789+0000","last_peered":"2024-06-11T19:26:21.390789+0000","last_clean":"2024-06-11T19:26:21.390789+0000","last_became_active":"2024-06-11T19:10:36.086670+0000","last_became_peered":"2024-06-11T19:10:36.086670+0000","last_unstale":"2024-06-11T19:26:21.390789+0000","last_undegraded":"2024-06-11T19:26:21.390789+0000","last_fullsized":"2024-06-11T19:26:21.390789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T22:06:29.653796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389520+0000","last_change":"2024-06-11T19:09:33.729275+0000","last_active":"2024-06-11T19:26:21.389520+0000","last_peered":"2024-06-11T19:26:21.389520+0000","last_clean":"2024-06-11T19:26:21.389520+0000","last_became_active":"2024-06-11T19:09:33.728948+0000","last_became_peered":"2024-06-11T19:09:33.728948+0000","last_unstale":"2024-06-11T19:26:21.389520+0000","last_undegraded":"2024-06-11T19:26:21.389520+0000","last_fullsized":"2024-06-11T19:26:21.389520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:29:28.908169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1329,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630555+0000","last_change":"2024-06-11T19:10:35.937546+0000","last_active":"2024-06-11T19:26:21.630555+0000","last_peered":"2024-06-11T19:26:21.630555+0000","last_clean":"2024-06-11T19:26:21.630555+0000","last_became_active":"2024-06-11T19:10:35.937161+0000","last_became_peered":"2024-06-11T19:10:35.937161+0000","last_unstale":"2024-06-11T19:26:21.630555+0000","last_undegraded":"2024-06-11T19:26:21.630555+0000","last_fullsized":"2024-06-11T19:26:21.630555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:12:14.668727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1330,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630527+0000","last_change":"2024-06-11T19:09:33.722503+0000","last_active":"2024-06-11T19:26:21.630527+0000","last_peered":"2024-06-11T19:26:21.630527+0000","last_clean":"2024-06-11T19:26:21.630527+0000","last_became_active":"2024-06-11T19:09:33.722180+0000","last_became_peered":"2024-06-11T19:09:33.722180+0000","last_unstale":"2024-06-11T19:26:21.630527+0000","last_undegraded":"2024-06-11T19:26:21.630527+0000","last_fullsized":"2024-06-11T19:26:21.630527+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:28:15.836943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630572+0000","last_change":"2024-06-11T19:10:36.062834+0000","last_active":"2024-06-11T19:26:21.630572+0000","last_peered":"2024-06-11T19:26:21.630572+0000","last_clean":"2024-06-11T19:26:21.630572+0000","last_became_active":"2024-06-11T19:10:36.016575+0000","last_became_peered":"2024-06-11T19:10:36.016575+0000","last_unstale":"2024-06-11T19:26:21.630572+0000","last_undegraded":"2024-06-11T19:26:21.630572+0000","last_fullsized":"2024-06-11T19:26:21.630572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:53:41.228041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1340,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630524+0000","last_change":"2024-06-11T19:09:33.729223+0000","last_active":"2024-06-11T19:26:21.630524+0000","last_peered":"2024-06-11T19:26:21.630524+0000","last_clean":"2024-06-11T19:26:21.630524+0000","last_became_active":"2024-06-11T19:09:33.728897+0000","last_became_peered":"2024-06-11T19:09:33.728897+0000","last_unstale":"2024-06-11T19:26:21.630524+0000","last_undegraded":"2024-06-11T19:26:21.630524+0000","last_fullsized":"2024-06-11T19:26:21.630524+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:49:13.439577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"20'7","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390521+0000","last_change":"2024-06-11T19:10:36.037487+0000","last_active":"2024-06-11T19:26:21.390521+0000","last_peered":"2024-06-11T19:26:21.390521+0000","last_clean":"2024-06-11T19:26:21.390521+0000","last_became_active":"2024-06-11T19:10:36.032874+0000","last_became_peered":"2024-06-11T19:10:36.032874+0000","last_unstale":"2024-06-11T19:26:21.390521+0000","last_undegraded":"2024-06-11T19:26:21.390521+0000","last_fullsized":"2024-06-11T19:26:21.390521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:58:37.764224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630669+0000","last_change":"2024-06-11T19:09:33.720915+0000","last_active":"2024-06-11T19:26:21.630669+0000","last_peered":"2024-06-11T19:26:21.630669+0000","last_clean":"2024-06-11T19:26:21.630669+0000","last_became_active":"2024-06-11T19:09:33.720599+0000","last_became_peered":"2024-06-11T19:09:33.720599+0000","last_unstale":"2024-06-11T19:26:21.630669+0000","last_undegraded":"2024-06-11T19:26:21.630669+0000","last_fullsized":"2024-06-11T19:26:21.630669+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:28:26.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1336,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630711+0000","last_change":"2024-06-11T19:10:35.982779+0000","last_active":"2024-06-11T19:26:21.630711+0000","last_peered":"2024-06-11T19:26:21.630711+0000","last_clean":"2024-06-11T19:26:21.630711+0000","last_became_active":"2024-06-11T19:10:35.976675+0000","last_became_peered":"2024-06-11T19:10:35.976675+0000","last_unstale":"2024-06-11T19:26:21.630711+0000","last_undegraded":"2024-06-11T19:26:21.630711+0000","last_fullsized":"2024-06-11T19:26:21.630711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:39:25.265311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630653+0000","last_change":"2024-06-11T19:09:33.725197+0000","last_active":"2024-06-11T19:26:21.630653+0000","last_peered":"2024-06-11T19:26:21.630653+0000","last_clean":"2024-06-11T19:26:21.630653+0000","last_became_active":"2024-06-11T19:09:33.724912+0000","last_became_peered":"2024-06-11T19:09:33.724912+0000","last_unstale":"2024-06-11T19:26:21.630653+0000","last_undegraded":"2024-06-11T19:26:21.630653+0000","last_fullsized":"2024-06-11T19:26:21.630653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:36:12.941264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389668+0000","last_change":"2024-06-11T19:10:36.090541+0000","last_active":"2024-06-11T19:26:21.389668+0000","last_peered":"2024-06-11T19:26:21.389668+0000","last_clean":"2024-06-11T19:26:21.389668+0000","last_became_active":"2024-06-11T19:10:36.071353+0000","last_became_peered":"2024-06-11T19:10:36.071353+0000","last_unstale":"2024-06-11T19:26:21.389668+0000","last_undegraded":"2024-06-11T19:26:21.389668+0000","last_fullsized":"2024-06-11T19:26:21.389668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:58:46.822805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1341,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630664+0000","last_change":"2024-06-11T19:09:33.725085+0000","last_active":"2024-06-11T19:26:21.630664+0000","last_peered":"2024-06-11T19:26:21.630664+0000","last_clean":"2024-06-11T19:26:21.630664+0000","last_became_active":"2024-06-11T19:09:33.724778+0000","last_became_peered":"2024-06-11T19:09:33.724778+0000","last_unstale":"2024-06-11T19:26:21.630664+0000","last_undegraded":"2024-06-11T19:26:21.630664+0000","last_fullsized":"2024-06-11T19:26:21.630664+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:09:32.697383+0000","last_clean_scrub_stamp":"2024-06-11T19:09:32.697383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:57:46.868411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1335,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389513+0000","last_change":"2024-06-11T19:10:36.060364+0000","last_active":"2024-06-11T19:26:21.389513+0000","last_peered":"2024-06-11T19:26:21.389513+0000","last_clean":"2024-06-11T19:26:21.389513+0000","last_became_active":"2024-06-11T19:10:36.035136+0000","last_became_peered":"2024-06-11T19:10:36.035136+0000","last_unstale":"2024-06-11T19:26:21.389513+0000","last_undegraded":"2024-06-11T19:26:21.389513+0000","last_fullsized":"2024-06-11T19:26:21.389513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:30:08.506219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389506+0000","last_change":"2024-06-11T19:10:36.052997+0000","last_active":"2024-06-11T19:26:21.389506+0000","last_peered":"2024-06-11T19:26:21.389506+0000","last_clean":"2024-06-11T19:26:21.389506+0000","last_became_active":"2024-06-11T19:10:36.044757+0000","last_became_peered":"2024-06-11T19:10:36.044757+0000","last_unstale":"2024-06-11T19:26:21.389506+0000","last_undegraded":"2024-06-11T19:26:21.389506+0000","last_fullsized":"2024-06-11T19:26:21.389506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:11:16.047309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390454+0000","last_change":"2024-06-11T19:10:36.011492+0000","last_active":"2024-06-11T19:26:21.390454+0000","last_peered":"2024-06-11T19:26:21.390454+0000","last_clean":"2024-06-11T19:26:21.390454+0000","last_became_active":"2024-06-11T19:10:36.000792+0000","last_became_peered":"2024-06-11T19:10:36.000792+0000","last_unstale":"2024-06-11T19:26:21.390454+0000","last_undegraded":"2024-06-11T19:26:21.390454+0000","last_fullsized":"2024-06-11T19:26:21.390454+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:42:36.937874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630349+0000","last_change":"2024-06-11T19:10:35.994191+0000","last_active":"2024-06-11T19:26:21.630349+0000","last_peered":"2024-06-11T19:26:21.630349+0000","last_clean":"2024-06-11T19:26:21.630349+0000","last_became_active":"2024-06-11T19:10:35.988056+0000","last_became_peered":"2024-06-11T19:10:35.988056+0000","last_unstale":"2024-06-11T19:26:21.630349+0000","last_undegraded":"2024-06-11T19:26:21.630349+0000","last_fullsized":"2024-06-11T19:26:21.630349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:24:15.498060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1332,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630386+0000","last_change":"2024-06-11T19:10:36.030504+0000","last_active":"2024-06-11T19:26:21.630386+0000","last_peered":"2024-06-11T19:26:21.630386+0000","last_clean":"2024-06-11T19:26:21.630386+0000","last_became_active":"2024-06-11T19:10:36.025371+0000","last_became_peered":"2024-06-11T19:10:36.025371+0000","last_unstale":"2024-06-11T19:26:21.630386+0000","last_undegraded":"2024-06-11T19:26:21.630386+0000","last_fullsized":"2024-06-11T19:26:21.630386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:14:08.888348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390627+0000","last_change":"2024-06-11T19:10:36.091418+0000","last_active":"2024-06-11T19:26:21.390627+0000","last_peered":"2024-06-11T19:26:21.390627+0000","last_clean":"2024-06-11T19:26:21.390627+0000","last_became_active":"2024-06-11T19:10:36.090801+0000","last_became_peered":"2024-06-11T19:10:36.090801+0000","last_unstale":"2024-06-11T19:26:21.390627+0000","last_undegraded":"2024-06-11T19:26:21.390627+0000","last_fullsized":"2024-06-11T19:26:21.390627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:34:25.520333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1332,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630306+0000","last_change":"2024-06-11T19:10:36.063862+0000","last_active":"2024-06-11T19:26:21.630306+0000","last_peered":"2024-06-11T19:26:21.630306+0000","last_clean":"2024-06-11T19:26:21.630306+0000","last_became_active":"2024-06-11T19:10:36.057852+0000","last_became_peered":"2024-06-11T19:10:36.057852+0000","last_unstale":"2024-06-11T19:26:21.630306+0000","last_undegraded":"2024-06-11T19:26:21.630306+0000","last_fullsized":"2024-06-11T19:26:21.630306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:16:37.800621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1336,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.389829+0000","last_change":"2024-06-11T19:10:36.056378+0000","last_active":"2024-06-11T19:26:21.389829+0000","last_peered":"2024-06-11T19:26:21.389829+0000","last_clean":"2024-06-11T19:26:21.389829+0000","last_became_active":"2024-06-11T19:10:36.042135+0000","last_became_peered":"2024-06-11T19:10:36.042135+0000","last_unstale":"2024-06-11T19:26:21.389829+0000","last_undegraded":"2024-06-11T19:26:21.389829+0000","last_fullsized":"2024-06-11T19:26:21.389829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:38:55.622181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1322,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630471+0000","last_change":"2024-06-11T19:10:36.026724+0000","last_active":"2024-06-11T19:26:21.630471+0000","last_peered":"2024-06-11T19:26:21.630471+0000","last_clean":"2024-06-11T19:26:21.630471+0000","last_became_active":"2024-06-11T19:10:36.026278+0000","last_became_peered":"2024-06-11T19:10:36.026278+0000","last_unstale":"2024-06-11T19:26:21.630471+0000","last_undegraded":"2024-06-11T19:26:21.630471+0000","last_fullsized":"2024-06-11T19:26:21.630471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:22:26.410474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1328,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630311+0000","last_change":"2024-06-11T19:10:36.034688+0000","last_active":"2024-06-11T19:26:21.630311+0000","last_peered":"2024-06-11T19:26:21.630311+0000","last_clean":"2024-06-11T19:26:21.630311+0000","last_became_active":"2024-06-11T19:10:36.028329+0000","last_became_peered":"2024-06-11T19:10:36.028329+0000","last_unstale":"2024-06-11T19:26:21.630311+0000","last_undegraded":"2024-06-11T19:26:21.630311+0000","last_fullsized":"2024-06-11T19:26:21.630311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T06:59:08.595933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1334,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630349+0000","last_change":"2024-06-11T19:10:36.085537+0000","last_active":"2024-06-11T19:26:21.630349+0000","last_peered":"2024-06-11T19:26:21.630349+0000","last_clean":"2024-06-11T19:26:21.630349+0000","last_became_active":"2024-06-11T19:10:36.082897+0000","last_became_peered":"2024-06-11T19:10:36.082897+0000","last_unstale":"2024-06-11T19:26:21.630349+0000","last_undegraded":"2024-06-11T19:26:21.630349+0000","last_fullsized":"2024-06-11T19:26:21.630349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:29:42.550377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1321,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.390495+0000","last_change":"2024-06-11T19:10:36.017667+0000","last_active":"2024-06-11T19:26:21.390495+0000","last_peered":"2024-06-11T19:26:21.390495+0000","last_clean":"2024-06-11T19:26:21.390495+0000","last_became_active":"2024-06-11T19:10:36.002730+0000","last_became_peered":"2024-06-11T19:10:36.002730+0000","last_unstale":"2024-06-11T19:26:21.390495+0000","last_undegraded":"2024-06-11T19:26:21.390495+0000","last_fullsized":"2024-06-11T19:26:21.390495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:34:23.834164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1328,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:21.630373+0000","last_change":"2024-06-11T19:10:36.076140+0000","last_active":"2024-06-11T19:26:21.630373+0000","last_peered":"2024-06-11T19:26:21.630373+0000","last_clean":"2024-06-11T19:26:21.630373+0000","last_became_active":"2024-06-11T19:10:36.075888+0000","last_became_peered":"2024-06-11T19:10:36.075888+0000","last_unstale":"2024-06-11T19:26:21.630373+0000","last_undegraded":"2024-06-11T19:26:21.630373+0000","last_fullsized":"2024-06-11T19:26:21.630373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:10:34.825826+0000","last_clean_scrub_stamp":"2024-06-11T19:10:34.825826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:07:12.839020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607764,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77264,"kb_used_data":9968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294576,"statfs":{"total":96636764160,"available":96557645824,"internally_reserved":0,"allocated":10207232,"data_stored":5691113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81956,"kb_used_data":10488,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289884,"statfs":{"total":96636764160,"available":96552841216,"internally_reserved":0,"allocated":10739712,"data_stored":6220041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640468,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86100,"kb_used_data":10484,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94285740,"statfs":{"total":96636764160,"available":96548597760,"internally_reserved":0,"allocated":10735616,"data_stored":6220036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640469,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73112,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298728,"statfs":{"total":96636764160,"available":96561897472,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640468,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67532,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94304308,"statfs":{"total":96636764160,"available":96567611392,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640470,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85604,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286236,"statfs":{"total":96636764160,"available":96549105664,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-11T19:26:53.173 INFO:tasks.ceph:pgid 7.17 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.173 INFO:tasks.ceph:pgid 7.16 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.173 INFO:tasks.ceph:pgid 7.15 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.173 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.173 INFO:tasks.ceph:pgid 7.b last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.174 INFO:tasks.ceph:pgid 7.a last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.174 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.174 INFO:tasks.ceph:pgid 7.8 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.174 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.174 INFO:tasks.ceph:pgid 7.e last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.174 INFO:tasks.ceph:pgid 7.d last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.175 INFO:tasks.ceph:pgid 7.c last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.175 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-06-11T19:09:25.660869+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=9, tm_sec=25, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.175 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-06-11T19:09:32.697383+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=9, tm_sec=32, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.175 INFO:tasks.ceph:pgid 7.6 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.175 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-06-11T19:09:32.697383+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=9, tm_sec=32, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.175 INFO:tasks.ceph:pgid 7.5 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.176 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-06-11T19:09:32.697383+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=9, tm_sec=32, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.176 INFO:tasks.ceph:pgid 7.4 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.176 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-06-11T19:09:32.697383+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=9, tm_sec=32, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.176 INFO:tasks.ceph:pgid 7.7 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.176 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-06-11T19:09:32.697383+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=9, tm_sec=32, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.176 INFO:tasks.ceph:pgid 7.1 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.176 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-06-11T19:09:32.697383+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=9, tm_sec=32, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.177 INFO:tasks.ceph:pgid 7.0 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.177 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-06-11T19:09:32.697383+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=9, tm_sec=32, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.177 INFO:tasks.ceph:pgid 7.3 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.177 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-06-11T19:09:32.697383+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=9, tm_sec=32, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.177 INFO:tasks.ceph:pgid 7.2 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.177 INFO:tasks.ceph:pgid 7.10 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.178 INFO:tasks.ceph:pgid 7.11 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.178 INFO:tasks.ceph:pgid 7.12 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.178 INFO:tasks.ceph:pgid 7.13 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.178 INFO:tasks.ceph:pgid 7.1c last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.178 INFO:tasks.ceph:pgid 7.1d last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.178 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.179 INFO:tasks.ceph:pgid 7.1f last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.179 INFO:tasks.ceph:pgid 7.18 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.179 INFO:tasks.ceph:pgid 7.19 last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.179 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.179 INFO:tasks.ceph:pgid 7.1b last_scrub_stamp 2024-06-11T19:10:34.825826+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=10, tm_sec=34, tm_wday=1, tm_yday=163, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=11, tm_hour=19, tm_min=26, tm_sec=49, tm_wday=1, tm_yday=163, tm_isdst=0) 2024-06-11T19:26:53.179 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-06-11T19:27:13.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-11T19:27:13.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.292+0000 7f253effb640 10 monclient: get_monmap_and_config 2024-06-11T19:27:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.292+0000 7f253effb640 10 monclient: build_initial_monmap 2024-06-11T19:27:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.292+0000 7f253effb640 10 monclient: monmap: 2024-06-11T19:27:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:27:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:27:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:27:13.293695+0000 2024-06-11T19:27:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:27:13.293695+0000 2024-06-11T19:27:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:27:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:27:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:27:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:27:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:27:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:27:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:27:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:27:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:27:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:27:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:27:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:27:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:27:13.295 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 1 Processor -- start 2024-06-11T19:27:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 1 -- start start 2024-06-11T19:27:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 10 monclient: init 2024-06-11T19:27:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 10 monclient: _reopen_session rank -1 2024-06-11T19:27:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 10 monclient: _add_conns ranks=[10,8,6,7,2,5,3,1,9,0,4] 2024-06-11T19:27:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f25380a4920 0x7f25380a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 10 monclient(hunting): picked mon.noname-f con 0x7f25380a4920 addr v2:172.21.15.117:3300/0 2024-06-11T19:27:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25380a5240 0x7f25380a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 10 monclient(hunting): picked mon.noname-e con 0x7f25380a5240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-06-11T19:27:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 0x7f25380a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 10 monclient(hunting): picked mon.noname-d con 0x7f25380a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:27:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:27:13.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 -- 0x7f25380a7140 con 0x7f25380a4920 2024-06-11T19:27:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:27:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f25380a2b50 con 0x7f25380a2530 2024-06-11T19:27:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:27:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f25380a2e30 con 0x7f25380a5240 2024-06-11T19:27:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 10 monclient(hunting): _renew_subs 2024-06-11T19:27:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.293+0000 7f253effb640 10 monclient(hunting): authenticate will time out at 1843.100976s 2024-06-11T19:27:13.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25380a5240 0x7f25380a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253d598640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 0x7f25380a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253d598640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 0x7f25380a5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:37104/0 (socket says 172.21.15.98:37104) 2024-06-11T19:27:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25380a5240 0x7f25380a59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:33262/0 (socket says 172.21.15.98:33262) 2024-06-11T19:27:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253d598640 1 -- 172.21.15.98:0/2389426781 learned_addr learned my addr 172.21.15.98:0/2389426781 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:27:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253d598640 10 monclient(hunting): get_auth_request con 0x7f25380a2530 auth_method 0 2024-06-11T19:27:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253d598640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:27:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253d598640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:27:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253d598640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:27:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 10 monclient(hunting): get_auth_request con 0x7f25380a5240 auth_method 0 2024-06-11T19:27:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:27:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:27:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:27:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:27:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:27:13.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:27:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253d598640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:27:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253d598640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:27:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253d598640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:27:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253cd97640 1 --2- 172.21.15.98:0/2389426781 >> v2:172.21.15.117:3300/0 conn(0x7f25380a4920 0x7f25380a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 10 monclient(hunting): handle_auth_done global_id 124550 payload 274 2024-06-11T19:27:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 1 -- 172.21.15.98:0/2389426781 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 msgr2=0x7f25380a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:27:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 1 --2- 172.21.15.98:0/2389426781 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 0x7f25380a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 1 -- 172.21.15.98:0/2389426781 >> v2:172.21.15.117:3300/0 conn(0x7f25380a4920 msgr2=0x7f25380a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:27:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 1 --2- 172.21.15.98:0/2389426781 >> v2:172.21.15.117:3300/0 conn(0x7f25380a4920 0x7f25380a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 10 monclient: _finish_hunting 0 2024-06-11T19:27:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 1 monclient: found mon.noname-e 2024-06-11T19:27:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:27:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-06-11T19:27:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f252ffff640 1 -- 172.21.15.98:0/2389426781 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25380a3110 con 0x7f25380a5240 2024-06-11T19:27:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253cd97640 10 monclient: get_auth_request con 0x7f25380a4920 auth_method 0 2024-06-11T19:27:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253cd97640 1 --2- 172.21.15.98:0/2389426781 >> v2:172.21.15.117:3300/0 conn(0x7f25380a4920 0x7f25380a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:27:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.294+0000 7f253d598640 1 --2- 172.21.15.98:0/2389426781 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 0x7f25380a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-11T19:27:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252ffff640 1 --2- 172.21.15.98:0/2389426781 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25380a5240 0x7f25380a59a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f25280097a0 tx=0x7f252802b6f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=1953aceb23acda35 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:27:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 1 -- 172.21.15.98:0/2389426781 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2528037070 con 0x7f25380a5240 2024-06-11T19:27:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:27:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-06-11T19:27:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 10 monclient: dump: 2024-06-11T19:27:13.312 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:27:13.312 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:27:13.312 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.312 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.312 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:27:13.312 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:27:13.312 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:27:13.312 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:27:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:27:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:27:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:27:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:27:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:27:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:27:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:27:13.313 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 10 monclient: _finish_auth 0 2024-06-11T19:27:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 10 monclient: _check_auth_tickets 2024-06-11T19:27:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:27:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 5 monclient: authenticate success, global_id 124550 2024-06-11T19:27:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:27:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 1 -- 172.21.15.98:0/2389426781 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2528030980 con 0x7f25380a5240 2024-06-11T19:27:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 10 monclient: handle_config config(0 keys) 2024-06-11T19:27:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 10 monclient: get_monmap_and_config success 2024-06-11T19:27:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 4 set_mon_vals no callback set 2024-06-11T19:27:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 1 -- 172.21.15.98:0/2389426781 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2528030bb0 con 0x7f25380a5240 2024-06-11T19:27:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:27:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-06-11T19:27:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f252f7fe640 10 monclient: dump: 2024-06-11T19:27:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:27:13.315 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:27:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:27:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:27:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:27:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:27:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:27:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:27:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:27:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:27:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:27:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:27:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:27:13.317 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 10 monclient: shutdown 2024-06-11T19:27:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:27:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 1 -- 172.21.15.98:0/2389426781 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25380a5240 msgr2=0x7f25380a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:27:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 1 --2- 172.21.15.98:0/2389426781 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25380a5240 0x7f25380a59a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f25280097a0 tx=0x7f252802b6f0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 1 -- 172.21.15.98:0/2389426781 shutdown_connections 2024-06-11T19:27:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 1 --2- 172.21.15.98:0/2389426781 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 0x7f25380a5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 1 --2- 172.21.15.98:0/2389426781 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25380a5240 0x7f25380a59a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 1 --2- 172.21.15.98:0/2389426781 >> v2:172.21.15.117:3300/0 conn(0x7f25380a4920 0x7f25380a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 1 -- 172.21.15.98:0/2389426781 >> 172.21.15.98:0/2389426781 conn(0x7f25380ab850 msgr2=0x7f25380a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:27:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.295+0000 7f253effb640 1 -- 172.21.15.98:0/2389426781 shutdown_connections 2024-06-11T19:27:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.296+0000 7f253effb640 1 -- 172.21.15.98:0/2389426781 wait complete. 2024-06-11T19:27:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.296+0000 7f253effb640 10 monclient: build_initial_monmap 2024-06-11T19:27:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.296+0000 7f253effb640 10 monclient: monmap: 2024-06-11T19:27:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:27:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:27:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:27:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:27:13.320 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:27:13.320 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:27:13.320 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:27:13.320 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:27:13.320 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:27:13.320 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:27:13.321 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:27:13.321 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:27:13.321 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:27:13.321 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:27:13.321 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:27:13.321 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:13.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.296+0000 7f253effb640 1 librados: starting msgr at 2024-06-11T19:27:13.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.296+0000 7f253effb640 1 librados: starting objecter 2024-06-11T19:27:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.296+0000 7f253effb640 1 Processor -- start 2024-06-11T19:27:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.296+0000 7f253effb640 1 -- start start 2024-06-11T19:27:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.296+0000 7f253effb640 1 librados: setting wanted keys 2024-06-11T19:27:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.296+0000 7f253effb640 1 librados: calling monclient init 2024-06-11T19:27:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.296+0000 7f253effb640 10 monclient: init 2024-06-11T19:27:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 10 monclient: _reopen_session rank -1 2024-06-11T19:27:13.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 10 monclient: _add_conns ranks=[6,1,3,5,0,7,2,8,4] 2024-06-11T19:27:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 0x7f25381215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 10 monclient(hunting): picked mon.noname-d con 0x7f25380a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:27:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25380a4920 0x7f2538121b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 10 monclient(hunting): picked mon.noname-f con 0x7f25380a4920 addr [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-06-11T19:27:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25380a5240 0x7f2538122070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 10 monclient(hunting): picked mon.noname-g con 0x7f25380a5240 addr [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] 2024-06-11T19:27:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:27:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f25380a6da0 con 0x7f25380a4920 2024-06-11T19:27:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:27:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 -- 0x7f2538138e50 con 0x7f25380a5240 2024-06-11T19:27:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 10 monclient(hunting): start opening mon connection 2024-06-11T19:27:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f25380a6420 con 0x7f25380a2530 2024-06-11T19:27:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 10 monclient(hunting): _renew_subs 2024-06-11T19:27:13.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253effb640 10 monclient(hunting): authenticate will time out at 1843.104596s 2024-06-11T19:27:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 0x7f25381215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 0x7f25381215f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:37118/0 (socket says 172.21.15.98:37118) 2024-06-11T19:27:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 1 -- 172.21.15.98:0/2497293438 learned_addr learned my addr 172.21.15.98:0/2497293438 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:27:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 10 monclient(hunting): get_auth_request con 0x7f25380a2530 auth_method 0 2024-06-11T19:27:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:27:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:27:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:27:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:27:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:27:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:27:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253d598640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25380a5240 0x7f2538122070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f252ffff640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25380a4920 0x7f2538121b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f252ffff640 10 monclient(hunting): get_auth_request con 0x7f25380a4920 auth_method 0 2024-06-11T19:27:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f252ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:27:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f252ffff640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:27:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f252ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:27:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253d598640 10 monclient(hunting): get_auth_request con 0x7f25380a5240 auth_method 0 2024-06-11T19:27:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253d598640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:27:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253d598640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:27:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253d598640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:27:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 10 monclient(hunting): handle_auth_done global_id 124557 payload 824 2024-06-11T19:27:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 1 -- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25380a5240 msgr2=0x7f2538122070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:27:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25380a5240 0x7f2538122070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 1 -- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25380a4920 msgr2=0x7f2538121b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:27:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25380a4920 0x7f2538121b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 10 monclient: _finish_hunting 0 2024-06-11T19:27:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 1 monclient: found mon.noname-d 2024-06-11T19:27:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:27:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:27:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f253cd97640 1 -- 172.21.15.98:0/2497293438 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2538124ad0 con 0x7f25380a2530 2024-06-11T19:27:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.297+0000 7f252ffff640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25380a4920 0x7f2538121b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-11T19:27:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f253cd97640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 0x7f25381215f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f253400c820 tx=0x7f253400cc20 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:27:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 10 client.?.objecter ms_handle_connect 0x7f25380a2530 2024-06-11T19:27:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 10 client.?.objecter resend_mon_ops 2024-06-11T19:27:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 1 -- 172.21.15.98:0/2497293438 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2534015070 con 0x7f25380a2530 2024-06-11T19:27:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:27:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:27:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 10 monclient: dump: 2024-06-11T19:27:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:27:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:27:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.331 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:27:13.331 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:27:13.331 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:27:13.331 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:27:13.331 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:27:13.331 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:27:13.331 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:27:13.331 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:27:13.332 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:27:13.332 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:27:13.332 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:27:13.332 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:13.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 10 monclient: _finish_auth 0 2024-06-11T19:27:13.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 10 monclient: _check_auth_tickets 2024-06-11T19:27:13.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:27:13.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 1 -- 172.21.15.98:0/2497293438 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2534007ca0 con 0x7f25380a2530 2024-06-11T19:27:13.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f253effb640 5 monclient: authenticate success, global_id 124557 2024-06-11T19:27:13.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f253effb640 10 monclient: _renew_subs 2024-06-11T19:27:13.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f253effb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:27:13.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f253effb640 1 -- 172.21.15.98:0/2497293438 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2538123860 con 0x7f25380a2530 2024-06-11T19:27:13.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 10 monclient: handle_config config(0 keys) 2024-06-11T19:27:13.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 1 -- 172.21.15.98:0/2497293438 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2534007e50 con 0x7f25380a2530 2024-06-11T19:27:13.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:27:13.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:27:13.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252d7fa640 10 monclient: dump: 2024-06-11T19:27:13.334 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:27:13.334 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:27:13.334 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.335 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.335 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:27:13.335 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:27:13.335 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:27:13.335 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:27:13.335 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:27:13.335 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:27:13.336 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:27:13.336 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:27:13.336 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:27:13.336 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:27:13.336 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:27:13.336 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:13.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f253effb640 10 client.124557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:27:13.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f253effb640 10 monclient: _renew_subs 2024-06-11T19:27:13.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f252effd640 4 set_mon_vals no callback set 2024-06-11T19:27:13.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f253effb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:27:13.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f253effb640 1 -- 172.21.15.98:0/2497293438 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f25380a4eb0 con 0x7f25380a2530 2024-06-11T19:27:13.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.298+0000 7f253effb640 1 librados: init done 2024-06-11T19:27:13.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f252d7fa640 1 -- 172.21.15.98:0/2497293438 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f2534018070 con 0x7f25380a2530 2024-06-11T19:27:13.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f252d7fa640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f252403ff80 0x7f2524042440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f252d7fa640 1 -- 172.21.15.98:0/2497293438 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f2534051600 con 0x7f25380a2530 2024-06-11T19:27:13.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f252d7fa640 10 client.124557.objecter ms_dispatch 0x7f2538000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:27:13.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f252d7fa640 3 client.124557.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:27:13.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f252d7fa640 3 client.124557.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:27:13.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f253effb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:27:13.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f253effb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:27:13.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f253effb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:27:13.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f253effb640 1 -- 172.21.15.98:0/2497293438 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2538123620 con 0x7f25380a2530 2024-06-11T19:27:13.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f252d7fa640 20 client.124557.objecter dump_active .. 0 homeless 2024-06-11T19:27:13.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f252ffff640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f252403ff80 0x7f2524042440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.299+0000 7f252ffff640 10 monclient: get_auth_request con 0x7f252403ff80 auth_method 0 2024-06-11T19:27:13.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.300+0000 7f252d7fa640 10 client.124557.objecter ms_handle_connect 0x7f252403ff80 2024-06-11T19:27:13.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.300+0000 7f252ffff640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f252403ff80 0x7f2524042440 secure :-1 s=READY pgs=4348 cs=0 l=1 rev1=1 crypto rx=0x7f2528034000 tx=0x7f2528008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:27:13.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.304+0000 7f252d7fa640 1 -- 172.21.15.98:0/2497293438 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2534010350 con 0x7f25380a2530 2024-06-11T19:27:13.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.304+0000 7f252d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:27:13.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.304+0000 7f252d7fa640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:27:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.522+0000 7f253effb640 1 -- 172.21.15.98:0/2497293438 --> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f25380a6720 con 0x7f252403ff80 2024-06-11T19:27:13.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.525+0000 7f252d7fa640 1 -- 172.21.15.98:0/2497293438 <== mgr.4111 v2:172.21.15.117:6824/2398011028 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+113909 (secure 0 0 0) 0x7f25380a6720 con 0x7f252403ff80 2024-06-11T19:27:13.525 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-06-11T19:27:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-06-11T19:27:13.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 10 librados: watch_flush enter 2024-06-11T19:27:13.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 10 librados: watch_flush exit 2024-06-11T19:27:13.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 20 client.124557.objecter shutdown clearing up homeless session... 2024-06-11T19:27:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 10 client.124557.objecter successfully canceled tick 2024-06-11T19:27:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 -- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f252403ff80 msgr2=0x7f2524042440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:27:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f252403ff80 0x7f2524042440 secure :-1 s=READY pgs=4348 cs=0 l=1 rev1=1 crypto rx=0x7f2528034000 tx=0x7f2528008030 comp rx=0 tx=0).stop 2024-06-11T19:27:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 10 monclient: shutdown 2024-06-11T19:27:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:27:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 -- 172.21.15.98:0/2497293438 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 msgr2=0x7f25381215f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:27:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 0x7f25381215f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f253400c820 tx=0x7f253400cc20 comp rx=0 tx=0).stop 2024-06-11T19:27:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 -- 172.21.15.98:0/2497293438 shutdown_connections 2024-06-11T19:27:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f252403ff80 0x7f2524042440 unknown :-1 s=CLOSED pgs=4348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25380a5240 0x7f2538122070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25380a4920 0x7f2538121b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 --2- 172.21.15.98:0/2497293438 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25380a2530 0x7f25381215f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 -- 172.21.15.98:0/2497293438 >> 172.21.15.98:0/2497293438 conn(0x7f25380ab850 msgr2=0x7f2538137fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:27:13.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 -- 172.21.15.98:0/2497293438 shutdown_connections 2024-06-11T19:27:13.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 -- 172.21.15.98:0/2497293438 wait complete. 2024-06-11T19:27:13.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.527+0000 7f253effb640 1 librados: shutdown 2024-06-11T19:27:13.537 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1212,"stamp":"2024-06-11T19:27:11.636946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":108,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":471376,"kb_used_data":60832,"kb_used_omap":9,"kb_used_meta":410486,"kb_avail":565759664,"statfs":{"total":579820584960,"available":579337895936,"internally_reserved":0,"allocated":62291968,"data_stored":35204514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":420338336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002579"},"pg_stats":[{"pgid":"7.17","version":"0'0","reported_seq":1343,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:59.707041+0000","last_change":"2024-06-11T19:26:59.707067+0000","last_active":"2024-06-11T19:26:59.707041+0000","last_peered":"2024-06-11T19:26:59.707041+0000","last_clean":"2024-06-11T19:26:59.707041+0000","last_became_active":"2024-06-11T19:10:36.005579+0000","last_became_peered":"2024-06-11T19:10:36.005579+0000","last_unstale":"2024-06-11T19:26:59.707041+0000","last_undegraded":"2024-06-11T19:26:59.707041+0000","last_fullsized":"2024-06-11T19:26:59.707041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:59.706943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:59.706943+0000","last_clean_scrub_stamp":"2024-06-11T19:26:59.706943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T07:09:01.856301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1343,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:55.592755+0000","last_change":"2024-06-11T19:26:55.592780+0000","last_active":"2024-06-11T19:26:55.592755+0000","last_peered":"2024-06-11T19:26:55.592755+0000","last_clean":"2024-06-11T19:26:55.592755+0000","last_became_active":"2024-06-11T19:10:35.960336+0000","last_became_peered":"2024-06-11T19:10:35.960336+0000","last_unstale":"2024-06-11T19:26:55.592755+0000","last_undegraded":"2024-06-11T19:26:55.592755+0000","last_fullsized":"2024-06-11T19:26:55.592755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:55.592668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:55.592668+0000","last_clean_scrub_stamp":"2024-06-11T19:26:55.592668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:23:23.338585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1330,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:58.566422+0000","last_change":"2024-06-11T19:26:58.566441+0000","last_active":"2024-06-11T19:26:58.566422+0000","last_peered":"2024-06-11T19:26:58.566422+0000","last_clean":"2024-06-11T19:26:58.566422+0000","last_became_active":"2024-06-11T19:10:36.087715+0000","last_became_peered":"2024-06-11T19:10:36.087715+0000","last_unstale":"2024-06-11T19:26:58.566422+0000","last_undegraded":"2024-06-11T19:26:58.566422+0000","last_fullsized":"2024-06-11T19:26:58.566422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:58.566355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:58.566355+0000","last_clean_scrub_stamp":"2024-06-11T19:26:58.566355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:50:54.336791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:56.204303+0000","last_change":"2024-06-11T19:26:56.204303+0000","last_active":"2024-06-11T19:26:56.204303+0000","last_peered":"2024-06-11T19:26:56.204303+0000","last_clean":"2024-06-11T19:26:56.204303+0000","last_became_active":"2024-06-11T19:10:36.036974+0000","last_became_peered":"2024-06-11T19:10:36.036974+0000","last_unstale":"2024-06-11T19:26:56.204303+0000","last_undegraded":"2024-06-11T19:26:56.204303+0000","last_fullsized":"2024-06-11T19:26:56.204303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:56.204201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:56.204201+0000","last_clean_scrub_stamp":"2024-06-11T19:26:56.204201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:01:33.272195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1344,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:27:00.553369+0000","last_change":"2024-06-11T19:27:00.553395+0000","last_active":"2024-06-11T19:27:00.553369+0000","last_peered":"2024-06-11T19:27:00.553369+0000","last_clean":"2024-06-11T19:27:00.553369+0000","last_became_active":"2024-06-11T19:10:36.065872+0000","last_became_peered":"2024-06-11T19:10:36.065872+0000","last_unstale":"2024-06-11T19:27:00.553369+0000","last_undegraded":"2024-06-11T19:27:00.553369+0000","last_fullsized":"2024-06-11T19:27:00.553369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:27:00.553281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:27:00.553281+0000","last_clean_scrub_stamp":"2024-06-11T19:27:00.553281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T22:20:22.171782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1338,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:57.104002+0000","last_change":"2024-06-11T19:26:57.104029+0000","last_active":"2024-06-11T19:26:57.104002+0000","last_peered":"2024-06-11T19:26:57.104002+0000","last_clean":"2024-06-11T19:26:57.104002+0000","last_became_active":"2024-06-11T19:10:36.076424+0000","last_became_peered":"2024-06-11T19:10:36.076424+0000","last_unstale":"2024-06-11T19:26:57.104002+0000","last_undegraded":"2024-06-11T19:26:57.104002+0000","last_fullsized":"2024-06-11T19:26:57.104002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:57.103907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:57.103907+0000","last_clean_scrub_stamp":"2024-06-11T19:26:57.103907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:11:01.119025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:55.157970+0000","last_change":"2024-06-11T19:26:55.157995+0000","last_active":"2024-06-11T19:26:55.157970+0000","last_peered":"2024-06-11T19:26:55.157970+0000","last_clean":"2024-06-11T19:26:55.157970+0000","last_became_active":"2024-06-11T19:10:36.024538+0000","last_became_peered":"2024-06-11T19:10:36.024538+0000","last_unstale":"2024-06-11T19:26:55.157970+0000","last_undegraded":"2024-06-11T19:26:55.157970+0000","last_fullsized":"2024-06-11T19:26:55.157970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:55.157883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:55.157883+0000","last_clean_scrub_stamp":"2024-06-11T19:26:55.157883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T06:53:09.017836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:58.284887+0000","last_change":"2024-06-11T19:26:58.284929+0000","last_active":"2024-06-11T19:26:58.284887+0000","last_peered":"2024-06-11T19:26:58.284887+0000","last_clean":"2024-06-11T19:26:58.284887+0000","last_became_active":"2024-06-11T19:10:36.085399+0000","last_became_peered":"2024-06-11T19:10:36.085399+0000","last_unstale":"2024-06-11T19:26:58.284887+0000","last_undegraded":"2024-06-11T19:26:58.284887+0000","last_fullsized":"2024-06-11T19:26:58.284887+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:58.284774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:58.284774+0000","last_clean_scrub_stamp":"2024-06-11T19:26:58.284774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:36:49.071832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1343,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:53.475862+0000","last_change":"2024-06-11T19:26:53.475901+0000","last_active":"2024-06-11T19:26:53.475862+0000","last_peered":"2024-06-11T19:26:53.475862+0000","last_clean":"2024-06-11T19:26:53.475862+0000","last_became_active":"2024-06-11T19:10:35.976679+0000","last_became_peered":"2024-06-11T19:10:35.976679+0000","last_unstale":"2024-06-11T19:26:53.475862+0000","last_undegraded":"2024-06-11T19:26:53.475862+0000","last_fullsized":"2024-06-11T19:26:53.475862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:53.475735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:53.475735+0000","last_clean_scrub_stamp":"2024-06-11T19:26:53.475735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:35:31.799937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1344,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:54.486959+0000","last_change":"2024-06-11T19:26:54.486982+0000","last_active":"2024-06-11T19:26:54.486959+0000","last_peered":"2024-06-11T19:26:54.486959+0000","last_clean":"2024-06-11T19:26:54.486959+0000","last_became_active":"2024-06-11T19:10:35.985449+0000","last_became_peered":"2024-06-11T19:10:35.985449+0000","last_unstale":"2024-06-11T19:26:54.486959+0000","last_undegraded":"2024-06-11T19:26:54.486959+0000","last_fullsized":"2024-06-11T19:26:54.486959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:54.486850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:54.486850+0000","last_clean_scrub_stamp":"2024-06-11T19:26:54.486850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:27:27.135736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1344,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:56.598921+0000","last_change":"2024-06-11T19:26:56.598955+0000","last_active":"2024-06-11T19:26:56.598921+0000","last_peered":"2024-06-11T19:26:56.598921+0000","last_clean":"2024-06-11T19:26:56.598921+0000","last_became_active":"2024-06-11T19:10:35.962677+0000","last_became_peered":"2024-06-11T19:10:35.962677+0000","last_unstale":"2024-06-11T19:26:56.598921+0000","last_undegraded":"2024-06-11T19:26:56.598921+0000","last_fullsized":"2024-06-11T19:26:56.598921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:56.598828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:56.598828+0000","last_clean_scrub_stamp":"2024-06-11T19:26:56.598828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T05:48:03.045051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1331,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:55.642660+0000","last_change":"2024-06-11T19:26:55.642682+0000","last_active":"2024-06-11T19:26:55.642660+0000","last_peered":"2024-06-11T19:26:55.642660+0000","last_clean":"2024-06-11T19:26:55.642660+0000","last_became_active":"2024-06-11T19:10:36.056338+0000","last_became_peered":"2024-06-11T19:10:36.056338+0000","last_unstale":"2024-06-11T19:26:55.642660+0000","last_undegraded":"2024-06-11T19:26:55.642660+0000","last_fullsized":"2024-06-11T19:26:55.642660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:55.642576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:55.642576+0000","last_clean_scrub_stamp":"2024-06-11T19:26:55.642576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T05:50:59.330505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"685'74","reported_seq":1449,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:51.598855+0000","last_change":"2024-06-11T19:26:51.598881+0000","last_active":"2024-06-11T19:26:51.598855+0000","last_peered":"2024-06-11T19:26:51.598855+0000","last_clean":"2024-06-11T19:26:51.598855+0000","last_became_active":"2024-06-11T19:09:26.673120+0000","last_became_peered":"2024-06-11T19:09:26.673120+0000","last_unstale":"2024-06-11T19:26:51.598855+0000","last_undegraded":"2024-06-11T19:26:51.598855+0000","last_fullsized":"2024-06-11T19:26:51.598855+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"685'74","last_scrub_stamp":"2024-06-11T19:26:51.598798+0000","last_deep_scrub":"685'74","last_deep_scrub_stamp":"2024-06-11T19:26:51.598798+0000","last_clean_scrub_stamp":"2024-06-11T19:26:51.598798+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:53:02.852612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1351,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:53.600588+0000","last_change":"2024-06-11T19:26:53.600620+0000","last_active":"2024-06-11T19:26:53.600588+0000","last_peered":"2024-06-11T19:26:53.600588+0000","last_clean":"2024-06-11T19:26:53.600588+0000","last_became_active":"2024-06-11T19:09:33.724870+0000","last_became_peered":"2024-06-11T19:09:33.724870+0000","last_unstale":"2024-06-11T19:26:53.600588+0000","last_undegraded":"2024-06-11T19:26:53.600588+0000","last_fullsized":"2024-06-11T19:26:53.600588+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:53.600491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:53.600491+0000","last_clean_scrub_stamp":"2024-06-11T19:26:53.600491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T22:42:07.401153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1344,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:55.507804+0000","last_change":"2024-06-11T19:26:55.507827+0000","last_active":"2024-06-11T19:26:55.507804+0000","last_peered":"2024-06-11T19:26:55.507804+0000","last_clean":"2024-06-11T19:26:55.507804+0000","last_became_active":"2024-06-11T19:10:35.988487+0000","last_became_peered":"2024-06-11T19:10:35.988487+0000","last_unstale":"2024-06-11T19:26:55.507804+0000","last_undegraded":"2024-06-11T19:26:55.507804+0000","last_fullsized":"2024-06-11T19:26:55.507804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:55.507726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:55.507726+0000","last_clean_scrub_stamp":"2024-06-11T19:26:55.507726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:13:46.468515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1338,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:51.613359+0000","last_change":"2024-06-11T19:26:51.613377+0000","last_active":"2024-06-11T19:26:51.613359+0000","last_peered":"2024-06-11T19:26:51.613359+0000","last_clean":"2024-06-11T19:26:51.613359+0000","last_became_active":"2024-06-11T19:09:33.728930+0000","last_became_peered":"2024-06-11T19:09:33.728930+0000","last_unstale":"2024-06-11T19:26:51.613359+0000","last_undegraded":"2024-06-11T19:26:51.613359+0000","last_fullsized":"2024-06-11T19:26:51.613359+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:51.613292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:51.613292+0000","last_clean_scrub_stamp":"2024-06-11T19:26:51.613292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:10:51.995760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1331,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:56.631270+0000","last_change":"2024-06-11T19:26:56.631310+0000","last_active":"2024-06-11T19:26:56.631270+0000","last_peered":"2024-06-11T19:26:56.631270+0000","last_clean":"2024-06-11T19:26:56.631270+0000","last_became_active":"2024-06-11T19:10:36.086670+0000","last_became_peered":"2024-06-11T19:10:36.086670+0000","last_unstale":"2024-06-11T19:26:56.631270+0000","last_undegraded":"2024-06-11T19:26:56.631270+0000","last_fullsized":"2024-06-11T19:26:56.631270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:56.631185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:56.631185+0000","last_clean_scrub_stamp":"2024-06-11T19:26:56.631185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T22:07:32.789444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1351,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:52.460209+0000","last_change":"2024-06-11T19:26:52.460234+0000","last_active":"2024-06-11T19:26:52.460209+0000","last_peered":"2024-06-11T19:26:52.460209+0000","last_clean":"2024-06-11T19:26:52.460209+0000","last_became_active":"2024-06-11T19:09:33.728948+0000","last_became_peered":"2024-06-11T19:09:33.728948+0000","last_unstale":"2024-06-11T19:26:52.460209+0000","last_undegraded":"2024-06-11T19:26:52.460209+0000","last_fullsized":"2024-06-11T19:26:52.460209+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:52.460123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:52.460123+0000","last_clean_scrub_stamp":"2024-06-11T19:26:52.460123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T06:09:35.873059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1338,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:54.122321+0000","last_change":"2024-06-11T19:26:54.122346+0000","last_active":"2024-06-11T19:26:54.122321+0000","last_peered":"2024-06-11T19:26:54.122321+0000","last_clean":"2024-06-11T19:26:54.122321+0000","last_became_active":"2024-06-11T19:10:35.937161+0000","last_became_peered":"2024-06-11T19:10:35.937161+0000","last_unstale":"2024-06-11T19:26:54.122321+0000","last_undegraded":"2024-06-11T19:26:54.122321+0000","last_fullsized":"2024-06-11T19:26:54.122321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:54.122206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:54.122206+0000","last_clean_scrub_stamp":"2024-06-11T19:26:54.122206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:10:38.405942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1339,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:50.704800+0000","last_change":"2024-06-11T19:26:50.704819+0000","last_active":"2024-06-11T19:26:50.704800+0000","last_peered":"2024-06-11T19:26:50.704800+0000","last_clean":"2024-06-11T19:26:50.704800+0000","last_became_active":"2024-06-11T19:09:33.722180+0000","last_became_peered":"2024-06-11T19:09:33.722180+0000","last_unstale":"2024-06-11T19:26:50.704800+0000","last_undegraded":"2024-06-11T19:26:50.704800+0000","last_fullsized":"2024-06-11T19:26:50.704800+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:50.704733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:50.704733+0000","last_clean_scrub_stamp":"2024-06-11T19:26:50.704733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:05:08.805881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1344,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:27:00.739844+0000","last_change":"2024-06-11T19:27:00.739872+0000","last_active":"2024-06-11T19:27:00.739844+0000","last_peered":"2024-06-11T19:27:00.739844+0000","last_clean":"2024-06-11T19:27:00.739844+0000","last_became_active":"2024-06-11T19:10:36.016575+0000","last_became_peered":"2024-06-11T19:10:36.016575+0000","last_unstale":"2024-06-11T19:27:00.739844+0000","last_undegraded":"2024-06-11T19:27:00.739844+0000","last_fullsized":"2024-06-11T19:27:00.739844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:27:00.739763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:27:00.739763+0000","last_clean_scrub_stamp":"2024-06-11T19:27:00.739763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:15:45.332001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1349,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:53.150520+0000","last_change":"2024-06-11T19:26:53.150546+0000","last_active":"2024-06-11T19:26:53.150520+0000","last_peered":"2024-06-11T19:26:53.150520+0000","last_clean":"2024-06-11T19:26:53.150520+0000","last_became_active":"2024-06-11T19:09:33.728897+0000","last_became_peered":"2024-06-11T19:09:33.728897+0000","last_unstale":"2024-06-11T19:26:53.150520+0000","last_undegraded":"2024-06-11T19:26:53.150520+0000","last_fullsized":"2024-06-11T19:26:53.150520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:53.150431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:53.150431+0000","last_clean_scrub_stamp":"2024-06-11T19:26:53.150431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T03:01:51.174758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"685'8","reported_seq":1345,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:54.638817+0000","last_change":"2024-06-11T19:26:54.638848+0000","last_active":"2024-06-11T19:26:54.638817+0000","last_peered":"2024-06-11T19:26:54.638817+0000","last_clean":"2024-06-11T19:26:54.638817+0000","last_became_active":"2024-06-11T19:10:36.032874+0000","last_became_peered":"2024-06-11T19:10:36.032874+0000","last_unstale":"2024-06-11T19:26:54.638817+0000","last_undegraded":"2024-06-11T19:26:54.638817+0000","last_fullsized":"2024-06-11T19:26:54.638817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"685'8","last_scrub_stamp":"2024-06-11T19:26:54.638736+0000","last_deep_scrub":"685'8","last_deep_scrub_stamp":"2024-06-11T19:26:54.638736+0000","last_clean_scrub_stamp":"2024-06-11T19:26:54.638736+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T20:58:02.327690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1351,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:52.578326+0000","last_change":"2024-06-11T19:26:52.578345+0000","last_active":"2024-06-11T19:26:52.578326+0000","last_peered":"2024-06-11T19:26:52.578326+0000","last_clean":"2024-06-11T19:26:52.578326+0000","last_became_active":"2024-06-11T19:09:33.720599+0000","last_became_peered":"2024-06-11T19:09:33.720599+0000","last_unstale":"2024-06-11T19:26:52.578326+0000","last_undegraded":"2024-06-11T19:26:52.578326+0000","last_fullsized":"2024-06-11T19:26:52.578326+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:52.578261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:52.578261+0000","last_clean_scrub_stamp":"2024-06-11T19:26:52.578261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T01:25:15.387690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1345,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:57.635220+0000","last_change":"2024-06-11T19:26:57.635259+0000","last_active":"2024-06-11T19:26:57.635220+0000","last_peered":"2024-06-11T19:26:57.635220+0000","last_clean":"2024-06-11T19:26:57.635220+0000","last_became_active":"2024-06-11T19:10:35.976675+0000","last_became_peered":"2024-06-11T19:10:35.976675+0000","last_unstale":"2024-06-11T19:26:57.635220+0000","last_undegraded":"2024-06-11T19:26:57.635220+0000","last_fullsized":"2024-06-11T19:26:57.635220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:57.635105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:57.635105+0000","last_clean_scrub_stamp":"2024-06-11T19:26:57.635105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T00:35:27.933204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1345,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:53.124267+0000","last_change":"2024-06-11T19:26:53.124267+0000","last_active":"2024-06-11T19:26:53.124267+0000","last_peered":"2024-06-11T19:26:53.124267+0000","last_clean":"2024-06-11T19:26:53.124267+0000","last_became_active":"2024-06-11T19:09:33.724912+0000","last_became_peered":"2024-06-11T19:09:33.724912+0000","last_unstale":"2024-06-11T19:26:53.124267+0000","last_undegraded":"2024-06-11T19:26:53.124267+0000","last_fullsized":"2024-06-11T19:26:53.124267+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:53.124147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:53.124147+0000","last_clean_scrub_stamp":"2024-06-11T19:26:53.124147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:37:25.488752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1344,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:59.542851+0000","last_change":"2024-06-11T19:26:59.542886+0000","last_active":"2024-06-11T19:26:59.542851+0000","last_peered":"2024-06-11T19:26:59.542851+0000","last_clean":"2024-06-11T19:26:59.542851+0000","last_became_active":"2024-06-11T19:10:36.071353+0000","last_became_peered":"2024-06-11T19:10:36.071353+0000","last_unstale":"2024-06-11T19:26:59.542851+0000","last_undegraded":"2024-06-11T19:26:59.542851+0000","last_fullsized":"2024-06-11T19:26:59.542851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:59.542764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:59.542764+0000","last_clean_scrub_stamp":"2024-06-11T19:26:59.542764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T23:20:11.481965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1351,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:54.587467+0000","last_change":"2024-06-11T19:26:54.587467+0000","last_active":"2024-06-11T19:26:54.587467+0000","last_peered":"2024-06-11T19:26:54.587467+0000","last_clean":"2024-06-11T19:26:54.587467+0000","last_became_active":"2024-06-11T19:09:33.724778+0000","last_became_peered":"2024-06-11T19:09:33.724778+0000","last_unstale":"2024-06-11T19:26:54.587467+0000","last_undegraded":"2024-06-11T19:26:54.587467+0000","last_fullsized":"2024-06-11T19:26:54.587467+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:54.587320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:54.587320+0000","last_clean_scrub_stamp":"2024-06-11T19:26:54.587320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T22:15:33.520047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1344,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:58.518570+0000","last_change":"2024-06-11T19:26:58.518609+0000","last_active":"2024-06-11T19:26:58.518570+0000","last_peered":"2024-06-11T19:26:58.518570+0000","last_clean":"2024-06-11T19:26:58.518570+0000","last_became_active":"2024-06-11T19:10:36.035136+0000","last_became_peered":"2024-06-11T19:10:36.035136+0000","last_unstale":"2024-06-11T19:26:58.518570+0000","last_undegraded":"2024-06-11T19:26:58.518570+0000","last_fullsized":"2024-06-11T19:26:58.518570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:58.518433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:58.518433+0000","last_clean_scrub_stamp":"2024-06-11T19:26:58.518433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:43:08.848270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1344,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:56.463594+0000","last_change":"2024-06-11T19:26:56.463594+0000","last_active":"2024-06-11T19:26:56.463594+0000","last_peered":"2024-06-11T19:26:56.463594+0000","last_clean":"2024-06-11T19:26:56.463594+0000","last_became_active":"2024-06-11T19:10:36.044757+0000","last_became_peered":"2024-06-11T19:10:36.044757+0000","last_unstale":"2024-06-11T19:26:56.463594+0000","last_undegraded":"2024-06-11T19:26:56.463594+0000","last_fullsized":"2024-06-11T19:26:56.463594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:56.463485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:56.463485+0000","last_clean_scrub_stamp":"2024-06-11T19:26:56.463485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:55:59.220195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1331,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:52.576181+0000","last_change":"2024-06-11T19:26:52.576181+0000","last_active":"2024-06-11T19:26:52.576181+0000","last_peered":"2024-06-11T19:26:52.576181+0000","last_clean":"2024-06-11T19:26:52.576181+0000","last_became_active":"2024-06-11T19:10:36.000792+0000","last_became_peered":"2024-06-11T19:10:36.000792+0000","last_unstale":"2024-06-11T19:26:52.576181+0000","last_undegraded":"2024-06-11T19:26:52.576181+0000","last_fullsized":"2024-06-11T19:26:52.576181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:52.576097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:52.576097+0000","last_clean_scrub_stamp":"2024-06-11T19:26:52.576097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T06:01:57.279693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1344,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:58.668244+0000","last_change":"2024-06-11T19:26:58.668269+0000","last_active":"2024-06-11T19:26:58.668244+0000","last_peered":"2024-06-11T19:26:58.668244+0000","last_clean":"2024-06-11T19:26:58.668244+0000","last_became_active":"2024-06-11T19:10:35.988056+0000","last_became_peered":"2024-06-11T19:10:35.988056+0000","last_unstale":"2024-06-11T19:26:58.668244+0000","last_undegraded":"2024-06-11T19:26:58.668244+0000","last_fullsized":"2024-06-11T19:26:58.668244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:58.668173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:58.668173+0000","last_clean_scrub_stamp":"2024-06-11T19:26:58.668173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:24:25.648020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1342,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:54.170448+0000","last_change":"2024-06-11T19:26:54.170448+0000","last_active":"2024-06-11T19:26:54.170448+0000","last_peered":"2024-06-11T19:26:54.170448+0000","last_clean":"2024-06-11T19:26:54.170448+0000","last_became_active":"2024-06-11T19:10:36.025371+0000","last_became_peered":"2024-06-11T19:10:36.025371+0000","last_unstale":"2024-06-11T19:26:54.170448+0000","last_undegraded":"2024-06-11T19:26:54.170448+0000","last_fullsized":"2024-06-11T19:26:54.170448+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:54.170348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:54.170348+0000","last_clean_scrub_stamp":"2024-06-11T19:26:54.170348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:10:45.396036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1330,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:57.588596+0000","last_change":"2024-06-11T19:26:57.588622+0000","last_active":"2024-06-11T19:26:57.588596+0000","last_peered":"2024-06-11T19:26:57.588596+0000","last_clean":"2024-06-11T19:26:57.588596+0000","last_became_active":"2024-06-11T19:10:36.090801+0000","last_became_peered":"2024-06-11T19:10:36.090801+0000","last_unstale":"2024-06-11T19:26:57.588596+0000","last_undegraded":"2024-06-11T19:26:57.588596+0000","last_fullsized":"2024-06-11T19:26:57.588596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:57.588506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:57.588506+0000","last_clean_scrub_stamp":"2024-06-11T19:26:57.588506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T05:19:51.677341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1341,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:57.235613+0000","last_change":"2024-06-11T19:26:57.235635+0000","last_active":"2024-06-11T19:26:57.235613+0000","last_peered":"2024-06-11T19:26:57.235613+0000","last_clean":"2024-06-11T19:26:57.235613+0000","last_became_active":"2024-06-11T19:10:36.057852+0000","last_became_peered":"2024-06-11T19:10:36.057852+0000","last_unstale":"2024-06-11T19:26:57.235613+0000","last_undegraded":"2024-06-11T19:26:57.235613+0000","last_fullsized":"2024-06-11T19:26:57.235613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:57.235531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:57.235531+0000","last_clean_scrub_stamp":"2024-06-11T19:26:57.235531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:24:10.539873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1345,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:57.507836+0000","last_change":"2024-06-11T19:26:57.507881+0000","last_active":"2024-06-11T19:26:57.507836+0000","last_peered":"2024-06-11T19:26:57.507836+0000","last_clean":"2024-06-11T19:26:57.507836+0000","last_became_active":"2024-06-11T19:10:36.042135+0000","last_became_peered":"2024-06-11T19:10:36.042135+0000","last_unstale":"2024-06-11T19:26:57.507836+0000","last_undegraded":"2024-06-11T19:26:57.507836+0000","last_fullsized":"2024-06-11T19:26:57.507836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:57.507748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:57.507748+0000","last_clean_scrub_stamp":"2024-06-11T19:26:57.507748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T04:12:14.154478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1331,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:51.736322+0000","last_change":"2024-06-11T19:26:51.736357+0000","last_active":"2024-06-11T19:26:51.736322+0000","last_peered":"2024-06-11T19:26:51.736322+0000","last_clean":"2024-06-11T19:26:51.736322+0000","last_became_active":"2024-06-11T19:10:36.026278+0000","last_became_peered":"2024-06-11T19:10:36.026278+0000","last_unstale":"2024-06-11T19:26:51.736322+0000","last_undegraded":"2024-06-11T19:26:51.736322+0000","last_fullsized":"2024-06-11T19:26:51.736322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:51.736247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:51.736247+0000","last_clean_scrub_stamp":"2024-06-11T19:26:51.736247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:18:39.634952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1337,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:55.112249+0000","last_change":"2024-06-11T19:26:55.112278+0000","last_active":"2024-06-11T19:26:55.112249+0000","last_peered":"2024-06-11T19:26:55.112249+0000","last_clean":"2024-06-11T19:26:55.112249+0000","last_became_active":"2024-06-11T19:10:36.028329+0000","last_became_peered":"2024-06-11T19:10:36.028329+0000","last_unstale":"2024-06-11T19:26:55.112249+0000","last_undegraded":"2024-06-11T19:26:55.112249+0000","last_fullsized":"2024-06-11T19:26:55.112249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:55.112160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:55.112160+0000","last_clean_scrub_stamp":"2024-06-11T19:26:55.112160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T21:15:55.123099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1343,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:27:01.761370+0000","last_change":"2024-06-11T19:27:01.761393+0000","last_active":"2024-06-11T19:27:01.761370+0000","last_peered":"2024-06-11T19:27:01.761370+0000","last_clean":"2024-06-11T19:27:01.761370+0000","last_became_active":"2024-06-11T19:10:36.082897+0000","last_became_peered":"2024-06-11T19:10:36.082897+0000","last_unstale":"2024-06-11T19:27:01.761370+0000","last_undegraded":"2024-06-11T19:27:01.761370+0000","last_fullsized":"2024-06-11T19:27:01.761370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:27:01.761281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:27:01.761281+0000","last_clean_scrub_stamp":"2024-06-11T19:27:01.761281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T22:55:37.985171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1330,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:53.600503+0000","last_change":"2024-06-11T19:26:53.600522+0000","last_active":"2024-06-11T19:26:53.600503+0000","last_peered":"2024-06-11T19:26:53.600503+0000","last_clean":"2024-06-11T19:26:53.600503+0000","last_became_active":"2024-06-11T19:10:36.002730+0000","last_became_peered":"2024-06-11T19:10:36.002730+0000","last_unstale":"2024-06-11T19:26:53.600503+0000","last_undegraded":"2024-06-11T19:26:53.600503+0000","last_fullsized":"2024-06-11T19:26:53.600503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:53.600441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:53.600441+0000","last_clean_scrub_stamp":"2024-06-11T19:26:53.600441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-13T02:51:13.995938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1337,"reported_epoch":685,"state":"active+clean","last_fresh":"2024-06-11T19:26:56.138320+0000","last_change":"2024-06-11T19:26:56.138344+0000","last_active":"2024-06-11T19:26:56.138320+0000","last_peered":"2024-06-11T19:26:56.138320+0000","last_clean":"2024-06-11T19:26:56.138320+0000","last_became_active":"2024-06-11T19:10:36.075888+0000","last_became_peered":"2024-06-11T19:10:36.075888+0000","last_unstale":"2024-06-11T19:26:56.138320+0000","last_undegraded":"2024-06-11T19:26:56.138320+0000","last_fullsized":"2024-06-11T19:26:56.138320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-11T19:26:56.138238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-11T19:26:56.138238+0000","last_clean_scrub_stamp":"2024-06-11T19:26:56.138238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-12T19:38:27.479650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607768,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77240,"kb_used_data":9968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294600,"statfs":{"total":96636764160,"available":96557670400,"internally_reserved":0,"allocated":10207232,"data_stored":5691113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81920,"kb_used_data":10488,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289920,"statfs":{"total":96636764160,"available":96552878080,"internally_reserved":0,"allocated":10739712,"data_stored":6220041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640472,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86076,"kb_used_data":10484,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94285764,"statfs":{"total":96636764160,"available":96548622336,"internally_reserved":0,"allocated":10735616,"data_stored":6220036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640473,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73076,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94298764,"statfs":{"total":96636764160,"available":96561934336,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640472,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67508,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94304332,"statfs":{"total":96636764160,"available":96567635968,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640474,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85556,"kb_used_data":9964,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286284,"statfs":{"total":96636764160,"available":96549154816,"internally_reserved":0,"allocated":10203136,"data_stored":5691108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-11T19:27:13.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-06-11T19:27:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.688+0000 7f87b4cd1640 10 monclient: get_monmap_and_config 2024-06-11T19:27:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.688+0000 7f87b4cd1640 10 monclient: build_initial_monmap 2024-06-11T19:27:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.688+0000 7f87b4cd1640 10 monclient: monmap: 2024-06-11T19:27:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:27:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:27:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:27:13.689911+0000 2024-06-11T19:27:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:27:13.689911+0000 2024-06-11T19:27:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:27:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:27:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-06-11T19:27:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.117:6789/0 mon.noname-f-legacy 2024-06-11T19:27:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:27:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:27:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:27:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:27:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:27:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:27:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:27:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-06-11T19:27:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.117:3300/0 mon.noname-f 2024-06-11T19:27:13.691 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:13.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.689+0000 7f87b4cd1640 1 Processor -- start 2024-06-11T19:27:13.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.689+0000 7f87b4cd1640 1 -- start start 2024-06-11T19:27:13.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.689+0000 7f87b4cd1640 10 monclient: init 2024-06-11T19:27:13.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 10 monclient: _reopen_session rank -1 2024-06-11T19:27:13.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 10 monclient: _add_conns ranks=[0,6,4,5,1,8,3,9,7,10,2] 2024-06-11T19:27:13.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87b00a4920 addr v1:172.21.15.98:6789/0 2024-06-11T19:27:13.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b00a2530 0x7f87b013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 10 monclient(hunting): picked mon.noname-d con 0x7f87b00a2530 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:27:13.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f87b013e1a0 0x7f87b0140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 10 monclient(hunting): picked mon.noname-c con 0x7f87b013e1a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-06-11T19:27:13.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87b00a7140 con 0x7f87b00a4920 2024-06-11T19:27:13.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:27:13.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f87b00a2b50 con 0x7f87b013e1a0 2024-06-11T19:27:13.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:27:13.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f87b00a2e30 con 0x7f87b00a2530 2024-06-11T19:27:13.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 10 monclient(hunting): _renew_subs 2024-06-11T19:27:13.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87b4cd1640 10 monclient(hunting): authenticate will time out at 1843.497172s 2024-06-11T19:27:13.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b00a2530 0x7f87b013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87aed9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f87b013e1a0 0x7f87b0140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b00a2530 0x7f87b013dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:37126/0 (socket says 172.21.15.98:37126) 2024-06-11T19:27:13.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87aed9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f87b013e1a0 0x7f87b0140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:53608/0 (socket says 172.21.15.98:53608) 2024-06-11T19:27:13.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 1 -- 172.21.15.98:0/1600365206 learned_addr learned my addr 172.21.15.98:0/1600365206 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:27:13.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 10 monclient(hunting): get_auth_request con 0x7f87b00a2530 auth_method 0 2024-06-11T19:27:13.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:27:13.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:27:13.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:27:13.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87ae59c640 1 --1- 172.21.15.98:0/1600365206 >> v1:172.21.15.98:6789/0 conn(0x7f87b00a4920 0x7f87b00a4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53144/0 (socket says 172.21.15.98:53144) 2024-06-11T19:27:13.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87aed9d640 10 monclient(hunting): get_auth_request con 0x7f87b013e1a0 auth_method 0 2024-06-11T19:27:13.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:27:13.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87aed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:27:13.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:27:13.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:27:13.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:27:13.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:27:13.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:27:13.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:27:13.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:27:13.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 10 monclient(hunting): handle_auth_done global_id 124566 payload 274 2024-06-11T19:27:13.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 1 -- 172.21.15.98:0/1600365206 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f87b013e1a0 msgr2=0x7f87b0140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-11T19:27:13.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 1 --2- 172.21.15.98:0/1600365206 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f87b013e1a0 0x7f87b0140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 1 -- 172.21.15.98:0/1600365206 >> v1:172.21.15.98:6789/0 conn(0x7f87b00a4920 legacy=0x7f87b00a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:27:13.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 10 monclient: _finish_hunting 0 2024-06-11T19:27:13.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 1 monclient: found mon.noname-d 2024-06-11T19:27:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:27:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:27:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.690+0000 7f87add9b640 1 -- 172.21.15.98:0/1600365206 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b00a3110 con 0x7f87b00a2530 2024-06-11T19:27:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87add9b640 1 --2- 172.21.15.98:0/1600365206 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b00a2530 0x7f87b013dc60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f87a00098a0 tx=0x7f87a002b840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=da90326a777e82f7 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:27:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 1 -- 172.21.15.98:0/1600365206 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f87a0031a10 con 0x7f87b00a2530 2024-06-11T19:27:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:27:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:27:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 10 monclient: dump: 2024-06-11T19:27:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:27:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:27:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:27:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:27:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:27:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:27:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:27:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:27:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:27:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:27:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:27:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:27:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:27:13.706 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 10 monclient: _finish_auth 0 2024-06-11T19:27:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 10 monclient: _check_auth_tickets 2024-06-11T19:27:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:27:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 1 -- 172.21.15.98:0/1600365206 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87a0037070 con 0x7f87b00a2530 2024-06-11T19:27:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87b4cd1640 5 monclient: authenticate success, global_id 124566 2024-06-11T19:27:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87b4cd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-11T19:27:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 10 monclient: handle_config config(0 keys) 2024-06-11T19:27:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87b4cd1640 10 monclient: get_monmap_and_config success 2024-06-11T19:27:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 1 -- 172.21.15.98:0/1600365206 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f87a0031c10 con 0x7f87b00a2530 2024-06-11T19:27:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:27:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87b4cd1640 4 set_mon_vals no callback set 2024-06-11T19:27:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:27:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.691+0000 7f87ad59a640 10 monclient: dump: 2024-06-11T19:27:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:27:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:27:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:27:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:27:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:27:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:27:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:27:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:27:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:27:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:27:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:27:13.710 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:27:13.710 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:27:13.710 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:13.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.692+0000 7f87b4cd1640 10 monclient: shutdown 2024-06-11T19:27:13.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.692+0000 7f87b4cd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:27:13.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.692+0000 7f87b4cd1640 1 -- 172.21.15.98:0/1600365206 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b00a2530 msgr2=0x7f87b013dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:27:13.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.692+0000 7f87b4cd1640 1 --2- 172.21.15.98:0/1600365206 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b00a2530 0x7f87b013dc60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f87a00098a0 tx=0x7f87a002b840 comp rx=0 tx=0).stop 2024-06-11T19:27:13.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.692+0000 7f87b4cd1640 1 -- 172.21.15.98:0/1600365206 shutdown_connections 2024-06-11T19:27:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.692+0000 7f87b4cd1640 1 --2- 172.21.15.98:0/1600365206 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f87b013e1a0 0x7f87b0140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.692+0000 7f87b4cd1640 1 --2- 172.21.15.98:0/1600365206 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b00a2530 0x7f87b013dc60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.692+0000 7f87b4cd1640 1 -- 172.21.15.98:0/1600365206 >> 172.21.15.98:0/1600365206 conn(0x7f87b00ab850 msgr2=0x7f87b00a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:27:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.692+0000 7f87b4cd1640 1 -- 172.21.15.98:0/1600365206 shutdown_connections 2024-06-11T19:27:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.692+0000 7f87b4cd1640 1 -- 172.21.15.98:0/1600365206 wait complete. 2024-06-11T19:27:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.692+0000 7f87b4cd1640 10 monclient: build_initial_monmap 2024-06-11T19:27:13.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.692+0000 7f87b4cd1640 10 monclient: monmap: 2024-06-11T19:27:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-06-11T19:27:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-11T19:27:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-06-11T19:27:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-06-11T19:27:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-06-11T19:27:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-06-11T19:27:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-06-11T19:27:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.noname-f 2024-06-11T19:27:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-06-11T19:27:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.noname-g 2024-06-11T19:27:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-06-11T19:27:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.noname-h 2024-06-11T19:27:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-06-11T19:27:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.noname-i 2024-06-11T19:27:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-06-11T19:27:13.713 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:13.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 1 librados: starting msgr at 2024-06-11T19:27:13.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 1 librados: starting objecter 2024-06-11T19:27:13.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 1 Processor -- start 2024-06-11T19:27:13.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 1 -- start start 2024-06-11T19:27:13.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 1 librados: setting wanted keys 2024-06-11T19:27:13.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 1 librados: calling monclient init 2024-06-11T19:27:13.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 10 monclient: init 2024-06-11T19:27:13.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 10 monclient: _reopen_session rank -1 2024-06-11T19:27:13.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 10 monclient: _add_conns ranks=[5,7,6,4,0,2,8,3,1] 2024-06-11T19:27:13.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 1 --2- >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f87b00a2530 0x7f87b0135c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 10 monclient(hunting): picked mon.noname-h con 0x7f87b00a2530 addr [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] 2024-06-11T19:27:13.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 1 --2- >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f87b00a4920 0x7f87b0136180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 10 monclient(hunting): picked mon.noname-i con 0x7f87b00a4920 addr [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] 2024-06-11T19:27:13.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b013e1a0 0x7f87b009ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 10 monclient(hunting): picked mon.noname-d con 0x7f87b013e1a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-06-11T19:27:13.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:27:13.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 1 -- --> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] -- mon_getmap magic: 0 -- 0x7f87b0138e50 con 0x7f87b00a2530 2024-06-11T19:27:13.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:27:13.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f87b00a84b0 con 0x7f87b013e1a0 2024-06-11T19:27:13.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 10 monclient(hunting): start opening mon connection 2024-06-11T19:27:13.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 1 -- --> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] -- mon_getmap magic: 0 -- 0x7f87b00a6da0 con 0x7f87b00a4920 2024-06-11T19:27:13.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 10 monclient(hunting): _renew_subs 2024-06-11T19:27:13.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87b4cd1640 10 monclient(hunting): authenticate will time out at 1843.500847s 2024-06-11T19:27:13.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87aed9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b013e1a0 0x7f87b009ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87aed9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b013e1a0 0x7f87b009ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:37130/0 (socket says 172.21.15.98:37130) 2024-06-11T19:27:13.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87aed9d640 1 -- 172.21.15.98:0/3431979687 learned_addr learned my addr 172.21.15.98:0/3431979687 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-06-11T19:27:13.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87aed9d640 10 monclient(hunting): get_auth_request con 0x7f87b013e1a0 auth_method 0 2024-06-11T19:27:13.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-06-11T19:27:13.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87aed9d640 10 monclient(hunting): _init_auth method 2 2024-06-11T19:27:13.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.693+0000 7f87aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-06-11T19:27:13.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-06-11T19:27:13.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-06-11T19:27:13.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-06-11T19:27:13.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 10 monclient(hunting): handle_auth_done global_id 124575 payload 824 2024-06-11T19:27:13.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87ae59c640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f87b00a2530 0x7f87b0135c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 1 -- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f87b00a4920 msgr2=0x7f87b0136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:27:13.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87add9b640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f87b00a4920 0x7f87b0136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f87b00a4920 0x7f87b0136180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 1 -- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f87b00a2530 msgr2=0x7f87b0135c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:27:13.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f87b00a2530 0x7f87b0135c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 10 monclient: _finish_hunting 0 2024-06-11T19:27:13.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 1 monclient: found mon.noname-d 2024-06-11T19:27:13.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-11T19:27:13.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-06-11T19:27:13.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 1 -- 172.21.15.98:0/3431979687 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b013dcf0 con 0x7f87b013e1a0 2024-06-11T19:27:13.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87ae59c640 10 monclient: get_auth_request con 0x7f87b00a2530 auth_method 0 2024-06-11T19:27:13.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87ae59c640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f87b00a2530 0x7f87b0135c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-11T19:27:13.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87aed9d640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b013e1a0 0x7f87b009ecb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f87a800efc0 tx=0x7f87a800c490 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:27:13.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 10 client.?.objecter ms_handle_connect 0x7f87b013e1a0 2024-06-11T19:27:13.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 10 client.?.objecter resend_mon_ops 2024-06-11T19:27:13.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 1 -- 172.21.15.98:0/3431979687 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f87a8018020 con 0x7f87b013e1a0 2024-06-11T19:27:13.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:27:13.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-06-11T19:27:13.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 10 monclient: dump: 2024-06-11T19:27:13.720 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:27:13.720 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:27:13.720 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.720 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.720 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:27:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:27:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:27:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:27:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:27:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:27:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:27:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:27:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:27:13.722 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:27:13.722 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:27:13.722 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:13.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 10 monclient: _finish_auth 0 2024-06-11T19:27:13.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 10 monclient: _check_auth_tickets 2024-06-11T19:27:13.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-11T19:27:13.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 1 -- 172.21.15.98:0/3431979687 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87a800cd30 con 0x7f87b013e1a0 2024-06-11T19:27:13.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87b4cd1640 5 monclient: authenticate success, global_id 124575 2024-06-11T19:27:13.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 10 monclient: handle_config config(0 keys) 2024-06-11T19:27:13.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 1 -- 172.21.15.98:0/3431979687 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f87a8014d60 con 0x7f87b013e1a0 2024-06-11T19:27:13.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87b4cd1640 10 monclient: _renew_subs 2024-06-11T19:27:13.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87b4cd1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:27:13.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87b4cd1640 1 -- 172.21.15.98:0/3431979687 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87b0122310 con 0x7f87b013e1a0 2024-06-11T19:27:13.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 10 monclient: handle_monmap mon_map magic: 0 2024-06-11T19:27:13.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-06-11T19:27:13.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87acd99640 4 set_mon_vals no callback set 2024-06-11T19:27:13.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f879effd640 10 monclient: dump: 2024-06-11T19:27:13.724 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-06-11T19:27:13.724 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 9daa5251-9ae7-4a07-8f69-80112d2163e1 2024-06-11T19:27:13.724 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.724 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-06-11T19:08:43.892957+0000 2024-06-11T19:27:13.724 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-06-11T19:27:13.724 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-06-11T19:27:13.724 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-06-11T19:27:13.724 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.f 2024-06-11T19:27:13.724 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-06-11T19:27:13.725 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.g 2024-06-11T19:27:13.725 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-06-11T19:27:13.725 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] mon.h 2024-06-11T19:27:13.725 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-06-11T19:27:13.725 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] mon.i 2024-06-11T19:27:13.725 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-06-11T19:27:13.725 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:13.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87b4cd1640 10 client.124575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-11T19:27:13.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87b4cd1640 10 monclient: _renew_subs 2024-06-11T19:27:13.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87b4cd1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:27:13.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87b4cd1640 1 -- 172.21.15.98:0/3431979687 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f87b0122030 con 0x7f87b013e1a0 2024-06-11T19:27:13.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.694+0000 7f87b4cd1640 1 librados: init done 2024-06-11T19:27:13.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f879effd640 1 -- 172.21.15.98:0/3431979687 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 49411+0+0 (secure 0 0 0) 0x7f87a8005d10 con 0x7f87b013e1a0 2024-06-11T19:27:13.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f87b4cd1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-11T19:27:13.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f87b4cd1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:27:13.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f87b4cd1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:27:13.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f87b4cd1640 1 -- 172.21.15.98:0/3431979687 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87b00a7360 con 0x7f87b013e1a0 2024-06-11T19:27:13.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f879effd640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f879003fdb0 0x7f8790042270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-11T19:27:13.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f879effd640 1 -- 172.21.15.98:0/3431979687 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(685..685 src has 517..685) ==== 5557+0+0 (secure 0 0 0) 0x7f87a8052310 con 0x7f87b013e1a0 2024-06-11T19:27:13.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f879effd640 10 client.124575.objecter ms_dispatch 0x7f87b0000b90 osd_map(685..685 src has 517..685) 2024-06-11T19:27:13.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f879effd640 3 client.124575.objecter handle_osd_map got epochs [685,685] > 0 2024-06-11T19:27:13.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f879effd640 3 client.124575.objecter handle_osd_map decoding full epoch 685 2024-06-11T19:27:13.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f879effd640 20 client.124575.objecter dump_active .. 0 homeless 2024-06-11T19:27:13.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f87ae59c640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f879003fdb0 0x7f8790042270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-11T19:27:13.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.695+0000 7f87ae59c640 10 monclient: get_auth_request con 0x7f879003fdb0 auth_method 0 2024-06-11T19:27:13.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.696+0000 7f87ae59c640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f879003fdb0 0x7f8790042270 secure :-1 s=READY pgs=4349 cs=0 l=1 rev1=1 crypto rx=0x7f87a4000f60 tx=0x7f87a4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-11T19:27:13.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.696+0000 7f879effd640 10 client.124575.objecter ms_handle_connect 0x7f879003fdb0 2024-06-11T19:27:13.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.700+0000 7f879effd640 1 -- 172.21.15.98:0/3431979687 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f87b00a7360 con 0x7f87b013e1a0 2024-06-11T19:27:13.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.700+0000 7f879effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-11T19:27:13.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.700+0000 7f879effd640 10 monclient: _finish_command 1 = system:0 2024-06-11T19:27:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.879+0000 7f87b4cd1640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-06-11T19:27:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.879+0000 7f87b4cd1640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-06-11T19:27:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.879+0000 7f87b4cd1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-06-11T19:27:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.879+0000 7f87b4cd1640 1 -- 172.21.15.98:0/3431979687 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f87b00a4d00 con 0x7f87b013e1a0 2024-06-11T19:27:13.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.886+0000 7f879effd640 1 -- 172.21.15.98:0/3431979687 <== mon.6 v2:172.21.15.98:3303/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f87a801a840 con 0x7f87b013e1a0 2024-06-11T19:27:13.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.886+0000 7f879effd640 10 monclient: handle_config config(1 keys) 2024-06-11T19:27:13.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.886+0000 7f87acd99640 4 set_mon_vals no callback set 2024-06-11T19:27:13.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.886+0000 7f87acd99640 10 set_mon_vals mon_health_to_clog = false 2024-06-11T19:27:13.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.886+0000 7f879effd640 1 -- 172.21.15.98:0/3431979687 <== mon.6 v2:172.21.15.98:3303/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v4)=0 v4) ==== 123+0+0 (secure 0 0 0) 0x7f87a8022070 con 0x7f87b013e1a0 2024-06-11T19:27:13.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.886+0000 7f879effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-06-11T19:27:13.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.886+0000 7f879effd640 10 monclient: _finish_command 2 = system:0 2024-06-11T19:27:13.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 10 librados: watch_flush enter 2024-06-11T19:27:13.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 10 librados: watch_flush exit 2024-06-11T19:27:13.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 20 client.124575.objecter shutdown clearing up homeless session... 2024-06-11T19:27:13.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 10 client.124575.objecter successfully canceled tick 2024-06-11T19:27:13.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 1 -- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f879003fdb0 msgr2=0x7f8790042270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:27:13.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f879003fdb0 0x7f8790042270 secure :-1 s=READY pgs=4349 cs=0 l=1 rev1=1 crypto rx=0x7f87a4000f60 tx=0x7f87a4009040 comp rx=0 tx=0).stop 2024-06-11T19:27:13.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 10 monclient: shutdown 2024-06-11T19:27:13.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-11T19:27:13.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 1 -- 172.21.15.98:0/3431979687 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b013e1a0 msgr2=0x7f87b009ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-11T19:27:13.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b013e1a0 0x7f87b009ecb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f87a800efc0 tx=0x7f87a800c490 comp rx=0 tx=0).stop 2024-06-11T19:27:13.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 1 -- 172.21.15.98:0/3431979687 shutdown_connections 2024-06-11T19:27:13.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:6824/2398011028,v1:172.21.15.117:6825/2398011028] conn(0x7f879003fdb0 0x7f8790042270 unknown :-1 s=CLOSED pgs=4349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f87b013e1a0 0x7f87b009ecb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:3303/0,v1:172.21.15.117:6792/0] conn(0x7f87b00a4920 0x7f87b0136180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 1 --2- 172.21.15.98:0/3431979687 >> [v2:172.21.15.117:3302/0,v1:172.21.15.117:6791/0] conn(0x7f87b00a2530 0x7f87b0135c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-11T19:27:13.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.888+0000 7f87b4cd1640 1 -- 172.21.15.98:0/3431979687 >> 172.21.15.98:0/3431979687 conn(0x7f87b00ab850 msgr2=0x7f87b00a91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-11T19:27:13.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.889+0000 7f87b4cd1640 1 -- 172.21.15.98:0/3431979687 shutdown_connections 2024-06-11T19:27:13.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.889+0000 7f87b4cd1640 1 -- 172.21.15.98:0/3431979687 wait complete. 2024-06-11T19:27:13.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-06-11T19:27:13.889+0000 7f87b4cd1640 1 librados: shutdown 2024-06-11T19:27:13.898 INFO:teuthology.misc:Shutting down mds daemons... 2024-06-11T19:27:13.899 INFO:teuthology.misc:Shutting down osd daemons... 2024-06-11T19:27:13.899 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-06-11T19:27:13.899 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:13.950 INFO:tasks.ceph.osd.0:Stopped 2024-06-11T19:27:13.950 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-06-11T19:27:13.950 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.005 INFO:tasks.ceph.osd.1:Stopped 2024-06-11T19:27:14.005 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-06-11T19:27:14.005 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.044 INFO:tasks.ceph.osd.2:Stopped 2024-06-11T19:27:14.044 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-06-11T19:27:14.044 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.086 INFO:tasks.ceph.osd.3:Stopped 2024-06-11T19:27:14.086 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-06-11T19:27:14.086 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.157 INFO:tasks.ceph.osd.4:Stopped 2024-06-11T19:27:14.158 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-06-11T19:27:14.158 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.242 INFO:tasks.ceph.osd.5:Stopped 2024-06-11T19:27:14.243 INFO:teuthology.misc:Shutting down mgr daemons... 2024-06-11T19:27:14.243 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-06-11T19:27:14.243 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.271 INFO:tasks.ceph.mgr.x:Stopped 2024-06-11T19:27:14.271 INFO:teuthology.misc:Shutting down mon daemons... 2024-06-11T19:27:14.271 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-06-11T19:27:14.271 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.288 INFO:tasks.ceph.mon.a:Stopped 2024-06-11T19:27:14.288 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-06-11T19:27:14.288 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.346 INFO:tasks.ceph.mon.b:Stopped 2024-06-11T19:27:14.346 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-06-11T19:27:14.346 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.398 INFO:tasks.ceph.mon.c:Stopped 2024-06-11T19:27:14.399 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-06-11T19:27:14.399 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.452 INFO:tasks.ceph.mon.d:Stopped 2024-06-11T19:27:14.453 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-06-11T19:27:14.453 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.507 INFO:tasks.ceph.mon.e:Stopped 2024-06-11T19:27:14.507 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-06-11T19:27:14.508 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.522 INFO:tasks.ceph.mon.f:Stopped 2024-06-11T19:27:14.522 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-06-11T19:27:14.523 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.578 INFO:tasks.ceph.mon.g:Stopped 2024-06-11T19:27:14.579 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-06-11T19:27:14.579 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.633 INFO:tasks.ceph.mon.h:Stopped 2024-06-11T19:27:14.633 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-06-11T19:27:14.633 INFO:teuthology.orchestra.run:waiting for 300 2024-06-11T19:27:14.687 INFO:tasks.ceph.mon.i:Stopped 2024-06-11T19:27:14.688 INFO:tasks.ceph:Checking cluster log for badness... 2024-06-11T19:27:14.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-06-11T19:27:14.741 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi098.front.sepia.ceph.com 2024-06-11T19:27:14.741 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-06-11T19:27:14.952 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi098.front.sepia.ceph.com 2024-06-11T19:27:14.952 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-06-11T19:27:15.073 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi098.front.sepia.ceph.com 2024-06-11T19:27:15.073 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-06-11T19:27:15.141 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi117.front.sepia.ceph.com 2024-06-11T19:27:15.141 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-06-11T19:27:15.294 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi117.front.sepia.ceph.com 2024-06-11T19:27:15.295 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-06-11T19:27:15.408 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi117.front.sepia.ceph.com 2024-06-11T19:27:15.408 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-06-11T19:27:15.464 INFO:tasks.ceph:Archiving mon data... 2024-06-11T19:27:15.465 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/data/mon.a.tgz 2024-06-11T19:27:15.466 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-06-11T19:27:15.481 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.N15MsnZLCy 2024-06-11T19:27:15.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.N15MsnZLCy 2024-06-11T19:27:15.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.N15MsnZLCy 2024-06-11T19:27:15.841 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.N15MsnZLCy is 1MB 2024-06-11T19:27:15.921 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.N15MsnZLCy 2024-06-11T19:27:15.937 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/data/mon.b.tgz 2024-06-11T19:27:15.937 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-06-11T19:27:15.991 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.fWGCH6pKXv 2024-06-11T19:27:15.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.fWGCH6pKXv 2024-06-11T19:27:16.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.fWGCH6pKXv 2024-06-11T19:27:16.338 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.fWGCH6pKXv is 1MB 2024-06-11T19:27:16.418 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.fWGCH6pKXv 2024-06-11T19:27:16.433 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/data/mon.c.tgz 2024-06-11T19:27:16.433 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-06-11T19:27:16.487 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.Lzxzwfb9xo 2024-06-11T19:27:16.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Lzxzwfb9xo 2024-06-11T19:27:16.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.Lzxzwfb9xo 2024-06-11T19:27:16.830 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.Lzxzwfb9xo is 1MB 2024-06-11T19:27:16.911 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.Lzxzwfb9xo 2024-06-11T19:27:16.926 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-d to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/data/mon.d.tgz 2024-06-11T19:27:16.926 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-06-11T19:27:16.981 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.jSQ6jTfWoe 2024-06-11T19:27:16.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-d -- . > /tmp/tmp.jSQ6jTfWoe 2024-06-11T19:27:17.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.jSQ6jTfWoe 2024-06-11T19:27:17.326 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.jSQ6jTfWoe is 1MB 2024-06-11T19:27:17.432 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.jSQ6jTfWoe 2024-06-11T19:27:17.447 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-e to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/data/mon.e.tgz 2024-06-11T19:27:17.447 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-06-11T19:27:17.502 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.0i5CkGFOyY 2024-06-11T19:27:17.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-e -- . > /tmp/tmp.0i5CkGFOyY 2024-06-11T19:27:17.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.0i5CkGFOyY 2024-06-11T19:27:17.851 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.0i5CkGFOyY is 1MB 2024-06-11T19:27:17.965 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.0i5CkGFOyY 2024-06-11T19:27:17.981 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-f to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/data/mon.f.tgz 2024-06-11T19:27:17.981 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-06-11T19:27:18.000 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.v4A3wqOZSo 2024-06-11T19:27:18.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-f -- . > /tmp/tmp.v4A3wqOZSo 2024-06-11T19:27:18.343 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.v4A3wqOZSo 2024-06-11T19:27:18.427 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.v4A3wqOZSo is 1MB 2024-06-11T19:27:18.538 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.v4A3wqOZSo 2024-06-11T19:27:18.553 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-g to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/data/mon.g.tgz 2024-06-11T19:27:18.554 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-06-11T19:27:18.608 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.14OnLFXNpS 2024-06-11T19:27:18.609 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-g -- . > /tmp/tmp.14OnLFXNpS 2024-06-11T19:27:18.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.14OnLFXNpS 2024-06-11T19:27:18.958 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.14OnLFXNpS is 1MB 2024-06-11T19:27:19.065 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.14OnLFXNpS 2024-06-11T19:27:19.081 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-h to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/data/mon.h.tgz 2024-06-11T19:27:19.081 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-06-11T19:27:19.136 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.w5SmW8B3vY 2024-06-11T19:27:19.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-h -- . > /tmp/tmp.w5SmW8B3vY 2024-06-11T19:27:19.405 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.w5SmW8B3vY 2024-06-11T19:27:19.488 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.w5SmW8B3vY is 1MB 2024-06-11T19:27:19.596 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.w5SmW8B3vY 2024-06-11T19:27:19.611 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-i to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/data/mon.i.tgz 2024-06-11T19:27:19.611 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-06-11T19:27:19.665 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.yqTmu40giL 2024-06-11T19:27:19.666 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-i -- . > /tmp/tmp.yqTmu40giL 2024-06-11T19:27:19.933 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.yqTmu40giL 2024-06-11T19:27:20.016 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.yqTmu40giL is 1MB 2024-06-11T19:27:20.095 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.yqTmu40giL 2024-06-11T19:27:20.110 INFO:tasks.ceph:Cleaning ceph cluster... 2024-06-11T19:27:20.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-06-11T19:27:20.113 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-06-11T19:27:20.197 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-06-11T19:27:20.229 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-06-11T19:27:20.229 INFO:tasks.ceph:Archiving crash dumps... 2024-06-11T19:27:20.231 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/remote/smithi098/crash 2024-06-11T19:27:20.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-11T19:27:20.266 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/crash to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/remote/smithi117/crash 2024-06-11T19:27:20.267 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-11T19:27:20.292 INFO:tasks.ceph:Compressing logs... 2024-06-11T19:27:20.292 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-11T19:27:20.310 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-11T19:27:20.335 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44769.log 2024-06-11T19:27:20.348 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40216.log 2024-06-11T19:27:20.349 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48928.log 2024-06-11T19:27:20.349 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44769.log: /var/log/ceph/ceph-client.admin.40216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49853.log 2024-06-11T19:27:20.349 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48928.log: gzip 90.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.44769.log.gz --verbose 2024-06-11T19:27:20.349 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/ceph-client.admin.38346.log 2024-06-11T19:27:20.349 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.40216.log.gz 2024-06-11T19:27:20.349 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40258.log 2024-06-11T19:27:20.349 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.48928.log.gz 2024-06-11T19:27:20.349 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50063.log 2024-06-11T19:27:20.350 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38346.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49853.log.gz 2024-06-11T19:27:20.350 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38481.log 2024-06-11T19:27:20.350 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38346.log.gz 2024-06-11T19:27:20.350 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38304.log 2024-06-11T19:27:20.350 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.40258.log.gz 2024-06-11T19:27:20.350 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41686.log 2024-06-11T19:27:20.350 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.50063.log.gz 2024-06-11T19:27:20.350 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47352.log 2024-06-11T19:27:20.350 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38481.log.gz 2024-06-11T19:27:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44601.log 2024-06-11T19:27:20.351 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38304.log.gz 2024-06-11T19:27:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47688.log 2024-06-11T19:27:20.351 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.41686.log.gz 2024-06-11T19:27:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43485.log 2024-06-11T19:27:20.351 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47352.log.gz 2024-06-11T19:27:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44937.log 2024-06-11T19:27:20.352 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44601.log.gz 2024-06-11T19:27:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44157.log 2024-06-11T19:27:20.352 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.47688.log.gz 2024-06-11T19:27:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42855.log 2024-06-11T19:27:20.352 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.43485.log.gz 2024-06-11T19:27:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43569.log 2024-06-11T19:27:20.352 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.44937.log.gz 2024-06-11T19:27:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45846.log 2024-06-11T19:27:20.352 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.44157.log.gz 2024-06-11T19:27:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49769.log 2024-06-11T19:27:20.353 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42855.log.gz 2024-06-11T19:27:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51060.log 2024-06-11T19:27:20.353 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.43569.log.gz 2024-06-11T19:27:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49097.log 2024-06-11T19:27:20.353 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.45846.log.gz 2024-06-11T19:27:20.354 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43989.log 2024-06-11T19:27:20.354 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49769.log.gz 2024-06-11T19:27:20.354 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46056.log 2024-06-11T19:27:20.354 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51060.log.gz 2024-06-11T19:27:20.354 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37810.log 2024-06-11T19:27:20.354 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49097.log.gz 2024-06-11T19:27:20.355 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42568.log 2024-06-11T19:27:20.355 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43989.log.gz 2024-06-11T19:27:20.355 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45315.log 2024-06-11T19:27:20.355 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.46056.log.gz 2024-06-11T19:27:20.355 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46688.log 2024-06-11T19:27:20.355 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37810.log.gz 2024-06-11T19:27:20.355 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49223.log 2024-06-11T19:27:20.355 INFO:teuthology.orchestra.run.smithi098.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.42568.log.gz 2024-06-11T19:27:20.356 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42981.log 2024-06-11T19:27:20.356 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1%/var/log/ceph/ceph-client.admin.49223.log: -- replaced with /var/log/ceph/ceph-client.admin.45315.log.gz 2024-06-11T19:27:20.356 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39114.log 2024-06-11T19:27:20.356 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46688.log.gz 2024-06-11T19:27:20.356 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41190.log 2024-06-11T19:27:20.356 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49223.log.gz 2024-06-11T19:27:20.356 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39114.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-06-11T19:27:20.356 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42981.log.gz 2024-06-11T19:27:20.356 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49475.log 2024-06-11T19:27:20.357 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39114.log.gz 2024-06-11T19:27:20.357 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38389.log 2024-06-11T19:27:20.357 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5%/var/log/ceph/ceph-client.admin.49475.log: -- replaced with /var/log/ceph/ceph-client.admin.41190.log.gz 2024-06-11T19:27:20.357 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.d.log 2024-06-11T19:27:20.357 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46478.log 2024-06-11T19:27:20.357 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49475.log.gz 2024-06-11T19:27:20.357 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45273.log 2024-06-11T19:27:20.357 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38389.log.gz 2024-06-11T19:27:20.357 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40038.log 2024-06-11T19:27:20.358 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51144.log 2024-06-11T19:27:20.358 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.46478.log.gz 2024-06-11T19:27:20.358 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40038.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45273.log.gz 2024-06-11T19:27:20.358 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45653.log 2024-06-11T19:27:20.393 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37672.log 2024-06-11T19:27:20.394 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37764.log 2024-06-11T19:27:20.394 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38075.log 2024-06-11T19:27:20.394 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38555.log 2024-06-11T19:27:20.394 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36584.log 2024-06-11T19:27:20.394 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38472.log 2024-06-11T19:27:20.394 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38555.log: /var/log/ceph/ceph-client.admin.36584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38126.log 2024-06-11T19:27:20.395 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38472.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.37672.log.gz 2024-06-11T19:27:20.395 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36569.log 2024-06-11T19:27:20.395 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38170.log 2024-06-11T19:27:20.395 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38554.log 2024-06-11T19:27:20.395 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38170.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.38075.log.gz 2024-06-11T19:27:20.395 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38077.log 2024-06-11T19:27:20.395 INFO:teuthology.orchestra.run.smithi117.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.38555.log.gz 2024-06-11T19:27:20.395 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38554.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38472.log.gz 2024-06-11T19:27:20.395 INFO:teuthology.orchestra.run.smithi117.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.38126.log.gz 2024-06-11T19:27:20.396 INFO:teuthology.orchestra.run.smithi117.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.36584.log.gz 2024-06-11T19:27:20.396 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37281.log 2024-06-11T19:27:20.396 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37244.log 2024-06-11T19:27:20.396 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38077.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.38170.log.gz 2024-06-11T19:27:20.396 INFO:teuthology.orchestra.run.smithi117.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.36569.log.gz 2024-06-11T19:27:20.396 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38469.log 2024-06-11T19:27:20.396 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33553.log 2024-06-11T19:27:20.396 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37244.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.38554.log.gz 2024-06-11T19:27:20.397 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38469.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-06-11T19:27:20.397 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37244.log.gz 2024-06-11T19:27:20.397 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.33553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36785.log 2024-06-11T19:27:20.397 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33553.log.gz 2024-06-11T19:27:20.397 INFO:teuthology.orchestra.run.smithi117.stderr: /var/log/ceph/ceph-client.admin.36613.log 2024-06-11T19:27:20.397 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37189.log 2024-06-11T19:27:20.397 INFO:teuthology.orchestra.run.smithi117.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.37281.log.gz 2024-06-11T19:27:20.397 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36613.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.38077.log.gz 2024-06-11T19:27:20.397 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37780.log 2024-06-11T19:27:20.398 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.g.log 2024-06-11T19:27:20.398 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37189.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.38469.log.gz 2024-06-11T19:27:20.398 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38553.log 2024-06-11T19:27:20.398 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37780.log: /var/log/ceph/ceph-mon.g.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37620.log 2024-06-11T19:27:20.398 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38553.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.37189.log.gz 2024-06-11T19:27:20.398 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37722.log 2024-06-11T19:27:20.398 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37620.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.38553.log.gz 2024-06-11T19:27:20.398 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38169.log 2024-06-11T19:27:20.399 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37722.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.37780.log.gz 2024-06-11T19:27:20.399 INFO:teuthology.orchestra.run.smithi117.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.36785.log.gz 2024-06-11T19:27:20.399 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-06-11T19:27:20.399 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37084.log 2024-06-11T19:27:20.399 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.38169.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38169.log.gz 2024-06-11T19:27:20.399 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36868.log 2024-06-11T19:27:20.399 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37084.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.37620.log.gz 2024-06-11T19:27:20.399 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37864.log 2024-06-11T19:27:20.400 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36868.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.37084.log.gz 2024-06-11T19:27:20.400 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36768.log 2024-06-11T19:27:20.400 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37864.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.37764.log.gz 2024-06-11T19:27:20.400 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37069.log 2024-06-11T19:27:20.400 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36768.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.36868.log.gz 2024-06-11T19:27:20.400 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-06-11T19:27:20.400 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37069.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.36768.log.gz 2024-06-11T19:27:20.400 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-06-11T19:27:20.408 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.5.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.37069.log.gz 2024-06-11T19:27:20.408 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36891.log 2024-06-11T19:27:20.409 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.3.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.37722.log.gz 2024-06-11T19:27:20.409 INFO:teuthology.orchestra.run.smithi117.stderr: 96.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-06-11T19:27:20.409 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38471.log 2024-06-11T19:27:20.409 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36854.log 2024-06-11T19:27:20.409 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38471.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.36891.log.gz 2024-06-11T19:27:20.409 INFO:teuthology.orchestra.run.smithi117.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.38471.log.gz 2024-06-11T19:27:20.409 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37793.log 2024-06-11T19:27:20.410 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36854.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-06-11T19:27:20.424 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37793.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.37793.log.gz 2024-06-11T19:27:20.432 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36551.log 2024-06-11T19:27:20.432 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.4.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.37864.log.gz 2024-06-11T19:27:20.432 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38167.log 2024-06-11T19:27:20.432 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36551.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.36551.log.gz 2024-06-11T19:27:20.432 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.i.log 2024-06-11T19:27:20.436 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38167.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38167.log.gz 2024-06-11T19:27:20.436 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38127.log 2024-06-11T19:27:20.449 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mon.i.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37296.log 2024-06-11T19:27:20.449 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38127.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.38127.log.gz 2024-06-11T19:27:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.51144.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45930.log 2024-06-11T19:27:20.452 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40038.log.gz 2024-06-11T19:27:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45653.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-06-11T19:27:20.452 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6%/var/log/ceph/ceph-client.admin.45930.log: -- replaced with /var/log/ceph/ceph-client.admin.51144.log.gz 2024-06-11T19:27:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43317.log 2024-06-11T19:27:20.452 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45653.log.gz 2024-06-11T19:27:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38065.log 2024-06-11T19:27:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.43317.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45930.log.gz 2024-06-11T19:27:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38988.log 2024-06-11T19:27:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38065.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28463.log 2024-06-11T19:27:20.453 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9%/var/log/ceph/ceph-client.admin.38988.log: -- replaced with /var/log/ceph/ceph-client.admin.43317.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49139.log 2024-06-11T19:27:20.453 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:20.453 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38065.log.gz 2024-06-11T19:27:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.tmp-client.admin.28463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50514.log 2024-06-11T19:27:20.454 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28463.log.gz 2024-06-11T19:27:20.454 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38988.log.gz 2024-06-11T19:27:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.49139.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38525.log 2024-06-11T19:27:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44685.log 2024-06-11T19:27:20.454 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49139.log.gz 2024-06-11T19:27:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38525.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.50514.log.gz 2024-06-11T19:27:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44199.log 2024-06-11T19:27:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46182.log 2024-06-11T19:27:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44685.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38525.log.gz 2024-06-11T19:27:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44199.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.e.log 2024-06-11T19:27:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46182.log: gzip 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44685.log.gz 2024-06-11T19:27:20.455 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose 90.0% -- replaced with /var/log/ceph/ceph-client.admin.44199.log.gz 2024-06-11T19:27:20.455 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/ceph-client.admin.48234.log 2024-06-11T19:27:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44895.log 2024-06-11T19:27:20.455 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46182.log.gz 2024-06-11T19:27:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45888.log 2024-06-11T19:27:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45189.log 2024-06-11T19:27:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44895.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.48234.log.gz/var/log/ceph/ceph-client.admin.45888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42400.log 2024-06-11T19:27:20.456 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42022.log 2024-06-11T19:27:20.456 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.45888.log.gz 2024-06-11T19:27:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42400.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44895.log.gz 2024-06-11T19:27:20.456 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-06-11T19:27:20.456 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45189.log.gz 2024-06-11T19:27:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.42022.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43527.log 2024-06-11T19:27:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:gzip 91.7% -- replaced with /var/log/ceph/ceph-client.admin.42400.log.gz 2024-06-11T19:27:20.457 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37814.log 2024-06-11T19:27:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43527.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42022.log.gz 2024-06-11T19:27:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39366.log 2024-06-11T19:27:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51186.log 2024-06-11T19:27:20.457 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.43527.log.gz 2024-06-11T19:27:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39366.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37814.log.gz 2024-06-11T19:27:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46520.log 2024-06-11T19:27:20.458 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39366.log.gz 2024-06-11T19:27:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47478.log 2024-06-11T19:27:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41232.log 2024-06-11T19:27:20.458 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51186.log.gz 2024-06-11T19:27:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50556.log 2024-06-11T19:27:20.458 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.46520.log.gz 2024-06-11T19:27:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41232.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.47478.log.gz 2024-06-11T19:27:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41148.log 2024-06-11T19:27:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50556.log: 89.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.41232.log.gz 2024-06-11T19:27:20.459 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42610.log 2024-06-11T19:27:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47436.log 2024-06-11T19:27:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42610.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.50556.log.gz 2024-06-11T19:27:20.459 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41148.log.gz 2024-06-11T19:27:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33108.log 2024-06-11T19:27:20.459 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42610.log.gz 2024-06-11T19:27:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47562.log 2024-06-11T19:27:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46350.log 2024-06-11T19:27:20.460 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.47436.log.gz 2024-06-11T19:27:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33453.log 2024-06-11T19:27:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46350.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47562.log.gz 2024-06-11T19:27:20.460 INFO:teuthology.orchestra.run.smithi098.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.33108.log.gz 2024-06-11T19:27:20.460 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.46350.log.gz 2024-06-11T19:27:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44267.log 2024-06-11T19:27:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40430.log 2024-06-11T19:27:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38067.log 2024-06-11T19:27:20.461 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33453.log.gz 2024-06-11T19:27:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50976.log 2024-06-11T19:27:20.461 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44267.log.gz 2024-06-11T19:27:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42232.log 2024-06-11T19:27:20.461 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40430.log.gz 2024-06-11T19:27:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50976.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38067.log.gz 2024-06-11T19:27:20.462 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50976.log.gz 2024-06-11T19:27:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46856.log 2024-06-11T19:27:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46772.log 2024-06-11T19:27:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49181.log 2024-06-11T19:27:20.462 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42232.log.gz 2024-06-11T19:27:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37809.log 2024-06-11T19:27:20.462 INFO:teuthology.orchestra.run.smithi098.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.46856.log.gz 2024-06-11T19:27:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49181.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46772.log.gz 2024-06-11T19:27:20.463 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.49181.log.gz 2024-06-11T19:27:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46562.log 2024-06-11T19:27:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49433.log 2024-06-11T19:27:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50306.log 2024-06-11T19:27:20.463 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37809.log.gz 2024-06-11T19:27:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49433.log: gzip -5 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46562.log.gz 2024-06-11T19:27:20.463 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46646.log 2024-06-11T19:27:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49895.log 2024-06-11T19:27:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46646.log: 88.8% 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49433.log.gz 2024-06-11T19:27:20.464 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50306.log.gz 2024-06-11T19:27:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48798.log 2024-06-11T19:27:20.464 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.46646.log.gz/var/log/ceph/ceph-client.admin.49895.log: 2024-06-11T19:27:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51649.log 2024-06-11T19:27:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48798.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49895.log.gz 2024-06-11T19:27:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49307.log 2024-06-11T19:27:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42274.log 2024-06-11T19:27:20.465 INFO:teuthology.orchestra.run.smithi098.stderr: 91.2%/var/log/ceph/ceph-client.admin.49307.log: -- replaced with /var/log/ceph/ceph-client.admin.48798.log.gz 2024-06-11T19:27:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37816.log 2024-06-11T19:27:20.465 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49307.log.gz 2024-06-11T19:27:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42274.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51649.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38736.log 2024-06-11T19:27:20.465 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37816.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42274.log.gz 2024-06-11T19:27:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45441.log 2024-06-11T19:27:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46436.log 2024-06-11T19:27:20.465 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37816.log.gz 2024-06-11T19:27:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38736.log: /var/log/ceph/ceph-client.admin.45441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50808.log 2024-06-11T19:27:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46436.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38736.log.gz 2024-06-11T19:27:20.466 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45441.log.gz 2024-06-11T19:27:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38568.log 2024-06-11T19:27:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47188.log 2024-06-11T19:27:20.466 INFO:teuthology.orchestra.run.smithi098.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.46436.log.gz 2024-06-11T19:27:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50808.log: /var/log/ceph/ceph-client.admin.38568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38433.log 2024-06-11T19:27:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47188.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38568.log.gz 2024-06-11T19:27:20.467 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50808.log.gz 2024-06-11T19:27:20.467 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47188.log.gz 2024-06-11T19:27:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41770.log 2024-06-11T19:27:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.38433.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46224.log 2024-06-11T19:27:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39324.log 2024-06-11T19:27:20.467 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38433.log.gz 2024-06-11T19:27:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46224.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41770.log.gz 2024-06-11T19:27:20.467 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46224.log.gz 2024-06-11T19:27:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39198.log 2024-06-11T19:27:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42148.log 2024-06-11T19:27:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50598.log 2024-06-11T19:27:20.468 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39324.log.gz 2024-06-11T19:27:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39198.log.gz 2024-06-11T19:27:20.468 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39996.log 2024-06-11T19:27:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42148.log: /var/log/ceph/ceph-client.admin.50598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47310.log 2024-06-11T19:27:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39996.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.50598.log.gz 2024-06-11T19:27:20.468 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42148.log.gz 2024-06-11T19:27:20.469 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40384.log 2024-06-11T19:27:20.469 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39996.log.gz 2024-06-11T19:27:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42897.log 2024-06-11T19:27:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39156.log 2024-06-11T19:27:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40384.log: /var/log/ceph/ceph-client.admin.42897.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.47310.log.gz 2024-06-11T19:27:20.469 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40384.log.gz 2024-06-11T19:27:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41938.log 2024-06-11T19:27:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39156.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.42897.log.gz 2024-06-11T19:27:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40556.log 2024-06-11T19:27:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43947.log 2024-06-11T19:27:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41938.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.39156.log.gz/var/log/ceph/ceph-client.admin.40556.log: 2024-06-11T19:27:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40686.log 2024-06-11T19:27:20.470 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.40556.log.gz/var/log/ceph/ceph-client.admin.43947.log: 2024-06-11T19:27:20.470 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.41938.log.gz 2024-06-11T19:27:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42484.log 2024-06-11T19:27:20.470 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43947.log.gz 2024-06-11T19:27:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40080.log 2024-06-11T19:27:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40686.log: /var/log/ceph/ceph-client.admin.42484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38862.log 2024-06-11T19:27:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40080.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40686.log.gz 2024-06-11T19:27:20.471 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42484.log.gz 2024-06-11T19:27:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46308.log 2024-06-11T19:27:20.471 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40080.log.gz 2024-06-11T19:27:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40300.log 2024-06-11T19:27:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38862.log: /var/log/ceph/ceph-client.admin.46308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38068.log 2024-06-11T19:27:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40300.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38862.log.gz 2024-06-11T19:27:20.472 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46308.log.gz 2024-06-11T19:27:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45021.log 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40300.log.gz 2024-06-11T19:27:20.472 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38068.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-06-11T19:27:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45021.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38068.log.gz 2024-06-11T19:27:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38946.log 2024-06-11T19:27:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41560.log 2024-06-11T19:27:20.472 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6%/var/log/ceph/ceph-osd.1.log: -- replaced with /var/log/ceph/ceph-client.admin.45021.log.gz 2024-06-11T19:27:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38946.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.38946.log.gz 2024-06-11T19:27:20.473 INFO:teuthology.orchestra.run.smithi098.stderr: 96.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-06-11T19:27:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47898.log 2024-06-11T19:27:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48150.log 2024-06-11T19:27:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51018.log 2024-06-11T19:27:20.473 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.41560.log.gz 2024-06-11T19:27:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48150.log: gzip -5 --verbose -- 90.1% /var/log/ceph/ceph-client.admin.48024.log 2024-06-11T19:27:20.473 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47898.log.gz 2024-06-11T19:27:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51018.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48150.log.gz 2024-06-11T19:27:20.473 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51018.log.gz 2024-06-11T19:27:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40514.log 2024-06-11T19:27:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51438.log 2024-06-11T19:27:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50105.log 2024-06-11T19:27:20.474 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48024.log.gz 2024-06-11T19:27:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51438.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.40514.log.gzgzip 2024-06-11T19:27:20.474 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-06-11T19:27:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50105.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51438.log.gz 2024-06-11T19:27:20.474 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50105.log.gz 2024-06-11T19:27:20.475 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45063.log 2024-06-11T19:27:20.475 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43401.log 2024-06-11T19:27:20.475 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40598.log 2024-06-11T19:27:20.475 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43401.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45063.log.gz 2024-06-11T19:27:20.475 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43401.log.gz 2024-06-11T19:27:20.475 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45802.log 2024-06-11T19:27:20.475 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43191.log 2024-06-11T19:27:20.475 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45802.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40598.log.gz 2024-06-11T19:27:20.475 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42939.log 2024-06-11T19:27:20.476 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43191.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45802.log.gz 2024-06-11T19:27:20.476 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43275.log 2024-06-11T19:27:20.476 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6%/var/log/ceph/ceph-client.admin.42939.log: -- replaced with /var/log/ceph/ceph-client.admin.43191.log.gz 2024-06-11T19:27:20.476 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47520.log 2024-06-11T19:27:20.476 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43275.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.42939.log.gz 2024-06-11T19:27:20.476 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43275.log.gz 2024-06-11T19:27:20.476 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45231.log 2024-06-11T19:27:20.476 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50850.log 2024-06-11T19:27:20.476 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45231.log: /var/log/ceph/ceph-client.admin.47520.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45231.log.gz 2024-06-11T19:27:20.477 INFO:teuthology.orchestra.run.smithi098.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.47520.log.gz 2024-06-11T19:27:20.477 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40342.log 2024-06-11T19:27:20.477 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37506.log 2024-06-11T19:27:20.477 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40342.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.50850.log.gz 2024-06-11T19:27:20.477 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45105.log 2024-06-11T19:27:20.477 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37506.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.40342.log.gz 2024-06-11T19:27:20.477 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38694.log 2024-06-11T19:27:20.477 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37506.log.gz 2024-06-11T19:27:20.477 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43065.log 2024-06-11T19:27:20.478 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.45105.log.gz 2024-06-11T19:27:20.478 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38694.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-06-11T19:27:20.478 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43065.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38694.log.gz 2024-06-11T19:27:20.478 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50189.log 2024-06-11T19:27:20.478 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43065.log.gz 2024-06-11T19:27:20.478 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47982.log 2024-06-11T19:27:20.478 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.50189.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50189.log.gz 2024-06-11T19:27:20.479 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37674.log 2024-06-11T19:27:20.479 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47982.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47982.log.gz 2024-06-11T19:27:20.479 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40854.log 2024-06-11T19:27:20.479 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37674.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37674.log.gz 2024-06-11T19:27:20.479 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42691.log 2024-06-11T19:27:20.479 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40854.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40854.log.gz 2024-06-11T19:27:20.479 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41980.log 2024-06-11T19:27:20.479 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42691.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42691.log.gz 2024-06-11T19:27:20.479 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51481.log 2024-06-11T19:27:20.480 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41980.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41980.log.gz 2024-06-11T19:27:20.480 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45611.log 2024-06-11T19:27:20.480 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51481.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51481.log.gz 2024-06-11T19:27:20.480 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40728.log 2024-06-11T19:27:20.480 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45611.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.45611.log.gz 2024-06-11T19:27:20.480 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-06-11T19:27:20.480 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40728.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40728.log.gz 2024-06-11T19:27:20.480 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47024.log 2024-06-11T19:27:20.483 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38076.log 2024-06-11T19:27:20.483 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37296.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37296.log.gz 2024-06-11T19:27:20.483 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38128.log 2024-06-11T19:27:20.484 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38076.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.38076.log.gz 2024-06-11T19:27:20.484 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36879.log 2024-06-11T19:27:20.484 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38128.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.38128.log.gz 2024-06-11T19:27:20.484 INFO:teuthology.orchestra.run.smithi117.stderr: 90.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-06-11T19:27:20.494 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-06-11T19:27:20.494 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36879.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.36879.log.gz 2024-06-11T19:27:20.494 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.h.log 2024-06-11T19:27:20.499 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48318.log 2024-06-11T19:27:20.499 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47024.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.47024.log.gz 2024-06-11T19:27:20.499 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43149.log 2024-06-11T19:27:20.499 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48318.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48318.log.gz 2024-06-11T19:27:20.499 INFO:teuthology.orchestra.run.smithi098.stderr: 90.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-06-11T19:27:20.499 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44031.log 2024-06-11T19:27:20.499 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43149.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43149.log.gz 2024-06-11T19:27:20.500 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50387.log 2024-06-11T19:27:20.500 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44031.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44031.log.gz 2024-06-11T19:27:20.500 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49937.log 2024-06-11T19:27:20.500 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50387.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50387.log.gz 2024-06-11T19:27:20.500 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50471.log 2024-06-11T19:27:20.500 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49937.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49937.log.gz 2024-06-11T19:27:20.500 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46982.log 2024-06-11T19:27:20.500 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50471.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.50471.log.gz 2024-06-11T19:27:20.500 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43653.log 2024-06-11T19:27:20.501 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46982.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46982.log.gz 2024-06-11T19:27:20.501 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45357.log 2024-06-11T19:27:20.501 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43653.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43653.log.gz 2024-06-11T19:27:20.501 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50682.log 2024-06-11T19:27:20.501 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45357.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.45357.log.gz 2024-06-11T19:27:20.501 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44073.log 2024-06-11T19:27:20.501 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50682.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50682.log.gz 2024-06-11T19:27:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50231.log 2024-06-11T19:27:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44073.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44073.log.gz 2024-06-11T19:27:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44853.log 2024-06-11T19:27:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50231.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.50231.log.gz 2024-06-11T19:27:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49601.log 2024-06-11T19:27:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44853.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44853.log.gz 2024-06-11T19:27:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43779.log 2024-06-11T19:27:20.502 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49601.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.49601.log.gz 2024-06-11T19:27:20.503 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38220.log 2024-06-11T19:27:20.503 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43779.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43779.log.gz 2024-06-11T19:27:20.503 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39030.log 2024-06-11T19:27:20.503 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38220.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38220.log.gz 2024-06-11T19:27:20.503 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45972.log 2024-06-11T19:27:20.508 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39030.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39030.log.gz 2024-06-11T19:27:20.508 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37464.log 2024-06-11T19:27:20.508 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45972.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.45972.log.gz 2024-06-11T19:27:20.508 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42813.log 2024-06-11T19:27:20.509 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37464.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37464.log.gz 2024-06-11T19:27:20.509 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44349.log 2024-06-11T19:27:20.509 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42813.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.42813.log.gz 2024-06-11T19:27:20.509 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47394.log 2024-06-11T19:27:20.509 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44349.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44349.log.gz 2024-06-11T19:27:20.509 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40812.log 2024-06-11T19:27:20.509 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47394.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.47394.log.gz 2024-06-11T19:27:20.510 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37207.log 2024-06-11T19:27:20.513 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47646.log 2024-06-11T19:27:20.514 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40812.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40812.log.gz 2024-06-11T19:27:20.514 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44811.log 2024-06-11T19:27:20.514 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47646.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47646.log.gz 2024-06-11T19:27:20.514 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48276.log 2024-06-11T19:27:20.514 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44811.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.44811.log.gz 2024-06-11T19:27:20.514 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49012.log 2024-06-11T19:27:20.516 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48276.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48276.log.gz 2024-06-11T19:27:20.516 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50429.log 2024-06-11T19:27:20.517 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49012.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49012.log.gz 2024-06-11T19:27:20.517 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43863.log 2024-06-11T19:27:20.518 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50429.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50429.log.gz 2024-06-11T19:27:20.518 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38058.log 2024-06-11T19:27:20.519 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mon.h.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33382.log 2024-06-11T19:27:20.520 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43863.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43863.log.gz 2024-06-11T19:27:20.520 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44115.log 2024-06-11T19:27:20.522 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38058.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38058.log.gz 2024-06-11T19:27:20.522 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45567.log 2024-06-11T19:27:20.523 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44115.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44115.log.gz 2024-06-11T19:27:20.523 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40770.log 2024-06-11T19:27:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45567.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.45567.log.gz 2024-06-11T19:27:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45483.log 2024-06-11T19:27:20.525 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37207.log: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.37207.log.gz 2024-06-11T19:27:20.527 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40770.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40770.log.gz 2024-06-11T19:27:20.527 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47108.log 2024-06-11T19:27:20.528 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45483.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.45483.log.gz 2024-06-11T19:27:20.528 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43737.log 2024-06-11T19:27:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47108.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.47108.log.gz 2024-06-11T19:27:20.530 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51732.log 2024-06-11T19:27:20.531 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43737.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43737.log.gz 2024-06-11T19:27:20.531 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41106.log 2024-06-11T19:27:20.533 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51732.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51732.log.gz 2024-06-11T19:27:20.533 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39240.log 2024-06-11T19:27:20.534 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41106.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41106.log.gz 2024-06-11T19:27:20.534 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41728.log 2024-06-11T19:27:20.541 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39240.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39240.log.gz 2024-06-11T19:27:20.541 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49727.log 2024-06-11T19:27:20.541 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41728.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41728.log.gz 2024-06-11T19:27:20.541 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43695.log 2024-06-11T19:27:20.541 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49727.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49727.log.gz 2024-06-11T19:27:20.542 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44559.log 2024-06-11T19:27:20.542 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43695.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.43695.log.gz 2024-06-11T19:27:20.542 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43905.log 2024-06-11T19:27:20.545 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37044.log 2024-06-11T19:27:20.545 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.33382.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33382.log.gz 2024-06-11T19:27:20.546 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37560.log 2024-06-11T19:27:20.548 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44559.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.44559.log.gz 2024-06-11T19:27:20.548 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50021.log 2024-06-11T19:27:20.548 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43905.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.43905.log.gz 2024-06-11T19:27:20.548 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38262.log 2024-06-11T19:27:20.548 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50021.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.50021.log.gz 2024-06-11T19:27:20.549 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41518.log 2024-06-11T19:27:20.549 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38262.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38262.log.gz 2024-06-11T19:27:20.549 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51396.log 2024-06-11T19:27:20.551 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41518.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41518.log.gz 2024-06-11T19:27:20.551 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51228.log 2024-06-11T19:27:20.551 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51396.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51396.log.gz 2024-06-11T19:27:20.551 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41644.log 2024-06-11T19:27:20.553 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51228.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51228.log.gz 2024-06-11T19:27:20.553 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46098.log 2024-06-11T19:27:20.554 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36810.log 2024-06-11T19:27:20.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41644.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.41644.log.gz 2024-06-11T19:27:20.556 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50724.log 2024-06-11T19:27:20.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46098.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46098.log.gz 2024-06-11T19:27:20.556 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42106.log 2024-06-11T19:27:20.557 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37560.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.37560.log.gz 2024-06-11T19:27:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50724.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50724.log.gz 2024-06-11T19:27:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46266.log 2024-06-11T19:27:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42106.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42106.log.gz 2024-06-11T19:27:20.559 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49643.log 2024-06-11T19:27:20.560 INFO:teuthology.orchestra.run.smithi117.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.37044.log.gz 2024-06-11T19:27:20.561 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46266.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.46266.log.gz 2024-06-11T19:27:20.561 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39912.log 2024-06-11T19:27:20.562 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37140.log 2024-06-11T19:27:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49643.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.49643.log.gz 2024-06-11T19:27:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48614.log 2024-06-11T19:27:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39912.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39912.log.gz 2024-06-11T19:27:20.564 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44727.log 2024-06-11T19:27:20.565 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48614.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.48614.log.gz 2024-06-11T19:27:20.565 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44979.log 2024-06-11T19:27:20.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44727.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.44727.log.gz 2024-06-11T19:27:20.567 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51354.log 2024-06-11T19:27:20.568 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44979.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.44979.log.gz 2024-06-11T19:27:20.569 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51523.log 2024-06-11T19:27:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51354.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51354.log.gz 2024-06-11T19:27:20.570 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51102.log 2024-06-11T19:27:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51523.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51523.log.gz 2024-06-11T19:27:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38778.log 2024-06-11T19:27:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51102.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51102.log.gz 2024-06-11T19:27:20.573 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49055.log 2024-06-11T19:27:20.574 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37500.log 2024-06-11T19:27:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38778.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.38778.log.gz 2024-06-11T19:27:20.575 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39072.log 2024-06-11T19:27:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49055.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49055.log.gz 2024-06-11T19:27:20.576 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48530.log 2024-06-11T19:27:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39072.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39072.log.gz 2024-06-11T19:27:20.578 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37815.log 2024-06-11T19:27:20.579 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48530.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.48530.log.gz 2024-06-11T19:27:20.580 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39492.log 2024-06-11T19:27:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37815.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37815.log.gz 2024-06-11T19:27:20.581 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41896.log 2024-06-11T19:27:20.581 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38470.log 2024-06-11T19:27:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39492.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39492.log.gz 2024-06-11T19:27:20.583 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48446.log 2024-06-11T19:27:20.583 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37500.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37500.log.gz 2024-06-11T19:27:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41896.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41896.log.gz 2024-06-11T19:27:20.585 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42442.log 2024-06-11T19:27:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48446.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.48446.log.gz 2024-06-11T19:27:20.586 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47856.log 2024-06-11T19:27:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42442.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42442.log.gz 2024-06-11T19:27:20.588 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46140.log 2024-06-11T19:27:20.589 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47856.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47856.log.gz 2024-06-11T19:27:20.589 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40174.log 2024-06-11T19:27:20.590 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36498.log 2024-06-11T19:27:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46140.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.46140.log.gz 2024-06-11T19:27:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43023.log 2024-06-11T19:27:20.592 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40174.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.40174.log.gz 2024-06-11T19:27:20.593 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46730.log 2024-06-11T19:27:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43023.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43023.log.gz 2024-06-11T19:27:20.594 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48192.log 2024-06-11T19:27:20.595 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46730.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46730.log.gz 2024-06-11T19:27:20.596 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46814.log 2024-06-11T19:27:20.596 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38470.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.37140.log.gz 2024-06-11T19:27:20.598 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48192.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.48192.log.gz 2024-06-11T19:27:20.598 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47066.log 2024-06-11T19:27:20.598 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46814.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46814.log.gz 2024-06-11T19:27:20.599 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43611.log 2024-06-11T19:27:20.600 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47066.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.47066.log.gz 2024-06-11T19:27:20.600 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46604.log 2024-06-11T19:27:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43611.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.43611.log.gz 2024-06-11T19:27:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37632.log 2024-06-11T19:27:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46604.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46604.log.gz 2024-06-11T19:27:20.604 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37812.log 2024-06-11T19:27:20.605 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37295.log 2024-06-11T19:27:20.605 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36498.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.38470.log.gz 2024-06-11T19:27:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37632.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37632.log.gz 2024-06-11T19:27:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41812.log 2024-06-11T19:27:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37812.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37812.log.gz 2024-06-11T19:27:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44433.log 2024-06-11T19:27:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41812.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.41812.log.gz 2024-06-11T19:27:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48572.log 2024-06-11T19:27:20.610 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44433.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44433.log.gz 2024-06-11T19:27:20.610 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49265.log 2024-06-11T19:27:20.611 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48572.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.48572.log.gz 2024-06-11T19:27:20.611 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45759.log 2024-06-11T19:27:20.612 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49265.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49265.log.gz 2024-06-11T19:27:20.613 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38904.log 2024-06-11T19:27:20.614 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45759.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45759.log.gz 2024-06-11T19:27:20.615 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38057.log 2024-06-11T19:27:20.615 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38904.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.38904.log.gz 2024-06-11T19:27:20.616 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48402.log 2024-06-11T19:27:20.618 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38057.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38057.log.gz 2024-06-11T19:27:20.618 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37548.log 2024-06-11T19:27:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48402.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.48402.log.gz 2024-06-11T19:27:20.620 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46940.log 2024-06-11T19:27:20.621 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37548.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37548.log.gz 2024-06-11T19:27:20.621 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51607.log 2024-06-11T19:27:20.622 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37556.log 2024-06-11T19:27:20.624 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46940.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46940.log.gz 2024-06-11T19:27:20.624 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47940.log 2024-06-11T19:27:20.625 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51607.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51607.log.gz 2024-06-11T19:27:20.625 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43107.log 2024-06-11T19:27:20.626 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37295.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.37295.log.gz 2024-06-11T19:27:20.627 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47940.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47940.log.gz 2024-06-11T19:27:20.627 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39408.log 2024-06-11T19:27:20.627 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43107.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.43107.log.gz 2024-06-11T19:27:20.628 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50892.log 2024-06-11T19:27:20.629 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39408.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.39408.log.gz 2024-06-11T19:27:20.630 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41274.log 2024-06-11T19:27:20.631 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50892.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50892.log.gz 2024-06-11T19:27:20.631 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51692.log 2024-06-11T19:27:20.632 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41274.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41274.log.gz 2024-06-11T19:27:20.633 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50934.log 2024-06-11T19:27:20.634 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33462.log 2024-06-11T19:27:20.636 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51692.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51692.log.gz 2024-06-11T19:27:20.636 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49685.log 2024-06-11T19:27:20.636 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50934.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50934.log.gz 2024-06-11T19:27:20.636 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40642.log 2024-06-11T19:27:20.637 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49685.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.49685.log.gz 2024-06-11T19:27:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40122.log 2024-06-11T19:27:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40642.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-06-11T19:27:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39450.log 2024-06-11T19:27:20.640 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40122.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40122.log.gz 2024-06-11T19:27:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37408.log 2024-06-11T19:27:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39450.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39450.log.gz 2024-06-11T19:27:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41476.log 2024-06-11T19:27:20.643 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37845.log 2024-06-11T19:27:20.645 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37408.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37408.log.gz 2024-06-11T19:27:20.646 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49391.log 2024-06-11T19:27:20.646 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41476.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41476.log.gz 2024-06-11T19:27:20.646 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46014.log 2024-06-11T19:27:20.647 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49391.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.49391.log.gz 2024-06-11T19:27:20.647 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50147.log 2024-06-11T19:27:20.649 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46014.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.46014.log.gz 2024-06-11T19:27:20.649 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43233.log 2024-06-11T19:27:20.650 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50147.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50147.log.gz 2024-06-11T19:27:20.650 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43821.log 2024-06-11T19:27:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43233.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.43233.log.gz 2024-06-11T19:27:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40980.log 2024-06-11T19:27:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43821.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43821.log.gz 2024-06-11T19:27:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48488.log 2024-06-11T19:27:20.655 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.33462.log: 90.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37753.log 2024-06-11T19:27:20.655 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37845.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33462.log.gz 2024-06-11T19:27:20.656 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37556.log.gz 2024-06-11T19:27:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40980.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.40980.log.gz 2024-06-11T19:27:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50640.log 2024-06-11T19:27:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48488.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48488.log.gz 2024-06-11T19:27:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37590.log 2024-06-11T19:27:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50640.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.50640.log.gz 2024-06-11T19:27:20.660 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45399.log 2024-06-11T19:27:20.661 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37590.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37590.log.gz 2024-06-11T19:27:20.661 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48886.log 2024-06-11T19:27:20.662 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45399.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.45399.log.gz 2024-06-11T19:27:20.662 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40472.log 2024-06-11T19:27:20.663 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38468.log 2024-06-11T19:27:20.664 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48886.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.48886.log.gz 2024-06-11T19:27:20.664 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41064.log 2024-06-11T19:27:20.666 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40472.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.40472.log.gz 2024-06-11T19:27:20.666 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38652.log 2024-06-11T19:27:20.667 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41064.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.41064.log.gz 2024-06-11T19:27:20.667 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47604.log 2024-06-11T19:27:20.668 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38652.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38652.log.gz 2024-06-11T19:27:20.669 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44643.log 2024-06-11T19:27:20.670 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47604.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47604.log.gz 2024-06-11T19:27:20.670 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41022.log 2024-06-11T19:27:20.671 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44643.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.44643.log.gz 2024-06-11T19:27:20.672 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38820.log 2024-06-11T19:27:20.673 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37753.log: 90.6%gzip -5 --verbose -- /var/log/ceph/ceph-mon.f.log 2024-06-11T19:27:20.673 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37845.log.gz 2024-06-11T19:27:20.675 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41022.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.41022.log.gz 2024-06-11T19:27:20.675 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40896.log 2024-06-11T19:27:20.675 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38820.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38820.log.gz 2024-06-11T19:27:20.675 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43443.log 2024-06-11T19:27:20.677 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40896.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40896.log.gz 2024-06-11T19:27:20.677 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45147.log 2024-06-11T19:27:20.678 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43443.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.43443.log.gz 2024-06-11T19:27:20.678 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45695.log 2024-06-11T19:27:20.680 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45147.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.45147.log.gz 2024-06-11T19:27:20.680 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43359.log 2024-06-11T19:27:20.680 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45695.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.45695.log.gz 2024-06-11T19:27:20.681 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49979.log 2024-06-11T19:27:20.682 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43359.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43359.log.gz 2024-06-11T19:27:20.683 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38468.log: 92.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37168.log 2024-06-11T19:27:20.684 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38066.log 2024-06-11T19:27:20.684 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49979.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.49979.log.gz 2024-06-11T19:27:20.685 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40938.log 2024-06-11T19:27:20.686 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38066.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38066.log.gz 2024-06-11T19:27:20.686 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51270.log 2024-06-11T19:27:20.687 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mon.f.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.38468.log.gz 2024-06-11T19:27:20.689 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40938.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40938.log.gz 2024-06-11T19:27:20.689 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47730.log 2024-06-11T19:27:20.689 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51270.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51270.log.gz 2024-06-11T19:27:20.690 INFO:teuthology.orchestra.run.smithi117.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.36498.log.gz 2024-06-11T19:27:20.691 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51312.log 2024-06-11T19:27:20.692 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47730.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.47730.log.gz 2024-06-11T19:27:20.692 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39282.log 2024-06-11T19:27:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51312.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51312.log.gz 2024-06-11T19:27:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49517.log 2024-06-11T19:27:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39282.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39282.log.gz 2024-06-11T19:27:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48108.log 2024-06-11T19:27:20.695 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49517.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.49517.log.gz 2024-06-11T19:27:20.696 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37753.log.gz 2024-06-11T19:27:20.697 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41354.log 2024-06-11T19:27:20.698 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48108.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48108.log.gz 2024-06-11T19:27:20.698 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48842.log 2024-06-11T19:27:20.699 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41354.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41354.log.gz 2024-06-11T19:27:20.699 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42358.log 2024-06-11T19:27:20.700 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48842.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48842.log.gz 2024-06-11T19:27:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-06-11T19:27:20.701 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37502.log 2024-06-11T19:27:20.702 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42358.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42358.log.gz 2024-06-11T19:27:20.703 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48066.log 2024-06-11T19:27:20.708 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38156.log 2024-06-11T19:27:20.715 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37502.log: 89.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37853.log 2024-06-11T19:27:20.715 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37502.log.gz 2024-06-11T19:27:20.722 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38156.log: 94.8% 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38155.log 2024-06-11T19:27:20.722 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38156.log.gz 2024-06-11T19:27:20.723 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42316.log 2024-06-11T19:27:20.724 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48066.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.48066.log.gz 2024-06-11T19:27:20.729 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37853.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36537.log 2024-06-11T19:27:20.730 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37853.log.gz 2024-06-11T19:27:20.730 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46394.log 2024-06-11T19:27:20.730 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42316.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42316.log.gz 2024-06-11T19:27:20.733 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38155.log: -- replaced with /var/log/ceph/ceph-client.admin.36810.log.gz 2024-06-11T19:27:20.736 INFO:teuthology.orchestra.run.smithi117.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.38155.log.gz 2024-06-11T19:27:20.738 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46898.log 2024-06-11T19:27:20.739 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46394.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.46394.log.gz 2024-06-11T19:27:20.744 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33422.log 2024-06-11T19:27:20.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37562.log 2024-06-11T19:27:20.756 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.33422.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33422.log.gz 2024-06-11T19:27:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42526.log 2024-06-11T19:27:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46898.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46898.log.gz 2024-06-11T19:27:20.764 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36564.log 2024-06-11T19:27:20.765 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37303.log 2024-06-11T19:27:20.766 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42526.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42526.log.gz 2024-06-11T19:27:20.770 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37562.log: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.36537.log.gz 2024-06-11T19:27:20.777 INFO:teuthology.orchestra.run.smithi117.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37562.log.gz 2024-06-11T19:27:20.780 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38610.log 2024-06-11T19:27:20.781 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37303.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37303.log.gz 2024-06-11T19:27:20.782 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36480.log 2024-06-11T19:27:20.789 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50766.log 2024-06-11T19:27:20.790 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38610.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.38610.log.gz 2024-06-11T19:27:20.792 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36516.log 2024-06-11T19:27:20.796 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36480.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.37168.log.gz 2024-06-11T19:27:20.800 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37011.log 2024-06-11T19:27:20.808 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48360.log 2024-06-11T19:27:20.809 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.50766.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.50766.log.gz 2024-06-11T19:27:20.810 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33342.log 2024-06-11T19:27:20.819 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44475.log 2024-06-11T19:27:20.819 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48360.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48360.log.gz 2024-06-11T19:27:20.820 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37011.log: 91.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37278.log 2024-06-11T19:27:20.821 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.33342.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.36516.log.gz 2024-06-11T19:27:20.829 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37011.log.gz 2024-06-11T19:27:20.832 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33513.log 2024-06-11T19:27:20.832 INFO:teuthology.orchestra.run.smithi117.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33342.log.gz 2024-06-11T19:27:20.835 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37278.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.37278.log.gz 2024-06-11T19:27:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49811.log 2024-06-11T19:27:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44475.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2024-06-11T19:27:20.852 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47772.log 2024-06-11T19:27:20.853 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49811.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.49811.log.gz 2024-06-11T19:27:20.854 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.33513.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33513.log.gz 2024-06-11T19:27:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47814.log 2024-06-11T19:27:20.862 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47772.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.47772.log.gz 2024-06-11T19:27:20.870 INFO:teuthology.orchestra.run.smithi117.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.36480.log.gz 2024-06-11T19:27:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45525.log 2024-06-11T19:27:20.878 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.47814.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47814.log.gz 2024-06-11T19:27:20.880 INFO:teuthology.orchestra.run.smithi117.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.admin.36564.log.gz 2024-06-11T19:27:20.886 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41602.log 2024-06-11T19:27:20.887 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45525.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45525.log.gz 2024-06-11T19:27:20.903 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42190.log 2024-06-11T19:27:20.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41602.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41602.log.gz 2024-06-11T19:27:20.912 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49559.log 2024-06-11T19:27:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42190.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42190.log.gz 2024-06-11T19:27:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49349.log 2024-06-11T19:27:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49559.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49559.log.gz 2024-06-11T19:27:20.940 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41854.log 2024-06-11T19:27:20.949 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.49349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51565.log 2024-06-11T19:27:20.953 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41854.log: 88.4% 90.0% -- replaced with /var/log/ceph/ceph-client.admin.49349.log.gz 2024-06-11T19:27:20.958 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42064.log 2024-06-11T19:27:20.962 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.51565.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41854.log.gz 2024-06-11T19:27:20.967 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33527.log 2024-06-11T19:27:20.972 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42064.log: -- replaced with /var/log/ceph/ceph-client.admin.51565.log.gz 2024-06-11T19:27:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44517.log 2024-06-11T19:27:20.980 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33527.log: 88.3% 90.1% -- replaced with /var/log/ceph/ceph-client.admin.42064.log.gz 2024-06-11T19:27:20.985 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44391.log 2024-06-11T19:27:20.990 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44517.log: -- replaced with /var/log/ceph/ceph-client.admin.33527.log.gz 2024-06-11T19:27:20.990 INFO:teuthology.orchestra.run.smithi117.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-06-11T19:27:20.994 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48970.log 2024-06-11T19:27:20.998 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44391.log: 89.0% 90.1% -- replaced with /var/log/ceph/ceph-client.admin.44517.log.gz 2024-06-11T19:27:21.005 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.48970.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48970.log.gz 2024-06-11T19:27:21.008 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44391.log.gz 2024-06-11T19:27:21.031 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.e.log.gz 2024-06-11T19:27:21.032 INFO:teuthology.orchestra.run.smithi098.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.d.log.gz 2024-06-11T19:27:21.238 INFO:teuthology.orchestra.run.smithi117.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.36854.log.gz 2024-06-11T19:27:21.255 INFO:teuthology.orchestra.run.smithi117.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.g.log.gz 2024-06-11T19:27:21.378 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-06-11T19:27:21.571 INFO:teuthology.orchestra.run.smithi117.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.i.log.gz 2024-06-11T19:27:21.755 INFO:teuthology.orchestra.run.smithi098.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-06-11T19:27:21.769 INFO:teuthology.orchestra.run.smithi117.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.h.log.gz 2024-06-11T19:27:22.055 INFO:teuthology.orchestra.run.smithi098.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-06-11T19:27:22.153 INFO:teuthology.orchestra.run.smithi117.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.36613.log.gz 2024-06-11T19:27:22.518 INFO:teuthology.orchestra.run.smithi117.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.f.log.gz 2024-06-11T19:27:27.282 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-06-11T19:27:27.425 INFO:teuthology.orchestra.run.smithi117.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-06-11T19:27:27.465 INFO:teuthology.orchestra.run.smithi117.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-06-11T19:27:27.739 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-06-11T19:27:27.778 INFO:teuthology.orchestra.run.smithi117.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-06-11T19:27:27.781 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-06-11T19:27:27.781 INFO:teuthology.orchestra.run.smithi117.stderr:real 0m7.432s 2024-06-11T19:27:27.781 INFO:teuthology.orchestra.run.smithi117.stderr:user 0m28.031s 2024-06-11T19:27:27.781 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m0.945s 2024-06-11T19:27:28.382 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-06-11T19:27:28.384 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-06-11T19:27:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m8.059s 2024-06-11T19:27:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m26.647s 2024-06-11T19:27:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m1.117s 2024-06-11T19:27:28.385 INFO:tasks.ceph:Archiving logs... 2024-06-11T19:27:28.385 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/remote/smithi098/log 2024-06-11T19:27:28.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-11T19:27:32.437 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/remote/smithi117/log 2024-06-11T19:27:32.438 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-11T19:27:36.338 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-11T19:27:36.348 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-11T19:27:36.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-11T19:27:36.351 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-11T19:27:36.426 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-11T19:27:36.436 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-11T19:27:36.436 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-11T19:27:36.439 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-11T19:27:36.453 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-06-11T19:27:36.454 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-06-11T19:27:36.458 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-11T19:27:36.458 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-06-11T19:27:36.458 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv01.front.sepia.ceph.com 3 6 377 44 +451us[ +458us] +/- 43ms 2024-06-11T19:27:36.458 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 41 -93us[ -93us] +/- 88ms 2024-06-11T19:27:36.458 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -793us[ -793us] +/- 87ms 2024-06-11T19:27:36.458 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-11T19:27:36.460 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-11T19:27:36.460 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-06-11T19:27:36.460 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv01.front.sepia.ceph.com 3 6 377 39 +447us[ +453us] +/- 43ms 2024-06-11T19:27:36.460 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 40 -88us[ -81us] +/- 88ms 2024-06-11T19:27:36.460 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 -789us[ -782us] +/- 87ms 2024-06-11T19:27:36.460 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-11T19:27:36.461 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-11T19:27:36.569 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-11T19:27:36.569 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-11T19:27:36.582 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-11T19:27:36.657 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-11T19:27:36.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-11T19:27:36.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T19:27:36.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-11T19:27:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T19:27:36.782 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-11T19:27:36.794 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-11T19:27:36.809 INFO:teuthology.task.internal:Duration was 1541.729997 seconds 2024-06-11T19:27:36.809 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-11T19:27:36.824 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-11T19:27:36.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-11T19:27:36.826 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-11T19:27:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-11T19:27:36.866 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-11T19:27:37.026 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-11T19:27:37.027 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-06-11T19:27:37.027 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-11T19:27:37.088 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-06-11T19:27:37.088 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-11T19:27:37.111 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-11T19:27:37.111 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-11T19:27:37.132 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-11T19:27:37.176 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-11T19:27:37.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-11T19:27:37.197 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-11T19:27:37.313 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-11T19:27:37.323 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-11T19:27:37.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-11T19:27:37.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-11T19:27:37.372 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-11T19:27:37.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-11T19:27:37.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-11T19:27:37.411 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-06-11T19:27:37.436 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-06-11T19:27:37.474 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-11T19:27:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T19:27:37.488 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-11T19:27:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T19:27:37.528 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-11T19:27:37.538 INFO:teuthology.task.internal:Transferring archived files... 2024-06-11T19:27:37.539 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/remote/smithi098 2024-06-11T19:27:37.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-11T19:27:37.573 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-06-10_22:21:08-rados-wip-lflores-testing-2-2024-06-10-1925-distro-default-smithi/7749861/remote/smithi117 2024-06-11T19:27:37.573 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-11T19:27:37.627 INFO:teuthology.task.internal:Removing archive directory... 2024-06-11T19:27:37.627 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-11T19:27:37.629 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-11T19:27:37.657 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-11T19:27:37.668 INFO:teuthology.task.internal:Not uploading archives. 2024-06-11T19:27:37.668 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-11T19:27:37.681 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-11T19:27:37.681 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-11T19:27:37.684 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-11T19:27:37.697 INFO:teuthology.orchestra.run.smithi098.stdout: 265430 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 11 19:27 /home/ubuntu/cephtest 2024-06-11T19:27:37.712 INFO:teuthology.orchestra.run.smithi117.stdout: 265304 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 11 19:27 /home/ubuntu/cephtest 2024-06-11T19:27:37.712 INFO:teuthology.orchestra.run.smithi117.stdout: 403682 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 11 19:09 /home/ubuntu/cephtest/mnt.0 2024-06-11T19:27:37.712 INFO:teuthology.orchestra.run.smithi117.stdout: 403683 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0 2024-06-11T19:27:37.712 INFO:teuthology.orchestra.run.smithi117.stdout: 418170 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-11T19:27:37.712 INFO:teuthology.orchestra.run.smithi117.stdout: 418176 92 -rw-r--r-- 1 ubuntu ubuntu 92565 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_asio.log 2024-06-11T19:27:37.712 INFO:teuthology.orchestra.run.smithi117.stdout: 418181 4 -rw-r--r-- 1 ubuntu ubuntu 1241 Jun 11 19:11 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_pool.log 2024-06-11T19:27:37.713 INFO:teuthology.orchestra.run.smithi117.stdout: 418204 4 -rw-r--r-- 1 ubuntu ubuntu 621 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_list.log 2024-06-11T19:27:37.713 INFO:teuthology.orchestra.run.smithi117.stdout: 418180 4 -rw-r--r-- 1 ubuntu ubuntu 2117 Jun 11 19:11 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_misc.log 2024-06-11T19:27:37.713 INFO:teuthology.orchestra.run.smithi117.stdout: 418173 8 -rw-r--r-- 1 ubuntu ubuntu 5852 Jun 11 19:16 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_aio_pp.log 2024-06-11T19:27:37.713 INFO:teuthology.orchestra.run.smithi117.stdout: 418202 4 -rw-r--r-- 1 ubuntu ubuntu 579 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_cmd.log 2024-06-11T19:27:37.713 INFO:teuthology.orchestra.run.smithi117.stdout: 418194 4 -rw-r--r-- 1 ubuntu ubuntu 712 Jun 11 19:11 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_service_pp.log 2024-06-11T19:27:37.713 INFO:teuthology.orchestra.run.smithi117.stdout: 418203 4 -rw-r--r-- 1 ubuntu ubuntu 515 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_ec_io.log 2024-06-11T19:27:37.713 INFO:teuthology.orchestra.run.smithi117.stdout: 418182 4 -rw-r--r-- 1 ubuntu ubuntu 2063 Jun 11 19:11 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_snapshots.log 2024-06-11T19:27:37.713 INFO:teuthology.orchestra.run.smithi117.stdout: 418186 4 -rw-r--r-- 1 ubuntu ubuntu 1150 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_stat.log 2024-06-11T19:27:37.714 INFO:teuthology.orchestra.run.smithi117.stdout: 418208 4 -rw-r--r-- 1 ubuntu ubuntu 1568 Jun 11 19:11 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_read_operations.log 2024-06-11T19:27:37.714 INFO:teuthology.orchestra.run.smithi117.stdout: 418210 4 -rw-r--r-- 1 ubuntu ubuntu 1690 Jun 11 19:11 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_snapshots.log 2024-06-11T19:27:37.714 INFO:teuthology.orchestra.run.smithi117.stdout: 418206 4 -rw-r--r-- 1 ubuntu ubuntu 1338 Jun 11 19:11 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_misc.log 2024-06-11T19:27:37.714 INFO:teuthology.orchestra.run.smithi117.stdout: 418207 4 -rw-r--r-- 1 ubuntu ubuntu 931 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_pool.log 2024-06-11T19:27:37.714 INFO:teuthology.orchestra.run.smithi117.stdout: 418174 4 -rw-r--r-- 1 ubuntu ubuntu 2528 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_io.log 2024-06-11T19:27:37.714 INFO:teuthology.orchestra.run.smithi117.stdout: 418199 4 -rw-r--r-- 1 ubuntu ubuntu 706 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_cls.log 2024-06-11T19:27:37.714 INFO:teuthology.orchestra.run.smithi117.stdout: 418205 4 -rw-r--r-- 1 ubuntu ubuntu 639 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_ec_list.log 2024-06-11T19:27:37.714 INFO:teuthology.orchestra.run.smithi117.stdout: 418200 4 -rw-r--r-- 1 ubuntu ubuntu 478 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_handler_error.log 2024-06-11T19:27:37.715 INFO:teuthology.orchestra.run.smithi117.stdout: 418211 4 -rw-r--r-- 1 ubuntu ubuntu 975 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_write_operations.log 2024-06-11T19:27:37.715 INFO:teuthology.orchestra.run.smithi117.stdout: 418189 4 -rw-r--r-- 1 ubuntu ubuntu 587 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cmd_pp.log 2024-06-11T19:27:37.715 INFO:teuthology.orchestra.run.smithi117.stdout: 418201 4 -rw-r--r-- 1 ubuntu ubuntu 1516 Jun 11 19:11 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_io.log 2024-06-11T19:27:37.715 INFO:teuthology.orchestra.run.smithi117.stdout: 418177 4 -rw-r--r-- 1 ubuntu ubuntu 1848 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_lock.log 2024-06-11T19:27:37.715 INFO:teuthology.orchestra.run.smithi117.stdout: 418193 4 -rw-r--r-- 1 ubuntu ubuntu 1196 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_c_write_operations.log 2024-06-11T19:27:37.715 INFO:teuthology.orchestra.run.smithi117.stdout: 418185 4 -rw-r--r-- 1 ubuntu ubuntu 3781 Jun 11 19:12 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_snapshots_pp.log 2024-06-11T19:27:37.715 INFO:teuthology.orchestra.run.smithi117.stdout: 418192 4 -rw-r--r-- 1 ubuntu ubuntu 2744 Jun 11 19:11 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_service.log 2024-06-11T19:27:37.715 INFO:teuthology.orchestra.run.smithi117.stdout: 418179 4 -rw-r--r-- 1 ubuntu ubuntu 1993 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_lock_pp.log 2024-06-11T19:27:37.716 INFO:teuthology.orchestra.run.smithi117.stdout: 418195 4 -rw-r--r-- 1 ubuntu ubuntu 1680 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_c_read_operations.log 2024-06-11T19:27:37.716 INFO:teuthology.orchestra.run.smithi117.stdout: 418178 8 -rw-r--r-- 1 ubuntu ubuntu 7018 Jun 11 19:13 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_list.log 2024-06-11T19:27:37.716 INFO:teuthology.orchestra.run.smithi117.stdout: 418175 8 -rw-r--r-- 1 ubuntu ubuntu 5624 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_io_pp.log 2024-06-11T19:27:37.717 INFO:teuthology.orchestra.run.smithi117.stdout: 418183 8 -rw-r--r-- 1 ubuntu ubuntu 4225 Jun 11 19:12 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_misc_pp.log 2024-06-11T19:27:37.717 INFO:teuthology.orchestra.run.smithi117.stdout: 418190 4 -rw-r--r-- 1 ubuntu ubuntu 2700 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_watch_notify.log 2024-06-11T19:27:37.718 INFO:teuthology.orchestra.run.smithi117.stdout: 418209 4 -rw-r--r-- 1 ubuntu ubuntu 800 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_watch_notify.log 2024-06-11T19:27:37.718 INFO:teuthology.orchestra.run.smithi117.stdout: 418188 8 -rw-r--r-- 1 ubuntu ubuntu 4112 Jun 11 19:11 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_watch_notify_pp.log 2024-06-11T19:27:37.718 INFO:teuthology.orchestra.run.smithi117.stdout: 418187 4 -rw-r--r-- 1 ubuntu ubuntu 1253 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_stat_pp.log 2024-06-11T19:27:37.718 INFO:teuthology.orchestra.run.smithi117.stdout: 418191 28 -rw-r--r-- 1 ubuntu ubuntu 25071 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cmd.log 2024-06-11T19:27:37.718 INFO:teuthology.orchestra.run.smithi117.stdout: 418184 20 -rw-r--r-- 1 ubuntu ubuntu 19642 Jun 11 19:26 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_tier_pp.log 2024-06-11T19:27:37.718 INFO:teuthology.orchestra.run.smithi117.stdout: 418197 4 -rw-r--r-- 1 ubuntu ubuntu 1029 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_open_pools_parallel.log 2024-06-11T19:27:37.718 INFO:teuthology.orchestra.run.smithi117.stdout: 418196 8 -rw-r--r-- 1 ubuntu ubuntu 5517 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_list_parallel.log 2024-06-11T19:27:37.718 INFO:teuthology.orchestra.run.smithi117.stdout: 418172 4 -rw-r--r-- 1 ubuntu ubuntu 4036 Jun 11 19:14 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_aio.log 2024-06-11T19:27:37.719 INFO:teuthology.orchestra.run.smithi117.stdout: 418198 4 -rw-r--r-- 1 ubuntu ubuntu 1295 Jun 11 19:10 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_delete_pools_parallel.log 2024-06-11T19:27:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-11T19:27:37.720 INFO:teuthology.orchestra.run.smithi117.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-06-11T19:27:37.720 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi117 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-11T19:27:37.728 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-11T19:27:37.743 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-11T19:27:37.759 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi117 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-11T19:27:37.762 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} duration: 1541.7299971580505 failure_reason: 'Unit test failed (workunit test rados/test.sh) on smithi117 with status 1: ''(total 4 failed) FAILURE: Test `CrcZeroWrite` of `LibRadosIoECPP`. Reason: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-4216-g37bbe1d8/rpm/el9/BUILD/ceph-19.0....''' flavor: default owner: scheduled_lflores@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=525c3dbf9f57437b9f48582eef8032aa status: fail success: false 2024-06-11T19:27:37.762 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-11T19:27:37.870 INFO:teuthology.run:FAIL